JP3907838B2 - Voice conversion device and voice conversion method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To execute natural speech conversion even if target speeches do not exist in a corresponding play portion in the case where the generating source of the speeches is entirely different like in the case where the tones of a musical instrument are converted to resemble the previously determined singing of a singer. SOLUTION: The sinusoidal wave components and residual components of the input speeches are deformed in accordance with the sinusoidal wave component and residual components extracted from the input speeches and the target sinusoidal wave components and target residual components extracted from the target speeches and the converted speeches are formed by synthesizing the components. In this case, when the portions, which are the portions of the target voices and are required to be made correspondent to the input speeches, are silent portions, the sinusoidal wave components and residual components of the input speeches are deformed in accordance with the preset silent part sinusoidal wave components and silent part residual components in place of the target sinusoidal wave components and target residual components and, therefore, the natural speech conversion may be executed even when the generating source of the speech is entirely heterogeneous and the target speeches do not exist in the corresponding playing portions.

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a voice conversion device and a voice conversion method, and in particular, converts a sound of a musical instrument to resemble a predetermined singer's singing voice or converts a singer's singing voice to resemble a predetermined musical instrument's sound. In particular, the present invention relates to a voice conversion device and a voice conversion method capable of performing voice conversion even when a voice generation source is completely different.
[0002]
[Prior art]
Various voice conversion devices that change the frequency characteristics of the input voice and the like have been developed. For example, in a karaoke device, the pitch of a singer's singing voice is converted to convert a male voice into a female voice. There are those that convert to voice or vice versa (see, for example, US Pat. No. 5,567,901 and Japanese Patent Publication No. 8-508581).
[0003]
[Problems to be solved by the invention]
In the above conventional voice conversion device, voice is converted by pitch conversion, but voice quality itself is not converted.
By the way, for example, if there is a function to convert the input instrument sound to resemble someone's singing voice, it will be possible to output the singing voice by playing the instrument, which is very interesting for karaoke devices etc. It is.
However, it has been difficult to perform such processing with a conventional audio conversion device.
[0004]
By the way, when such a voice conversion device is to be realized as a karaoke device, when providing a function of converting the input instrument sound into someone's singing voice, someone's singing voice (target voice) is set in advance. A configuration may be considered in which voice conversion of an instrument sound input based on the target voice is performed.
When such a configuration is adopted, if an instrument sound is input in a portion where the target sound does not exist, such as a prelude part, an interlude part or a follower part of the music corresponding to the target sound, the sound is Conversion may not be possible.
[0005]
Therefore, an object of the present invention is to change the sound of an instrument to resemble a predetermined singer's singing voice (= target sound), or to resemble a singer's singing voice to a predetermined instrument's sound (= target sound). Even if the sound source is completely different as in the case of conversion, it is possible to easily obtain a more natural converted sound, and even if the target sound does not exist in the corresponding performance part, the natural sound An object of the present invention is to provide an audio conversion device and an audio conversion method capable of performing conversion.
[0006]
[Means for Solving the Problems]
  In order to solve the above problem, the configuration of claim 1 is:Compatible with instrument soundsComponent extraction means for extracting a sine wave component and a residual component from the input audio signal;Corresponds to singing voiceTarget sine wave component and target residual component extracted from target speech signalThe sine wave component is transformed to generate a new sine wave component, and the residual component is transformed to generate a new residual component.New component generation means;Generated by the new component generation meansA synthesized voice signal generating means for synthesizing the new sine wave component and the new residual component to generate and output a converted voice signal; and when generating the converted voice signal, the target voice signal part, Silencer detection means for detecting whether or not the part to be associated with the input audio signal is a silence part, and when the part to be associated is a silence part, a silent part sine wave component and a silence part remaining Silent part component output means for outputting a difference component, the new component generation means,When the part to be matched is a silent part, the sine wave component is transformed based on the silent part sine wave component to generate a new sine wave component, and based on the silent part residual component While the residual component is deformed to generate a new residual component, and the portion to be matched is not a silent portion, the new sine wave component is modified by deforming the sine wave component based on the target sine wave component And generating a new residual component by modifying the residual component based on the target residual componentIt is characterized by that.
[0007]
  The configuration of claim 2 is:Component extraction means for extracting a sine wave component and a residual component from an input voice signal corresponding to a singing voice, a target sine wave component and a target residual component extracted from a target voice signal corresponding to a musical instrument sound, and the sine wave component New component generation means for generating a new sine wave component by deformation, generating a new residual component by changing the residual component, and the new sine wave component and the new residual generated by the new component generation means A converted voice signal generating means for synthesizing components and generating and outputting a converted voice signal; and when generating the converted voice signal, a portion of the target voice signal that should correspond to the input voice signal is silent. A silent part detecting means for detecting whether or not the part is a silent part, and outputting a preset silent part sine wave component and silent part residual component when the corresponding part is a silent part Silent part component output means, and when the part to be associated is a silent part, the new component generation means transforms the sine wave component based on the silent part sine wave component to generate a new sine wave And generating a new residual component by transforming the residual component based on the silent part residual component, and when the portion to be associated is not a silent part, the target sine wave The sine wave component is deformed based on the component to generate a new sine wave component, and the residual component is deformed based on the target residual component to generate a new residual component.It is characterized by that.
[0008]
According to a third aspect of the present invention, in the configuration according to the first or second aspect of the present invention, the structure includes a silent detection unit that detects whether or not the input voice signal is an unvoiced sound, and the input voice signal is detected by the silent detection unit. Is detected as an unvoiced sound, the target sound signal is output as the converted sound signal.
[0009]
  The configuration of claim 4 is:Compatible with instrument soundsA component extraction step of extracting a sine wave component and a residual component from the input audio signal;Deforming the sine wave component to generate a new sine wave component;The residual component is transformedTheA new component generating step for generating a new residual component, a converted speech generating step for generating a converted speech by synthesizing the new sine wave component and the new residual component, and a target speech for generating the converted speech A silent part detecting step for detecting whether or not the part corresponding to the input voice is a silent part.WhenThe novel component generation step comprisesWhen the part to be matched is a silent part, the sine wave component is transformed based on a preset silent part sine wave component to generate a new sine wave component, and a preset silent part residual component Based on the target sine wave component extracted from the target voice corresponding to the singing voice, when the part to be matched is not a silent part, The sine wave component is transformed to generate a new sine wave component, and the residual component is transformed based on the target residual component extracted from the target voice corresponding to the singing voice to generate a new residual component.It is characterized by that.
[0010]
  The configuration according to claim 5 is:A component extraction step for extracting a sine wave component and a residual component from an input voice signal corresponding to a singing voice, a new sine wave component is generated by modifying the sine wave component, and a new residual is generated by modifying the residual component A new component generating step for generating a component, a converted speech generating step for generating a converted speech by synthesizing the new sine wave component and the new residual component, and a portion of the target speech in generating the converted speech. And a silent part detecting step for detecting whether or not the part to be associated with the input voice is a silent part, and the new component generating step is performed when the part to be associated is a silent part. The sine wave component is modified based on a preset silent part sine wave component to generate a new sine wave component, and the residual component is modified based on a preset silent part residual component to generate a new residual. Difference component On the other hand, if the portion to be matched is not a silent portion, a new sine wave component is generated by modifying the sine wave component based on the target sine wave component extracted from the target sound corresponding to the instrument sound. And generating a new residual component by modifying the residual component based on the target residual component extracted from the target speech corresponding to the instrument sound.It is characterized by that.
[0011]
According to a sixth aspect of the present invention, in the configuration of the fourth or fifth aspect, the method further includes a silent detection step of detecting whether or not the input voice is an unvoiced sound, and the input voice is an unvoiced sound in the silent detection step. When it is detected that there is, the target voice is output as the converted voice.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
Next, preferred embodiments of the present invention will be described with reference to the drawings.
[1] Outline processing of embodiment
First, an outline process of the embodiment will be described.
FIG. 1 shows a schematic processing flowchart.
First, SMS (Spectral Modeling Synthesis) analysis including FFT (Fast Fourie Transform) is performed on the instrument sound (input audio signal Sv) performed by the performer (me) in real time (step S101), and a sine wave component ( Sine component) is extracted, a residual component (Residual component) is generated from the input speech signal and the sine wave component in units of frames (step S102), and at the same time, it is determined whether the input speech signal is an unvoiced sound. Then, from the extracted sine wave component, pitch (Pitch), amplifier (Amplitude) and spectral shape (Spectral Shape) which are original attribute data are further extracted (step S103).
[0013]
Next, referring to the frame information of the specific singer (Target) corresponding to the frame being processed (target frame information: attribute data, residual component, consonant sound signal, etc.), the specific singer held It is determined whether or not the (Target) singing voice is a silent part (step S104).
If it is determined in step S104 that it is not a silent part (step S104; NO), attribute data (target attribute data = pitch, amplifier, and spectral) of a specific singer (Target) stored (stored) in advance. From the shape, target attribute data of a frame corresponding to the frame of the input audio signal Sv is extracted (step S105).
[0014]
  On the other hand, when it is determined in step S104 that the singing voice of the specific singer (Target) is silent (step S104; YES), the player (me) plays the instrument.do itDespite the input of the audio signal Sv, there is no inconvenience because there is no audio to be converted. Therefore, in the present embodiment, as will be described in detail later, silence part conversion processing is performed (step S106), and preset silence part data (silence part attribute data and silence part residual component) are converted into target attributes. It will be used as data or target residual component.
[0015]
Subsequently, it is determined whether or not the input voice is an unvoiced sound (step S107).
If it is determined in step S107 that the input sound is not an unvoiced sound (step S107; NO), since the input sound signal Sv is a voiced sound, a sound conversion process described below is performed (steps S108 to S110). ).
Next, new attribute data (new attribute data = pitch, amplifier, and spectral shape) is generated by appropriately selecting and combining the original attribute data and target attribute data, which are attribute data extracted from the input audio signal Sv. (Step S108).
[0016]
  Based on the new attribute data obtained subsequently, a new sine wave component of the frame is generated (step S109). Further, a new residual component is generated based on the residual component of the specific singer (Target) held from the residual component extracted in step S102 (step S110), and together with the new sine wave component generated in step S109. Inverse FFT is performed to obtain a converted audio signal (step S112). By the way, in the determination of step S107,Input voiceIs determined to be an unvoiced sound (step S107; YES), since the singing voice is a consonant part, the voice conversion process is not performed and the consonant voice signal of the specific singer (Target) is output as it is (step S107). S111). This is because the consonant part becomes unnatural unless data extracted from the singing voice is used.
[0017]
In this way, the converted voice signal generated by the inverse FFT conversion process in step S112 or the voice signal of the specific singer (Target) output in step S111 is synthesized with the musical sound signal that is accompanied by karaoke and output. (Step S113), the process proceeds to Step S101, and the next frame is processed. Thus, in the process of cycling through the processing from step S101 to step S113, the audio signal Sv input by the player (me) playing the instrument is as if the singing voice sung by a specific singer (Target). Is output as an audio signal converted as shown in FIG.
[0018]
[2] Outline configuration of embodiment
FIG. 2 is a block diagram showing a schematic configuration of the speech conversion apparatus according to the embodiment.
In FIG. 2, an SMS (Spectral Modeling Synthesis) analysis unit 101 performs SMS analysis on input speech (for example, instrument sound), and acquires a sine wave component SINme and a residual component Rme (f).
More specifically, in the SMS analysis, the input voice signal is subjected to FFT conversion, a local peak is obtained from the result after the FFT conversion, the pitch Pme-sta is extracted from the obtained local peak, and the local peaks are linked in time series. As a result, the sine wave component SINme (frequency component and amplifier component) is acquired.
[0019]
Further, the residual component Rme (f) is extracted by subtracting the sine wave component SINme from the input audio signal.
That is, the sine wave component SINme is a component that can be expressed by the synthesis of sine waves, and the residual component Rme (f) is a remaining component that cannot be expressed as a sine wave component in the input audio signal.
[0020]
Next, the SMS analysis unit 101 detects an unvoiced sound (consonant: 't', 'k', etc.) according to the magnitude of the high-frequency component at the local peak, or zero crossing per unit time on the time axis. According to the number, it is detected that it is unvoiced (such as 's'), and unvoiced sound information U / Vinf indicating whether it is unvoiced sound is supplied as a switching control signal to be supplied to the switching unit SW3.
These analysis results are output to the silent part detection unit 103 as input voice information INFme that is information related to the input voice signal.
Further, the residual component Rme (f) is output to the residual component processing unit 104.
[0021]
Next, the target data storage unit 102a of the target information storage unit 102 is a voice (hereinafter referred to as a target voice) of a singer (a singer who is a target of imitation: TARGET) subjected to voice conversion subjected to SMS analysis in advance. Various data (sine wave component SINtar, residual component Rtar (f)) are stored. These are output to the silent part detection unit 103 as target information INFtar-sync.
The target silence data storage unit 102b of the target information storage unit 102 includes a prelude part, an interlude part, etc., and is input when the target sound corresponding to the input sound (= instrument sound) is a silent part (SILENT part). Various data (silent part sine wave component SILENTsine, silent part residual component SILENTresidual (f)) prepared in advance for use in voice conversion of the voice signal are stored.
[0022]
  Next, the silent part detecting unit 103 receives the input voice information INFme andTarget information INF tar-syncWhether the target audio signal is a silent part (SILENT part) by determining whether or not the target sine wave component SINtar exists at a position corresponding to the position where the target audio signal corresponding to the input audio signal exists. Is output to the switching units SW1 and SW2 as silence control information S / Pinf indicating whether or not the target voice signal is a silence part. As a result, the switching unit SW1 sends the target residual component Rtar (f) stored in the target data storage unit 102a to the residual component processing unit 104 if the target audio signal is not a silent part according to the silence information S / Pinf. Supply.
[0023]
  If the target speech signal is a silent part, the silent part residual component SILENTresidual (f) stored in the target data storage unit 102b is sent to the residual component processing unit 104 instead of the target residual component Rtar (f). Supply. Further, according to the silence information S / Pinf, the switching unit SW2 converts the target sine wave component SINtar stored in the target data storage unit 102a into a sine wave if it is not a silence part of the target audio signal.Overtone componentIf it is a silent part of the target audio signal that is output to the processing unit 106, the silent part sine wave component SILENTsine stored in the target data storage part 102b is replaced with the sine wave instead of the target sine wave component SINtar.Overtone componentThis is supplied to the processing unit 106.
[0024]
  The residual component processing unit 104 converts the residual component Rme (f) of the input audio signal (= instrument sound) based on the target residual component Rtar (f) or the silent part residual component SILENTresidual (f). Perform difference component transformation and reverse generated new residual component Rnew (f)Fast Fourier transformThe data is output to the processing unit 108. Next, sine waveOvertoneThe component processing unit 106 transforms the sine wave component SINme of the sound signal of the musical instrument sound based on the sine wave component SINtar or the silent part sine wave component SILENTsine of the singer's sound signal to be subjected to sound conversion. To create a new sine wave component SINnewAttribute transformation partIt supplies to 107. The attribute transformation unit 107 performs attribute transformation such as changing the amplitude, pitch, and spectral shape as necessary on the sine wave component SINnew subjected to the voice conversion process, and generates a new sine wave component SINnew ′ generated. ,Inverse fast Fourier transformThe data is supplied to the processing unit 108.
[0025]
  Inverse fast Fourier transformThe processing unit 108 adds the attribute-converted sine wave component SINnew ′ and the residual component Rnew (f) on the frequency axis, and then performs inverse FFT conversion, and supplies the result to the final processing unit 110.Inverse fast Fourier transformThe processing unit 109 synthesizes a signal to be output when the input voice is silent. That is, the target sine wave component SINtar and the target residual component Rtar (f), or the silent part sine wave component SILENTsine and the silent part residual component SILENTresidual (f) are added on the frequency axis, and then subjected to inverse FFT transform, and the target silent The sound is output to the final processing unit 110. If the switching unit SW3 is an unvoiced sound according to the unvoiced sound information U / Vinf supplied from the SMS analysis unit 101, the switching unit SW3 will be described later from the U (Unvoice) side.Inverse fast Fourier transformThe synthesized signal from the processing unit 109 (= unvoiced sound on the target side) is supplied to the final processing unit 110. If it is not unvoiced sound, the V (Voice) sideInverse fast Fourier transformSynthetic sound signal (sinusoidal component SINnew) from the processing unit 108'And residual componentR new (f)And a combined signal) are supplied to the final processing unit 110.
[0026]
The final processing unit 110 synthesizes and outputs the audio signal subjected to the inverse FFT conversion and the music.
As a result, it is possible to convert an input voice such as a musical instrument sound into a natural singing voice. Similarly, it is also possible to convert a singer's singing voice as an input voice into a natural instrument sound.
[0027]
[3] Detailed configuration of the embodiment
Next, the detailed configuration of the embodiment will be described with reference to FIGS.
In this embodiment, the voice conversion device (voice conversion method) according to the present invention is applied to a karaoke device, and is configured as a voice conversion device that can convert an input voice that is a musical instrument sound into a natural singing voice of a singer. This is an example.
3 and 4 are block diagrams showing the detailed configuration of the voice conversion device.
[0028]
[3.1] Operation of SMS analysis unit
[3.1.1] Operation of microphone, analysis window generation unit, and input audio signal extraction unit
In FIG. 3, the microphone 1, the analysis window generation unit 2, the input speech signal extraction unit 3, the fast Fourier transform unit 4, the peak detection unit 5, the unvoiced / voiced detection unit 6, the pitch detection unit 7, the peak cooperation unit 8, and interpolation synthesis. Unit 9, residual component detection unit 10, fast Fourier transform unit 11, residual component holding unit 12, sine wave component holding unit 13, average amplifier calculation unit 14, amplifier normalization unit 15, spectral shape calculation unit 16, pitch The normalization unit 17, the original frame information holding unit 18, and the static change / vibrato change separation unit 19 constitute an SMS analysis unit 101.
First, the microphone 1 collects musical instrument sounds of a musical instrument player (me) who wants to perform voice conversion, and outputs them to the input voice signal cutout unit 3 as an input voice signal Sv.
[0029]
In parallel with this, the analysis window generator 2 generates an analysis window (for example, a hamming window) AW having a fixed period (for example, 3.5 times) of the pitch period detected in the previous frame. And output to the input voice signal cutout unit 3. When the initial state or the previous frame is a silent sound (including silent sound), an analysis window having a preset fixed period is output to the input voice signal cutout unit 3 as an analysis window AW.
As a result, the input voice signal cutout unit 3 multiplies the input analysis window AW and the input voice signal Sv, cuts out the input voice signal Sv in units of frames, and outputs the frame voice signal FSv to the fast Fourier transform unit 4. The
More specifically, the relationship between the input audio signal Sv and the frame is as shown in FIG. 5, and each frame FL is set to partially overlap the previous frame FL.
[0030]
[3.1.2] Operation of fast Fourier transform
Then, the frame audio signal FSv is analyzed in the fast Fourier transform unit 4 and a local peak is detected by the peak detection unit 5 from the frequency spectrum output from the fast Fourier transform unit 4 as shown in FIG. .
More specifically, a local peak marked with x is detected for the frequency spectrum as shown in FIG. This local peak is expressed as a combination of a frequency value and an amplifier (amplitude) value.
That is, as shown in FIG. 6, local peaks are detected and represented for each frame as (F0, A0), (F1, A1), (F2, A2),..., (FN, AN). It will be.
[0031]
  Then, as schematically shown in FIG. 5, each frame is output to the unvoiced / voiced detection unit 6 and the peak link unit 8 as one set (hereinafter referred to as a local peak set). The unvoiced / voiced detection unit 6 detects unvoiced ('t', 'k', etc.) according to the magnitude of the high frequency component based on the input local peak for each frame, and detects unvoiced / voiced. The signal U / Vme is sent to the pitch detector 7,And output to the crossfader 30. Alternatively, it is detected that the voice is unvoiced according to the number of zero crosses per unit time on the time axis (such as “s”), and the original unvoiced / voiced detection signal U / Vme is output to the pitch detector 7.
[0032]
Furthermore, if the unvoiced / voiced detection unit 6 does not detect that the input frame is unvoiced, the unvoiced / voiced detection unit 6 outputs the input local peak set to the pitch detection unit 7 as it is.
Further, unvoiced / voiced information is supplied to the crossfader 30 as U / Vinf.
[0033]
[3.1.3] Operation of pitch detector and peak link unit
The pitch detector 7 detects the pitch Pme of the frame corresponding to the local peak set based on the input local peak set.
As a more specific method of detecting the frame pitch Pme, for example, Maher, RC and J.W. Beauchamp: “Fundamental Frequency Estimation of Musical Signal using a two-way Mismatch Procedure” (Journal of Acounstical Society of America 95 (4) : 2254-2263).
[0034]
Next, the local peak set output from the peak detection unit 5 is determined in the peak linkage unit 8 for linkage between the previous and subsequent frames, and the local peaks recognized to be linked are local peaks so as to form a series of data strings. Cooperation process to connect
[0035]
Here, this cooperation processing will be described with reference to FIG.
Assume that a local peak as shown in FIG. 7A is detected in the previous frame, and a local peak as shown in FIG. 7B is detected in the current frame.
In this case, the peak link unit 8 performs local peaks corresponding to the local peaks (F0, A0), (F1, A1), (F2, A2),..., (FN, AN) detected in the previous frame. It is checked whether or not is detected even in the current frame. Whether or not there is a corresponding local peak is determined by whether or not the local peak of the current frame is detected within a predetermined range centered on the frequency of the local peak detected in the previous frame.
[0036]
More specifically, in the example of FIG. 7, corresponding local peaks are detected for local peaks (F0, A0), (F1, A1), (F2, A2). For FK, AK (see FIG. 7A), the corresponding local peak (see FIG. 7B) is not detected.
When the peak linking unit 8 detects corresponding local peaks, they are connected in time series order and output as a set of data strings. If no corresponding local peak is detected, the data is replaced with data indicating that there is no corresponding local peak for the frame.
Here, FIG. 8 shows an example of changes in the local peak frequency F0 and frequency F1 over a plurality of frames.
[0037]
Such a change is similarly recognized for the amplifiers (amplitudes) A0, A1, A2,. In this case, the data string output from the peak cooperation unit 8 is a discrete value output at every frame interval.
The peak value output from the peak cooperation unit 8 is hereinafter referred to as a deterministic component. This means that the original signal (that is, the audio signal Sv) is a component that is definitely replaced as a sine wave element. Further, each replaced sine wave (strictly speaking, the frequency and amplifier (amplitude) which are parameters of the sine wave) will be referred to as a sine wave component.
[0038]
Next, the interpolation synthesis unit 9 performs an interpolation process on the deterministic component output from the peak cooperation unit 8 and performs waveform synthesis using a so-called oscillator method based on the deterministic component after the interpolation. In this case, the interpolation interval is performed at an interval corresponding to the sampling rate (for example, 44.1 KHz) of the final output signal output from the output unit 34 described later. The solid line shown in FIG. 8 indicates an image when the interpolation processing is performed for the frequencies F0 and F1 of the sine wave component.
[0039]
[3.1.4] Configuration of interpolation / synthesis unit
Here, the configuration of the interpolation / synthesis unit 9 is shown in FIG.
The interpolation / synthesis unit 9 includes a plurality of partial waveform generation units 9a, and each partial waveform generation unit 9a has a frequency (F0, F1,...) And an amplifier (amplitude) of a specified sine wave component. A corresponding sine wave is generated. However, since the sine wave components (F0, A0), (F1, A1), (F2, A2),... In the first embodiment change each time according to the interpolation interval, The waveform output from the partial waveform generator 9a is a waveform according to the change. That is, the sine wave components (F0, A0), (F1, A1), (F2, A2),... Are sequentially output from the peak cooperation unit 8, and interpolation processing is performed for each sine wave component. Each partial waveform generator 9a outputs a waveform whose frequency and amplitude vary within a predetermined frequency region. The waveforms output from the partial waveform generators 9a are added and synthesized in the adder 9b. Therefore, the output signal of the interpolation synthesis unit 9 is a sine wave component synthesis signal SSS obtained by extracting a deterministic component from the input audio signal Sv. These analysis information and a residual signal to be described later are combined and output to the silence detector 21 as input audio signal information INFme which is information relating to the input audio signal.
[0040]
[3.2] Target information storage unit
On the other hand, the target sine wave component SINtar and the target residual component Rtar are analyzed in advance and held in advance in the target data storage unit 102a on the target information holding unit 102 (for example, a hard disk).
In this case, the target sine wave component SINtar includes the average amplifier static component Atar-sta, the average amplifier vibrato component Atar-vib, the pitch static component Ptar-sta, the pitch vibrato component Ptar-vib, and the spectral shape Star. There is (f).
When the voice conversion device of this embodiment is used as a karaoke device, the pitch static component Ptar-sta and the pitch vibrato component Ptar-vib constituting the target sine wave component SINtar when the key is raised or lowered from the reference. Also, correction processing is performed to raise and lower the same amount. For example, when the key is raised by 50 [cent], the pitch static component Ptar-sta and the pitch vibrato component Ptar-vib must also be raised by 50 [cent].
[0041]
Further, when the tempo of the karaoke apparatus is raised or lowered, it is necessary to read out the target sine wave component SINtar and the target residual component Rtar at a timing corresponding to the changed tempo.
In this case, if the target sine wave component SINtar and the target residual component Rtar corresponding to the timing corresponding to the necessary frame do not exist, the targets of the two frames existing at timings before and after the timing of the necessary frame. The sine wave component SINtar and the target residual component Rtar are read out, and interpolation processing is performed by using these two sets of the target sine wave component SINtar and the target residual component Rtar, and the target sine wave component SINtar and the target residual of the frame at the necessary timing. The component Rtar is generated.
[0042]
In this case, the vibrato component (the average amp vibrato component Atar-vib and the pitch vibrato component Ptar-vib) is not suitable as it is because the vibrato cycle itself is changed and is not suitable. It is necessary to perform interpolation processing that does not occur. Alternatively, if the target attribute data is not data representing the trajectory of the vibrato itself but the parameters of the vibrato period and the vibrato depth are held and the actual trajectory is obtained by calculation, this problem can be avoided. Information on these target sounds is output to the silent part detection unit 21 as target information INFtar-sync.
[0043]
[3.3] Silent part detector
When the silent part detector 21 functioning as the silent part detector 103 detects that the target voice corresponding to the input voice is not silent based on the input voice signal information INFme and the target information INFtar-sync, the silent information S By outputting / Pinf as a switching control signal, both the switching unit SW1 and the switching unit SW2 are set to the target data storage unit 20a side.
As a result, the residual component selection unit 25 of the residual component processing unit 104 receives the target residual component Rtar, and the sine wave component attribute data selection unit 23 receives the target sine wave component SINtar. Become.
[0044]
Further, when the silent part detecting unit 21 functioning as the silent part detecting unit 103 detects that the target voice corresponding to the input voice is silent, the silent part detecting part 21 switches by outputting the silent information S / Pinf as a switching control signal. Both the unit SW1 and the switching unit SW2 are switched from the target data storage unit 20a side to the target silent data storage unit 20b side.
As a result, the silence component residual component SILENTresidual (f) is input to the residual component selection unit 25 of the residual component processing unit 104 instead of the target residual component Rtar, and the sine wave component attribute data selection unit 23 The silent sine wave component SILENTsine is input instead of the target sine wave component SINtar.
[0045]
[3.3.1] Detailed operation of silent part detector
Here, a detailed operation when the silent part detecting unit 21 detects that the target voice is a silent part will be described with reference to FIG.
When it is detected that the target voice corresponding to the input voice is silent, the silent part detection unit 21 outputs both the switching unit SW1 and the switching unit SW2 by outputting the silent information S / Pinf as a switching control signal. The target data storage unit 2a side is switched to the target silence data storage unit 2b side.
As a result, the silence component residual component SILENTresidual (f) is input to the residual component selection unit 25 of the residual component processing unit 104 instead of the target residual component Rtar, and the sine wave component attribute data selection unit 23 The silent sine wave component SILENTsine is input instead of the target sine wave component SINtar.
[0046]
Thereby, as shown in FIGS. 10A and 10B, when the target data (= SINtar + Rtar) is not a silent part, for example, when the target data Y1 corresponding to the input audio signal X1 exists, the synthesis is performed. As shown in FIG. 10D, the audio signal is synthesized based on the input audio signal X1 and the target data Y1 (indicated by “X1 * Y1” in the figure). Similarly, when the target data Y3 corresponding to the input audio signal X3 exists, the synthesized audio signal is synthesized based on the input audio signal X3 and the target data Y3 as shown in FIG. (Indicated in the figure by “X3 * Y3”).
Further, as shown in FIGS. 10A and 10B, when the target data (= SINtar + Rtar) is a silent portion, for example, the input audio signal X2 as in the case of the input audio signal X2 or the input audio signal X4. Alternatively, when the target data corresponding to the input voice signal X4 is a silent part, the synthesized voice signal is based on the input voice signal X2 and the target silence data Z (= SILENTsine + SILENTresidual (f); see FIG. 10C). They are synthesized (indicated by X2 * Z in the figure) or synthesized based on the input audio signal X4 and the target silence data Z (indicated by X4 * Z in the figure).
[0047]
As a result, when an input audio signal (= instrument sound) is input, a more natural audio signal is output as compared to the case where the input audio signal is output as it is even if the target audio is a silent portion. Will be.
When the input sound signal (= instrument sound) is an unvoiced sound, the unvoiced sound of the target sound that is output from the inverse fast Fourier transform unit 35 of the inverse FFT processing unit 109 is directly passed through the mixer 33 of the final processing unit 110. Will be output.
[0048]
[3.4] Residual component processing unit
[3.4.1] Operation of residual component detector
Next, the residual component detection unit 10 constituting the SMS analysis unit 101 has a residual component signal SRD (time waveform) that is a deviation between the sine wave component synthesis signal SSS output from the interpolation synthesis unit 9 and the input speech signal Sv. ) Is generated. This residual component signal SRD includes many unvoiced components included in the speech. On the other hand, the aforementioned sine wave component composite signal SSS corresponds to the voiced component.
By the way, in order to resemble the voice of a singer as a target of conversion (Target), if processing is performed only for voiced sounds, it is not necessary to process unvoiced sounds.
Therefore, in the present embodiment, the speech conversion process is performed on the deterministic component corresponding to the voiced vowel component.
[0049]
  More specifically, the residual component signal SRD is converted into a frequency waveform by the fast Fourier transform unit 11 constituting the SMS analysis unit 101, and the obtained residual component signal (frequency waveform) is converted to Rme (f). Is stored in the residual component storage unit 12 of the residual component processing unit 104.
[3.4.2] Operation of residual component selector
  On the other hand, the residual component selection unit 25 constituting the residual component processing unit 104 isTarget residual component input from the target frame information holding unit 20 Rtar (f) Or silent part residual component SILENTresidual (f) Either one ofBased on the residual component signal (frequency waveform) Rme (f) held in the residual component holding unit 12 and the residual component attribute data selection information inputted from the controller 29, new residual component attribute data is obtained. A residual component Rnew (f) is generated.
[0050]
  TheFurthermore, with respect to the new residual component Rnew (f), as shown in FIG. 11, in order to simulate the same state as the new spectral shape, the high-frequency component of the residual component, that is, the high-frequency component portion More realistic sound can be reproduced by performing and controlling spectral tilt correction that compensates for the slope of the residual component according to the magnitude of the new amplifier component Anew.
[0051]
[3.5] Sine wave overtone component processing section
[3.5.1] Operation of sine wave component holding unit and average amplifier calculation unit
On the other hand, as shown in FIG. 12A, sine wave components (F0, A0), (F1, A1), (F1) output from the peak detector 5 constituting the SMS analyzer 101 via the peak link unit 8 ( F2, A2),... (F (N-1), A (N-1)) N sine wave components (hereinafter collectively referred to as Fn, An, where n = 0 to (N -1).) Is held in the sine wave component holding unit 13 of the sine wave overtone component processing unit 106, and the amplifier An is input to the average amplifier calculation unit 14, and the average amplifier Ame is calculated by the following equation for each frame. Calculated.
Ame = Σ (An) / N
[0052]
[3.5.2] Operation of amplifier normalization unit
Next, the amplifier normalization unit 15 normalizes each amplifier An with the average amplifier Ame according to the following equation to obtain a normalized amplifier A'n.
A’n = An / Ame
[3.5.3] Operation of the spectral shape calculation unit
Then, in the spectral shape calculation unit 16 constituting the SMS analysis unit 101, as shown in FIG. 12B, the frequency Fn and the sine wave component (Fn, A′n) obtained by the normalization amplifier A′n are obtained. An envelope (envelope) as a breakpoint is generated as a spectral shape Sme (f).
In this case, the value of the amplifier at the frequency between the two breakpoints is calculated by, for example, linearly interpolating the two breakpoints. Note that the interpolation method is not limited to linear interpolation.
[0053]
[3.5.4] Operation of pitch normalization unit
Subsequently, in the pitch normalization unit 17 constituting the SMS analysis unit 101, each frequency Fn is normalized by the pitch Pme detected by the pitch detection unit 7 to obtain a normalized frequency F'n.
F'n = Fn / Pme
[3.5.5] Operation of original frame information holding unit
As a result, the original frame information holding unit 18 constituting the SMS analysis unit 101 has the average amplifier Ame, the pitch Pme, and the spectral shape Sme (f) which are the original attribute data corresponding to the sine wave component included in the input audio signal Sv. ), The normalized frequency F′n is held.
In this case, the normalized frequency F′n represents a relative value of the frequency of the harmonic sequence. If the harmonic structure of the frame is handled as a complete harmonic structure, it is not necessary to hold it.
[0054]
[3.5.6] Operation of static change / vibrato-like change separation unit
Subsequently, among the original attribute data held in the original frame information holding unit 18, the average amplifier Ame and the pitch Pme are further filtered by the static change / vibrato change separation unit 19, so as to be static. Separately and keep the target change component and the vibrato change component. It is also possible to separate the jitter changing component, which is a higher frequency changing component, from the vibrato changing component.
More specifically, the average amplifier Ame is separated into an average amplifier static component Ame-sta and an average amplifier vibrato component Ame-vib.
[0055]
  Further, the pitch Pme is separated and held into a pitch static component Pme-sta and a pitch vibrato-like component Pme-vib. As a result, as shown in FIG. 12C, the original frame information data of the corresponding frame includes the average amplifier static component Ame-sta and the average amplifier vibrato component Ame corresponding to the sine wave component of the input audio signal Sv. -vib, pitch static component Pme-sta, pitch vibrato component Pme-vib, spectral shape Sme (f),as well asIt will be held in the form of normalized frequency F'n.
[0056]
[3.5.7] Operation of sine wave component attribute data selection unit
  Subsequently, the sine wave component attribute data selection unit 23 constituting the sine wave overtone component processing unit 106Target frame information holding unit 20The average amplifier static component Ame-sta corresponding to the sine wave component of the input audio signal Sv, the average amplifier vibrato component Ame-vib, one of the target sine wave component SINtar and the silent sine wave component SILENTsine Pitch static component Pme-sta, Pitch vibrato-like component Pme-vib, Spectral shape Sme (f), Normalized frequency F'n,Based on the sine wave component attribute data selection information input from the controller 29, a new amplifier component Anew, a new pitch component Pnew, and a new spectral shape Snew (f), which are new sine wave component attribute data, are generated.
[0057]
By the way, in general, when the amplifier component is large, a bright sound extending to a high frequency is obtained, and when the amplifier component is small, the sound is concealed. Therefore, with respect to the new spectral shape Snew (f), in order to simulate such a state, as shown in FIG. 11, the high-frequency component of the spectral shape, that is, the slope of the spectral shape of the high-frequency component portion. Is controlled by performing spectral tilt correction that compensates according to the magnitude of the new amplifier component Anew, so that more realistic sound can be reproduced.
[0058]
[3.5.8] Operation of attribute data transformation unit
Subsequently, with respect to the generated new amplifier component Anew, new pitch component Pnew, and new spectral shape Snew (f), a sine wave harmonic overtone is generated based on the sine wave component attribute data deformation information input from the controller 29 as necessary. Further modification is performed by the attribute data modification unit 24 constituting the component processing unit 106. For example, deformation such as extending the spectral shape as a whole is performed.
[0059]
[3.5.9] Operation of sine wave component generator
Subsequently, the sine wave component generation unit 26 constituting the sine wave overtone component processing unit 106 is not accompanied by the deformation output from the attribute data deformation unit 24, or includes a new amplifier component Anew, a new pitch component Pnew and Based on the new spectral shape Snew (f), new sine wave components (F ″ 0, A ″ 0), (F ″ 1, A ″ 1), (F ″ 2, A ″ 2) in the frame, ..., N sine wave components of (F "(N-1), A" (N-1)) (hereinafter, these are collectively referred to as a new sine wave component SINnew).
[0060]
[3.6] Sine wave deformation section
Furthermore, the obtained new sine wave component SINnew (= new frequency F ″ n and new amplifier A ″ n) is used as the sine wave deformation unit 107 based on the sine wave component deformation information input from the controller 29 as necessary. Further deformation is performed by the functioning sine wave component deformation unit 27 to output a modified new sine wave component SIN′new. For example, only the new-order amplifier A ″ n (= A ″ 0, A ″ 2, A ″ 4,...) Of even-order components is increased (for example, doubled), and so on. As a result, it is possible to give the converted speech further variety.
[0061]
[3.7] Inverse fast Fourier transform processing unit
Next, the inverse fast Fourier transform unit 28 constituting the inverse fast Fourier transform processing unit 108 uses the obtained new frequency F ″ n, new amplifier A ″ n (= new sine wave component), and new residual component Rnew (f). A new voiced sound time axis signal is obtained by storing in the FFT buffer, sequentially performing inverse FFT, performing overlap processing so that the obtained time axis signals partially overlap, and adding them. A converted voice signal (converted voiced voice signal) is generated.
At this time, based on the sine wave component / residual component balance control signal input from the controller 29, the mixing ratio of the sine wave component and the residual component is controlled to obtain a more realistic voiced signal. In this case, generally, a rough voice can be obtained by increasing the mixing ratio of the residual components.
[0062]
In this case, when the new frequency f ″ n, the new amplifier a ″ n (= new sine wave component) and the new residual component Rnew (f) are stored in the FFT buffer, they are converted with different pitches and appropriate pitches. By further adding a sine wave component, harmony can be obtained as a converted audio signal. Further, by giving a harmony pitch suitable for the accompaniment sound by the sequencer 31, a musical harmony suitable for the accompaniment can be obtained.
In addition, the inverse fast Fourier transform unit 35 constituting the inverse fast Fourier transform processing unit 109 selects the selected sine wave component SINSLCT (= target sine wave component SINtar or silent part sine wave component SILENTsine selected by the switching unit SW2 by S / Pinf. ) And the selected residual component RSLCT (= one of the target residual component Rtar or the silent residual component SILENTresidual (f)) selected in the switching unit SW1 by S / Pinf is stored in the FFT buffer. Then, the inverse inverse FFT is performed, the obtained time axis signals are overlapped so that they partially overlap, and the addition process of adding them is performed to convert a new voiceless sound (including silence) time axis signal An audio signal (converted unvoiced audio signal) is generated.
[0063]
[3.8] Switching unit SW3
Next, when the input speech is unvoiced (U) based on the unvoiced sound information U / Vinf output from the silent / voiced detection unit 6, the crossfader 30 constituting the switching unit SW <b> 3 is an inverse fast Fourier transform unit 35. The target audio signal to be output is output to the mixer 33 of the final processing unit 110 as it is.
When the target sound is voiced (V), the converted sound signal output from the inverse fast Fourier transform conversion unit 28 is output to the mixer 33.
In this case, the reason why the cross fader 30 is used as the switching unit SW3 is to prevent the generation of a click sound when the switch is switched by performing a cross fade operation.
[0064]
[3.9] Final processing section
The final processing unit 110 includes a sequencer 31, a sound source unit 32, and a mixer 33.
The sequencer 31 outputs sound source control information for generating the accompaniment sound of karaoke to the sound source unit 32 as MIDI (Musical Instrument Digital Interface) data, for example.
Thereby, the sound source unit 32 generates an accompaniment signal based on the sound source control information and outputs the accompaniment signal to the mixer 33.
The mixer 33 mixes either the input audio signal Sv or the converted audio signal and the accompaniment signal, and outputs the mixed signal to the output unit 34.
The output unit 34 has an amplifier (not shown) and amplifies the mixed signal and outputs it as an acoustic signal.
[0065]
[4] Effects of the embodiment
As a result, when an input sound such as a musical instrument sound is input, it is converted into a natural sound such as a singing voice of the target singer and output.
In this case, even if there is a silent part in the singing voice of the target singer, the converted voice is output based on the previously prepared silent part data (silent part sine wave component + silent part residual component). Less natural converted sound is output.
[0066]
[5] Modification of embodiment
[5.1] First modification
In the above-described embodiment, the instrument sound is input and the analyzed singing voice is held in advance. However, the present invention is not limited thereto, and the singing voice is input and the instrument sound is output, or the intermediate tone color between the singing voice and the instrument sound. May be output.
[5.2] Second modification
The extraction of the sine wave component is not limited to the method used in this embodiment. In short, it is only necessary to extract a sine wave component included in the audio signal.
[0067]
[5.3] Third modification
In this embodiment, the target sine wave component and residual component are stored, but instead, the target speech itself is stored and read out, and the sine wave component and residual component are extracted by real-time processing. May be. That is, you may perform the process similar to the process performed with respect to the input audio | voice in this embodiment with respect to the voice of a target singer.
[5.4] Fourth modification
In the present embodiment, pitch, amplifier, and spectral shape are all handled as sine wave components, but it is also possible to handle at least one of them.
[0068]
【The invention's effect】
As described above, according to the present invention, the sound of a musical instrument is converted to resemble a predetermined singer's singing voice, or the singer's singing voice is converted to resemble a predetermined musical instrument's sound. Even if the sound source is completely different, it is possible to easily obtain a more natural converted sound and to perform natural sound conversion even when the target sound does not exist in the corresponding performance part. It becomes.
[Brief description of the drawings]
FIG. 1 is a flowchart of an overview process according to an embodiment.
FIG. 2 is a schematic configuration block diagram of the embodiment.
FIG. 3 is a block diagram (part 1) illustrating a detailed configuration of the embodiment.
FIG. 4 is a block diagram (part 2) illustrating a detailed configuration of the embodiment.
FIG. 5 is a diagram illustrating a state of a frame in the embodiment.
FIG. 6 is an explanatory diagram for explaining peak detection of a frequency spectrum in the embodiment.
FIG. 7 is a diagram illustrating cooperation of peak values for each frame in the embodiment.
.
FIG. 8 is a diagram illustrating a change state of a frequency value in the embodiment.
FIG. 9 is a diagram illustrating a change state of a deterministic component in a process in the embodiment.
FIG. 10 is an explanatory diagram of a detailed operation of a silent part detection unit.
FIG. 11 is a diagram for explaining spectral tilt compensation of a spectral shape;
FIG. 12 is an explanatory diagram of signal processing in the embodiment.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Microphone, 2 ... Analysis window production | generation part, 3 ... Input audio | voice signal extraction part, 4 ... Fast Fourier transform part, 5 ... Peak detection part, 6 ... Unvoiced / voiced detection part, 7 ... Pitch extraction part, 8 ... Peak Linking unit, 9 ... interpolation synthesis unit, 10 ... residual component detection unit, 11 ... fast Fourier transform unit, 12 ... residual component holding unit, 13 ... sine wave component holding unit, 14 ... average amplifier calculation unit, 15 ... amplifier Normalization unit, 16 ... Spectral shape calculation unit, 17 ... Pitch normalization unit, 18 ... Original frame information holding unit, 19 ... Static change / vibrato change separation unit, 20 ... Target frame information holding unit, 21 ... Silence Partial detection unit, 23 ... sine wave component attribute data selection unit, 24 ... attribute data transformation unit, 25 ... residual component selection unit, 26 ... sine wave component generation unit, 27 ... sine wave component transformation unit, 28 ... inverse fast Fourier Conversion unit, 29 ... Contro , 30 ... crossfader, 31 ... sequencer, 32 ... sound source unit, 33 ... mixer, 34 ... output unit, 35 ... inverse fast Fourier transform unit, 101 ... SMS analysis unit, 102 ... target information storage unit, 103 ... silent part detection 104, residual component processing unit, 106 ... sine wave overtone component processing unit, 107 ... sine wave deformation unit, 108 ... inverse fast Fourier transform processing unit, 109 ... inverse fast Fourier transform processing unit, 110 ... final processing unit.

Claims (6)

Component extraction means for extracting a sine wave component and a residual component from an input audio signal corresponding to a musical instrument sound ;
A target sine wave component and a target residual component extracted from the target voice signal corresponding to the singing voice ;
New component generation means for deforming the sine wave component to generate a new sine wave component, deforming the residual component to generate a new residual component ;
Converted speech signal generating means for combining the new sine wave component and the new residual component generated by the new component generating means to generate and output a converted speech signal;
When generating the converted audio signal, a silent part detecting means for detecting whether or not a part of the target audio signal that is to be made to correspond to the input audio signal is a silent part;
A silent part component output means for outputting a preset silent part sine wave component and silent part residual component when the part to be associated is a silent part;
The new component generation means generates a new sine wave component by deforming the sine wave component based on the silence portion sine wave component when the portion to be associated is a silence portion, and the silence portion When the residual component is deformed based on the residual component to generate a new residual component, and the portion to be matched is not a silent portion, the sine wave component is converted based on the target sine wave component. An audio conversion device characterized by generating a new sine wave component by deformation and generating a new residual component by modifying the residual component based on the target residual component . Component extraction means for extracting a sine wave component and a residual component from an input voice signal corresponding to a singing voice ;
A target sine wave component and a target residual component extracted from the target audio signal corresponding to the instrument sound ;
New component generation means for deforming the sine wave component to generate a new sine wave component, deforming the residual component to generate a new residual component ;
Converted speech signal generating means for combining the new sine wave component and the new residual component generated by the new component generating means to generate and output a converted speech signal;
When generating the converted audio signal, a silent part detecting means for detecting whether or not a part of the target audio signal that is to be made to correspond to the input audio signal is a silent part;
A silent part component output means for outputting a preset silent part sine wave component and silent part residual component when the part to be associated is a silent part;
The new component generation means generates a new sine wave component by deforming the sine wave component based on the silence portion sine wave component when the portion to be associated is a silence portion, and the silence portion When the residual component is deformed based on the residual component to generate a new residual component, and the portion to be matched is not a silent portion, the sine wave component is converted based on the target sine wave component. An audio conversion device characterized by generating a new sine wave component by deformation and generating a new residual component by modifying the residual component based on the target residual component . In the voice converter according to claim 1 or 2,
Comprising silent detection means for detecting whether or not the input voice signal is a silent sound;
When the unvoiced detection means detects that the input sound signal is unvoiced sound, the target sound signal is output as the converted sound signal. A component extraction step of extracting a sine wave component and a residual component from an input audio signal corresponding to a musical instrument sound ;
And novel component generating step of generating a new residual component to generate a new sinusoid, then deforming said residual component to deform the sine wave component,
A converted voice generation step of generating a converted voice by combining the new sine wave component and the new residual component;
Upon generating the conversion voice, said a portion of the target sound, and a silence detecting step portion to be corresponding to the input voice is detected whether the silent portion,
The new component generation step generates a new sine wave component by transforming the sine wave component based on a preset silent portion sine wave component when the portion to be associated is a silent portion, Based on the set silent part residual component, the residual component is transformed to generate a new residual component. On the other hand, if the part to be matched is not a silent part, it is extracted from the target voice corresponding to the singing voice. The sine wave component is modified based on the target sine wave component to generate a new sine wave component, and the residual component is modified based on the target residual component extracted from the target voice corresponding to the singing voice to generate a new residual sine wave component. A voice conversion method characterized by generating a difference component . A component extraction step of extracting a sine wave component and a residual component from an input audio signal corresponding to a singing voice ;
And novel component generating step of generating a new residual component to generate a new sinusoid, then deforming said residual component to deform the sine wave component,
A converted voice generation step of generating a converted voice by combining the new sine wave component and the new residual component;
Upon generating the conversion voice, said a portion of the target sound, and a silence detecting step portion to be corresponding to the input voice is detected whether the silent portion,
The new component generation step generates a new sine wave component by transforming the sine wave component based on a preset silent portion sine wave component when the portion to be associated is a silent portion, Based on the set silent part residual component, the residual component is transformed to generate a new residual component. On the other hand, if the part to be matched is not a silent part, it is extracted from the target voice corresponding to the instrument sound. The sine wave component is transformed based on the target sine wave component to generate a new sine wave component, and the residual component is transformed based on the target residual component extracted from the target sound corresponding to the instrument sound. A speech conversion method characterized by generating a new residual component . The voice conversion method according to claim 4 or 5,
Comprising a silent detection step of detecting whether the input voice is a silent sound,
The speech conversion method, wherein the target speech is output as the converted speech when the input speech is detected to be unvoiced in the silent detection step.

Images