JP2000003187A - Method and device for storing voice feature information - Google Patents

Abstract

PROBLEM TO BE SOLVED: To store precise voice feature information with small storage capacity by obtaining a specified multiplication and a specified difference or ratio related to plural frequencies showing a voice feature. SOLUTION: An input voice signal segmenting part 3 multiplies an analytic window AW having a period of fixed times of a pitch period generated by an analytic window generation part 2 with an input voice signal Sv. Then, the cut-out part 3 segments the input voice signal Sv in frame to output it to a fast Foulier transform part (FFT) 4 as a frame voice signal FSv. The signal FSv is analytic processed by the FFT 4, and a local peak value shown by combination between a frequency value and an amplifier value is detected from a frequency spectrum being its output by a peak value detection part 5. Plural frequencies Fk (k is natural number) showing such a voice characteristic are obtained, and a reference frequency Fo is multiplied with respective natural numbers (k), and the difference or the ratio between respective multiplication results Fo×k and respective frequencies Fk are obtained to store these difference or the ratio.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for generating an audio signal, and more particularly to a method and apparatus for storing audio characteristic information suitable for use in a karaoke apparatus.

[0002]

2. Description of the Related Art There have been developed various voice converters for changing the frequency characteristics and the like of an input voice and outputting the converted voice. For example, some karaoke devices convert the pitch of a singer's singing voice into a male voice. Some voices are converted to female voices and vice versa (for example, Japanese Translation of International Patent Application No. Hei 8-5085).
No. 81).

[0003]

In the conventional voice conversion apparatus, although voice conversion (for example, male voice → female voice, female voice → male voice) is performed, only voice quality is changed. It could not be converted to resemble the voice of a particular singer (eg, a professional singer). Also, if there is a function like imitation that makes not only voice quality but also singing style similar to a specific singer, it is very interesting in karaoke equipment etc. It was impossible.
Thus, the present inventors have provided a voice conversion device that can resemble the voice of a singer whose voice quality is a target.

[0004] However, in such a device, it is necessary to store voice feature information, so that a huge storage capacity is required. The present invention has been made in view of the above-described circumstances, and has as its object to provide a voice feature information storage method and a voice feature information storage device capable of storing voice feature information with high accuracy with a small storage capacity.

[0005]

According to a first aspect of the present invention, there is provided a configuration for obtaining a plurality of frequencies Fk (where k is a natural number) representing a characteristic of a voice, and a method of obtaining a reference frequency F0. Multiplying by each of said natural numbers k;
A step of obtaining a difference or a ratio between each of the multiplication results F0 × k and each of the frequencies Fk, and storing the difference or the ratio to store the feature of the voice. Further, in the configuration according to the second aspect, a step of obtaining a plurality of frequencies Fk (k is a natural number) representing a feature of the voice, a step of multiplying a reference frequency F0 by each of the natural numbers k, A step of obtaining a difference or ratio between each multiplication result F0 × k and each of the frequencies Fk, a step of selecting a difference or ratio exceeding a predetermined threshold value, and storing the selected difference or ratio. Thus, the feature of the voice is stored. Further, in the configuration according to the third aspect, a step of obtaining a plurality of frequencies Fk (where k is a natural number) representing a feature of the voice, a step of multiplying a reference frequency F0 by each of the natural numbers k, Obtaining a difference or ratio between each multiplication result F0 × k and each of the frequencies Fk, selecting a predetermined number of differences or ratios in descending order of these differences or ratios, and storing these selected differences or ratios And storing the characteristics of the voice.
Further, in the configuration of the fourth aspect, a process of obtaining a plurality of frequencies Fk (where k is a natural number) representing a feature of a voice, an amplitude value Ak corresponding to the frequency Fk, and a process of obtaining the amplitude value Ak Selecting a frequency Fk corresponding to a frequency not less than a predetermined value, multiplying a reference frequency F0 by each natural number k corresponding to the selected frequency Fk; Each selected frequency F
a step of obtaining a difference or a ratio with respect to k, and storing the difference or the ratio to store the feature of the voice. According to a fifth aspect of the present invention, the voice feature information storage method according to any one of the first to fourth aspects is performed.

[0006]

DESCRIPTION OF THE PREFERRED EMBODIMENTS [1] Outline Processing of Embodiment [2] Outline Processing of Embodiment Next, a preferred embodiment of the present invention will be described with reference to the drawings. First, the outline processing of the embodiment will be described. [2.1] Step S1 First, the voice (input voice signal) of the singer (me) trying to imitate is FFT (Fast Fourie Tr) in real time.
SMS (Spectral Modeling S)
analysis) and performs a sine wave component (Si
ne component), and generates a residual component (residual component) for each frame from the input audio signal and the sine wave component. In parallel with this, it is determined whether or not the input audio signal is unvoiced (including non-voiced sound). If the input audio signal is unvoiced, the following steps S2 to S6 are not performed, and the input audio signal is output as it is. Become. In this case,
As the SMS analysis, pitch synchronous analysis that changes the analysis window width according to the pitch in the previous frame is employed.

[2.2] Step S2 Next, when the input voice signal is a voiced sound, pitch (Pitch), amplifier (Amplitude) and spectral which are original attribute data are further extracted from the extracted sine wave component. -Extract a shape (Spectral Shape). Further, the extracted pitch and amplifier are separated into a vibrato component and components other than the vibrato component.

[2.3] Step S3 Attempt to imitate from the singer's attribute data (target attribute data = pitch, amplifier and spectral shape) which is stored (saved) in advance and is the target of imitation (Target). The target attribute data (= pitch, amplifier, and spectral shape) of the frame corresponding to the frame of the input voice signal of the singer (me) to be extracted is extracted.
In this case, the singer trying to imitate (m
If target attribute data of a frame corresponding to the frame of the input audio signal of e) does not exist, target attribute data is generated in accordance with a predetermined Easy Synchronization Rule, as described in detail later. Then, the same processing is performed.

[2.4] Step S4 The original attribute data corresponding to the singer (me) to be imitated next and the target attribute data corresponding to the singer to be imitated are appropriately selected and combined. , New attribute data (new attribute data = pitch, amplifier, and spectral shape). In addition, when using as a simple voice conversion instead of imitation,
It is also possible to obtain new attribute data by calculation based on both the original attribute data and the target attribute data, such as obtaining the new attribute data as an average of the original attribute data and the target attribute data.

[2.5] Step S5 A sine wave component of the frame is determined based on the new attribute data obtained in the subsequent step. [2.6] Step S6 Then, one of the sine wave component obtained and the residual component obtained in step S1 or the residual component of the singer to be a target (Target) to be imitated in advance (preserved). When,
To obtain a converted audio signal.

[2.7] Conclusion According to the converted voice signal obtained as a result of these processes, the reproduced voice is as if the singing voice of the singer trying to imitate is different from that of another singer (the target singing voice). Person) sings like a singer.

[3] Detailed Configuration of the Embodiment FIGS. 1 and 2 show detailed configuration diagrams of the embodiment. In addition,
The present embodiment is an example in which the voice conversion device (voice conversion method) according to the present invention is applied to a karaoke device and configured as a karaoke device that can perform imitation.

In FIG. 1, a microphone 1 collects the voice of a singer (me) trying to imitate, and
It is output to the input audio signal cutout unit 3 as v. In parallel with this, the analysis window generation unit 2 generates an analysis window (for example, a Hamming window) AW having a period that is a fixed multiple (for example, 3.5 times) of the period of the pitch detected in the previous frame. Is output to the input audio signal cutout unit 3. When the initial state or the previous frame is an unvoiced sound (including silence), an analysis window having a fixed period set in advance is output to the input audio signal cutout unit 3 as the analysis window AW.

Thus, the input audio signal extracting section 3 multiplies the input analysis window AW by the input audio signal Sv, cuts out the input audio signal Sv in units of frames, and converts the input audio signal Sv into a frame audio signal FSv as a fast Fourier transform section 4. Is output to

More specifically, the relationship between the input audio signal Sv and the frames is as shown in FIG. 3, and each frame FL is set so as to partially overlap the previous frame FL. Then, the frame audio signal FSv is analyzed in the fast Fourier transform unit 4, and a local peak is detected by the peak detecting unit 5 from the frequency spectrum output from the fast Fourier transform unit 4, as shown in FIG. .

More specifically, a local peak marked with x is detected in the frequency spectrum as shown in FIG. This local peak is represented as a combination of a frequency value and an amplifier (amplitude) value. That is, as shown in FIG. 4, (F0, A0), (F1, A1), (F
Local peaks are detected and represented for each frame, such as (2, A2),..., (FN, AN).

Then, as schematically shown in FIG. 3, one set (hereinafter referred to as a local peak set) is output to the unvoiced / voiced detection unit 6 and the peak coordination unit 8 for each frame. The unvoiced / voiced detection unit 6 detects that the voice is unvoiced ('t', 'k', etc.) according to the magnitude of the high-frequency component based on the input local peak for each frame.
The voiced detection signal U / Vme is supplied to a pitch detection unit 7, an easy synchronization processing unit 22, and a crossfader unit 3.
Output to 0. Alternatively, it is detected that there is no voice according to the number of zero crosses per unit time on the time axis ('s', etc.)
Then, the original unvoiced / voiced detection signal U / Vme is converted to a pitch detection unit 7,
The signals are output to the easy synchronization processing unit 22 and the crossfader unit 30.

Further, the unvoiced / voiced detecting section 6 outputs the inputted local peak set to the pitch detecting section 7 as it is when the input frame is not detected as unvoiced. The pitch detector 7 detects the pitch Pme of the frame corresponding to the local peak set based on the input local peak set.

As a more specific method of detecting the pitch Pme of a frame, for example, Maher, RCand J.W. Beauchamp: "
Fundamental Frequency Estimation of Musical Signal
using a two-way Mismatch Procedure "(Journal of A
counstical Society of America95 (4): 2254-2263).

Next, in the local peak set output from the peak detecting section 5, the link is determined for the preceding and succeeding frames in the peak linking section 8, and the local peaks recognized to be linked are formed into a series of data strings. A linking process for connecting a local peak to the data is performed.

Here, the cooperation processing will be described with reference to FIG. Now, a local peak as shown in FIG. 5A is detected in the previous frame, and FIG.
It is assumed that a local peak as shown in FIG.

In this case, the peak coordinating unit 8 calculates each local peak (F0, A0) detected in the previous frame,
(F1, A1), (F2, A2), ..., (FN, A
It is checked whether the local peak corresponding to N) has been detected in the current frame. The determination as to whether or not there is a corresponding local peak is made based on whether or not the local peak of the current frame is detected within a predetermined range centered on the frequency of the local peak detected in the previous frame.

More specifically, in the example of FIG. 5, the local peaks (F0, A0), (F1, A1), (F2, A
2)..., Corresponding local peaks are detected, but for local peaks (FK, AK) (see FIG. 5A), corresponding local peaks (FIG.
(See (B)) is not detected.

When the corresponding local peaks are detected, the peak linking unit 8 connects the local peaks in chronological order and outputs them as a set of data strings. If the corresponding local peak is not detected, the data is replaced with data indicating that there is no corresponding local peak for the frame. Here, FIG. 6 shows an example of changes in the frequency F0 and the frequency F1 of the local peak over a plurality of frames.

Such a change is caused by the amplifier (amplitude) A0,
A1, A2,... Are similarly recognized. In this case, the data string output from the peak linking unit 8 is a discrete value output at every frame interval. The peak value output from the peak linking unit 8 is hereinafter referred to as a deterministic component. This means a component that is deterministically replaced as a sine wave element in the original signal (that is, the audio signal Sv). Further, each of the replaced sine waves (strictly speaking, frequency and amplifier (amplitude) which are parameters of the sine wave) will be referred to as sine wave components.

Next, the interpolation synthesizing unit 9 performs an interpolation process on the deterministic component output from the peak linking unit 8, and synthesizes a waveform by a so-called oscillator method based on the deterministic component after the interpolation. The interpolation interval in this case is determined by the output unit 3 described later.
4 is performed at intervals corresponding to the sampling rate (for example, 44.1 KHz) of the final output signal output. The solid lines shown in FIG. 6 described above are the frequencies F0 and F1 of the sine wave components.
5 shows an image when the interpolation processing is performed for.

[3.1] Configuration of Interpolation Synthesis Unit The configuration of the interpolation synthesis unit 9 is shown in FIG. The interpolation / synthesis unit 9 includes a plurality of partial waveform generation units 9a, and each of the partial waveform generation units 9a adjusts a frequency (F0, F1,...) And an amplifier (amplitude) of a designated sine wave component. Generates a corresponding sine wave. However, the sine wave components (F0, A0), (F1, A1), (F2,
Since A2),... Change every moment according to the interpolation interval, the waveform output from each partial waveform generator 9a becomes a waveform according to the change.

That is, the sine wave components (F0, A0), (F1, A1), (F2, A2),
Are sequentially output and interpolation processing is performed on each of the sine wave components, so that each partial waveform generating section 9a outputs a waveform whose frequency and amplitude fluctuate within a predetermined frequency region. Then, the waveforms output from the respective partial waveform generators 9a are added and synthesized in an adder 9b. Therefore,
The output signal of the interpolation / synthesis unit 9 is a sine wave component synthesized signal SSS obtained by extracting a deterministic component from the input audio signal Sv.

[3.1.1] Data Configuration of Interpolation / Synthesis Unit 9 Here, the data configuration in the interpolation / synthesis unit 9 will be described. The interpolation / synthesis unit 9 has N + 1 pairs of frequency and amplifier for each partial component. To store these data with high accuracy, an enormous memory capacity is required. On the other hand, if the accuracy is reduced by a method such as reducing the effective digits of data, the fidelity of the audio signal is also reduced.

On the other hand, as a property of a human voice signal, the frequency of each partial component is almost an integral multiple of the pitch frequency.
Therefore, by utilizing this property and paying attention to the difference between a value of an integer multiple of the pitch frequency and the frequency of each partial component, it is considered that faithful reproduction can be performed with a small memory capacity. Specifically, it is preferable to employ any of the methods described below or a combination thereof.

(1) Method for storing the difference The frequency Fk (where k = 0 to N) can be expressed as follows. Fk = F0 × k + dFk Here, F0 × k is a value of an integral multiple of the pitch frequency,
dFk is a difference value between the value of this integral multiple and the actual frequency Fk. By storing the difference value dFk instead of actually storing the frequency Fk, the memory capacity for storing the frequency Fk can be reduced. However, the minimum and maximum possible values of the difference value dFk increase in proportion to the value “k”.

(2) Method of storing the ratio The frequency Fk (where k = 0 to N) can also be expressed by the following equation. Fk = F0 × k × rFk Here, rFk is a ratio between a value F0 × k, which is an integral multiple of the pitch frequency, and the actual frequency Fk. Thus, by storing the ratio rFk without actually storing the frequency Fk, the memory capacity for storing the frequency Fk can be reduced. This ratio rFk is easier to handle than the difference value dFk described above in that it is substantially constant regardless of the value “k”.

Here, it is sufficient to secure the range in which each of the frequencies Fk can be taken is about 50 Hz to 10 kHz.
On the other hand, although the range of the ratio rFk varies from person to person, according to observations by the present inventors, it is possible to faithfully reproduce most human voices if one pitch (about 100 cents) is secured. it can. It is sufficient to secure the accuracy of the ratio rFk of about one cent.

(3) Method of Storing Ratio as Logarithmic Value When storing the ratio rFk, if this ratio is converted to a logarithmic value, the memory capacity can be further reduced. If "cent" is used as the logarithmic value cFk, the logarithmic value c
Fk is obtained by the following equation. cFk = 1200 × log2 (rFk) Specifically, in order to express the range of +/− 100 cents with an accuracy of 1 cent by the logarithmic value cFk, the logarithmic value cFk
May be stored in 8 bits.

(4) Method of omitting storage of some frequencies Fk This method is a method that can be adopted in combination with any of the above methods (1) to (3). In the above methods (1) to (3), it is considered that “(N + 1)” difference values dFk, ratio rFk, or logarithmic value cFk are required. However, according to experiments by the present inventors, it has been found that the sound quality does not deteriorate much even if it is assumed that some frequencies Fk are integral multiples of the pitch frequency F0. In such a partial component, a value that is an integral multiple of the pitch frequency F0 can be used as the frequency Fk, and there is no need to store a logarithmic value cFk or the like.

Assuming that the methods (3) and (4) are used in combination, the following components (4.1) to (4.3) are used as the partial components for faithfully reproducing the frequency Fk. Can be determined. In addition, the methods (4.1) to (4.3) may be used alone or in combination. (4.1) The reproduction number M (where M <N + 1) is determined in advance, and M partial components are selected in ascending order of the logarithmic value cFk. (4.2) A threshold value is determined for the logarithmic value cFk, and a partial component whose logarithmic value cFk exceeds the threshold value is selected. (4.3) The size of the amplifier is a predetermined condition (for example, the maximum A
A partial component that satisfies (a value greater than −30 dB with respect to k) is selected.

When M pieces of frequency information relating to the partial components selected as described above are stored, a value M is stored in a predetermined area of the memory, and a value k indicating the order of the component,
It is preferable to store M sets of information for specifying the frequency (the logarithmic value cFk and the like). This method uses the number of partial components N + 1.
This is particularly effective when the reproduction number M is much smaller than the reproduction number M.

(5) Method for storing the amplifier Ak This method is the same as the method (4) or the method (1) to (3).
Is a method that can be adopted in combination with the above. In the present embodiment, as described above, the amplifier A
k is stored. If one byte (8 bits) is assigned to each amplifier Ak and the data accuracy is 1 dB, the amplifier Ak has a dynamic range of 256 dB. However, in practice, such a wide dynamic range is not required, and it is sufficient to secure about 128 dB.

In a normal computer, since the data length is set in units of 8 bits, if 7 bits are allocated to the amplifier Ak and a dynamic range of 128 dB is secured, one bit surplus occurs. Therefore, this one bit indicates whether or not there is information (for example, the logarithmic value cFk) for specifying the frequency. Hereinafter, this information is referred to as a flag xk.

In this case, it is not necessary to provide an independent storage area for storing the value k indicating the order of the component, so that the memory capacity can be further reduced. It should be noted that various modes are conceivable for expressing the decibel value of the amplifier Ak. For example, assuming that the maximum value of the amplifier Ak is 0 dB, it may be expressed in a range of 0 to -127 dB. When the maximum value has a value exceeding 0 dB, or when the required dynamic range is narrow and a high resolution is desired, the amplifier nAk is obtained by the following equation, and the amplifier nAk is stored instead of the amplifier Ak. Good. nAk = α · (Ak + β)

That is, in the above equation, the amplifier nAk is 0
Α or β may be determined so as to fall within the range of ~ 127. As an example, the required dynamic range is + 20dB ~
In the case of −40 dB, α = −127 / 60, β = −
If n = 20, nAk = 0 when the amplifier Ak is 20 dB,
When the amplifier Ak is −40 dB, nAk = 127, and the storage area can be used effectively. The values of α and β may be determined in advance and set to fixed values. If it is desired to change them according to the situation, one or both of them may be made variable by taking measures such as adding them to the data string. However, if there is data exceeding the range of 0 to 127 in the above method, a value of 0 or less is set to 0 so that the data falls within the range.
The above values will need to be aligned to 127.

(6) Conclusion The above methods (3), (4) and (5) are combined to find that the frequency F
For a sine wave component that should faithfully reproduce k, the amplifier Ak
(7 bits), flag xk (1 bit), and logarithmic value cF
With k (8 bits), a total memory capacity of 16 bits is required. If the frequency Fk can be approximated to an integral multiple of the pitch frequency F0, the amplifier Ak (7 bits)
And a flag xk (1 bit), a total memory capacity of 8 bits is required. Since general floating-point data requires a 32-bit length, combining the amplifier Ak and the frequency Fk requires 64 bits per sine wave component. In the present embodiment, since this can be reduced to 8 to 16 bits, the required memory capacity is 1/8 to 1/4.
It is possible to reduce to the extent.

[3.2] Operation of Residual Component Detecting Unit Next, the residual component detecting unit 10 calculates the deviation between the sine wave component synthesized signal SSS output from the interpolation synthesizing unit 9 and the input audio signal Sv. A residual component signal SRD (time waveform) is generated. This residual component signal SRD contains a lot of unvoiced components included in the voice. On the other hand, the above-mentioned sine wave component composite signal SSS corresponds to a voiced component. By the way, in order to resemble the voice of the singer who becomes the target (Target), if only the voiced sound is processed, it is not necessary to process the unvoiced sound. Therefore, in the present embodiment, speech conversion processing is performed on a deterministic component corresponding to a voiced vowel component. More specifically, regarding the residual component signal SRD,
The fast Fourier transform unit 11 converts the signal into a frequency waveform, and the obtained residual component signal (frequency waveform) is stored in the residual component storage unit 12 as Rme (f).

[3.3] Operation of Average Amplifier Operation Unit On the other hand, as shown in FIG. 8A, sine wave components (F0, A) output from the peak detection unit 5 via the peak linking unit 8
0), (F1, A1), (F2, A2),..., (F (N
-1), A (N-1)) N sine wave components (hereinafter, these are collectively referred to as Fn and An. N = 0 to (N-1).)
Is held in the sine wave component holding unit 13, and the amplifier An is input to the average amplifier operation unit 14, and the average amplifier Ame is calculated for each frame by the following equation. Ame = Σ (An) / N

[3.4] Operation of Amplifier Normalization Unit Next, in the amplifier normalization unit 15, each amplifier An is normalized by the average amplifier Ame according to the following equation, and the normalized amplifier A'n
Ask for. A'n = An / Ame

[3.5] Operation of Spectral Shape Calculation Unit The spectral shape calculation unit 16
As shown in FIG. 8B, an envelope (envelope) having a sine wave component (Fn, A'n) obtained by the frequency Fn and the normalizing amplifier A'n as a break point has a spectral shape Sme (f). Generate as In this case, the value of the amplifier at the frequency between the two breakpoints,
It is calculated by linear interpolation. The method of interpolation is not limited to linear interpolation.

[3.6] Operation of Pitch Normalization Unit Subsequently, the pitch normalization unit 17 normalizes each frequency Fn with the pitch Pme detected by the pitch detection unit 7 to obtain a normalized frequency F'n. F'n = Fn / Pme
As a result, the original frame information holding unit 18 outputs the average amplifier Ame, the pitch Pme, the spectral shape Sme (f), and the normalized frequency F ′, which are the original attribute data corresponding to the sine wave component included in the input audio signal Sv. n will be held.

In this case, the normalized frequency F'n represents the relative value of the frequency of the overtone sequence, and if the overtone structure of the frame is treated as a complete overtone structure, it is not necessary to hold it. Absent. In this case, if a male / female conversion is going to be performed, at this stage, the pitch is raised by an octave when the male to female conversion is performed, and the pitch is lowered by an octave when the female to male conversion is performed. It is preferable to perform a female voice pitch control process.

Subsequently, of the original attribute data held in the original frame information holding unit 18, the average amplifier Ame and the pitch Pme are further subjected to a filtering process and the like by the static change / vibrato change change separation unit 19. Thus, the static change component and the vibrato change component are separately held. In addition, it is also possible to configure so as to further separate a jitter variable component which is a higher frequency change component from a vibrato variable component.

More specifically, the average amplifier Ame is separated into an average amplifier static component Ame-sta and an average amplifier vibrato component Ame-vib and held. The pitch Pme is defined as a pitch static component Pme-sta and a pitch vibrato-like component Pme.
-vib and keep separately.

As a result, the original frame information data INFme of the corresponding frame is, as shown in FIG.
Average amplifier static component Ame-sta, average amplifier vibrato component Ame-vib, pitch static component Pme-sta, pitch vibrato component Pme-vib, which are original attribute data corresponding to the sine wave component of the input audio signal Sv, Spectral Shape S
me (f), the normalized frequency F'n, and the residual component Rme (f).

On the other hand, the target frame information data INFtar composed of the target attribute data corresponding to the singer to be imitated (target) is analyzed in advance and stored in advance in a hard disk or the like constituting the target frame information storage unit 20. Have been. In this case, of the target frame information data INFtar,
As target attribute data corresponding to the sine wave component,
Average amplifier static component Atar-sta, average amplifier vibrato component Atar-vib, pitch static component Ptar-sta, pitch vibrato component Ptar-vib, spectral shape St
There is ar (f).

The target frame information data IN
Among Ftar, target attribute data corresponding to the residual component includes a residual component Rtar (f).

[3.7] Operation of Key Control / Tempo Change Unit Next, the key control / tempo change unit 21 responds to the synchronization signal SSYNC from the target frame information holding unit 20 based on the synchronization signal SSYNC from the sequencer 31. Of target frame information INFtar of the frame to be read and read target frame information data IN
The target attribute data constituting the Ftar is corrected, and the read target frame information INF is read.
It outputs tar and a target unvoiced / voiced detection signal U / Vtar indicating whether the frame is unvoiced or voiced.

More specifically, a key control unit (not shown) of the key control / tempo change unit 21 is configured such that when the key of the karaoke apparatus is raised or lowered from the reference, the pitch static component Ptar- which is the target attribute data.
For the sta and pitch vibrato-like components Ptar-vib,
A correction process for raising and lowering by the same amount is performed. For example, 50
When the key is raised by [cent], the pitch static component P
The tar-sta and pitch vibrato-like component Ptar-vib must also be increased by 50 [cent].

A tempo change unit (not shown) of the key control / tempo change unit 21 reads the target frame information data INFtar at a timing corresponding to the changed tempo when the tempo of the karaoke apparatus is raised or lowered. There is a need to do. In this case, if there is no target frame information data INFtar corresponding to the timing corresponding to the required frame, the target frame information data INFtar of the two frames existing before and after the timing of the required frame is read out. Interpolation is performed using these two pieces of target frame information data INFtar to generate target frame information data INFtar of the frame at the necessary timing, and furthermore, target attribute data.

In this case, the vibrato-like components (average amp vibrato-like component Atar-vib and pitch vibrato-like component Ptar-vib) are unsuitable as they are, because the vibrato period itself changes and is unsuitable. It is necessary to perform interpolation processing so that the period does not change. Alternatively, this problem can be avoided by holding the parameters of the vibrato cycle and the vibrato depth instead of the data representing the vibrato trajectory itself as the target attribute data and calculating the actual trajectory by calculation.

[3.8] Operation of Easy Synchronization Processing Unit Next, the easy synchronization processing unit 22 adds the original frame information data INFme to the frame of the singer who wants to imitate (hereinafter referred to as the original frame). Despite the presence of, the singer's frame that is the target of the corresponding singer (hereinafter referred to as the target frame)
If the target frame information data INFtar does not exist in the target frame, the target frame information data INFta
An easy synchronization process is performed using r as the target frame information data INFtar of the target frame.

Then, the easy synchronization processing section 22 outputs the target attribute data (average amplifier static component Atar-sync-) of the sine wave component among the target attribute data included in the replaced target frame information data INFtar-sync described later. sta, average amp vibrato component Atar-sync-vib, pitch static component Ptar-sync-st
a, a pitch vibrato-like component Ptar-sync-vib and a spectral shape Star-sync (f)) are output to the sine wave component attribute data selection unit 23.

Further, the easy synchronization processing unit 22 generates target attribute data (residual component Rtar-sync (f)) relating to the residual component among target attribute data included in the replaced target frame information data INFtar-sync to be described later. ) Is output to the residual component selector 25.

In the processing by the easy synchronization processing section 22, the vibrato cycle itself changes with the vibrato-like components (average amp vibrato-like component Atar-vib and pitch vibrato-like component Ptar-vib) as they are. Therefore, it is necessary to perform interpolation processing so that the period does not change. Alternatively, this problem can be avoided by holding the parameters of the vibrato cycle and the vibrato depth instead of the data representing the vibrato trajectory itself as the target attribute data and calculating the actual trajectory by calculation.

[3.8.1] Details of Easy Synchronization Process Here, the easy synchronization process will be described in detail with reference to FIGS. 9 and 10. FIG. 9 is a timing chart of the easy synchronization process, and FIG. 10 is a flowchart of the easy synchronization process.

First, the easy synchronization processing section 22 sets a synchronization mode = "0" indicating a method of the synchronization processing (step S11). This synchronization mode =
“0” corresponds to a normal process in which the target frame information data INFtar exists in the target frame corresponding to the original frame.

Then, the original silence at a certain timing t /
The voiced detection signal U / Vme (t) changes from unvoiced (U) to voiced (V)
Is determined (step S12). For example, as shown in FIG. 9, at timing t = t1, the original unvoiced / voiced detection signal U / Vme (t) changes from unvoiced (U) to voiced (V).

In the determination in step S12, the original silent /
The voiced detection signal U / Vme (t) changes from unvoiced (U) to voiced (V)
(Step S12; Yes),
Original silence at the previous timing t-1 of timing t /
It is determined whether the voiced detection signal U / Vme (t-1) is unvoiced (U) and the target unvoiced / voiced detection signal U / Vtar (t-1) is unvoiced (U) (step S18). For example, FIG.
At time t = t0 (= t1-1), the original unvoiced / voiced detection signal U / Vme (t-1) is unvoiced (U).
And the target unvoiced / voiced detection signal U / Vtar (t-1) is unvoiced (U).

In the determination at step S18, the original silent /
When the voiced detection signal U / Vme (t-1) is unvoiced (U) and the target unvoiced / voiced detection signal U / Vtar (t-1) is unvoiced (U) (step S18; Yes), Since the target frame does not have the target frame information data INFtar, the synchronization mode is set to “1”, and the replacement target frame information data INFhold is set to the target of the frame existing in the backward direction (Backward) of the target frame. Frame information.

For example, as shown in FIG.
= T1 to t2, since the target frame information data INFtar does not exist, the synchronization mode is set to “1”, and the replacement target frame information data INFhold is set to a frame existing in the backward direction of the target frame (ie, , Target frame information data backward at timing t = frames existing at t2 to t3).

Then, the process proceeds to step S15,
It is determined whether or not the synchronization mode is "0" (step S15). If it is determined in step S15 that the synchronization mode is "0", the target frame information data I is added to the target frame corresponding to the original frame at the timing t.
If NFtar (t) exists, that is, since it is a normal process, the replaced target frame information data INFtar-t
Let sync be target frame information data INFtar (t).

INFtar-sync = INFtar (t) For example, as shown in FIG. 9, the target frame at timing t = t2 to t3 has target frame information data INFtar, so that INFtar-sync = INFtar (t). I do.

In this case, the replaced target frame information data INFtar-sync used in the subsequent processing
Target attribute data (average amplifier static component Atar-sync-sta, average amplifier vibrato-like component Atar-sy)
nc-vib, pitch static component Ptar-sync-sta, pitch vibrato-like component Ptar-sync-vib, spectral shape S
The tar-sync (f) and the residual component Rtar-sync (f) have substantially the following contents (step S16).

Atar-sync-sta = Atar-sta Atar-sync-vib = Atar-vib Ptar-sync-sta = Ptar-sta Ptar-sync-vib = Ptar-vib Star-sync (f) = Star (f) Rtar-sync (f) = Rtar (f)

If it is determined in step S15 that the synchronization mode is "1" or the synchronization mode is "2", the target frame information data INFtar (t) is added to the target frame corresponding to the original frame at the timing t. ) Does not exist, the replaced target frame information data I
Target frame information data I for replacing NFtar-sync
NFhold.

INFtar-sync = INFhold For example, as shown in FIG. 9, the target frame at the timing t = t1 to t2 does not have the target frame information data INFtar, and the synchronization mode becomes “1”. The target frame at the timing t = t2 to t3 includes the target frame information data IN
Since the Ftar exists, the process P1 of setting the replaced target frame information data INFtar-sync as replacement target frame information data INFhold, which is the target frame information data of the target frame at the timing t = t2 to t3, is performed. The target attribute data included in the replaced target frame information data INFtar-sync used is an average amplifier static component Atar-sync-s
ta, average amp vibrato component Atar-sync-vib, pitch static component Ptar-sync-sta, pitch vibrato component P
tar-sync-vib, spectral shape Star-sync (f)
And the residual component Rtar-sync (f) (step S1).
6).

As shown in FIG. 9, the timing t =
In the target frame from t3 to t4, the target frame information data INFtar does not exist, and the synchronization mode = “2”.
Since the target frame information data INFtar exists in the target frame at t3, the replaced target frame information data INFtar-sync is set at the timing t = t.
Replacement target frame information data IN which is the target frame information data of the target frames from 2 to t3
The target attribute data included in the replaced target frame information data INFtar-sync used in the subsequent processing includes an average amplifier static component Atar-sync-sta and an average amplifier vibrato-like component Atar-sy.
nc-vib, pitch static component Ptar-sync-sta, pitch vibrato-like component Ptar-sync-vib, spectral shape S
The result is a tar-sync (f) and a residual component Rtar-sync (f) (step S16).

In the determination in step S12, the original silent /
The voiced detection signal U / Vme (t) changes from unvoiced (U) to voiced (V)
(Step S12; No),
It is determined whether or not the target unvoiced / voiced detection signal U / Vtar (t) has changed from voiced (V) to unvoiced (U) (step S13). In the determination in step S13,
If the target unvoiced / voiced detection signal U / Vtar (t) changes from voiced (V) to unvoiced (U) (step S13; Yes), the original unvoiced / voiced signal at the previous timing t-1 of the timing t is output. The voiced detection signal U / Vme (t-1) is voiced (V) and the target unvoiced / voiced detection signal U / Vtar
It is determined whether or not (t-1) is voiced (V) (step S19).

For example, as shown in FIG.
Target unvoiced / voiced detection signal U / Vtar (t) at 3
Changes from voiced (V) to unvoiced (U) at timing t-1
= T2 to t3, the original unvoiced / voiced detection signal U / Vme
(t-1) is voiced (V) and the target unvoiced / voiced detection signal U / Vtar (t-1) is voiced (U). Step S
In the determination at 18, the original unvoiced / voiced detection signal U / Vme (t
-1) is voiced (V) and the target unvoiced / voiced detection signal U
If / Vtar (t-1) is voiced (V) (step S19; Yes), the target frame includes
Since the target frame information data INFtar does not exist, the synchronization mode is set to “2”, and the replacement target frame information data INFhold is set as the target frame information of the frame existing in the forward direction of the target frame.

For example, as shown in FIG.
= T3 to t4, since the target frame information data INFtar does not exist in the target frame, the synchronization mode is set to “2” and the replacement target frame information data INFhold is set to a frame existing in the forward direction of the target frame (that is, , Target frame information data forward at a timing t = frames t2 to t3).

Then, the process proceeds to step S15,
It is determined whether or not the synchronization mode is "0" (step S15), and the same processing is performed thereafter. In the determination in step S13, the target unvoiced /
The voiced detection signal U / Vtar (t) changes from voiced (V) to unvoiced (U)
(Step S13; No),
Original unvoiced / voiced detection signal U / Vme at timing t
(t) changes from voiced (V) to unvoiced (U), or
It is determined whether or not the target unvoiced / voiced detection signal U / Vtar (t) has changed from unvoiced (U) to voiced (V) (step S14).

In the determination at step S14, the original unvoiced / voiced detection signal U / Vme (t) at timing t changes from voiced (V) to unvoiced (U), and the target unvoiced / voiced detection signal U / Vtar ( If t) changes from unvoiced (U) to voiced (V) (step S14; Ye)
s), the synchronization mode is set to "0", the replacement target frame information data INFhold is initialized (cleared), the process proceeds to step S15, and the same process is performed. In the determination of step S14, the original unvoiced / voiced detection signal U / V at timing t
When me (t) does not change from voiced (V) to unvoiced (U), or when target unvoiced / voiced detection signal U / Vtar (t) does not change from unvoiced (U) to voiced (V). (Step S14; No), the process proceeds to Step S15, and the same process is performed thereafter.

[3.9] Operation of Sine Wave Component Attribute Data Selection Unit Subsequently, the sine wave component attribute data selection unit 23 replaces the replaced target frame information data INFtar-sync input from the easy synchronization processing unit 22. Attribute data on sine wave components (average amplifier static component Atar-sync-s)
ta, average amp vibrato component Atar-sync-vib, pitch static component Ptar-sync-sta, pitch vibrato component P
tar-sync-vib and spectral shape Star-sync
(f)) and the new amplifier component Anew and the new pitch component Pn, which are new sine wave component attribute data, based on the sine wave component attribute data selection information input from the controller 29.
Generate ew and a new spectral shape Snew (f).

That is, the new amplifier component Anew is generated by the following equation. Anew = A * -sta + A * -vib (* is me or tar-sy
nc) More specifically, as shown in FIG. 8D, the new amplifier component Anew is replaced with the average amplifier static component Ame-st of the original attribute data.
a or the average amplifier static component Atar-sync-sta of the target attribute data and the average amplifier vibrato component Ame-vib of the original attribute data or the average amplifier vibrato component Atar-sync-vib of the target attribute data
Is generated as a combination of any one of the above.

Further, regarding the new pitch component Pnew,
Generated by the following equation. Pnew = P * -sta + P * -vib (* is me or tar-sy
nc) More specifically, as shown in FIG. 8D, the new pitch component Pnew is defined as the pitch static component Pme-sta of the original attribute data or the pitch static component Ptar-sy of the target attribute data.
It is generated as a combination of one of nc-sta and the pitch vibrato component Pme-vib of the original attribute data or the pitch vibrato component Ptar-sync-vib of the target attribute data.

Also, a new spectral shape Snew
(f) is generated by the following equation. Snew (f) = S * (f) (However, * is me or tar-sync) By the way, generally, when the amplifier component is large, it becomes a bright sound that extends to a high frequency and has a small amplifier component. In this case, the sound will be muffled. Therefore, for the new spectral shape Snew (f), in order to simulate such a state, as shown in FIG. 11, the high-frequency component of the spectral shape, that is, the gradient of the spectral shape of the high-frequency component portion. A new amplifier component Anew
Spectral tilt compensation (sp
By performing ectral tilt correction) and controlling, more realistic sound can be reproduced.

Subsequently, the generated new amplifier component Ane
w, the new pitch component Pnew and the new spectral shape Snew (f) are further modified by the attribute data modifying unit 24 based on the sine wave component attribute data modification information input from the controller 29 as needed.
For example, a deformation such as extending the entire spectral shape is performed.

[3.10] Operation of Residual Component Selection Unit On the other hand, the residual component selection unit 25 sets the target attribute included in the replaced target frame information data INFtar-sync input from the easy synchronization processing unit 22. Target attribute data (residual component Rtar-sync (f)) relating to the residual component of the data, the residual component signal (frequency waveform) Rme (f) held in the residual component holding unit 12 and input from the controller 29 Based on the residual component attribute data selection information to be generated, a new residual component Rnew (f), which is new residual component attribute data, is generated.

That is, the new residual component Rnew (f) is generated by the following equation. Rnew (f) = R * (f) (* is me or tar-sync) In this case, whether to select me or tar-sync is the same as the new spectral shape Snew (f) It is more preferred to select

Further, as for the new residual component Rnew (f), in order to simulate the same state as the new spectral shape, as shown in FIG. 11, the high frequency component of the residual component, that is, the high frequency component, Performing a spectral tilt correction for compensating the gradient of the residual component of the component part according to the magnitude of the new amplifier component Anew,
By controlling, more realistic sound can be reproduced.

[3.11] Operation of Sine Wave Component Generation Unit Subsequently, the sine wave component generation unit 26
4, a new sine wave component (F "0, F" 0, Fnew) in the frame based on the new amplifier component Anew, new pitch component Pnew and new spectral shape Snew (f) without or with the deformation output A "0),
(F "1, A" 1), (F "2, A" 2), ...,
(F ″ (N−1), A ″ (N−1)) N sinusoidal components
These are collectively denoted as F "n, A" n. n = 0
(N-1). ).

More specifically, a new frequency F "n and a new amplifier A" n are obtained by the following equations. F "n = F'n.times.Pnew A" n = Snew (F "n) .times.Anew If it is considered as a model of a perfect harmonic structure, F" n = (n + 1) .times.Pnew.

[3.12] Operation of Sine Wave Component Deformer Further, the new frequency F "n and new amplifier A" obtained
n is further modified by the sine wave component transformation unit 27 based on the sine wave component transformation information input from the controller 29 as needed. For example, a new amplifier A "n (= A" 0, A "2, A" 4,...) Of even harmonic components.
..) Are increased (for example, doubled). As a result, it is possible to give the converted speech further variety.

[3.13] Operation of Inverse Fast Fourier Transform Unit Next, the inverse fast Fourier transform unit 28 calculates the new frequency F "n and new amplifier A" n (= new sine wave component) and new residual component. Rnew (f) is stored in the FFT buffer,
Inverse FFT is sequentially performed, and the obtained time axis signals are overlapped so as to partially overlap, and an addition processing of adding them is performed to generate a converted voice signal which is a new voiced sound time axis signal. .

At this time, based on the sine wave component / residual component balance control signal input from the controller 29,
A more realistic voiced signal is obtained by controlling the mixing ratio of the sine wave component and the residual component. In this case, generally,
When the mixing ratio of the residual components is increased, a rough voice is obtained. In this case, when the new frequency F "n, the new amplifier A" n (= new sine wave component) and the new residual component Rnew (f) are stored in the FFT buffer, they are converted at different pitches and at an appropriate pitch. Harmony can be obtained as a converted audio signal by further adding the sine wave component. Further, by giving a harmony pitch adapted to the accompaniment sound by the sequencer 31, musical harmony adapted to the accompaniment can be obtained.

[3.14] Operation of Crossfader Next, the crossfader unit 30 outputs the original unvoiced / voiced detection signal U
If the input audio signal Sv is unvoiced (U) based on / Vme (t), the input audio signal Sv is
Output to When the input audio signal Sv is voiced (V), the converted audio signal output from the inverse fast Fourier transform unit 28 is output to the mixer 33. In this case,
The reason why the crossfader unit 30 is used as the changeover switch is to prevent a click sound from occurring at the time of switchover by performing a crossfade operation.

[3.15] Operation of Sequencer, Sound Source Unit, Mixer, and Output Unit On the other hand, the sequencer 31 transmits sound source control information for generating a karaoke accompaniment sound to, for example, MIDI (Musical Instrument).
rument Digital Interface) sound source section 3 as data etc.
Output to 2. Thereby, the sound source section 32 generates an accompaniment signal based on the sound source control information, and outputs the accompaniment signal to the mixer 33.
The mixer 33 mixes either the input audio signal Sv or the converted audio signal and the accompaniment signal, and outputs the mixed signal to the output unit 34. The output unit 34 has an amplifier (not shown), amplifies the mixed signal, and outputs it as an acoustic signal.

[4] Modifications of Embodiment [4.1] First Modification In the above description, either the original attribute data or the target attribute data is selectively used as the attribute data. It is also possible to obtain a converted audio signal having an intermediate attribute by performing an interpolation process using both the original attribute data and the target attribute data. However, according to such a configuration, a converted voice that is not similar to any of the singer trying to imitate and the singer to be imitated may be obtained. Also, especially when the spectral shape is obtained by interpolation processing, the singer trying to imitate pronounces "a", and the singer to be imitated pronounces "i". There is a possibility that sounds other than "A" or "I" may be output as converted voices, and care must be taken when handling them.

[4.2] Second Modification The extraction of the sine wave component is not limited to the method used in this embodiment. In short, it is only necessary to extract a sine wave included in the audio signal.

[4.3] Third Modification In the present embodiment, the sine wave component and the residual component of the target are stored. Instead, the target voice itself is stored and read out to read in real time. The sine wave component and the residual component may be extracted by the processing. That is, processing similar to the processing performed on the voice of the singer trying to imitate in the present embodiment may be performed on the voice of the target singer.

[4.4] Fourth Modification In the present embodiment, all of the pitch, amplifier, and spectral shape are handled as attribute data.
It is also possible to handle at least one of them.

[4.5] Fifth Modification The data configuration in the interpolation / synthesis unit 9 of the present embodiment is the same as the data configuration in other units (for example, a section from the pitch normalization unit 17 to the sine wave component deformation unit 27). Needless to say, it is good. In particular, since the target frame information holding unit 20 stores the target frame information for one piece of music, the effect of reducing the data amount by using the above data configuration is great.

[5] Effects of the Embodiment As described above, according to the present embodiment, the difference value dFk, the ratio rFk or the logarithmic value cFk is stored for the frequency Fk. Accurate voice feature information can be stored. Furthermore, seven bits in one byte are allocated to the amplifier Ak to secure a dynamic range of 128 dB and determine whether or not there is information (such as the logarithmic value cFk) for specifying the frequency in the remaining one bit. In the embodiment shown, the storage capacity can be further reduced. Further, instead of the amplifier Ak, “nAk = α ·
By storing the amplifier nAk by “(Ak + β)”, it is possible to ensure the highest possible resolution according to the desired dynamic range.

[0101]

As described above, according to the present invention, since the characteristics of voice are stored by storing the difference or ratio between each multiplication result F0 × k and each frequency Fk, a high storage capacity can be obtained with a small storage capacity. Accurate voice feature information can be stored.

[Brief description of the drawings]

FIG. 1 is a block diagram (part 1) illustrating a configuration of an embodiment of the present invention.

FIG. 2 is a block diagram (part 2) showing a configuration of an embodiment of the present invention.

FIG. 3 is a diagram illustrating a state of a frame according to the embodiment.

FIG. 4 is an explanatory diagram for describing peak detection of a frequency spectrum in the embodiment.

FIG. 5 is a diagram illustrating cooperation of peak values for each frame in the embodiment.

FIG. 6 is a diagram illustrating a change state of a frequency value in the embodiment.

FIG. 7 is a diagram showing a change state of a deterministic component in a process in the embodiment.

FIG. 8 is an explanatory diagram of signal processing in the embodiment.

FIG. 9 is a timing chart of an easy synchronization process.

FIG. 10 is a flowchart of an easy synchronization process.

FIG. 11 is a diagram for explaining spectral tilt compensation of a spectral shape.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Microphone, 2 ... Analysis window generation part, 3 ... Input audio signal extraction part, 4 ... Fast Fourier transform part, 5 ... Peak detection part, 6 ...
Unvoiced / voiced detection unit, 7: pitch extraction unit, 8: peak linking unit, 9: interpolation synthesis unit, 10: residual component detection unit, 11: fast Fourier transform unit, 12: residual component holding unit, 13: sine Wave component holding unit, 14: average amplifier calculation unit, 15: amplifier normalization unit, 16: spectral shape calculation unit, 17
... Pitch normalizing section, 18 ... Original frame information holding section, 19
... Static change / vibrato change separation section, 20 ... Target frame information holding section, 21 ... Key control / tempo change section, 22 ... Easy synchronization processing section, 23 ... Sine wave component attribute data selection section, 24 ... Attribute Data transformation unit, 25: Residual component selection unit, 26: Sine wave component generation unit, 27: Sine wave component transformation unit, 28: Inverse fast Fourier transform unit, 29: Controller, 30: Crossfader unit, 31: Sequencer, 32: sound source section, 33: mixer, 34: output section.

Claims (5)

[Claims] 1. A plurality of frequencies Fk representing characteristics of a voice
(Where k is a natural number); a step of multiplying a reference frequency F0 by each of the natural numbers k; a step of calculating a difference or a ratio between each multiplication result F0 × k and each of the frequencies Fk; A voice feature information storage method, wherein the feature of the voice is stored by storing a difference or a ratio. 2. A plurality of frequencies Fk representing characteristics of a voice.
(Where k is a natural number); a step of multiplying a reference frequency F0 by each of the natural numbers k; a step of calculating a difference or a ratio between each multiplication result F0 × k and each of the frequencies Fk; A voice feature information storage method, comprising: selecting a difference or a ratio exceeding a predetermined threshold value; and storing the selected difference or ratio to store the voice feature. 3. A plurality of frequencies Fk representing characteristics of a voice.
(Where k is a natural number); a step of multiplying a reference frequency F0 by each of the natural numbers k; a step of calculating a difference or a ratio between each multiplication result F0 × k and each of the frequencies Fk; A voice feature information storage method, comprising: selecting a predetermined number of differences or ratios in descending order of the differences or ratios; and storing the voice characteristics by storing the selected differences or ratios. 4. A plurality of frequencies Fk representing characteristics of a voice
(Where k is a natural number), a process of obtaining an amplitude value Ak corresponding to these frequencies Fk, a process of selecting a frequency Fk corresponding to a frequency equal to or more than a predetermined value among the amplitude values Ak, Multiplying each natural number k corresponding to the selected frequency Fk; obtaining a difference or ratio between each multiplication result F0 × k and each selected frequency Fk; A voice feature information storage method, wherein the voice feature is stored by storing. 5. A voice feature information storage device that executes the voice feature information storage method according to claim 1.