JP2001117578A - Device and method for adding harmony sound - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a harmony sound adding device which can add natural voice harmony sound to singing voice. SOLUTION: The device is equipped with a phoneme recognition part 2 which recognizes phonemes of an input voice signal, a phoneme feature parameter storage part 3 which stores phoneme feature parameters by the phonemes, a harmony parameter control part 6 which reads phoneme feature parameters corresponding to the phonemes according to the recognized phoneme information and outputs parameter control information on harmony sound to be added according to the phoneme feature parameters and harmony information showing the harmony sound to be added, and a harmony sound synthesis part 7 which performs processes such as pitch shifting, amplitude control, formant shifting, and spectrum tilt control for the input voice signal according to the number of channels of the harmony sound to be added and the read parameter control information and adds and outputs process results, and adds the harmony sound having the features of the phonemes of the input voice signal to the input voice signal and outputs the resulting signal.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a harmony sound adding device for adding a harmony sound generated based on an input audio signal to an input audio signal.

[0002]

2. Description of the Related Art An example of a conventional harmony sound adding device is as follows.
This will be described with reference to FIG. In the harmony sound adding device shown in FIG. 4, the voice input from the microphone 101 is pitch-shifted in the pitch shifter 102 by the number of channels of the harmony sound to be added, N, and is added to the adder 103.
, The harmony sound is synthesized, and the output unit 104 outputs a signal to which a plurality of harmony sounds are added. The generation of the harmony sound in this case is based on MIDI (Musical Instrument) indicating the melody of the harmony.
ment digital interface) data and musical score information (hereinafter collectively referred to as harmony information).

Some other conventional harmony sound adding apparatuses can set a formant shift amount for each harmony sound. In such a harmony sound adding device, it is possible to perform male voice ← → female voice conversion, so-called gender change, by controlling the amount of formant shift.

[0004]

In the conventional method as described above, the pitch and amplitude of the added harmony sound are often uniquely controlled by the harmony information.
Sometimes it was a monotonous and mechanical harmony sound. In addition, since the amplitude is controlled in accordance with the amplitude of the input voice, the unnaturalness of the harmony sound has become more remarkable. As for the pitch, to improve the naturalness,
There have been proposals such as causing a fixed pitch change only at the beginning of the sound or adding a vibrato in a fixed manner, but both were fixed changes and were unnatural. As for such a method, a method of holding several fixed patterns and randomly changing the setting at the beginning of each sound has been considered, but an unnatural portion is added by that method. And did not get very good results.

SUMMARY OF THE INVENTION In view of the above circumstances, it is an object of the present invention to provide a harmony sound adding device and a method capable of adding a natural harmony sound to a voice sung in, for example, karaoke. I do.

[0006]

In order to solve the above problems, the invention according to claim 1 comprises a phoneme recognizing means for recognizing phonemes of an input voice signal, a pitch, amplitude, formant shift reference value, or spectrum tilt reference value. Phoneme feature parameter storage means for storing a phoneme feature parameter for each phoneme prepared in advance comprising at least one of the values, and generating a harmony sound signal by pitch-shifting the input voice signal to a harmony pitch, Upon generation of the harmony sound signal, a phoneme feature parameter corresponding to the phoneme is read from the phoneme feature parameter storage unit based on the phoneme information recognized by the phoneme recognition unit, and a pitch is set for each phoneme according to the read phoneme feature parameter. Shift, amplitude control, formant shift, or spectral tilt control Characterized in that it comprises a harmony note synthesizing means for performing at least one treatment of le.

As described above, according to the present invention, in synthesizing a harmony sound, the control parameters of the harmony sound are obtained from the phoneme information of the input speech signal selected by a predetermined phoneme recognition method, and the characteristic parameters for each phoneme prepared in advance. The main feature is that a harmony sound having the characteristics of the phoneme is synthesized by taking into account (pitch, amplitude, temporal change of spectrum, etc.).

[0008]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an embodiment of a harmony sound adding device according to the present invention. The harmony sound adding device shown in FIG. 1 is one in which the present invention is applied to a karaoke device, and includes a singer's microphone 1.
A predetermined process is performed on the input voice from the melody to obtain one or more harmony sounds, these are synthesized, and the accompaniment sound output from the accompaniment performance section 9 is synthesized and output. .

The microphone 1 collects the voice of the singer. The collected voice signals are sent to a phoneme recognition unit 2, a voice analysis unit 2 *,
And the harmony sound synthesis unit 7. The phoneme recognition unit 2 extracts a phoneme of the input voice signal using a known phoneme recognition method, compares the phoneme with a plurality of phoneme information registered in advance,
A phoneme number corresponding to the most similar phoneme is output based on the comparison result. The larger the number of phonemes to be registered, the better, but the capacity of the phoneme feature parameter storage unit 3 described below becomes large, so it may be about 20 to 100.
Further, the voice pitch is detected and output by the voice analysis unit 2 *. This pitch information is used when calculating the difference between the output pitch of each harmony sound specified by the harmony information 4 and the pitch of the input audio signal.

The pitch detection by the voice analysis unit 2 * may be performed by the voice recognition unit 2. Also, the voice analysis unit 2
This pitch detection by * is necessary only when difference information is required, and can be omitted in other cases. For example, in the case where the frequency of the singing voice is not changed by the difference from the harmony melody but the frequency conversion is performed so as to match the harmony melody specified by the harmony information 4, the difference information becomes unnecessary.

The phoneme feature parameter storage unit 3 stores a plurality of feature data for each phoneme, which are obtained in advance from singing data or the like, for each phoneme. In the present embodiment, the elements of the feature data include a pitch, an amplitude, a formant shift reference value, and a spectrum tilt reference value. Note that the feature data can be formed from at least one of these elements. Hereinafter, each element will be described.

(1) Pitch: Since each phoneme has a characteristic pitch fluctuation (fluctuation), it is data for reproducing the fluctuation. Here, a value obtained by sampling a characteristic pitch of each phoneme (preferably excluding a vibrato component) at a certain time interval (for example, 5 ms) is used as a ratio from the average pitch. However, if only a small amount of data can be stored, the data may have the data of about 100 ms at the beginning of each phoneme, and the subsequent data may be stored as repeated data.
The repetitive data can be obtained by storing data for a certain time (for example, for 200 ms) with a stable portion and the time (or the number of samples), and repeating the data in the time cycle.

(2) Amplitude: Since each phoneme has a characteristic amplitude fluctuation (fluctuation), it is data for reproducing the fluctuation. Here, the manner of holding the amplitude data of the waveform corresponding to each phoneme is the same as in the case of the pitch, where the amplitude envelope of each phoneme is defined as a ratio from the largest portion of the amplitude envelope or a ratio from the average level. It can be sampled in a format and held. Regarding the control of the amplitude of each phoneme with respect to the harmony sound, the harmony sound has a predetermined amplitude (in this case, in this case, like a method of creating a harmony sound by connecting sound waveforms pitch-shifted to the harmony pitch on the time axis). If it has an amplitude corresponding to the input voice), it may be omitted. The control using the amplitude data as the feature data is particularly effective when the harmony sound can be controlled irrespective of the input voice amplitude or when a method in which the amplitude of the harmony sound is fixed is adopted.

(3) Formant shift reference value: a value used as a reference when setting a different shift amount for each phoneme when performing a formant shift, and is set for each phoneme. Here, the reason why the formant shift reference value is required will be described. For example, when an unspecified number of people speak various words, the difference between the average formant frequencies (for example, the frequency between the first formant and the second formant) of each person is different for each phoneme with almost the same tendency. It turns out that there is. This is due to the vocalization mechanism. Therefore, when performing a formant shift, a harmony sound with a natural feeling cannot be obtained if the same shift amount is given to all phonemes. For this reason, the data format of the formant shift reference value is set for each phoneme and has no time-varying time information. As a specific data format, any form can be adopted as long as the formant shift reference value is normalized for each phoneme. For example, a simple example is a logarithmic value (for example, cents). If the difference between the average formants of the three phonemes obtained as an example is 20 cents, 10 cents, and 50 cents, the formant shift reference value is the maximum value of 50 cents. 10
As 0, they are 40, 20, and 100, respectively.

(4) Spectral tilt reference value: The slope of the spectrum (spectral tilt) differs for each phoneme, but is feature data for determining the reference value in that case. This spectrum tilt reference value is a value used as a reference when setting a different tilt amount for each phoneme when performing tilt control of the spectrum, and is set for each phoneme. In this embodiment, the inclination of the spectrum of the harmony sound to be added is changed according to the difference between the pitch of the input sound and the pitch of the output sound. That is, the tilt control of the spectrum is performed. In this case, if the tilt control is performed uniformly so that all the phonemes have the same spectral gradient, the same unnatural problem as in the case of the formant shift described above may occur. . Therefore, in the present embodiment, by using the spectrum tilt reference value to make the control condition of the spectrum tilt different for each phoneme, such a problem can be corrected. This parameter is not time information but one value for each phoneme. Any value can be used as long as the value for each phoneme is normalized and held. A specific example will be described below.

As a method of obtaining the spectral tilt reference value, first, for a certain person, for each phoneme (each phoneme is classified by phoneme number p), at a plurality of pitches (classified by frequency number f) from a low sound to a high sound. Have the user pronounce the sound and analyze the spectrum of the sound signal. Then, the spectrum inclination of the spectrum analysis result is obtained for each phoneme and each pitch. The spectral tilt Xfp (spectral tilt coefficient, where f: frequency number, p: phoneme number) for each phoneme and each pitch can be calculated by, for example, the following equation.

(Equation 1) Here, i is the frequency index of the spectrum analysis result, N is the maximum value of the index, x is the frequency value of each index, y is the magnitude value of the component of each index, and i = 0 is the spectral component index of the lowest frequency. Is represented.

Next, from the pitch and the spectral gradient Xfp for each phoneme, the pitch (pitch difference) versus the gradient rate, that is, the rate of change of the spectral gradient value Xfp with respect to the pitch is determined for each phoneme p. The calculation is performed by the same method (calculation similar to the above formula) when obtaining the spectrum inclination Xfp. This is represented by pitch-to-inclination ratio Yp (p: phoneme number). Then, an average pitch-to-slope rate Y for all phonemes is obtained from the pitch-to-slope rate Yp for each phoneme. Next, for each phoneme, a value Yp / Y is obtained by dividing the pitch-to-slope rate Yp by the average pitch-to-slope rate Y, and this is used as the tilt reference value. A plurality of values obtained by changing the spectrum tilt change amount Y obtained here using the pitch difference value as a parameter are stored in a predetermined storage device of the phoneme feature parameter storage unit 3 or the harmony parameter control unit 6 in a pitch. It is stored as a tilt change amount table corresponding to (pitch difference value).

The contents stored in the phoneme feature parameter storage unit 3 have been described above. Next, the harmony information 4 in FIG. 1 is information indicating the pitch (pitch) of the harmony to be added.
This may be provided from MIDI standard music data or the like, or may be included in musical score information as sequence information. The formant shift degree setting unit 51 is a means for the operator to set the formant shift amount using a predetermined operation element when the harmony sound is not a self-voice but is a female voice for a male or a male voice for a female. Alternatively, when the set amount is changed in accordance with the song, the set amount may be included in the musical score information and held as sequence information, and the value may be called and set.

The harmony parameter control section 6 includes a phoneme feature parameter value corresponding to the input phoneme output from the phoneme feature parameter storage section 3, harmony information 4, a formant shift degree setting section 51, and a harmony thickness degree setting section. A means for generating and outputting each control parameter (harmony parameter control information) of the harmony sound synthesizing unit 7 from each value such as a harmony thickness degree designated by an operator operating a predetermined operation element in 5. . The harmony parameter control information includes a pitch, an amplitude, a formant shift amount, and a spectrum tilt coefficient. Hereinafter, each parameter of the harmony parameter control information will be described.

(1) Pitch parameter: The pitch parameter is obtained by adding or multiplying the pitch for each harmony sound specified from the harmony information 4 and the pitch information of the phoneme characteristic parameter. This parameter determines the pitch shift amount of the harmony sound. In generating the "pitch parameter", addition or multiplication of the pitch information of the phoneme feature parameters is performed, so that fine pitch fluctuation corresponding to each phoneme can be controlled.

(2) Amplitude parameter: harmony information 4
Is a parameter for determining the amplitude of each harmony sound in the harmony sound synthesis unit 7 obtained by adding or multiplying the amplitude for each sound designated from the above and the amplitude information of the phoneme feature parameter. In generating the “amplitude parameter”, the addition or multiplication of the amplitude information of the phoneme feature parameters is performed, so that a subtle amplitude variation corresponding to the phoneme can be controlled.

(3) Formant shift amount parameter:
The formant shift amount for each harmony sound is determined from three pieces of information. The first information is a formant shift reference value for each phoneme in the phoneme feature parameter. The second information is the formant shift degree set by the formant shift degree setting unit 51. This parameter is
An offset amount by which the formant shift amount of the harmony sound is shifted depending on the number of the harmony sound. As a way of holding the value, for example, it is assumed to have a cent value, and the value is multiplied by the harmony thickness (when it is set to 0 to 1.0), and the result is the offset amount of the formant shift amount of each harmony sound. Becomes The third information is a parameter determined based on the harmony thickness set by the harmony thickness setting unit 5.

The formant shift amount is determined based on these three pieces of information. That is, the formant shift amount is determined by the phoneme, the formant shift degree, and the thickness degree. A plurality of patterns of the formant shift reference value may be set in advance, and may be selectable by a selection operation using the harmony thickness degree setting unit 5. When the formants are shifted for each of the harmony sounds in accordance with the thickness of the harmony, an effect is obtained as if a plurality of people are singing, and the thickness of the harmony sound can be controlled.

(4) Spectral tilt coefficient: a parameter for determining the control condition of the spectral tilt control. To determine the spectral tilt coefficient, first, a difference between the input pitch and the output pitch of each sound is obtained from the pitch of the input voice from the voice analysis unit 2 * and the pitch information of each harmony sound in the harmony information 4. Based on the difference, a tilt change amount common to each phoneme corresponding to the difference is obtained from the tilt change amount table in the phoneme feature parameter storage unit 3. The amount of tilt change common to each phoneme,
From the tilt reference value in the phoneme feature parameter storage unit 3, a spectrum tilt change amount corresponding to the input phoneme is obtained, and further, a spectrum tilt coefficient is obtained from the spectrum tilt change amount corresponding to the input phoneme and the pitch difference value. it can. This makes it possible to specify a spectral tilt suitable for the phonemes and pitches of the input voice, and avoids the monotony that the spectral tilt is the same for all phonemes at any pitch, such as the harmony sound of the conventional device, and a natural Harmony sounds can be generated.

Next, the harmony sound synthesizer 7 of FIG. 1 will be described. The harmony sound synthesizing section 7 adds harmony sounds to an input audio signal and synthesizes them, and has N (N-channel) processing circuits 1 to N for adding harmony to the input audio signals. Note that a processing unit 71 is configured by the processing circuits 1 to N. Harmony sound synthesizer 7
Performs control in accordance with the harmony parameter control information for each harmony sound from the harmony parameter control unit 6 and adds each harmony sound as a control result to the addition unit 7
The result is added and output.

Here, as for the output of the original voice signal sung, for example, for one channel of the harmony sound (for example, channel 1), the output is performed without performing any processing. In the case of the original audio signal, for one channel, only a process for performing a delay process for synchronizing with a delay of another channel is performed, or a process having a smaller pitch shift as compared with other channels, for example, an input process is performed. A configuration may also be adopted in which only a pitch shift that corrects a pitch shift of a voice (a pitch shift from a melody) is performed and output. In addition, a path may be separately provided for the input voice to pass through, and the input voice may be synthesized by the adding unit 72.

Next, the mixing amplifier 8 adds the harmony sound from the harmony sound synthesizing section 7 and the accompaniment sound from the accompaniment sound playing section 9 and outputs the result.

FIG. 2 shows the harmony sound synthesizer 7 shown in FIG. 1 using an SMS analysis / synthesis method (SMS: Spectral Mod).
FIG. 3 is a block diagram illustrating an example of a configuration in the case of realizing by using eling synthesis (spectral modeling / synthesis). The SMS analysis / synthesis is described in Japanese Unexamined Patent Publication No. Hei 7-325583, "Sound Analysis and Synthesis Method and Apparatus", and Japanese Unexamined Patent Publication No. Hei 11-133959, "Speech Converter".

The SMS analyzer 701 converts the input voice signal into
After clipping in a predetermined frame unit, it is converted into a frequency spectrum by FFT (Fast Fourier Transform), a sine wave component and a residual component are extracted from the spectrum analysis result by SMS analysis, and output in frame units. N processing units 1 (702-1), processing units 2 (702-2),.
The processing unit N (702-1) is a signal processing circuit corresponding to each channel of the harmony sound, and based on N harmony parameter control information supplied to each circuit from the harmony parameter control unit 6, The sine wave component and the residual component are subjected to processes such as amplitude control, pitch control, and spectrum tilt, and output. Adder 70
3 adds and outputs the results processed by the processing units 1 to N (702-1 to N). Inverse FFT section 704 converts the added result into waveform information by inverse FFT and outputs the waveform information.

FIG. 3 shows the processing units 1 to N (702-1) of FIG.
FIG. 2 is a block diagram showing a configuration of the present invention. Each processing unit 702
(702-1 to N) are a control unit 7021 that performs pitch shift, formant shift, spectral tilt, and amplitude control on the sine wave component based on the harmony parameter control information and outputs the result. A residual component composite filter 7 that performs a filtering process for controlling the frequency component of the residual component based on information about the frequency such as the pitch shift of the
022.

According to the configurations shown in FIGS. 2 and 3, pitch, amplitude, formant shift, and tilt control can be performed in the frequency domain by using the SMS analysis / synthesis method, and the synthesis of each harmony sound is also possible. , Can be performed by inverse FFT after addition in the frequency domain, so that many harmony sounds can be easily synthesized.

(Operation of the Embodiment) According to the above-described configuration, the singing voice input from the microphone 1 is transmitted to the phoneme recognition unit 2.
In, the phoneme is recognized. According to the recognized phoneme, feature parameters for each phoneme are output from the phoneme feature parameter storage unit 3 and supplied to the harmony parameter control unit 6. The harmony parameter control unit 6 obtains a pitch shift amount for forming a pitch of a harmony sound, and further, based on the supplied phoneme feature parameters and output signals from the harmony thickness setting unit 5 and the formant shift degree setting unit 51, A harmony parameter including control for each phoneme is generated and supplied to the harmony synthesis unit 7. Thus, the harmony synthesizing unit 7 further performs fine pitch control, amplitude control, and formant shift corresponding to phonemes when generating a harmony sound that shifts the pitch of an input voice to a harmony pitch. As for the spectrum tilt, control corresponding to both phonemes and pitches is performed. Through the above-described processing, a natural and thick harmony sound is added to the singing voice, and a harmony effect with an unprecedented sound can be obtained.

The embodiment of the present invention can be constituted by a combination of a semiconductor integrated circuit for signal processing and a microprogram or the like set in the semiconductor integrated circuit. It can also be realized by a combination with a program to be executed. Further, in the case of a configuration including a computer and a program, the program executed by the computer can be recorded on a computer-readable recording medium and distributed.

[0034]

As described above, according to the present invention,
Because the harmony sound is accompanied by the characteristics of the singer's phonemes,
A more natural harmony sound can be obtained. Also, by setting the formant shift amount for each harmony sound, it is possible to obtain an effect as if a plurality of people were singing. Further, at this time, if the formant shift amount is set to a shift amount corresponding to each phoneme, unnaturalness of phonemes when the shift amount is fixed is eliminated, and a more natural effect can be obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of a harmony sound adding device according to the present invention.

FIG. 2 is a block diagram showing a configuration of a harmony sound synthesizer 7 in FIG. 1;

FIG. 3 is a block diagram showing a configuration of processing units 1 to N in FIG. 2;

FIG. 4 is a block diagram showing an embodiment of a conventional harmony sound adding device.

[Explanation of symbols]

1 microphone, 2 phoneme recognition unit, 2 * voice analysis unit, 3
Phoneme feature parameter storage unit, 4 ... Harmony information, 5 ... Harmony thickness degree setting unit, 6 ... Harmony parameter control unit,
7: Harmony sound synthesizer, 8: Mixing amplifier, 9:
Accompaniment part, 701 ... SMS analysis part, 702, 702-
1-701-N: processing unit, 703: adding unit, 704: inverse FFT unit, 7021: control unit, 7022: residual component composite filter.

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] December 9, 1999 (1999.12.
9)

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] 0029

[Correction method] Change

[Correction contents]

The SMS analyzer 701 converts the input voice signal into
After clipping in a predetermined frame unit, it is converted into a frequency spectrum by FFT (Fast Fourier Transform), a sine wave component and a residual component are extracted from the spectrum analysis result by SMS analysis, and output in frame units. N processing units 1 (702-1), processing units 2 (702-2),.
The processing unit N (702- N ) is a signal processing circuit corresponding to each channel of the harmony sound, and based on N pieces of harmony parameter control information supplied from the harmony parameter control unit 6 to each circuit. The sine wave component and the residual component are subjected to processes such as amplitude control, pitch control, and spectrum tilt, and output. Adder 70
3 adds and outputs the results processed by the processing units 1 to N (702-1 to N). Inverse FFT section 704 converts the added result into waveform information by inverse FFT and outputs the waveform information.

Continued on the front page (72) Inventor Alex Rothkos Spain Barcelona 08002 Merce 12 (72) Inventor Pedro Keino Spain Barcelona 08002 Merce 12 (72) Inventor Jodi Bonada Spain Barcelona 08002 Merce 12 F-term (reference) 5D015 BB02 KK02 KK04 5D045 AB30 5D108 BD07 BF02 BF06 5D378 JC02 JC10 KK02 MM97

Claims (1)

[Claims] 1. A phoneme recognizing means for recognizing a phoneme of an input voice signal, and a phoneme feature parameter for each phoneme prepared in advance comprising at least one of a pitch, an amplitude, a formant shift reference value, or a spectrum tilt reference value. A phoneme feature parameter storage means for storing a harmony pitch, and generating a harmony sound signal by pitch-shifting the input voice signal to a harmony pitch, based on phoneme information recognized by the phoneme recognition means when generating the harmony sound signal. Reading a phoneme feature parameter corresponding to the phoneme from the phoneme feature parameter storage means, and performing at least one of pitch shift, amplitude control, formant shift, or spectral tilt control for each phoneme according to the read phoneme feature parameter. Harmony sound synthesis means and Harmony sound adding device, characterized in that it comprises.