JP2003216147A - Encoding method of acoustic signal - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an encoding method of an acoustic signal capable of separating the acoustic signal in which a plurality of tones are mixed, into three or more tones. <P>SOLUTION: Frequency analysis is performed to the acoustic signal and separated into time series phonemic data to be constituted of starting time, ending time, frequency and strength (S1). Then, a unit tone parameter is calculated based on a distribution state of other pieces of phonemic data to be overlapped on the respective pieces of phonemic data at the same time and with different frequencies (S2). Next, pieces of the phonemic data of which the frequency and time are similar among the respective pieces of phonemic data are linked with one piece of linked phonemic data (S3). Furthermore, a linked tone parameter is calculated based on temporal changes of the frequency/a strength value of the phonemic data to be a base to be linked for the respective pieces of the linked phonemic data (S4). The linked phonemic data are sorted into a plurality of groups based on the calculated unit tone parameter and the linked tone parameter (S5). Thus, code data sorted into a plurality of tone groups can be obtained. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

The present invention relates to broadcast media (radio, television), communication media (CS video / audio distribution, Internet music distribution, communication karaoke), package media (CD, MD, cassette, video, LD, CD). −
Production of various audio contents provided in ROM, game cassettes, solid-state memory media for portable music players, etc., as well as music performance recording signals, musical score publishing, MIDI data for communication karaoke distribution, and automatic performance data for electronic musical instruments with performance guide function. , Automatic music transcription technology for automatically creating ringing melody data for mobile phones, PHS, pagers, etc.

[0002]

2. Description of the Related Art A time series signal represented by an acoustic signal is
A plurality of periodic signals are included as its constituent elements. Therefore, a method of analyzing what kind of periodic signal is included in a given time series signal has been known for a long time. For example, Fourier analysis is widely used as a method for analyzing frequency components included in a given time series signal.

By using such a time-series signal analysis method, it is possible to encode an acoustic signal. With the spread of computers, it has become easy to sample the analog sound signal that is the original sound at a predetermined sampling frequency, quantize the signal strength at each sampling, and capture it as digital data. If a method such as Fourier analysis is applied to the data and the frequency components included in the original sound signal are extracted, the original sound signal can be encoded by the code indicating each frequency component.

On the other hand, MIDI (Musical Instrume) was born from the idea of encoding musical instrument sounds by electronic musical instruments.
The nt Digital Interface) standard has also been actively used with the spread of personal computers. Code data according to this MIDI standard (hereinafter referred to as MID
Basically, the I data) is data that describes the operation of the musical instrument playing, such as which keyboard key of the musical instrument was played and with what strength. The MIDI data itself contains the actual sound. Waveform not included. Therefore, when reproducing the actual sound, the MI that stores the waveform of the instrument sound is stored.
Although a DI sound source is required separately, its high coding efficiency has been attracting attention, and the MIDI coding and decoding technology is currently used for musical instrument performance, musical instrument practice, composition, etc. using a personal computer. It is widely adopted in software that does.

Therefore, a time-series signal typified by an acoustic signal is analyzed by a predetermined method to extract a periodic signal which is a constituent element thereof, and the extracted periodic signal is M
Proposals have been made to encode using IDI data. For example, JP-A-10-247099, JP-A-11-73199, and JP-A-11-73200.
JP, JP-A-11-95753, JP, 2000
In JP-A-999009, JP-A-2000-99092, and JP-A-2000-99093, it is possible to analyze a frequency that is a component of an arbitrary time-series signal and create MIDI data from the analysis result. Various possible methods have been proposed.

[0006]

In recent years, research on the coding of acoustic signals has progressed, and attempts have been made to separately code the acoustic signals in which a plurality of timbres are mixed for each timbre. Specifically, a method of analyzing a multi-channel input signal using a 2-channel stereo or a microphone array by using an independent component analysis is generally used. However, in the recorded record medium, the source is often monaural, and it is extremely unusual and impractical when a multi-channel signal such as a microphone array is provided. Therefore, the present applicant has proposed the following method as a method applicable to monaural signals.

Basically, the MIDI encoding method proposed in each of the above-mentioned publications or specifications is used. However, in Japanese Patent Application No. 2000-319175, the degree of frequency fluctuation, and in Japanese Patent Application No. 2001-321968. Overtone distribution degree / duration, Japanese Patent Application No. 2001-8750
In No. 1, the level distribution degree is used as a parameter to calculate 1
Channel division is performed by dimensional threshold processing. With these, it is possible to separate into two representative sound sources such as piano and vocal, but the number of channels that can be divided is only two, and the problem that phonemes that span both are distributed to either one. is there.

In the method using the tone color management database proposed in Japanese Patent Application No. 2001-35378, a piano
A database of spectrogram patterns is constructed for each timbre group such as vocals, matching is performed with the analyzed phoneme patterns in a brute force manner, and timbre groups with a high matching rate are selected. With this method, it is possible to separate as many tone color groups as possible by expanding the database. However, a huge number of matching operations must be performed, and it is difficult to obtain the desired separation accuracy inefficiently. Also in this method, the phonemes spanning both groups were distributed to either one.

In view of the above points, it is an object of the present invention to provide an audio signal encoding method capable of separating an audio signal in which a plurality of tones are mixed into three or more tones. To do.

[0010]

In order to solve the above problems, according to the present invention, as a method of encoding an acoustic signal, a frequency analysis is performed on a given acoustic signal to start / end time / frequency / strength. A phoneme data generation step for decomposing into time-series phoneme data composed of, unit timbre parameters based on a distribution state of other phoneme data having different frequencies at the same start time and the same end time for each of the phoneme data. A unit timbre parameter calculation step of calculating and assigning to each of the phoneme data, a frequency of a plurality of temporally continuous phoneme data in each of the phoneme data is similar, and is followed by the end time of the preceding phoneme data. If the start times of the unit phoneme data are similar, the preceding phoneme data and the following phoneme data are integrated into one concatenated phoneme data,
Generates concatenated phoneme data composed of the start time of the preceding phoneme data, the end time of the following phoneme data, and the frequency, intensity value, and unit tone color parameters of either the preceding phoneme data or the following phoneme data. Phoneme connection step, for each of the connected phoneme data, a connected tone color parameter is calculated based on a distribution state of frequency / intensity values of a plurality of phoneme data that are configured, and a connected tone color parameter calculation is given to each of the connected phoneme data. Step, based on a plurality of parameters including at least unit tone color parameters and connected tone color parameters of each connected phoneme data, performing a phoneme classification step of classifying each connected phoneme data into a plurality of tone color groups, start time and end time・ Frequency / intensity value ・ Character data composed of the classified tone color groups is obtained. To. According to the present invention, a plurality of tone color parameters such as a unit tone color parameter obtained from the characteristics of phoneme data obtained by frequency analysis, and a connected tone color parameter obtained from the characteristics of connected phoneme data obtained by connecting phoneme data. Since the concatenated phoneme data is classified into a plurality of timbre groups based on the above, it becomes possible to separate an acoustic signal in which a plurality of timbres are mixed into two or more timbres, and in particular, it is separated into three or more timbres. It is also suitable for doing.

[0011]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below in detail with reference to the drawings. (1. Basic Principle of Acoustic Signal Coding Method) First, the basic principle of the acoustic signal coding method according to the present invention will be described.
Since this basic principle is disclosed in each of the above-mentioned publications or specifications, only the outline thereof will be briefly described here.

As shown in FIG. 1A, it is assumed that an analog acoustic signal is given as a time series signal. In the example of FIG. 1, the horizontal axis represents time t, and the vertical axis represents amplitude (intensity) to show this acoustic signal. Here, first, a process of taking in the analog acoustic signal as digital acoustic data is performed. This uses the conventional general PCM method,
The analog acoustic signal may be sampled at a predetermined sampling frequency, and the amplitude may be converted into digital data by using a predetermined number of quantization bits. here,
For convenience of explanation, the waveform of the acoustic data digitized by the PCM method is also shown as the same waveform as the analog acoustic signal of FIG.

Then, a plurality of unit sections are set on the time axis of the acoustic signal to be analyzed. In the example shown in FIG. 1A, six times t1 to t6 are equally spaced on the time axis t.
Is defined, and five unit sections d1 to d5 whose start point and end point are the respective time points are set. In the example of FIG. 1, all unit sections having the same section length are set without overlapping on the time axis, but the section setting is performed so that adjacent unit sections partially overlap on the time axis. It doesn't matter.

When the unit section is set in this way, a representative frequency is selected for each acoustic signal (hereinafter referred to as section signal) for each unit section. Each section signal usually contains various frequency components,
For example, a frequency component having a large intensity ratio of the components may be selected as the representative frequency. Here, the representative frequency is generally a so-called fundamental frequency, but a harmonic frequency such as a formant frequency of voice or a peak frequency of a noise sound source may be treated as a representative frequency. Although only one representative frequency may be selected, more accurate encoding becomes possible if a plurality of representative frequencies are selected depending on the acoustic signal. FIG. 1B shows an example in which three representative frequencies are selected for each unit section and one representative frequency is encoded as one representative code (in the figure, it is shown as a note for convenience). Has been done. Here, three tracks T1 for accommodating a representative code (note) are provided.
T2 and T3 are provided so that the three representative codes selected for each unit section are accommodated in different tracks.

For example, the representative codes n (d1,1), n (d1,2), n (d1,) selected for the unit section d1.
3) are housed in the tracks T1, T2, T3, respectively. Here, each code n (d1,1), n (d1,
2) and n (d1,3) are codes indicating note numbers in the MIDI code. The note number in the MIDI code takes 128 values from 0 to 127, and each indicates one key on the keyboard of the piano. Specifically, for example, the representative frequency is 440 Hz
Is selected, the frequency is note number n = 6
Since it corresponds to 9 (corresponding to "Ra sound (A3 sound)" at the center of the keyboard of the piano), n = 69 is selected as the representative code. However, FIG. 1B is a conceptual diagram showing the representative code obtained by the above-described method in the form of a musical note, and in fact, each musical note is also provided with data relating to its strength. For example, the track T1 includes note numbers n (d1,1), n (d2,1) ... Pitch data and e (d1,1), e (d
Data indicating the strength of 2, 1, 1 ... The data indicating this intensity is determined by the degree to which the component of each representative frequency is included in the original section signal. Specifically, the data indicating the intensity is determined based on the correlation value of the section signal of the periodic function having each representative frequency. Also, FIG.
In the conceptual diagram shown in (b), the position of each unit section on the time axis is shown by the position of the note in the lateral direction. The data shown as is added to each note.

It is not always necessary to adopt the MIDI format as the format for encoding the acoustic signal, but since the MIDI format is the most popular as this type of encoding format, the MIDI format code data is practically used. Is preferably used. In the MIDI format, “note on” data or “note off” data exists with “delta time” data interposed. The "Note On" data is
The "note-off" data is data for instructing the start of playing a specific sound by designating a specific note number N and velocity V. The "note-off" data is a data of a specific note designated by a specific note number N and velocity V. This is data for instructing the end of performance. The "delta time" data is data indicating a predetermined time interval. Velocity V is a parameter indicating, for example, the speed at which the piano keyboard is pushed down (velocity at note-on) and the speed at which the finger is released from the keyboard (velocity at note-off), and operation to start playing a specific sound. Alternatively, it indicates the strength of the performance ending operation.

In the above method, the i-th unit section di
About J note numbers n (d
i, 1), n (di, 2), ..., N (di, J) are obtained for each of these intensities e (di, 1), e
(Di, 2), ..., E (di, J) are obtained. Therefore, the code data in the MIDI format can be created by the following method. First, as the note number N described in the “note on” data or the “note off” data, the obtained note number n (d
i, 1), n (di, 2), ..., N (di, J) may be used as they are. On the other hand, as the velocity V described in the “note-on” data or the “note-off” data, the obtained intensities e (di, 1), e (d
i, 2), ..., E (di, J) may be standardized by a predetermined method. The “delta time” data may be set according to the length of each unit section. In the description here, in order to explain the basic principle in an easy-to-understand manner, the concept of "track" is used to distribute each code, but in the MIDI standard,
A plurality of codes are recorded on one channel. Currently, one channel per MIDI standard
Since it is possible to pronounce 6 to 64 sounds at the same time,
There is no concept of a track, and the three codes shown in FIG. 1 are recorded in one channel and sounded as a chord. In this sense, the channels described later are different from the tracks described above.

(2. Concrete Method for Obtaining Correlation with Periodic Function) In the method based on the above-mentioned basic principle, one or a plurality of representative frequencies are selected for the section signal and have the representative frequency. The section signal is represented by the periodic signal. Here, the selected representative frequency is literally a frequency representing the signal component in the unit section. As a specific method of selecting the representative frequency, there are a method using a short-time Fourier transform and a method using a generalized harmonic analysis method, as described later. In either method, the basic idea is the same, and a plurality of periodic functions with different frequencies are prepared in advance as harmonic signals, and the correlation with respect to the section signal in the unit section is selected from the plurality of periodic functions. A method of finding a high periodic function and selecting the frequency of this highly correlated periodic function as the representative frequency is adopted. That is, when the representative frequency is selected, the calculation for obtaining the correlation between the plurality of periodic functions prepared in advance and the section signal in the unit section is performed. Therefore, here, a specific method for obtaining the correlation with the periodic function will be described.

It is assumed that a trigonometric function as shown in FIG. 2 is prepared as a plurality of periodic functions. These trigonometric functions are composed of a pair of a sine function and a cosine function having the same frequency, and 128 standard frequencies f (0) to
For each of f (127), a pair of sine and cosine functions will be defined. Here, a pair of functions including a sine function and a cosine function having the same frequency will be defined as a periodic function for the frequency. That is, the periodic function for a specific frequency is composed of a pair of sine function and cosine function. Thus, the reason why the periodic function is defined by a pair of sine function and cosine function is to consider that the correlation value is influenced by the phase when the correlation value of the periodic function with respect to the signal is obtained. The variables F and k in each trigonometric function shown in FIG. 2 are variables corresponding to the sampling frequency F and the sample number k for the interval signal X. For example, a sine wave for the frequency f (0) is represented by sin (2πf (0) k / F), and given an arbitrary sample number k, the same time position as the kth sample forming the interval signal is given. The amplitude value of the periodic function at is obtained. Here, 128 standard frequencies f (0)
~ F (127) is defined by the following [Formula 1].

[Formula 1] f (n) = 440 × 2 ^{γ (n)} γ (n) = (n−69) / 12 where n = 0, 1, 2, ..., 127

Defining the standard frequency by such an equation is convenient when finally performing encoding using MIDI data. This is because there are 128 standard frequencies f (0) to f (12) set by such a definition.
7) is to take frequency values forming a geometric series, and M
This is because the frequency corresponds to the note number used in the IDI data. Therefore, 128 shown in FIG.
The standard frequencies f (0) to f (127) are the frequencies set at equal intervals (semitone unit in MIDI) on the frequency axis shown by the logarithmic scale. Therefore, in the present application, all the note number axes in the graphs shown in the figures will be shown on a logarithmic scale.

(2.1. Short-Time Fourier Transform Method) Next, a specific description will be given of how to obtain the correlation of each periodic function with respect to the section signal of an arbitrary section. For example, as shown in FIG. 3, it is assumed that the section signal X is given to a certain unit section d. Here, it is assumed that the unit section d having the section length L is sampled at the sampling frequency F and w sample values are obtained in total, and the sample number is 0, as shown in the figure. 1, 2, 3,
..., k, ..., w-2, w-1 (the w-th sample indicated by a white circle is a sample included at the beginning of the next unit section adjacent to the right). In this case, for any sample number k, the amplitude value X (k) is given as digital data. In the short-time Fourier transform, it is usual to multiply X (k) by a window function W (k) such that the center weight is close to 1 and the weights at both ends are close to 0 for each sample. That is, the following correlation calculation is performed by treating X (k) × W (k) as X (k), and a cosine wave-shaped Hamming window is generally used as the shape of the window function. Where w
Is also described as a constant in the following description, but it is generally an integer multiple of F / f (n) that varies depending on the value of n and becomes maximum within the range L. It is desirable to set to.

The principle of obtaining the correlation value with the sine function Rn having the nth standard frequency f (n) for such a section signal X will be described. The correlation value A (n) between the two can be defined by the following [Formula 2].

[Formula 2] A (n) = (2 / w) Σ _{k = 0, w−1} x (k) sin (2πf _n k / F) B (n) = (2 / w) Σ _{k = 0, w-1} x (k) cos (2πf _n k / F) {E (n)} ² = {A (n)} ² + {B (n)} ²

In the above [Formula 2], X (k) is the amplitude value of the sample number k in the interval signal X, as shown in FIG. 3, and sin (2πf _n k / F) is on the time axis. It is the amplitude value of the sine function Rn at the same position of. In order to avoid complicated expressions, f
(N) is expressed as f _n . The first arithmetic expression of [Equation 2] is an expression for obtaining the inner product of the amplitude value of the interval signal X and the amplitude vector of the sine function Rn for the dimensions of all sample numbers k = 0 to w−1 in the unit interval d. Can be said.

Similarly, the second arithmetic expression of the above [Equation 2] is an expression for obtaining the correlation value between the interval signal X and the cosine function having the nth standard frequency f (n). The correlation value of is given by B (n). Note that both the first arithmetic expression for obtaining the correlation value A (n) and the second arithmetic expression for obtaining the correlation value B (n) are finally multiplied by 2 / w. Is for normalizing the correlation value, and since w is generally changed depending on n as described above, this coefficient is also a variable depending on n.

The effective value of the correlation between the interval signal X and the standard periodic function having the standard frequency f (n) is the third value of the above [Formula 2].
As shown in the following equation, of the square root of the sum of squares of the correlation value A (n) with the sine function and the correlation value B (n) with the cosine function,
It can be indicated by a positive value, E (n). If the frequency of the standard periodic function having a large effective value of the correlation is selected as the representative frequency, the interval signal X
Can be encoded.

That is, one or a plurality of standard frequencies whose correlation value E (n) is greater than a predetermined standard may be selected as the representative frequency. The selection condition that “the correlation value E (n) is greater than or equal to a predetermined reference” is, for example, a threshold that is set in advance and the correlation value E (n) exceeds the threshold. Frequency f (n)
May be set as a representative frequency, but an absolute selection condition may be set. For example, relative selection conditions such as selecting up to the Qth in the order of the magnitude of the correlation value E (n). May be set.

(2.2. Generalized Harmonic Analysis Method) Here,
A method of generalized harmonic analysis useful for encoding an acoustic signal according to the present invention will be described. As already described, when the acoustic signal is encoded, some representative frequencies having a high correlation value are selected for the section signals in each unit section. Generalized harmonic analysis is a method that enables selection of representative frequencies with higher accuracy, and its basic principle is as follows.

It is assumed that the signal S (j) exists in the unit section d as shown in FIG. 4 (a). here,
As will be described later, j is a parameter for iterative processing (j = 1 to J). First, for this signal S (j), correlation values for all 128 periodic functions as shown in FIG. 2 are obtained. Then, the frequency of one periodic function for which the maximum correlation value is obtained is selected as the representative frequency,
A periodic function having the representative frequency is extracted as an element function. Then, the inclusion signal G as shown in FIG.
Define (j). The content signal G (j) is added to the extracted element function as its amplitude and the signal S of the element function.
It is a signal obtained by multiplying the correlation value for (j). For example, as shown in FIG. 2 as a periodic function, when a pair of sine function and cosine function are used and the frequency f (n) is selected as the representative frequency, the sine function A (n) having the amplitude A (n) is used. ) Sin (2πf _n k / F) and amplitude B
Cosine function B (n) cos (2πf _n k / with (n)
The signal composed of the sum of F and F is the content signal G (j) (in FIG. 4B, only one function is shown for convenience of illustration). Here, since A (n) and B (n) are normalized correlation values obtained by the above [Formula 2], the inclusion signal G (j) is eventually included in the signal S (j). It can be said that it is a signal component having the frequency f (n) that is set.

When the content signal G (j) is obtained in this way, the difference signal S (j + 1) is obtained by subtracting the content signal G (j) from the signal S (j). Figure 4 (c) shows
The difference signal S (j + 1) thus obtained is shown. This difference signal S (j + 1) is the original signal S
It can be said that the signal is composed of the remaining signal components obtained by removing the signal component having the frequency f (n) from (j). Therefore, by increasing the parameter j by 1, the difference signal S (j + 1) is treated as a new signal S (j), and the same processing is performed for the parameter j from j = 1 to 1.
If you repeat J times while incrementing by 1 to J, J
Individual representative frequencies can be selected.

The J contained signals G (1) to G (J) output as a result of such a correlation calculation are signals which are constituent elements of the original section signal X, and are the original section signals. When X is coded, information indicating the frequency and amplitude (intensity) of these J contained signals may be used as code data. Although it has been described that J is the number of representative frequencies, it may be the same as the number of standard frequencies f (n), that is, J = 128, and it is customary to do so for the purpose of obtaining the frequency spectrum. Is.

In this way, when a predetermined number of frequency groups are selected for each unit section, "information indicating the pitch of the sound" corresponding to each frequency of this frequency group, corresponding to the signal strength of each selected frequency. "Information indicating the sound intensity", "Information indicating the sound production start time" corresponding to the start point of the relevant unit section, "Indicating the sound production end time corresponding to the start point of the unit section subsequent to the relevant unit section" By generating a predetermined number of code data including four pieces of information, it is possible to encode the section signal X in the unit section with a predetermined number of code data. If MIDI data is created as code data, note number is used as "information indicating pitch of tone", velocity is used as "information indicating intensity of tone", and "start time of sound generation" Note-on time is used as "information to indicate", and "information to indicate the end time of sound generation"
The note-off time may be used as

(3.1. Audio Signal Coding Method According to the Present Invention) The audio signal coding method according to the present invention will be described below with reference to the flowchart shown in FIG. First, a unit section is set over the entire section on the time axis of the acoustic signal, frequency analysis is performed to calculate an intensity value corresponding to each frequency, and four values of frequency, intensity value, start point and end point of the unit section are set. Phoneme data consisting of information is calculated (step S
1). The method of frequency analysis in step S1 is
Above (basic principle) and (specific method of frequency analysis)
Is as described in the section. Here, the unit section may be set by overlapping consecutive unit sections, or as shown in FIG. 1A, the end point of the preceding unit section and the start point of the following unit section are the same point. However, in the following description, for convenience, the latter setting will be described as an example.

To calculate the phoneme data, specifically, the correlation strength of the interval signal is calculated for 128 types of periodic functions as shown in FIG. 2, the frequency of the periodic function, the calculated correlation strength, and the unit interval. It is performed by defining four pieces of information of the starting point and the ending point of the phoneme data. However, in this embodiment, phoneme data corresponding to all prepared periodic functions are acquired instead of selecting a representative frequency as in the case of the above-described basic principle. The phoneme data [m, n] is obtained by performing the process of step S1 for all unit intervals.
A group of (0 ≦ m ≦ M−1, 0 ≦ n ≦ N−1) is obtained. Here, N is the total number of periodic functions (N = 12 in the above example).
8) and M are the total number of unit sections set in the acoustic signal. That is, a phoneme data group including M × N phoneme data is obtained.

Subsequently, a unit tone color parameter is calculated using the start time, end time, frequency, and intensity value, which are components of the obtained phoneme data (step S2). In this embodiment, the fluctuation distribution parameter Py and the overtone distribution parameter Po are calculated as the unit tone color parameters. Furthermore, when a stereo signal is used as an acoustic signal,
The velocity is calculated for each of the right and left sides, and the stereo localization parameter Ps is calculated based on the ratio.

Specifically, the fluctuation distribution parameter Py is a parameter indicating a local frequency fluctuation distribution, and the fluctuation distribution parameter Py (n) corresponding to each note number n is calculated by the following [Equation 3]. It is calculated.

[Formula 3] Py (n) = {V (n-1) + V (n + 1) + 2V (n-2) + 2V (n + 2)
} × 6 / V (n)

As shown in the above [Formula 3], the fluctuation distribution parameter has a tone (note number n-1) lower by one semitone than itself for each phoneme data (note number n),
Higher half tone (note number n + 1), lower half tone (note number n-2), higher tone by two semitones (note number n + 2) indicates how strong the intensity value V is compared to itself. Is. Since this fluctuation distribution parameter Py is standardized in the range of 0 to 11 in [Equation 3], the closer Py is to 0, the higher the piano sound tendency, and the closer Py is to 11, the vocal sound is. The tendency is high.

The harmonic overtone distribution parameter Po is a value for determining whether the unit phoneme data is a fundamental sound or a harmonic overtone of other unit phoneme data. Specifically, the harmonic distribution parameter Po (n) corresponding to the note number n is calculated using the following [Formula 4].

[Formula 4] Po (n) = {6V (n) + V (n + 12) + V (n + 19) + V (n + 24)
+ V (n + 28) + V (n + 31) + V (n + 34) + V (n + 36)-V (n-12)-
V (n-19)-V (n-24)-V (n-28)-V (n-31)-V (n-34)-
V (n-36)} / V (n)

In the above [Formula 4], V (n) represents the intensity value of the note number n, and V (n + 12), V (n + 19), V
(n + 24), V (n + 28), V (n + 31), V (n + 34), V (n + 36) are the second overtone, the third overtone, and the fourth overtone of the note number n, respectively.
The intensity value of the 5th overtone, 6th overtone, 7th overtone, and 8th overtone is V (n-12)
, V (n-19), V (n-24), V (n-28), V (n-31), V (n-34), V (n
-36) indicates the intensity value of the basic sound when the note number n is assumed to be the second overtone, the third overtone, the fourth overtone, the fifth overtone, the sixth overtone, the seventh overtone, and the eighth overtone, respectively. After all, the overtone distribution parameter Po (n) calculated by the above [Formula 4] is
It is standardized to fall within the range of 0 to 11, and when there are many sounds with frequencies that are integral multiples of itself, that is, in the case of basic sounds, the value is close to 11 and is a fraction of its own integer.
When there are many sounds with the frequency of, that is, in the case of overtones, the value is close to 0.

When a stereo signal is used as the acoustic signal, frequency analysis is performed on the signal from each channel in step S1 to calculate phoneme data. Therefore, M × N pieces of phoneme data are obtained for each channel, but since the start point, the end point, and the frequency of each unit section are the same, the phoneme data of both channels are collected, and L as intensity value
The intensity value from the (left) channel and the intensity value from the R (right) channel are set as V _L and V _R , respectively. Then, in step S2, the stereo localization parameter Ps is calculated as one of the unit tone color parameters by the following [Formula 5].

[0044] [Equation 5] Ps (n) = 6-6 [{ V L (n) - V R (n)} / V R (n)] 1/2: V L (n)> V R (n ) when _{= 6 + 6 [{V R} (n) - V L (n)} / V L (n)] 1/2: when _{V R (n)> V L} (n) = 6: V R (n ) = V _L (n)

The stereo localization parameter Ps is standardized so as to fall within the range of 0 to 11 in [Equation 5], and the maximum value 6 is obtained when the intensity value of the left and right channels is the same for the sound of a certain note number. And the larger the intensity value of the left channel compared to the intensity value of the right channel,
It takes a value close to 0, and as the intensity value of the right channel is larger than the intensity value of the left channel, it takes a value closer to 11. In general, a musical instrument sound is often recorded biased to one of the channels as compared with vocals. Therefore, when the value of the stereo localization parameter Ps is far from 6, it can be determined to be a musical instrument sound. .

When the unit timbre parameters are calculated, a plurality of phoneme data which have the same frequency and are continuous in the time series direction are connected as one connected phoneme data (step S3). In order to perform this process, phoneme data whose intensity value does not reach a predetermined standard in the process of step S1 or step S2 is deleted in advance. In this case, as the predetermined reference, a reference is set to the extent that it is determined that the sound is noise or the like and is not the target performance recording signal. FIG. 6 is a conceptual diagram for explaining connection of phoneme data. FIG. 6A is a diagram showing a state of a phoneme data group before connection. In FIG. 6 (a), each of the rectangles partitioned in a grid shows phoneme data, and the shaded rectangles are deleted because the intensity value does not reach the predetermined standard in step S3. It is phoneme data, and other rectangles show valid phoneme data. In step S3, the same frequency (same note number)
In order to connect the phoneme data continuous in the time t direction, the concatenation process is performed on the phoneme data group shown in FIG. 6A to obtain a connected phoneme data group as shown in FIG. 6B. For example, the phoneme data A1 and A shown in FIG.
2 and A3 are connected to obtain connected phoneme data A as shown in FIG. 6 (b). At this time, a frequency common to the phoneme data A1, A2, A3 is given as the frequency of the newly obtained concatenated phoneme data A, and the intensity value is the maximum of the intensity values of the phoneme data A1, A2, A3. The section start time t1 of the first phoneme data A1 is given as the start time, and the section end time t4 of the last phoneme data A3 is given as the end time. For both phoneme data and concatenated phoneme data, frequency (note number), intensity value, start time, end time 4
Since three phoneme data are integrated into one concatenated phoneme data, the data amount is 3
It is reduced by a factor of 1. This means that, when finally encoded in MIDI, it is represented as one long note instead of three short notes.

Further, in step S3, the unit tone color parameter of the phoneme data having the maximum intensity value among the phoneme data which is the source of the connection is set as the unit tone color parameter of the connected phoneme data.

Subsequently, the connected tone color parameters are calculated using the start time, end time, frequency, and intensity value, which are components of the obtained connected phoneme data (step S4). In this embodiment, the level distribution parameter Pl is calculated as the connected tone color parameter. Specifically, the level distribution parameter Pl (n) corresponding to each note number n is a parameter that indicates the distribution of intensity values in the time series direction, and is one of the original unit phoneme data that is connected to one connected phoneme data. , V _{i is} the intensity value of the preceding unit phoneme data, and V _{i + 1} is the intensity value of the following unit phoneme data, the following formula 6 is calculated.

[Equation 6] Pl (n) = 36 × Σ _{i = 0, I} | V _{i + 1} −V _i | × (t _{i + 1} −
t _i ) / V _max Σ _{i = 0, I} (t _{i + 1} −t _i ).

In the above [Equation 6], I is the number of the basic phoneme data to be connected to one concatenated phoneme data minus one, and t _i is the time of the start point of each phoneme data. Is shown. Further, V _max indicates the maximum _value of V _{0 to} V _I. The reason that 36 is multiplied at the beginning is to standardize the value of Pl (n) in the range of 0 to 11. The calculated concatenated tone color parameter is added as one element of the concatenated phoneme data in addition to the parameters of start time, end time, frequency, intensity value, and unit tone color.

When the connected tone color parameters are calculated, each connected phoneme data is classified into a plurality of tone color groups based on the unit tone color parameter and the connected tone color parameter given to each connected phoneme data (step S5). In particular,
A weighting coefficient for classification is prepared in advance for each output channel, and an appropriate value for each group of each concatenated phoneme data is calculated by the following [Equation 7] using these weighting coefficients and each timbre parameter. calculate.

[Equation 7] Ch1 = a ₁ n + a ₂ D + a ₃ V + a ₄ Py + a ₅ Po + a ₆
Ps + a ₇ Pl Ch2 = b ₁ n + b ₂ D + b ₃ V + b ₄ Py + b ₅ Po + b ₆
Ps + b ₇ Pl Ch3 = c ₁ n + c ₂ D + c ₃ V + c ₄ Py + c ₅ Po + c ₆
Ps + c ₇ Pl

In the above [Formula 7], a ₁ ... A ₇ , b _{1
··· b 7, c 1 ··· c} 7 each connecting phoneme data note number (frequency), duration (end time - starting time), velocity (intensity), fluctuation distribution parameters, harmonic distribution parameter, stereo position parameters , Are weighting factors for classification corresponding to seven parameters including level distribution parameters. In [Formula 7], Ch1, C
h2 and Ch3 are classified group numbers, and MID
When encoding to the I standard, it corresponds to the channel. Each piece of connected phoneme data will be classified into a group in which its proper value is maximum. For example, the weighting coefficient a ₁
The ·· a ₇ piano, b ₁ ··· b ₇ mixture of piano and vocals,
When c ₁ ... c ₇ is set to a value matched to the vocal characteristic, the value of Ch1 becomes large in the connected phoneme data having the characteristic of piano, and the value of Ch1 becomes large in the connected phoneme data having the characteristic of the mixed sound of piano and vocal. The value of Ch2 becomes large, and the value of Ch3 becomes large in the concatenated phoneme data having the vocal characteristic, and the value is classified into three types from the value ratio of Ch1, Ch2, and Ch3. When none of the values of Ch1, Ch2, and Ch3 is significantly increased, the phoneme data is unclassifiable by the set weighting factor. Analysis using a large number of variables in this way is called multivariate analysis.

As described above, it is preferable that the audio signal is converted into the widely used MIDI standard. In this case, the concatenated phoneme data is converted into MIDI code data, but classification into a plurality of groups in step S5 may be performed after conversion into MIDI code data.

(3.2. Other Embodiments) In the first embodiment, all processing is performed by the arithmetic processing unit such as a computer, but the state of the connected phoneme data obtained from the acoustic signal is confirmed. However, it is also possible for the operator to determine a specific threshold value for each timbre parameter. Here, such a second embodiment will be described. A flow chart of the second embodiment of the present invention is shown in FIG. As shown in FIG. 7, also in the second embodiment, the process up to the calculation of the connected tone color parameter in step S4 is performed in the same manner as in the first embodiment. In the second embodiment, when the connected tone color parameters are calculated, a group that is a classification destination of the connected phoneme data is designated (step S
11). For example, "piano group", "vocal group", etc. are designated. In the case where the process after step S11 is performed after the MIDI conversion has already been performed, the channel which is the recording destination of the code data is designated as the group. For example, a channel for recording a piano sound and a channel for recording a vocal sound are designated. For example, when recording a piano sound on channel ch1, ch1 is specified and GM (General
Based on the MIDI standard, set the program number "1" of the piano to ch1.

Then, threshold values of each tone color parameter for classifying the connected phoneme data into the designated group are set (step S12). For this setting, first, a connected phoneme data display screen as shown in FIG. 8 is displayed on the display device. In FIG. 8, the horizontal axis represents time and the vertical axis represents scale (MIDI standard note number). On the vertical axis, the keyboard is shown at the left end, and each key on the keyboard corresponds to one note number. On such a screen, each connected phoneme data is displayed as a rectangle. In the rectangle indicating the concatenated phoneme data, the left end position indicates the start time, the right end position indicates the end time, the upper end position indicates the scale, and the upper and lower lengths indicate the intensity value.

On the connected phoneme data display screen shown in FIG. 8, rectangles representing the connected phoneme data are displayed in different colors. The number of colors can be set as necessary, but in the present embodiment, a maximum of 12 colors can be displayed. Actually, when one threshold value is specified, the connected phoneme data having the tone color parameter within the range of less than the threshold value (0 to threshold value −1) or more than the threshold value (threshold value to 11) is extracted. Connected phoneme data having timbre parameters within one threshold range is extracted. Further, if the two threshold values are set such that the end point is smaller than the start point, such as "10 and 1," it is interpreted that "10, 11, 0, 1" is designated. If the extracted connected phoneme data and the other connected phoneme data are set to be classified into two channels and the color-coding mode of the display screen is set to the channel, the connected phoneme data is displayed in two colors. At this time, all the connected phoneme data and the extracted connected phoneme data may be overlapped and classified into two channels. As a concrete threshold value setting method using the connected phoneme data display screen, the threshold value of the tone color parameter is set after the type of the tone color parameter is designated. At this time, if the tone color parameters have values of 0 to 11, it is possible to perform color-coded display according to the tone color parameter threshold value.

For example, when the fluctuation distribution parameter is set as the timbre parameter and the threshold value is set to "less than 6," the value of the fluctuation distribution parameter is 6 or more (6 to 11) on the connected phoneme data display screen. Or less than 6 (0 to
Depending on whether it is 5), it will be divided into two colors.
For example, connected phoneme data having a fluctuation distribution parameter value of less than 6 is red (piano), and the fluctuation distribution parameter value is 6
The above connected phoneme data is displayed in blue (other than piano, for example, vocal). Here, the operator looks at the shape, position, and arrangement of the rectangle that represents the connected phoneme data, and checks whether the connected phoneme data showing the shape, position, and arrangement of the piano sound is displayed in blue. Check. If the shape / position / arrangement state does not match the displayed color state, the value of the fluctuation distribution parameter is changed and color-coded display is performed again. In this way, changing the threshold and repeating the color-coded display, if the state of the shape, position, and arrangement of the piano sounds and the state of the displayed color match,
The threshold value of the fluctuation distribution parameter is determined, and a value less than the threshold value is set to be taken into ch1 which is a channel for recording a piano sound.

Similarly, the threshold values are determined while color-coded display is performed for other unit tone color parameters such as overtone distribution parameters, stereo localization parameters, and level distribution parameters that are connected tone color parameters. In this way, the concatenated phoneme data classified by the logical sum of the maximum four parameters with the threshold value specified for one channel is determined. Further, the thresholds of the tone color parameters are similarly set for the other channels.

When the tone color parameters to be classified into each group (channel in the case of MIDI) are set, the arithmetic processing unit such as a computer sets each connected phoneme data into each group according to the value of the tone color parameter included in each connected phoneme data. The process of classifying into is performed (step S13). If it has already been converted to MIDI as in the above example, it will be recorded in the set channel. Then, tone color instruction information such as a program number and a bank number based on the MIDI standard is added to each channel so that a proper tone color can be reproduced by a MIDI sound source. Since each concatenated phoneme data is classified based on the tone color parameter set for each group, it may be classified into a plurality of groups. It will no longer be grouped alone.

Although the preferred embodiments of the present invention have been described above, it goes without saying that the above encoding method is executed by a computer or the like. Specifically, a program for executing the steps shown in the flowchart of FIG. 5 in the above procedure is installed in the computer. And
After the audio signal is digitized by the PCM method or the like, the audio signal is taken into the computer, the processes of steps S1 to S5 are performed, and then the code data in the MIDI format or the like is output from the computer. The output code data is, for example, MI
In the case of DI data, it is reproduced as an acoustic signal using a MIDI sequencer and MIDI sound source. In particular, the second embodiment requires a display device for displaying a concatenated phoneme data display screen and a computer to which an instruction input device for setting a threshold value of a parameter is connected. In the second embodiment, the computer executes the processes of steps S1 to S4 in accordance with the installed program, receives the designation of the group as the classification destination of the concatenated phoneme data in step S11, and sets in step S12. The threshold value of each tone color parameter is accepted, and the process of step S13, that is, the process of classifying each connected phoneme data into each group is performed based on the accepted group and parameter.

[0062]

As described above, according to the present invention,
Frequency analysis is performed on a given acoustic signal and decomposed into time-series phoneme data consisting of start time, end time, frequency, and intensity, and frequency is calculated for each phoneme data at the same start time and end time. Unit tone color parameters are calculated based on the distribution state of other phoneme data different from each other and given to each phoneme data, and the frequency of a plurality of time-continuous phoneme data in each phoneme data is similar and precedes. When the end time of the phoneme data is similar to the start time of the subsequent unit phoneme data, the preceding phoneme data and the following phoneme data are integrated into one concatenated phoneme data, and the start time of the preceding phoneme data and the following End time of phoneme data,
And connected phoneme data that is composed of each parameter of the frequency, intensity value, and unit tone color of either the preceding phoneme data or the following phoneme data, and a plurality of phoneme data that is configured for each connected phoneme data. The connected tone color parameters are calculated based on the distribution state of the frequency / intensity values of each and are given to each connected phoneme data, and based on the plurality of parameters including at least the unit tone color parameter and the connected tone color parameter of each connected phoneme data, By classifying phoneme data into multiple timbre groups, start time, end time,
Since the code data composed of the frequency, the intensity value, and the tone color group is obtained, it is possible to separate the acoustic signal in which a plurality of tone colors are mixed into two or more tone colors.

[Brief description of drawings]

FIG. 1 is a diagram showing a basic principle of an audio signal encoding method of the present invention.

FIG. 2 is a diagram showing an example of a periodic function used in the present invention.

FIG. 3 is a diagram showing a method of calculating a correlation between a signal to be analyzed and a periodic signal.

FIG. 4 is a diagram showing a basic method of generalized harmonic analysis.

FIG. 5 is a flowchart showing a first embodiment of an audio signal encoding method according to the present invention.

FIG. 6 is a conceptual diagram for explaining connection of phoneme data.

FIG. 7 is a flowchart showing a second embodiment of an audio signal encoding method according to the present invention.

FIG. 8 is a diagram showing a connected phoneme data display screen.

[Explanation of symbols]

A1 to A3 ... Phoneme data A: Connected phoneme data d, d1 to d5 ... Unit section G (j) ... Inclusion signal n ・ ・ ・ Note number S (j), S (j + 1) ... Difference signal X, X (k) ... Section signal

Claims (8)

[Claims] 1. A phoneme data generation step for performing frequency analysis on a given acoustic signal to decompose into time-series phoneme data composed of start time, end time, frequency, and intensity, and With respect to the phoneme data, a unit tone color parameter is calculated based on a distribution state of other phoneme data having different frequencies at the same start time and the same end time, and a unit tone color parameter calculation step is given to each of the phoneme data, When the frequencies of a plurality of phoneme data consecutive in time among the phoneme data are similar and the end time of the preceding phoneme data is similar to the start time of the following unit phoneme data, the preceding phoneme data and the subsequent phoneme data are succeeded. Phoneme data to be combined into one concatenated phoneme data, and the start time of the preceding phoneme data, the end time of the following phoneme data, and the preceding phoneme data or A phoneme concatenation step of generating concatenated phoneme data composed of each frequency / intensity value / unit timbre parameter of any of the following phoneme data, and a frequency of a plurality of phoneme data constituted for each concatenated phoneme data. A connected tone color parameter calculation step of calculating a connected tone color parameter based on a distribution state of intensity values and giving it to each connected phoneme data, and a plurality of parameters including at least a unit tone color parameter and a connected tone color parameter of each connected phoneme data A phoneme classification step of classifying each of the concatenated phoneme data into a plurality of tone color groups on the basis of, and a code data composed of the start time, end time, frequency, intensity value, and the classified tone color group. An encoding method for an acoustic signal, comprising encoding an acoustic signal. 2. The acoustic signal is a stereo signal having at least L channel and R channel, and the phoneme data generating step corresponds to start time, end time, frequency, intensity value corresponding to L channel, and R channel. The unit timbre parameter calculating step gives a ratio of the strength value L and the strength value R as a unit timbre parameter, and the strength value of the phoneme data is either strength. The method for encoding an acoustic signal according to claim 1, wherein the value is given as a representative. 3. The phoneme classifying step, when classifying each concatenated phoneme data into a plurality of tone color groups, causes the concatenated phoneme data located at the boundary of the tone color groups to belong to a plurality of tone color groups. The method for encoding an acoustic signal according to claim 1, wherein 4. The phoneme classification step performs a multivariate analysis based on a plurality of parameters including at least a unit tone color parameter and a connected tone color parameter of each connected phoneme data, and classifies into any tone color group. The method for encoding an acoustic signal according to claim 1, wherein: 5. The phoneme classifying step graphically displays each concatenated phoneme data on a display device, displays each concatenated phoneme data by color according to a threshold value of a selected tone color parameter, and based on a specified threshold value of the tone color parameter. 2. The method of encoding an acoustic signal according to claim 1, wherein each concatenated phoneme data is classified into a plurality of tone color groups. 6. The unit timbre parameter calculating step calculates a unit timbre parameter based on intensity value distributions of phoneme data having frequencies close to each other at the same start time and the same end time. The method for encoding an acoustic signal according to claim 1. 7. The unit tone color parameter calculating step calculates a unit tone color parameter based on the intensity value distribution of the phoneme data whose frequency becomes an integral multiple or an integer fraction at the same start time and the same end time. The audio signal encoding method according to claim 1, wherein the audio signal encoding method is provided. 8. The connected tone color parameter calculating step calculates a distribution state of a ratio of a change in intensity value with respect to an interval of start times of phoneme data which are temporally adjacent to each other among a plurality of phoneme data constituting one connected phoneme data. 2. The method of encoding an acoustic signal according to claim 1, wherein the connection tone color parameter is calculated based on the basis.