JP2006251712A - Analyzing method for observation data, especially, sound signal having mixed sounds from a plurality of sound sources - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a framework that enables a wide-area time structure and a frequency structure to be estimated at the same time. <P>SOLUTION: An observation spectrum of a sound signal having mixed sounds from a plurality of sound sources is modeled with a superposition object model obtained by superposing a plurality of sound object models, the respective sound object models are represented with a model function having two variables of a frequency (x) and a time (t), and model parameters of the model function are optimized to estimate characteristics of the observation spectrum. The respective sound object models correspond to one harmonic structure. The model function includes a harmonic structure function including the frequency (x) as a variable and an envelope function including the time (t) as a variable. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

The present invention relates to an observation data analysis method, and more particularly to an acoustic signal analysis method in which sounds from a plurality of sound sources are mixed.

There have been many studies on the analysis of multi-acoustic signals in which sounds from multiple sound sources are mixed, but it is still one of the difficult problems. Recently proposed methods based on Kalman filters (Non-Patent Document 1), model approximation estimation in the signal and spectral domains (Non-Patent Documents 2 and 3) have made great progress in this field. However, the problem of multiple sound analysis is that the information in the frequency direction and the time direction should be processed at the same time, and these methods first resolve the problem and extract the information in the frequency dimension, then the information in the time direction. We were trying to solve the problem with a consolidated approach.
K. Nishi, S. Ando and S. Aida, “Optimum Harmonics Tracking Filter for Auditory Scene Analysis,” Proc. IEEE, ICASSP 96, pp. 573.576, 1996. S. Godsill and M. Davy, “BaysianHarmonic Models for Musical Pitch Estimation and Analysis,” Proc.ICASSP2002, Vol. 2, pp. 1769.1772, 2002. M. Goto, “A Predominant-F0 Estimation Method for CD Recordings: MAP Estimation Using EM Algorithm for Adaptive Tone Models,” Proc. ICASSP2001, Vol. 5, pp. 3365.3368, 2001. H. Kameoka, T. Nishimoto and S. Sagayama, “Separation of Harmonic Structures Based on Tied Gaussian Mixture Model and Information Criterion for Concur-rent Sounds,” Proc. ICASSP2004, AE-P5.9, May2004.

An object of the present invention is to provide a framework capable of simultaneously estimating a global time structure and a frequency structure, not an approach in which local partial information is integrated.

The technical means adopted by the present invention in order to solve such a problem is that observation data is modeled by a superimposed object model, each object model is represented by a two-variable model function, and the model parameter of the model function is optimized for observation. A feature of the value is estimated.

In one preferred embodiment, the observation data is an observation spectrum of an acoustic signal in which sounds from a plurality of sound sources are mixed, and the variables of the model function are a frequency x and a time t. Although the logarithmic frequency will be described in an embodiment described later, a linear frequency axis may be used as the frequency axis. The characteristics of the observation spectrum include frequency information (basic frequency, overtone frequency) of each sound and time information (rise time, time length). The characteristics of the observed spectrum further include the frequency component power ratio of each frequency component constituting the harmonic structure and the power spectrum envelope in the time direction.

で表される。ｐ_k(x,t)は、ｋ番目の音響オブジェクトモデルの一般式である。重畳オブジェクトモデルのパラメータには、各音響オブジェクトモデルを表すモデル関数のパラメータ、及び、各音響オブジェクトモデルの重みが含まれる。 The superimposed acoustic object model is

It is represented by p _k (x, t) is a general expression of the kth acoustic object model. The parameters of the superimposed object model include a parameter of a model function representing each acoustic object model and a weight of each acoustic object model.

In the case where the observation data is an acoustic signal, in a preferred aspect, one acoustic object model corresponds to one harmonic structure. In the embodiment described later, harmonicity is assumed. However, as long as an analytical parametric model can be assumed for the harmonic structure, the harmonic structure may be inharmonic.

で表される。 In the model function having two variables x and t in the present invention, the general expression of the k-th acoustic object model p _k (x, t) is exemplified for the case where the frequency component is represented by a normal distribution (Gaussian function).

It is represented by

In the case where the observation data is an acoustic signal, in one aspect, the model function includes a harmonic structure function including the frequency x as a variable and an envelope function including the time t as a variable. In an embodiment to be described later, as one preferable aspect, the k-th acoustic object model p (x, t | Θ _k ) is expressed by a product of two functions Φ _k (x) and Ψ _k (t). The present invention will be described based on this, but the function used is not limited to this. In an embodiment described later, a common envelope function (Gaussian basis function) is used for the entire harmonic structure.

The harmonic structure function may further include time t as a variable. The k-th acoustic object model p (x, t | Θ _k ) is exemplified by a product of two functions Φ _k (x, t) and Ψ _k (t). In this case, the harmonic structure function is a time-dependent function, and the value of the frequency x can change with time t. The case where the pitch locus projected on the xt plane is expressed by a polynomial or the like is exemplified.

A common envelope function is used for one harmonic structure. In another aspect, an independent envelope function is used for each harmonic component. The k-th acoustic object model p (x, t | Θ _k ) is represented by the product of two functions Φ _k (x, t) and Ψ _{n, k} (t). In this case, the power envelope function is prepared separately for each harmonic component. More specifically, for example, in a model having a separate attenuation curve (envelope function) for each harmonic (modeled to attenuate with separate curves at harmonics, third harmonics, fourth harmonics, ...) is there.

という形になり、xの関数とtの関数に分解できる。 In the embodiment described later, a model function having two variables x and y representing an acoustic object represents an analytical solution in a special case where the model function can be decomposed into a product of a function of x and a function of t. That is, in the above general formula, the envelope function for each harmonic component is similar (that is, Ψ _k ⁿ (t) is common regardless of n), and the pitch trajectory is parallel to the time axis ( In other words, under the assumption that μ _k (t) = μ _k ), the general formula is

It can be decomposed into a function of x and a function of t.

As described above, in the embodiment, it is assumed that the pitch trajectory of the musical object of the musical sound is parallel to the time axis, but in reality, the situation where these are parallel is limited. The assumption that the pitch trajectory is almost parallel to the time axis is a big problem, especially when the target multiple sound signal is a music signal, even if it is a voice or musical instrument sound, but it is not parallel in the playing method such as vibrato or glissando. It will not be. Further, the pitch trajectory of a musical sound object may be modeled by a polynomial or the like without assuming that the pitch trajectory is parallel to the time axis.

In one preferred embodiment, the harmonic structure function includes a fundamental frequency estimate that is a representative value of one unimodal distribution corresponding to a fundamental frequency component, and another unimodal distribution determined by the fundamental frequency estimate. The model parameter includes the representative value, weight, and variance of each unimodal distribution. The representative value parameter is composed of each representative value of each unimodal distribution of the harmonic structure model including the representative value that constitutes the fundamental frequency estimate, but only the fundamental frequency estimate is a free parameter in the representative value parameter. The other representative values are parameters constrained by the fundamental frequency estimation value. A number of distributions are known as the unimodal distribution, but in one preferred embodiment, the unimodal distribution is a normal distribution (including a log normal distribution). Examples of the representative value of the distribution include an average value, a median value, and a mode value. In one preferred embodiment, the representative value of the distribution is an average value. In an embodiment described later, an acoustic object is represented by a harmonic structure model modeled by a constrained mixed normal distribution, and the harmonic structure model modeled by a constrained mixed normal distribution corresponds to a fundamental frequency component. It has a fundamental frequency estimate that is the average μ _k of one normal distribution and an average μ _k + logn of another normal distribution that is determined by the fundamental frequency estimate. Weighting parameter r _k ⁿ represents the frequency component power ratio of each frequency component constituting the harmonic structure of the audio object k. The dispersion parameter σ _k represents the width of each frequency component constituting the harmonic structure of the acoustic object k. In one embodiment, it may be given to the model as a known parameter.

であり、モデルパラメータは、各ガウス分布の代表値、重み、分散を含む。代表値は、主として、音響オブジェクトの立ち上がり時刻の推定に用いるパラメータであり、後述する実施の形態では、（先頭の）ガウス分布の平均ｏ_ｋであるが、代表値はこれには限定されない。各ガウス分布の重みｃ_ｋ ^ｙは、時間方向のパワーエンベロープ曲線を決定するパラメータである。各ガウス分布の分散φ_ｋは、音響オブジェクトの時間長を決定するパラメータである。一つの好ましい態様では、各ガウス関数は、先頭のガウス関数の分散パラメータ（一つの好適な例では、標準偏差パラメータ）に基づく所定の等間隔αφ_ｋで配置されている。 In one preferred embodiment, the envelope function includes a plurality of Gaussian functions arranged continuously in the time axis direction,

And the model parameters include representative values, weights, and variances of each Gaussian distribution. Representative value is primarily a parameter used to estimate the rise time of the audio object, in the embodiment described below, (the beginning of) but the average o _k of Gaussian, the representative value is not limited thereto. Weights c _k ^y of each Gaussian distribution is a parameter that determines the power envelope curve in the time direction. The variance φ _k of each Gaussian distribution is a parameter that determines the time length of the acoustic object. In one preferred embodiment, the Gaussian functions are arranged at predetermined equal intervals αφ _k based on the dispersion parameter (in one preferred example, the standard deviation parameter) of the leading Gaussian function.

から構成されている。具体的には、エンベロープ関数は、２つのシグモイド関数（同一でも、同一でなくてもよい）の横軸をずらした差のいわゆる二重シグモイド関数であり、パラメータは、o_k ⁽⁰⁾,o_k ⁽¹⁾,a_k,n,b_k,n,A_k,nである。 In another aspect, the envelope function is a function that combines two sigmoid functions,

It is composed of Specifically, the envelope function is a so-called double sigmoid function of a difference in which the horizontal axes of two sigmoid functions (which may or may not be the same) are shifted, and the parameters are o _k ⁽⁰⁾ , o _k ⁽¹⁾ , a _{k, n} , b _{k, n} , A _{k, n} .

であり、パラメータは、o_k,a_k,n,b_k,n,A_k,nである。 In another aspect, the envelope function is an extreme value distribution function,

, And the parameter is a _{o k, a k, n,} b k, n, A k, n.

であり、パラメータはo_k,λ_k,nである（但し、pは定数、Γはガンマ関数である）。 Furthermore, in another aspect, the envelope function is Generalized Gaussian Distribution (GDD),

And the parameters are o _k , λ _{k, n} (where p is a constant and Γ is a gamma function).

In one preferred embodiment, the model function parameter optimization method is MAP estimation. However, the optimization method applied to the present invention is not limited to MAP estimation, and may be other optimization methods. Also good. Moreover, in one preferable aspect, the estimation algorithm of model parameter optimization is EM algorithm.

The present invention can also be provided as an acoustic analysis system, a computer program for acoustic analysis, or a recording medium on which the program is recorded.

The observation data analysis method of the present invention is preferably applied to an acoustic signal, but the superimposed object model according to the present invention uses the original information of the projected data from the data projected on the two-dimensional plane. Can be extended to restore. In another aspect, the observation data is image data including a plurality of objects. As a simple example, the target object is a rectangular parallelepiped object, which is modeled with an object model corresponding to the rectangular parallelepiped object and its shadow, and the characteristics of the target object are restored. When the observation data is image data, it can be applied to scene analysis such as robot vision.

According to the present invention, it is possible to model an acoustic signal in which sounds from a plurality of sound sources are mixed with the superimposed object model and simultaneously estimate the global geometric structure of each acoustic object and the superimposed acoustic object in terms of time and frequency. The sound signal can be analyzed with high accuracy.

The invention will be described based on one preferred embodiment, multispectral separation using a Gaussian basis acoustic object model.

[A] Acoustic object model
[A-1] Formulation of Problem As shown in FIG. 1, the observed spectrum of an acoustic signal in which sounds from a plurality of sound sources are mixed includes a fundamental wave component associated with time trajectories of a plurality of pitches (fundamental frequencies) and This is a complex distribution in which multiple harmonic components are superimposed. Considering separation of such a mixed distribution into spectra, spectrum overlap becomes a problem in short-time analysis. In the present invention, the observed spectral distribution is a kind of micro energy pattern histogram, and is assigned to a number of strip-like regions in the time-frequency plane, and each region is a plurality of predicted spectra of each acoustic object. Make up the ingredients. In this specification, an observation pattern is arbitrarily decomposed, and each decomposed pattern is called a cluster. That is, the cluster means the distribution of the observed patterns that are decomposed, and the clustering means that the observed patterns are decomposed into clusters. If an appropriate degree of clustering attribution is determined, the observed composite distribution can be separated by a probabilistic method.

The power spectrum of musical sounds distributed on the time-frequency plane forms a kind of object (hereinafter referred to as an acoustic object) in which comb structures in the frequency direction are continuous in the time direction. FIG. 2 shows one acoustic object, and one acoustic object is composed of a plurality of object elements allocated on the frequency-time plane, and the plurality of object elements are one element corresponding to one fundamental frequency component. And a plurality of elements corresponding to overtone (including non-integer multiple) components. In the present invention, the spectrum time pattern of a music signal composed of a large number of musical sounds is considered to be superimposed on each musical sound object, and the acoustic object decomposition is analyzed as a fuzzy clustering problem of acoustic energy distributed in two dimensions of time and frequency. Formulate it.

と設定する。ただし、x, t, f(x, t)はそれぞれ対数周波数、時間（フレーム）、ウェーブレット変換により得られた観測スペクトル（パワースペクトル密度）、Ｔ₀, Ｔ₁, Ω₀, Ω₁ はそれぞれ時間と対数周波数の下限と上限を指し、Ｋはクラスタ数、ｋはクラスタのインデックスを表す。 The model p (x, t | Θ _k ) that geometrically forms one acoustic object in each cluster can be defined by the parameter Θ _k (Θ = {Θ _k | k = 1,... K}). Function

And set. However, x, t, f (x, t) are logarithmic frequency, time (frame), observed spectrum (power spectral density) obtained by wavelet transform, and T ₀ , T ₁ , Ω ₀ , Ω ₁ are time, respectively. Indicates the lower and upper limits of the logarithmic frequency, K represents the number of clusters, and k represents the cluster index.

で与えられる。つまり、ｐ(k|x,t,Θ)f(x, t)は確率的に分離された音響オブジェクトという意味をなす。Ｄ(x,t｜Θ_k)は、k番目のモデルが座標(x, t)においてどれだけ支配的であるかを反映した(擬)距離関数である。より直感的にはモデルと観測スペクトルの積分値がいずれも等しい場合、すなわち、ｐ(x,t｜Θ_k)が、

を満たす場合には、ｐ(k|x,t,Θ)f(x, t)Ｄ(x,t｜Θ_k)は、２つの分布、ｐ(x,t｜Θ_k)とｐ(k|x,t,Θ)f(x,t)、が近くなるほど大きな値を取ることになる。 P (k | x, t, Θ) is a probability indicating how much of the spectral component belongs to the k th cluster at the coordinates (x, t),

Given in. That is, p (k | x, t, Θ) f (x, t) means a stochastic separated acoustic object. D (x, t | Θ _k ) is a (pseudo) distance function that reflects how dominant the k th model is in coordinates (x, t). More intuitively, if both the integral value of the model and the observed spectrum are equal, that is, p (x, t | Θ _k ) is

If p is satisfied, p (k | x, t, Θ) f (x, t) D (x, t | Θ _k ) has two distributions, p (x, t | Θ _k ) and p (k | x, t, Θ) f (x, t), the larger the value becomes.

From the above, it can be reduced to the problem of optimal approximation of the time series distribution of the observed spectrum by some geometric model. Here, the objective function, D (x, t | Θ k) = logp (x, t | Θ k) is under certain conditions that should be noted that the Q function and the same shape in the EM algorithm. In the following, a two-dimensional distribution model that reflects both the harmonic structure and time continuity of an acoustic object is formulated.

ここで、

とする。 [A-2] Assuming that the pitch trajectory of the acoustic object of the Gaussian base acoustic object model musical sound is parallel to the time axis, the cut at a specific time t of the kth acoustic object model as shown in FIG. Such a function reflects the harmonic structure Φ _k (x). Therefore, assuming that the acoustic object model is obtained by multiplying the harmonic structure model function Φ _k (x) by the envelope function Ψ _k (t) as shown in FIG. 4 along the time axis, the kth acoustic object model p ( x, t | Θ _k ) can be expressed as a product of two functions and power (energy) w _k .

here,

And

[A-3] Harmonic structure function Φ _k (x)
As a preferable embodiment of the harmonic structure function constituting the model function, the harmonic structure model function already proposed by the inventors of the present application can be used. First, the harmonic structure model will be described. In short-time spectrum analysis, the frequency components of different signals overlap due to the spread of fundamental frequency components and harmonic components, making it difficult to separate adjacent frequency components and accurately detect fundamental frequencies or harmonic frequencies. The frequency component observed in this way is regarded as the frequency distribution or probability distribution of each frequency, and by approximating the distribution with a Gaussian distribution, a spectrum with a single harmonic structure is mixed with multiple Gaussian distributions. Model as a distribution. As shown in FIG. 3, by approximating the spectrum spread shape with a Gaussian distribution, the frequency value can correspond to the average estimation of the Gaussian distribution, and the energy of the frequency component can correspond to the weight estimation of the mixed Gaussian distribution. To maintain harmony, only the average of one Gaussian distribution (fundamental frequency estimate) corresponding to the fundamental frequency component has a degree of freedom, and the average position of all remaining normal distributions is determined according to its position. The A single harmonic structure modeled by such a constrained mixed Gaussian distribution is referred to as a “harmonic structure model” in this specification. The Gaussian distribution is a suitable example of a distribution function that can be applied to the harmonic structure model, and the harmonic structure model may be configured using other unimodal distribution functions. The average is one preferable example of the representative value of the distribution, and the median and the mode may be used instead of the average.

で表す。ただし、μ_kは対数基本周波数推定値、ｒ_ｋ ^ｎ(n=1,…, N、Σｒ_ｋ ^ｎ＝１、ｎは調波構造モデルにおけるガウス基底のインデックスである )はn 次高調波成分パワー比に対応する。 Assuming harmonicity, if the nth log frequency component is logn away from the basic logarithmic frequency, the basic logarithmic frequency is estimated to be μ _k , and the nth partial logarithmic frequency is estimated to be μ _k + logn. That is, if the fundamental frequency estimation value is set as μ _k , each average μ _k of the harmonic structure model _k is expressed by μ _k , μ _k + log2, μ _k + logn,. . . μ _k + logN. By approximating the distribution of each frequency component with a Gaussian distribution, one harmonic structure is modeled with a weighted sum of Gaussian bases. When this is formulated, a harmonic structure model is obtained by assuming harmonicity and approximating one frequency component distribution with a Gaussian function.

Represented by However, mu _k is logarithmic fundamental frequency ^{_{estimate, r k n (n = 1}} , ..., N, Σr k n = 1, n is the index of Gaussian basis of the harmonic structure model) the n-th harmonic component power Corresponds to the ratio.

として表される。ただし、Yはガウス基底の数、ｏ_kは先頭のガウス基底の中心であり、音響オブジェクトの立ち上がり時刻の推定に密接に関係し、ｃ_ｋ ^y(y =0,…,Y- 1) はエンベロープ曲線を規定する各ガウス基底の重み値を表す。複数のガウス関数の中心を標準偏差パラメータφ_kと等しい間隔（α＝１の場合で言うと）で配置した特殊な拘束をもったガウス基底関数は、各基底が孤立するのを防いで曲線の滑らかさを保つと同時にφ_kの値あるいは/およびαの値に応じて時間方向に線形伸縮する性質を持ち、さまざまな時間長の音響オブジェクトに広く対応できる。 [A-4] Envelope function Ψ _k (t)
The envelope function Ψ _k (t) is desirably a function that can flexibly cope with various fluctuations of the power spectrum envelope. For example, when it comes to music signals, depending on the instrument and musical expression, the attack, sustain and release will be quite different. Therefore, the envelope function Ψ _k (t) is expressed from a plurality of Gaussian bases, and each Gaussian base is related to the envelope shape and weights c _k ^y (y = 0,..., Y−1, Σc _k ^y = 1). , Y is an index of the Gaussian basis of the envelope model). The feature of this model is that the interval between adjacent Gaussian functions is expressed based on the standard deviation parameter φ _k of each Gaussian function, and the envelope function Ψ _k (t) is

Represented as: However, Y is the number of Gaussian basis, o _k is the center of the head of Gaussian basis, closely related to the estimation of the rise time of the audio ^{_{object, c k y (y = 0}} , ..., Y- 1) envelope Represents the weight value of each Gaussian basis that defines the curve. Gaussian basis functions with special constraints in which the centers of multiple Gaussian functions are arranged at equal intervals (in the case of α = 1) with the standard deviation parameter φ _k , prevent each basis from being isolated. While maintaining smoothness, it has the property of linear expansion and contraction in the time direction according to the value of φ _k and / or α, and can be widely applied to acoustic objects of various time lengths.

[A-5] Superimposed Object Model Observation of an acoustic signal in which sounds from a plurality of sound sources are mixed using a superimposed object model in which one object model corresponding to one harmonic structure as described above is superimposed. Model the spectrum. Table 1 shows model parameters of the superimposed object model. Table 1 exemplifies suitable model parameters, and the model parameters according to the present invention are not limited to those shown in Table 1.

k: Index of each acoustic object (acoustic stream) model, and actually corresponds to the index of the acoustic object. The observed spectrum of the mixed sound is modeled using K acoustic object models.

n: Gaussian basis index in the harmonic structure model, which actually corresponds to the index of each frequency component of the harmonic structure. One harmonic structure is modeled using N Gaussian functions.

y: Gaussian basis index in the power envelope model. One envelope curve is modeled using Y Gaussian functions.

μ _k : the average of the first Gaussian basis in the harmonic structure model, and actually corresponds to the fundamental logarithmic frequency.

μ _k + logn: The average of the nth Gaussian basis in the harmonic structure model, and actually corresponds to the nth logarithmic frequency element.

w _k : The weight of the k th acoustic object model, which actually means the relative dominance of the k th acoustic object.

r _k ⁿ : Gaussian basis weight in the harmonic structure model of the kth acoustic object model, and actually corresponds to the frequency component power ratio.

c _k ^y : Weight of the Gauss basis in the power envelope model of the kth acoustic object model, and actually corresponds to a curve in the time direction of the power envelope.

o _k : average of the first Gaussian basis in the power envelope model of the kth acoustic object model, and in one example, corresponds to the rise time (onset time) of the kth acoustic object.

σ _k : Standard deviation of the Gauss basis in the harmonic structure model of the kth acoustic object model, and actually corresponds to the width of each frequency component.

φ _k : Gaussian base interval and standard deviation in the power envelope model of the kth acoustic object model, and is actually related to the time length of the kth acoustic object.

を利用する。ただし、ｄ_r、ｄ_c は事前分布の寄与の大きさ、β(d_r)、β(d_c)はそれぞれ正規化係数を表す。事前分布は、ＭＡＰ推定におけるラグランジュの未定乗数の計算を大幅に簡単化できるという有利な点を有する。尚、この分布以外にもディリクレ分布も同じ目的に適用可能である。 [B] Optimal parameter estimation
[B-1] Assumption of prior distribution The assumption of prior distribution is effective when it is desired to give flexible constraints on specific parameters. For _example, for _r ^{k n} and _c ^{k y,} somewhat commonsense prediction value _r ^{k n} (bar) envisaged for each component ratio and power envelope of the harmonic _structure, too extreme departure from the _c ^{k y} (bar) Parameter constraints can be added (see FIG. 5). Here, a prior distribution (see Non-Patent Document 3) that can greatly simplify the calculation of Lagrange's undetermined multiplier in MAP estimation,

Is used. Here, d _r and d _c are the magnitudes of prior distribution contributions, and β (d _r ) and β (d _c ) are normalization coefficients, respectively. The prior distribution has the advantage that it can greatly simplify the calculation of Lagrange's undetermined multiplier in MAP estimation. In addition to this distribution, the Dirichlet distribution can be applied for the same purpose.

という補助関数に書き直せる。ただし、λ_r ^(k)，λ_c ^(k)，λ_ｗはラグランジュの未定乗数である。尚、式（９）において、f(x,t)を正規化し、重みの総和を１としてもよい（この場合、Ｆ＝１となる）。 [B-2] Optimum approximate parameter estimation of a mixed acoustic object model under constraint conditions higher than MAP estimation using EM algorithm is a problem of the same type as MAP estimation by EM algorithm (monotonic increase of auxiliary function by iterative calculation). The objective function in Equation (1) corresponds to the auxiliary function, and Equation (1) is

Can be rewritten as an auxiliary function. Here, λ _r ^(k) , λ _c ^(k) , and λ _w are Lagrange's undetermined multipliers. In equation (9), f (x, t) may be normalized and the sum of weights may be set to 1 (in this case, F = 1).

The local optimum parameter is obtained by the following iterative calculation.
(1) Substitute Θ (hat) into Θ (bar) updated in the M-step before E-step, and calculate Θ (bar) to the auxiliary function R (Θ, Θ (bar)). This step corresponds to updating the attribution probability density p (k, n, y | x, t, Θ).
(2) Under the M-step attribution probability density p (k, n, y | x, t, Θ) fixed, the parameter of Θ (bar) is updated, and the auxiliary function R (Θ, Θ (bar) ).

[B-3] Parameter Update Formula in M-Step The calculation result of each model parameter update formula in M-step is shown. In the following expression, for the sake of brevity, the integration ranges in the time direction (T ₁ , T ₂ ) and the frequency direction (Ω ₁ , Ω ₂ ) are omitted.

The update formula of the basic logarithmic frequency μ _k is as follows. Thereby, the fundamental frequency of the kth acoustic object is estimated.

Update equation of relative power r _k ⁿ spectral components are as follows. Thereby, the frequency component power ratio of each frequency in the harmonic structure of the kth acoustic object is estimated.

The update formula of the width σ _k of each frequency component in the harmonic structure is as follows. The width of each frequency component of the harmonic structure of the kth acoustic object is estimated. Here, the width is the same between the frequency components.

Update equation of the rising time o _k are as follows. The rise time of the kth acoustic object is estimated.

Updating expression elements c _k ^y of power envelope curve is as follows. The power envelope curve of the kth acoustic object is determined by a weighted sum of a plurality of Gaussian bases constituting the envelope function.

The update formula for the time length element φ _k is as follows. The time length of the kth acoustic object is estimated.

The formula for updating the power (energy) of the kth acoustic object in the superimposed acoustic object is as follows.

[C] Experimental Example [C-1] Experimental Example 1
As test data of the method according to the present invention, two actual music signals (16 kHz sampling frequency) were used from the music database for RWC research. The power spectrum time series was output by Gabor wavelet transform (frame shift 20ms, frequency resolution 16.7cent, minimum frequency 50Hz). The time length of the analysis section (temporal frequency plane) was 3 s (150 frames). The initial values of the parameters (μ _k , o _k | k = 1,..., K) for the EM algorithm are 70 peaks (maximum points of power spectral density) in order from the given spectral distribution. Determined. In the iteration of the EM algorithm, the total number of acoustic objects was estimated by thresholding. That is, a model whose weight parameter w _k is equal to or smaller than a certain threshold value is judged to be silent and removed.

A specific example of the optimization model estimated from the actual spectrum and the corresponding three-dimensional display and gray scale display of the time-frequency spectrum are shown in FIG. FIG. 6A shows the observed spectrum distribution in three dimensions (logarithmic frequency axis, time axis, and axis representing energy intensity), and FIG. 6C shows an observation corresponding to FIG. 6A. It is a gray scale display of a spectrogram of a spectrum (horizontal axis: time, vertical axis: logarithmic frequency). FIG. 6B is a three-dimensional display (logarithmic frequency axis, time axis, and axis representing energy intensity) of the superimposed acoustic object model with the optimum parameters, and corresponds to FIG. FIG. 6B is an overlay of the acoustic objects shown in FIG. FIG. 6D is a grayscale display (horizontal axis: time, vertical axis: frequency) of the optimized superimposed acoustic object model corresponding to FIG. 6B. As shown in FIGS. 6B and 6D, not only the pitch of the superimposed acoustic object but also the onset time (rise time), time length, offset time, and power envelope are appropriately estimated. It is also possible to extract and reconstruct individual acoustic objects by sinusoidal synthesis using the optimal attribution probability.

[C-2] Experimental example 2
As a performance evaluation standard of the method according to the present invention, the pitch correct rate was calculated from the attached reference MIDI data. Further, as a comparison target, a method (Non-patent Document 4) for estimating a pitch trajectory by HMM based on model estimation information for each frame was selected. Compared to the conventional method, the method according to the present invention showed higher performance than the conventional method (Table 2), and the effect of modeling the time direction and the frequency direction at the same time was confirmed.

INDUSTRIAL APPLICABILITY The present invention can be used for speech recognition in a real environment, high performance speech recording in a multi-speaker environment, automatic scoring in a karaoke system, and music signal analysis for accompaniment data creation.

This is an observation spectrum obtained by wavelet transforming an actual music performance signal at times T ₀ to T ₁ and frequencies Ω ₀ to Ω ₁ . It is a figure explaining the parametric model (acoustic object model) of a kth acoustic object spectrum, Comprising: One acoustic object (kth) on a frequency-time plane is represented. A Gaussian basis harmonic structure model is shown. A Gaussian basis power envelope model is shown. Is a diagram illustrating a prior distribution of the weight parameter r _k ^n. (A) Three-dimensional display of observed spectrum distribution (frequency axis, time axis, axis representing energy intensity); (b) Three-dimensional display of superimposed acoustic object model with optimum parameters (frequency axis, time axis, axis representing energy intensity) (C) Gray scale display of a given spectrogram (horizontal axis: time, vertical axis: frequency); (d) Gray scale display of an optimization model (horizontal axis: time, vertical axis: frequency);

Claims (23)

The observation data is modeled by a superimposed object model formed by superimposing multiple object models, each object model is represented by a two-variable model function, and the model parameters of the model function are optimized to estimate the characteristics of the observation data This is a method for analyzing observation data. An observation spectrum of an acoustic signal in which sounds from a plurality of sound sources are mixed is modeled by a superimposed object model formed by superimposing a plurality of acoustic object models, and each acoustic object model is represented by two variables of frequency x and time t. A method of analyzing an acoustic signal, characterized by estimating a characteristic of an observed spectrum by optimizing a model parameter of the model function. 3. The acoustic signal analysis method according to claim 2, wherein the characteristics of the observed spectrum include frequency information and time information of each sound. 4. The acoustic signal analysis method according to claim 3, wherein the characteristics of the observed spectrum further include a frequency component power ratio of each frequency component constituting the harmonic structure and a power spectrum envelope in the time direction. 5. The acoustic signal analysis method according to claim 2, wherein each acoustic object model corresponds to one harmonic structure. 6. The acoustic signal analysis method according to claim 5, wherein the model function includes a harmonic structure function including a frequency x as a variable and an envelope function including a time t as a variable. 7. The harmonic structure function according to claim 6, wherein a fundamental frequency estimated value that is a representative value of one unimodal distribution corresponding to a fundamental frequency component and another unimodal distribution determined by the fundamental frequency estimated value. A method for analyzing an acoustic signal having a representative value, wherein the model parameter includes a representative value, weight, and variance of each unimodal distribution. 8. The method for analyzing an acoustic signal according to claim 7, wherein the unimodal distribution is a Gaussian distribution. 9. The method for analyzing an acoustic signal according to claim 7, wherein the representative value of the distribution is an average. 10. The acoustic signal analysis method according to claim 6, wherein the harmonic structure function further includes a time t as a variable. 11. The acoustic signal analysis method according to claim 10, wherein the feature of the observation data includes a pitch locus on the xt plane. 12. The acoustic signal analysis method according to claim 6, wherein a common envelope function is used for one harmonic structure. 12. The method for analyzing an acoustic signal according to claim 6, wherein an independent envelope function is used for each harmonic component. 14. The envelope function according to claim 12, wherein the envelope function is a plurality of Gauss functions continuously arranged in the time axis direction, and the model parameter includes a representative value, a weight, and a variance of each Gaussian distribution. Analysis method of acoustic signal. 15. The method of analyzing an acoustic signal according to claim 14, wherein the Gaussian functions are arranged at predetermined equal intervals based on a dispersion parameter of the leading Gaussian function. 16. The method for analyzing an acoustic signal according to claim 6, wherein the envelope function is composed of a function obtained by combining two sigmoid functions. 16. The method of analyzing an acoustic signal according to claim 6, wherein the envelope function is an extreme value distribution function or GDD. 18. The method of analyzing an acoustic signal according to claim 2, wherein the parameter of the superimposed object model includes a parameter of a model function representing each acoustic object model and a weight of each acoustic object model. 19. The acoustic signal analysis method according to claim 2, wherein the parameter optimization is performed by MAP estimation. 19. The acoustic signal analysis method according to claim 2, wherein the estimation algorithm for model parameter optimization is an EM algorithm. A computer program for causing a computer to execute the method according to any one of claims 2 to 20. A recording medium on which a computer program for causing a computer to execute the method according to claim 2 is recorded. An observation spectrum of an acoustic signal in which sounds from a plurality of sound sources are mixed is modeled by a superimposed object model formed by superimposing a plurality of acoustic object models, and each acoustic object model is represented by two variables of frequency x and time t. An acoustic signal analysis system characterized by estimating the characteristics of an observed spectrum by optimizing model parameters of the model function.

Images