JP2003015684A - Method for extracting feature from acoustic signal generated from one sound source and method for extracting feature from acoustic signal generated from a plurality of sound sources - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a computerized method for extracting features from acoustic signals generated from one or a plurality of sound sources. SOLUTION: The acoustic signal are first windowed and filtered to produce a spectral envelope for each source. The dimensionality of the spectral envelope is then reduced to produce a set of features for the acoustic signal. The features in the set are clustered to produce a group of features for each of the sources. The features in each group include spectral features and corresponding temporal features characterizing each source. Each group of features is a quantitative descriptor that is also associated with a qualitative descriptor. Hidden Markov models are trained with sets of known features and stored in a database. The database can then be indexed by sets of unknown features to select or recognize like acoustic signals.

Description

Detailed Description of the Invention

[0001]

FIELD OF THE INVENTION The present invention relates generally to the field of acoustic signal processing, and more particularly to a method for recognizing, indexing and searching acoustic signals.

[0002]

2. Description of the Related Art Up to now, little research has been done on extracting features of environmental sounds and ambient sounds. Most prior art acoustic signal representations have focused on human voice and music. On the other hand, footsteps, traffic sounds, door slamming sounds, laser gun sounds, knocking sounds, knocking sounds, thundering sounds, rustling leaves,
There is no suitable representation for many sound effects heard in movies, television, video games and virtual environments such as the sound of water flowing. These environmental acoustic signals are generally much more difficult to extract features than voice and music.
Because their signals are often noisy,
This is because it contains a large number of superimposed components and higher order structural components such as repetition and scattering.

One particular application in which such representation schemes can be used is video processing. Methods are available for extracting, compressing, searching, and classifying video objects. See, for example, various MPEG standards. There is no way to process such an "audio band" object, except when the "audio band" object is speech. For example, it may be desirable for John Wayne to search a video library to identify the locations of all the videos galloping on a horse while shooting a six-shot pistol. Indeed, it is possible to visually identify John Wayne or a horse. However, it is very difficult to sort out the rhythmic squeaking noise of a galloping horse and the intermittent sounding of a revolver.
By recognizing audio frequency band events, it is possible to delineate motions in the video.

Another application in which the expression can be used is in sound synthesis. Before a sound can be synthesized and generated by a method other than trial and error, the characteristics of the sound must be specified.

In the prior art, representations for sounds other than speech generally include, for example, reproducing the timbre of a particular instrument, identifying that particular instrument, distinguishing the sound of a submarine from the sounds of the surrounding sea. In doing so, I have focused on sounds other than a certain class of sounds, such as recognizing underwater mammals by their calls. Each of these applications requires a particular array of acoustic features that is not generalized beyond the particular application.

In addition to these particular applications, other research has focused on developing generalized analytical representations of acoustic scenes. This study became known as "Computational Auditory Scene Analysis." These systems require a great deal of computational work due to the complexity of their algorithms. Typically, these systems utilize heuristics from artificial intelligence and various inference schemes.

While such systems provide useful insights into the challenges of acoustic representation, it has never been shown that the system's performance is satisfactory with respect to the classification and synthesis of mixed state acoustic signals. It has not been.

[0008] In still another application form, the expression of sound is used to generate environmental sounds, background noise, sound effects (sound effects), animal sounds,
Media in the audible frequency range can be indexed, including a wide range of sound phenomena including voice, non-voice cry and music. This will allow the use of automatically extracted indices to design sound recognition tools for searching media in the audio frequency band. With these tools, a soundtrack containing a lot of content, such as a movie or a news show, can be described by a semantic description of the content.
Alternatively, it can be searched by the similarity to the query of the target audible frequency band sound. For example, it is desirable to locate all video clips where the lion roars or the image roars.

There are numerous feasible approaches to automatic classification and indexing. Wold, etc. (IEEE Multimedia,
pp.27-36, 1996), Martin et al., "Musical inst
rument identification: a pattern-recognition appro
ach ”(Presented at the 136th Meeting of the Acous
The tic Society of America, Norfolk, VA, 1998) describes a strict classification for musical instruments. Zhang et al. "Cont
ent-based classification and retrieval of audio ''
(SPIE 43rd Annual Meeting, Conference on Advanced
Signal Processing Algorithms, Architectures and I
mplementationsVIII, 1998) describe a system for training a model using spectrogram data, which is described in Borecczky et al., “A hidden Markov model fr
amework for video segmentation using audio and ima
ge features "(Proceedings of ICASSP'98, pp.3741-
3744, 1998) uses the Markov model.

[0010]

The indexing and searching of audio frequency band sound media is particularly closely related to the emerging MPEG-7 standard for multimedia. The standard requires an integrated interface for common sound classes. Encoder compatibility is a factor in design. In doing so, a “sound” database with indices provided by one embodiment can be compared to a database extracted by a different embodiment.

[0011]

Features are extracted from an acoustic signal generated from one or more sound sources by a computerized method. The acoustic signal is first windowed and filtered to generate a spectral envelope for each source. Then the dimensionality of the spectral envelope is reduced to 1 for that acoustic signal.
A set of features is generated. The features in the set are clustered to produce a group of features for each sound source. The features within each group include spectral features and corresponding temporal features that characterize each sound source.

A feature of each group is a quantitative descriptor, which is also associated with a qualitative descriptor. Hidden Markov models are trained by a known set of features,
Stored in the database. At that time, the database selects or recognizes similar acoustic signals,
It can be indexed by a set of unknown features.

[0013]

DETAILED DESCRIPTION OF THE INVENTION FIG. 1 shows a method 100 for extracting spectral features 108 and temporal features 109 from a mixture 101 of signals according to the present invention. The method 100 of the present invention has the purpose of characterizing and extracting features from recorded sounds for the purpose of classifying sound sources, and in structured multimedia applications such as parameter synthesis. Change and reuse (re-purpose)
Can be used for The method can also be used to extract features from other linear mixtures as well as multidimensional mixtures. The mixture can be obtained from one sound source or multiple sound sources such as a stereo sound source.

To extract features from the recorded signal, the method according to the invention makes use of a statistical technique based on independent component analysis (ICA). With a contrast function defined by a cumulative extension up to the 4th order, the ICA transform produces a rotation of the basis of the time-frequency observation matrix 121.

The resulting basis components are as statistically independent as possible, revealing individual features within the mixture sound source 101, such as the structure of the sound. These characteristic structures can be used to classify signals or identify new signals with predictable characteristics.

The representation according to the invention is able to synthesize the behavior of multiple tones from a small set of features. The representation according to the invention is also able to synthesize complex acoustic event structures such as collisions, bounces, hits, rubs, etc., as well as acoustic object properties such as material, size, shape.

In the method 100, the audio band sound mixture 101 is first processed by a bank 110 of logarithmic filters. Each filter produces a bandpass signal 111 for a given frequency range. Typically 40-5
The bandpass signal 111 of 0 is generated, and many signals are generated in the low frequency range rather than the high frequency range so as to mimic the frequency response characteristics of the human ear. Alternatively, the filter may be a constant Q (CQ) or wavelet filter bank, or the filter may be a short-time fast Fourier transform representation (S
They can be arranged linearly as in the case of (TFT).

In step 120, each bandpass signal is "windowed" into short, eg 20 msec, segments to generate an observation matrix. Each matrix can contain hundreds of samples. Steps 110 and 1
Details of 20 are shown in more detail in FIGS. Note that windowing can be done before filtering.

In step 130, singular value decomposition (SVD) is applied to the observation matrix 121 to generate a matrix 131 having a reduced dimensionality. The SVD was first described in 1873 by the Italian geometricist Bertrami.
Singular value decomposition is a well-defined generalization of principal component analysis (PCA). Singular value decomposition of an m × n matrix is an arbitrary factorization of the form

X = UΣV ^T

However, U is an m × m orthogonal matrix, that is, U has an orthonormal column, V is an n × n orthogonal matrix,
Σ is a m × n diagonal matrix of singular values with components σ _ij = 0, where i is not equal to j.

As an advantage, and in contrast to PCA, SVD is capable of decomposing non-square matrices, thereby either spectrally or temporally without the need for covariance matrix computations. The observation matrix can be decomposed directly in the direction. Since SVD directly decomposes a non-square matrix without the need to find a covariance matrix, the resulting basis is less sensitive to dynamic range issues than PCA.

The method of the present invention applies optional Independent Component Analysis (ICA) to the reduced dimension matrix 131 in step 140. ICAs that use an iterative online algorithm based on a pseudo-neuro architecture for blind signal separation are well known.
Recently, numerous neural network architectures have been proposed to solve the ICA problem. For example, "Adapt given to Sejnowski on January 17, 1995.
ive system for broadband multisignal discriminatio
See U.S. Pat. No. 5,383,164 entitled "n in a channel with reverberation."

The ICA produces spectral features 108 and temporal features 109. The spectral features, represented as vectors, correspond to estimates of the statistically most independent components within the segmentation window. Temporal features, also represented as vectors, describe the evolution of spectral components during the course of that segment.

Each pair of spectrum and time vector is
Vector cross products can be combined to reconstruct the partial spectrum for a given input spectrum.
Independent time domain signals can be estimated if these spectra are reversible as the filterbank representation does. For each independent component described in that scheme, a matrix of compatibility scores for the components in the previous segment is available. This allows the components to be tracked over time by estimating the most likely consecutive correspondence. Equal to the backward compatibility matrix only when looking forward in time.

Independent component decomposition of the audio band sound track can be used to estimate individual signal components within the audio band sound track. The separation problem is awkward when a full rank signal matrix (N linear mixture of N sources) is not available, but by using the independent components of a short temporal section of the frequency domain representation An approximation to the sound source at can be given. These approximations can be used for classification and recognition tasks, and comparisons between sounds.

As shown in FIG. 3, the time-frequency distribution (TFD) has a power spectral density (PSD) to reduce the contribution of lower frequency components that carry more energy in some acoustic regions. ) 115
Can be normalized by

FIGS. 4 and 5 respectively show the temporal and spatial decomposition for percussion instruments played in regular rhythms. The observable structure reveals a broad band segmental component corresponding to the shake and a horizontal layered structure corresponding to the ringing of the metal shell.

Applications for Acoustic Features of Sound The present invention can be used in numerous applications.
The extracted features can be considered as separable components of the acoustic mixture, which represent unique structures within the source mixture. The extracted features are determined by pattern recognition techniques to recognize or identify their components 1
Can be compared to a set of a priori classes. These classifiers can be in the area of analysis models with phonemes, sound effects, musical instruments, animal sounds or any other corpus. The extracted features can be recombined individually using an inverse filter bank, thereby achieving "purification" of the acoustic mixture of the sound source. One example application is the separation of singers, drums and guitars from recorded sounds, for repurposed reusing of some components, or for automatic analysis of the structure of music. is there. Another example is to separate the actor's voice from the background noise and pass a clear audio signal to the speech recognizer for automatic subtitle translation of the movie.

The spectral and temporal features can be considered individually to identify various characteristics of the acoustic structure of individual sound objects within the mixture. Spectral features can describe properties such as material, size, shape, while temporal features can describe behavior such as bouncing, breaking, slamming, and the like.
Thus, beating a cup can be distinguished from bouncing the cup or beating an earthenware. The extracted features can be modified and resynthesized to produce a modified synthetic case of the sound of the source. If the input sound is a single sound event that includes multiple acoustic features such as slamming a cup, the individual features can be controlled for resynthesis. This is useful for media applications with models such as producing sound in a virtual environment.

Indexing and Searching The invention can also be used to include many different types of sounds, such as sound effects, animal sounds, musical instruments, sounds, overlapping sounds, environmental sounds, masculine sounds, feminine sounds. You can also index and search large multimedia databases.

In this context, phonetic descriptions are generally divided into two types: qualitative descriptions using letters with category labels and quantitative descriptions using probabilistic model states. Category labels provide qualitative information about the sound content. The description in this form is suitable for textual query applications, such as Internet search engines, or any processing tool that uses textual fields.

In contrast, the quantitative descriptor contains compact information about a segment of the audio band sound and can be used for the numerical evaluation of the similarity of sounds. For example, these descriptors can be used to identify a particular instrument in a video or audio recording.
The qualitative and quantitative descriptors are adapted to the application of an example query search for audio band sounds.

Sound Recognition Descriptors and Description Schemes Qualitative Descriptors While segmenting a recorded audio band sound into classes, it is desired to obtain relevant semantic information about its content. For example, recognizing a scream in the video soundtrack can indicate fear or danger, and a laugh can indicate a comedy. In addition, the sound can indicate the presence of a person,
Therefore, the video segment to which these sounds belong can be used as a candidate when searching for clips containing people. The sound category and taxonomy descriptors provide a means for organizing category concepts into a hierarchical structure that enables this type of complex relational search strategy.

Sound Categories As shown in FIG. 6 for a simple taxonomy 600, description schemes (DS) are used to name sound categories. As an example, a dog barking sound may be given the qualitative category label “dog” 610 with “barking” 611 as a subcategory. Further, "groans" 612 or "howls" 613 can be a desirable subcategory of "dogs". The first two subcategories are closely related, but the third
The subcategories of are completely different sound events. Therefore, FIG. 6 shows that the four categories are organized into a taxonomy with “dog” 610 as the root node. Each category has at least one relational link 601 to another category within its taxonomy. By default, the contained category is considered to be a narrower category (NC) 601 than the contained category. However, in this example, "groan" is generally synonymous with "croak" but is defined as less preferred. To obtain such a structure,
The following relationships are defined as part of the description scheme of the present invention.

BC-Wide category means that the associated category is more general in meaning than the containing category. NC-Narrower category means that the associated category is more specific in meaning than the containing category. US-Use associated categories that are generally synonymous with the current category because they are preferred over the current category.
UF-Use of the current category is preferred over nearly synonymous associated categories. RC-The associated category is not synonymous, somewhat synonymous, wider or narrower, but associated with the containing category.

The following XML Schema code illustrates how to use the Definition Definition Language (DDL) to exemplify the qualitative description scheme for the category classification scheme shown in FIG.

[0038]

[Equation 2]

The category and scheme attributes together provide a unique identifier that can be used to refer to categories and taxonomies from quantitative description schemes such as the probabilistic model described in more detail below. Label descriptors give meaningful semantic labels for each category, and relationship descriptors describe the relationships among the categories of the taxonomy according to the invention.

Classification Scheme As shown in FIG. 7, categories can be combined with classification scheme 700 by relational links to create a richer classification scheme. For example, “bark” 611 is a subcategory of “dog” 610, and “dog” 610 is a subcategory of “pet” 701. The same applies to the category "cat" 710. The cat 710 has the categories of sounds “crowing” 711 and “throating sound” 71.
Have two. The following is an example of a simple classification scheme for "pets", which includes two categories, "dogs" and "cats."

In order to implement this classification scheme by extending the predefined scheme, a second scheme named "cat" is illustrated as follows.

[0042]

[Equation 3]

To combine these categories here, a classification scheme called "pet" is illustrated with reference to a predefined scheme.

[0044]

[Equation 4]

Here, the classification system called "pet" includes all of the category elements "dog" and "cat", and the additional category "pet" as a root.
The above qualitative classification methods are sufficient for character indexing applications.

The following section describes quantitative descriptors for classification and indexing that can be used with qualitative descriptors to form a complete phonetic indexing and search engine.

Quantitative Descriptor The Sound Recognition Quantitative Descriptor describes the characteristics of an audible signal that will be used with a statistical classifier. Sound recognition quantitative descriptors can be used for general sound recognition, including sound effects and musical instruments. In addition to the suggested descriptors, any other descriptor defined in the structure of audio band sounds can be used for classification.

The most widely used feature for the classification of audio frequency band spectral basis features is a spectral representation such as a power spectrum slice or frame. Each spectral slice is typically an n-dimensional vector, where n is the number of spectral channels and has up to 1024 channels of data. The number of dimensions is approximately 3 by the logarithmic frequency spectrum as represented by the structure descriptor of the audio frequency band sound.
It can be reduced to 2 channels. Therefore, the features derived by the spectrum are generally not compatible with stochastic model classifiers due to their high dimensionality. Probability classifiers work best with fewer than 10 dimensions.

Therefore, low dimensional basis functions generated by singular value decomposition (SVD) as described above and below are preferred. The audio spectrum sound spectrum basis descriptor is then a container for the basis functions used to project that spectrum into a low-dimensional subspace suitable for the stochastic model classifier.

The present invention determines the basis for each class and subclass of sounds. The basis captures the statistically most regular features of the sound feature space. The reduction of the number of dimensions is performed by projecting the spectrum vector on the matrix of the basis function derived from the data as described above. Basis functions are stored in columns of a matrix, where the number of rows corresponds to the length of the spectral vector and the number of columns corresponds to the number of basis functions. The base projection is the matrix product of the spectrum and the base vector.

Spectrogram Reconstructed from Basis Functions FIG. 8 shows a spectrogram 800 reconstructed from four basis functions according to the present invention. Its concrete spectrogram is for "pop" music. The left spectral basis vector 801 is combined with the basis projection vector 802 using the vector cross product. The resulting outer product matrices are added together to produce the final reconstruction. The basis functions are chosen to maximize the information in a smaller number of dimensions than the original data. For example, the basis functions correspond to uncorrelated features extracted using principal component analysis (PCA) or Karhunen-Loeve transform (KLT), or statistically independent extracted by independent component analysis (ICA). Can correspond to the components of. The KLT or Hotelling transform is the preferred inverse correlation transform when the quadratic statistic, or covariance, is known. This reconstruction is described in more detail with reference to FIG.

To serve the purposes of classification, a basis for all classes is derived. Thus, the classification space contains the most statistically significant components of that class. The following DDL instantiation defines a basis projection matrix that reduces the series of 31 channel logarithmic frequency spectra in five dimensions.

[0053]

[Equation 5]

The low edge, high edge, and resolution attributes are the lower and upper frequency limits of the basis function,
And the spacing of the spectral channels in octave band notation. In the classification structure according to the invention, the basis functions for all classes of sounds are stored along with the probabilistic model for that class.

Sound Recognition Features The features used for sound recognition can be aggregated into one description scheme that can be used for a variety of different applications. The default audio spectrum projection descriptors play a good role in the classification of many sound types, for example sounds obtained from sound effects libraries, and sample discs of musical instruments.

The base features are derived from the audio frequency band sound spectrum envelope extraction process as described above. The audio frequency spectrum projection descriptor is a container for reduced dimensionality features obtained by projection of the spectral envelope over a set of basis functions, also as described above. For example, the audible frequency band sound spectrum envelope is extracted by a sliding window FFT analysis along with resampling into frequency bands arranged in logarithm. In the preferred embodiment, the analysis frame period is 10 msec. However, a sliding extraction window of 30 msec duration is used in the Hamming window. The 30 msec interval is chosen to provide sufficient spectral resolution and generally resolve the 62.5 Hz wide first channel of the octave band spectrum. The size of the FFT analysis window is the next largest power of two samples. In case of 30 msec at 32 kHz, there are 960 samples,
It means that the FFT will be performed on 1024 samples. At 44.1 kHz for 30 msec, there are 1323 samples, but the FFT is 2048.
It will be executed in samples and samples outside the window will be set to zero.

9a and 9b show the three spectral basis components 901-903 for the time index 910 and the generated basis for the frequency index 920 for the "laughter" spectrogram 1000 in FIGS. 10a and 10b. Projections 911 to 913 are shown. The format here is similar to that shown in FIGS. 4 and 5. Figure 1
0a shows the logarithmic scale spectrogram of laughter, and FIG. 10b shows the reconstructed spectrogram. Both figures plot the time and frequency index on the x and y axes, respectively.

In addition to the base descriptors, a large sequence of other quantitative descriptors is used to identify special classes of sounds, such as harmonic envelopes and fundamental frequency features often used for instrument classification. A classifier can be defined using various properties.

One convenience of dimensionality reduction as done by the present invention is that any descriptor based on a scalable set of descriptors can be added to the spectral descriptor at the same sampling rate. Moreover, suitable bases can be calculated for the entire extended feature set in the same manner as spectrally based bases.

Spectrogram Summarization Using Basis Functions Another application for the feature description scheme of sound recognition according to the present invention is efficient spectrogram representation. To visualize and summarize the spectrogram, the audio frequency band sound spectrum basis projections and audio frequency band sound spectrum basis features can be used as a very efficient storage mechanism.

To reconstruct the spectrogram, the present invention uses Equation 2, which is described in more detail below. Formula 2
Constructs a two-dimensional spectrogram from the cross product of each basis function and its corresponding spectrogram basic projection, which is also shown in FIG. 8 as described above.

Stochastic Model Description Finite State Model Since the spectral features fluctuate over time, the phenomenon of sound is dynamic. This very large temporal variation gives the acoustic signal a characteristic "fingerprint" for recognition. Therefore, the model of the present invention divides the acoustic signal generated by a particular sound source or class of sounds into a finite number of states. The division is based on spectral features. Individual sounds are described by their trajectories through this state space. This model is shown in FIGS. 11a and 11b.
Will be described in more detail below. Each state can be represented by a continuous probability distribution such as a Gaussian distribution.

The dynamic behavior of a class of sounds passing through the state space is represented by a k × k transition matrix that describes the probability of transition to the next state given the current state. The transition matrix T models the probability of a transition from state i at time t-1 to state j at time t. The initial state distribution is a k × 1 vector of probabilities and is also typically used in finite state models. The kth element of this vector is the probability of being in state k in the first observation frame.

Gaussian Distribution Type The multidimensional Gaussian distribution is used to model states during sound classification. The Gaussian distribution is parameterized by a 1 × n vector of mean m and an n × n covariance matrix K. However, n is the number of features in each observation vector. Given the Gaussian parameters, the formula for calculating the probabilities for a particular vector x is:

[0065]

[Equation 6]

The continuous hidden Markov model is a finite state model having a continuous probability distribution model for state observation probabilities. The following DDL instantiation is an example of the use of a stochastic model description scheme to represent a continuous Hidden Markov Model with Gaussian states. In this example, floating point numbers are rounded to two decimal places for display purposes only.

[0067]

[Equation 7]

In this example, the "stochastic model" is instantiated as a Gaussian distribution type, which is derived from the base stochastic model class.

Sound Recognition Model Description Method Up to now, the method according to the present invention has separated tools without using any structure of application form. The following data types combine the above descriptors and description schemes into a unified structure for sound classification and indexing. The sound segment can be indexed with a category label based on the output of the classifier. In addition, the probabilistic model parameters can be used for indexing sounds in the database. Indexing by model parameters such as state is needed by the example query application when the query category is unknown or when a matching criterion narrower than the range of categories is needed.

Sound Recognition Model The sound recognition model description method specifies a stochastic model of a class of sounds such as a Hidden Markov Model or Gaussian Mixture Model. The example below shows the "croak" sound category 61 of FIG.
2 is an illustration of a Hidden Markov Model of 1. The probabilistic model and associated basis functions for that class of sounds are defined in the same way as for the example described above.

[0071]

[Equation 8]

Sound Model State Path This descriptor references a finite state probability model and describes the dynamic state path of the sound through that model. By segmenting the sound into model states or by sampling the state path at regular intervals, the sound can be indexed in two ways. In the first case, each audio band tone segment contains a reference to a state, and the duration of that segment indicates the effective duration for that state. In the second case, the sound is described by a series of sampled indices that refer to model states. Tone categories with relatively long state durations are efficiently described using a one-segment, one-state approach. Sounds with relatively short state durations are more efficiently described using a series of sampled state indices.

FIG. 11a shows the log spectrogram (frequency vs. time) 1100 of the dog bark sound 611 of FIG. FIG. 11b shows a sound model state pass sequence of states through a continuous Hidden Markov Model for the roaring model of FIG. 11a over the same time interval. In FIG. 11b, the x-axis is the time index and the y-axis is the state index.

Sound Recognition Classifier FIG. 12 shows a sound recognition classifier that uses one database 1200 for all required components of the classifier. The sound recognition classifier describes the relationships between multiple probabilistic models,
It defines the ontology of the classifier. For example,
Hierarchical recognizers have a wide range of animal-like sound classes at the root node and dogs: barks and cats: barks at the root nodes, as described in FIGS. 6 and 7. Such a finer class can be classified. This method uses the descriptor descriptor structure of the graph to define the correspondence between the ontology of the classifier and the taxonomy of the sound categories, and the hierarchical sound model is categorical for the given taxonomy. Be used to extract the description.

FIG. 13 shows a system 1300 for constructing a database of models. The system shown in FIG. 13 is an extension of the system shown in FIG. Here, the input acoustic signal is windowed before being filtered to extract the spectral envelope.
The system can capture audio band input 1301 in the form of, for example, a WAV format audio file. The system extracts audible frequency band sound features from the file and trains Hidden Markov Models with these features. The system also uses a directory of sound samples for each sound class. The hierarchical directory structure defines the ontology that corresponds to the desired taxonomy. One hidden Markov model is trained for each directory of its ontology.

Audio Frequency Band Sound Feature Extraction System 1300 of FIG. 13 illustrates a method for extracting audio frequency band sound spectrum basis functions and features from an acoustic signal, as described above. The input acoustic signal 1301 is 1
It can be produced by one sound source, eg a person, an animal, an instrument, or by a number of sound sources, eg a person and an animal, an instrument, or a synthetic sound. In the latter case, the acoustic signal is a mixture. Input sound signal is 10ms first
Window processing is performed on the ec frame (1310). Note that in FIG. 1, the input signal is bandpass filtered before windowing. Here, the acoustic signal is first windowed, then filtered (1320), and the short-time logarithmic frequency spectrum (sh
ort-time logarithmic-in-frequency spectrum) is extracted. Filtering is squared size (squared-
magnitude) Perform a time-frequency power spectrum analysis, such as a short time Fourier transform. The result is a matrix with M frames and N frequency bins. The spectral vector x is the row of this matrix.

Step 1330 performs logarithmic scale normalization. Each spectrum vector x is calculated from the power spectrum on the decibel scale 1331 and z = 10 log.
It is converted by ₁₀ (x). Step 1332 determines the L2 norm of the vector element as follows.

[0078]

[Equation 9]

The new unit norm spectral vector is then determined as the spectral envelope (~) X by z / r, which is each slice z divided by its power r, resulting in the normalized spectral envelope (~) X1340. Is passed to the base extraction process 1360. Note that (~) X
Indicates that ~ is attached to X.

The spectral envelope (-) X makes each vector look like a row in the form of an observation matrix. The resulting matrix size is M × N. However, M is the number of time frames and N is the number of frequencies (frequency bins). The matrix will have the following structure:

[0081]

[Equation 10]

Basis Extraction Basis functions are extracted using the singular value decomposition SVD 130 of FIG. SVD is command [U, S, V] = SVD
Performed using (X, 0). It is preferred to use a "brief" SVD. The concise SVD omits unnecessary rows and columns during SVD factorization. In the present invention, since the row basis function is not necessary, the SVD extraction efficiency is high. SVD factors a matrix as follows. (~)
X = USV ^T However, (˜) X is decomposed into a matrix product of three matrices, U is a row basis, S is a diagonal singular value matrix, and V is
Is the transposed column basis function. The basis is the first K
It is reduced by retaining only B basis functions, ie only the first K columns of V.

[0083]

[Equation 11]

However, K is typically in the range of 3-10 basis functions for sound feature applications. The singular values contained in the matrix S are used to determine the proportion of information retained for the K basis functions.

[0085]

[Equation 12]

However, I (K) is the ratio of information held in the case of K basis functions, and N is the spectrum (spec
The total number of basis functions equal to the number of tral bins). S
The VD basis functions are stored in the columns of that matrix.

For maximum compatibility between applications, the basis functions include columns with unit L2 norm,
The function maximizes k-dimensional information relative to other possible basis functions. The basis functions can be orthogonal as provided by PCA extraction or non-orthogonal as provided by ICA extraction. See below. The basic projection and reconstruction are described by the following analysis-synthesis formula.

[0088]

[Equation 13]

Where X is the spectral envelope and Y
Is a spectral feature and V is a temporal feature. Spectral features are extracted from the m × k observation matrix of features,
X is an m × n spectral data matrix in which spectral vectors are organized in rows, and V is an n × k matrix of basis functions organized in columns.

The first equation corresponds to feature extraction and the second equation corresponds to spectral reconstruction. See FIG. 8. However, V ⁺ represents the pseudo inverse matrix of V in the case of non-orthogonality.

Independent Component Analysis After the reduced SVD basis V has been extracted, an optional step can perform basis rotation in maximally statistically independent directions.
This separates the independent components of the spectrogram and is useful for all applications requiring maximum separation of features. It is possible to use any of the well-known and widely introduced independent component analysis (ICA) processes to find statistically independent bases using the previously obtained basis functions. it can. For example, JAD
E or FastICA, Cardoso,
J. F. And Laheld, B .; H. By "Equiva
riant adaptive source separation "(IEEE Trans. On
Signal Processing, 4: 112- 114, 1996) or H
Yvarinen, A, “Fast and robust fixed-
point algorithms for independent component analysi
s '' (IEEE Trans. On Neural Networks, 10 (3): 626-6
34, 1999).

The following use of ICA uses the set of vectors as a statistically independent vector [(−) V ^T _k , A] =
Decompose into ica (V ^T _k ). However, the new basis is
It is obtained as the product of the SVD input vector and the pseudo-inverse of the estimated mixing matrix A given by the ICA process. The ICA basis has the same size as the SVD basis and is stored in the columns of the basis matrix. Ratio of information held I
(K) is equivalent to SVD when using a given extraction method. The basis function (−) V _K 1361 is the database 1
Can be stored in 200. In addition, (-) V is-
Indicates that V is attached above V.

When the input acoustic signal is a mixture produced from multiple sources, the set of features produced by SVD is by any known clustering technique having a dimensionality equal to that of the features. It can be clustered as a group. This brings similar features together in the same group. Thus, each group contains features of the acoustic signal produced by one sound source. The number of groups to be used in clustering can be set manually or automatically, depending on the level of discrimination desired.

Utilization of Spectral Subspace Basis Function In order to determine the projection or temporal feature Y, the spectral envelope matrix X is multiplied with the basis vector of the spectral feature V. This step is for SVD and ICA
The basis function is the same in any case, that is, (~) _Yk = (~) X (-) _Vk . However, Y
Is a matrix of features whose dimensionality is reduced after projection of the spectrum on the basis V.

For independent spectrogram reconstruction and visualization, the present invention extracts the unnormalized spectral projections by omitting the normalization step 1330 extraction. That is, _Yk = X (-) _Vk . Now, to reconstruct an independent spectrogram,
The X _k component as shown in FIG. 8 uses the individual vector pairs corresponding to the Kth projection vector y _k and the Kth inverse basis vector v _k , and the reconstruction formula X _k = y _k (−) v
Apply ⁺ _k . However, the “+” operator indicates the transpose for the SVD basis function, which is orthogonal, or is the pseudo-inverse matrix for ICA and is non-orthogonal.

Spectrogram Summarization with Independent Components One use form for these descriptors is to efficiently represent the spectrogram with less data than the complete spectrogram. Using the independent component basis, each spectrogram reconstruction, such as that shown in FIG. 8, generally corresponds to a source object in the spectrogram.

Much of the difficult work in designing model acquisition and training sound classifiers is spent collecting and preparing training data. The range of sounds will reflect the range of categories of sounds.
For example, the barking of dogs can include individual barking, multiple barkings in sequence, or barking of many dogs at once. The model extraction process adapts to a range of data such that a narrower range of samples produces a more specialized classifier.

FIG. 14 illustrates a process 140 for extracting features 1410 and basis functions 1420 from an acoustic signal produced by a known sound source 1401 as described above.
Indicates 0. Then, they are used to train a Hidden Markov Model (1440). The trained models are stored in the database 1200 along with their corresponding features. During training, an unsupervised clustering process is used to partition the n-dimensional feature space into k states. The feature space is occupied by observation vectors with reduced dimensionality. The process determines the optimal number of states for a given data by truncating the transition matrix when giving an initial guess for k. Typically, 5-10 states are sufficient for good classifier performance.

The hidden Markov model is Forward-
It is trained on a variation of the well known Baum-Welch process, also known as the Backward process. These processes are extended by the use of entropic priors and the implementation of deterministic annealing of the expected maximum (EM) process.

Appropriate HMM training process 143
For more on 0, see "Pattern dis
covery via entropy minimization "(Proceedings, Un
certainty'99. Society of Artificial intelligence a
nd Statistics # 7, Morgan Kaufmann, 1999) and Bran
`` Structure discovery in conditional probab by d
ility models via an entropic prior and parameter e
xtinction ”(Neural Computation, 1999).

After each HMM for each known sound source has been trained, its model is stored in persistent storage 1200 along with its basis functions, ie, the set of sound features. Corresponding to the overall sound category taxonomy, when multiple sound models are being trained, both HMMs are collected in a larger sound recognition classifier data structure,
Thereby, an ontology of the model as shown in FIG. 12 is generated. The ontology is used to index new sounds with qualitative and quantitative descriptors.

Sound Descriptor FIG. 15 shows an automatic extraction system 1500 for indexing sounds in a database using a pre-trained classifier stored as a DDL file. The unknown sound is read from a medium sound source format such as the WAV file 1501. The unknown sound is spectrally projected 1520 as described above. Then, using that projection, or set of features, from database 1200 to H
One of the MMs is selected (1530). The Viterbi decoder 1540 can be used to provide both the optimal model and the state path through the model for that unknown sound. That is, there is one model state for each windowed frame of the sound. Figure 11b
Please refer to. Each sound is then indexed by its category, model reference and model state path, and its descriptor is written to the database in DDL format.
The indexed database 1599 may then be searched to find a matching sound using any of the stored descriptors as described above, eg, all dog barks. it can. A generally similar sound may then be provided in the results list 1560.

FIG. 16 shows ten sound classes 1601 to 1601.
1610, respectively, crowing of birds, applause of applause, barking of dogs, roaring sounds, footsteps, sounds of breaking cups, gunshots, sports shoes,
Shows classification performance for laughter and phone calls. The performance of the system is determined by using the sound effect labels as specified by the expert sound effect library to
Measured against Truth. The results shown are for new sounds that are not used during training of the classifier, and thus exemplify the generalization ability of the classifier. Its average performance is about 95% accurate.

Sample Search Application The following section gives an example of how to use the description scheme to perform a search using both DDL matching and media source format queries.

Example Query Using DDL As shown in FIG. 17 in simplified form, a phonetic query is presented to system 1700 using a DDL-formatted sound model state path description 1710. The system reads the query and populates an internal data structure with its descriptive information. This description is matched (1550) with the description retrieved from the sound database 1599 stored on disk. A sorted result list 1560 of the most similar sounds is returned.

The matching step 1550 can use the sum of squared errors (SSE) between the state path histograms. This matching procedure requires very little computation and can be calculated directly from the stored state path descriptors.

The state path histogram is the total length of time a note spends in each state divided by the total length of the note, thereby giving a discrete probability density function with the state index as a random variable. SS between the query sound histogram and the histogram of each sound in the database
E is used as a distance metric. The fact that the distance is 0 implies that they are exactly the same, and when the distance increases with a value other than 0, it means that they are much different. This distance metric is used to rank the sounds in the database for similarity, returning the desired number of the closest listed first from the top.

FIG. 18a shows the state path.
b shows a state path histogram for a laughing sound query. Figure 19a shows the state path and Figure 19b shows
A histogram for the five best matching sounds for the query is shown. All matching sounds are from the same class as the query, indicating that the system is working correctly.

To take advantage of the ontology's structure, the sounds in the equal or narrower categories, as defined by the taxonomy, are returned as matching sounds. Thus, the "dog" category will return sounds that belong to all categories associated with "dog" in a taxonomy.

Example Queries Using Audio Band Sounds The system may also perform queries using audio band signals as input. Here, the input for the example query application is a query by audible frequency band sound, instead of a query by DDL description. In this case,
The audio frequency band feature extraction process is performed first, that is, the spectrogram and envelope extraction, followed by the projection on the set of stored basis functions for each model in the classifier.

The resulting reduced dimensionality features are passed to the Viterbi decoder for the given classifier and the HMM with the maximum likelihood score for the given feature is selected. . The Viterbi decoder generally functions as a model matching algorithm for that classification scheme. The model reference and state path are recorded and the results are collated against a precomputed database as in the first case.

It should be understood that various other adaptations and modifications may be made within the spirit and scope of the invention. Therefore, the purpose of the appended claims is to cover all such variations and modifications as fall within the true spirit and scope of the invention.

[Brief description of drawings]

1 is a flow chart of a method for extracting features from a mixture of signals according to the present invention.

FIG. 2 is a block diagram of filtering and windowing steps.

FIG. 3 is a block diagram of the steps of normalizing, reducing and extracting.

FIG. 4 is a graph of characteristics of a metal percussion instrument.

FIG. 5 is a graph of characteristics of a metal percussion instrument.

FIG. 6 is a block diagram of a descriptive model for a dog barking.

FIG. 7 is a block diagram of a descriptive model regarding the sound of a pet.

FIG. 8 is a spectrogram reconstructed from four spectral basis functions and basis projections.

FIG. 9a is a base-projective envelope for laughter.

9b is an audible frequency band sound spectrum for the laughter of FIG. 9a.

FIG. 10a is a spectrogram on a logarithmic scale for laughter.

FIG. 10b is a reconstructed spectrogram for laughter.

FIG. 11a is a spectrogram on a logarithmic scale when a dog barks.

FIG. 11b is a sequence diagram of a sound model state path through a continuous hidden Markov model when the dog of FIG. 11a barks.

FIG. 12 is a block diagram of a sound recognition classifier.

FIG. 13 is a block diagram of a system for extracting sounds according to the present invention.

FIG. 14 is a block diagram of a process for training a Hidden Markov Model according to the present invention.

FIG. 15 is a block diagram of a system for identifying and classifying sounds according to the present invention.

16 is a graph of performance of the system of FIG.

FIG. 17 is a block diagram of a sound inquiry system according to the present invention.

18a is a block diagram of a laughing state path. FIG.

FIG. 18b is a histogram of laughing state paths.

FIG. 19a is a diagram showing matching laughing state paths.

FIG. 19b is a histogram of matching laughing state paths.

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Michael A. Kasei             Que, Massachusetts, United States             Bridge, Chauncey Street             26, number 9 F-term (reference) 5D015 AA06 HH23

Claims (18)

[Claims] 1. A method for extracting features from an acoustic signal generated from one sound source, the method comprising windowing and filtering the acoustic signal to generate a spectral envelope. Reducing the dimensionality of the spectral envelope to generate the feature of
A method for extracting features from an acoustic signal generated from one sound source, the method comprising: spectral features characterizing one sound source and steps including corresponding temporal features. 2. The source of claim 1, further comprising the step of multiplying the spectral features with the temporal features using an outer product to reconstruct the spectrogram of the acoustic signal. A method for extracting features from an acoustic signal. 3. To separate the features in the set:
The method for extracting features from an acoustic signal generated from a single sound source according to claim 1, further comprising applying an independent component analysis to the set of features. 4. The spectral envelope is logarithmically scaled prior to reducing the dimensionality of the spectral envelope, L
The method for extracting features from an acoustic signal generated from one source according to claim 1, further comprising the step of normalizing by 2 to a decibel scale and a unit L2 norm. 5. A method for extracting features from an acoustic signal generated from a plurality of sound sources, the method comprising windowing and filtering the acoustic signal to generate a spectral envelope. Reducing the dimensionality of the spectral envelope to produce a feature of the plurality of sources, and clustering the features in the set to produce a group of features for each source of the plurality of sources. A step of: extracting the features from an acoustic signal generated from a plurality of sound sources, the features in each group comprising spectral features characterizing each of the sound sources and corresponding temporal features; Method. 6. The feature of each group is a quantitative descriptor of each of the sound sources, and associates a qualitative descriptor with each of the quantitative descriptors to generate a category for each of the sound sources. The method for extracting features from acoustic signals generated from a plurality of sound sources according to claim 5, further comprising steps. 7. The method of organizing categories in a database as classified sound sources of a taxonomy, and associating each of the categories in the database with at least one other category by an associative link. 7. A method for extracting features from an acoustic signal generated from a plurality of sound sources according to claim 6, comprising. 8. The category is a description definition language (DD).
Method for extracting features from acoustic signals generated from a plurality of sound sources according to claim 7 stored in the database using L). 9. A particular category within a DDL instantiation is generated from a plurality of sources as defined in claim 8 which defines a basis projection matrix that reduces the series of logarithmic frequency spectra of the particular source to a smaller number of dimensions. A method for extracting features from an acoustic signal. 10. The category includes environmental sounds, background noise,
7. A method for extracting features from acoustic signals generated from multiple sound sources according to claim 6, including sound effects, overlapping sounds, animal sounds, voices, non-voice calls and music. 11. The method for extracting features from an acoustic signal generated from a plurality of sound sources according to claim 7, further comprising the step of combining generally similar categories in the database as a hierarchy of classes. 12. The specific quantitative descriptor for extracting features from an acoustic signal generated from a plurality of sound sources according to claim 6, further comprising a harmonic envelope descriptor and a fundamental frequency descriptor. Method. 13. The temporal feature describes a trajectory of the spectral feature over time, and the acoustic signal generated by a specific sound source is subjected to a finite number of states based on the corresponding spectral feature. To represent each of the states by a continuous probability distribution, and to model the probability of transition to the next state when the current state is given, the temporal features are represented by a transition matrix. The method for extracting features from an acoustic signal generated from a plurality of sound sources according to claim 5, further comprising: 14. The continuous probability distribution has a mean value m of 1 ×
is a Gaussian distribution parameterized by an n vector and an n × n covariance matrix K, where n is the number of spectral features in each spectral envelope and the probability of a particular spectral envelope x is Number 1] A method for extracting features from an acoustic signal generated from a plurality of sound sources according to claim 13, provided by: 15. Each of the sources is known, and for each of the known sources, training a hidden Markov model with the set of features, and each training with an associated set of spectral features. The method for extracting features from acoustic signals generated from a plurality of sound sources according to claim 5, further comprising the step of storing the hidden Hidden Markov Model in a database. 16. The set of acoustic signals belongs to a known category, extracting a spectral basis for the acoustic signals, and training a hidden Markov model using the temporal features of the acoustic signals. , Storing each trained Hidden Markov Model having said associated spectral basis function, the method for extracting features from an acoustic signal generated from a plurality of sound sources according to claim 5. 17. A step of generating an unknown acoustic signal from an unknown sound source, a step of windowing and filtering the unknown signal to generate an unknown spectral envelope, and a set of unknown features. Reducing the dimensionality of the unknown spectral envelope to generate the set, the set including unknown spectral features that characterize the unknown sound source and corresponding unknown temporal features. The method further comprising the steps of: and selecting one of the stored Hidden Markov Models that best fits the unknown feature set to identify the unknown source.
A method for extracting features from an acoustic signal generated from a plurality of sources as described. 18. The sound generated from multiple sound sources of claim 17, wherein a plurality of the stored hidden Markov models are selected to identify a plurality of unknown sound sources that are generally similar to the unknown sound source. A method for extracting features from a signal.