KR20050084039A - Summarizing digital audio data - Google Patents

Abstract

An embodiment is related to automatic summarization for digital audio raw data (12), more specifically, for identifying pure music and vocal music (40,60) from digital audio data by extracting distinctive features from music frames (73,74,75,76), designing a classifier and determining the classification parameters (20) using adaptive learning/training algorithm (36), and identifying music into pure music or vocal music according to the classifier. For pure music, temporal, spectral and cepstral features are calculated to characterise the musical content, and an adaptive clustering method is used to structure the musical content according to calculated features. The summary (22,24,26,48,52,70,72) is created according to clustered result and domain-based music knowledge (50,150). For vocal music, voice related features are extracted and used to structure the musical content, and similarly, the music summary is created in terms of structured content and heuristic rules related to music genres.

Description

Digital audio data summarizing method {SUMMARIZING DIGITAL AUDIO DATA}

The present invention relates to data analysis, such as audio data indexing and classification. More specifically, the present invention relates to a method of automatically summarizing digital music raw data for various applications such as, for example, content-based music search and web-based online music distribution.

Due to the rapid development of computer network and multimedia technology, the size of digital multimedia data collection has been rapidly increased. This development requires a concise and informative summary of the vast collection of multimedia data that best captures the essential elements of the original content among large-scale information organization and processing. To date, many techniques have been proposed and developed for automatically generating text, speech and video summary. However, music summerization is to determine the most common and salient theme of a given music that can be used as a representative of the music and easily recognized by the listener. Since raw digital music data is a collection of featureless bytes that are only available in the form of highly unstructured monolithic sound files, music summering presents a particular challenge.

United States Patent No. 6,225,548 to International Business Machines Corporation, registered May 1, 2001, relates to music summerization, which uses the repeating nature of a musical instrument design interface (MIDI) composition to incorporate the main melody theme segment of a given piece of music. A summary system for automatically recognizing MIDI data formats is disclosed. The detection engine uses an algorithm that models melody recognition and music summerization problems as various string processing problems, and handles the problems. The system recognizes the longest segment with no normal repeats on each track of a musical piece in MIDI format. These segments are the basic unit of music composition and will be the melody of the piece of music. However, the MIDI format data is not sampled raw audio data, i.e., active audio sound. Instead, MIDI format data plays audio data, including synthesizer instructions or MIDI notes. In particular, the synthesizer produces an actual sound from instructions of MIDI format data. Compared with the actual audio sound, the MIDI data will not provide a common playback experience and unlimited sound palettes for both the instrument and sound effects. On the other hand, the MIDI data is in a structured format, and it is easy to generate a summary according to the structure. Therefore, MIDI summarization is not practical for real-time playback applications. Therefore, it is necessary to generate a music summary from real one digital audio data.

Speech by Beth Logan and Stephen Chu (IEEE International Conference on Audio, Speech and Signal processing, Orlando, USA, 2000, vol. 2, pp749-752) entitled "Music Summering with Key Phrases" A method of summering music by parameterizing each song using the "Mel-cepstral" feature known to be used in an application is disclosed. This feature of speech recognition can be applied along with various clustering techniques to determine the song structure of a musical piece using voice. Then, you use it to summarize the key phrases given in this structure. Such a summation method is suitable for a specific genre of music such as rock or folk music, but this method is difficult to apply to pure music or musical instrument genre such as classical or jazz music. The "mel-septral" feature will not uniquely reflect the nature of the music content, in particular pure music, such as, for example, musical instrument playing music. Thus, the quality of the summation in this way is not satisfactory, especially for applications that require music summerization of all types of music genres.

Thus, automatic music summery of digital music source data that can be applied to music indexing of all types of music genres, for example for use in content-based music search and web-based music distribution of real-time playback applications. I need a ration.

These and other features, objects, and advantages of embodiments of the present invention will become more readily understood and more apparent to those skilled in the art from the detailed description set forth below in connection with the drawings.

1 is a block diagram of a system used to generate an audio file summary according to one embodiment of the invention;

2 is a flowchart illustrating a method of generating an audio file summary according to an embodiment of the present invention;

3 is a flowchart of a training process for generating classification parameters of the classifier of FIGS. 1 and 2 in accordance with one embodiment of the present invention;

4 is a more detailed flowchart of the pure music summerization of FIG. 2 in accordance with an embodiment of the present invention;

FIG. 5 illustrates a block diagram of the vocal music summerization of FIG. 2 in accordance with an embodiment of the present invention. FIG.

6 illustrates a graph illustrating segmentation of audio raw data into overlapping frames, in accordance with an embodiment of the present invention; And

FIG. 7 illustrates a two-dimensional representation of the distance matrix of the frames of FIG. 6 in accordance with an embodiment of the present invention. FIG.

Embodiments of the present invention inherently provide for automatic summation of digital audio data, such as highly structured musical raw data. One embodiment provides a summary for audio files such as pure and / or vocal music such as classical, jazz, pop, rock or playing music. Another feature of one embodiment is the use of an adaptive training algorithm to design a classifier for identifying pure music and vocal music. Another feature of one embodiment is the use of an adaptive clustering algorithm to structure the music content and apply domain-based music knowledge to generate music summaries for pure and vocal music. One embodiment extracts distinct features from music frames, determines classification parameters using an adaptive learning / training algorithm, and identifies music as pure music or vocal music according to the classifier, thereby identifying pure music and vocal music. It provides automatic summation of digital audio raw data for identification from digital audio data. In pure music, temporal, spectral and spectral features are calculated to characterize the music content, and an adaptive clustering method is used to structure the music content according to the calculated features. The summary is generated according to the clustered results and domain-based music knowledge. In vocal music, voice related features are extracted and used to structure the music content, and similarly the music summary is created with respect to heuristic rules and structured content related to the genre of music.

According to one embodiment of the invention,

Analyzing the audio data to identify a representation of the audio data having one or more calculated feature characteristics of audio data; Based on the representation, classifying the audio data into a predetermined category selected from two or more categories; And generating an acoustic signal representing the thermalization of the digital audio data, wherein the thermalization provides a method of summarizing digital audio data depending on the selected predetermined category.

In other implementations, the analyzing may further comprise segmenting audio data into segment frames, and overlapping the frames, and / or classifying the respective ones. Classifying the frames into a predetermined category by collecting training data from a frame, and determining classification parameters by using a training calculation.

According to another embodiment of the present invention,

A feature extractor for receiving audio data and analyzing the audio data to identify a representation of the audio data having one or more calculated feature characteristics of the audio data; A classifier in communication with the feature extractor, classifying the audio data based on the representation received from the feature extractor into a predetermined category selected from two or more categories; And a summerizer in communication with the classifier for generating an acoustic signal representing the summerization of the digital audio data, wherein the summerization comprises digital audio data depending on the category selected by the classifier. A device for summarizing is provided.

In other implementations, the apparatus may further comprise a segmenter in communication with the feature extractor to receive an audio file and segment the audio data into segment frames, overlapping the frames with respect to the feature extractor. You may. The apparatus may further comprise a classification parameter generator in communication with the classifier, wherein the classifier classifies each of the frames into a predetermined category by collecting training data from each frame, and the classification parameter generator The classification parameters are determined by using a training calculation in.

According to another embodiment of the present invention,

In order to summarize the digital audio data, a computer program product comprising the medium having computer readable program code means embodied in a computer usable medium is provided, the computer program product analyzing the audio data, Computer readable program code means for identifying a representation of the audio data having one or more calculated feature characteristics of the audio data; Computer readable program code for classifying the audio data into a predetermined category selected from two or more categories based on the expression; And computer readable program code for generating an acoustic signal representing the thermalization of the digital audio data, wherein the thermalization is dependent on the selected category.

1 is a block diagram illustrating components and / or modules of a system 100 used to generate an audio summary according to one embodiment of the invention. The system may receive an audio file, such as music content 12, at segment 114. Music sequence 12 is segmented into frames, and features are extracted into each frame in feature extractor 116. Classifier 118 based on classification parameters supplied from classification parameter generator 120 classifies the feature-extracted frames into categories, such as pure music sequence 140 or vocal music sequence 160. Pure music is defined as music content without a singing voice, and vocal music is defined as music content with a singing voice. An audio summary is generated in one of the music summerizers 122, 124 where the audio content performs a summerization of any of the specifically designed audio content for the category categorized by the classification 118, and the audio knowledge module or It may be calculated with the help of information of specific categories of audio content remaining in the lookup table 150. Although both summerizers are shown in FIG. 1, if all audio files contain only one type of music content such as pure music or vocal music, only one summerizer may be required for one type of audio file. I can understand that. 1 shows two summerizers that can be implemented for two general types of music, such as, for example, pure music summerizer 122 and vocal music summerizer 124. The system then provides an audio sequence summary such as, for example, music summary 26.

The embodiment shown in FIG. 1 and the methods discussed herein may be implemented within computer architectures and / or on computer architectures generally well known in the art. The functionality of the described embodiments of the invention may be implemented in either hardware or software. In software sense, the components of a system can be a process, a program, or a part thereof, which typically performs a particular function or related functions. In hardware sense, a component is a functional hardware unit designed for use with other components. For example, a component may be implemented using discrete electrical components or may form part of an overall electronic circuit, such as an application specific integrated circuit (ASIC). Many other possibilities exist and one skilled in the art will appreciate that the system may also be implemented in a combination of hardware and software components.

Personal computer or servers are examples of computer architectures in which embodiments are implemented. These computer architectures include a central processing unit (CPU) with a microprocessor, random access memory (RAM) and read only memory (ROM) for temporary and permanent storage of information, and mass storage devices such as hard drives, diskettes, or CD ROMs. It comprises a component and / or modules such as. Such computer architectures also include a bus for interconnecting the components and controlled information and for communicating between the components. In addition, user input and output interfaces are generally provided, such as a display, printer, speaker, etc., for outputting a keyboard, mouse, microphone and the like for user input. In general, each input / output interface is connected to the bus by a controller and implemented in controller software. Of course, it is apparent that any number of input / output devices may be implemented in such systems. The computer system is typically controlled and managed by operating system software residing on the CPU. There are several commonly used and well known operating systems. Thus, embodiments of the invention may be implemented within and / or on such computer architectures.

2 illustrates a block diagram of the components of the system and / or method 10 used to automatically generate an audio summary such as a music summary according to one embodiment of the invention. This embodiment begins with receiving incoming audio data. Incoming audio data, such as audio file 12, may comprise a music sequence or content, for example. The music content is first segmented into frames in segmentation step 14. Then, in the feature extraction step 16, features such as linear prediction coefficients, zero crossing rates and mel-frequency cepstral coefficients are extracted and together By calculating and forming the feature vector of each frame, the characteristics of the music content are represented. The feature vector of each frame of the entire music sequence is passed through the classifier to categories such as pure or vocal music. It will be appreciated that any number of categories may be used. The classification parameters 20 of the classifier 18 are determined by the training / classification process shown in FIG. 3. Once classified into audio categories, such as music categories such as pure music 40 or vocal music 60, each category is then summed up and provided to the audio summary 26 to finish. For example, pure music summerization step 22 is shown in detail in FIG. Similarly, vocal music summerization step 24 is shown in detail in FIG.

3 shows a conceptual diagram of a diagram of an embodiment training / classification parameter process 38 for generating classification parameters 20 of a classifier 18 (shown in FIG. 2) according to one embodiment of the invention. To illustrate. In order to identify the music content into different categories, such as pure music or vocal music, a classifier 18 is provided. The classification parameters 20 for the classifier 18 are determined by the training process 38. The training process analyzes the music training sample data to find the best way to classify the music frames into classifications such as, for example, vocal 60 or non-vocal 40 classes. The training audio 30 should be sufficient to be statistically significant, such as, for example, that the training data must originate from several sources and include several musical genres. Training sample audio data may also be segmented into fixed-length and overlapping frames as discussed in the segmentation of FIG. 2. Features such as linear prediction coefficients, zero crossing rate, mel-frequency septal coefficients and the like are extracted from each frame. The features selected for each frame best characterize the classification, for example the features selected for the vocal classes are the features that best characterize the vocal classes. The computed features are clustered by training algorithms 36, such as hidden Markov models, neural networks, and support vector machines, to generate classification parameters 20. . While these other training algorithms may be used, some training algorithms may be more suitable for any particular application. For example, the support vector machine training algorithm may perform good classification results, but the training time is longer than other training algorithms. The training process needs to be performed only once, but may be performed several times. The derived classification parameters are used to identify different classifications of audio content such as non-vocal or pure music and vocal music.

FIG. 4 illustrates a conceptual block diagram of one embodiment of pure music summerization, and FIG. 5 illustrates a conceptual block diagram of one embodiment of vocal music summerization. The purpose of summerization is to analyze given audio data, such as music sequences, and extract important frames to reflect the prominent theme of the music. Based on the calculated features of each frame, an adaptive clustering method is used to group the structure of music frames and music content. Since adjacent frames overlap, the length of the overlap is determined for frame grouping. In the initial stage, it is difficult to accurately determine the length of the overlap. The length of the overlap can be adaptively adjusted if the clustering result is not ideal for frame grouping. An example of a general clustering algorithm is described below:

(1) As shown in FIG. 6, in the segmenter 144 or the segmentation steps 42 and 62, the segment music signal becomes N fixed-lengths 73, 74, 75, and 76, for example, FIG. 6. Provide 50% overlapping frames 77, 78, 79 as shown in, labeling each frame with the number i (i = 1, 2, ... N), the initial set of clusters being all frames admit. The segmentation process in steps 42 and 62 will also follow the same procedure of the segmentation process performed in other situations, such as segmentation steps 14 and 32 as detailed and shown in FIGS. 2 and 3. May be;

(2) For each frame, feature extraction values are calculated in feature extraction steps 44, 64 specified for a particular category of audio file, such as, for example, linear prediction coefficients, zero crossing rate, and mel-frequency septal coefficients. Form the feature vector:

Here, LPC _i denotes a linear prediction coefficient, ZCR _i denotes a zero crossing rate, and MFCC _i denotes a mel-frequency septal coefficient.

(3) Calculate the distances between all pairs of music frames i and j, using, for example, Mahalanobis distances:

Where R is the covariance matrix of the feature vector. Since R ⁻¹ is symmetrical, R ⁻¹ is a semi or positive matrix. R ⁻¹ may be diagonalized to R ⁻¹ = P ^T ΛP, where Λ is a diagonal matrix and P is an orthogonal matrix. Equation (2) can be simplified for Euclidean distance as follows:

Since Λ and P may be calculated directly from R ⁻¹ , the complexity of calculating the vector distance can be reduced from O (n ² ) to O (n).

(4) As shown in FIG. 7, the calculated distance is embedded in the two-dimensional representation 80. Since matrix S 80 includes a similarity metric calculated for all frame combinations, the frame indexes i and j such that the i, j th element of S is D (i, j).

(5) For each row of the two-dimensional matrix S, if the distance between two predetermined frames is less than a pre-defined threshold, for example in this embodiment the pre-defined threshold is a value such as 1.0. In this case, the frames are grouped into the same cluster.

(6) If the final clustering result is not ideal, adjust the overlap length of the two frames as shown by arrow 45 in FIG. 4 and arrow 65 in FIG. 5 and repeat steps (2) to (5). For example, in the above embodiment, the ideal result means that the number of clusters is much smaller than the number of initial clusters after clustering. If the result is not ideal, the overlap may be adjusted by changing the overlapping length from 50% to 40%, for example.

With reference to clustering for certain categories, FIG. 4 shows a summerization process for future / non-vocal music, and FIG. 5 shows a summerization process for vocal music. In FIG. 4, the pure music content 40 is first segmented 42, e.g., with the lengths of the fixed-length and overlapping frames described above, and then feature extraction 44 is performed in each frame described above. The extracted features may include amplitude envelopes, power spectra, mel-frequency septal coefficients, and the like, which may characterize pure music content in the temporal, spectral, and septal domains. It can be appreciated that other features can be extracted to characterize pure music content, which is not limited to the features listed here. Based on the calculated features, the adaptive clustering 46 algorithm is applied to group the frames and obtain the structure of the music content. The segmentation and adaptive clustering algorithm may be the same as described above. For example, if the clustering result is not ideal in decision steps 47 and 69 after the first pass, segmentation steps 42 and 62 and feature extraction steps 44 and 64 are repeated in frames with different overlapping relationships. . This process is repeated in the queuing steps 47 and 69 as shown by arrows 45 and 65 until the desired clustering result is achieved. After clustering, frames with similar features are grouped into identical clusters that represent the structure of the music content. The summary generation 48 is then performed in terms of this structure and domain-based music knowledge 50. According to music knowledge, most distinct or expressive music themes must occur repeatedly in the overall music work.

The length of the summary 52 should be long enough to represent most distinct or expressive proficiency of the overall music. Typically, for 3-4 minute pieces of music, 30 seconds is the appropriate length of the summary. An example of generating a summary of a music task is described as follows:

(1) Identify the cluster containing the maximum amount of frames. The labels of these frames are f ₁ , f ₂ , ... ... f _n , where f ₁ <f ₂ <...... <f _n ;

(2) From these frames, select the frame with the minimum label f _i according to the following rule:

for m = 1 to k,

If frame (f _i + m) and frame (f _j + m) belong to the same cluster, i, j ∈ [1, n], i <j, k are numbers for determining the length of the summary;

(3) Frames (f _i +1), (f _i +2), ..., and (f _i + k) are the final summary of the music.

5 illustrates a conceptual block diagram of vocal music summerization according to one embodiment. Vocal music content 60 is first segmented 62 into fixed-length and overlapping frames that can be performed in the same manner as described above. Feature extraction 64 is performed in each frame. The extracted features include linear prediction coefficients, zero crossing rate and mel-frequency septal coefficients, etc., which can characterize vocal music content. Of course, as described above with respect to non-vocal music, it can be seen that other features may be extracted to characterize vocal music content, and are not limited to the features listed here. Based on the calculated features, vocal frames 66 are placed and other non-vocal frames are discarded. Adaptive clustering algorithm 68 is applied to group these vocal frames and obtain the structure of vocal music content. The segmentation and adaptive clustering algorithm may be the same as above, for example, if the clustering result is not ideal, segmentation step 62 and feature extraction step 64 are repeated with frames having different overlapping relationships. The process is repeated as shown in decision step 69 until the desired clustering result is achieved and branches 65 in FIG. 5. Finally, the music summary 70 is generated based on the clustered results and the music knowledge 50 related to vocal music.

The saturation process 72 for vocal music is similar to that of pure music, but there are a number of differences, such as may be stored as music knowledge 50, such as the music knowledge module or lookup table 150 of FIG. The first difference is feature extraction. For pure music, power-related features such as amplitude envelope and power spectrum are used because voice-related features may better represent the characteristics of pure music content. The amplitude envelope is calculated in the time domain, while the spectral power is calculated in the frequency domain. In the case of vocal music, voice-related features such as linear prediction coefficients, zero crossing rate, and mel-precision septal coefficients are used because they may better express the characteristics of the vocal music content.

Another difference between the pure music and vocal buccal summerization process is the summary generation. For pure music, the summary is still pure music. However, in the case of vocal music, the summary should start with the vocal part and it is desirable to have a music title that is called out in the summary. There are some other rules related to the music genre, which may be stored as music knowledge 50. For example, in pop and rock music, the main melody part is repeated in the same way without major changes. Pop and rock music typically follow a similar fashion or pattern, such as the ABAB format, where A stands for bus and B stands for refrain. The main theme (refrain) part is most often done, followed by buses, bridges, etc. However, jazz music is typically performed on the fly by musicians, so most of the parts are changed and also create various problems in determining the main melody part. Usually, since jazz music does not have a refresh, the main part of jazz music is a bus.

In essence, an embodiment of the present invention provides a relatively simple and characteristic means by which a representation of music information comprising a characteristic relative difference value represents, indexes, and / or retrieves music information. It comes from realisation. It has also been found that these relative difference values provide a relatively uncomplicated structural representation for unstructured monolithic musical raw digital data.

In the above, the method, system, and computer program material which provide the summation of digital audio raw data are disclosed. Only some embodiments are described. However, it will be appreciated by those skilled in the art that various changes and / or modifications may be made without departing from the scope of the present invention.

Claims (30)

In the method of summarizing digital audio data, Analyzing the audio data to identify a representation of the audio data having at least one calculated feature characteristic of the audio data; Based on the expression, classifying the audio data into a predetermined category selected from two or more categories; And Generating an acoustic signal representing a thermalization of the digital audio data, And wherein said thermosizing depends on said selected predetermined category. The method of claim 1, Said analyzing further comprises segmenting audio data into segment frames, and overlapping said frames. The method of claim 2, The classifying further comprises classifying the frames into a predetermined category by collecting training data from each frame, and determining classification parameters by using a training calculation. The method according to any one of claims 1 to 3, Wherein the calculated feature comprises perceptual and subjective features related to music content. The method of claim 3, The training calculation comprises a statistical learning algorithm, wherein the statistical learning algorithm is a Hidden Markov Model, a Neural Network, or a Support Vector Machine. The method according to any one of claims 1 to 5, And the type of acoustic signal is music. The method according to any one of claims 1 to 6, The acoustic signal type is vocal music or pure music. The method according to any one of claims 1 to 7, Wherein the calculated feature is an amplitude envelope, a power spectrum, or a mel-frequency septal coefficient (MFCC). The method according to any one of claims 1 to 8, Wherein the summerization is generated for heuristic rules and clustered results related to pure or vocal music. The method according to any one of claims 1 to 9, Wherein the calculated feature is associated with pure or vocal music content and is a linear prediction coefficient, zero crossing rate, or MFCC. In the apparatus for summarizing digital audio data, A feature extractor for receiving audio data and analyzing the audio data to identify a representation of the audio data having one or more calculated feature characteristics of the audio data; A classifier in communication with the feature extractor, classifying the audio data based on the representation received from the feature extractor into a predetermined category selected from two or more categories; And And a summerizer in communication with the classifier to generate an acoustic signal representing a thermalization of the digital audio data. And wherein said summerization depends on said category selected by said classifier. The method of claim 11, And a segmenter in communication with the feature extractor, receiving an audio file, segmenting the audio data into segment frames, and overlapping the frames with respect to the feature extractor. The method of claim 12, And further comprising a classification parameter generator in communication with the classifier, wherein the classifier classifies each of the frames into a predetermined category by collecting training data from each frame, and uses training calculations in the classification parameter generator. Thereby determining the classification parameters. The method according to any one of claims 11 to 13, The calculated feature comprises perceptual and subjective features related to music content. The method according to any one of claims 11 to 14, The training calculation includes a statistical learning algorithm, wherein the statistical learning algorithm is a Hidden Markov Model, a Neural Network, or a Support Vector Machine. The method according to any one of claims 11 to 15, Wherein the acoustic signal is music. The method according to any one of claims 11 to 16, Wherein the acoustic signal is vocal music or pure music. The method according to any one of claims 11 to 17, Wherein the calculated feature is an amplitude envelope, a power spectrum, or an MFCC. The method according to any one of claims 11 to 18, Wherein the summerizer generates a summerization on empirical rules and clustered results related to pure or vocal music. The method according to any one of claims 11 to 19, Wherein the calculated feature is associated with pure or vocal music content and is a linear prediction coefficient, zero crossing rate or mel-frequency. A computer program for summarizing said digital audio data comprising said medium having computer readable program code means embodied in a computer usable medium for deriving the summarizing of digital audio data. Computer readable program code means for analyzing the audio data and identifying a representation of the audio data having one or more calculated feature characteristics of the audio data; Computer readable program code for classifying the audio data into a predetermined category selected from two or more categories based on the expression; And Computer readable program code for generating an acoustic signal representing the thermalization of the digital audio data, And wherein said summerization is dependent on said selected category. The method of claim 21, And the analyzing further comprises segmenting the audio data into segment frames and overlapping the frames. The method of claim 22, The classifying comprises classifying the frames into a predetermined category by collecting training data from each frame and determining classification parameters by using a training calculation. The method according to any one of claims 21 to 23, wherein The calculated feature comprises perceptual and subjective features related to music content. The method according to any one of claims 21 to 24, The training calculation includes a statistical learning algorithm, wherein the statistical learning algorithm is a Hidden Markov Model, a Neural Network, or a Support Vector Machine. water. The method according to any one of claims 21 to 25, And the acoustic signal is music. The method according to any one of claims 21 to 26, And the acoustic signal type is vocal music or pure music. The method according to any one of claims 21 to 27, And said calculated feature is an amplitude envelope, a power spectrum, or an MFCC. The method according to any one of claims 21 to 28, wherein The summerization is generated for empirical rules and clustered results related to pure or vocal music. The method according to any one of claims 21 to 29, wherein Wherein the calculated feature is associated with pure or vocal music content and is a linear prediction coefficient, zero crossing rate or mel-frequency.