JP4985134B2 - Scene classification device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a scene classifier capable of enhancing precision of section detection. <P>SOLUTION: This scene classifier is provided with a time peak detecting part 5 for detecting a time peak in a spectrogram, and a frequency directional feature amount extracting part 6 for extracting a feature amount indicated in the time leak, and uses also the feature amount concerned in the time peak, as an index for the section detection. The scene classifier is provided further with a mutual feature amount extracting part 7, and classifies scenes, using also a new feature amount defined by an interaction between a frequency peak and the time peak, as an index. <P>COPYRIGHT: (C)2009,JPO&amp;INPIT

Description

  The present invention relates to a technique for classifying multimedia content scenes (scenes) mainly accompanied by music scenes based on acoustic signals.

  In order to view a recorded broadcast program, there is a demand for a form in which only a desired scene is viewed in a short time. For example, a scene in which an artist sings as a highlight scene of a music program is detected, and only that scene is viewed. In order to realize such a function, a technique for classifying scenes of multimedia contents is provided.

  Japanese Patent Application Laid-Open No. 2004-228561 has a technique for detecting a scene having a large feature amount as a music section by using a ratio of [Lch power−Rch power] to [Lch power + Rch power] of a two-channel sound signal as a feature amount representing stereo feeling. Disclosed. According to this technique, the amount of calculation for detecting a music section can be relatively reduced.

In the technique of Non-Patent Document 1, spectrum peaks between adjacent analysis frames are connected if the frequency value and the normalized distance of the logarithmic power value in the two-dimensional space are close to each other. Find out. This continuous spectrum peak is defined as a sinusoidal segment, and a feature amount associated therewith is classified using a dictionary that has been statistically learned in advance to detect a speech segment or a music segment.
JP 2006-301134 A "Sound category detection in acoustic signals with mixed voice, musical instrument sound and singing voice using temporal characteristics of Sinusoidal Segment" (Waseda Univ.) Toru Taniguchi et al. Acoustical Society of Japan, September 2005

  In the technique of Patent Document 1, there is a problem that when a sound scene recorded by a plurality of channels has a stereo component, a scene that does not belong to a music section such as laughter or applause is detected as a music section. Also, for vocal main songs (a cappella, rap tone, etc.), the stereo component of the music section is small, and the accuracy of section detection is reduced.

In Non-patent Document 1, because sinusoidal segments are spectral peak sequences that are continuous in the time direction, the music is not remarkable in its temporal persistence, such as a lap tone or a song with a fast tempo. On the other hand, it is conceivable that the accuracy of section detection decreases. This causes a problem that a music detection error occurs.
The present invention has been made in view of the above circumstances, and an object thereof is to provide a scene classification device with improved accuracy of section detection.

  In order to achieve the above object, according to one aspect of the present invention, spectrum calculation for dividing multimedia content including an acoustic signal into a plurality of temporally continuous sections and calculating a spectrum of the acoustic signal for each of the sections A frequency peak detection unit that detects a frequency peak that is a local maximum point in the frequency direction in the spectrum, a time direction feature amount extraction unit that extracts a first feature amount indicating a feature of the frequency peak, and the spectrum as a time A time peak detection unit that detects a time peak that is a local maximum point in a time direction in spectrograms that are continuously arranged, a frequency direction feature amount extraction unit that extracts a second feature amount indicating the feature of the time peak, and Based on the acoustic features of the section indicated by the first feature quantity and the second feature quantity, the plurality of sections are defined as a first music section. Scene classification device is provided which is characterized by comprising an acoustic classifier unit for classifying the second music segment.

  By taking such means, it is possible to use not only the frequency peak but also the feature amount shown in the spectrogram, that is, the time peak, when classifying the sections. By using the time peak, it is possible to quantitatively evaluate an acoustic feature that cannot be captured by the frequency peak. Therefore, it is possible to further increase the accuracy of section detection.

  According to the present invention, it is possible to provide a scene classification device that improves the accuracy of section detection.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Here, an apparatus for classifying scenes included in multimedia contents based on acoustic signals and detecting a music scene will be described.
Multimedia contents are roughly classified into those including both video signals and audio signals and those consisting of audio signals. Examples of the former include a stream to be broadcast on television, a recording of this, or a recording by a moving image recording device such as a home video. Examples of the latter include a stream broadcast on radio, a recording of this, or a recording recorded by a recording device such as an IC recorder.

FIG. 1 is a functional block diagram showing an embodiment of a scene classification apparatus according to the present invention. In FIG. 1, multimedia content is input to a content input unit 1 and an acoustic signal is extracted.
That is, the content input unit 1 is an interface for inputting multimedia content accompanied by at least an acoustic signal to the apparatus. For example, if the medium is a DVD (Digital Versatile Disk), it has a DVD reader, and if the medium is recorded on an HD or the like, it has a data transmission bus. In short, the content input unit 1 has an appropriate configuration according to the form of input multimedia content.

  The content input unit 1 extracts an acoustic signal from content provided in various forms. In particular, if the acoustic signal is analog, digital conversion is performed and digital data is output. In this case, it is convenient to keep the sampling frequency constant regardless of the content form because it is not necessary to be aware of the format of the acoustic signal in the subsequent processing. The acoustic signal extracted by the content input unit 1 is given to the spectrum calculation unit 2.

  The spectrum calculation unit 2 divides the acoustic signal into frames having a certain arbitrary time length, and calculates a spectrum for each frame (section or segment). That is, the spectrum calculation unit 2 calculates a spectrum from the input acoustic signal by a technique such as FFT (Fast Fourier Transform) or LPC analysis. This makes it possible to analyze the frequency information of the acoustic signal. It should be noted that the processing load in the subsequent stage can be reduced by cutting high frequency information of about 4 kHz or higher, which is not so sensitive acoustically. Conversely, if an auditory characteristic is taken into consideration, the linear spectrum may be converted into a mel scale spectrum. In particular, in order to detect a music section, it is preferable to convert it to a Cent logarithmic frequency scale corresponding to a musical scale. The spectrum calculated by the spectrum calculation unit 2 is given to the frequency peak detection unit 3 and the time peak detection unit 5.

  The frequency peak detector 3 detects the peak of the calculated spectrum in the frequency direction for each section (segment). That is, the frequency peak detector 3 detects a peak in the frequency direction at a certain time of the acoustic signal as a frequency peak. The frequency peak means a maximum point in the frequency direction that appears in a spectrum plotted in a graph of sound power versus frequency.

  As preprocessing, fine components of the spectrum may be removed by a smoothing filter such as a moving average filter or a median filter. By removing the fine component of the spectrum, it is possible to detect a global frequency peak instead of a local frequency peak. By doing in this way, it is possible to extract a feature amount in consideration of human auditory characteristics.

  The detection data in the frequency peak detection unit 3 is given to the time direction feature quantity extraction unit 4. The time direction feature amount extraction unit 4 mainly connects a plurality of frequency peaks that are continuous in the time direction. Then, a statistic such as the average or variance of the number, length, continuity, direction, each peak value statistic, each frequency value statistic, etc. of this time continuous peak train is extracted as a feature quantity. This feature amount is referred to as a time continuous feature amount.

  That is, the time-direction feature quantity extraction unit 4 counts the number of frequency peaks at each sampling time, and first extracts statistics such as the average and variance. Next, if the frequency peaks are mainly continuous in the time direction, these are concatenated and the number of time continuous peak sequences, as well as the time length, continuity, direction, peak value statistic, frequency, Statistics such as average and variance of value statistics are extracted as time-continuous feature quantities. The time-continuous feature amount is given to the acoustic classification unit 8 and is used for detecting a voice section, a music section, and the like from the multimedia content. The detected scene is output from the output unit 9.

  The number of time continuous peak trains is the number of time continuous peak trains in a section (segment). The time length of the time continuous peak train is the length of the time continuous peak train in the time direction. Continuity refers to the degree of no omission in the time continuous peak train. The direction is an amount indicating how much the time continuous peak train is inclined in the time direction. Each peak value statistic indicates the average or variance of the power of each frequency peak in the time continuous peak sequence, and each frequency value statistic indicates the average or variance of the frequency value of each frequency peak in the time continuous peak sequence. Indicates.

  It is considered that the feature amount extraction related to the time continuity of the frequency peak is effective in detecting an acoustic pattern in which an auditory stimulus is sustained in the time direction. For example, the sound from a guitar or synthesizer has a frequency peak structure so that it has high continuity, and the average peak value is large. In addition, in a song with a slow tempo such as a ballad, the time length becomes long, and when there is vibrato (frequency fluctuation of the frequency), the frequency value dispersion becomes large. Furthermore, in the case of voices that are uttered with long vowels such as “Ah” and “Wow”, the direction is inclined and the frequency value variance becomes large. On the contrary, in the case of an acoustic signal such as steady noise that has little density in terms of frequency such as applause and background sound, the number of frequency peak values is small. As described above, the time-continuous feature amount is effective for capturing the characteristics of acoustic signals such as music and voice, but is not suitable for capturing acoustic signals such as percussion instrument accents and plosives. Moreover, since the time continuous feature amount is based on the premise that the frequency peak is continuous in the time direction, it may be very difficult to extract the feature depending on the music genre.

Incidentally, the scene classification apparatus of FIG. 1 includes a time peak detection unit 5, a frequency direction feature amount extraction unit 6, and a mutual feature amount extraction unit 7. Among these, the time peak detection part 5 arrange | positions the spectrum calculated in the spectrum calculation part 2 continuously in time, and produces | generates a spectrogram. And the peak in the time direction in a certain frequency of this spectrogram is detected. That is, the time peak detector 5 detects a time-direction peak at a certain frequency as a time peak. The time peak means a local maximum point appearing in a spectrogram plotted on a graph of sound power versus time.
As preprocessing, fine components in the time direction power train may be removed by a smoothing filter such as a moving average filter or a median filter. By removing the fine component of the time direction power train, a global time peak can be detected instead of a local time peak. By doing in this way, it is possible to extract a feature amount in consideration of human auditory characteristics. The detection data in the time peak detection unit 5 is given to the frequency direction feature amount extraction unit 6.

  The frequency direction feature amount extraction unit 6 connects the spectrograms that are mainly continuous in the frequency direction. Then, statistics such as the average and variance of the number, length, continuity, direction, each peak value statistic, each time value statistic, and the like of this frequency continuous peak train are extracted as feature quantities. This feature amount is referred to as a frequency continuous feature amount.

  That is, the frequency direction feature quantity extraction unit 6 counts the number of time peaks for each frequency, and first extracts statistics such as the average and variance. Next, if time peaks are mainly continuous in the frequency direction, they are connected, and the number of frequency continuous peak trains, as well as the bandwidth length, continuity, direction, peak value statistic, time value of frequency continuous peak trains A statistical quantity such as a statistical quantity such as an average or variance is extracted as a frequency continuous feature quantity.

  The number of frequency continuous peak trains indicates the number of frequency continuous peak trains in a section (segment). The band length of the frequency continuous peak sequence indicates the length in the frequency direction of the frequency continuous peak sequence, that is, the length of the band. The continuity indicates the degree of no missing in the frequency continuous peak train. The direction is an amount indicating how much the frequency continuous peak train is inclined in the frequency direction. Each peak value statistic indicates the average or variance of the power of each time peak in the frequency continuous peak sequence, and each time value statistic indicates the average or variance of the time value of each time peak in the frequency continuous peak sequence. Indicates.

  The feature amount extraction regarding the frequency continuity of the time peak is physically effective for capturing a power increase, that is, a sharp increase in volume and a change in volume. For example, in the case of singing or playing such as a lap tone, the number of frequency continuous peak rows increases, and when a percussion instrument such as a drum is played, the band length becomes long. Furthermore, in the case of an acoustic signal whose power increases primarily in a wide frequency band, such as a consonant / s / utterance, the band length is extremely long and continuity is high. As described above, the frequency continuous feature amount extracted by the frequency direction feature amount extraction unit 6 is effective as a feature amount for capturing features such as music and speech that cannot be captured by the time continuous feature amount. This continuous frequency feature quantity is given to the mutual feature quantity detection unit 7 and the acoustic classification unit 8 together with the continuous time feature quantity (from the time direction feature quantity extraction unit 4).

  The mutual feature quantity detection unit 7 extracts a mutual feature quantity defined by the degree of mutual influence between the time continuous feature quantity and the frequency continuous feature quantity. That is, the mutual feature quantity extraction unit 7 extracts a mutual feature quantity indicating a degree of mutual influence from the time continuous feature quantity continuous in the time direction and the frequency continuous feature quantity continuous in the frequency direction. That is, the mutual feature amount is an amount that is not established unless both the time continuous feature amount and the frequency continuous feature amount exist. This mutual feature quantity is given to the acoustic classification unit 8 together with the time continuous feature quantity and the frequency continuous feature quantity.

  The acoustic classification unit 8 classifies each frame by using at least one of the time continuous feature value, the frequency continuous feature value, and the mutual feature value. That is, the acoustic classification unit 8 classifies each section (segment) into a voice section or a music section, and the other from the time continuous feature quantity, the frequency continuous feature quantity, and the mutual feature quantity.

A simple method for classifying frames is applied to each feature quantity X with a weight W provided for each classification pattern, and a linear sum is calculated using the following equations (1) and (2). If the value exceeds the threshold value for each classification pattern, there is a method in which the target frame belongs to a specified classification pattern.

  For example, if the music has a slow tempo, the length of the continuous peak sequence will be long, so the weight of the classification pattern: “the length of the continuous peak sequence” in the “slow tempo song” is increased. Can be set.

  In addition, the weights may be optimized using learning data prepared in advance using a neural network, or statistics such as GMM (Gaussian mixture model), VQ (vector quantization), and SVM (support vector machine). The frames may be classified using a typical model. By using a statistical classification model, it is possible to effectively use features that do not contribute to classification by features alone or that are not clear how they relate to each classification pattern. It becomes.

  Furthermore, the sound classification unit 8 may determine the scene in addition to the frame classification. For example, in a section where a section (segment) classified as a music section appears frequently, the section is indexed as a music scene.

In this way, the user can recognize the delimited portion of the multimedia content as a meaningful scene, so that the multimedia content can be viewed for a short time or edited easily. can get. The results classified as described above are given to the output unit 9 and output in an appropriate form based on a request from the user.

  The output unit 9 outputs the output from the acoustic classification unit 8 in an appropriate form. For example, the result of classification in units of sections (segments) output from the sound classification unit 8 may be output as it is to a video output device such as a display, may be output as text data, or specified as electronic data. Or may be displayed and transmitted after being converted into a description language. If the output of the sound classification unit 8 is indexed information, the result may be output together with time information and indexed summary information by the various methods described above. Further, for example, when a music scene or the like is designated, only the corresponding scene may be output and reproduced by AV (Audio Visual). In this way, the user can view only the scene that the user wants to see.

  FIG. 2 is a diagram for explaining the operation of the frequency peak detection unit 3 and the time peak detection unit 5. FIG. 2A is a spectrogram in which spectra over a plurality of frames are continuously arranged in time, and this corresponds to one section (segment). However, this is for simplifying the explanation and is not necessarily limited to the two-dimensional data structure of frequency and time.

  If the graph of FIG. 2A is cut out by frequency (horizontal dotted line), a logarithmic power graph with respect to time (FIG. 2B) is obtained. The maximum point (* in the figure) of this graph is a time peak, which is the amount detected by the time peak detector 5. Further, when the graph of FIG. 2A is cut out by time (vertical dotted line), a logarithmic power graph with respect to frequency (FIG. 2C) is obtained. The maximum point (x in the figure) of this graph is the frequency peak, which is the amount detected by the frequency peak detector 3.

Existing technology used only frequency peaks. In this embodiment, a time peak is also used together, and a mutual feature quantity that can be newly defined by combining both is also used for classification of a section (segment). Next, the mutual feature amount will be described.
FIG. 3 is a diagram for explaining processing in the mutual feature quantity extraction unit 7. The mutual feature quantity extraction unit 7 extracts a mutual feature quantity that affects each other from the time continuous feature quantity and the frequency continuous feature quantity. As shown in FIG. 3, the mutual feature amount is defined at a portion where a time continuous peak sequence (horizontal line) and a frequency continuous peak sequence (vertical line) intersect, and the number (“◯” and “ □ "Total number of"), or how to interact.

  As shown by the “□” in the figure, the way in which the frequency continuous peak sequence is located at the end of the time continuous peak sequence is the same as the instrument with a sharp increase in power and the instrument with the frequency peak structure. Indicates that it sounded at the timing. This means that, for example, it can be determined that there is a high possibility that a drum and a guitar have been played at the same time, and such a segment is very likely to be a music segment. A music scene can be extracted using this fact.

  In this way, the mutual feature quantity extraction unit 7 extracts the mutual feature quantity, which is an index that the time continuous feature quantity and the frequency continuous feature quantity affect each other, so that only two types of feature quantities are extracted. It becomes possible to extract information that cannot be obtained.

  FIG. 4 is a diagram illustrating that a scene can be determined based on the section classification. As shown in FIG. 4, when sections (segments) classified as music sections are temporally continuous at a high rate, the sound classification unit 8 can index these sections as music scenes. That is, the sound classification unit 8 detects a scene including a music section over a predetermined number within a specified time as a music scene in the multimedia content. Next, points of the scene classification apparatus in FIG. 1 will be described from different viewpoints.

FIG. 5 is a diagram for explaining processing in the time direction feature quantity extraction unit 4. The frequency peak features include the number of frequency peaks, the power value of this frequency peak, the frequency value of this frequency peak, the number of time continuous peak trains, the length of this time continuous peak train, and the continuous of this time continuous peak train. , The direction of this time continuous peak train, the statistic of each peak value, and the statistic including the mean or variance of the statistic of each frequency value. This will be described below with reference to equations (3) and (4).

  FPK (t, n) = (FPKf (t, n), FPKp (t, n)) in the equations (3) and (4) is a vector indicating the n-th peak at a certain time t, and FPKf (t , n) represents the time value of the nth peak at a certain time t, and FPKp (t, n) represents the power value of the nth peak at a certain time t. NFPK (t, n) is a vector obtained by normalizing FPK (t, n) for each dimension. Then, normalization can absorb the difference in dimension between frequency and power.

  The time direction feature quantity extraction unit 4 calculates the first and second vectors when connecting the frequency peaks in the time direction. The first vector has NFPK (t, i), which is the i-th frequency peak at a certain time t, as an end point, and NFPK (t-1, j), which is the j-th frequency peak at a time t−1. Is a vector. That is, the first vector is stretched between the end point and the start point. The second vector is a vector having the kth frequency peak NFPK (t + 1, k) at time t + 1 as the end point and the above NFPK (t, i) as the start point. That is, the second vector is stretched between the end point and the start point.

  Then, the time direction feature amount extraction unit 4 connects both vectors according to the degree of similarity between the first vector and the second vector. The degree of similarity can be expressed by, for example, the small angle between the first vector and the second vector, or the size of the inner product of the first vector and the second vector. The time direction feature amount extraction unit 4 determines the similarity of both vectors by determining the threshold values of these amounts.

  In FIG. 5, the crosses are frequency peaks detected every time. For example, if the vector V11 obtained from P10 and P11 in FIG. 5 is the first vector and the vector V12 obtained from P11 and P12 is the second vector, the angle formed by these two vectors is small. Are concatenated. On the other hand, if the vector V21 obtained from P20 and P21 is the first vector and the vector V22 obtained from P21 and P22 is the second vector, the angles formed by these vectors are large, and the two are not connected.

  FIG. 6 is a diagram showing a comparison between the existing technique and the method of this embodiment in frequency peak connection processing. FIG. 6A shows an existing technique, in which the closest frequency peaks are connected in adjacent time frames. In this method, discontinuous connection occurs as can be seen at a glance, and as a result, the accuracy of the time continuous feature amount extracted by the time direction feature amount extraction unit 4 is lowered.

  On the other hand, the time direction feature quantity extraction unit 4 of this embodiment introduces the concept of a vector and connects the peaks in consideration of the direction between the peaks. As a result, as shown in FIG. 6B, discontinuous and subjectively inconsistent connections can be avoided, and the accuracy of time-continuous feature values can be improved. Thus, by considering the direction in the connection between peaks, it is possible to avoid the connection of vectors that should not be connected. An already connected vector may be used as the first vector. In this way, processing can be simplified.

FIG. 7 is a diagram for explaining the processing in the frequency direction feature amount extraction unit 6. The time peak features include the number of time peaks, the power value of this time peak, the frequency value of this time peak, the number of frequency continuous peak trains, the length of this frequency continuous peak train, the continuation of this frequency continuous peak train There is a statistic that includes the mean or variance of at least one of the sex, the direction of this frequency continuous peak sequence, the statistic of each peak value, and the statistic of each time value. This will be described below with reference to equations (5) and (6).

  In equations (5) and (6), TPK (f, n) = (TPKt (f, n), TPKp (f, n)) is a vector indicating the nth peak of a certain frequency f, and TPKt (f , n) represents the time value of the nth peak of a certain frequency f, and TPKp (f, n) represents the power value of the nth peak of a certain frequency f. NTPK (f, n) is a vector obtained by normalizing TPK (f, n) for each dimension. Then, normalization can absorb the difference in dimension between time and power.

  The frequency direction feature quantity extraction unit 6 calculates third and fourth vectors when connecting the time peaks in the frequency direction. The third vector has an NTPK (f, i) that is the i-th time peak at a certain frequency f as an end point, and an NTPK (f-1, j) that is the j-th time peak at a frequency f−1. Is a vector. That is, the third vector is stretched between the end point and the start point. The fourth vector is a vector having the kth time peak NTPK (f + 1, k) at the frequency f + 1 as an end point and the above NTPK (f, i) as a start point. That is, the fourth vector is stretched between the end point and the start point.

  Then, the frequency direction feature quantity extraction unit 6 connects both vectors according to the degree of similarity between the third vector and the fourth vector. The degree of similarity can be expressed by, for example, the small angle formed by the third vector and the fourth vector, or the size of the inner product of the third vector and the fourth vector. The frequency direction feature amount extraction unit 6 determines the similarity between both vectors by determining the threshold values of these amounts.

  In FIG. 7, * indicates a time peak detected for each frequency. For example, if the vector V31 formed by P30 and P31 in FIG. 7 is the third vector and the vector V32 formed by P31 and P32 is the fourth vector, the angle formed by these two vectors is small. Connected. Conversely, if the vector V41 formed by P40 and P41 is the third vector, and the vector V42 formed by P41 and P42 is the fourth vector, the angle formed by these two vectors is large and the two are not connected. .

  As described above, by considering the direction also in the time peak connection, it becomes possible to avoid the connection of the vectors that should not be connected, as in the time direction feature quantity extraction unit 4, and the extraction is performed as a result. It becomes possible to improve the accuracy of the continuous frequency feature quantity. An already connected vector may also be used for the third vector, and in this way, the processing can be simplified.

  As described above, this embodiment includes the time peak detection unit 5 that detects the time peak in the spectrogram and the frequency direction feature amount extraction unit 6 that extracts the feature amount indicated by the time peak, and features related to the time peak. The quantity is also used as an index for interval detection. Furthermore, a mutual feature amount extraction unit 7 is provided, and a new feature amount defined by the interaction between the frequency peak and the time peak is also used as an index to classify the scene.

  That is, in this embodiment, in the spectrogram, not only the characteristic amount (frequency peak) indicating that the characteristic state having higher power than the surrounding spectrogram is continuous in the time direction, but also that the above state is continuous in the frequency direction. The feature quantity (time peak) shown is also extracted and used for scene classification. This makes it possible to detect a music genre such as a spectrum peak that is difficult to show time continuity, such as a lap tone or a song with a fast tempo, which is difficult with existing technology, with high accuracy. In addition to music, it is possible to detect an arbitrary acoustic section such as speech, and further, it can be expected to improve accuracy in various classifications such as music type and voice gender determination in the detected section. .

  For these reasons, it is possible to provide a scene classification device that improves the accuracy of section detection. In addition, by applying this to devices including multimedia content players, various scene classifications such as music scenes and talk scenes, as well as the type of music and the sex of the person who is speaking, can be made with high accuracy. Can be realized. As a result, the user can watch only the scene he / she wants to watch for a short time, or can quickly find the scene he / she wants to edit by a simple operation.

  The present invention is not limited to the above embodiment. That is, the present invention is not limited to a stationary device such as a DVD recorder, but can also be applied to an image viewing device using a mobile communication terminal, so-called one-segment. In addition, according to the present invention, the accuracy of feature quantity extraction can be improved, so by using various classification models, not only music but also mechanical noise, male voice, female voice, animal voice, etc. It can be applied to various scene classifications.

  Furthermore, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment.

The functional block diagram which shows embodiment of the scene classification | category apparatus concerning this invention. The figure for demonstrating the effect | action in the frequency peak detection part 3 and the time peak detection part 5. FIG. The figure for demonstrating the process in the mutual feature-value extraction part. The figure which shows that a scene can be determined based on an area classification. The figure for demonstrating the process in the time direction feature-value extraction part. The figure which compares and shows the technique of this technique and the technique of this embodiment in the connection process of a frequency peak. The figure for demonstrating the process in the frequency direction feature-value extraction part.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Content input part, 2 ... Spectrum calculation part, 3 ... Frequency peak detection part, 4 ... Time direction feature-value extraction part, 5 ... Time peak detection part, 6 ... Frequency direction feature-value extraction part, 7 ... Mutual feature-value extraction Part, 8 ... acoustic classification part, 9 ... output part

Claims (5)

A spectrum calculation unit that divides multimedia content including an acoustic signal into a plurality of temporally continuous sections and calculates a spectrum of the acoustic signal for each of the sections;
A frequency peak detector that detects a frequency peak that is a maximum point in the frequency direction in the spectrum;
A time direction feature quantity extraction unit for extracting a first feature quantity indicating the feature of the frequency peak;
A time peak detection unit for detecting a time peak that is a local maximum point in a time direction in a spectrogram in which the spectrum is continuously arranged in time;
A frequency direction feature quantity extraction unit for extracting a second feature quantity indicating the feature of the time peak;
A third feature quantity extraction unit for extracting a third feature quantity indicating a degree of interaction between the first feature quantity and the second feature quantity;
Based on the acoustic features of the section indicated by the first feature quantity , the second feature quantity , and the third feature quantity , the plurality of sections are divided into a first music section and a second music section. An acoustic classification unit for classification ;
Including
The acoustic classification unit detects a scene including the first music section over a specified number of times within a specified time as a music scene in the multimedia content,
The time direction feature amount extraction unit includes the feature of a time continuous peak sequence obtained by connecting the frequency peaks in the time direction, and extracts the first feature amount.
The frequency direction feature quantity extraction unit includes the feature of a frequency continuous peak sequence obtained by connecting the time peaks in the frequency direction, and extracts the second feature quantity.
The scene classification device, wherein the third feature amount represents the number of intersections of the time-related continuous peak sequence and the frequency continuous peak sequence, or how to intersect . The first feature amount is
Number of frequency peaks, power value of this frequency peak, frequency value of this frequency peak, number of said time continuous peak train, length of this time continuous peak train, continuity of this time continuous peak train, this time continuous peak 2. The scene classification apparatus according to claim 1 , wherein the scene classification device includes an average or a variance of at least one of a column direction, a statistic of each peak value, and a statistic of each frequency value. The time direction feature amount extraction unit includes:
A first vector spanned between first and second frequency peaks adjacent in the time direction, and a first vector spanned between the first frequency peak and a third frequency peak adjacent to the first frequency peak in the time direction. if the similarity between the two-vector or specified threshold value, the scene classification apparatus according to claim 1, characterized in that coupling the first and second frequency peaks. The second feature amount is
Number of time peaks, power value of this time peak, frequency value of this time peak, number of said frequency continuous peak train, length of this frequency continuous peak train, continuity of this frequency continuous peak train, this frequency continuous peak 2. The scene classification apparatus according to claim 1 , wherein the scene classification apparatus includes a statistic including an average or a variance of at least one of a column direction, a statistic of each peak value, and a statistic of each time value. The frequency direction feature extraction unit
A third vector spanned between first and second time peaks adjacent in the frequency direction; and a third vector spanned between the first time peak and a third time peak adjacent to the first time peak in the frequency direction. 4 if the similarity between the vector more than specified threshold value, the scene classification apparatus according to claim 1, characterized in that coupling the first and second time peak.

Images