JP5034516B2 - Highlight scene detection device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To detect highlight scene without using reference information and without being greatly influenced by encoding distortion. <P>SOLUTION: A spectrum is detected by a spectrum calculation module 22 by dividing an audio data included in a multi-media content for each fixed section. Then, a cheering voice feature amount composed of a frequency value (peak frequency) and its power value, which become a maximum value and a local maximal value in a specified bandwidth in which a cheering voice feature appears in the detected spectrum, is detected by a cheering voice feature amount extracting module 23. Then, in a cheering voice section judgement module 24 and in preset time length, the section in which the state of the detected cheering voice feature amount being higher than a threshold exists for a time length more than a judgement threshold rate is judged as a cheering section, and a display data for expressing the judgement result of the cheering voice section is created and output by an output control module 25. <P>COPYRIGHT: (C)2008,JPO&amp;INPIT

Description

  The present invention relates to an apparatus for detecting a highlight scene from content of a live program such as a concert or a sports program.

  In a recorder that aims to record and reproduce multimedia contents such as broadcast programs, a viewing mode is required in which only the highlight scene is viewed in a short time. In particular, in sports programs, scenes that are highlights such as scoring scenes for the entire content are short in time, and thus there is a high need for short-time viewing.

  As a solution, for example, in a home recorder, a technique has been developed for detecting a section where cheers are greatly raised as a highlight scene (hereinafter referred to as a cheer section) when a video signal is recorded or after recording. By adopting such a technique, the user can view only the highlight scene in the entire content of the sports program in a short time.

As a technique for detecting a cheering section, a feature of a cheering section is prepared in advance as reference information, a similarity with the reference information is calculated for each input signal, and a section in which the similarity is larger than a threshold is calculated. A technique for detecting a cheering section is known (see, for example, Patent Document 1). Reference information includes information using the spectrum itself and information using statistical information of a plurality of spectra. In Patent Document 1, a similarity calculation method using vector quantization is used as a technique for detecting a cheering section, which corresponds to a method using statistical information.
Japanese Patent No. 3475317

  However, in the method of preparing the reference information as described above and detecting the cheering section by calculating the similarity using the reference information, the detection accuracy depends on the reference information, and as a result, the cheering section There is a problem that the detection performance becomes unstable. For example, if the recording environment of the audio signal used to create the reference information and the input signal to be detected are different, the expected detection performance may not be obtained. In addition, when the encoded audio signal is decoded and the decoded audio signal is used as an input signal to be detected, the detection performance is significantly deteriorated because the reference information includes distortion due to encoding. There is a possibility. This is particularly noticeable in the case of encoding with a low bit rate.

  The present invention has been made paying attention to the above circumstances, and the object of the present invention is to enable detection of a highlight scene without using reference information and without being greatly affected by distortion caused by encoding. An object of the present invention is to provide a highlight scene detection apparatus with high accuracy.

  In order to achieve the above object, one aspect of the present invention receives content data including an audio signal, divides the audio signal included in the received content data into predetermined intervals, and detects the spectrum for each of these intervals. To do. And, from the spectrum in the preset band among the detected spectrum, the frequency that takes the maximum value and the maximum value is set as the peak frequency, and the cheering feature amount including the peak frequency and the power value of the peak frequency is detected. The section where the detected cheering feature amount is higher than the determination threshold is equal to or more than the determination threshold rate with respect to the predetermined determination time length is determined as the cheering section. .

  In general, there is a stable frequency peak in the time direction in a specific frequency band in a section including a cheer as an index of a highlight scene of an audio signal. This frequency peak is an essential feature of cheers, and since it exists in a relatively low frequency range, it is less susceptible to the effects of the recording environment and the encoding distortion on the audio signal. Therefore, as described above, the detection target band is first identified among the frequency bands of the input audio signal, and the frequency peak representing the cheering feature amount and its power value are detected in this specific frequency band, and this detection information is obtained. Based on the determination of the cheering section, it is possible to obtain relatively stable and high detection performance with respect to differences in recording environment and distortion due to audio encoding.

  That is, according to the present invention, it is possible to detect a highlight scene without using reference information and without being greatly affected by the distortion caused by encoding, thereby providing a highlight scene detection device with high detection accuracy. it can.

  Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a configuration of a highlight scene detection apparatus according to the first embodiment of the present invention.
The highlight scene detection apparatus 1A of this embodiment is connected to various broadcast recording / playback apparatuses such as a video recorder, a video camera, a personal computer with a television recording / playback function, and a portable terminal with a television recording / playback function, or Used in a state of being built in the recording / playback apparatus.

  The highlight scene detection apparatus 1A includes a control unit 2A composed of, for example, a central control unit (CPU). A storage unit 6 and an interface group are connected to the control unit 2A via a bus 7. The interface group includes an operation information input interface (operation information input I / F) 3, a content input interface (content input I / F) 4, and an output interface (output I / F) 5.

The operation information input I / F 3 is connected to the key switch group and detects the operation of the key switch group by the user. The content input I / F 4 receives multimedia content data output from a broadcast program recording / playback apparatus (not shown), and includes a recording medium I / F and a line input terminal. In addition, an A / D converter that converts the received analog signal into a digital signal is provided in case the multimedia content is an analog signal. The output I / F 5 outputs information representing a cheering section in the content detected by the control unit 2A to a content reproduction device or the like.
When the highlight scene detection device 1A is incorporated in a broadcast program recording / playback device, the content input I / F 4 has a function of receiving data read from the multimedia content recording unit in the device. .

  The control unit 2A includes an input control module 21, a spectrum calculation module 22, a cheer feature amount extraction module 23, a cheer section determination module 24, and an output control module 25 as control functions according to the present invention. Each of these modules 21 to 25 is realized by causing the CPU to execute an application program.

The input control module 21 receives an operation input signal from the operation information input I / F 3 and determines its type. For example, a signal for requesting detection of a highlight scene, a signal for selecting and specifying a detection mode, and a signal for selecting and specifying an output mode are determined.
Further, the input control module 21 takes in the data of multimedia contents via the content input I / F 4. Then, audio data is extracted from the captured multimedia content data, and the extracted audio data is divided at regular intervals and temporarily stored in the storage unit 6 as audio frame data. In the case where the extracted audio data is compressed data encoded by an audio encoding method such as AAC (Adaptive Audio Coding), the compressed data is decoded and then divided into fixed time frames to obtain audio frame data. .

  Furthermore, the input control module 21 down-samples the audio frame data to 8 kHz and converts the audio frame data into a general-purpose PCM (Pulse Code Modulation) signal when passing the audio frame data to the spectrum calculation module 22. Here, the reason why the sampling frequency is a relatively low value of 8 kHz is that the frequency band used in the subsequent processing is sufficient from 0 to 2 kHz, and about 4 kHz even when the bit rate is low by audio encoding such as AAC. The frequency components up to are less likely to lose information due to compression, and the sampling frequency before downsampling is often an integer multiple of 8 kHz, so the downsampling process is relatively simple. . Therefore, in some cases, the downsampling frequency may be 6 kHz or 12 kHz, and a value of 8 kHz is not essential.

  The spectrum calculation module 22 calculates the logarithmic power spectrum of the audio frame data passed from the input control module 21. The spectrum calculation method includes a method based on Fourier transform such as DFT (Discrete Fourier Transform) and FFT (Fast Fourier Transform), a method based on linear prediction analysis such as LPC (Linear Predictive Coding) spectrum, a bandpass filter and power. A computational signal processing based method is used.

  The cheering feature quantity extraction module 23 detects, from the spectrum calculated by the spectrum calculating module 22, a frequency value (peak frequency) that takes a maximum value and a maximum value within a specific band in which cheering features appear, and its power value. . Then, the detected peak frequency and its power value are passed to the cheering section determination module 24 as information representing the cheering feature amount.

  Based on the information indicating the cheering feature amount passed from the cheering feature amount extraction module 23, the cheering section determination module 24 indicates that the state that is higher than the cheering feature amount threshold value in the preset time length is the time length. On the other hand, the section which exists more than the judgment threshold rate is detected. Then, the detected section is determined as a cheer section.

  The output control module 25 generates output data in which the section determined as the cheering section by the cheering section determination module 24 is represented by a time bar, and outputs the generated output data to the output I / F 5. The form of the time bar is determined in accordance with an output mode designated in advance via the operation information input I / F 3.

  The storage unit 6 stores the audio content data input via the content input I / F 4 and is calculated when each module of the control unit 2A executes a series of processes for determining a cheering section. It is used to temporarily store information representing cheering feature values and information representing determination results of cheering sections.

Next, the operation of the highlight scene detection apparatus 1A configured as described above will be described.
In this case, audio content data of sports program multimedia content is taken from an external broadcast program recording device, a cheer section is detected from the audio content data, and information representing the result is output to the broadcast program recording device. Will be described as an example.

  In the highlight scene detection device 1A, first, the detection mode of the cheering section and the output mode of the detection result are set as follows. That is, when the user performs the selection operation of the detection mode and the output mode with an input device (not shown), these mode selection designation signals are taken into the input control module 21 of the control unit 2A via the operation information input I / F3. The input control module 21 is identified and stored. A specific example of the method for selecting and specifying the detection mode and the output mode will be described later.

  When the setting process of the detection mode and the output mode is completed, the highlight scene detection device 1A shifts to the cheering section detection mode, and first performs a process of calculating a spectrum of audio data and a process of extracting cheering feature values. Run like so. FIG. 2 is a flowchart showing the processing procedure and processing contents of the control unit 2A.

  That is, the control unit 2A monitors the input of the highlight scene detection request in step S21. In this state, it is assumed that the user performs a highlight scene detection request operation on the input device. Then, the control unit 2A detects the requested operation by the input control module 21 via the operation information input I / F3. Subsequently, the control unit 2A monitors the input control module 21 through the content input I / F 40 for the input of multimedia content data from a broadcast program recording / playback apparatus (not shown).

  In this state, when the multimedia content data sent from the broadcast program recording / playback apparatus is received by the content input I / F 40, the control unit 2A causes the input control module 21 to perform multimedia via the content input I / F4. Import content data. Then, audio data is extracted from the captured multimedia content data, and the extracted audio data is divided into fixed time intervals to obtain audio frame data, which is further downsampled to 8 kHz and passed to the spectrum calculation module 22.

  The spectrum calculation module 22 captures the downsampled audio data for each frame in step S22. In step S23, a logarithmic power spectrum is calculated by a method based on Fourier transform or a method based on linear prediction analysis. It should be noted that no matter what calculation method is used, the frequency resolution is preferably at least about 30 Hz. Thus, by calculating the spectrum with a frequency resolution of 30 Hz or higher, it is possible to improve the extraction accuracy of the cheer feature quantity in the cheer feature quantity extraction module 23 in the subsequent stage.

Subsequently, the control unit 2A executes the extraction process of the cheering feature value by the cheering feature value extraction module 23 as follows.
In general, cheers are a collection of a very large number of screams, and therefore the frequency characteristics of cheers can be thought of as smoothed by superimposing the frequency characteristics of individual screams. In the case of males, the first formant frequency of the vowel parts “a” and “o” that last for a long time, such as “War” and “War”, which are peculiar to screams, is 600 to 800 Hz, and the second formant frequency is around 1000 Hz. Since it exists, the frequency peak appears around 600 to 1000 Hz. In addition, the screaming voice tends to increase the pitch frequency and the sound pressure if the degree of excitement is strong, and to decrease the pitch frequency and the sound pressure if the degree of excitement is weak. For this reason, the higher the peak frequency in the range of 600 to 1000 Hz and the higher the sound pressure, the stronger the degree of excitement. Conversely, the lower the peak frequency and the lower the sound pressure, the weaker the degree of excitement. Therefore, in the spectrum of cheers, which is a collection of screams, there is a peak frequency in the range of 600 to 1000 Hz.

  Therefore, the cheering feature quantity extraction module 23 first detects, in step S24, a frequency value having a maximum value in a specific band range of 600 to 1000 Hz where cheering features appear from the spectrum calculated by the spectrum calculation module 22. . Subsequently, in step S25, the frequency having the maximum value in the specific band of 600 to 1000 Hz is detected. Then, based on the detection result of the maximum value and the detection result of the maximum value, the presence / absence of a frequency value having a maximum value and a maximum value is determined in step S26, and when a frequency value satisfying the condition is detected. Let this frequency value be the peak frequency.

  For example, as shown in FIG. 4A, when a frequency value having a maximum value and a maximum value in the range of 600 to 1000 Hz can be detected, this frequency value becomes the peak frequency. On the other hand, as shown in FIG. 4B, if the maximum value is not the maximum value even if the maximum value is detected, it is determined that there is no peak frequency (= 0). A series of processing from step S24 to step S26 is processing for detecting a peak frequency, and step S24 and step S25 may be in any order.

  When the peak frequency is detected, the cheering feature amount extraction module 23 proceeds to step S27, where the power value at the detected peak frequency is detected. Then, information representing the cheering feature amount represented by the detected power value and the peak frequency value is generated, and this information is stored in the storage unit 6 in step S28.

  For example, the weighted sum of the peak frequency PeakFreq and its power value PeakPow is obtained as shown in the following equation 1 and this is used as the cheering feature amount Feat, or the threshold value with the power value PeakPow as shown in the following [equation 2]. In the case of ThPowA or more, a bonus term Bns is added to the value of the peak frequency PeakFreq and this is used as a cheering feature amount Feat. Further, without using the power value PeakPow of the peak frequency PeakFreq as a feature amount as it is, for example, as shown in [Equation 3], when the power value PeakPow is lower than a certain threshold value ThPowB, the peak frequency is in the range of 600 to 1000 Hz. Even if it exists, it may be determined that it is not a peak due to cheers, and the cheer feature value may be set to zero. Further, as shown in [Equation 4], the threshold ThPowV may be increased as the peak frequency PeakFreq is higher. Α and β represent the weights of the weighted sum of the peak frequency PeakFreq and its power value PeakPow, respectively, and γ represents a coefficient for setting the threshold ThPowV to a threshold corresponding to the peak frequency PeakFreq.

  The cheering feature value generation method described above uses the characteristic that the peak frequency is higher and the power is larger as the degree of excitement in cheering is stronger. Not only the generation method shown in [Equation 4] but also various modifications such as a combination thereof are possible.

  Thus, when the extraction process of the cheering feature amount is completed for one frame of the audio data, the control unit 2A determines whether or not there is a next frame of the audio data in step S29. If there is a next frame, the process returns to step S22, and the spectrum calculation and cheering feature amount extraction processes in steps S22 to S28 are repeated.

  On the other hand, when the above-described extraction process of the cheering feature amount is completed for all the frames of the audio data, the control unit 2A next executes the cheering section determination process and the determination result output process as follows. FIG. 3 is a flowchart showing the processing procedure and processing contents.

  That is, the cheering section determination module 24 of the control unit 2A first sets a threshold value among the cheering feature amounts extracted by the cheering feature amount extraction module 23 for each determination time length L set in advance in step S31. The total time length exceeding is calculated, and the ratio of the calculated total time length to the determination time length is calculated. Then, the calculated ratio is compared with the determination threshold value in step S32, and when the ratio exceeds the determination threshold value, it is determined in step S33 that the determination time L is a cheering section. On the other hand, when the ratio is equal to or less than the determination threshold value, it is determined in step S34 that the determination time L is a non-cheering section. Then, the above determination result is stored in the storage unit 6.

  For example, it is assumed that a cheering feature amount as shown in FIG. In this case, the cheering section determination module 24 calculates a total time length Σl (= l1 + l2 + l3) of times l1, l2, and l3 of cheering feature values exceeding the threshold value. Then, the ratio of the calculated total time length Σl (= l1 + l2 + l3) with respect to the determination time L is calculated, and the value of the calculated ratio is compared with the determination threshold rate as in the following equation.

  Then, if the ratio of the calculated total time length Σl (= l1 + l2 + l3) to the determination time L is larger than the determination threshold rate, that is, if the expression of [Equation 5] is satisfied, this interval is determined as a cheering interval. For example, when the determination threshold rate is 0.7, in the example of FIG. 5A, Σl / L exceeds the determination threshold rate = 0.7, so this interval is determined to be a cheering interval. The On the other hand, in the example of FIG. 5B, Σl / L is not determined as a cheering section because the determination threshold rate is 0.7 or less.

  Here, the determination time length L is a value that controls the length of the cheering section to be detected. The longer the value L, the shorter the cheering section can be excluded from the detection target, but it is too long. Only a very long cheering section will be detected. In addition, the shorter the cheering interval, the shorter the cheering interval can be detected. However, if it is too short, the cheering feature threshold that suddenly exceeds the cheering feature value due to analysis errors, etc. It will be. Therefore, it is appropriate to set the determination time length L to about several seconds in advance. In addition, the value of the determination time length L can be set to a value desired by the user in response to the user's setting request input at the input device.

  The determination threshold rate is a value for controlling the probability of cheering determined as a cheering section. Increasing this value improves detection accuracy but increases the possibility of detection omission. On the other hand, the lower the determination threshold rate, the lower the detection accuracy, but the lower the possibility of detection omission. For this reason, it is desirable to set the determination threshold rate to an appropriate value of about 0.7 to 0.9 in advance. As with the determination time length L, this determination threshold is also set to an arbitrary value in the range of greater than 0.0 and less than or equal to 1.0 according to the user's setting request input by the input device. It is possible.

  If the determination threshold rate is used for the determination of the cheering section, a stable result can be obtained even when a sudden loss occurs in the cheering section due to an analysis error or the like. However, it is not always necessary to use the threshold rate, and there are other methods that bring about the same effect, such as taking a moving average of cheering feature values and using median filtering. That is, the threshold determination rate is not essential, and various improvements or changes can be made within a range that does not depart from this spirit.

  The detection threshold value of the cheering feature value is a value for controlling the detection level of the cheering magnitude (swelling), and the higher the value, the higher the cheering degree can be detected. On the other hand, if the detection threshold value of the cheering feature amount is low, even a cheering that is not so high will be detected. Therefore, it is desirable that the detection threshold value of the cheering feature amount is set to an appropriate value in advance. However, as with the determination time length L and the determination threshold rate, a user setting request input at the input device is required. Depending on, it may be set to an arbitrary value larger than 0.

  As described above, the cheering section determination module 24 can arbitrarily control the determination result of the cheering section by selectively setting a plurality of determination conditions, so that cheering detection can be performed according to more detailed needs. Is possible. However, in order for the user to appropriately control these determination conditions, experience or troublesome operations are required.

  Therefore, a plurality of highlight detection modes may be prepared in advance, and when the user selects an arbitrary mode from these modes, the determination condition may be variably set to an appropriate value accordingly. For example, as shown in FIG. 6, three highlight detection modes are prepared, and a user selects one of these modes and inputs a condition value, thereby determining according to the condition value. Make sure the condition is set.

  In FIG. 6, the first detection mode from the top is for detecting up to the upper X position according to the degree of highlight swell. In the second detection mode from the top, detection is performed from the top in accordance with the degree of the climax of the highlight so that the total time of the detected cheering section is X minutes. In the third detection mode from the top, detection is performed from the top in accordance with the degree of highlight so that the total time of the detected section is X% with respect to the time length of the entire content. By preparing such a highlight detection mode in advance, the user does not need to directly input the value of the determination condition used by the cheering section determination module 24, so that it is always appropriate with a simple operation regardless of experience. It is possible to set various determination conditions.

  Specifically, the detection threshold value of the cheering feature amount is first set to about 3 seconds, which is a short determination time length, and to about 0.7, which is a small determination threshold value so that a large number of cheer sections are detected. Is set to a lower value and a cheering section is detected. Next, the degree of highlight is calculated as a cheer score for these cheer sections. The degree of highlight is an index for judging how successful the highlight is. For this reason, the cheering score becomes larger as the cheering lasts for a longer time, the cheering feature value is larger, and there are fewer omissions in the cheering section.

  As an example of a cheer score, the area of a figure drawn by cheer feature values as shown in FIG. Thereby, a cheering score can be calculated by a relatively simple method. The cheering section determination module 24 selects cheering sections according to the highlight detection mode desired by the user in descending order of the cheering scores, and outputs the selected cheering sections. In this way, by taking a form in which a cheering section is detected in such a setting that a large number of cheering sections are once detected, and a cheering section is selected and output according to requirements desired by the user, the user can Even if the desired condition is changed, it is not necessary to perform the cheering section detection process again. For this reason, the processing amount required for detection of a cheering section can be significantly reduced.

  In addition, when the number of cheering sections is so small that the user's desired condition is not satisfied, for example, the total of the lengths of the cheering sections detected is 60 minutes even though the user designates the top 60 minutes as detection. If not, it is possible to cope with the problem by adjusting the determination time length L, the determination threshold rate, and the detection threshold value of the cheering feature amount so that more cheering sections are detected. In such a case, if the determination time length L and the determination threshold rate are lowered too much, there is a high possibility that a section that is not cheering will be detected for the reason described above, so the detection threshold value of the cheer feature amount is lowered. Is most effective.

  As described above, when a plurality of highlight detection modes are prepared in advance as a method for detecting a cheering section, and a user selects and designates a desired mode from these detection modes, determination is made according to the designated mode. By automatically adjusting the detection threshold of time length L, judgment threshold rate, and cheering feature value, the user can detect highlight scenes under the conditions desired by the user without troublesome operations. It becomes.

  When the determination result of the cheering section is obtained as described above, the control unit 2A subsequently executes the output process of the cheering section determination result by the output control module 25. That is, the output control module 25 first determines a preset output form in step S36. Then, output data is generated in one of steps S37 to S40 according to the determined output form, and the generated output data is output from the output I / F 5 to a recording / reproducing apparatus (not shown) or the like in step S41. .

For example, the output form includes a first form for displaying the position of the highlight scene, a second form for compressing and displaying only the highlight scene, a third form for displaying the highlight scene by color, and a highlight form. There is a fourth form in which light scenes are ranked and displayed.
Among these, first, when the first form is selected by the user, the output control module 25 performs the time positions T1 to Tn of the highlight scene in the content as shown in FIG. Is generated, and display data of this time bar is output from the output I / F 5 in step S41.

  Next, when the second form is selected by the user, the output control module 25 skips the sections other than the highlight scene, for example, as shown in FIG. Are generated, and display data of the time bar is output from the output I / F 5 in step S41.

  If the third form is selected, the output control module 25 represents the time position of the highlight scene in the content, for example, as shown in FIG. A time bar is generated by color-coding according to the cheer score, and the display data of the generated time bar is output from the output I / F 5 in step S41.

  When the fourth form is selected, the output control module 25 indicates the time position of the highlight scene in the content as shown in FIG. 8D, for example, as shown in FIG. A time bar that is displayed with the information indicating the cheer score and the ranking is generated, and the display data of the generated time bar is output from the output I / F 5 in step S41.

  In this way, by generating and outputting a time bar representing the determination result of the cheering section according to the output form selected and specified by the user, the user can find out what time position the detected highlight scene exists and which With such a length, it becomes possible to confirm how much the climax is in the form that he desires. For this reason, very useful information can be obtained when viewing in a short time. In addition, when editing multimedia contents, information that is very useful for editing called a highlight scene can be obtained, so that editing work can be performed efficiently and in a short time.

  As a content reproduction control method using the output time bar display data, the following can be considered. That is, a time bar indicating the position of the highlight scene of the multimedia content is supplied to the recording / playback apparatus that is playing back the multimedia content and displayed on the display. When the user selects a highlight scene displayed on the time bar in the recording / playback apparatus, only the highlight scene is played back. Further, an automatic skip mode is set in the recording / playback apparatus, and only highlight scenes are sequentially played back by skipping scenes other than the highlight scene according to the time bar.

  The time bar information is used for content distribution control by a content distribution server provided on the Internet, and is used for recording content on a recording medium, in addition to being used for content reproduction control in a recording / reproducing apparatus. It can be used for control to selectively record only the section corresponding to the highlight scene. Furthermore, information representing the attribute of the highlight scene section may be displayed as text data.

  As described above, in the first embodiment, the audio data included in the multimedia content input from the recording / playback apparatus is divided by the input control module 21 for each predetermined section, and the spectrum calculation module 22 for each section. Detect the spectrum. Next, the cheering feature amount extraction module 23 makes a cheering feature comprising a frequency value (peak frequency) having a maximum value and a maximum value within a specific band in which cheering features appear in the detected spectrum and its power value. Detect the amount. Then, in the cheering section determination module 24, a section where a state where the detected cheering feature amount is higher than a threshold value for a preset time length is greater than or equal to the determination threshold rate with respect to the time length is defined as a cheering section. The output control module 25 generates display data representing the determination result of the cheering section and outputs the display data to the recording / playback apparatus.

  Therefore, since the reference information is not used for detecting the cheering section, it is possible to always detect the cheering section with stable performance without being affected by the difference in the recording environment. Moreover, the peak of the spectrum existing in the 600 to 1000 Hz band peculiar to cheers is that the higher the cheer excitement, the higher the frequency and the greater the power, and the weaker the cheer excitement, the lower the frequency and the lower the power. A characteristic amount for determining a cheering section is detected using the characteristic. For this reason, it becomes possible to determine a cheering section stably and with high accuracy even with respect to encoding distortion of an audio signal.

  In the present embodiment, a plurality of highlight detection modes are prepared and displayed in advance, and when the user selects and designates a desired mode from these detection modes, a determination time is determined according to the designated detection mode. The detection threshold of the length L, the judgment threshold rate, and the cheering feature amount is automatically adjusted. Therefore, the highlight scene can be detected under the conditions desired by the user without performing a troublesome input setting operation.

  Further, a time bar representing the determination result of the cheering section is generated and output according to the output form selected and designated by the user. Therefore, the user can confirm at which time position in the content the detected highlight scene exists, and at what length and how much it rises in the form he desires. For this reason, very useful information can be obtained when viewing in a short time. In addition, when editing multimedia content, information that is very useful for editing called a highlight scene can be obtained, so that editing work can be performed efficiently and in a short time.

(Second Embodiment)
FIG. 9 is a block diagram showing a configuration of a highlight scene detection apparatus 1B according to the second embodiment of the present invention. In the figure, the same parts as those in FIG.
In the control unit 2B, in addition to the input control module 21, the spectrum calculation module 22, the cheer feature amount extraction module 23, the cheer segment determination module 24, and the output control module 25 described in the first embodiment, the cheer segment information normalization is performed. A module 26 and a cheering pattern similarity calculation module 27 are newly provided.

The cheering section information normalization module 26 normalizes the cheering section determination result obtained by the cheering section determination module 24 by dividing the cheering section into a plurality of subsections and using the threshold value of the cheering feature amount. The determination result of the cheering section is patterned.
The cheering pattern similarity calculation module 27 calculates the similarity between the cheering section patterned by the normalization process and a reference cheering pattern prepared in advance, and the cheering section is divided into a plurality of cheering sections according to the calculated similarity. Classify into cheer patterns.

Next, cheering section normalization processing and pattern classification processing operations performed by the apparatus configured as described above will be described.
First, normalization processing of the cheering section is performed as follows. FIG. 10 is a flowchart showing the processing procedure and processing contents. In step S101, the cheer section information normalization module 26 of the control unit 2B first calculates an average value and a standard deviation of each cheer feature value of the cheer section detected by the cheer section determination module 24. Next, a threshold value of the cheering feature value is calculated from the calculated average value and standard deviation, and the cheering section to be normalized is divided into three parts: a first half part, an intermediate part, and a second half part. First, in step S102, it is determined whether or not a cheering feature amount equal to or greater than the threshold value exists in the first half. As a result of this determination, if there is a cheering feature value greater than or equal to the threshold value, it is then determined in step S103 whether or not there is a cheering feature value greater than or equal to the threshold value in the latter half. If the result of this determination is that there is a cheering feature amount greater than or equal to the threshold value, it is determined in step S104 whether or not there is a cheering feature amount greater than or equal to the threshold value in the intermediate portion.

  As a result of the above determinations, if there is a cheering feature amount greater than or equal to the threshold value in any of the first half, the second half, and the middle of the cheering section, the cheering section information normalization module 26 performs the above in 6 in step S105. The normalization pattern of the cheering section is determined as pattern A. If there is a cheering feature amount greater than or equal to the threshold value in the first half portion and the second half portion of the cheering interval, but no cheering feature amount greater than or equal to the threshold value exists in the middle portion, the cheering interval in step S106. Is determined as a pattern B. Further, if there is a cheering feature amount greater than or equal to the threshold value in the first half part and no cheering feature value greater than or equal to the threshold value in the second half part, the normalization pattern of the cheering section is determined as pattern D in step S107. To do.

  On the other hand, if it is determined in step S102 that there is no cheering feature amount greater than or equal to the threshold value in the first half portion, the cheer section information normalization module 26 determines whether or not a cheer feature amount greater than or equal to the threshold value exists in the second half portion. In step S108, if it exists, the normalization pattern of the cheering section is determined as pattern C in step S109. On the other hand, if it is determined in step S108 that there is no cheering feature quantity greater than or equal to the threshold value, the cheering section information normalization module 26 determines whether or not there is a cheering feature quantity greater than or equal to the threshold value in the intermediate portion. Is determined in step S110. If it exists, the normalization pattern of the cheering section is determined as the pattern E in step S111, and if it does not exist, the normalization pattern of the cheering section is determined as the pattern A in step S112.

  11A to 11E schematically show the respective normalization patterns A to E. The horizontal axis indicates time, and the vertical axis indicates cheering feature amounts. The pattern A is a pattern having a flat cheering feature amount, and corresponds to, for example, a cheering without any inflection, such as no particular excitement. Pattern B is a pattern that once decreases and increases from a state where the cheering feature amount is high. Taking baseball as an example, if the runner is in the scoring area, the hit will be small immediately after the ambiguous hit, but it corresponds to the case where the ball falls and hits and gets big when the score enters . Pattern C is a pattern in which the cheering feature amount increases from a low state. For example, in soccer, it corresponds to a case where a cheer is greatly increased by deciding a shot from a state where the scoring chance is increased. The pattern D is a pattern that decreases from a state where the cheering feature amount is high. For example, in soccer, this corresponds to a case where a cheerful lingering sound persists after a sudden cheering such as a long shot. The pattern E is a pattern in which the cheering feature value is changed from a low state to a high state and becomes low. For example, in baseball, when a batter hits a home run, the cheering swells immediately after the batting and heals a little, and the cheering rises again at the moment the ball enters the spectator seat.

  Thus, in the section detected as the cheering section, when there is no portion where the degree of cheering is large as in pattern A, when in the first half and the second half as in pattern B, only in the second half as in pattern C , It is possible to grasp to some extent what kind of cheer the detected cheers are by classifying them when they are only in the first half like pattern D and when they are near the center like pattern E. .

  The threshold value used for the normalization process is calculated as follows. FIG. 12 is a diagram for explaining the calculation method. That is, the threshold value is calculated by adding the average value of the cheering feature amount in the cheering section and the standard deviation. Thus, by calculating a threshold value using a statistical quantity such as an average value or a standard deviation, it becomes possible to deal with variations in cheering feature quantities.

  Further, as shown in [Equation 6] below, the sum of the standard deviation multiplied by a positive real number α and the average value is used as a threshold value, and the threshold value is variably controlled by controlling this coefficient α. It may be.

  In this way, as the coefficient α is increased, the cheering feature amount exceeding the threshold value is decreased and the pattern A is almost classified into the pattern A. Therefore, the coefficient α is in the range of 1.0 to 2.0 based on experience. It is appropriate to set to.

  Further, the standard deviation may be calculated using information of a plurality of cheering sections instead of calculating for each cheering section. By doing so, not only the variation in the cheering section but also the influence of the variation outside the cheering section, that is, for each cheering section, can be considered. In the example of FIG. 12, since the cheering feature amount equal to or greater than the threshold exists in the first half portion of the cheering section, it is determined in step S102 that the cheering feature amount equal to or greater than the threshold exists. Since the above cheer feature amount does not exist, it is determined in step S103 that it does not exist. For this reason, the normalization pattern of the cheering section is determined as the pattern D.

  As described above, when normalizing the cheering interval, time information can be normalized by dividing the cheering interval into a first half part, an intermediate part, and a second half part and quantizing. In addition, since the binarization is performed depending on whether or not the cheering feature amount is equal to or greater than the threshold value, the cheering feature amount can also be normalized. However, since the cheering feature amount is the information itself indicating the excitement of the cheering, it is binarized. It is preferable to keep the value without. For example, in each of the first half part, the middle part, and the second half part, the cheering feature amount is less than the threshold value, and the cheering feature amount is not less than the threshold value. It is conceivable to maintain the average value of the voices and the average of the cheering feature values of only that part. By holding the value of the cheering feature amount in this way, more detailed classification is possible in the cheering pattern classification module 27 in the subsequent stage.

  An example of the expression of the normalized cheer information obtained by the cheer section information normalization module 26 is shown in FIG. In the figure, 131 indicates the number of cheer sections from the start position of the content, and 132 indicates the start time and end time of the cheer sections. Reference numeral 133 indicates which cheer pattern is normalized by the cheer section information normalization module 26, and 134 indicates a normalized value of the cheer feature amount. By setting it as such an output form, since the information regarding the time, pattern, and the magnitude | size (degree) of a cheer regarding a cheering section is known, in a cheering pattern classification | category module 27 of a latter stage, a more detailed classification | category is attained.

  As described above, in the cheer section information normalization module 26, the section determined as the cheer section by the cheer section determination module 24 is normalized to a cheer pattern and a cheer feature value prepared in advance. As a result, the determination information of the cheering section becomes information that is robust with respect to the length of the cheering section and the variation in the cheering feature amount, so that the cheering section can be classified with higher accuracy.

  In the above description, the case where five cheer patterns are prepared has been described as an example. However, the number of cheer patterns is not limited to this, and any number may be set as long as it is plural. However, if the number is too large, the robustness against variations due to normalization is lost, and if the number is too small, the number of cheer patterns is reduced. Therefore, it is important to set the number appropriately.

  Next, in the cheering pattern classification module 27, processing for classifying the cheering section determination information normalized by the cheering section information normalizing module 26 is performed. The classification method is basically the cheer pattern itself normalized by the cheer section information normalizing module 26. For example, when the cheering pattern is the pattern C shown in FIG. 11, the cheering feature amount is increased from a low state as described above. This corresponds to a case where the value rises greatly.

  On the other hand, when trying to classify in more detail, the same cheering pattern may be given superiority or inferiority using the value of the cheering feature value. For example, as shown in FIG. 14, in the same cheering pattern C, the value of the cheering feature value at the time of rising is set so that (b) is larger than (a). In other words, in this example, (b) indicates that the cheering was greater, so that superiority or inferiority can be given. In this way, by comparing the cheering features of those who are prospering between the same cheering patterns, or by comparing with the linear sum of those who are prospering and those who are not prominent, the same cheering patterns Can be better or worse.

  In this way, the following effects can be obtained. That is, in the first embodiment, when restrictions such as the total playback time are imposed according to various requests of the user input at the input device, it depends on the length of the cheer section and the size of the feature amount. The superiority and inferiority were given by cheering scores, and cheering sections were selected in order from the top based on this superiority. However, in the second embodiment, it is possible to rank all cheering sections by prioritizing cheering patterns in advance and giving superiority or inferiority within the cheering patterns.

  The second embodiment can be modified in various ways as follows. For example, as shown in FIG. 15, a highlight pattern selection menu may be displayed to allow the user to select a desired cheer pattern. Further, as shown in FIG. 8 (c), the cheer patterns are classified and displayed by color and shape, so that the user can instantly display the cheer patterns as positional information and the structure of cheer sections in the entire content. It becomes possible to easily grasp, and the user can realize various viewing modes.

  As described above, in the second embodiment, the cheering pattern classification module 27 can classify cheering sections using the normalized cheering section determination information obtained by the cheering section information normalization module 26. Thus, the user can not only know the position of the cheering section, but can easily grasp the cheering instantly. Accordingly, it is possible to expect the effects that it is possible to grasp the flow of the entire sports content, and that the information when editing the content recorded with events such as athletic meet becomes more useful information.

  In the second embodiment, the cheering section information normalization module 26 and cheering pattern classification module 27 perform normalization processing and pattern classification processing to classify cheering sections into a plurality of cheering patterns using information expressed by patterns and values. The case where the superiority or inferiority is given is explained as an example. However, the present invention is not limited to this, and various methods can be adopted as the pattern classification processing method for the cheering section without departing from the gist of the present invention. For example, a target scene such as a goal scene or a home run scene may be statistically learned in advance, and normalization of a cheering section and calculation of similarity may be performed based on a stochastic distance with respect to the target pattern. Specifically, statistical learning is performed in advance for each scene such as a goal scene or home run scene using various statistical models such as vector quantization, clustering, and GMM. And when classifying a cheering section, the cheering pattern is classified as belonging to a pattern having the highest probability that the determination information of each cheering section can occur stochastically with respect to the learned information. If the probability at this time is the magnitude of the similarity, priority can be given.

(Third embodiment)
In the second embodiment, the cheering segment classification processing is performed by normalizing and classifying the temporal transition pattern of the cheering feature value. However, with the above processing method, it is difficult to correctly specify a systematic support scene such as a scene where a group sings and supports a song with the same pitch or a support scene using a musical instrument. Organized support often has little to do with the excitement of sports content. Rather, organized support collapses when the audience enthusiastically swells. For this reason, the detection accuracy of the cheering section and the improvement of the classification accuracy of the cheering section can be expected by not actively setting the organized support section as the cheering section or by distinguishing from the cheering section as the organized cheering section.

  In the third embodiment, when the cheering feature value is extracted, the cheering feature value is extracted and it is determined whether or not it is systematic support. For example, in the cheer feature amount extraction module 23 shown in FIG. 1, from the spectrum calculated by the spectrum calculation module 22, a frequency (peak frequency) having a maximum value and a maximum value within a partial band where cheer features appear. While detecting the power value, the power value of the frequency around the peak frequency is calculated. Then, when the difference between the power value of the peak frequency and the power value of the peripheral frequency is larger than the threshold value, an organization support flag indicating that the system is in an organized support state is output.

  Hereinafter, the determination processing operation of the cheering section and the organized support section according to the third embodiment will be described. FIG. 16 is a flowchart showing the processing procedure and processing contents. In addition, about the structure of an apparatus, FIG. 1 is used and demonstrated.

  The cheering feature amount extraction module 23 first detects a peak frequency from the spectrum calculated by the spectrum calculation module 22 in step S121. Next, in step S122, it is determined whether or not the detected peak frequency is 0. If it is determined to be 0, the section is determined as a non-cheering section in step S123.

  In contrast, it is assumed that the peak frequency is determined to be other than 0 in step S122. In this case, the cheering feature amount extraction module 23 calculates the bandwidth of the cheering feature amount including the peak frequency in step S124. For example, as shown in FIG. 17, the frequency width at a position 3 dB lower than the peak frequency power value is calculated as the bandwidth. Then, it is determined in step S125 whether or not the calculated bandwidth is smaller than a preset threshold value. As a result of this determination, if it is determined that the bandwidth is equal to or greater than the threshold, the section is determined to be a normal cheer section in step S126. On the other hand, when it is determined in step S125 that the bandwidth is smaller than the threshold value, the cheering feature amount extraction module 23 determines that the section is organized support in step S127.

  Here, the reason why it can be regarded as systematic support when the bandwidth of the peak frequency is smaller than the threshold will be described below. In other words, organized support means that many people sing the same song or play the same instrument. Accordingly, the pitch frequencies that are different in the case of normal cheers are almost the same when the same song is sung. For this reason, the harmonics structure due to the pitch, that is, the peaks and valleys of the frequency, remarkably appear on the spectrum as shown in FIGS. 17 (a) and 17 (b), for example. Similarly, when a cheering instrument is ringing, the peaks and valleys of the frequency appear prominently in the spectrum due to the harmonic structure of the fundamental frequency of the instrument. For this reason, in the case of organized support, the frequency peak becomes sharp and the bandwidth becomes narrow.

  As described above, according to the third embodiment, the cheering section and the organized support section can be divided according to the bandwidth of the peak frequency. For this reason, the accuracy of the detection of the cheering section and the classification of the cheering section is improved, and various viewing modes satisfying the user can be realized. Further, the organized support section itself is useful information for grasping the content. As a result, the user can view and edit the content in a shorter time.

(Other embodiments)
In each of the above-described embodiments, the case where the highlight scene detection device is provided separately from the recording / playback device has been described as an example. However, the highlight scene detection device may be provided in the recording / playback device. Further, the highlight scene detection device may be added to or incorporated in the imaging device or the content distribution server in addition to the recording / playback device.
In addition, the configuration of the highlight scene detection apparatus and the processing procedure and processing contents by the control unit can be variously modified and implemented without departing from the gist of the present invention.

  In short, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in each embodiment. Furthermore, you may combine suitably the component covering different embodiment.

It is a block diagram which shows the structure of the highlight scene detection apparatus which concerns on 1st Embodiment of this invention. It is a flowchart which shows the procedure of the spectrum calculation process by the control unit of the highlight scene detection apparatus shown in FIG. It is a flowchart which shows the procedure and process content of a cheering area determination process by the control unit of the highlight scene detection apparatus shown in FIG. 1, and its output process. It is a figure for demonstrating the peak frequency detection processing operation by the control unit of the highlight scene detection apparatus shown in FIG. It is a figure for demonstrating the cheering zone determination processing operation by the control unit of the highlight scene detection apparatus shown in FIG. It is a figure which shows an example of the highlight detection mode input menu displayed by the control unit of the highlight scene detection apparatus shown in FIG. It is a figure which shows an example of a cheer score obtained by the control unit of the highlight scene detection apparatus shown in FIG. It is a figure which shows several examples of the time bar produced | generated by the control unit of the highlight scene detection apparatus shown in FIG. It is a block diagram which shows the structure of the highlight scene detection apparatus concerning the 2nd Embodiment of this invention. It is a flowchart which shows the procedure and process content of a cheering area normalization process by the control unit of the highlight scene detection apparatus shown in FIG. It is the figure which showed typically the normalization pattern obtained by the control unit of the highlight scene detection apparatus shown in FIG. It is a figure for demonstrating the calculation method used in the control unit of the highlight scene detection apparatus shown in FIG. It is a figure which shows an example of the expression of the normalized cheering information obtained by the control unit of the highlight scene detection apparatus shown in FIG. It is a figure which shows the example in the case of giving the superiority or inferiority to a cheering pattern using the value of a cheering feature-value in the control unit of the highlight scene detection apparatus shown in FIG. It is a figure which shows the example of a display of the highlight pattern selection menu which is a modification of 2nd Embodiment. It is a flowchart which shows the procedure and process content of the determination process of a cheering area and a systematic support area in 3rd Embodiment of this invention. It is a figure for demonstrating the bandwidth calculation method of the cheering feature-value containing the peak frequency in 3rd Embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1A, 1B ... Highlight scene detection apparatus, 2A, 2B ... Control unit, 3 ... Operation information input interface (operation information input I / F), 4 ... Content input interface (content input I / F), 5 ... Output interface ( (Output I / F), 6 ... storage unit, 7 ... bus, 21 ... input control module, 22 ... spectrum calculation module, 23 ... cheer feature quantity extraction module, 24 ... cheer section determination module, 25 ... output control module, 26 ... Cheer section information normalization module, 27 .. cheer section determination module.

Claims (4)

Means for receiving content data including an audio signal;
A spectrum detector for dividing the audio signal included in the received content data into predetermined intervals and detecting the spectrum for each of the intervals;
Based on the detected peak frequency and the power value , a peak frequency indicating a maximum value and a maximum value and a power value of the peak frequency are detected from a spectrum within a preset band among the detected spectra. Feature amount detecting means for detecting a cheer feature amount from a predetermined relational expression ;
Determining means for determining a section where the detected cheering feature amount is higher than a determination threshold value as a cheering section for a predetermined determination time length with respect to a predetermined determination time length; Highlight scene detection device characterized by the above. Means for receiving the designation information when the user designates and inputs the output form of the determination result of the cheering section;
Means for adjusting at least one of the determination time length, a determination threshold value and a determination threshold rate according to the received designation information;
The determination result of the cheering section obtained by the determination unit based on the adjusted determination time length, determination threshold value or determination threshold rate is edited and output in the output form represented by the designation information. The highlight scene detection apparatus according to claim 1, further comprising: means. Normalization means for normalizing the determination result of the cheer section obtained by the determination means with information representing the length of cheer and the size of the cheer;
The highlight scene detection device according to claim 1, further comprising a classifying unit that classifies the determination result of the normalized cheer section into a plurality of preset cheer patterns. The feature amount detection means includes:
From the spectrum in a band that has been set in advance among the detected spectrum by the spectrum detecting means, and means for detecting a power value of the peak frequency and the peak frequency,
From the spectrum of the preset in-band, and means for detecting a power value of a frequency near the peak frequency,
The difference between the power value of the detected peak frequency and the power value of the surrounding frequency is calculated, and the cheering feature amount is detected when the difference of the calculated power value is larger than a preset value. 4. The highlight scene detection device according to claim 1, further comprising means for outputting an organization support flag indicating that the cheering section is in an organized support state.

Images