JP4416133B2 - Audio summary information extraction device and recording medium - Google Patents

Description

  The present invention extracts audio information and audio video summary information for extracting summary information (summary) for efficiently grasping the contents from uncompressed or compressed audio information or compressed audio video information. The present invention relates to an apparatus and a recording medium. Also, providing high-speed and efficient browsing of audio information or audio-video information by extracting summary information at high speed in the compressed data area from uncompressed or compressed audio information or compressed audio-video information The present invention relates to a playback apparatus for audio / video outline information.

  For automatic video summarization, for example, Tanaka, Wakimoto, Kanda “Development of Automatic Video Information Summarization and Browsing Technology by Scene Detection”, IEICE Technical Report, IE99-20 (1999) In this research presentation, after video scene change points are detected, a hierarchical structure is created and a video summary is automatically created by assigning priorities to each scene. . In particular, a high priority is given to the scene immediately after the turning point of the topic. In the hierarchical structure, priority is set so that the higher the higher the higher the hierarchy is.

  In addition, “Using content models to build audio-video summaries” by J.Saarela and B.Merialdo, SPIE Conference on Strorage and Retrieval for Image and Video Databases VII (1999) (second prior art) The creation of video summary information (summary) is regarded as an optimization problem with constraints. Constraints include minimum shot length, audio and video synchronization, video continuity, and audio and video redundancy. Then, a video content model (symbol description) is manually constructed to create a summary.

  R. Lienhart et al. “Video Abstracting”, Communications of ACM, Vol. 40, No. 12 (1997) (third prior art) aims to create summary information specialized for movie trailers. . The main creation procedures are splitting the video into shots, analysis of clips containing special events, clip selection, and clip aggregation. Special events include actor face recognition, conversation identification, character information extraction from titles, and events such as shootings and explosions.

  Extraction of summary information of audio video according to these conventional techniques has the following problems. First, in the first prior art, efficient summarization is provided by video hierarchical structuring. However, hierarchical structuring needs to be performed manually, and for long audio video information, summary information is extracted. It may take a lot of time to make a hierarchical structure as a pre-process necessary for the process. Further, since the priority is given depending on the hierarchy to which the scene belongs, the work is often required substantially in practice. In addition, it is possible to extend to audio video because it is a single video, but because it does not use the characteristics as audio, in some cases it does not use audio information containing important content, It is possible that adequate summary information is not available.

  In the second prior art, the video content model must be manually constructed, and a method of effectively using audio characteristics is not employed. At the same time, the classification of video segments requires advanced recognition technology that is difficult to realize on compressed data, so it is expected that the processing required for extracting summary information will increase. With regard to the third prior art as well, these high-level processes (entering the meaning of contents) are necessary.

  As described above, in the prior art, manual processing is often involved in the process from the input of the audio video information to the output of the summary information, and the audio information is not effectively used. Information creation is not considered. Also, no effective method has been studied so far for automatic summary information extraction from compressed or uncompressed audio alone.

  Also, from the structural aspect of the summary information, there is no guarantee that the summary information is uniformly extracted from the entire audio video information. Further, in the first and second prior arts, the summary information designated from outside is not guaranteed. A method of performing control to approach the length is not sufficiently employed.

  The present invention has been made in view of the above-described prior art, and an object thereof is an outline for efficiently grasping the contents from uncompressed or compressed audio information or compressed audio video information. In a device that extracts information (summary), it analyzes the spatio-temporal characteristics of audio and video in the compressed data area, evaluates them comprehensively as necessary, and determines the specified length of audio or audio-video information. SUMMARY OF THE INVENTION An object of the present invention is to provide an audio summary information, audio video summary information extraction device, playback device, and recording medium that enable uniform summary information to be extracted from the entire content uniformly and at high speed.

In order to achieve the above object, the present invention provides means for extracting subband data from input compressed audio content or means for converting input audio content into subband data; It means for calculating the sum of the sub-band energy weighted by, from the sum of the sub-band energy, means for calculating a sub-band energy summation at a certain unit time, time position where the sub-band energy sum is maximized at the unit time Means for determining the time position at which the sum of subband energy in the unit time is α times the maximum value (where 0 <α <1) as the summary information end time; Audio information for the start time and summary information end time interval is used as summary information. The first feature is that it has means for extracting data.

Further, the present invention provides a computer for extracting audio summary information, wherein at least one of means for extracting subband data from input compressed audio content and means for converting input audio content to subband data, A means for calculating a sum of subband energy weighted by bands from subband data, a means for calculating a subband energy sum in a unit time from the sum of subband energy, and a subband energy sum in the unit time being maximum Means for determining as a summary information start time, and determining as a summary information end time a time position at which the total subband energy in the unit time is α times the maximum value (where 0 <α <1); And the summary information start time and summary information end time. A second feature is that a computer-readable recording medium in which a program for extracting audio summary information for functioning as audio information for the section is extracted as summary information is provided.

  According to the present invention, it is possible to extract summary information for quickly and efficiently grasping the contents of uncompressed or compressed audio information or compressed audio video information. In addition, this extraction enables high-speed browsing of audio-video information.

  In addition, the length of the extracted summary information can be arbitrarily specified, and at the same time, the summary information is uniformly extracted from the audio video information, so that it is possible to efficiently grasp the entire content.

  Also, by describing the time information included in the summary information, it becomes possible to describe the feature as the summary information of the corresponding audio video information, and it can be applied to MPEG-7, which is the standardization of content description. Is possible.

  In addition, it is possible to provide reproduction of high-performance outline information such as converting the display speed of the extracted outline information.

  Hereinafter, the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an audio video summary information extraction apparatus according to the first embodiment of the present invention.

First, the compressed audio video information AV is input to the content analysis unit 1 that analyzes the temporal structure of the audio video information AV. The content analysis unit 1 has a configuration as shown in FIG. 2, and the compressed audio video information AV is first input to the shot division unit 11. The shot division unit 11 divides the video information into shot units S, then counts the number of shots for the entire content, and holds the total number of shots as NS. Next, the division number determination unit 12 uses the total shot number NS input from the shot division unit 11, the content length CL obtained from the audio video information AV, and the outline information length SL specified from the outside, as follows: The division number NP is determined by the equation (1).
NP = NS × SL / CL (1)
However, it is assumed that SL> CL / NS. For example, if the content length CL is 1 hour and the summary information length SL is 5 minutes, the total number of shots NS is 240, and the division number NP is 20. Then, the content is divided into NP equal parts in the content division 13. The equally divided sections SAV and the shots S belonging to each section are mapped.

In the following, processing is performed for each section SAV input by the divided section input 14.
First, the first shot Sn is input by the shot input 15. When the shot length SHLn of the input shot Sn is equal to or shorter than the length T specified in advance, the shot length evaluation unit 16 excludes it from the candidates for the summary information SUM and proceeds to input processing for the next shot Sn + 1. On the other hand, if the shot length SHLn of the shot Sn is longer than T, the corresponding shot Sn is set as a candidate for the outline information SUM and is sent to the representative frame extraction unit 17.

  The representative frame extraction unit 17 extracts the first frame SFSn or feature frame KFSn of the shot as the representative frame RFSn of the shot Sn. For example, the method described in Japanese Patent Application No. 2000-065259 can be used to extract the feature frame KFSn. Further, the representative frame feature value extraction unit 18 extracts the feature value CHFSn of the frame RFSn extracted by the representative frame extraction unit 17, and the shot feature value extraction unit 19 extracts the feature value CHSn of the shot Sn.

  One or both of the feature value CHFSn of the start frame SFSn or the feature frame KFSn as the representative frame RFSn for the shot Sn and the feature value CHSn of the shot Sn are sent to the representative shot evaluation unit 1A. As the feature value CHFSn of the first frame SFSn or the feature frame KFSn, for example, a descriptor defined in MPEG-7 (Moving Picture Experts Group phase 7) can be used.

  In the representative shot evaluation unit 1A, the feature value CHF of the representative frame RF regarding all shots already registered as the representative shot RS in the target divided section SAV and the shot Sn sent from the representative frame feature value extraction unit 18 The similarity is determined between the feature value CHFSn of the representative frame RFs. Here, if it is determined that the similarity between the feature value CHF and the feature value CHFSn of the input shot Sn is large, it is regarded as a repetitive shot, and the corresponding shot Sn is excluded from the candidates for the summary information SUM. Then, the process proceeds to input processing for the next shot Sn + 1. On the other hand, if it is determined that the similarity between the feature value CHF and the feature value CHFSn of the shot Sn is small, it is regarded as an independent shot, the corresponding shot Sn is set as a candidate for the summary information SUM, and the representative shot registration 1B After registering the shot at, it is sent to the audio level evaluation unit 2. It should be noted that the feature value related to the shot Sn registered as the representative shot is stored separately and used for evaluation of the representative shot thereafter.

  In the determination of the shot similarity, similarly, the feature value CHS of the representative shot related to the shot that has already been registered as the representative shot RS and the feature value CHSn of the shot Sn sent from the shot feature value extraction unit 19 are substituted. Or you may use together.

  In the audio level evaluation unit 2, regarding the shot Sn input from the content analysis unit 1, the subband data SDSn of the audio part of the shot Sn is extracted by the subband data extraction unit 21 of FIG. Then, after the silence analysis unit 22 calculates the feature value as silence, the silence determination unit 23 determines that all sections of the shot Sn, or Y% or more of the sections of the shot Sn are determined to be silent. The shot Sn to be excluded is excluded from the candidates for the summary information SUM, and the process proceeds to input processing for the next shot Sn + 1. On the other hand, in the determination by the silence determination unit 23, when the audio portion of the shot Sn is (100-Y)% or more of the section of the shot Sn, the shot Sn is sent to the low level sound analysis unit 24. As the silence analysis method in the silence analysis unit 22 and the silence determination method in the silence determination unit 23, for example, the method described in Japanese Patent Application No. 10-235543 can be used.

  Similarly, the low-level sound analysis unit 24 estimates the level LSn of the corresponding audio from the subband data SDSn extracted by the subband data extraction unit 21, and the audio having a level sufficiently lower than the specified level THLL is recorded in the shot Sn. If all the sections or Z% or more of the section of the shot Sn occupies, the corresponding shot Sn is excluded from the candidates for the summary information SUM, and the process proceeds to the input process for the next shot Sn + 1. On the other hand, when the audio part of the shot Sn is an audio in which (100-Z)% or more of the section of the shot Sn exceeds the audio level THLL in the low level sound analysis unit 24, the corresponding shot Sn is selected as the candidate of the summary information SUM. And sent to the high level sound analysis unit 26.

In the high level sound analysis unit 26, based on the subband data SDSn relating to the shot Sn obtained from the subband data extraction unit 21, an audio unit of a level THHL or higher in the shot Sn by the unit time density calculation unit 261 in FIG. Calculate the time density Dsn. At this time, when the audio information is encoded with MPEG audio, the unit time density Dsn per second can be obtained as follows, for example.
Dsn = (NAF _THHL / NAF) × AAL _THHL (2)

Here, NAF _THHL is the number of frames per second having a level higher than THHL, NAF is the number of audio frames per second, and AAL _THHL is the average level with respect to NAF _THHL .

  In addition, the subband energy sum calculation unit 262 calculates the sum SBE of the subband energy weighted by the subband, and based on this, the unit time subband energy calculation unit 263 calculates the subband energy sum SBsn per unit time. . The unit time subband energy SBsn per second can be obtained, for example, by the following equation (3).

  Here, αk is a weight for subband k, and sbk is the energy of subband k in a frame.

  Next, when the unit time density determination unit 264 has a unit time density Dsn that exceeds a certain threshold THD, the corresponding shot Sn shifts to the unit time subband energy determination unit 265. When the unit time subband energy SBsn exceeding the threshold THSB is present, the unit time subband energy determination unit 265 determines the shot Sn including the corresponding audio as a candidate for summary information, exits the high-level sound analysis routine, and moves The process proceeds to the activity evaluation unit 3.

  On the other hand, if the unit time density DSn of the audio level THHL or higher is less than the threshold value THD, or if the unit time subband energy SBSn is less than the threshold value THSB, the shot Sn including the corresponding audio is excluded from the summary information candidates. Then, the process proceeds to input processing for the next shot Sn + 1.

  In the configuration of FIG. 3, all of the silence analysis unit 22, the low level sound analysis unit 24, and the high level sound analysis unit 26 are not necessarily required, and at least one may be provided.

  In the motion activity evaluation unit 3, regarding the shot Sn input from the audio level evaluation unit 2, the motion vector extraction unit 31 in FIG. 5 extracts the motion vector information MV of all frames belonging to the shot Sn. Then, the motion activity calculation unit 32 calculates the motion activity MASn of the entire shot Sn using the motion vector information MV, and uses the motion activity information calculation unit 32 to calculate a unit time (for example, 1 second, 1 Motion activity MA in a frame etc.). The processing in the motion activity calculation unit 32 and the unit time motion activity 33 can be expressed as follows, for example, for MPEG-encoded video. Here we assume motion activity per second.

  Where ASMV is the sum of absolute values of motion vectors with a size greater than or equal to X, NMB is the number of macroblocks with motion vectors whose size is greater than or equal to X in the frame, and NPSn is the number of predicted encoded frames in a shot. , NVF is the number of video frames per second.

  In the movement activity determination unit 34, the obtained movement activity MA is compared with a specified threshold THMA, and when MA exceeds THMA, the corresponding shot Sn is excluded from the candidates for the summary information SUM, and the next shot Move on to Sn + 1 input processing. On the other hand, if the motion activity MA in a unit time of the shot Sn does not exceed the threshold value THMA in the motion activity determination unit 34, the corresponding shot Sn is set as a candidate for the summary information SUM.

  Through the above processing, in the section SAV in which the audio video information AV is divided at equal intervals, the shot S selected as the candidate for the summary information SUM is input to the summary information registration unit 4, and the information of the shot S is stored in the shot memory of FIG. 41 is stored. Next, the determination unit 42 determines whether or not the processing of all shots in the section SAV has been completed. When this determination is negative, processing of the next shot is performed.

  On the other hand, when the processing of all shots in the section SAV is completed, the total SSL of the shot lengths related to all the shots S in the section SAV is obtained by the summary information length determination unit 43 in the average summary information length MSL (= SL / NP) is judged to be close enough. If the total shot length SSL in the section SAV and the average summary information length MSL are considered sufficiently close, the summary information extraction process in the section SAV is terminated, and time information and the like are registered in the summary information registration 44 . Then, the process proceeds to input processing for the next section SAVn + 1. At this time, various feature values related to the representative shot registered in the representative shot registration 1B are cleared. On the other hand, if SSL and MSL are not considered to be close, the threshold value changing unit 47 changes some or all threshold values used for determination as candidates for the summary information SUM, and has been extracted so far For the candidate of the summary information SUM, the processing from the shot length evaluation unit 16 to the motion activity evaluation unit 3 is recursively repeated until SSL and MSL are sufficiently close.

  These processes are performed for all the sections SAV, finally, the summary information SUM of the audio video information AV is obtained, and the summary information SUM is registered. At this time, if the description of the summary information is designated, the process proceeds to the summary information description unit 5, and if not designated, the process is terminated.

  As shown in FIG. 7, the summary information description unit 5 describes at least the time information of all the shots S extracted as the summary information SUM by the summary information description unit 51, and the summary information output unit 52. To output as a summary information description file. As a format to be described, for example, a format defined by MPEG-7 can be used. When the description is finished for all the sections SAV, all the processes are finished. Further, all the shots S extracted as the summary information SUM can be combined and saved as a separate file. At this time, it can be saved as an audio video file, or audio and video can be saved separately.

  Next, the function of the audio / video outline information extracting apparatus of the above-described embodiment can be realized by software (program). The software can be recorded on a recording medium such as an optical disk, a floppy (registered trademark) disk, or a hard disk.

  FIG. 8 shows an example of a program recorded on the recording medium 100. The recording medium 100 includes a compressed audio / video information content analysis function 111, an audio level evaluation function 112, a motion activity evaluation function 113, A summary information registration function 114 and a summary information description function 115 are recorded. The movement activity evaluation function 113 may be omitted.

  The content analysis function 111 evaluates the function of dividing video information into shots, the function of dividing input content into equal intervals according to a certain standard, and the length of shots included in the equal intervals. It can be composed of at least one of a function and a function for determining repetitive shots. The audio level evaluation function 112 can be constituted by a function for determining silence, a function for determining low level sound, and a function for determining high level sound.

   The motion activity evaluation function 113 has a function of extracting motion vector data of a frame belonging to the shot, a function of calculating motion activity from the extracted motion vector data, and a function of calculating motion activity in unit time. And a function for determining summary information candidates using the motion activity in the unit time.

  In addition, the extracted summary information can be combined as audio video or audio and video individually, and the combined summary information can be recorded on the recording medium 100 as a separate file.

  The recording medium 100 includes a transmission medium that temporarily records and holds data, such as a network.

  FIG. 9 is a block diagram showing a configuration of an audio summary information extraction apparatus according to the second embodiment of the present invention.

  First, when the compressed audio information CA is input, the subband data extraction unit A1 extracts the subband data SD. The extracted subband data SD is sent to the high level sound evaluation unit A3. The operation of the subband data extraction unit A1 is the same as that of the subband data extraction unit 51 in the silence / low level sound evaluation unit 5 shown in the first embodiment. On the other hand, when uncompressed audio information UA is input, the subband analysis unit A2 performs subband analysis on the input audio, and the analyzed subband data SD is similarly sent to the high-level sound evaluation unit A3. .

  The high level sound evaluation unit A3 has the same functions as the subband energy sum calculation unit 262 and the unit time subband energy sum calculation unit 263 included in the high level sound analysis unit 26 shown in the first embodiment. The subband energy total calculation unit A31 and the unit time subband energy total calculation unit A32 in FIG. 10 respectively calculate the subband energy total SBE and the subband energy total SB in unit time from the input subband data SD. Is done.

Next, in the summary information start time determination unit A33, the time position at which the unit time subband energy sum SB is maximized is determined as the summary information start time T_start. Also, the summary information end time determination unit A34 determines the time position at which the unit time subband energy sum SB is α times the maximum value (0 <α <1) as the summary information end time T_end. At this time, T_start < T_end.

  The summary information registration unit A4 registers the summary information based on the summary information start time T_start and the summary information end time T_end determined by the high level sound evaluation unit A3. Then, at least the time information is described in the summary information description part A5 and output as a summary information description file. As a format to be described, for example, a format defined by MPEG-7 can be used.

  If there is a plurality of audio information, the above process is performed on all audio information.

  The function of the audio summary information extraction device of the above-described embodiment can be realized by software (program), and the software can be recorded on a recording medium 100 such as an optical disk, a floppy (registered trademark) disk, a hard disk, or the like. . The extracted audio summary information can be recorded on the recording medium 100 as an individual file.

  FIG. 11 shows an example of a program recorded on the recording medium 100. The recording medium 100 includes a subband data extraction function 121 or / and a subband analysis function 122, a high level sound evaluation function 123, A summary information registration function 124 and a summary information description function 125 are recorded.

  FIG. 12 is a block diagram showing an embodiment of an audio / video outline information reproducing apparatus according to the present invention.

When the audio video summary information SUM extracted by the above means is input, the audio video separation unit P1 separates the summary information into the video element VSUM and the audio element ASUM. Next, the video speed conversion unit P2 converts the video playback speed by spatially thinning out the video elements VSUM in accordance with the conversion speed parameter SP given from the outside. Similarly, the audio element ASUM is thinned out in time at the same rate as the video element VSUM in accordance with the conversion speed parameter SP in the audio speed conversion unit P3 to convert the audio playback speed. Audio playback speed conversion can be realized by, for example, periodic skipping of audio frames or a combination of repeated playback and periodic skipping. Here, if the audio is to be 1.5 times faster, the former <frame number to play> 1, 2, 4, 5, 7, 8, 10, 11, ...
Is achieved by playing back two consecutive frames and skipping the next one frame. In the latter,
<Frame number to play> 1, 1, 4, 4, 7, 7, 10, 10, ...
Is achieved by repeatedly playing the same frame twice and skipping the next two frames.

  The speed-converted video element VSUM ′ and audio element ASUM ′ are input to the audio video multiplexing / synchronizing unit P4, multiplexed and synchronized, and obtained speed-converted audio video summary information SUM ′. It is done. The obtained audio video summary information SUM 'is displayed and reproduced.

It is a block diagram which shows the whole structure of one Embodiment of this invention. It is a block diagram which shows the detailed structure of the content analysis part of FIG. It is a block diagram which shows the detailed structure of the audio level evaluation part of FIG. It is a block diagram which shows the detailed structure of the high level sound analysis part of FIG. It is a block diagram which shows the detailed structure of the movement activity evaluation part of FIG. It is a block diagram which shows the detailed structure of the summary information registration part of FIG. It is a block diagram which shows the detailed structure of the summary information description part of FIG. It is a figure which shows the outline | summary of the program recorded on a recording medium. It is a block diagram which shows the structure of the audio summary information extraction apparatus of other embodiment of this invention. It is a block diagram which shows the detailed structure of the high level sound evaluation part of FIG. It is a figure which shows the outline | summary of the program recorded on a recording medium. It is a block diagram which shows the structure of the audio summary information reproducing | regenerating apparatus of other embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Content analysis part, 2 ... Audio level evaluation part, 3 ... Motion activity evaluation part, 4 ... Outline information registration part, 5 ... Outline information description part, A1 ... Subband Data extraction unit, A2 ... subband analysis unit, A3 ... high level sound evaluation unit, A4 ... summary information registration unit, A5 ... summary information description unit, P1 ... audio video separation unit, P2 ... Video speed conversion unit, P3 ... Audio speed conversion unit, P4 ... Audio video multiplexing / synchronization unit, 100 ... Recording medium.

Claims (5)

Means for extracting subband data from the input compressed audio content;
Means for calculating a sum of subband energies weighted by bands from the subband data;
From the sum of the sub-band energy, means for calculating a sub-band energy summation at a certain unit time,
The time position at which the subband energy sum in the unit time is maximum is determined as the summary information start time, and the time position at which the subband energy sum in the unit time is α times the maximum value (where 0 <α <1) , As a summary information end time,
An audio summary information extracting apparatus comprising means for extracting audio information in a section of the summary information start time and summary information end time as summary information. Means for converting the input audio content into subband data;
Means for calculating a sum of subband energies weighted by bands from the subband data;
Means for calculating a subband energy sum in a unit time from the sum of the subband energies;
The time position at which the subband energy sum in the unit time is maximum is determined as the summary information start time, and the time position at which the subband energy sum in the unit time is α times the maximum value (where 0 <α <1) , As a summary information end time,
Extractor audio summary information, characterized by comprising means for extracting the audio information of the summary information start time and summary information end time interval as summary information. The apparatus for extracting audio summary information according to claim 1 or 2 ,
A device for extracting audio summary information, comprising means for describing start time and end time of the summary information or start time and duration as time information of the extracted summary information. In the audio outline information extracting device according to any one of claims 1 to 3 ,
A device for extracting audio summary information, comprising means for outputting and saving the extracted summary information as a separate file. A computer to extract audio summary information,
At least one of means for extracting subband data from the input compressed audio content and means for converting the input audio content to subband data;
Means for calculating a sum of subband energy weighted by bands from the subband data;
Means for calculating a subband energy sum in a unit time from the sum of the subband energies;
The time position at which the subband energy sum in the unit time is maximum is determined as the summary information start time, and the time position at which the subband energy sum in the unit time is α times the maximum value (where 0 <α <1) , As a summary information end time, and
It means for extracting the audio information of the summary information start time and summary information end time interval as summary information,
The computer-readable recording medium which recorded the program which extracts the audio | voice outline information for making it function as.

Images