KR100404322B1 - A Method of Summarizing News Video Based on Multimodal Features - Google Patents

Abstract

The present invention relates to a multi-modal feature based news video summary method.

In the present invention, multi-modal features are extracted from the news video containing the closed caption signal, and based on this, the main section of the news video is automatically detected to summarize the news video. To this end, a digital content acquisition step of converting and storing the input target video signal into a digital signal, a multimodal feature extraction step of extracting features from each of the elements of the multimodal constituting the acquired digital content, and the extracted multimodal feature A main section detection step of detecting a main section of the news video based on the, and a video summary description step of structurally describing the summary information of the news video based on the main section detection results.

As a result, stable performance can be obtained by reducing the complexity of speech recognition and improving reliability.

Description

A method of summarizing news video based on multimodal features}

The present invention relates to a news video summarization method based on a multimodal feature. More specifically, the present invention relates to a multimodal feature that divides a news video including closed caption information into units of news articles and summarizes news videos by segments. To news based video summary method.

Conventional techniques for news videos or general video summaries are proposed by M. A. Smith and T. Kanade [Articles: Video Skimming and Characterization through the Combination of Image and Language Understanding Technique, Publication: Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp 775-781, Year of Publication: 1997]. This paper proposes a summary method using video multi-modal features such as image analysis and language recognition technology. This summary method detects camera movement, face area and subtitle area from video signal, and obtains it through speech recognition process from audio signal. (transcript) indexes key words, and then merges them and takes them as key sections. However, these methods do not go through a separate article structure analysis of the news video, making it difficult to provide structured summaries, and because the main segments of the audio are extracted on a word-by-word basis, speech is often played back when playing a summary of the news video. It is cut off and the meaning is not delivered properly. In addition, since the speech recognition process is performed to automatically generate the script, the reliability and the complexity are also low compared to the case of using the closed caption signal on the video.

Another prior art is Q. Huang et al. [Article: Automated Generation of News Content Hierarchy by Integrating Audio, Video, and Text Information, Place: Proc. IEEE Conference on Acoustics, Speech, and Signal Processing, pp 3025-3028. Here, we proposed a method of hierarchically structuring the content of news video by utilizing all the multimodal features of the video. In this method, when closed caption data is given along with the video, it can be used, and it is accompanied with the process of analyzing the article structure of the news video by separating the advertisement part and detecting the metabolic part of the anchor through the analysis of the audio signal. However, this method does not give much information on the visual side as it can contain only anchor shots in the video summary. Furthermore, when they are included in the summary, the entire anchor shot paragraph is included at the same time. Bulky has its drawbacks. In addition, it is assumed in advance that the closed caption data is synchronized with an audio signal in time. In fact, most of the closed caption data provided in the news video is not, and in this case, the processing method is very unclear.

SUMMARY OF THE INVENTION An object of the present invention for solving the above-mentioned problems of the prior art is to divide a news video provided with an input closed caption into news article units and to summarize news videos by divided news article units, thereby providing content-based video browsing. In addition, the indexing results of the closed caption data generated in the middle are to provide a multi-modal feature-based news video summary method that is used when constructing a database for text-based video retrieval.

1 is a flow diagram illustrating a multi-modal feature based news video summary method according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating a digital content acquisition process in detail among the multi-modal feature based news video summary method illustrated in FIG. 1;

FIG. 3 is a detailed diagram illustrating a multimodal feature extraction process in the multi-modal feature based news video summary method of FIG. 1 according to the present invention.

※ Explanation of code about main part of drawing ※

210: video compression encoder 220: closed caption decoder

310: Shot boundary detector 320: Audio-closed caption synchronizer

330: closed caption text analyzer

In the multi-modal feature-based news video summary method containing closed caption data according to the present invention for achieving the above object, Digital content acquisition step of converting and storing the broadcast signal of the news video to a digital signal Wow ; A multi-modal feature extraction step of extracting a feature from each element of the multi-modal constituting the acquired digital content; A main section detecting step of detecting a main section of the news video based on the extracted multi-modal features; And a video summary description step of structurally describing summary information of the news video based on the detected main section.

Preferably, the closed caption data is stored in a program that can be executed by the computer to perform the listed multimodal (Multimodal) feature-based news video summary way the recording medium, a broadcast signal of the news video to a digital signal Digital content acquisition step of converting and storing; A multi-modal feature extraction step of extracting a feature from each element of the multi-modal constituting the acquired digital content; A main section detecting step of detecting a main section of the news video based on the extracted multi-modal features; A computer readable recording medium having stored therein a program comprising a video summary description step of structurally describing summary information of the news video based on the detected main section.

Hereinafter, a multimodal feature based news video summary method according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings. 1 is a flowchart illustrating a multi-modal feature based news video summary method according to an embodiment of the present invention.

The information summarization method of a news video characterized by multimodal (multimodal signals including audio, video, and textual information), as shown, a digital content acquisition step S110 and a multimodal feature extraction step. (S120), a main section detection step S130, and a video summary description step S140.

First, in the digital content acquisition step S110, an analog news video broadcast signal containing a closed caption signal is input, separated into an audiovisual signal and a closed caption signal, and then converted into a digital format and outputted. At this time, the audiovisual signal is converted into a compressed digital signal using one of several conventionally known compression coding schemes. On the other hand, the closed caption signal is decoded and stored in a text format, but the time information in which the corresponding subtitle appears is also included. Save it.

Next, the multi-modal feature extraction step (S120) extracts a series of features necessary for detecting the main section from each of the multi-modal elements (video, audio, and closed caption text data) of the digital content obtained in the digital content acquisition step (S110). Step. That is, by detecting the shot boundary from the video signal and applying the speech recognition method to the audio track based on the closed caption data, time synchronization between the audio signal and the closed caption data is achieved and the closed caption data is divided by speaker and news story unit. Split and index. At this time, the index importance such as the frequency of each extracted index word in each split unit is also calculated.

Next, the main section detecting step S130 detects a few main sentences for each article unit based on the index result of the closed caption data obtained through the multi-modal feature extraction step S120, and then includes the main section to be included in the summary video. It is a step of determining. The process of detecting the main sentence is to calculate the importance of all the sentences in each paragraph, and then extract as the number of sentences of the higher priority. At this time, the importance of the sentence is calculated by weighted sum of the importance of the index words included in each sentence based on the index importance of each word extracted in the closed caption data indexing process. The section in which each main sentence extracted through the above method is located in the actual video is determined as the main section to be included in the summary video. In addition, by using the video shot boundary information obtained in the multi-modal feature extraction step (S120) includes correcting so that both ends of the main section is not visually disturbing.

Meanwhile, the main sections extracted from each paragraph of the introductory section and the detailed section are classified into different levels, and then, in the next step, the video summarization technique S140, the video summarization is formed in a hierarchical form.

Finally, the video summary description step S140 receives the video shot boundary information extracted in the multi-modal feature extraction step S120 and the main section information detected in the main section detection step S130, and then is determined by various standardization mechanisms. Summarize the video using the technical definition language. At this time, each major section detected in the main section detecting step (S130) is described in a hierarchical form according to its level, and event subject information is automatically, semi-automatically or manually extracted for each major section, and the information is added to the technical data. Include.

FIG. 2 is a diagram illustrating a digital content acquisition step in detail in the multi-modal feature-based news video summary method of FIG. 1 according to the present invention. After receiving an analog news video including a closed caption signal, FIG. It is a process of dividing the audiovisual signal and closed caption signal into digital signals and outputting them.

As shown, the audiovisual signal is converted into a digital signal compressed through the video compression encoder 210, and the closed caption signal is output as a file in a text format through the closed caption decoder 220. At this time, when the closed caption is decoded and stored, time information in which the subtitle appears is also stored.

For example, when time information is extracted for each word unit of the closed caption, whenever a space character appears in the closed caption, the corresponding field value of the analog input signal may be used as the time information.

Here, the field value is a count value that can count each field distinguished by the vertical synchronous scanning signal in the analog broadcast signal, and can be initialized to 0 at an arbitrary time point, and the value is set to 1 for each field. It means increasing value.

In addition, when the video compression encoder 210 and the closed caption decoder 220 are configured as separate modules, a time reference point sharing control signal for sharing the reference point of time information is separately required between them. To this end, when the field value is initialized in the closed caption decoder 220, a time reference point sharing control signal is transmitted to the video compression encoder 210 so that a zero time point and the video compression encoder 210 of the closed caption decoder 220 are transmitted. The error between the zero time points is calculated and then the time information extracted for each word in the closed caption decoder 220 is corrected by the error.

FIG. 3 is a detailed diagram illustrating a multimodal feature extraction process in the multi-modal feature based news video summary method of FIG. 1 according to the present invention. In this step, each feature of the multimodal to be used in the main section extraction step is performed.

First, shot boundary detector 310 extracts a shot boundary from a video signal by any method. The extracted shot boundary information is described later as part of the description data in the video summary description step, and is used to correct this when both ends of the detected main section are visually disturbing in the main section detection step.

In general, the closed caption data comes out behind the audio signal in the case of real-time subtitles in which the short reporter listens to the contents of the news dialogue being performed live. In order to compensate for such a time difference, the audio-caption subtitle synchronizer 320 receives the closed caption data and performs an arbitrary speech recognition process in a state where the recognition target is clear, and extracts each word of each subtitle in the digital content acquisition process. Updated time information. Through such a process, the closed caption data in which time information of each word is synchronized with an audio signal in time is inputted by the closed caption text analyzer 330.

The closed caption text analyzer 330 divides and indexes the closed caption data by speaker and article unit by using speaker tag information included in the closed caption data.

The process of indexing closed-caption data consists of a noun extractor or a dictionary, and extracts phrases suitable as index words from the closed-caption data, and index frequency, which is the frequency in which these index words appear in each partition unit. TF) and Inverse Document Frequency (IDF), which are the number of splitting units in which these index words appear.

The extracted index word frequency value (TF) and reverse literature frequency value (IDF) are used as a measure for calculating the importance of each index word, and the expression of the frequency value (TF) * reverse literature frequency value (IDF) is used as the index word importance value. Use it as a formula.

While the invention has been described above based on the preferred embodiments thereof, these embodiments are intended to illustrate rather than limit the invention. It will be apparent to those skilled in the art that various changes, modifications, or adjustments to the above embodiments can be made without departing from the spirit of the invention. Therefore, the protection scope of the present invention will be limited only by the appended claims, and should be construed as including all such changes, modifications or adjustments.

As described above, according to the present invention, by applying the speech recognition method based on the closed caption data to the news video, not only the video section corresponding to the sentence representatively representing each news article as the main section, In the process, it is possible to obtain stable performance by reducing the complexity of speech recognition and improving reliability.

Claims (8)

A first step of processing a news video signal including closed caption data into an audiovisual signal of a digital signal and a closed caption signal in a text format including time information in which subtitles appear; Applying a voice recognition technique to synchronize time between the audio signal of the audiovisual signal and the closed caption data; A third step of classifying and classifying the closed caption data by speaker and article unit; Analyzing the indexing result of the closed caption data, and detecting the main section of each sentence unit in each of the introduction unit and the above-mentioned detail unit for each article unit including an introduction unit including an anchor shot and a detailed unit including a detailed report of a reporter. 4 steps; And And a fifth step of hierarchically constructing summary information of the news video by dividing the main section detected by the introduction section and the main section detected by the detail section into different levels. How to summarize news video. The method of claim 1, When summarizing the main section into the news video, using the shot boundary information detected from the video signal of the audio-visual signal, correcting both ends of the main section without visually disturbing the multi-modal. Feature based news video summary method. delete delete delete delete delete A first step of processing a news video signal including closed caption data into an audiovisual signal of a digital signal and a closed caption signal in a text format including time information in which subtitles appear; Applying a voice recognition technique to synchronize time between the audio signal of the audiovisual signal and the closed caption data; A third step of classifying and classifying the closed caption data by speaker and article unit; Analyzing the indexing result of the closed caption data, and detecting the main section of each sentence unit in each of the introduction unit and the above-mentioned detail unit for each article unit including an introduction unit including an anchor shot and a detailed unit including a detailed report of a reporter. 4 steps; And A computer program storing a program capable of executing a fifth step of hierarchically configuring the summary information of the news video by dividing the main section detected by the introduction section and the main section detected by the detail section into different levels. Recordable Media.