KR20030014804A - Apparatus and Method for Database Construction of News Video based on Closed Caption and Method of Content-based Retrieval/Serching It - Google Patents

Abstract

PURPOSE: A system and a method for generating a closed-caption-based news video database and a contents-based searching method are provided to simultaneously search a text and videos of a news item through a text searcher and an item searcher. CONSTITUTION: A system for generating a closed-caption-based news video database includes an embedded text structuring unit(203), an index and additional information extraction unit(204), and a synchronizing unit(206). The embedded text structuring unit segments a closed caption text(201) into significant units to divide it into structured embedded texts. The index and additional information extraction unit extracts indexes and additional information from the closed caption text to construct a term dictionary(207). The synchronizing unit synchronizes the closed caption text with audio signals of a news video(202).

Description

Apparatus and Method for Database Construction of News Video based on Closed Caption and Method of Content-based Retrieval / Serching It}

The present invention relates to a closed caption-based news video database generating device and method, a content-based search / search method, and a computer-readable recording medium recording a program for realizing the method.

In particular, the present invention relates to news video retrieval in the field of content-based multimedia retrieval, terminology dictionary construction by structuring closed subtitle news document, extracting index word and additional information thereof, time between closed subtitle string and audio information of audio track. Recorded apparatus and method for creating closed-captioned video database for searching news strings and videos synchronized with the main function elements, and content-based retrieval devices and methods, and programs for realizing the method. The present invention relates to a computer-readable recording medium.

The prior art of the news video retrieval method is as follows.

First, "Content Based Annotation and Retrieval of News Videos" (ICME2000: IEEE International Conference on Multimedia and Expo 2000) proposed by "M. Bertini" uses content segmentation and classification, text recognition, and character recognition. A method for video retrieval has been proposed.

In the above method, the video is first divided into shot units, and the divided shots are classified into anchor shots in which anchors appear and other service shots. Provide news video search function.

Accordingly, in order to add a semantic annotation, the frame detects a frame in which the caption exists in the video, separates the text from the background, recognizes the separated text, and uses it as a textual shot descriptor. Used. In addition, the words recognized through speech recognition for anchor shots were used as descriptors.

This method of the related art deals with the problem that the description and the search unit are composed of shots of physical units, not synonymous units, and anchor shots only, and furthermore, by describing only words recognized through speech recognition, The problem is that all words that occur are not used as a database.

Meanwhile, in "Telop-on-demand: Video structuring and retrieval based on text recognition" (ICME2000), jointly published by H. Kuwano and five others, news video subtitles (Embedded text) and its size, font and location We have proposed a method of constructing a database by structuring a video using information such as, extracting and recognizing subtitles.

Since the important meaning of the news is displayed in the subtitles by the subtitle editor, the method was used for structuring using additional information such as the subtitle size, font, and position. In addition, the method of this paper also provides a function to compose a summary video into a video frame including text and to view the entire video in a short time.

However, since the method uses only subtitles existing in the news video, it does not use much other information such as voice information used in the news, and thus has a very limited disadvantage in constructing a database. Since there are various styles of editing subtitles such as text, it is problematic to use only the size of subtitles.

The present invention has been proposed in order to solve the above problems, to structure a closed caption document, to extract a key word and its additional information for the closed caption document to build a term dictionary, the division of the unit Closed caption-based news video database generation device that extracts the title of the article and synchronizes the closed caption string with the news video, and as a result, provides a subtitle string of the news and the corresponding synchronized video through text query or article searcher. And a method, a content-based search / search method, and a computer-readable recording medium having recorded thereon a program for realizing the method.

1 is a diagram illustrating an overall system configuration of an apparatus for generating a closed caption based news video database according to the present invention.

Figure 2 is a detailed configuration diagram of an embodiment of a digital news information processing unit included in the closed caption-based news video database generating apparatus according to the present invention.

3 illustrates an example structured closed caption document in accordance with the present invention.

4 is a flow diagram of an embodiment of a content-based retrieval method for a closed caption based news video database according to the present invention;

5 is a flow diagram of an embodiment of a content-based search method for a closed caption based news video database according to the present invention;

* Explanation of symbols for the main parts of the drawings

101: digital news video database generator

102: digital news video search unit

103: digital news video information acquisition unit

104: digital news video information processing unit

105: news broadcast program

106: Digital News Database

According to an aspect of the present invention, there is provided a closed caption-based news video database generating device comprising: subtitle document structuring means for dividing a closed caption document into a structured subtitle document by hierarchically dividing the closed caption document into meaningful units; Index word and additional information extracting means for extracting an index word and additional information from the closed caption document hierarchically divided into meaningful units in the subtitle document structuring means to construct a term dictionary; And audio subtitle synchronizing means for synchronizing the character string of the closed caption document with the audio signal section of the digital news video.

In addition, the present invention provides a sub-unit title extracting means for extracting a title of a unit article from a closed caption document hierarchically divided into meaningful units in the subtitle document structuring means; And digital news video information acquiring means for acquiring the closed caption document in digital form and the digital news video.

The present invention also provides a closed caption-based news video database generation method applied to a closed caption-based news video database generating apparatus. The structured caption document information is generated by hierarchically dividing the closed caption document into meaningful units. A first step of making; A second step of generating term dictionary information by extracting an index word and additional information from the structured subtitle document information generated in the first step; A third step of generating title information of the unit article from the structured caption document information generated in the first step; A fourth step of generating synchronized time code information by synchronizing the character string of the closed caption document with an audio signal section of digital news video; And a fifth step of generating a database linking the text of the closed caption document and the digital news video based on the generated information and the digital news video.

The present invention also provides a content-based retrieval method for a news video database generated by a closed caption-based news video database generating apparatus, comprising: a first step of generating a closed caption-based news video database; A second step of searching a news video database generated in the first step according to a text query of a user; A third step of prioritizing and outputting the search result of the second step; And a fourth step of outputting the closed caption string and the corresponding news video according to the user's selection of the output result of the third step.

The present invention also provides a content-based search method for a news video database generated by a closed caption-based news video database generating apparatus, comprising: a first step of generating a closed caption-based news video database; And a second step of outputting the closed caption string and the corresponding news video according to the user's selection through the article explorer.

The present invention also provides a database generating apparatus having a processor, comprising: a first function of generating structured caption document information by hierarchically dividing a closed caption document into meaningful units; A second function of extracting index words and additional information from the structured subtitle document information generated in the first function to generate term dictionary information; A third function of generating title information of a unit article from the structured caption document information generated in the first function; A fourth function of generating synchronized time code information in synchronization with a character string of the closed caption document and an audio signal section of a digital news video; And a computer-readable recording medium having recorded thereon a program for realizing a fifth function of creating a database linking the text of the closed caption document with the digital news video based on the generated information and the digital news video. .

The present invention also provides a search apparatus having a processor, comprising: a first function of generating a closed caption-based news video database; A second function of searching a news video database generated in the first function according to a text query of a user; A third function of giving priority to and outputting a search result of the second function; And a computer readable recording medium having recorded thereon a program for realizing a fourth function of simultaneously outputting a closed caption string and a corresponding news video according to a user's selection of an output result of the third function.

The present invention also provides a search apparatus having a processor, comprising: a first function of generating a closed caption-based news video database; And a computer-readable recording medium having recorded thereon a program for realizing the second function of outputting the closed caption string and the news video according to the user's selection through the article explorer.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram illustrating an overall system configuration of an apparatus for generating a closed caption based news video database according to the present invention.

As shown in FIG. 1, FIG. 1 illustrates a digital news video database generation unit 101 for constructing a digital news video database and a search system using the entire system of the closed caption-based news video database generating device according to the present invention. And a digital news video retrieval unit (102).

Looking at each part in more detail, the digital news video database generation unit 101 processes and retrieves the digital news video information acquisition unit 103 for acquiring a closed caption document and news video in digital form and the acquired digital news video information. The digital news video information processing unit 104 is processed into a form suitable for the present invention.

In addition, the digital news video information acquisition unit 103 performs a function of obtaining a digital news video and a closed caption document when the input signal is analog.

Here, the closed caption document is a character string that can be viewed through the caption decoder and is distinguished from the embedded text displayed in the video. The data from the subtitle decoder is recorded by the registrar, which contains the string information of the voice pronounced by the speaker, additional information such as information about the speaker such as "anchor", "interview", "reporter", and other punctuation marks. Document. The document also contains information about the time when the subtitles appear, in word units.

The digital news video information acquisition unit 103 obtains the digital news video from the news broadcasting program 105 using the video encoder. If the input signal is a digital signal, the digital news video information acquisition unit 103 is omitted.

In addition, the digital news video information processing unit 104 performs a function of processing the closed caption document and the news video of the acquired digital form into data of a form suitable for an effective multi-modal search. Detailed description thereof will be provided with reference to FIG. 2.

Then, the digital news video search unit 102 searches the digital news video database 106 generated by the digital news video database generation unit 101 by using a character string or the like, by the user's selection. Is output with the news video image synchronized to the search result.

2 is a detailed block diagram of an embodiment of a digital news information processing unit included in a closed caption-based news video database generating apparatus according to the present invention.

As shown in FIG. 2, the digital news video information processing unit included in the closed caption-based news video database generating apparatus according to the present invention is used for hierarchical segmentation of the closed caption document 201 in meaningful units. An index word for constructing a glossary by extracting index words and additional information from closed caption documents hierarchically divided into meaningful units in the closed caption document structuring 203 and the closed caption document structuring 203. And the sub information extractor 204 and the subtitle extractor 205 and the closed caption which extract the title of the unit article from the closed caption document hierarchically divided into meaningful units in the subtitle document structurer 203. And an audio subtitle synchronizer 206 for synchronizing the character string of the document 201 with the audio signal interval of the digital news video 202.

Looking at each component in more detail, the closed caption document structurer 203 received the closed caption document 201 from the digital news information acquisition unit 103 is the closed caption document 201 in a meaningful unit The structured caption document 208 is extracted by performing hierarchical segmentation.

Here, the hierarchical division into meaningful units for the closed caption document 201 is required for use as a unit of news search. Generally, since the search is performed in articles, it is generally divided into articles. It may be used to search by dividing it into smaller units to refine and search.

More specifically, in a news video, an article is usually composed of multiple speakers, such as anchors, journalists, and interviews. Hierarchical partitioning can be done in units. When hierarchical splitting is performed for each speaker unit in an article, splitting information on the speaker is generally provided by a closed caption decoder. Therefore, it divides according to the segmentation information provided by the closed caption decoder.

In addition, the division of the article may use a method of schematizing and dividing the conversion pattern of some types of speakers accordingly.

In addition, a unique identification number of a division unit is assigned to manage the database in the division unit.

In addition, the index word and the additional information extractor 204 that receive the closed caption document hierarchically divided into meaningful units from the subtitle document structurer 203 extract the index word and the additional information from the hierarchical partitioned closed caption document. Create a dictionary 207.

Here, the index word means a noun-type word determined to be useful from a search point of view, and may be extracted using an arbitrary method such as a general morpheme analyzer method or a noun extractor.

In addition, the additional information on the frequency of the document in which the term occurs, the unique number (ID) of the document to which the term belongs, the frequency of the partition unit in which the term occurred, the unique number (ID) of the partition unit in which the term occurred, Term frequency; Among them, information related to frequency is used as a weight when showing search results having a rank in the search process. In other words, the lower the frequency of the document in which the term appears, the higher the frequency of the term is assigned to the document.

The information related to the frequency included in the additional information is incorrectly used for the related frequency information according to the unit of search. For example, if you are searching by article, use the frequency of the article with the index.

The information on the speaker of a term is information about the speaker to which the term belongs, and is used to limit the scope of the search to anchors, reporters, reporters, and the like.

The index word and its additional information may be stored in any structured form for efficient use in terms of storage and retrieval.

The division unit title extractor 205 extracts the title of the divided unit article from the closed caption document divided hierarchically by the subtitle document structurer 203. Here, the divided unit article refers to an article or a speaker unit within the article, which is a unit of hierarchical division in the subtitle document structurer 203.

Here, the title may be manually assigned by the person or automatically determined. In the case of automatic, the title is selected by selecting a part of the index word extracted from the corresponding article or the speaker unit within the article. As described above, a method of determining a title by selecting a part of the index word is performed using the order in which the index word is extracted or the weight of the index word. Each title extracted in this way is used to provide a search result through a text query.

The audio-subtitle synchronizer 206 synchronizes the closed caption string of the closed caption document 201 with the audio signal section of the digital news video 202.

In the digital news video 202, video data and audio data are synchronized in principle. However, the closed caption text and the time information of the closed caption document 201 may not be accurate with the audio information of the audio track generated by the speaker in the video program. There is no motive.

In general, the time code included in the closed caption character string of the closed caption document 201 stores time information about 2 to 7 seconds after the time when the corresponding voice is generated in the video program. Therefore, in order to segment the closed caption document 201 based on the content and to accurately segment the corresponding video using the content, it is necessary to synchronize the time code in the subtitle document with the video / audio signal interval of the video program using speech recognition technology. The process is necessary.

Accordingly, semantic segmentation using closed caption information brings about an effect of segmenting the corresponding video based on the content.

That is, the audio-subtitle synchronizer 206 extracts the closed caption string of the closed caption document 201 and the closed caption string of the closed caption document 201 from the time information, which is displayed in the digital news video, and the digital news video 202. The audio information is used as an input, and the time information of the input subtitle string is updated to the synchronized time code 210 which is actually pronounced time information.

At this time, in the voice recognition process, the calculation amount may be reduced by selecting only a predetermined section as a search range with reference to the time information input as an input.

Then, the digital news video database 106 generated by the digital news video information processing unit 104 is connected to the closed caption document 208, the index word, and the additional information extractor 204 structured by the subtitle document structurer 203. Terminology 207 generated by the subtitle title extractor 205, subunit title 209 generated by the subtitle title extractor 205, and closed time strings of the closed-captioned text strings by the audio-subtitle synchronizer 206 ) And digital news video 202.

3 is an illustration of a structured closed caption document in accordance with the present invention.

The structured subtitle document 208 according to the present invention is composed of information on the speaker and the like, a unique identification number assigned in division units, division information, and synchronized time information.

As shown in FIG. 3, those marked with <EVENT>, <ANCHOR>, etc. correspond to the partition information 303. In addition to the splitting information 303, information 301 for the speaker and the like and a unique identification number 302 allocated for each splitting unit are inserted.

In the case of the anchor as the information 301 for the speaker in FIG. 3, "1", the report "2", and the interview "3" are uniquely assigned. The number assignment for the speaker can be made by the user. The information 301 for the speaker is used as one of the additional information of the index word in the process of constructing the dictionary. This can be used to select the scope of the search in the search process. For example, in the case of limiting the search range to the anchor, only the case where the additional information about the speaker is the anchor 1 in the term dictionary is searched. The unique identification number assigned to each division unit may be uniquely assigned by the user without a duplicate number for each division unit. Assignment rules can be used by the user. In addition, it includes time information 304 and the like synchronized with a character string and a digital video voice signal in units of sentences.

4 is a flowchart illustrating an embodiment of a content-based retrieval apparatus for a closed caption-based news video database according to the present invention.

As shown in FIG. 4, the content-based retrieval apparatus for a closed caption-based news video database according to the present invention searches a term dictionary based on a user's text query as an input and performs a closed caption-based news. It shows the process of providing news videos and closed caption documents from video databases.

First, a closed caption-based news video database is created (401), and when a user makes a text query (402), a search is performed on a glossary in the news video database (403), and the searched results are ranked. Output 404.

Here, the ranking of the search results is based on the weight calculated using the index word frequency and the inverse document frequency, which is information on the ratio of the number of documents including the query word to the total number of search documents.

In addition, a search unit based on a user's text query may be specified as a date, an article, a speaker unit within an article, and the ranking may be determined using information related to the frequency of the search unit. In addition, since the terminology includes information on the speaker to which the term belongs, the scope of the search can be limited to the whole, anchors, reporters, interviews, and the like.

The search result as described above first outputs the date, title, etc. of the searched news to the user (404). When the user selects from the results (405), the closed caption string and the news video of the selected article are extracted and provided to the user (406). This operation is repeated until the end of the user's query (407).

At this time, the internal operation of the closed caption string and news video extraction is displayed to the user by accessing the corresponding string and video in the structured closed caption document using the document number of the term searched as a result of the searcher and the unique number of the division unit in the document. It depends on how to do it.

In the structured closed caption document, a division tag is inserted for each division unit, and a unique number of the division unit is inserted to accurately extract a range of a corresponding character string.

And, outputting the news video synchronized with the closed caption string uses information about the video section matched in divided units using the synchronized timecode existing in the digital news database as a result of the synchronization of the audio-subtitle synchronizer.

5 is a flowchart illustrating a content-based search method for a closed caption-based news video database according to the present invention.

First, a closed caption-based news video database is created (401), and when the user searches through the article navigator and selects a desired article (402), the closed caption string and the news video are provided (403). Then, the process 402-403 is repeatedly performed until the user finds the desired article (404).

Here, the article navigator includes a table of contents of the article and a hierarchical display of the titles of the articles according to the needs of the user, so that the user can effectively navigate the articles.

The title contents mean that the title contents are created by concatenating subtitle strings and video section information corresponding to the respective titles extracted by the division unit title extractor. A hierarchical Table of Contents such as Year / Month / Day / Article / Speaker can be configured, and the user can use this to search for the desired news.

For example, if a user selects a specific day, the article title corresponding to the day can be viewed, and if a specific article is selected, the title of the article can be viewed. Based on these titles, when the user finally selects the article they want to view, the text and video detectors provide the text and video of the news.

In the present invention, the word "search" of the search method of FIG. 5 is used differently from the word "search" of the search method of FIG. 4, and the meaning of "search" in the digital news search unit is a glossary by a text query. It is used in a broad sense including both search method and news search method through the article searcher.

Preferably, the digital news video database and searcher is located on a server, and the text query and article navigator is located on a client to configure a news string and synchronized news video search system of a server / client structure capable of remote service. Can be.

In this case, the server additionally needs a function for streaming news video, and the client may use a web browser.

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications, and changes are possible in the art without departing from the technical spirit of the present invention. It will be apparent to those of ordinary knowledge.

As described above, the present invention structured the closed caption document of the news video based on contents, constructs a glossary from the closed caption document, and performs temporal matching between the subtitle string and the audio track of the digital video to structure the document. Induces the structured effect of the content-based video, and extracting the title in the unit of division has the effect that you can search the text and video of the news article at the same time through a text searcher, article searcher and the like.

Claims (18)

In the closed caption based news video database generating device, Subtitle document structuring means for dividing the closed caption document into a structured subtitle document by performing hierarchical division into meaningful units; Index word and additional information extracting means for extracting an index word and additional information from the closed caption document hierarchically divided into meaningful units in the subtitle document structuring means to construct a term dictionary; And Audio subtitle synchronizing means for synchronizing the character string of the closed caption document with the audio signal section of the digital news video Closed caption-based news video database generation device comprising a. The method of claim 1, Division unit title extraction means for extracting the title of the unit article from the closed caption document hierarchically divided into meaningful units in the subtitle document structuring means; And Digital news video information acquisition means for acquiring the closed caption document in digital form and the digital news video; Closed caption-based news video database generation device further comprising. The method according to claim 1 or 2, The meaningful unit for hierarchical division by the subtitle document structuring means, Closed caption-based news video database generating device, characterized in that the article unit. The method according to claim 1 or 2, The meaningful unit for hierarchical division by the subtitle document structuring means, Closed caption-based news video database generating device, characterized in that the speaker unit in the article. The method of claim 1, The division unit title extraction means, Closed caption-based news video database generating device, characterized in that the title is given by the user. The method of claim 1, The audio subtitle synchronization means, The closed caption document input using the text of the closed caption document, the time caption of the closed caption document in the digital news video, and the audio information extracted from the digital news video by using a voice recognition method. Closed caption-based news video database generating device, characterized in that for updating the time information of the character string with the synchronized time code which is the actual pronunciation time information. The method of claim 2, The division unit title extraction means, The closed caption-based news video database generating device, characterized in that a title is selected by selecting a part of the index word of the divided unit article. In the closed caption-based news video database generating apparatus applied to the closed caption-based news video database generating device, A first step of generating structured subtitle document information by hierarchically dividing the closed caption document into meaningful units; A second step of generating term dictionary information by extracting an index word and additional information from the structured subtitle document information generated in the first step; A third step of generating title information of the unit article from the structured caption document information generated in the first step; A fourth step of generating synchronized time code information by synchronizing the character string of the closed caption document with an audio signal section of digital news video; And A fifth step of generating a database linking the text of the closed caption document and the digital news video based on the generated information and the digital news video; Closed caption-based news video database generation method comprising a. The method of claim 8, The structured subtitle document information, A closed caption-based news video database generation method comprising information on a speaker, etc., a unique identification number allocated in division units, split information, and synchronized time information. The method of claim 8, The additional information of the second step is In the structured subtitle document information generated in the first step, the frequency of the document in which the term occurs, the unique number (ID) of the document to which the term belongs, the frequency of the division unit in which the term occurs, and the uniqueness of the division unit in which the term occurs. Closed caption-based news video database generation method comprising a number (ID), the frequency of the corresponding term occurred in the division unit. A content-based retrieval method for a news video database generated by a closed caption based news video database generating device, Generating a closed caption based news video database; A second step of searching a news video database generated in the first step according to a text query of a user; A third step of prioritizing and outputting the search result of the second step; And A fourth step of outputting a closed caption string and a corresponding news video according to a user's selection of the output result of the third step; Content-based search method for a closed caption-based news video database comprising a. The method of claim 11, Prioritization of the third step, A closed-caption-based news video database based on weights calculated using index word frequency and inverse document frequency, which is information about the ratio of the number of documents including the query word to the total number of search documents. Based search method. A content-based search method for a news video database generated by a closed caption based news video database generating device, Generating a closed caption based news video database; And The second step of outputting the closed caption string and the news video according to the user's selection through the article navigator Content-based search method for a closed caption-based news video database comprising a. The method of claim 13, The article explorer, A content-based search method for a closed caption-based news video database, wherein the title table of the article and the title table are hierarchically displayed according to a user's request. The method of claim 14, The title table of contents is A content-based search method for a closed caption based news video database, comprising a year, month, day, article, and speaker. In a database generating device having a processor, A first function of generating structured subtitle document information by hierarchically dividing the closed caption document into meaningful units; A second function of extracting index words and additional information from the structured subtitle document information generated in the first function to generate term dictionary information; A third function of generating title information of a unit article from the structured caption document information generated in the first function; A fourth function of generating synchronized time code information in synchronization with a character string of the closed caption document and an audio signal section of a digital news video; And A fifth function of generating a database linking the text of the closed caption document and the digital news video based on the generated information and the digital news video; A computer-readable recording medium having recorded thereon a program for realizing this. In a search device equipped with a processor, A first function of creating a closed caption based news video database; A second function of searching a news video database generated in the first function according to a text query of a user; A third function of giving priority to and outputting a search result of the second function; And A fourth function of outputting a closed caption string and a corresponding news video according to a user's selection of an output result of the third function; A computer-readable recording medium having recorded thereon a program for realizing this. In a search device having a processor, A first function of creating a closed caption based news video database; And A second function that outputs the closed caption string and the news video according to the user's selection through the article navigator A computer-readable recording medium having recorded thereon a program for realizing this.