JP2007513398A - Method and apparatus for identifying high-level structure of program - Google Patents

Abstract

An apparatus and method for restoring a high-level configuration of a program such as a television or video program using an unsupervised clustering algorithm with manual analysis is provided. The method has three stages: a first stage, referred to herein as a text type clustering stage, a second stage of a genre / subgenre identification stage in which the genre / subgenre type of the target program is detected, and a structure here. It consists of a third final stage called the restoration stage. The structure restoration phase uses a schematic model to represent the program structure. Once restored, the high-level structure of the program may be effectively used to restore further information, including but not limited to temporal events, text events, program events, and the like.

Description

  The present invention relates generally to the technical field of video analysis, and more particularly to identifying high-level structures of programs such as television and video programs using classifiers for various types of video text aspects that appear in the program.

  As video becomes increasingly popular, more efficient methods of analyzing the content contained therein are becoming increasingly necessary and important. In essence, video contains a complex amount of data that makes analysis difficult and a huge amount of data. An important analysis is to understand the high-level structure of the video that can provide the basis for further detailed analysis.

  Several analysis methods are known. By Yeung et al., "Video Browsing using Clustering and Scene Transitions on Compressed Sequences" (Multimedia Computing and Networking 1995, Vol.SPIE2417, pp.399-413, February 1995), "Time-constrained Clustering for Segmentation of Video into Story by Yeung et al. Units "(ICPR, Vol. C. pp. 375-380 August 1996), Zhong et al.," Clustering Methods for Video Browsing and Annot. tion "(SPIE Conference on Storage and Retrieval for Image and Video Databases, Vol.2670, February 1996)," ViBE by Chen et al: A New Paradigm for Video Database Browsing and Search "(Proc.IEEE Workshop on Content-Based Access of Image and Video Databases, 1998), and “Automatic Parsing of TV Soccer Programs” by Gong et al. (Proceedings of the International Conference of Multimedia Computing and systems (ICMCS), see May 1995).

  Gong et al. Describe a system that uses domain knowledge and a domain-specific model in structural analysis of soccer video. As with other prior art systems, the video is first segmented into shots. A shot is defined as every frame between shutter opening and closing. Spatial features (playing field lines) extracted from the frames in each shot are used to classify each shot into different categories such as penalty area, midfield, corner area, corner kick, goal shot, etc. . Note that this work relies heavily on accurately segmenting the video into shots prior to feature extraction. Each shot does not represent an event that occurs in a soccer video.

  Zhong et al. Describe a system for analyzing sports video. The system detects boundaries of high-level thematic units such as baseball pitching and tennis serve. Each semantic unit is further analyzed to extract interesting events such as the number of strokes in tennis, play type, return to the net, and baseline return. A color-based adaptive filtering method is applied to each shot keyframe to detect a particular view. Complex features such as edges and moving objects are used to verify and refine the detection results. Note that this work is also greatly affected by accurately segmenting the video into shots prior to feature extraction. In short, Gong and Zhong regard video as a connection of basic units in which each unit is a shot. The resolution of feature analysis is not finer than the shot level. The task is very detailed and relies heavily on color-based filtering to detect a particular view. Furthermore, the system becomes useless if the color palette of the video changes.

  Therefore, in general, the prior art is as follows. First, the video is segmented into shots. Thereafter, key frames are extracted from each shot and grouped into scenes. A scene transition graph and a hierarchical tree are used to represent these data structures. The problem with these approaches is the mismatch between low level shot information and high level scene information. These only work when interesting content changes correspond to shot changes.

  In many applications, such as soccer video, interesting events such as “play” cannot be defined by shot changes. Each play may have multiple shots with a similar color distribution. Transition between plays is difficult to detect by simple frame clustering based on simple shot characteristics.

  In many situations where there is substantial camera movement, the shot detection process is from this low-level feature without this type of segmentation taking into account the domain-specific high-level syntax and content model of the video. Therefore, it tends to segment by mistake. Therefore, it is difficult to mediate the gap between low and high level features based on shot level segmentation. In addition, much information is lost during the shot segmentation process.

  Different domain images have very different features and structures. Domain knowledge can greatly facilitate the analysis process. For example, in a sport video, there are usually a certain number of transition syntaxes, camera control rules, views, and cameras imposed by game rules such as every play in soccer, every serve in tennis, and every inning in baseball.

  Tan et al., "Rapid estimation of camera motion from compressed video with application to video annotation" in the (IEEE Trans.on Circuits and Systems for Video Technology, 1999), Zhang et al., "Automatic Parsing and Indexing of News Video" (Multimedia Systems, Vol.2, pp.256-266, 1995) describes video analysis of news and baseball. However, for more complex and broader images, only a very few systems consider high-level structures.

  For example, a soccer video has a problem that a soccer game has a relatively loose structure compared to other videos such as news and baseball. Except for the structure of play units, the content flow is totally unpredictable and can occur randomly. There are numerous movements and view changes in the video of a soccer game. Solving this problem is useful for automatic content filtering for soccer fans and professionals.

  The above problem is more interesting in the broader context of video structure analysis and content understanding. The main structural issue is the time sequence of high-level video states such as soccer game play and break game states. It is desired to automatically parse a continuous video stream into an alternating sequence of the two game states.

  Most conventional structural analysis methods focus on detecting domain-specific events. The structural analysis independent of event detection has the following effects. Typically, only 60% or less of content corresponds to play. Therefore, considerable information reduction could be realized by eliminating the video portion corresponding to the break. Also, content characteristics in play and break are different, and the event detector could be optimized with knowledge of such previous conditions.

  Structural analysis of related technologies is mostly related to sports video analysis including soccer and other various games and general video segmentation. In soccer video, the prior art is related to shot classification, scene reconstruction by Gong as described above, “Analysis and Presentation of Soccer High Digital Video” by Proc. See “Detecting Semantic Events in Soccer Games: Towers A Complete Solution” (Proc. ICME 2001, August 2001).

  Hidden Markov Models (HMMs) have been used to distinguish between general video classification and various types of programs such as news and commercials. See "Joint video scene segmentation and classification based on hidden Markov model" (Proc. ICME 2000, pp. 1551-1554 Vol. 3, July 2000) by Huang et al.

  Heuristic rules based on domain specific features and dominant color ratios are also utilized to segment play and breaks. "Algorithms and system for segmentation and structure analysis in soccer video" (Proc. ICME 2001, August 2001), US Patent Application No. 9/24 filed April 20, 2001, Xu et al. See “Method and System for High-Level Structure Analysis and Event Detection in Domain Specific Videos”. However, these feature changes are difficult to quantify with explicit low-level decision rules.

  Therefore, there is a need for a framework that retains all the information of the low-level features of the video and better represents the feature sequence. At this time, it is possible to have a domain-specific syntax and a content model in order to specify a high-level structure that enables video classification and segmentation in a high-level program structure as well as a simple shot.

  The main idea of the present invention is to identify the high-level structure of programs such as televisions and video programs that use unsupervised clustering algorithms in cooperation with analysts.

  More particularly, the present invention provides an apparatus and method for automatically determining the high level structure of programs such as television and video programs. The method of the present invention comprises three stages: a first stage, referred to herein as a text type clustering stage, a second stage of a genre / subgenre identification stage in which the genre / subgenre type of the target program is detected, and Then, it is comprised from the 3rd final stage called a structure restoration stage. The structure restoration phase uses a schematic model to represent the program structure. The graphical model used for training can be a Petri net constructed manually or a hidden Markov model constructed automatically using a Baum-Welch training algorithm. To clarify the structure of the target program, the Viterbi algorithm may be used.

  In the first stage (ie, text type clustering), duplicate text is detected from a frame of a target program such as a television or video program that is of interest to the user. For each line of text detected in the target program, various text features such as position (line, column), height, font type and color are extracted. A feature vector is formed from the extracted text features for each line of detected text. Next, feature vectors are grouped into clusters based on unsupervised clustering techniques. At this time, the clusters are labeled according to the type of text (name plate, score, opening credit, etc.) described by the feature vector.

  In the second stage (ie, genre / sub-genre identification), training processing is performed, whereby training videos representing various genres / sub-genre types are described above in the first stage to determine their respective cluster distributions. Analysis according to the method. Once acquired, the cluster distribution functions as a genre / sub-genre identifier of various genres / sub-genre types. For example, comedy movies have a certain cluster distribution and baseball games have a different cluster distribution. However, each appropriately represents their genre / sub-genre type. As a result of the training process, the genre / sub-genre type of the target program is the cluster distribution obtained previously in the first stage (text type clustering), and the genre / sub-genre type of the various genres / sub-genre types acquired in the second stage. It may be determined by comparing with the cluster distribution.

  In the third final stage (ie, the high-level program structure restoration stage), the high-level structure of the target program is first restored by constructing a database of higher-level graphical models, so that the model can be / The flow of video text in a sub-genre type program is schematically represented. When a graphical model database is constructed using the text detection result determined in step 140 and the cluster distribution result determined in step 160, one schematic model is identified from a plurality of stored models. And extracted. The selected graphical model, along with text detection and cluster information, is used to restore the high level structure of the program.

  The high-level structure of a program such as a video or television program is not limited to the following, but as a recommended device, search for time events, text events and / or program events in the target program and multimedia summary of the target program. The present invention can be effectively used in a wide range of applications including the following configurations.

  The above features of the present invention will become more readily apparent and understood by referring to the following detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings.

  In the following detailed description of the present invention, numerous specific details are given in order to provide a thorough understanding of the present invention, but the invention may be practiced without these specific details. In some embodiments, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention. Furthermore, the various examples used to explain the principles of the present invention in FIGS. 1-6 and the patent literature described below are merely illustrative and should not be construed as limiting the scope of the present invention.

  In the following description, a preferred embodiment of the present invention, usually implemented as a software program, will be clearly described. Those skilled in the art will readily recognize that the equivalent of such software may be configured by hardware. Since video processing algorithms and systems are well known, this description is particularly concerned with algorithms and systems that form part of, or more directly coordinate with, the systems and algorithms according to the present invention. Such algorithms and systems not specifically shown or described herein, and other features of hardware and / or software that generate and process related video signals are known in the art, May be selected from algorithms, components and elements. Given a system and method as described below by the present invention, software not specifically illustrated, suggested or described herein, according to the prior art, is useful for implementing the present invention. And within the scope of those skilled in the art.

  Further, as used herein, a computer program can be, for example, a magnetic storage medium such as a magnetic disk (such as a hard drive or floppy disk) or a magnetic tape, an optical disk such as an optical tape, an optical tape, or a machine-readable barcode. Computer-readable storage including storage media, solid state electronic storage devices such as RAM (Random Access Memory) and ROM (Read Only Memory), or any other physical device or medium utilized to store computer programs It may be stored on a medium.

  The following description uses the terms defined below.

The genre / subgenre genre is the type, category, or type of literature or artwork, and the subgenre is a category that belongs to a certain genre. An example of the genre is “sports” having sub-genres such as basketball, baseball, soccer, and tennis. Examples of other genres are “movies” with sub-genres such as comedy, tragedy, musical, and step. Examples of other genres include “news”, “music show”, “nature”, “talk show”, “kids show”, and the like.

Target program A target program is a video or television program in which the end user is interested. It is provided as an input to the process of the present invention. Processing on a target program in accordance with the principles of the present invention provides the following possibilities. (1) Allows the end user to receive a multimedia summary of the target program, (2) restores the high-level structure of the target program, (3) determines the genre / subgenre of the target program, (4) desires the program Or detection of predetermined content in the target program that may be undesired content, and (5) receiving information about the target program (ie, as a recommended device).

Clustering clustering divides a vector so that vectors having similar contents belong to the same group, and each group is as different as possible.

Clustering Algorithm A clustering algorithm is performed by detecting groups of similar items and grouping them into categories. When no category is specified, this is sometimes referred to as unsupervised clustering. When a category is specified in advance, this is sometimes referred to as supervised clustering.

  1-3, the method of the present invention according to one embodiment is shown.

  FIG. 1 is a flowchart illustrating a first stage of the present invention according to one embodiment, referred to herein as a text type clustering stage 100, where the superimposed or attached text is a television or video that the user is interested in. It is detected from the frame of the target program such as a program.

  FIG. 2 is a flow chart illustrating the second stage of the present invention according to one embodiment, referred to herein as genre / sub-genre identification, during which the training process is performed, thereby providing various genre / sub-genre types. Are analyzed to determine their cluster distribution. Once acquired, the cluster distribution is used as a genre / subgenre identifier of various genres / subgenre types. As a result of the training process, the genre / sub-genre type of the target program may be determined by comparing its cluster distribution with the cluster distribution of various genres / sub-genre types obtained during training.

  FIG. 3 is a flow chart illustrating the third stage of the present invention according to one embodiment, called the target program structure restoration stage, during which the high-level structure of the target program first builds a higher-level graphical model database. Thus, each model graphically represents the flow of video text through a genre / sub-genre type program. When the database is constructed, the results previously obtained at the processing stage such as text detection and cluster distribution results for the target program are one of those stored in the database to restore the high level structure of the program. Used to identify and select two graphical models.

  Note that not all of the steps described in the process flow diagrams described below may be performed in addition to those shown. Also, some steps may be performed substantially simultaneously with or during other steps. After reading this specification, one of ordinary skill in the art will be able to determine that a step is available for a particular requirement.

I. First Stage—Text Type Clustering As shown in the flow chart of FIG. 1, the first stage, text type clustering stage 100, generally includes the following steps.

110—Detection of the presence of text in a “target program” of interest to an end user, such as a television or video program 120—Identify and extract text features of each line of video text detected in the target program 130—Specify and extract Forming Feature Vectors from Features 140—Organizing Feature Vectors into Clusters 150—Labeling Each Cluster According to the Type of Video Text Present in the Cluster Each of the above general steps is described in more detail.

  In step 110, the process begins by analyzing the “target” television or video program to detect the presence of text contained within each video frame of the target program. A more detailed description of video text detection is included in US Pat. No. 6,608,930 “Method and System” issued to Agnihotri et al. On Aug. 19, 2003, which is hereby incorporated by reference in its entirety. for Analyzing Video Content Using Detected Text in Video Frames ". Text types that can be detected from the target program may include, for example, starting and ending credits, scores, title text, nameplates, and the like. Alternatively, text detection may be implemented according to the MPEG-7 standard that describes how to segment a still or moving video object.

  In step 120, text features are identified and extracted from the text detected in step 110. Examples of text features include position (row and column), height (h), font type (f) and color (r, g, b). Others are possible. For location features, for the purposes of the present invention, a video frame is considered to be divided into a 3 × 3 grid that yields nine designated areas. The row and column parameters of the location feature define the area where the text exists. For the font type (f) feature, “f” indicates the font type being used.

In step 130, for each row of the detected text, extracted text features are grouped into a single feature vector F _v.

In step 140, the feature vector F _V is converted into clusters {C1, C2, C3,. . . }. The grouping consists of feature vector F _V1 and clusters {C1, C2, C3,. . . } And F _V2 to associate the feature vector F _V1 with the cluster with the highest similarity. Unsupervised clustering algorithms may be utilized to class ring feature vector F _V on the basis of this similarity.

  In one embodiment, the distance metric used is

として計算される各自のテキスト特徴における差の絶対値の和として計算されるマンハッタン距離である。

Is the Manhattan distance calculated as the sum of the absolute values of the differences in their text features.

  It should be noted that the weighting factors w1-w4 and “Dist” may be determined empirically.

  In step 150, each cluster {C1, C2, C3,. . . } Is labeled according to the text type of the cluster. For example, cluster C1 may have a feature vector that describes text that is always delivered in yellow and always located in the lower right portion of the screen. Thus, cluster C1 is labeled "subsequent program notification" because the described feature represents text that notifies subsequent shows. As another example, cluster C2 may have a feature vector that describes text that is always delivered in blue by the surrounding black banner and that is always located in the upper left portion of the screen. Thus, cluster C2 is labeled "Sport Score" because the text feature is used to always display the score.

  The process of labeling clusters, ie step 150, may be performed manually or automatically. The effect of the manual approach is that the cluster labels become more intuitive, such as “title text”, “news update”, etc. Automatic labeling generates labels such as “text type 1” and “text type 2”.

II. Second Stage-Genre / Sub-Genre Identification The second stage, ie, the genre / sub-genre identification stage 200 as shown in the flowchart of FIG. 2, generally includes the following steps.

210—Perform Genre / Sub-Genre Specific Training a—Several training videos N of a certain genre / sub-genre type are provided as input 210. b—Text detection is performed for each training video N.

    210. c—Text features are identified and extracted for each line of detected text in each training video N.

    210. d-Step 210. A feature vector is formed from the text features extracted in c.

    210. e-cluster type {C1, C2, C3,. . . } And step 210. By using the distance metric to correlate with the feature vectors formed in d, the cluster types {C1, C2, C3,. . . } Is derived from the feature vector.

  220—Genre feature vectors are constructed for the genre / sub-genre type of the target program.

  Table I is exemplarily provided to help understand how genre feature vectors are used to define various genre / sub-genre types. In step 210, the rows of Table I show the various genres / sub-genre types, and columns 2-5 show the cluster distribution (count) that occurs after genre / sub-genre identification.

ジャンル／サブジャンル特定から決定されるジャンル特徴ベクトルは、映画／ウェスタン＝｛１３，４４，８，４３｝、スポーツ／野球｛５，３３，８，４｝などの各ジャンル／サブジャンルタイプを特徴付ける。

The genre feature vector determined from the genre / sub-genre specification characterizes each genre / sub-genre type such as movie / western = {13, 44, 8, 43}, sports / baseball {5, 33, 8, 4}. .

  In step 220, the genre / sub-genre type of the target program is determined. Here, the cluster distribution of the target program (previously calculated in step 140) is compared with the cluster distribution determined in step 210 for various genres / sub-genre types. The genre / sub-genre type of the target program is determined by determining which of the cluster distributions determined in step 210 is closest to the cluster distribution of the target program determined in step 140. Threshold determination may be used to ensure sufficient similarity. For example, the class distribution of the target program is required to have a similarity score of at least 80% of the nearest cluster distribution determined in step 210 to declare that the target program genre / sub-genre identification was successful. It may be.

Before describing the third stage 300 of Petri nets , ie, the high-level structure restoration stage 300, an outline of some basic principles of graphical modeling can be obtained by focusing on Petri net theory as a basis, as described below. Given.

  The basics of Petri nets are well known and James L. of the University of Texas at Austin. It is given in detail in “Petri Net Theory and the Modeling of Systems” by Peterson. This book is published by Plentice-Hall, Inc. of Englewood Cliffs, New Jersey and is incorporated herein by reference.

  Briefly, a Petri net is a specific type of directed graph that consists of two types of nodes called locations and transitions that have a directed arc from one location to another, or from one transition to another. It is. Locations are used to collect tokens, elements are used to represent what circulates in the system, and transitions move tokens between locations.

  In FIG. 4, an exemplary Petri net system with its location, transitions, arcs and tokens is shown. The Petri net shown in FIG. 4 is a schematic model for modeling an introductory segment of the movie “The Player”. In the movie, the beginning of the movie credit is indicated at three text positions, referred to herein as L1, L2 and L3. The appearance and subsequent disappearance of text in the introductory segment at locations L1, L2, and L3 is modeled graphically by Petri nets with respect to system states and their changes. More specifically, as will be described later, the system state is modeled as one or more conditions, and changes in the system state are modeled as transitions.

  With continued reference to FIG. 4, “Place” of an example Petri net is represented by a circle and labeled P1 to P6, which represents “Condition” in this example. For example, one condition of Petri in FIG. 4 is “text appearing at movie screen position L1”. This condition is associated with location P5 for modeling. Transitions are represented by rectangles and are labeled t1-t8 and represent events. For example, one event in the Petri net of FIG. 4 is “Text starts at movie screen position L1”. This event is associated with t2 for modeling.

  The condition and event concept is one interpretation of transitions and places as used in Petri net theory. As shown, each transition t1-t8 has a certain number of input / output locations representing the pre-condition and post-condition of the event. Since the event takes place, the preconditions need to be satisfied.

  In FIG. 5, for the Petri net as an example of FIG. 4, an outline of linked events and the conditions before and after are given. The precondition is described in column 1, the postcondition is described in column 3, and an event that links the preceding and following conditions is described in column 2.

  The Petri net of FIG. 4 is an example of a systematic flow of text that describes a small segment of a television or video program. Thus, the Petri net of FIG. 4 can be fully characterized as a “subordinate” Petri net. The present application utilizes an “upper” Petri net that is partially composed of “lower” Petri nets, as described below.

III. Third Stage—Restoring the High Level Structure of the Target Program The third stage, ie, the high level structure restoration stage 300 as shown in the flowchart of FIG. 3, generally includes the following steps.

310—Purpose: Restoring the high-level structure of the target program a—Structure of database of higher-level graphical model 310. b—Identify hotspots within each superior schematic model 310. c—Extraction of text detection results previously generated for the target program in step 140 (see FIG. 1)
310. d—Extraction of results of cluster distribution previously generated for the target program in step 160 (see FIG. 1)
310. Using the result of the cluster distribution of the e-target program, identification and extraction of a part of the higher-level graphical model from a plurality of higher-level graphical models stored in the database 310. f-Step 210. Using the part of the upper schematic model identified in e and the result of text detection, step 210. c. closest to the text detection event sequence of the target program extracted in step c. Identification of one high-level graphical model from part of the model identified in e. This single high-level graphical model schematically represents the high-level structure of the target program.

  Each of the above general steps is described in more detail.

  Step 310. In a, a plurality of higher-level schematic models (such as Petri nets) describing the systematic flow of video text are configured in the entire program. Each graphical model uniquely describes the flow of video text of a certain genre / sub-genre type. These multiple models are stored in a database for future reference in order to determine the genre / subgenre type of the target program that the user is interested in.

  In one embodiment, the graphical model is a manually configured upper Petri net. In order to manually construct such a model, the system designer analyzes cluster mapping and video text detection in the program for various program genres / sub-genre types.

  In another embodiment, the graphical model is automatically configured as a hidden Markov model using the Baum-Welch algorithm.

  Regardless of the manual or automatic configuration method, some key features of the high-level graphical model are: (1) the high-level graphical model models the program-level flow; (2) the schematic model is It has a transition that is an effective simplified representation of the sub-schematic model. In other words, the upper model is partially composed of the lower graphical model. This key feature is further illustrated with reference to FIG.

  FIG. 6 is an illustration of an upper Petri net that is one type of upper schematic model. The upper Petri net of FIG. 6 schematically shows the systematic flow of video text in the figure skating program. That is, it models a program level systematic flow. As is well known, the figure skating program consists of several program events as listed in Table II below.

前条件がイベントをトリガーするのに必要とされ、後条件がイベントの結果として発生する。本例での条件は、（条件ａ−プログラムがスタートした）、（条件ｂ−スケーターが紹介された）、（条件ｃ−スケーターのスコアが存在する）、及び（条件ｄ−最終順位が示される）、として規定されるかもしれない。

A pre-condition is required to trigger the event, and a post-condition occurs as a result of the event. The conditions in this example are (Condition a-Program started), (Condition b-Skater introduced), (Condition c-Skater score exists), and (Condition d-Final ranking). ), May be defined as:

  It should be understood that events 1-5 of the upper net in FIG. 6 are actually a simplified representation of the lower Petri net. For example, the first event, i.e., the start of credit, can be expanded as a lower Petri net such as that shown in FIG.

  Step 310. b-Within each superior graphical model configured in step 210a, several regions of interest (hot spots) may be identified. These hot spots may have a variable range. These hot spot areas correspond to events of particular interest to the end user. For example, event 2 “acting a skater” may have a higher importance as a program event of interest than the start of event 1 credits. A so-called “hot spot” may be assigned a rank order according to its relative importance. Furthermore, the lower Petri nets that make up the upper Petri nets may also be identified for so-called hot spots.

Step 310. c—Extraction of text detection results previously generated for the target program in step 140 (see FIG. 1)
Step 310. d—Extraction of results of cluster distribution previously generated for the target program in step 160 (see FIG. 1)
Step 310. e-Step 210. Step 210. Using the cluster distribution for the target program previously extracted at d. A portion of the high-level graphical model generated at a is identified and selected from the database. A part of the upper model is selected by determining which upper model has the same cluster as that specified for the target program.

  Step 310. f-Step 310. c. using the text detection data for the target program previously extracted in step c. One upper Petri net is specified from a part of the net specified in d. To identify one upper Petri net, the text detection data is compared with the systematic flow of each Petri net that is part of the Petri net to identify one Petri net that satisfies the text event sequence for the target program.

  As a result of identifying one graphical model that most closely approximates the high level structure of the target program, information about the target program may be easily obtained. Such information may include, for example, temporal events, text events, program events, program structures, summaries, and the like.

  As an example, program event information can be identified using text detection data from the target program along with one identified high-level graphical model. Table III represents the fictitious text detection data of the target program.

  As shown in the first row of Table III, the detected text event cluster type (column 1), text event occurrence time (column 2), text event duration (column 3), and text event must occur. Generate data about time boundary information that specifies the upper and lower limits of the time that must be reached. For simplicity of explanation, it should be understood that the table represents a greatly reduced text event sequence that occurs during the duration of the program.

ターゲットプログラムに関する情報が、テーブルＩＩＩに示されるように、テキスト検出データから直接抽出することが可能であるということが理解されるべきである。このような情報には、例えば、あるテキストクラスタタイプの出現回数、発生期間及び／又は発生時間などが含まれる。当業者は、テキスト検出データから抽出可能な他のデータの組み合わせを想起することができる。さらに、テキスト検出データがターゲットプログラムの構造を最も良く表す特定された上位図式的モデルと比較されるとき、プログラムイベントやプログラム構造などのターゲットプログラムに関する追加的情報が導出されるかもしれない。例えば、テーブルＩＩＩを参照するに、最初の３つの行は、以下の順序でテキストクラスタタイプの発生を記述している。すなわち、テキストクラスタタイプ１、それに続いて再びテキストクラスタタイプ１、それに続いてテキストクラスタタイプ２となる。このシーケンス又はテーブルからの他の何れかのシーケンスは、シーケンス｛１，２，２｝が図式的モデルのプログラムイベントを構成するか判断するため、ハイレベル図式的モデルと共に利用されてもよい。当該シーケンスがプログラムイベントを構成する場合、プログラムイベントはマルチメディアサマリに含めるため、特定の適用においては抽出されてもよい。｛１，２，２｝などの何れか選択されたシーケンスがプログラムイベントを構成するかに関する決定は、当該シーケンスがテーブルの第４列に指定される時間境界内に発生するかに基づくものとされる。この時間境界情報は、上位図式的モデルの一部として構成される時間境界と比較される。これの一例が時間処理されたペトリネットである。

It should be understood that information about the target program can be extracted directly from the text detection data as shown in Table III. Such information includes, for example, the number of occurrences, occurrence period, and / or occurrence time of a certain text cluster type. One skilled in the art can conceive of other data combinations that can be extracted from the text detection data. Further, when the text detection data is compared with the identified high-level schematic model that best represents the structure of the target program, additional information about the target program, such as program events and program structure, may be derived. For example, referring to Table III, the first three lines describe the occurrence of a text cluster type in the following order: That is, text cluster type 1, followed by text cluster type 1 again, followed by text cluster type 2. This sequence or any other sequence from the table may be used with the high-level graphical model to determine if the sequence {1, 2, 2} constitutes a graphical model program event. If the sequence constitutes a program event, the program event may be extracted in a particular application for inclusion in the multimedia summary. The decision as to whether any selected sequence such as {1, 2, 2} constitutes a program event shall be based on whether the sequence occurs within the time boundary specified in the fourth column of the table. The This time boundary information is compared to a time boundary that is configured as part of the higher graphical model. An example of this is a time-processed Petri net.

  It will be understood that the embodiments and variations shown and described herein are merely illustrative of the principles of the present invention and that various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention. Should.

  In interpreting the appended claims, a) the term “comprising” does not exclude the presence of other elements or steps than those listed in a given claim, and b) precedes an element. The term “a” does not exclude the presence of a plurality of such elements, c) any reference signs in the claims do not limit the scope thereof, and d) a plurality of “means” It can be expressed by a structure or function realized by the same item, hardware or software, and e) each disclosed element is a hardware part (such as each electronic circuit), a software part (such as computer programming), or these It should be understood that any combination of these may be used.

FIG. 1 is a flow diagram illustrating text type clustering steps of the present invention according to one embodiment. FIG. 2 is a flow diagram illustrating the genre / sub-genre identification stage of the present invention according to one embodiment. FIG. 3 is a flow diagram illustrating the high level structure restoration stage of the present invention according to one embodiment. FIG. 4 is an exemplary schematic model showing a program event for a movie. FIG. 5 is an overview of the conditions before and after the graphical model of FIG. FIG. 6 is an example of an upper Petri net.

Claims (22)

A method for restoring the high-level structure of a target program,
a) generating text detection data for the target program;
b) generating a genre / sub-genre feature vector for the target program using the text detection data generated in step (a);
c) constructing a plurality of super schematic models;
d) identifying a portion of the upper graphical model using cluster distribution data of the target program;
e) using a text detection data of the target program to identify a single superior graphical model from a portion of the model;
Have
The method, wherein the single high-level graphical model corresponds to a high-level structure of the target program. The method of claim 1, further comprising:
A method comprising the step of constructing a program summary using the single superior graphical model with the text detection data. The method of claim 2, comprising:
The step of configuring the program summary further includes:
Determining one or more events important to the viewer;
Retrieving the important event from the text detection data;
Extracting the important event from the text detection data;
Including the extracted event in the program summary;
A method characterized by comprising: The method of claim 1, further comprising:
Configuring the program summary using the single superior schematic model with the text detection data;
The configuring step includes:
Searching for program events;
Ranking program events identified in the searching step based on a predetermined ranking;
Selecting a portion of the identified program event based on the ranking;
A method characterized by comprising: The method of claim 4, comprising:
The step of searching for the program event includes:
Determining a text event sequence that collectively defines program events;
Retrieving the text event sequence from the text detection data;
Identifying the text event sequence in the text detection data, comparing the text event sequence to a corresponding node of the higher graphical model;
Determining whether the occurrence time sequence of the text event sequence obeys time constraints associated with corresponding nodes of the high-level graphical model;
A method characterized by comprising: The method of claim 1, further comprising:
A method comprising: searching for information having a text type, a similarity with a program other than the target program, a text pattern, a program event, and a program event pattern in the target program. The method of claim 6, comprising:
The method according to claim 1, wherein the information to be searched in the target program uses information provided by the text detection data and the high-level schematic model. The method of claim 1, comprising:
The graphical model is one of a Petri net model, a hidden Markov model, and a combination of the Petri net model and the hidden Markov model. The method of claim 1, comprising:
The method of claim 1, wherein the target program is one of a television and a video program. The method of claim 1, comprising:
Generating text detection data for the target program comprises:
i) detecting the presence of text in the target program;
ii) identifying and extracting text features of the detected text;
iii) forming a text feature vector from the identified and extracted features;
A method characterized by comprising: The method of claim 10, comprising:
The method of detecting the presence of text in the target program is performed according to the MPEG-7 standard. The method of claim 10, comprising:
The method wherein the identified and extracted text features include text position, text height, text font type, and text color. The method of claim 10, comprising:
The method of detecting the presence of text in the target program further comprises detecting the presence of text in a video feature of the target program. The method of claim 10, comprising:
Generating a genre / sub-genre feature vector for the target program;
comparing the text feature vector for the target program generated in iii) with a plurality of predetermined genre / sub-genre feature vectors for various genre / sub-genre types;
Associating a text feature vector for the target program with a genre / sub-genre feature vector having the highest similarity;
Defining a genre / sub-genre feature vector group identified in the step of associating text feature vectors for the target program;
A method characterized by comprising: The method of claim 1, comprising:
The plurality of upper graphical models graphically model a program genre / sub-genre type at a program level. The method of claim 12, comprising:
The transition element of the upper graphical model can be constructed from a lower graphical model having program text and timing information. The method of claim 16, comprising:
The sub-graphical model is modeled as a Petri net. The method of claim 17, comprising:
The transition element is assigned a priority with respect to other transition elements of the higher model. The method of claim 1, comprising:
The method of generating genre feature vector cluster data for the target program is performed according to an unsupervised clustering algorithm. 20. The method of claim 19, wherein
The unsupervised clustering algorithm compares corresponding text features based on a distance metric. 21. The method of claim 20, wherein
The distance metric is

A method characterized by calculating as: A system for restoring the high-level structure of the target program,
A memory for storing computer readable code;
A database for storing multiple upper Petri nets;
Operatively coupled to the memory, generating text detection data for the target program, generating a genre / sub-genre feature vector for the target program using the text detection data; A graphical model is constructed, the cluster distribution data of the target program is used to identify a part of the higher-level graphical model, and the text detection data of the target program is used to identify a part of the model. A processor configured to identify a high-level graphical model;
Have
The system, wherein the single high-level graphical model corresponds to a high level structure of the target program.

Images