KR100319160B1 - How to search video and organize search data based on event section - Google Patents

Abstract

The present invention relates to a system for retrieving a moving picture, and more particularly, to a system for retrieving moving picture content for each event section by expressing a content of a moving picture such as a movie or a drama in a form of a plurality of events. It is expressed as an entire story line consisting of case paragraphs representing unit events, divided into actual video sections for each event represented by one unit event paragraph, and summarized and searched for the contents of the video by divided sections. The present invention relates to a video retrieval method and a retrieval data construction method based on the.

Conventionally, techniques for searching video and summarizing video contents are mostly expressed using a key frame or other existing content-based browsing techniques.

The present invention expresses the contents of a video such as a movie, a drama as an entire plot, wherein the entire plot is expressed in the form of an event paragraph for each unit event according to the contents of the drama or movie, and for each event corresponding to each unit event By dividing the actual video section, each divided section provides a video search method and a search data structure based on the event section so that the search method including the existing browsing method can be appropriately applied and searched.

Description

Video retrieval method and retrieval data composition method based on event section

The present invention relates to a system for retrieving a moving picture, and more particularly, to a system for retrieving moving picture content for each event section by expressing a content of a moving picture such as a movie or a drama in a form of a plurality of events. It is expressed as an entire story line consisting of case paragraphs representing unit events, divided into actual video sections for each event represented by one unit event paragraph, and summarized and searched for the contents of the video by divided sections. The present invention relates to a video retrieval method and a retrieval data construction method based on the.

Conventional techniques for video retrieval and video summary are mostly expressed using key frames or other existing content-based browsing techniques.

Therefore, it is not only difficult and complicated to summarize and search the entire contents of a movie or drama with a few extracted keyframes or character-oriented content-based browsing techniques, but it also leads to ambiguity in expressing the contents from a few representative scenes. Difficult to present and retrieve accurate information.

In particular, since the entire story of a movie such as a drama or a movie has a characteristic that the entire story proceeds by mutual coupling of unit events, the key story or character center is not presented without showing a close coupling relationship between such unit events. Was forced to perform a search of.

As a conventional video retrieval technique, a technique of summarizing and retrieving the entire contents by extracting keyframes from a video and representing a temporal relationship between each keyframe is presented. (US Patent No. 5,708,767: Method and apparatus for video browsing) based on content and structure).

However, this method can be used for simple video files, but there is a limit to applying it to a large amount of video files in which unit events are closely related to each other and change over time such as dramas and movies.

In other words, in case of a huge amount of video, the number of key frames and the relationship between them becomes much larger, so it is not only difficult to express such a large number of key frames and relationships, but also if the user expresses the graph Even understanding itself becomes hard.

Conventional video retrieval technology provides a keyframe based video summary and retrieval function (A Distribute System for Editing and Browsing compressed Video Over the Network; Horace J.Meng, IEEE Signal Processing Society 1997 Workshop on Mumtimedia Signal Processing) , New Jersey).

This technique is hierarchical, depending on how long the keyframes are extracted from, which means that the keyframes are extracted from the larger sections at the top and the number of keyframes is extracted from the smaller sections. It consists of a tree structure.

However, this structure has a problem that it is difficult to browse the entire contents in the representative scene, and also it is difficult to find the desired part only by the key frame because the key frame of the upper layer is extracted in a too large section.

The present invention expresses the contents of a video such as a movie, a drama as an entire plot, wherein the entire plot is expressed in the form of an event paragraph for each unit event according to the contents of the drama or movie, and for each event corresponding to each unit event By dividing the actual video section, this divided section provides the structure of the video search method and the search data based on the event section so that the search method including the existing browsing method can be applied and searched appropriately.

The present invention presents a video event such as a movie or drama in a unit event paragraph (text or in the form of a text and a representative scene) which can be grasped at a glance from the mutual connection of unit events according to the content development, and based on such an event section By summarizing and retrieving video contents, it can be applied to a large amount of videos compared to the conventional retrieval system using only keyframes or representative scenes, and can summarize and search video contents simply and easily. To provide a video retrieval method and retrieval data structure based on an event section.

1 is a block diagram showing an embodiment of a video production and searcher according to the present invention;

FIG. 2 is a diagram illustrating a browser in which a unit event in the entire plot is expressed in a paragraph, and the relationship of the events is represented graphically by key frames and the temporal relationship between them.

FIG. 3 is a diagram illustrating a browser expressing a unit event in the entire story in paragraphs and a keyframe representing a scene representing the unit event in the present invention.

4 is a diagram illustrating a browser expressing a unit event in the entire plot in the present invention as a paragraph, and a relationship between a person and a place corresponding to the unit event and a representative scene.

5 is a view showing a browser showing a representative scene for each case paragraph constituting the plot in the present invention;

6 illustrates the structure of search data for the implementation of FIGS. 4 and 5;

7 and 8 are diagrams illustrating a browser for expressing a unit event in an entire story in a paragraph and explaining multiple levels of a person and a place corresponding to the unit event in the present invention.

The present invention is composed of a set of case paragraphs representing one unit event by dividing the entire plot of the video content by the unit case as a meaningful event unit constituting the video, and indicates a video section corresponding to each case paragraph A video retrieval data structure based on an event section, characterized in that the information is made by setting the information, wherein the case paragraph representing the unit event may be composed of text only or a representative scene with the text, the unit event The maximum may be content of the entire video, and the minimum may be a change in the frame level.

The present invention is composed of a set of case paragraphs representing one unit event by dividing the entire plot of the video content by the unit case as a meaningful event unit constituting the video, and indicates a video section corresponding to each case paragraph Search information setting process for setting information, and if a random unit event is selected from the unit event paragraphs according to the set search information, the search information display process expresses information on the corresponding video section in response thereto. It is a video search method based on an event section.

In addition, the present invention may be expressed in the form of an event paragraph indicating the unit event in the form of text only, or in the form of text and other elements to help understand the contents of the unit event, such as the representative scene in the unit event section. .

In addition, the present invention can be applied to the existing keyframe arrangement method, the scene transition (Scene Transition) method, the search information display process, by presenting a graph of the relationship between the object (Object) and the place (Place) by the object and place You can also apply the search method from the relation with.

Figure 1 shows one embodiment of an overall block diagram of the video retrieval system of the present invention and the present invention is not limited by the illustration.

In FIG. 1, for convenience of explanation, the video provider and the user are separated.

The video provider provides search information as shown in Figs. 2 to 6 through the process of constructing the event relationship-oriented video search data of the present invention, and the user uses the search information as shown in Figs. Event-driven video search will be available.

The digital video data 101 is stored in the input video buffer 102, the digital video data stored in the video buffer 102 is controlled by the video processing unit 103 as a file of the multimedia video stream, and the control processing unit ( 104) produces each piece of information required or reads and processes the information.

The user interface and display unit 105 provides an interface for displaying the video data processed by the video processing unit 103 and the control processing unit 104 as well as inputting user information for producing search data.

Corresponding information of the video data retrieval data structure by event section processed by the control processing unit 104 is stored in the video information processing memory 106, and the video data stored in the video buffer 102 is stored in the video information processing memory. Together, it is stored in the storage space 107 as an appropriate format.

The video data and search related information stored in the storage space 107 are provided to the user in the form of a recording medium or through a wired or wireless broadcasting network.

The user side detects and reads video data provided from the storage space 107 and search data for each event section, and responds to a user request.

That is, the data processing of the video processing unit 109 and the control processing unit 110 is performed on the video data stored in the video buffer 108, and at this time, the video data processing memory 112 uses the event data for each event section. And process the response to the user interface and the display unit 111.

That is, according to the event relationship-oriented video retrieval system shown in Fig. 1, the provider side completes the data structure as shown in Figs. 2 to 6 through the user interface and the display unit 105, and stores the completed information in an appropriate storage space. The data is stored together with the data, and the user side performs a video search as shown in FIGS. 2 to 6 through the user interface and the display unit 111.

A video search method based on the event section of the present invention implemented in the video search system as described above, and a browser according to a search data structure is a collection of event paragraphs expressing unit events in video content such as movies or dramas. If a case is selected from the set of case paragraphs, the representative scene of the video section corresponding to the unit event is displayed in the form of a graph, keyframe or object-place according to time and event development. Include a part to show as a relationship.

Therefore, the event relationship-oriented video search data as described above is produced in the following procedure.

First, the content of the video is shown as a plot of a series of events for the entire video such as drama or movie.

The storyline of a series of events consists of one or more paragraphs, each of which is described in the form of adding a representative scene with text or text.

Here, the meaning of each case paragraph is part of the video, so the section shown by the section becomes a section within the video, and the section becomes a unit event section of the video.

Therefore, a partial section of the video is divided by each case paragraph, and the divided video section information is added as a frame number, for example, and the existing keyframe type arrangement method or the scene transition method is applied to each divided section. Alternatively, a section desired by the user is searched by applying a search method from the object-place relationship to be proposed in the present invention.

As such, the results of applying the content browsing method to the unit event can provide very clear search results because the content of the unit event is browsed.

Such information may be included in a header of a multimedia stream as an example.

Then, the video reproducing (searching) system detects and reads the information contained in the header of the multimedia stream, and can perform event-oriented content search for the video.

2 illustrates an example of an event relationship-based search browser implemented through the above steps.

The video story browser 200 has a story display unit 201 in which a set of event paragraphs 201a are expressed for each unit event. In this story display unit 201, an event paragraph 201a corresponding to an arbitrary unit event may be selected. In this case, the key frame unit 202 for displaying the contents of the video section of the unit event as a transition graph in which a representative scene (Scene1-Scene4) is connected by a link is presented.

3 shows another example of an event relationship-based search browser implemented through the above steps.

The video story browser 300 has a story display unit 301 in which a set of event paragraphs 301a are expressed for each unit event, and an event paragraph 301a corresponding to an arbitrary unit event has been selected in the story display unit 301. In this case, a key frame (Scene1-Scene7) and a key frame unit 302 for displaying a description of each key frame are provided.

FIG. 4 illustrates an example of a browser in which a video content is expressed in a plot composed of a set of event paragraphs per unit event, and the content in each event section is expressed as a relationship between an object (character in this case) and a place.

That is, the video story browser 400 has a story display unit 401 in which a set of event paragraphs 401a are expressed for each unit event, and when a random unit event paragraph is selected in the story display unit 401, the characters in the section. And a key frame portion 402 for displaying a relationship graph between the location and the place and a representative scene (key frame) corresponding to the relationship selection.

That is, the key-frame portion 402 displays the character-place relationship graph screen 403 and the representative scene portion 404 corresponding to an arbitrary relationship.

For example, when the story display unit 401 selects an event paragraph 401a corresponding to an arbitrary unit event, the person (human A, B, C) appearing in the unit event section includes the contents of the video section of the unit event. (405, 406, 407) and a place where each person appears as a relationship between the hospital 408, the room 409, and the outdoor 410, and among them, the relationship 411 between the person B 406 and the hospital 408. If you select, the representative scenes for the section in which the person B appears in the hospital are displayed as key frames (Key Frame 14, Key Frame 17, Key Frame 1).

Thus, a brief description will be given of a video retrieval technique based on the relationship of an object (a person): a place for retrieval in a video section corresponding to an event paragraph of each unit event.

First, the main objects and places are set for the contents of a movie such as a movie or a drama. For example, as the main objects, the main characters necessary for the development of the event among the characters are set. Set the place,

Next, the relationship between the set main object and the place is set.

For example, if character A, character B, and character C are set as main objects, and hospital, room, and outdoor are set as places, the place where character A appears (relevant) is hospital, room, character B. Place is related to room, outdoor, and character C is related to the place and character.

Since this relationship will actually be the section in which the object appears in that place in the video, it is indexed.

Next, as a setting phase of the representative scene, a scene that can represent the appearance section of the object related to the place from the relation between the set object and the place is set, and the frame position is set to one representative frame. To display.

According to the above example, from the relationship between the character A-hospital, the character A represents a section in which the character appears in the hospital, and sets a 'surgical scene frame' and displays the position of the frame.

Next, an appropriate comment is attached to each of the appearance sections obtained in the previous steps.

According to the above example, the operation scene frame is set as the representative scene from the relationship between the character A-hospital, and a description to summarize the content of the appearance section is added as text.

As described above, the relationship between the main object and the place is set based on the contents of the video, and the indexing of the appearance section, the representative frame setting of the appearance section, and the annotation are performed (events). Basic information of video search based is completed.

Such information may be included in a header of a multimedia stream as an example.

Then, the video reproducing (searching) system can detect and read the information contained in the header of the multimedia stream and perform a content-based search for the video.

That is, if a random relationship 411 is selected from the set character: place relationship graph, the person B 406 presents the representative scene of the corresponding section using the section information appearing in the hospital 408, and the representative When one scene is selected from the scenes, the subdivisions represented by the scenes are shown so that the contents of the video can be summarized and searched based on the relationship between the object (eg, person) and the place.

5 is a view showing a browser showing a representative scene for each case paragraph constituting the plot in the present invention.

That is, the video story browser 500 has a story display unit 501 in which a set of event paragraphs per unit event and a key frame representing the unit event are represented together. When the unit event paragraph is selected, it has a key frame unit 502 for displaying a graph of the relationship between the characters in the section and a place and a representative scene (key frame) corresponding to the relationship selection.

That is, the key frame unit 502 displays the character-place relationship graph screen 503 and the representative scene unit 504 corresponding to an arbitrary relationship. The search method and the display of the search result are similar to those of FIG. same.

However, in FIG. 5, the selection and display of the representative scene according to the relationship between the person: place and the selection of the relationship are performed only by selecting the corresponding key frame instead of the unit event paragraph.

Fig. 6 shows an example of applying the person-place relationship graph as described above as an example of a search data structure for applying the present invention.

Reference numeral 601 denotes an entire plot, in which case paragraphs 601a are configured for each unit event.

Reference numeral 602 denotes a scene represented by the event section for each unit event, and displays the key frames (positional information) and the actual video section meant for each case paragraph as the link 604.

Reference numeral 603 denotes a portion including a content browsing structure in each section corresponding to the video section information 602, that is, a relationship graph structure of person 603a, 603b-place 603c, 603d, and as a link 605. The relationship information of the person-place corresponding to the unit event section is presented.

In this case, the subordinate (dependent) additional information 603b is added to the person 603a as the relationship information of the sub-layer subordinated to the person, and the relationship information of the sub-layer subordinated to the place also 603d for the place 603c. ) Was added.

This is to relate sub-class information such as 'patient room', 'operating room', 'hospital corridor', etc. to a place called 'hospital'.

7 and 8 are diagrams illustrating an example of a user interface for describing multiple levels of a place in the present invention as described above.

First, in FIG. 7, the search browser 700 has a relationship display unit 701 of an object and a place, and a story display unit 702 in which a set of event paragraphs are expressed for each unit event (in this story display unit, a unit event as shown in FIG. 5). A representative scene (key frame) representing the event may be displayed together), and the relationship display unit 701 of an object and a place corresponding to a paragraph of a unit event displays a graph of a relationship between an object and a place 703. And a key frame 704 corresponding to the selected relationship and a description 705 of the event contents of the section.

In Fig. 7, three characters (Character A, B, C) are presented as objects, and a hospital (Hospital), a character's house (A's house), and a park (The park) are presented as places.

In addition, character A is related to the hospital and the park, character B is related to the hospital, and character C is related to the house of the character A.

In this case, when the relationship between the character B and the hospital (706) is selected, the representative scene of the section in which the character B appears in the hospital is presented as a key frame 704, and the event description (705) for the section is presented together. Seems to be.

Meanwhile, as described above, the display unit-extension button marker 707 indicates that there is a level corresponding to the lower hierarchy of the place called 'hospital'.

This extended button marker may be represented as a menu.

Therefore, when the user selects the extended button marker 707, the object-place relationship is presented at the level corresponding to the lower hierarchy of the place called 'hospital' as shown in FIG.

That is, in Fig. 8, the search browser 800 also includes the object-place relationship display unit 801, the story display unit 802, the object-place relationship graph display unit 803, the key frame 804 corresponding to the selected relationship, and the contents of the event. With the explanation (805), the lower level of the lobby of the hospital and the room of the hospital is presented for the place of the upper layer called 'hospital'. The relationship with character B (806a, 806b) is presented for the level of.

Therefore, the user can select the relationship 806a between the character B and the hospital lobby or the relationship 806b between the character B and the sickroom from the search browser of FIG. 8, and the key frame 804 corresponding to the selected relationship and the description thereof. 805 is presented as a search result.

In addition, if the button marker 807 is placed to group the hospital lobby and the hospital room as shown in the figure to return to the upper level hospital, the button marker 807 is returned to the search screen of FIG.

As described above with reference to FIGS. 1 to 8, the present invention was able to retrieve the contents of a video from the relationship between unit events.

1 to 8, one video file, that is, one drama or one movie has been described.

However, by applying the concept of the event section based video retrieval system of the present invention, in the video retrieval browser of the present invention; All the video files are represented by a set of event paragraphs representing a single unit event, and the structure of each video file is represented by a set of event paragraphs representing a unit event. A video retrieval system that performs detailed retrieval through a coarse retrieval from the case to each video file from.

For example, even if a series is divided into several video files, the unit events are closely related to each other due to the nature of the series, and thus, a plurality of video files are usually contiguous in content development. The content of each video file is regarded as a single video file and represented by the event section based search structure of FIGS. 1 to 8, and the content of each video file is added to the event section based search as shown in FIGS. 1 to 8. By expressing more details in the structure, it becomes possible to roughly search the entire contents of a plurality of video files and to search in detail about another video file.

The present invention provides a video search method and a search data structure based on an event section.

According to the present invention, since the entire plot of a drama or a movie is expressed in the form of a set of unit events constituting the drama or a movie, the video story development is simply expressed as a representative scene like the existing keyframe-based content summary and retrieval method. Constraints can be removed, and the content of a drama or movie can be summarized and searched at a glance from the combination of unit events and unit events.

In addition, the entire storyline of a drama or movie is composed of event paragraphs for each unit case, so that the sections are divided by each event meaning each paragraph and the content is browsed within the sections, so that the entire contents of the conventional method are key. Using frames or other browsing methods to solve the problem of complexity and difficulty of the search tool, and summarize and search the video content easily and easily.

In addition, the present invention can be applied to the type of video, such as drama, movie, so that the user can search for the desired content by unit events, the form of the partial storyline of the story based on the entire storyline with the context of the story development story Can be summarized and searched.

In addition, the present invention has a structure in which the contents of the plurality of video files are expressed in a set relationship of unit events, and the structure in which the contents of each video file are expressed in a set relationship of one unit events, The search can be performed in a rough manner from the entire paragraph, and the contents of one video file can be retrieved in detail from the relevant paragraph. Therefore, a batch search of a plurality of video files is possible.

Claims (9)

The entire plot of the video content is composed of text information, and the information is divided into paragraphs, and the event paragraphs are classified into case paragraphs, and the classified case paragraphs are arranged in temporal order. Configure information for indicating a video section corresponding to each of the case paragraphs, and set information for summarizing the event contents for each unit event corresponding to the video section, and configure search data for searching for a video. How to construct a video search data based on the event interval, characterized in that. The method of claim 1, wherein the event paragraph representing the unit event further comprises a scene representing the unit event. The entire storyline of the video content is composed of text information, and it is divided into event paragraphs as a meaningful event unit constituting the video by classifying paragraphs from this information, and the event paragraphs are arranged by arranging the classified event paragraphs in a chronological order. A search information setting process of setting information indicating a video section corresponding to each case paragraph, and setting information indicating a video section corresponding to each case paragraph; In response to selecting a unit event in the unit event paragraph according to the set search information, a search for expressing information set in the corresponding video section using information indicating a video section corresponding to the selected unit event in response thereto. Video retrieval method based on the event section characterized in that the information display process. 4. The method of claim 3, wherein the set of event paragraphs representing the unit event is set as an event unit constituting a plot of the entire video files, and is set as an event unit constituting the plot of each video file. Performs event section-based content retrieval for a plurality of video files from event paragraphs for the entire video file, and performs event section-based content retrieval for each video file from event paragraphs for each video file. Video search method based on the event section, characterized in that. [5] The method according to claim 3 or 4, wherein when a part corresponding to the above case paragraph is selected, the content of only the section corresponding to the unit event is expressed by an arbitrary browsing method. How to search for videos. The method of claim 2 or 4, wherein when the part corresponding to the above case paragraph is selected, the content of only the section corresponding to the unit event is set in a keyframe arrangement method, a scene transition graph, or a person-place relationship graph. Video search method based on the event section, characterized in that represented by. The entire plot of the video content is expressed as a set of event paragraphs representing unit events and displayed on a predetermined designated screen area, and when a user selects any unit event paragraph from the displayed unit event paragraphs through the user key interface, In response, the information indicating the video section set in the case paragraph is read to find the corresponding video section, and based on the event section, the information set for the video section of the case paragraph is displayed in a predetermined area of the screen. Video search browser. The video search browser according to claim 7, wherein the information about the video section is expressed by an arbitrary browsing method. The video search based on the event section according to claim 7 or 8, wherein the information about the video section is expressed in a set form of a keyframe arrangement method, a scene transition graph, or a person-place relationship graph. browser.