KR100370249B1 - A system for video skimming using shot segmentation information - Google Patents

Abstract

The present invention relates to a video skimming method in which shot segmentation can be used to understand the contents of the entire video in a short time, and the user can navigate the video stream to a desired position by quickly searching the video stream.

According to the present invention, skimming is performed by playing only a part of the shot (front part, rear part, center part, or a combination thereof) based on the shots present in all videos, and skipping the remaining part. In the skimming method of the present invention, the length of a portion to be reproduced in each shot is the same for all shots, or different lengths for each shot based on video similarity (average image / motion / audio similarity) in the shot. Play with In addition, the present invention continuously reproduces segments selected as sections to be played in each shot, or decodes only some frames (eg, I frames in MPEG) in the section to be skipped and plays back in the section.

Description

A system for video skimming using shot segmentation information}

The present invention is a system for multimedia search and browsing, and in particular, using video shot segmentation information, a video skimming system is provided to allow a quick understanding of the entire content of the video and quick movement to a desired part (location). skimming) method.

In particular, the present invention is to skim the content of the video based on the shot made by video segmentation, and to play the video skimming through the partial playback in which the continuous playback or skip technique of a specific portion in the shot is introduced. The present invention relates to a video skimming method that enables the user to fully understand the content of the entire video in a short time, and allows the user to quickly navigate to a desired position by quickly searching the video stream.

With the development of mass media and the ease of production of multimedia contents, the amount of media that the public encounters every day has become very large. As the multimedia content is enormous, there is a demand for an automated system that selects data desired by the user, and researches on ways to solve the problem have been actively conducted. In particular, with the development of digital technology, video contents are being stored and distributed in digital form, and digitalization of such media will be accelerated when digital broadcasting becomes popular.

For such digital video content, some users may only want to watch sports-related news in the news, while other viewers may only want securities-related news in the news. Some users may also be required to watch only scenes in which a particular person appears in a show program. Various studies have been actively conducted to accommodate the needs of various users.

In addition, the user has a desire to grasp the entire contents of the video content within a limited time. It is a "highlight" to accommodate this need. In general, highlights can be understood as newly formed content by gathering important scenes from video content, which can be accessed through "sports highlights", "movie trailers", "headline news", and the like. However, extracting highlights from video content is a very difficult area to automate with current technology. Therefore, it is common to rely on the manual work of most people. As mentioned above, it is almost impossible to manually provide highlights for all video content as the amount of media explodes, which requires a lot of manpower. Therefore, there is a need for an automated system that allows users to quickly understand the contents of the sketch.

With the development of digital technology, key frames are used for use when the user wants to move to a desired position in video content. Using video summaries using keyframes has the advantage of allowing the user to quickly move to the desired location. However, a large number of key frames are required to easily find a user's desired location using key frames, and it is very inconvenient to require a large number of selection tasks for the user because it is difficult to display a large number of key frames in a limited display space. In addition, the method using a key frame generally has a disadvantage that it is difficult to understand the content of the entire video.

Recently, various video indexing techniques have been studied to find desired scenes in digital video. For users who only want to see a scene from a specific person, index the information about the person's appearance in the video by extracting the scene from the video and recognizing who the person is. And researches on indexing them are underway. However, video has various genres, and data to be indexed by genres is very different, and it is known that it is difficult to implement an automation system that can extract information meaningful to users with a high level of accuracy with current technology.

On the other hand, unlike in analog video, it is possible to prevent the deterioration of image quality in performing the fast forward / fast reverse function in digital video.

The fast playback method commonly used in digital video is to increase the number of frames to be decoded per unit time to display only a part of the frames or to skip and decode the frames.

However, the method of increasing the number of frames to decode per unit time has a disadvantage in that the maximum speed is affected by the performance of the terminal device. Therefore, in the case of fast forward / fast reverse of digital video, a method of decoding and displaying frames by skipping a portion is generally used. Use Fast-forward / fast-rewind technology in digital video is the most reasonable alternative to responding to the needs of users who want to grasp the entire contents within a limited time or move to a desired position, but most of the skipping part Since the user uses a constant interval in time, the user misses the scene of the part that he / she wants, or a relatively insignificant part plays a disadvantage.

In the present invention, in response to the user's request to grasp the contents of the entire video within a limited time or to move to a desired position in a digital video environment, relatively insignificant portions of the video are reproduced or the user actually We propose a method of constructing an automated video skimming system that minimizes the probability of missing a desired scene.

In addition, the present invention is characterized by performing video skimming based on the video segmentation technology that can be automated with high accuracy as known, and by performing video skimming using the shot segmentation information of the video, We propose a video skimming method that enables users to move quickly to a desired location while providing a search and browsing environment that is summarized enough to understand the contents of a video without watching all of them.

1 is a diagram for explaining a concept of shot segmentation;

2 is a view for explaining the concept of a skimming method using shot segmentation information of the present invention;

3 is a diagram illustrating various examples of a method of selecting a portion to be played back and a portion to be skipped in video skimming.

4 is a diagram illustrating an example of a dynamic unit reproduction length selection method using dissimilarity characteristics of a shot;

5 is a view for explaining a high-speed skimming method using skipping

6 is a diagram illustrating an example of a system configuration for video skimming using shot information;

The present invention provides a method of recognizing a segment of an individual shot from shot segmentation information of a video stream, selecting a specific portion within the recognized individual shot segment as video information to be reproduced reflecting contents of the shot, and each individual segment. A video skimming method using shot segmentation information comprising the step of continuously playing the video information selected for each shot.

In addition, in the present invention, the shot segmentation information is characterized by displaying shot information, which is a production and editing unit of a video stream, together with temporal information such as a start position and a duration, or a start position and an end position.

In the present invention, the selection of the portion to be reproduced in each shot is characterized by using the first half, the second half, the middle, the first half and the second half of the shot at the same time.

Also, in the present invention, the determination of the length to be reproduced in each shot is performed by selecting and reproducing segments of the same length in each shot, or reproducing if the similarity is high based on the average value of the image / motion / audio similarity in the individual shots. If the length is reduced and the similarity is low, the length to be played in the shot is increased.

Also, in the present invention, the image / motion / audio similarity in the shot may be similarity between frames, motion vectors, and audio data having different temporal positions in the shot.

In the present invention, when the length of the segment selected as the portion to be reproduced in the individual shot is calculated to be larger than the length of the shot, the length of the portion to be reproduced in the individual shot is reduced to less than or equal to the length of the shot.

Also, in the present invention, the sections to be selected and reproduced for the video skimming are reproduced at a normal video playback rate, or the number of frames to be decoded per unit time is higher than normal to reproduce the playback section at high speed, or the frames within the section to be played back. It is characterized in that the playback section is reproduced at high speed by skipping and reproducing every few frames without decoding all of them.

Also, in the present invention, when the fast skimming using the skip is applied to a video stream using a coding scheme using an interframe compression such as MPEG, a frame to be decoded can obtain frame data by decoding only the corresponding frame without decoding another frame. Characterized in that I frame.

A video skimming method using the shot segmentation information of the present invention made as described above will be described in more detail with reference to the accompanying drawings.

With the development of digital video technology and the development of image / video recognition technology, users can search / filter and browse only the desired part at the desired time of the desired video.

The most basic techniques for non-linear video browsing and retrieval are shot segmentation and shot clustering, both of which are used to analyze video. Is the key technology. Therefore, many studies are focused on shot segmentation until now, and research on shot clustering technology is being started.

Based on various studies, shot segmentation can be automated and most algorithms can be implemented with high accuracy of over 90%.

Shot refers to a sequence of video frames obtained from one camera without interruption, which is the most basic unit for analyzing or constructing video.

In general, video consists of a series of shots. Shot segmentation refers to a technique of dividing a video into individual shots, and FIG. 1 illustrates the shot segmentation process. In general, most shot segmentation algorithms are based on the property that image / motion / audio similarity exists within the same shot and that image / motion / audio dissimilarity is measured between two different shots.

In general, video highlighting is a method of selecting segments that are semantically important in the content development of a video stream and continuously playing these segments.

However, it is very difficult to automate the selection of semantically important segments in content deployment for various video content.

The present invention proposes a skimming method to reproduce a shorter length than the original video stream by playing only a portion of the shot and skipping the remaining portion based on the shots present in all the videos.

Such a skimming method of the present invention has the advantage that it is possible to build a fully automated system because the shot segmentation technology can be automated, and the problem that occurs during fast forward / fast reverse for general digital video, that is, a non-critical scene is reproduced for a long time. Or to miss important scenes.

2 is a diagram summarizing the concept of the present invention. In FIG. 2, gray portions indicate portions to be reproduced in the skimming method using shot segmentation information, and others indicate portions to be skipped.

In the video skimming method of the present invention shown in Fig. 2, how to select the portion to be played back and the portion to be skipped within each shot, how to select the playback length of the portion to be played back, and the playback method within the playback section Consider what to do.

First, a method of selecting a portion to be reproduced and a portion to be skipped in each shot will be described.

The method of selecting a portion to be played in each shot can be unconditionally selected as the front part of the shot, the rear part, or the middle part. FIG. 3 is a diagram illustrating a portion to be played back and a portion to be skipped in a video skimming method using the front part, the rear part, the middle part, or the front / rear part of the shot simultaneously.

However, although there are differences depending on the genre of the video, the results of the experiment show that skipping the front part of the shot and playing the back part generally shows a high level of user satisfaction. The reason for this is that the ending part of the shot (e.g. soccer goal scene, etc.) is more important than the introduction or development part in using the content of the shot, and when using a method such as step-by-step charting in a program such as news. This is because part of the content is expressed at the beginning of the shot and the entire content is expressed at the end.

However, depending on the genre of the video, the first part of the shot is generally important. For example, an education broadcast mainly for problem solving is used. In these broadcasts, there is information about what problems are dealt with at the beginning of the shot, and after that, problem solving is followed. To play the part you want, playing the front part of the shot gives more information to the user than playing the back part. Can provide.

Therefore, according to the present invention, the position to be played in the shot may be differently selected according to the genre of the video, and the skimming may be implemented by using the front part, the middle part, and the rear part in the same shot.

The following describes a method of selecting a length to be reproduced in the present invention.

The method of selecting the length to be played in each shot can be divided into a method of selecting segments of the same length for all the selected shots and a method of differently selecting the length of playback for each shot by using the characteristics of the shots. .

The characteristics of the shot used at this time may be based on the average image / motion / audio similarity within a shot. In other words, the larger the image, motion, and audio similarity within a shot, the more likely the scene is monotonous. Therefore, the scene is skipped a lot, and the smaller the image, motion, and audio similarity within a shot, the more complex the content. Since it can be determined as a scene, the length of the unit segment to be reproduced can be dynamically adjusted by using a method of reducing the skip for such a scene.

This method does not depend on the time length of the shot and skips a large portion of content and skips a large portion of content, which is more understandable to users than playing a segment of the same length for all selected shots. It is a way to provide a level of video skimming.

4 shows an example of a method of selecting a length to be played back and skipped based on image / motion / audio similarity in a shot.

In the graph of FIG. 4, the horizontal axis represents time and the vertical axis represents a value accumulated by measuring image / motion / audio dissimilarity in a shot. Such dissimilarity data is generally data representing shot characteristics that can be extracted by a shot segmentation algorithm.

An example of dissimilarity is the difference in color histograms between adjacent frames or frames spaced apart at regular intervals.

In Figure 4, shot A and shot B are similar in length, but the average change rate of shot B is larger than the average change rate of Satte A. Therefore, the situation in which more parts are reproduced in shot B than in shot A is represented.

In this case, if the length of the shot is not considered in setting the playback section, an error situation (when the shot length is very short) may be larger than the length of the shot. If the length is larger than the shot, the entirety of the shot may be selected as the playing section or a part may be selected as the playing section in consideration of the length of the shot.

Next, a reproduction method in the reproduction section will be described.

The video skimming according to the present invention can be applied not only in the forward direction but also in the reverse direction.

By continuously playing the segments selected as the sections to be played in each shot, the user can obtain the overview information of the contents in a short time while understanding the entire contents, and does not need any additional interference in searching for a desired position.

In the video skimming method of the present invention, a method of reproducing segments selected as sections to be reproduced in each shot may be divided into two types.

The first method is to reproduce each segment in the same manner as the normal playback method, and the second method is to decode some frames in the playback section and play back using skipping again in the section.

Since the normal reproduction method is very general, a detailed description thereof will be omitted, and a reproduction method using skipping within the interval by decoding only some frames in the reproduction interval will be described.

The playback method using skipping in a section by decoding only some frames in a playback section is a method for implementing high speed skimming. In this case, the frames to be displayed may be designated as frames spaced apart at regular intervals, and may be designated as I frames having no inter-frame dependency in a method using inter-frame compression such as MPEG.

5 is a diagram illustrating an example of a fast skimming method using skipping in a reproduction section. Using this method, the user can experience the effect of playing video files at high speed while actually getting a lot of information.

The video skimming system according to the video skimming method using the shot segmentation information of the present invention made as described above; In order to search and browse digital video data as multimedia data, user interface means for inputting a user command for video skimming, and a corresponding video file is skimmed based on indexing information according to a user command input to the user interface means. And a video information file for providing digital video data and index information of the video to the control means, and a display means for reproducing the video skimmed by the control means. .

In the present invention, the index information includes shot segmentation information for the video stream.

Also, in the present invention, the user interface means includes means for designating a summary level as a degree of video skimming, or means for designating a speed of a playback section during video skimming. You can choose.

Also, in the present invention, the control means reads the video index information related to the shot segmentation information from the index file according to the skimming condition using the user's input or the basic setting, calculates the segments to be played according to the video skimming condition, and then applies the corresponding media file in the related media file. The segments are continuously reproduced and output to the display means.

6 shows an embodiment of a skimming system configuration of the present invention for video skimming.

As shown in FIG. 6, the video skimming system of the present invention includes a user interface unit 601 for inputting a user command such as a degree of video skimming and a double speed to be used in skimming, and inputted to the user interface unit 601. A main control unit 602 for skimming a corresponding video file based on indexing information according to a user command, a media file 603 for providing digital video stream information to the main control unit 602, and the media file And an index file 604 for providing indexing information, and a display device unit 605 on which a video skimmed by the main control unit 602 is reproduced.

In the video skimming system of FIG. 6, an index file 604 may be included in a media file 603. The display device unit 605 is an output device such as a monitor or a speaker for displaying a video stream, and the user interface unit 601 means an input means such as a keyboard, a mouse, a remote controller, a button, etc. that receives a user input.

The media file 603 is a file in which video (audio) data is stored, and the index file 604 is a file in which index information of a video including shot segmentation information is stored.

The user requests video skimming using the user interface 601.

When you require video skimming, you can specify the level of summarization (the amount of skimming) and the speed you want to use for skimming. That is, the user designates in minutes how to compress and watch the entire video by using the user interface unit 601, and the main control unit 602 responds to the user's input by the media file 603 and the index file 602 accordingly. From this information, it is determined which part of the shot is to be played for skimming, and how many times each segment is to be played. After this process, the main controller 602 decodes the media file 603 and displays the frames on the display device unit 605 to provide a video skimming function to the user.

As described above, the present invention proposes a video skimming method that simultaneously responds to a user's request to grasp the entire contents within a limited time or move to a desired location in a digital video environment.

The present invention solves the problem by allocating segments to be played by shots in order to minimize the probability that a relatively small non-critical portion can be reproduced or a user misses a desired scene. It was.

The video skimming method of the present invention is a method capable of minimizing the necessity of user input for a user's request to move to a desired position.

By using the video skimming function of the present invention, the user can grasp the entire contents in a short time, and in grasping the entire contents, there is an advantage of simply passing a boring part without missing an important part.

In addition, the user can use the video skimming of the present invention even when the user wants to move to a desired position, which has the advantage that the user input request is much smaller than the keyframe method.

As a result, the present invention can be used for the purpose of playing video highlights, and when used with a high-speed playback method in playing sections to be played for each shot, it can be utilized as a function of quickly searching for a desired scene while minimizing a user's input request. .

Claims (16)

A shot segmentation for dividing a video into each individual shot and a method of skimming the video using the segment, Recognizing the section of the individual shot from the shot segmentation information for the video stream, Selecting a portion of the section in the recognized individual shot section as the video information to be reproduced, in order to allocate the segments to be played by shot, Each of Continuously playing the selected video information for each shot in the shot section; Video skimming method using the shot segmentation information, characterized in that comprises a. The shot segmentation information of claim 1, wherein the shot segmentation information is displayed by displaying shot information, which is a unit of production and editing of a video stream, together with temporal information such as a start position and a duration, or a start position and an end position. Video skimming method. The video skimming method using shot segmentation information according to claim 1, wherein the selection of a portion to be played in each shot is performed by using the first half, the second half, the middle half, the first half, and the second half of the shot at the same time. The method of claim 1, wherein the determination of the length to be played in each shot is performed by selecting and playing back segments of the same length in each shot. The method of claim 1, wherein the determination of the length to be reproduced in each shot is performed based on the average value of the image / motion / audio similarity in the individual shots. The video skimming method using the shot segmentation information, characterized in that to increase the. 6. The video skimming method according to claim 5, wherein the image / motion / audio similarity in the shot is similarity of frames, motion vectors, and audio data having different temporal positions in the shot. The method of claim 4 or 5, wherein when the length of the segment selected as the portion to be reproduced in the individual shot is calculated to be larger than the length of the shot, the length of the portion to be reproduced in the individual shot is reduced to the length of the shot or less. A video skimming method using shot segmentation information. The video skimming method using shot segmentation information according to claim 1, wherein the sections to be selected and reproduced for the video skimming are reproduced at a normal video playback speed. The video skimming method using shot segmentation information according to claim 1, wherein the sections to be selected and reproduced for the video skimming are reproduced using a faster playback speed than the normal speed. 10. The video skimming method of claim 9, wherein the playback section is reproduced at high speed by increasing the number of frames to be decoded per unit time than normal. 10. The video skimming method according to claim 9, wherein the playback section is reproduced at high speed by skipping and replaying a few frames in the middle without decoding all the frames in the section to be played back. The method of claim 11, wherein when the fast skimming using the skip is applied to a video stream using a coding scheme using an interframe compression such as MPEG, a frame to be decoded decodes only the corresponding frame without decoding another frame. A video skimming method using shot segmentation information, characterized in that the obtained I frame. A media information file providing user interface means for inputting user commands for video skimming, index information including digital video data and shot segmentation information for recognizing individual shot sections and playback sections of the video data stream, and the user; Control means for selecting a playback section in the individual shot section and continuously playing the selected playback sections based on the indexing information with respect to the corresponding video file according to a user command input to the interface means; and the video skimmed by the control means. Display means for reproducing; Video skimming system using the shot segmentation information, characterized in that comprises a. 15. The method of claim 13, wherein the index information is shot segmentation information for the video stream, and includes shot information, which is a unit of production and editing of the video stream, including temporal information such as a start position and a duration, or a start position and an end position. Video skimming system using shot segmentation information. 15. The method according to claim 13 or 14, wherein the user interface means includes means for designating a summary level as a degree of video skimming, or means for designating a double speed of a playback section during video skimming. A video skimming system using shot segmentation information, characterized in that the level of summary or playback speed can be selected. 15. The apparatus of claim 13 or 14, wherein the control means reads the video index information related to the shot segmentation information from the index file according to the skimming condition by using the user's input or the basic setting, and calculates the segments to be played back according to the video skimming condition. Video segmentation information using the shot segmentation information, characterized in that the corresponding media files are continuously played and output to the display means.