KR20030067133A - Intelligent DVR system using a content based automatic video parsing - Google Patents

Abstract

PURPOSE: An intelligent digital video recording system utilizing a technique of automatically segmenting a motion picture is provided to improve accessibility of motion pictures and to easily find a desired portion of a motion picture. CONSTITUTION: An apparatus for automatically segmenting digital motion-picture data includes a motion picture capturing unit(101) for converting an NTSC signal into a digital signal to extract standard continuous frames, and a calculator(103) for analyzing the frames generated by the motion picture capturing unit to calculate a characteristic quantity of a pixel value difference and histogram difference. The characteristic quantity of the pixel value difference and histogram difference, calculated by the characteristic quantity calculator, and the frames generated by the motion picture capturing unit are analyzed to calculate a characteristic quantity of histogram dispersion of a differential image and pixel value dispersion of each frame.

Description

Intelligent DVR system using a content based automatic video parsing}

According to the present invention, when video recording of a digital video recorder system (Digital Video Recorder System), digital video data is automatically divided and structured based on the content, and the representative frame is selected from the divided unit video to search and browse a flexible video. In more detail, the present invention relates to the development of a digital video recorder system, and more particularly, to a digital video recorder having a function of automatically dividing a video scene and extracting a representative frame capable of representing the content of the divided scene. The present invention relates to a system incorporating a content-based video browser using structure information of a recorded video.

Conventional digital video recorder system has solved the problem of huge data size to some extent with the development of several compression technologies, but browsing is very difficult and requires a lot of time due to the sequential characteristics, which are the characteristics of video information. There is a problem.

In general, one of the greatest features of digitized video is that, unlike video tape, it has instant access at any point. In other words, unlike video tapes that only have linear access that relies on the Fast Forward (Fast) or Fast Backward (FB) functions, a non-linear approach that enables instant movement to the desired point of view is possible. It is the part where the screen changes-the movement of an object, the change of lighting, etc.

However, a practical problem is that the user cannot know where to jump even if a jump function is provided. In general, the player of the digital video recorder system provides a function of moving to a desired position of the video by using a function such as a slide bar, but it can not be said to be useless when the part of the video does not know where the desired video is located. none. After all, in the case of recorded digital video, like the VCR, it has no choice but to rely on the FF function or the FB function to find the desired part. This can be seen as a great match.

As a countermeasure to this problem, an approach to structure video data by dividing a video having sequential characteristics based on contents has been sought and many studies have been made. However, until now, the division is often performed by manual labor, and thus, the division and structuring takes too much time and the waste of manpower is severe.

In order to solve this problem, studies for automating partitioning work have been actively conducted. For example, the first method is to find the transition point by calculating the difference in pixel values between successive frames, and the second is to find the transition point by calculating the difference in histograms between successive frames. Third, various methods have been proposed, such as a method for finding a scene transition point by grasping motion between successive frames.

However, so far, the accuracy is insufficient, and it was difficult to cope with the gradual scene change.

In addition, as a means for representing a divided video, it is necessary to select a frame that can represent each divided video unit, that is, a representative frame. In a conventional study, only one piece per second is selected for this part. Or, a mechanical approach has been adopted to select the first, last, and middle frames of a divided unit video.

However, since the representative frame is an important factor that can influence the quality of service, such as when a user browses or searches by a video feature, more careful selection is required.

The present invention extracts the internal structure information of the video recorded by the digital video recorder to solve the above problems and improve the accessibility of the video to create a TOC (Table Of Contents) of the video data and change the scene of the entire video The digital video data not only makes it easy to find the desired part but also selects a representative frame that is meaningful in content by performing a structured work that divides the part of the next scene into the one level based on the part that is present. It is an object of the present invention to provide a digital video recorder system and a method using an automatic splitting device.

The first means for achieving the object of the present invention is to analyze the digital image frame obtained by the analog / digital conversion of the NTSC signal, such as pixel value difference, histogram difference, histogram dispersion of the difference image and pixel value dispersion of each frame, etc. A first scene for determining a scene change by integrating a feature amount calculating means for calculating a feature amount and a feature amount of pixel value difference, histogram difference, and histogram dispersion of the difference image calculated by the feature amount calculating means using an artificial neural network; Switch point detection means, second scene change point detection means for monitoring a gradual scene change point by monitoring pixel value dispersion of each frame calculated by the feature amount calculation means, and scenes detected by the first and second scene change point detection means. Adopts the selection method suitable for each pattern by dividing the change trend of pixel value difference by pattern by switching point It is configured to include a representative frame selection means for selecting a representative frame over which information significant.

The second means for achieving the object of the present invention is to analyze the digital image frame obtained by analog / digital conversion of the NTSC signal, such as pixel value difference, histogram difference, histogram dispersion of the differential image and pixel value dispersion of each frame, etc. A first scene in which a scene change is determined by integrating a feature quantity calculation process of calculating a feature quantity and a feature quantity of pixel value difference, histogram difference, and histogram dispersion of difference images calculated in the feature quantity calculation process using an artificial neural network. A second scene change point detection step of detecting a progressive scene change point by monitoring a switch point detection process, a pixel value distribution of each frame calculated in the feature amount calculation process, and a scene detected in the first and second scene change point detection processes Adopts the selection method suitable for each pattern by dividing the change trend of pixel value difference by pattern by switching point Consists of a representative frame selection process of selecting a representative frame over which information significant.

The search and browsing information of the video data obtained through the above process is stored in one file format-a file format developed to store information related to video files-so that the structured video can be represented and easily searched for. A feature of the present invention is that a browser is provided that visually analyzes a series of moving images in a hierarchical structure.

1 is a block diagram of a digital video recorder system according to an embodiment of the present invention

2 is a representative distribution diagram of the differential image histogram in the case of continuous frames

3 is a representative distribution diagram of a differential image histogram when a scene change appears

4 is a change diagram of the differential image histogram when there is a momentary change in brightness on the screen

5 is a structural diagram of an artificial neural network used for integrating three feature quantities of pixel value difference, histogram difference, and variance of differential image histogram.

6 is a conceptual diagram of a method for detecting a fading effect according to a change in dispersion value

7 is an exemplary view showing a representative change pattern of pixel value difference

<Explanation of symbols for the main parts of the drawings>

100: camera 101: video capture unit

102: DVR recording unit 103: feature amount calculation unit

104: artificial neural network 105, 106: scene change point detection unit

107: representative frame selection unit 108: video browser

The intelligent digital video recorder system employing the automatic segmentation technology according to the present invention changes the next scene based on the part of the scene change in order to allow nonlinear access to the scene change point. We present a hierarchical data model that classifies parts up to a single level.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a digital video recorder system to which an automatic segmentation technology of a video is applied according to an exemplary embodiment of the present invention.

As shown in FIG. 1, the apparatus for automatically splitting digital video data according to an embodiment of the present invention includes a video capturing unit 101 for generating continuous single frames by A / D converting an NTSC signal, and the video capturing unit ( A feature amount calculator 103 for analyzing feature values such as pixel value difference, histogram difference, histogram variance of difference image, and pixel value variance of each frame by analyzing successive single frames generated in step 101); A scene change point detector 105 for determining a scene change by integrating the feature value of the pixel value difference, the histogram difference, and the histogram dispersion of the difference image calculated by the calculation unit 103 using the artificial neural network 104, and the feature amount A scene change point detector 106 which monitors the distribution of pixel values of each frame calculated by the calculation unit 103 and detects a progressive scene change point, and the scene change point detectors 105, ( The representative frame selecting unit 107 which selects a representative frame by monitoring the change trend of the pixel value difference by the scene change point detected at 106 and the search and browsing information of the video data acquired by the representative frame selecting unit 107. It consists of a video browser to provide a browsing service to the user.

The operation of the digital video recorder system to which the automatic segmentation technology of the video according to the embodiment of the present invention configured as described above will be described in detail with reference to FIGS. 2 and 7.

First, the video capturing 101 A / D converts an NTSC signal to generate each digital frame. The feature variable calculating section 103 calculates a feature variable for measuring the change obtained between successive frames from the generated frame. As shown in FIG. 1, the feature amounts obtained here include a difference in pixel values, a difference in histograms, and a variance value of the differential image histogram which is a feature of the present invention.

Next, the scene change point detector 105 detects the scene change based on a result of inputting and integrating these feature amounts into the artificial neural network 104.

In addition, the scene change point detector 105 simultaneously obtains the variance of pixel values for each frame and determines that fading is occurring when the value falls below a predetermined threshold value and determines the scene change point.

Meanwhile, in order to select a representative frame capable of representing the part from each of the divided unit videos after detection of the scene change point, the representative frame selecting unit 107 analyzes the change pattern of the difference of the pixel value which is particularly sensitive to motion. The representative representative frame is selected.

Hereinafter, the operation of each part will be described in detail.

As the feature amount used for the detection of the scene change point calculated by the feature amount calculation unit 103, the difference between the pixel value represented by the following equations (1) and (2), the difference in the histogram, etc. This has been used.

( One )

(I _m (x, y) is the pixel value of the m th frame coordinate (x, y), N is the number of pixels in the frame)

( 2 )

Where Im (x, y) is the pixel value at the coordinate (x, y) of the m-th frame, and N is the number of pixels in the frame.

Here, Hm is a histogram of the mth frame, and K is a range of pixel values.

However, such a feature has a disadvantage of being particularly weak to movement or vulnerable to a momentary change of brightness. On the contrary, in the present invention, such a disadvantage can be compensated by using histogram variance of difference images between successive frames.

When the histogram of the difference image between successive frames is obtained, the value is generally distributed in the lower side as shown in FIG. On the contrary, when a scene change occurs, there is no correlation between the frames before and after the boundary, and thus the distribution of the difference image shows a random distribution, and thus the histogram has an almost flat shape as shown in FIG. 3.

Therefore, the boundary of the shot can be detected using the difference in the characteristics of the histogram.

In the present invention, the variance of the differential image histogram, which is defined by Equation (3), is used as a feature to measure the characteristics of such a histogram.

(3)

Here, Hm and m + 1 are histograms of the difference images between the mth frame and the (m + 1) th frame, K is a range of difference values, and H is an average value of Hm and m + 1.

This value becomes smaller where the transition takes place and becomes larger where it does not. This feature has the advantage that the position where the peak appears only changes, especially when the change in brightness as a whole occurs as shown in Figs. 4A and 4B, and the dispersion value is not significantly affected.

This feature compensates for the weakness of the difference in general histograms, which are particularly sensitive to this change in brightness.

Next, a method of integrating a plurality of feature quantities using an artificial neural network in the scene change point detection unit 105 will be described.

The feature quantities used in the existing invention and histogram dispersion of the difference image proposed by the present invention have a suitable aspect for different situations. Therefore, by properly combining the respective feature amounts, it is possible to detect the scene change point more accurately.

The present invention incorporates these feature quantities using an artificial neural network. Artificial neural network has the advantage of being able to train with relatively small amount of teacher information because of its characteristics using non-periodic visibility variables, so it is easy to train and has strong characteristics such as noise effects.

The structure of the artificial neural network used in the present invention is shown in FIG. Here, the three feature quantities described above are normalized and input to the input layer, and two neurons indicating whether or not the scene is switched are arranged on the output layer. The hidden layer is one layer, and the number of neurons at that time uses the same number as the input layer.

Next, a method of detecting a progressive scene change in the scene change point detector 106 will be described.

The characteristic of a gradual transition is that a single color frame (a frame occupied by a single color without any shape, such as a black screen) must appear. In such a case, the dispersion of pixel values becomes extremely low in such a frame, so that the point where the scene changes can be easily detected.

That is, as shown in FIG. 6, when the sequence is fading out, the dispersion of pixel values rapidly decreases, and when the fading in starts, the value increases again. The goal of shot boundary detection is to detect point S in FIG.

In order to detect the point S, the dispersion of pixel values is monitored, and when the value falls below a certain threshold indicated by a dotted line in FIG. 6, the dispersion value is continued in the memory until the dispersion value rises above the threshold. That is, the variance between the points A and B is stored. The point having the lowest value among the stored points is recognized as a scene change point.

In particular, since the variance of pixel values can be easily obtained using the histogram already obtained for scene change detection from above, there is an advantage that little increase in processing time is possible. That is, the variance of pixel values in the state where the histogram of the frame is already obtained can be simply obtained by the following equation (4).

( 4 )

Here, H is a histogram, K is a range of pixel values, and N is the number of pixels. On the other hand, M is the average of the pixel values in the frame, which is also easily obtained from the histogram H.

Next, a representative frame selection method in the representative frame selecting unit 107 will be described.

Since the representative frame is to select a frame containing the subject of the shot from a plurality of consecutive frames, it is a problem that requires a high level of analysis of semantic information. However, with current image processing technology, image understanding of unspecified images can be a difficult problem. Therefore, it is important to find general rules for selecting semantically important frames by analyzing various kinds of video data.

In the present invention, a method of selecting a representative frame by analyzing a change pattern of a pixel value difference, which is particularly sensitive to motion, among the feature amounts used for detecting the scene change point is used.

Shown in FIG. 7 is a change pattern of a pixel value difference generally appearing. When the representative frame is selected based on the pattern, a relatively meaningful frame can be selected as the representative frame. Next, a representative frame selection method for each pattern will be described.

Pattern A is a case where there is almost no change on the screen. In this case, no matter which frame is selected, there is no problem, and only one frame is sufficient to represent the contents of the entire shot. In this case, therefore, the frame of the intermediate point is selected.

Pattern B is when a sudden camera operation is taking place or a large object moves at high speed. In such a case, the area where the difference in pixel values becomes large is often of a transient nature and is not important. Therefore, as indicated by the dots, the frame before the movement starts and the frame after the movement are selected as the representative frames.

Pattern C corresponds to a case where the camera is panning or following a moving object. In this case, there is no clue to select a frame of particular importance, and in practice, a frame having a similar degree of importance often changes continuously. In this case, the representative frame is selected while skipping at an appropriate interval. The detailed method is described below.

In the case of the change pattern C of the pixel value, since the content of the video is continuously changed, similar image characteristics are displayed between frames having a close temporal distance, but when the distance is far from each other, frames having completely different image characteristics appear.

The problem here is how to extract a sufficiently changed frame in a sequence of continuously changing frames.

In the present invention, in the case of pattern C, the representative frame is selected by the following method.

In step 1, F1 is selected as the representative frame, the selected F1 is replaced with Fs (Fs = F1), and n is set to 2 (n = 2).]

In step 2, if the difference function D (Fs, Fn) between the frames is larger than the threshold (D> threshold), Fn is selected as the representative frame, the Fn is replaced with Fs, and 1 is added to n. .

Step 2 is repeated until the last frame of the unit video divided in step 3.

Here, Fn means the nth frame of the divided unit video.

As a concrete method of implementing the difference function, a method of obtaining a difference in the histogram is used.

In a video browser, a hierarchical browser is provided to present basic information as well as additional information for browsing a structured video. The hierarchical browser loads all the information about the entire video clip, not just the specific scene of the video resulting from the search, so that the user can easily determine which part of the video the search is for.

As described above, the present invention can effectively solve the problems of the conventional digital video recorder (problem of browsing and flexible search / management) by segmenting a scene suitable for the purpose of the digital video recorder by applying the video automatic segmentation technology to the digital video recorder. Solved.

This can effectively structure the video data by dividing the video data based on the contents and select a representative frame that can represent each divided unit, thereby providing a more flexible search and browsing service to the user.

The present invention employs the histogram variance of the differential image as a feature amount for detecting scene transition points in addition to the difference in pixel values and histograms between successive frames, which has been used in the conventional invention, and the above three feature quantities are used as artificial neural networks. By integrating with, it is possible to improve the accuracy of automatic division.

The present invention adopts a method of using the variance value of each frame as a method for coping with a gradual change of the scene, which has been difficult to cope with the existing. By using this method, it is possible to provide useful information for selecting a representative frame that can represent each unit of a divided video.

The present invention adopts a method of analyzing a change pattern of the difference of pixel values used for scene division in a problem of selecting a representative frame which has a decisive influence on searching and browsing using video content information of a video. It is possible to select a representative frame that is more faithful to the contents of the moving picture than the same regular interval sampling method.

The present invention has the effect that the user can quickly and easily search and browse the desired video information based on the unique visual content of the video information.

Claims (8)

Digital video with a video capture that extracts standard continuous frames by analog-to-digital conversion of NTSC signals, and a feature-quantity calculator that calculates feature values for pixel value differences and histogram differences by analyzing the frames generated by the video capture. In the data automatic splitting device, Characteristic amount for calculating the histogram variance of the differential image and the characteristic amount of the pixel value variance of each frame by analyzing the pixel value difference calculated by the feature amount calculator, the feature amount of the histogram difference, and the frames generated by the video capture Calculation means. First scene change point detection means for integrating a feature value of the pixel value difference, histogram difference, and histogram variance of the difference image calculated by the feature variable calculating means using an artificial neural network to determine a scene change; Second scene change point detecting means for monitoring a gradual scene change point by monitoring the variance of pixel values of each frame calculated by the feature amount calculating means. Representative frame selection that selects a representative representative frame in terms of content by adopting a selection method suitable for each pattern by dividing a change trend of pixel value differences by patterns by scene change points detected by the first and second scene change point detection means. A digital video recorder comprising a digital video data automatic segmentation device comprising a means. Digital video data automatic segmentation method which has the process of extracting NTSC signal into continuous frame by analog / digital conversion and analyzing feature frames and calculating feature amount of pixel value difference and histogram difference To Characteristic amount calculation for calculating the histogram variance of the difference image and the feature amount of the pixel value variance of each frame by analyzing the feature values of the pixel value and histogram difference calculated in the feature amount calculation process and the frames generated in the frame extraction process process. A first scene change point detection process of determining a scene change by integrating a feature value of a pixel value difference, a histogram difference, and a histogram variance of the difference image calculated using the artificial neural network. And a second scene change point detection step of detecting progressive scene change points such as fade out and fade in by monitoring the distribution of pixel values of each frame calculated in the feature amount calculation process. Representative frame selection to select a representative representative frame in terms of content by adopting a selection method suitable for each pattern by dividing the change trend of pixel value difference by patterns according to the scene change points detected in the first and second scene change point detection processes. Digital video recorder applying digital video data automatic segmentation method characterized in that the process consists of The video browser that can search the video recorded by the digital video recorder based on the search and browsing information of the video data.