KR101050255B1 - Video scene segmentation system and method - Google Patents

Abstract

The disclosed video scene segmentation system includes a frame extractor for extracting a frame constituting a segmented video, a coefficient filter for determining a scheduled position for segmentation according to an energy difference between neighboring frames of a frame extracted by the frame extractor, and a coefficient filter. The motion analysis unit determines the continuity between the determined frames before and after the split scheduled position, and the split position determiner determines the scene division position of the video based on the continuity determined by the motion analyzer.

Video, scene, split

Description

System and Method of Scene Partitioning for Moving Picture}

The present invention relates to an image processing system, and more particularly, to a video scene segmentation system and method.

In recent years, the demand for multimedia devices such as DVD, HDTV, satellite TV, set-top boxes, and digital cameras is continuously increasing. Recently, the Internet Protocol Television (IPTV) service, which is an interactive television service provided using high-speed Internet, has been introduced, allowing viewers to select and watch a program they want to watch at a convenient time.

With the development of such multimedia devices, researches are being conducted to effectively manage video data, and there is a research for dividing and structuring video data into one field. The segmentation of video data is used not only to easily store, index, and search video data, but also to insert an advertisement in the middle of the video.

For example, cable TV, IPTV service, etc., instead of watching a movie for free to watch advertisements while watching a movie. In order to pause a movie being broadcast and watch an advertisement, it is important to stop the movie at an appropriate position such as changing the contents of the movie or changing the place where the performer is located.

Currently, a color histogram comparison method and a pixel unit comparison method are used to divide a scene for a movie such as a movie.

The pixel-by-pixel comparison method focuses on the small change in pixel values in the same scene, and compares the corresponding pixel values in a pair of consecutive frames to measure how many changes have occurred. Although this method is simple to implement, it is sensitive to camera movement, and thus it is not possible to accurately segment a scene when applied to a moving image.

On the other hand, the color histogram comparison method divides a scene by comparing the histogram difference of adjacent frames with a threshold value using the characteristic that the frames in the same scene have a similar color distribution. This method is less sensitive to camera movement but has the disadvantage of being sensitive to the effects of light. That is, when there is a sudden change in lighting, the same scene may be recognized as a different scene, and if the color distribution is similar despite the different scene, there is a problem in that it is not detected.

As described above, since RGB values such as pixel values and color distribution of the video are used as basic information, it is difficult to accurately determine the movement of the camera and the temporal / spatial continuity of the video. Accordingly, there may be a problem that an advertisement is output at an unexpected location, and in order to prevent this, an error of scene division must be corrected by manual operation, which causes troublesome work and an increase in time.

The present invention has been made to solve the above-mentioned disadvantages and problems, and there is a technical problem to provide a video scene segmentation system and method that can accurately detect whether or not a scene change.

Another object of the present invention is to provide a video scene segmentation system and method capable of dividing a video into a plurality of independent scenes.

In accordance with an aspect of the present invention, there is provided a video scene segmentation system including: a frame extracting unit configured to extract a frame constituting a segment target video;

A coefficient filtering unit for determining a preliminary split position according to an energy difference between neighboring frames of the frame extracted by the frame extractor; A motion analysis unit for determining the continuity between the frames before and after the split scheduled position determined by the coefficient filtering unit; And a split position determiner configured to determine a scene split position of the video based on the continuity determined by the motion analyzer.

Meanwhile, a video scene segmentation method according to an embodiment of the present invention is a video scene segmentation method in a video scene segmentation system including a frame extractor, a coefficient filter, a motion analyzer, and a segmentation position determiner. Extracting a frame constituting the target video; For each of the extracted frames, determining, by the coefficient filtering unit, a predetermined scheduled position by comparing an energy difference value and a threshold value between neighboring frames; Determining, by the motion analyzer, temporal / spatial continuity between frames before and after the segmentation scheduled position with reference to the segmentation scheduled position; And determining, by the division position determiner, a frame having a low temporal / spatial continuity as a division position.

According to the present invention, a frame meaningful for scene division is selected by comparing energy between frames constituting a moving image. In addition, time / spatial continuity can be simultaneously determined by filtering the coefficients of the intra frame and the predicted frame at the same time, thereby reducing the time required for scene division.

In addition, the motion of the background and the foreground of the video is estimated to determine whether the scene has been changed in actual contents. Accordingly, there is an advantage in that the scene transition time of the video can be accurately extracted and the scene is accurately divided.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the present invention.

1 is a view for explaining a connection relationship of a video scene segmentation system according to the present invention.

As shown, the video scene segmentation system 10 is connected to a plurality of Content Provider (CP) servers 30-1 to 30-n through the communication network 20, or the database 40 storing videos. It may be provided. Then, the video transmitted from the CP server (30-1 ~ 30-n), or the video stored in the database 40 is divided by scene.

When the scene is divided, energy of a frame, motion analysis and estimation of a camera are used, which will be described below with reference to FIGS. 2 to 4.

First, Figure 2 is a block diagram of a video scene segmentation system according to an embodiment of the present invention.

As shown in FIG. 2, the video scene segmentation system 10 may include a controller 110, a frame extractor 120, a coefficient filter 130, a motion analyzer 140, and a segmentation position controller for controlling the overall operation. The unit 150 and the memory 160 are included.

The frame constituting the video may be divided into an intra frame (I-frame), a predictive frame (P-frame), and a bidirectional frame (B-frame). I-frame refers to an independent frame that has little or no correlation with previous or subsequent pictures. The P-frame means a frame that depends on the previous picture, and the B-frame means a frame that depends on the previous picture and the after picture. In other words, spatial correlation can be known as an I-frame, and temporal correlation can be known as a P-frame or a B-frame. However, the B-frame is a frame predicted from a past and future frame with respect to the moving time of the video, and thus, since the movement of the camera cannot be found over time, it is not considered in the present invention.

Therefore, the I-frame and the P-frame are separated from the segmentation target video by using the frame extractor 120.

In general, P-frames are highly dependent on I-frames, but in some cases there may be frames that are less dependent on I-frames. Since the P-frame may be treated as an independent scene, the coefficient filtering unit 130 may have an energy difference, that is, an alternating current (AC) coefficient difference between a neighboring frame among the P-frames, higher than a preset threshold or direct current (DC). Current) Selects a frame having a higher coefficient difference than a preset threshold as a meaningful scene. In addition, an I-frame having a high correlation with a neighboring I-frame may exist among the I-frames, and the coefficient filtering unit 130 may set a threshold value in which an energy difference, that is, an AC coefficient difference, is set in advance among the I-frames. The frame that is higher or the DC coefficient difference is higher than the preset threshold is selected as a meaningful scene. That is, I-frames having high correlation between neighboring I-frames are treated as one scene.

The DC coefficient refers to an average value of brightness or color difference in each block included in one frame, and the AC coefficient refers to an average value of edge or noise. The greater the difference in DC coefficients between frames, the lower the similarity between frames, and the greater the difference in AC coefficients between frames means that the background of the frame is more likely to be switched.

Therefore, in the present invention, I-frames and P-frames of the video to be divided are extracted, and a frame having a large difference in AC coefficient between adjacent I-frames and a frame having a large DC coefficient difference between adjacent P-frames is selected as a split scheduled position. It is.

To this end, the frame extractor 120 includes an I-frame extraction module 122 and a P-frame extraction module 124, as shown in FIG. Meanwhile, the coefficient filtering unit 130 includes a comparison module 132 and a selection module 134.

Frames are extracted by the I-frame extraction module 122 and the P-frame extraction module 124 of the frame extraction unit 120, respectively, for the split target frame. In addition, the comparison module 132 compares the difference between the AC coefficient and the DC coefficient between neighboring I-frames with a preset threshold, respectively, and compares the difference between the AC coefficient and the DC coefficient between P-frames with the preset threshold, respectively. do. In addition, the selection module 134 selects a frame estimated as a meaningful scene change position, that is, a frame having a DC coefficient difference or an AC coefficient difference higher than a threshold value, as the scheduled division position according to the comparison result of the comparison module 132.

However, since it is not possible to accurately determine whether the scene is changed only by comparing the energy, the motion analyzer 140 checks whether the background or the foreground of the frame has been changed from the previous frame with respect to the frames determined as the scheduled division positions.

That is, the motion analyzer 140 determines temporal / spatial continuity through comparison between neighboring frames with reference to the frames selected as the division scheduled positions by the selection module 134 of the coefficient filtering unit 130. Temporal / spatial continuity is achieved by analyzing global motion or local motion of a frame. For this purpose, the motion analyzer 140 may analyze the global motion analysis module 142 as illustrated in FIG. 4. And local motion analysis module 144.

The global motion analysis module 142 checks whether the background image has changed when the corresponding frame is compared with the previous frame through the camera motion analysis. In other words, it is determined whether the camera is a two-dimensional motion (temporal continuity) by a photographing technique such as panning, yawing, zooming, or three-dimensional motion (spatial continuity) by spatial movement. To this end, the global motion analysis module 142 analyzes a distribution pattern of residual coefficients, that is, AC coefficients, for each frame at a predetermined position to be divided. It can be determined whether the change or the energy change due to the actual scene change.

The local motion analysis module 144 checks whether the foreground image, that is, the video object, has changed when the corresponding frame is compared with the previous frame. This is to check whether the person or the person's motion has changed. After separating the video object for each frame at the position to be divided, analyzing the distribution pattern of the surplus coefficients, or AC coefficients, between neighboring frames, the temporal continuity and the spatial continuity are analyzed. Judge.

Various feature extraction algorithms can be used for global motion analysis and local motion analysis, for example principal component analysis for motion vectors / independent component analysis for surplus coefficients, matching tracking analysis Matching Pursuit Analysis and the like can be used. The principal component analysis of the motion vector can determine the two-dimensional movement of the camera (background) or video object (foreground), and the independent component analysis of the surplus coefficients determines the three-dimensional movement of the camera or video object. can do. In addition, in the case of a matching tracking analysis method, there is an advantage that a temporal feature and a spatial feature can be simultaneously extracted.

Meanwhile, the split position determiner 150 determines a frame having a low temporal / spatial continuity as the split position according to the analysis result of the motion analyzer 140 to divide the video, and stores the divided video in the memory 160. do.

As described above, the present invention extracts a frame having a large energy difference from a neighboring frame among frames constituting the split target video to determine a split scheduled position. At this time, the accuracy of scene division is improved by considering not only I-frames independent of neighboring frames but also P-frames.

Then, temporal / spatial continuity is determined from the frames before and after the segmentation scheduled position through motion analysis. As a result, by extracting a frame having low temporal / spatial continuity, and determining a split position therefrom, it is possible to accurately distinguish between frames in which a scene change has been performed, thereby improving scene segmentation efficiency.

Accordingly, in the present invention, the "scene" finally divided from the video may be defined as a scene divided based on the temporal / spatial continuity of the camera or video object according to the playback time of the video.

After the final division of the scene of the video as described above, the division position determination unit 150 may include the reproduction time information, that is, the reproduction order in the metadata of each divided scene. In addition, it is also possible to provide an application service such as generating a snapshot in units of scenes using the play time information to show the contents of the entire video at a glance.

Meanwhile, the scene segmentation method of the video scene segmentation system 10 will be described below with reference to FIGS. 5 to 7.

5 is a flowchart illustrating a video scene segmentation method according to an embodiment of the present invention.

The video scene segmentation system 10 receives a segmentation target video from a CP server or a database held by itself (S10).

Then, the frame extractor 120 extracts the I-frame and the P-frame constituting the video, and the coefficient filtering unit 130 determines the segmentation scheduled position by comparing the energy difference value and the threshold value between the frames. (S20). In a preferred embodiment of the present invention, the coefficient filtering unit 130 calculates the AC coefficient and DC coefficient of the I-frame, the AC coefficient and DC coefficient of the P-frame, and the difference between the AC coefficient and DC coefficient with the neighboring frame, respectively An I-frame larger than a threshold of P, and a P-frame having a difference in AC coefficient and DC coefficient with a neighboring frame larger than each threshold may be selected as a division scheduled position.

Thereafter, the motion analysis unit 140 determines the temporal / spatial continuity by comparing frames before and after the split scheduled position with reference to the split scheduled position determined in step S20 (S30). That is, the feature is extracted from the frames before and after the segmentation scheduled position and compared to determine whether there is a two-dimensional or three-dimensional change.

Then, according to the determination result of step S30, the split position determiner 150 determines a frame having a low temporal / spatial continuity as the actual scene change position of the video (S40), and divides the video at the corresponding position (S50).

Referring to FIG. 6, the division scheduled position determination process illustrated in FIG. 5 will be described. First, the I-frame extraction module 122 and the P-frame extraction module 124 of the frame extractor 120 are included in the corresponding video, respectively. The extracted I / P frames are extracted (S201).

The comparison module 132 of the coefficient filtering unit 130 calculates the energy of each frame, that is, the AC coefficient and the DC coefficient (S203), and compares the energy difference between the frames with a preset threshold (S205).

Thereafter, the selection module 134 determines a frame in which the energy difference value between the frames of the comparison result of the step S205 is larger than the threshold value as the scheduled division position (S207).

After determining the division scheduled position as described above, a motion analysis process is performed as shown in FIG. 7.

That is, the global motion analysis module 142 of the motion analysis unit 140 analyzes the movement of the camera by comparing the frames before and after the split scheduled position (S301), so that a scene change by the movement of the camera is performed, that is, It is determined whether the motion is global (S303).

When it is determined that the motion is global, the background feature of the corresponding frame is extracted, and the temporal / spatial continuity is determined by comparing the feature with the neighboring frame (S309). That is, it is determined whether two-dimensional motion (temporal continuity) by a shooting technique such as panning, yawing, zooming, or three-dimensional motion (spatial continuity) by spatial movement of the camera. To this end, the global motion analysis module 142 analyzes a distribution pattern of residual coefficients, that is, AC coefficients, for each frame at a predetermined position to be divided. It is determined whether it is energy change or energy change due to actual scene change.

In operation S40, the neighboring frame and the frame having low temporal / spatial continuity are determined as the split positions.

On the other hand, when the determination result of step S303 is not the global motion, that is, in the case of local motion, the local motion analysis module 144 separates the video object from the frame, and then distributes the distribution pattern of the surplus coefficient, that is, AC coefficient, between neighboring frames. The analysis determines the temporal / spatial continuity (S309). Similarly, when temporal / spatial continuity with respect to the foreground feature is low, the corresponding frame is determined as the split position (S40).

As described above, the present invention selects a preliminary division position based on an energy difference value between neighboring frames. The temporal / spatial continuity is determined by comparing the frames before and after the segmentation scheduled position, and a frame having low continuity is determined as the segmentation position.

Therefore, the part where the scene change is performed on the contents of the moving picture can be easily detected at high speed, and the accuracy and speed of scene division can be improved. In addition, when the scene segmentation technique is applied to the field of advertisement insertion, it is possible to interrupt the broadcasting of the video at the appropriate position and transmit the advertisement, as well as to insert an advertisement that matches the previously developed contents. The effect can be maximized.

Those skilled in the art to which the present invention described above belongs will understand that the present invention can be implemented in other specific forms without changing the technical spirit or essential features. Therefore, the above-described embodiments are to be understood as illustrative in all respects and not as restrictive. The scope of the present invention is shown by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

1 is a view for explaining a connection relationship of a video scene segmentation system according to the present invention;

2 is a block diagram of a video scene segmentation system according to an embodiment of the present invention;

3 is a block diagram of a frame extractor and a coefficient filter shown in FIG. 2;

4 is a configuration diagram of a motion analyzer shown in FIG. 2;

5 is a flowchart illustrating a video scene segmentation method according to an embodiment of the present invention;

6 is a flowchart for explaining a division scheduled position determination process shown in FIG. 5;

FIG. 7 is a flowchart for describing a motion analysis process illustrated in FIG. 5.

Description of the Related Art [0002]

10: video scene segmentation system 110: control unit

120: frame extraction unit 130: coefficient filtering unit

140: motion analysis unit 150: division position determination unit

160: memory

Claims (16)

A frame extracting unit configured to extract an intra frame (I-frame) and a predictive frame (P-frame) constituting a segmentation video; Computing the energy of each of the I-frame and P-frame, and compares the energy difference with the neighboring homogeneous frame with a predetermined threshold and the comparison module of the comparison module as a result of comparing the frame with the energy greater than the threshold A coefficient filtering unit including a selection module that determines a division scheduled position; A motion analysis unit for determining the continuity between the frames before and after the split scheduled position determined by the coefficient filtering unit; And A split position determiner configured to determine a scene split position of the video based on the continuity determined by the motion analyzer; Video scene segmentation system comprising a. delete delete The method of claim 1, The comparison module calculates at least one of an alternating current (AC) coefficient and a direct current (DC) coefficient from the I-frame, and compares the AC coefficient difference and the DC coefficient difference with adjacent I-frames with respective thresholds. Video scene segmentation system, characterized in that. The method of claim 1, The comparison module calculates at least one of an alternating current (AC) coefficient and a direct current (DC) coefficient from the P-frame, and compares an AC coefficient difference and a DC coefficient difference with an adjacent P-frame with respective thresholds. Video scene segmentation system, characterized in that. The method of claim 1, The motion analysis unit comprises a global motion analysis module for analyzing the motion of the camera for each of the frames before and after the split scheduled position determined by the coefficient filtering unit. The method of claim 6, The global motion analysis module may determine whether a background image is changed temporally or spatially by analyzing a camera movement according to a difference value of an alternating current (AC) coefficient for each of the frames before and after the segmentation scheduled position. Scene segmentation system. The method of claim 6, The motion analysis unit further comprises a local motion analysis module for analyzing the change of the video object for each of the frames before and after the split scheduled position determined by the coefficient filtering unit. The method of claim 8, And the local motion analysis module determines whether the foreground image changes temporally or spatially according to a difference value of an alternating current (AC) coefficient for each of the frames before and after the segmentation scheduled position. The method of claim 1, And the division position determiner divides a moving image scene according to a scene dividing position of the moving image, and stores reproduction order information in metadata of each divided scene. A video scene segmentation method in a video scene segmentation system including a frame extractor, a coefficient filter, a motion analyzer, and a segment position determiner, Extracting an intra frame (I-frame) and a predictive frame (P-frame) from the video to be divided by the frame extractor; For each of the extracted frames, determining, by the coefficient filtering unit, a predetermined scheduled position by comparing an energy difference value and a threshold value between neighboring frames; Determining, by the motion analyzer, temporal / spatial continuity between frames before and after the segmentation scheduled position with reference to the segmentation scheduled position; And And determining, by the division position determiner, a frame having a low temporal / spatial continuity as a division position. The determining of the scheduled division position includes: calculating at least one of an alternating current (AC) coefficient and a direct current (DC) coefficient from the I-frame; Comparing the AC coefficient difference and the DC coefficient difference with neighboring I-frames with respective thresholds; Determining an I-frame in which the AC coefficient difference value and the DC coefficient difference value are larger than the respective threshold values as the scheduled division positions; Calculating at least one of an Alternate Current (AC) coefficient and a Direct Current (DC) coefficient from the P-frame; Comparing the AC coefficient difference value and the DC coefficient difference value between neighboring P-frames with respective threshold values; And Determining a P-frame in which the AC coefficient difference value and the DC coefficient difference value are larger than the respective threshold values as the scheduled division positions; Video scene segmentation method comprising a. delete delete The method of claim 11, The determining of the temporal / spatial continuity may include: determining whether a background change has been made by analyzing the movement of the camera; And Determining temporal / spatial continuity by comparing features with neighboring frames when a background change is made as a result of the determination; Video scene segmentation method comprising a. The method of claim 14, Determining whether a foreground change is made when the background change is not performed as a result of the determination; And Determining temporal / spatial continuity by comparing features with neighboring frames when the foreground change is made; Video scene segmentation method further comprising. The method of claim 11, After the determining of the dividing position, dividing the scene of the video according to the dividing position by the dividing position determining unit; And Storing division order information in metadata of the divided scenes by the division position determiner; Video scene segmentation method further including.

Images