KR101068288B1 - Content-Based Video Copy Detection Method Using Segment Feature Extraction - Google Patents

Abstract

Content-based video detection method using the segment feature according to the present invention, the segment extraction step of extracting a segment to be used for detecting duplicate video; A segment comparison step of comparing a segment extracted from the input video to segments extracted from the original video to find a segment included in the original video similar to the segment included in the input video to detect duplication of the input video; And a video detection step of determining which video is copied from the segment comparison result of the input video.

Description

Content-Based Video Copy Detection Method Using Segment Feature Extraction}

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for detecting a duplicate video. In particular, a segment feature for extracting a segment created by merging frames having similar feature values, and comparing the video in segments by using the segment information, detects a duplicate video. It relates to a content-based video detection method using a.

As the internet environment develops rapidly, the means of transferring knowledge is rapidly changing from texts and images to video. In this process, the need for copyright protection against illegal copying of video media is also increasing.

There is a content-based copy detection method as a method for copyright protection of video illegal copying. Such a content-based copy detection method detects a duplicate video by using a feature that can be extracted from a video, and has an advantage that a feature can be extracted even for a previously distributed video.

Since video has a larger capacity than other media, it is important to extract a feature that can efficiently identify a video for video detection. However, since the existing content-based copy detection methods use the features of all frames, unnecessary data and operation costs are required. That is, the existing duplicated video detection research requires a lot of processing time because all frames are compared.

Specifically, the content-based video copy detection is a study for determining whether illegal copying is performed between an input video for comparison and an original video collected in advance in order to determine whether or not to copy. These studies extract feature values from two videos to determine whether the video is illegally copied and then determine whether the video is illegal by comparing the extracted feature values.

In the content-based video copy detection technique disclosed in Ref. [1], the feature is classified into two types according to the extraction range, and the performances of the corresponding methods are compared. The local descriptor method extracts and uses the partial region feature of the video frame and the partial region feature of the continuous frame, whereas the global descriptor method uses the entire image information included in the frame. Is used as a feature value.

The local descriptor method determines whether a video is copied based on a Harris interest point detector and a differential description of the surrounding area.

In Ref. [2], the local area feature of interest is expressed as the second derivative of Gray level 2D signal, and the ViCopT method disclosed in Ref. You have assigned a Motion or Background label.

In addition, the STIP method disclosed in Ref. [4] used feature points capable of capturing temporal and spatial changes to capture spatiotemporal events. However, the method of using a Harris interest point detector may not detect an interest point on an image to which an editing effect such as contrast reduction is applied, and thus its performance may be degraded.

Among the global descriptor methods, a motion histogram feature representing a clip is proposed by quantizing a relative motion of a block divided by a specific number in consecutive frames and statistically representing each motion in a time domain. 5].

In reference [6], the image is divided into N windows to obtain an average contrast value for each window, and the average brightness values of each window are arranged in ascending order and displayed in discrete rank for the frame t. We propose an ordinal measurement method that is less sensitive to changes and that focuses on the relative order of contrast values.

In addition, Ref. [7] proposed a temporal ordinal measurement method for displaying the image divided into N windows as equality of windows having the same position for each frame for a specific time.

In addition, Reference [8] further describes motion matching (see Reference [5]), the original measurement method (see Reference [6]), and color histogram intersection method, which are used as video copy detection methods. By comparing the performance of, we showed that the search efficiency of the original measurement method is excellent.

Since the content descriptor copy detection method of the Global Descriptor method performs a video search by comparing the frame unit, an unnecessary operation of comparing all adjacent frames having similar feature values is necessary.

references

[1] J. Law-To, L. Chen, A. Joly, I. Laptev, O. Buisson, V. Gouet-Brunet, N. Boujemaa, and F. Stentiford, "Video copy detection: a comparative study," In ACM International Conference on Image and Video Retrieval, Amsterdam, The Netherlands, July 2007.

[2] A. Joly, O. Buisson, and C. Frelicot. Content-based copy detection using distortion-based probabilistic similarity search. ieee Transactions on Multimedia, 2007.

[3] J. Law-To, O. Buisson, V. Gouet-Brunet, and N. Boujemaa, "Robust voting algorithm based on labels of behavior for video copy detection," In ACM Multimedia, MM'06, 2006.

[4] I. Laptev and T. Lindeberg. Space-time interest points. In International Conference on Computer Vision, 2003.

[5] D.N. Bhat, S.K. Nayar, “Ordinal measures for image correspondence,” In IEEE Trans. on PAMI, Vol. 20, No. 4, pp. 415-423, 1998

[6] L. Chen and F. W. M. Stentiford, "Video sequence matching based on temporal ordinal measurement," Technical report no. 1, UCL Adastral, 2006.

[7] Ki-Ho Hyun, Jae-Cheol Lee, "Content-based Video Copy Detection Using Directional Histogram of Motion," Journal of KIISE: Software and Applications, Vol. 30, No. 5, 6, pp. 497-502, Jun., 2003.

[8] A. Hampapur, K. Hyun, and R. Bolle., “Comparison of Sequence Matching Techniques for Video Copy Detection,” In SPIE. Storage and Retrieval for Media Databases 2002, vol. 4676, pp. 194-201, San Jose, CA, USA, Jan. 2002.

[9] M. J. Swain and D. H. Ballard, Color indexing, International Journal of Computer Vision, vol. 7, no. 1, pp. 11-32, Nov. 1991.

The present invention has been made in view of the above, and it is possible to divide a frame included in an image into a plurality of windows, extract feature values for each window, combine frames having similar feature values into one segment, An object of the present invention is to provide a content-based video detection method using a segment feature that detects a duplicate video through segmental search.

Content-based video detection method using a segment feature according to the present invention for achieving the above object,

A segment extraction step of extracting a segment to be used for detecting a duplicate video;

A segment comparison step of comparing a segment extracted from the input video to segments extracted from the original video to find a segment included in the original video similar to the segment included in the input video to detect duplication of the input video;

And a video detection step of determining which video is copied from the segment comparison result of the input video.

According to the present invention, by comparing the video in segments, the search time can be shortened while the detection accuracy is equivalent to that of the existing frame-based method.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic configuration diagram of a video detecting method according to the present invention. The present invention combines a plurality of consecutive similar frames included in a video into segments, and features of segments of an input video and an original video have. This is a method of determining whether a video is duplicated by comparing the results.

As shown in FIG. 1, first, in a preprocessing step for detecting a video, a segment is extracted from a reference video set through a representative frame extraction process and a segment feature value extraction process, thereby constructing a segment database. Subsequently, in the search step, in order to detect whether the input video (Query video clip) is duplicated, the segment is extracted through the representative frame extraction process and the segment feature value extraction process in the summary video as in the preprocessing step. Compares the segments extracted from the input video with the segments extracted from the video extracted from the original video, registers the values created for segment comparison in the result set, and then compares the original video with the results of all segments Detect.

Hereinafter, a method of detecting a video in segment units will be described.

The process for detecting a video on a segment basis includes a segment extraction step, a segment comparison step, and a video detection step.

1. Segment Extraction Step

The segment extraction step is a step of extracting a segment to be used for detecting a duplicate video, and the detailed segment extraction step is composed of two steps as shown in FIG.

In the present invention, a segment is a unit for comparing a video, and the segment is composed of a representative frame which is a start frame of the segment determined using the intensity information of the frame, and other frames including similar motion information. Accordingly, the segment extraction step includes a representative frame extraction step of extracting a representative frame and a segment feature value extraction step of generating a segment by extracting motion information.

1.1 Representative Frame Extraction Stage

First, the representative frame extraction step will be described.

A segment consists of a set of similar frames. Therefore, the contrast characteristic values of the segment start frame and the previous frame have a difference of more than a certain level. In the representative frame extracting step, the representative frame is extracted using the intensity information of the pixels included in the frame. However, expressing one frame with one feature value is difficult to use as a unique feature value of the frame. Therefore, in the representative frame extracting step, the frame is divided into regions having a specific size, and the representative frame is determined by using a difference in the average value of the contrast of the pixels included in each region.

First, a frame of a video is divided into N _Frame = N _x × N _y windows. The average intensity of each divided window is calculated using the intensity information of the pixels included in the divided windows. The average intensity of a window i L _i, is calculated by equation (1).

---- (1)

---- (One)

Where W and H are the width and height of the window, and x and y represent the pixel coordinates inside the window i. I (x, y) is the intensity value of the pixel corresponding to the coordinates of x and y.

All the frames have the intensity value of the frame as the intensity value of the inside of each divided window obtained by Equation (1). Therefore, the sum total of the difference of the average contrast value of each window existing at the same position in the previous frame and the current frame is used for determining the representative frame. If the sum of the differences in the average intensity values of each window is larger than the experimentally obtained value, the current frame is selected as the representative frame. When the representative frame is selected, the segment from the previous representative frame to the current representative frame becomes a segment.

In general, the segment of the duplicated video and the original video is composed identically. However, if the representative frame started by artificial editing is copied differently, the representative frame of the segment is selected differently in succession. Therefore, in the present invention, the video is divided into frames equal to 1 second, but when a scene change occurs, a segment is generated only up to the frame before the scene change, and a new segment is generated after the scene change.

1.2 Segment Feature Extraction Step

In the segment feature value extraction step, the motion features of the representative frame and the successive frames are extracted to find the frame that is the end point of the segment from the representative frame.

To this end, the direction of change is quantized by searching by comparing the window of the previous frame with the corresponding window position and the surrounding window in the current frame. The current frame is referred to as _t , and the windows at the same location in frame t and frame _t-1 are called B _t and B _t-1 , respectively.

FIG. 3 is a diagram for explaining a process of extracting segment feature values. As shown in FIG. 3, an oral specific value (see Ref. [2]) is obtained from neighboring windows adjacent to eight directions with respect to one window. In this case, the tracking area is defined based on the window B _t of the current frame, and the audition measurement values of all the windows in the tracking area are compared with the auditor specific values of the window B _t-1 of the previous frame.

Of these, the direction of the frame t having the smallest distance difference from the window B _t-1 is quantized. Histograms corresponding to the number of directions of the frame are generated by statistically processing the direction values calculated in all windows of the frame t-1. This constitutes a representative frame and a segment in the direction vectors in the histogram that has a N _x × N _y of the average intensity over the window.

2 segment comparison step

In this step, in order to detect duplication of the input video, the segment extracted from the input video and the segments extracted from the original video are compared to find a segment included in the original video similar to the segment included in the input video.

To find similar segments, this segment comparison step uses a representative frame comparison step and a direction vector histogram comparison step.

Since one segment is composed of a frame having similar contrast and motion information, the representative frame of the same segment has a similar brightness value as the other frames. Therefore, in the representative frame comparison step, the similarity between the segments is calculated using the difference in the intensity of the representative frame. However, since the representative frame of the segment can be selected differently by artificial editing, the motion characteristics of the contrast are compared in the direction vector histogram comparison step to compensate for the error occurring when the input segments are only partially the same.

Comparison of segments requires comparison of input segments with all original segments. However, the segments of the input video have a feature that is similar in order to the segments of the original video. Therefore, in order to reduce the search speed of the segment, the input segment currently being compared is first compared with the segment existing behind the most similar original segment.

2.1 Representative Frame Comparison Step

Since one segment has sequentiality and similarity with respect to spatial features, if the segment of the input video and the segment of the original video are similar, the intensity component of the representative frame of each segment is similar. Therefore, in order to compare the similarity of the segments, the intensity of the representative frame is compared using Equation (2) above. Equation (2) represents the difference in contrast characteristics between the input segment and the original segment. Therefore, it can be said that the original segment with the smallest Dist _frame is a duplicated original segment.

---- (2)

---- (2)

Where Q is the input video segment, T is the original video segment, and N is the number of windows that the representative frame has. I (Q _i ) and I (T _i ) are the intensity values of the i-th window of the representative frame of each segment. As the Dist _frame value is smaller, the representative frames of the two segments have higher similarity.

The representative frame is the start frame of the segment, and one segment is composed of frames having similar brightness characteristic values. Therefore, the representative frame of the segment can be used as the representative feature value of the segment. However, the representative frame may be changed by artificial editing of the video, and since the segment is necessarily generated every certain time, the representative frame of the segment may deviate continuously. Therefore, to compensate for this case, segments are further compared using motion feature values.

2.2 Direction Vector Histogram Comparison

By comparing the direction vector histogram, when a frame included in the original segment and the input segment is only partially identical, an error occurring in the difference between the representative frames mentioned in the representative frame comparison step may be compensated for.

The histogram intersection proposed by Swan and Ballard (see Ref. [9]) is used to measure the similarity of the direction vector histogram. The higher the Dist _motion value, the more similar the motion characteristics of the two segments.

---- (3)

---- (3)

---- (4)

---- (4)

Here, Q denotes a frame of the input video, T denotes a frame of the original video, and i denotes each quantized direction component. l is the number of quantized directional components, and 0 means that the window has not moved. h _k (Q _i ) and h _k (T _i ) are the number of windows having the movement of each direction component i in the kth motion in the segment. n means the number of frames integrated into the segment.

Dist _frames generated during the comparison of representative frames are measured smaller for similar videos, and Dist _motion is measured larger for similar videos. Therefore, to evaluate the similarity between segments, equation (5) is used to determine the final similarity between segments. Select the original segment with the highest similarity value as the duplicated source segment.

---- (5)

---- (5)

3. Video Detection Step

In the video detection step, it is determined from which image is copied from the segment comparison result of the input video. Through the video detecting step, the detected results of the duplicate segments of the n input segments divided from the input image may be detected as the result of the duplicate image unit, and if an error occurs in the search results for some segments, This can be complemented by the results of the segment.

The input video edited by artificial editing is not only composed of one video but also composed of a plurality of unspecified videos. Therefore, in this step, a video detection method capable of searching such a plurality of edited videos is performed.

If the segment serial number of the input video is Q and the segment serial number of the original video determined to be similar to Q is R (Q), the process of detecting the search result of each segment as a video is as follows.

1) If the next segment R (Q) +1 of R (Q) and the original segment R (Q + 1) determined to be similar to the next segment Q + 1 of the input segment Q are equal, Tol _TRUE is increased.

2) If Tol _TRUE is larger than L, which is an identification criterion of one video, it detects that segments of Q to Q + Tol _TRUE are similar to videos of original R (Q) to R (Q) + Tol _TRUE . In this case, L was determined as an experimental result value.

3) If R (Q + 1) ≠ R (Q) +1, increase Tol _FALSE . Tol _FALSE is the number of consecutive noises.

4) flew less than the reference R that identifies the different video Tol _FALSE R (Q) + Tol _FALSE +1 = R (if the Q + Tol _FALSE +1), a segment of up to 1 + Q ~ Q + Tol _FALSE is error The noise is corrected and adjusted to a continuous segment between R (Q) and R (Q + Tol _FALSE +1).

5) If Tol _FALSE is greater than the criterion R identifying that it is another video, it indicates that Q + 1 and later are no longer segments of the video as before Q. Therefore, the video detection is determined in the process of 2) for the segment up to Q, and the process of 1) is repeated from the Q + 1 segment.

That is, each detection step includes: 1) verifying the continuity of the result of the segment unit, 2) detecting a similar video through the continuity, 3) giving an error correction opportunity, 4) determining the noise, and 5) determining the start of another image. Can be summarized as steps.

4. Experiment and Results

By the method according to the present invention, the representative frame of the segment is divided into a total of 465 windows (31 × 15), and the quantization direction number of the motion feature is quantized in nine directions including the stop and eight directions. The capacity of one segment with information of 24 frames is approximately 1 kb, excluding the header information.

As an input image for comparing the search performance, 25 partial images, which are part of a movie, and 25 edited images, which are made by combining partial images of several movies, were prepared. The 50 input images are 5 minutes each.

For the experiment, a 100-hour movie with a resolution of about 800 × 336 was prepared as the original video. Segments of the original video are assumed to be extracted and stored in advance.

4.1 Analyzing Data and Processing Time

The search time was calculated by sequentially searching the input video for 5 minutes and 15 minutes for 3 hours and 100 hours original video, respectively. The comparison method, Ordinal Measurement (see Ref. [5]), performs a sequential search with a window size of 3 × 3. The method is repeated 20 times each and the performance of the search time is compared with the average value.

Table 1 Feature Size Comparison

Table 1 shows the magnitudes of the comparative features produced through the pre-treatment process used in the method of ordinal measurement and the method of the present invention. The method according to the present invention includes 5,117 segments in an average video of 1 hour, on average 1 segment has 17 frames, and the size of each segment is 5,016 bits. That is, in the method according to the present invention, the feature size is about 25.7 M bits (5,117 x 5,016). On the other hand, since the size of the feature of 17 frames in the conventional Ordinal Measurement is 612 bits, the feature size is about 3.1 M bits (5,117 x 612). Similarly, in the case of a three-hour video, the present invention has a feature size of about 9.11 Mbytes, while in the auditoral measurement method it has a feature size of about 1.11 Mbytes. Therefore, the present invention has about eight times as many feature sizes as compared with the conventional ordinal measurement method. However, compared to the original video clip, the feature size of the present invention is reduced to about one hundredth of a level.

Table 2 Search Time Comparison

Table 2 shows the experimental results in comparison with the conventional original measurement (Ordinal Measurement) method. The search time according to the present invention was shown to be reduced to a level of one hundredth of that of the conventional ordinal measurement method. This is because the comparison information is less than that of the same amount of frame operations corresponding to the unit cost of the segment, and the search time is reduced because the segments included in the same video are first searched in consideration of the order characteristic. The search time is proportional to the number of segments of the input video and the number of segments of the original video. The search time decreases as the input video is continuous and the change is small.

4.2 Accuracy Analysis

With 50 videos, I retrieved 100 hours of original video. To measure the accuracy of the search performance, the average precision method was used to measure not only the results of the frame but also the accuracy of the segments. The average precision method is to give more weight to the relevant results found in the higher rank among the search results. The average precisions of the search results generated for each segment of each input video were calculated by Equation (6), and the search performance was calculated from the arithmetic mean.

---- (6)

---- (6)

Where R is actually the number of original segments similar to one input segment, and i is the rank of the detection result. xi is a binary function that represents 1 if the result of the i th rank is related, or 0, and pi is the precision of the result cut-off to rank i.

The method according to the present invention detects a newly generated video by collecting and combining the video without converting the information of the video. Both partial and edited images were detected, and the average of 99.7% was detected in the results of video segment. The method according to the invention shows that the accuracy of the segment unit is also high.

Meanwhile, the search performance of the video segment may be reduced as follows.

The first is a case where a segment of an original video divided by a specific time unit and a segment of an input video are shifted at regular intervals at the boundary where the input video is edited.

Secondly, in a scene where there is no motion for a long time, the segments of a certain time unit have almost the same value, so the surrounding segment is detected instead of the exact segment of the original video.

Third, when the length of the segment of the input video is shorter than the split reference time, the result value is inferior because the amount of information that can be compared with other segments is small.

As described above, the present invention provides a method of detecting an illegal copy video by combining frames having similarities into one segment, so that not only the illegal copy video partially extracted from the original but also the duplicate video extracted from different videos are combined. Both can be detected and can be detected at a faster time than the frame-by-frame comparison method.

1 is a schematic configuration diagram of a video detecting method according to the present invention;

2 is a view showing a segment extraction process according to the present invention,

3 is a diagram for explaining a process of extracting segment feature values according to the present invention.

Claims (7)

The representative frame is extracted by using the intensity information of the pixels included in the frame, and the video frame is divided into N _Frame = N _x × N _y window regions, and the difference between the average values of the intensity of the pixels included in each window region is determined. A representative frame extracting step of determining a representative frame by using the second frame; and in order to find a frame that is the end point of the segment from the representative frame, the window of the previous frame is compared and searched including the corresponding window position and the surrounding window in the current frame, and the change direction A segment feature extracting step of extracting motion feature values of the representative frame and the successive frames by quantizing a segment, and extracting a segment to be used for detecting a duplicate video; A segment comparison step of comparing a segment extracted from the input video to segments extracted from the original video to find a segment included in the original video similar to the segment included in the input video to detect duplication of the input video; And a video detecting step of determining which image is copied from the segment comparison result of the input video. delete The method of claim 1, wherein the average intensity of each divided window is calculated using the intensity information of the pixels included in each divided window, and the average intensity L _i of the window _i is:

(W and H are the width and height of the window, x and y are the pixel coordinates inside window i, and I (x, y) is the intensity value of the pixel corresponding to the coordinates of x and y.) Content-based video detection method using the segment feature, characterized in that calculated by. The method of claim 1 wherein the segment comparison step to find similar segments, To compare the similarity between the input segment and the original segment,

(Where Q is the input video segment, T is the original video segment, N is the number of windows the representative frame has, I (Q _i ) and I (T _i ) are the i-th windows of each segment's representative frame. Contrast value) A representative frame comparing step of comparing the contrast between the input segment and the original segment to show the difference in the contrast characteristic, and comparing the contrast of the representative frame; When the frames included in the original segment and the input segment are only partially identical, in order to compensate for errors occurring in the difference between the representative frames in the representative frame comparison step,

Where Q is the frame of the input video, T is the frame of the original video, i is each quantized direction component, l is the number of quantized direction components, and h _k (Q _i ) and h _k (T _i ) are segments The number of windows with the movement of each directional component i in the k th motion in n, where n is the number of frames incorporated into the segment) Measure similarity by

And a direction vector histogram comparing step of determining similarity between the final segments by using the segment-based content detection method. The method of claim 4, wherein in the representative frame comparison step, the original segment having the smallest Dist _frame is a duplicated original segment. The method of claim 4, wherein, in the final determination of similarity between segments, an original segment having the largest similarity value is selected as a duplicated original segment. The method of claim 1, wherein the detecting of the video comprises: If the segment serial number of the input video is similar to Q and Q, and the segment serial number of the original video is set to R (Q), Continuity of segment-by-segment result that increases Tol _TRUE if next segment R (Q) +1 of R (Q) and original segment R (Q + 1) determined to be similar to next segment Q + 1 of input segment Q are equal Confirmation step; If Tol _TRUE is greater than L (where L is the result determined by the experiment), which is the identification criterion of one video, the segments of Q to Q + Tol _TRUE are the original R (Q) to R (Q) + Tol _TRUE A similar video detection step through continuity for detecting that the video is similar to as many videos; Providing an error correction opportunity for increasing Tol _FALSE (where Tol _FALSE is the number of consecutively occurring noises) when R (Q + 1) ≠ R (Q) +1; Tol is _FALSE flew less than the reference R to identify that other video if the _{R (Q) + Tol FALSE +1} = R (Q + Tol FALSE +1), Q + 1 ~ segment to Q + Tol _FALSE is error correctable A noise determination step of judging as noise and adjusting to a continuous segment between R (Q) and R (Q + Tol _FALSE + 1); If Tol _FALSE is greater than the criterion R identifying that it is another video, it indicates that after Q + 1 is no longer a segment of the same video as before Q, and then by a similar video detection step through the continuity for the segment up to Q. And a determination step from the Q + 1 segment to the start of another image which repeats the continuity check of the result of the segment unit again from the Q + 1 segment, thereby detecting the segment unit search result as a movie. Content-based video detection method using.

Images