Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
  • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
  • G06F16/7847—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content

Definitions

• the present invention relates to a method and apparatus for comparing videos.
• video content may be uploaded by users to the site and made available to others via search engines. It is believed that current web video search engines provide a list of search results ranked according to their relevance scores based on a particular a text query entered by a user. The user must then consider the results to find the video or videos of interest.
• duplicate video content may include video sequences with identical or approximately identical content but which are in different file formats, have different encoding parameters, and/or are of different lengths.
• Other differences may be photometric variations, such as color and/or lighting changes, and/or minor editing operations in spatial or and temporal domain, such as the addition or alteration of captions, logos and/or borders.
• duplicate videos can make it difficult or inconvenient for a user to find the content he or she actually wants.
• based on sample queries from YouTube, Google Video and Yahoo! Video on average it was found that there are more than 27% near-duplicate videos listed in search results, with popular videos being those that are most duplicated in the results.
• users Given a high percentage of duplicate videos in search results, users must spend significant time to sift through them to find the videos they need and must repeatedly watch similar copies of videos which have already been viewed.
• the duplicate results depreciate users' experience of video search, retrieval and browsing.
• such duplicated video content increases network overhead by storing and transferring duplicated video data across networks.
• sequence matching an interval of time with multiple frames provides a basis for comparing the similarity of a query video and a target video.
• this involves extracting a sequence of features, which may be, for example, ordinal, motion, color and centroid- based features, from both the query video frames and the target video frames.
• the extracted feature sequences are then compared to determine the similarity distance between the videos. For example, where ordinal signatures are used, each video frame is first partitioned into Nl x N2 blocks and the average intensity of each block is calculated. Then, for each frame, the blocks are ranked according to their average intensities. The ranking order is considered to be that frame's ordinal measure.
• the sequence of ordinal measures for one video is compared with that of the other to assess their similarity.
• Sequence matching enables the start of the overlapping position between duplicate videos to be determined. Sequence matching approaches are suitable for identifying almost identical videos and copies of videos with format modifications, such as coding and frame resolution changes, and those with minor editing in the spatial and temporal domains. In particular, using spatial and temporal ordinal signatures allows detection of video distortions introduced by video
• digitalization/encoding processes for example, changes in color, brightness and histogram equalization, encoding parameters
• display format conversions for example converting to letter-box or pillar-box
• modification of partial content for example, cropping and zooming in
• Sequence matching techniques involve a relatively easy calculation and provide a compact representation of a frame, particularly when using ordinal measures. Sequence matching tends to be computationally efficient and real time computations may be carried out for processing live video. For example, an ordinal measure with 2 x 2 partitions of a frame needs only 4-dimensions to represent each frame, requiring fewer comparison points between two frames.
• Keyframe matching techniques usually segment videos into a series of keyframes to represent the videos. Each keyframe is then partitioned into regions and features are extracted from salient local regions. The features may be, for example, color, texture, corners, or shape features for each region. Keyframe matching is capable of detecting approximate copies that have undergone a substantial degree of editing, such as changes in temporal order or insertion/deletion of frames. However, since there are simply too many local features in a keyframe, it is computationally expensive to identify keyframes, extract local features from each keyframe and conduct metric distance comparison between them to match a video clip against a large number of videos in database.
• a method for comparing a query video and a target video includes partitioning frames of the query video and frames of the target video into blocks and calculating the mean intensity value for each block.
• a plurality of query time series is produced for the query video, each query time series representing temporal variation in mean intensity value for blocks from the same location in different frames of the query video.
• a plurality of target time series is produced for the target video, each target time series representing temporal variation in mean intensity value for blocks from the same location in different frames of the target video.
• the query time series and the target time series are used in determining if alignment exists between the query video and the target video.
• time series may be produced which can be compared for similarities.
• Duplicate videos show similarities in the their respective time series, which may be used to identify that they are related.
• a method in accordance with the invention offers efficient video duplication detection by reducing the comparison space between two videos.
• An embodiment includes segmenting the query time series and the target time series into a respective set of discrete linear segments and performing local sequence alignment of those linear segments.
• Linear segmentation enables mean video intensities to be compressed into a discrete list of linear inclines/declines which may then be compared for alignment.
• the overlapping video regions usually do not span the entire length of video sequences and similar regions could be isolated. Therefore, local alignment of linear segments is needed.
• the Smith- Waterman algorithm is well-known for determining similar regions between two nucleotide or protein sequences. The Smith-Waterman algorithm compares string segments of all possible lengths and optimizes the similarity measure. The present inventors have realized that the Smith- Waterman algorithm may be extended to perform local alignment for video intensity segments. Instead of comparing strings, intensity linear segments are compared to find local optimal alignment between videos.
• the Smith- Waterman algorithm is a dynamic programming algorithm to provide optimized search. It is fairly demanding of time and memory resources: the computational complexity is 0(MN) and the storage is 0(min(M, N)), where M and N are the lengths of the sequences under comparison.
• a sequence of major inclines/declines is selected as representations of key signatures of compared videos.
• a heuristic method is applied to provide fast alignment of those major inclines/major declines by excising alignments that unlikely to result in a successful alignment before performing the more time-consuming Smith- Waterman algorithm. This reduces computational cost.
• the heuristic method expedites the execution of the matching algorithm by filtering out very dissimilar videos and by narrowing down the potential matched regions for similar videos.
• An embodiment in accordance with the invention may be advantageous where it is not feasible to know the types of user modifications in advance before applying video duplication detection techniques, allowing sequence matching techniques to be used. In addition, it retains the advantages of using sequence matching approaches, which is to provide efficient detection.
• Detecting duplicate video with frame changes using an embodiment in accordance with the invention may be used by video hosting websites as a user feature; or used by video content providers to keep track of royalty payments and to detect possible copyright infringements; or used by communication "pipes" (e.g. Internet Service Providers (ISPs), peer-to-peer (P2P) system providers, content distribution network (CDN)) to reduce network traffic and to manage the storage of video content.
• ISPs Internet Service Providers
• P2P peer-to-peer
• CDN content distribution network
• It could assist video hosting websites in removing or aggregating near-duplicate videos to provide service for users to search, retrieval and browsing. It could also facilitate video content-based searching by finding similar videos, for example, with high quality (HD) or 3D.
• HD high quality
• a pre-existing video duplication system may be modified to include an embodiment in accordance with the invention, to enhance the ability to handle user modifications, such as frame insertions, deletions, or substitutions.
• a device is programmed or configured to perform a method in accordance with the first aspect.
• a data storage medium is provided for storing a machine-executable program for performing a method in accordance with the first aspect.
• Figure 1 schematically illustrates videos to be compared and a stage in a comparison process
• FIG. 2 schematically illustrates a method in accordance with the invention
• Figure 3 schematically illustrates changes in intensity with time for one block
• FIG. 4 schematically illustrates linear segmentation
• Figure 5 schematically illustrates changes in intensity for compared videos
• Figure 6 schematically illustrates a matrix used in the method of Figure 2;
• FIG. 7 schematically illustrates steps in matching used in the method of Figure 2;
• FIG 8 schematically illustrates steps in matching used in the method of Figure 2;
• FIG. 9 schematically illustrates an apparatus in accordance with the invention.
• a query video 1 comprising a plurality of frames is to be compared one or more target videos to determine if they are duplicates.
• each frame in the query video 1 is partitioned into Nl x N2 blocks.
• the mean intensity value for each block is calculated at 3.
• the calculated mean intensity value is plotted against frame number to produce a query time series at 4.
• all blocks are processed provide Nl x N2 time series associated with the video 1.
• selected blocks are involved, thus resulting in fewer than Nl x N2 time series being produced.
• a target video 5 shown in Figure 1 is based on the query video 1 but has been modified with histogram equalization, added brightness and border and frame deletion.
• target time series shown at 6 are obtained. It can be seen that the changes in intensity for blocks from the target video 5 are generally similar in form to those of the video 1. For example, at frame number 806 for the query time series at 4, the mean intensity for one block increases while that of another decreases so that they cross over. A similar cross over can be seen at frame 739 for the target time series at 6.
• the next step at 7 in Figure 2 is to capture information provided by temporal changes in the query and target time series by using piecewise linear segmentation techniques.
• segmenting the time series the video is compressed and most of the essential information in the temporal changes of video intensities is captured. Due to user modification, video distortion and format conversions, one would not expect find exact matches in video duplicate detection and ignoring minor changes of temporal intensities makes the video duplicate detection process relatively insensitive to noise.
• Figure 3 a illustrates variation in mean intensity for part of one time series such as that shown at 4 or 6 in Figure 1.
• Figure 3b illustrates the part of the time series shown in Figure la after linear segmentation has been applied.
• a Bottom-Up algorithm is used to segment the time series.
• the Bottom-Up approach is a well-known approximation algorithm in time series. It starts from the finest possible approximation and iteratively merges segments until a stopping criterion is met. In this case, linear interpolation is used rather than linear regression to find the approximating line since linear interpolation can be obtained in constant time with low computational complexity.
• the quality of fit for a potential segment is evaluated using residual error. A residual error is calculated by taking all the vertical differences between the best-fit line and the actual data points, squaring them and then summing them together.
• the fast linear segmentation of the time series is achieved by an interpolation method using extraction of major maxima and major minima points as extrema points.
• Figure 4a shows a linear approximation using maxima and minima points.
• Jump points correspond to rapid changes in values, such as, for example, a jump up or down, within a short time distance. For intensity curves of video block series, these jumps typically indicate shot boundaries, caused by hard cuts or fades in/out.
• the linear segmentation technique is extended to also include jump points so that the extrema points used in the linear segmentation method are maxima points, minima points and jump points, as illustrated in Figure 4b.
• inclines/declines in the time series are selected at 9 as providing significant video signatures. This enables the search space for aligning linear segments to be reduced.
• linear segments with longer distance and deeper height usually represent conspicuous changes in a scene. They are therefore chosen as major inclines.
• Matching consecutive major inclines indicates video copies following similar behavior with the same sequence of major scene changes.
• linear segments of deep heights but of very short lengths are typically associated with shot boundaries, such as hard cuts or fades. Such linear segments often contain less information than those representing changes within a scene.
• a shot boundary can be determined if the linear segments from all partitioned blocks have deep heights within a same short distance occurring at a same time (i.e. the same starting frame IDs). Those linear segments representing shot boundaries are ignored in the process of selecting major inclines.
• the major inclines/declines of a query video and a target video are compared, as illustrated in Figure 5, to find approximate alignments with consecutive matched inclines/declines that are likely to lead to a successful alignment.
• an Ml by M2 matrix is generated, where Ml and M2 are the lengths of the major inclines/declines sequences under comparison. If two major inclines/declines at i and j match, value "1" is put in matrix (i, j).
• ratioi 0.9.
• the minimal distance between the two corresponding frame sequences is at most the threshold constant dist when "sliding* the shorter sequence along the longer sequence, where p ranges over the beginning of the sliding frame position in the longer video.
• the ordinal signature measurement calculates the distance between two frame sequences F ⁇ and F 2
• L - , is the length of the shorter sequence.
• Reward scores are assigned to diagonal matched lines and penalty scores to gaps, that is, mismatches, when connecting neighboring diagonal lines.
• a score is obtained by adding the reward scores of each of the connected diagonals and subtracting the gap penalties. If the score of a linked approximate alignment exceeds a given threshold, a check is made to determine if the previously ignored initial short aligned segments around the linked segments can be joined to form an approximate alignment with gaps, as shown in Figure 8 (d). Finally, the local approximate alignments having final scores exceeding a threshold are selected for further examination.
• the next stage at 15 is to conduct fine-grain alignment of all intensity linear segments of compared videos by applying the Smith- Waterman algorithm. Based on the approximate alignments of major inclines/declines found previously, lists of linear intensity segments that could lead to successful alignment can be determined.
• the Smith- Waterman algorithm only needs to examine a restricted range of linear segments.
• the Smith- Waterman algorithm uses edit distance to find the optimal alignment. It constructs a scoring matrix H as follows:
• x and y are the lists of linear segments that are potentially aligned
• M and N are the lengths of x and y sequences
• ⁇ ⁇ , , _y .) is a scoring scheme. If JC, and . , match, ⁇ ⁇ ⁇ is positive and if they don't match, it is negative. For insertion and deletion, ⁇ )( ⁇ , ,-) and ⁇ (-, _y ) are negative.
• the Smith- Waterman algorithm finds the local alignment by searching for the maximal score in matrix Hand then tracking back the optimal path depending on the direction of movement used to construct the matrix. It maintains this process until a score of 0 is reached.
• the video similarity distance is calculated at 16 by applying existing sequence matching techniques for the matched linear segments. In this embodiment, ordinal
• the changes of intensity values of partitioned blocks are first considered as time series. Then; the time series are segmented into a list of discrete linear representations. Local sequence alignment is performed of those linear segments to find optimal matching position. Then video similarity distance is calculated based on the potential alignment position. If the best matching similarity distance is less than a given threshold, two videos are considered as duplicate. To handle changes of frames, gaps, the result of frame insertions, deletions, and substitutions, are permitted to exist when in comparing linear sequence segments.
• a video management apparatus includes a database or store 19 which holds video files.
• the database 19 may be one which is generally accessible to users via the Internet or may, for example, be a library or other depository with restricted access. Other types of store or database may be used instead or in addition to these possibilities.
• a user transmits a video Q that he or she wants to add to the database 19 by submitting the video Q via a user interface 20.
• the video Q is sent to the video database 19 and also to a partitioner 21.
• the partitioner 21 partitions each frame of the video Q into Nl x N2 blocks.
• a calculator 22 calculates the mean intensity values for each of the blocks.
• mean intensity value data is received by a segmenter 23 from the calculator 22.
• the segmenter 23 segments the changes of mean intensities of each block.
• a sorter 24 then sorts the linear segments from all blocks based on the segment starting frame IDs into a sorted list.
• a selector 25 receives the sorted list and selects major inclines/major declines from the sorted list.
• an aligner 26 attempts to find an approximate match between the selected major inclines and major declines of the query video and those of one or more target videos that have undergone similar processing. The results are tested by a first comparator 27. If there is no similarity, judged against a given threshold parameter, then the query video and target video or videos are deemed to not be duplicates and the duplication detection process stops at 28.
• a banded Smith- Waterman algorithm is applied by processor 29 and the results applied to a similarity distance calculator 30.
• the output of the similarity distance calculator 30 is checked against a given threshold by a second comparator 31. If there is insufficient similarity, the compared videos are deemed not to be duplicates and the process stops at 32.
• a frame matcher 33 checks unmatched frame positions for video insertions, deletions or substitutions.
• the results of the duplicate detection process are sent to the video database 19 to be used in managing the stored videos. If the query video is not found to be a duplicate, the video database 19 accepts it for storage. If the query video is found to be a duplicate, then in one embodiment, the video database 19 rejects it with or without a message to the user to inform them.
• the query video is found to be a duplicate, it is accepted into the video database 19 but it is denoted as a duplicate, preferably with a reference to the target video that it matches.
• Duplicate videos may be collected together in a group. When a search performed on the database calls up one of the group, other group members may be suppressed from the search results or are given a lower ranking in the search results than they would otherwise merit, so that any duplicates tend to be presented after other non-duplicates.
• the video management apparatus of Figure 9 may be modified so that videos held in the video database 19 are partitioned and processed at 21 and 22 prior to the query video being submitted.
• data obtained when a video is submitted to be examined for duplicates may be retained and sent to be stored at the video database 19. If that video is subsequently not accepted into the database 19, the data is deleted. When the video is accepted into the database, the data associated with it is retained and is available for use in the aligner 26.
• videos in the video database 19 may be partitioned and processed in Stage 1 and Stage 2 without necessarily having been used in testing for duplicates. For example, the data processing may be carried out as part of a preparation phase before opening the database to receive new videos.
• processors may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software.
• the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared.
• explicit use of the term "processor” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and non volatile storage.
• DSP digital signal processor
• ASIC application specific integrated circuit
• FPGA field programmable gate array
• ROM read only memory
• RAM random access memory
• non volatile storage Other hardware, conventional and/or custom, may also be included.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Library & Information Science (AREA)
• Theoretical Computer Science (AREA)
• Data Mining & Analysis (AREA)
• Databases & Information Systems (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Image Analysis (AREA)

Priority Applications (4)

Applications Claiming Priority (4)

Publications (2)

Family

ID=45922716

Family Applications (1)

Country Status (5)

Cited By (3)

Families Citing this family (5)

Family Cites Families (7)

• 2012
  • 2012-01-04 WO PCT/IB2012/000269 patent/WO2012093339A2/en active Application Filing
  • 2012-01-04 CN CN201280011854.9A patent/CN103430175B/zh not_active Expired - Fee Related
  • 2012-01-04 JP JP2013547935A patent/JP5685324B2/ja not_active Expired - Fee Related
  • 2012-01-04 EP EP12711444.5A patent/EP2661710A2/en not_active Withdrawn
  • 2012-01-04 KR KR1020137017739A patent/KR101556513B1/ko not_active IP Right Cessation

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events