KR102004929B1 - System and method for searching similarity of multimedia files - Google Patents

Abstract

The present invention relates to a multimedia file similarity retrieval system and method, and a multimedia file similarity retrieval system according to an exemplary embodiment of the present invention includes a video shot classifying unit for classifying consecutive similar frames in a multimedia file into one video shot; A representative frame selection unit for selecting a predetermined frame among the frames of the video shot classified by the video shot classification unit as a representative frame; A fingerprint application unit for extracting video features by applying a video fingerprint technique to a representative frame selected by the representative frame selection unit; And a similarity measuring unit for comparing the video characteristic of the original multimedia file extracted by the fingerprint application unit with the video characteristic of the changed multimedia file to thereby measure the similarity of the changed multimedia file. Thus, illegal distribution And can determine whether or not it has changed.

Description

{System and method for searching similarity of multimedia files}

The present invention relates to a multimedia file similarity retrieval system and method, and more particularly, to a multimedia file similarity retrieval system and method based on media recognition information detection and video fingerprint-based matching.

The main method of file forensics is file similarity and redundancy analysis. File similarity and duplicate detection techniques are important mechanisms for file synchronization and network deduplication.

The block hashing technique is the most important factor used in file similarity and redundancy search. It divides a file into a number of small blocks and extracts a hash key for identifying each block using a hash function. Dividing a file into small blocks is called chunking, and the divided unit becomes a chunk. Chunk-based deduplication is a very effective technique for deduplication.

There are two chunking methods of dividing a file into several blocks: fixed-length chunking and variable-length chunking.

Fixed-length chunking divides a file into several blocks of fixed size and then applies a hash function to extract a hash key from each block. This is easier to implement than variable-length chunking in a very simple way. However, the disadvantage of this method is that if some data is inserted or deleted at the block boundary, the result of the hash value can be changed to a completely different value. Therefore, this method is generally not expected to be good in terms of accuracy through similarity analysis.

Variable length chunking is achieved by dividing each block using a predefined special value called an anchor, thereby preventing the data moving problem of the fixed length chunking scheme. Thus, variable-length chunking is more flexible than fixed-length chunking, but requires more computation time to achieve good performance in finding file similarities.

That is, the byte-based comparison and block-based technique used for measuring the file similarity has a too large overhead for the amount of data to be processed and the processing time to be applied to a large-capacity file such as multimedia. Furthermore, due to the nature of multimedia, the accuracy of similarity measurement can be significantly reduced because the content of the byte can be completely different even though it is a file of the same image when viewed with eyes. There has been a problem in that it is a very inefficient similarity retrieval method when the existing method used in retrieving the file similarity is applied to the multimedia file as it is.

It is another object of the present invention to provide a multimedia file similarity search system and method based on media recognition information detection and video fingerprint based matching.

According to an aspect of the present invention, there is provided a multimedia file similarity searching system comprising: a video shot classifying unit for classifying consecutive similar frames in a multimedia file into one video shot; A representative frame selection unit selecting a predetermined frame among the frames of the video shot classified by the video shot classification unit as a representative frame; A fingerprint application unit for extracting video features by applying a video fingerprint technique to a representative frame selected by the representative frame selection unit; And a similarity measuring unit for comparing the video characteristic of the original multimedia file extracted by the fingerprint application unit and the video characteristic of the changed multimedia file to measure the similarity.

The video shot classification unit may detect a video shot boundary by comparing histogram similarities based on a color space of hue, saturation, and brightness (HSV) in detecting a video shot boundary for classifying the video shots.

The representative frame selection unit may select a first frame of each video shot as a representative frame.

The fingerprint application unit may extract a video feature by applying a perception image hashing technique to a representative frame selected by the representative frame selection unit using a video fingerprint technique.

The perceptual image hashing technique can obtain low frequency DCT coefficients related to the perception image using the DCT based hash technique.

The multimedia file similarity retrieval method according to an exemplary embodiment of the present invention includes a video shot classification step of classifying consecutive similar frames in a multimedia file into one video shot; A representative frame selecting step of selecting a predetermined one of the frames of the video shot classified in the video shot sorting step as a representative frame; A fingerprint applying step of extracting a video feature by applying a video fingerprint technique to the representative frame selected in the representative frame selection step; And a similarity measuring step of comparing the video characteristic of the original multimedia file extracted in the fingerprint application step with the video characteristic of the changed multimedia file to measure the similarity, thereby achieving the above object.

According to the above-described configuration, the present invention can judge illegally distribution and change of original contents using similarity search between multimedia files.

In addition, the present invention can ensure independence from changes in general video processing steps, including lossy compression, resizing, frame rate or format conversion, etc., through boundary detection of video shots.

Further, since the present invention uses a video fingerprint, the storage complexity can be lowered and the comparison matching time can be shortened.

FIG. 1 is a schematic block diagram of a multimedia file search system according to an embodiment of the present invention. Referring to FIG.
2 is a diagram illustrating boundary detection of a video shot according to an embodiment of the present invention.
3 is a view for explaining selection of a representative frame according to an embodiment of the present invention.
4 is a diagram for explaining generation of a hash list according to an embodiment of the present invention.
5 is a diagram illustrating an example of a comparison between a hash list of original multimedia generated according to an embodiment of the present invention and a modified hash list of multimedia.
6 is a flowchart illustrating a multimedia file searching method according to an embodiment of the present invention.

Embodiments of a multimedia file similarity retrieval system and method according to the present invention will now be described with reference to the accompanying drawings. In the following description of the present invention, it is to be understood that the terminology used herein is for the purpose of describing particular and exemplary embodiments of the invention, Will be.

FIG. 1 is a schematic block diagram of a multimedia file search system according to an embodiment of the present invention. FIG. 2 is a view for explaining boundary detection of a video shot according to an embodiment of the present invention. 4 is a view for explaining generation of a hash list according to an embodiment of the present invention, and FIG. 5 is a flowchart illustrating a method of generating a hash list according to an embodiment of the present invention FIG. 4 is a diagram illustrating a comparison between a hash list of original multimedia generated according to a modified multimedia hash list and a modified multimedia hash list.

1, the multimedia file search system includes a video shot classification unit 110, a representative frame selection unit 120, a fingerprint application unit 130, and a similarity measurement unit 140.

The video shot classifying unit 110 classifies consecutive similar frames in the multimedia file into one video shot. The video shot classification unit 110 detects a boundary of a video shot by comparing histogram similarities based on a color space of hue, saturation, and brightness (HSV) in detecting a video shot boundary for classifying video shots.

Boundary detection of video shots is the first step in performing video analysis, and it is even easier to construct a multimedia file search system without reference to all video frames. The boundary detection of this video shot is to detect the boundary between video shots to make the video data more compact. Based on the visual similarity and temporal relationship, the video is segmented into similar frame segments to find the beginning and end of the video shot.

Video can be divided into several video shots by grouping similar frames as a series of frames. From the segmented video shots, the first frame of each video shot can be used as the boundary anchor of each segmented video chunk.

The following four approaches can be used for boundary detection of video shots.

· Color / Luminance histogram: Computes the color of every pixel in each frame and uses this value to create a histogram. The histograms of the adjacent frames are successively compared with each other.

· Luminance values: Compares the luminance values but is affected by lighting changes. To overcome these drawbacks, one or more average values are used in the appropriate color space.

· Edges: Find edges in each frame. The edge information of the continuous frame is compared, and the detection of the change by the comparison means that there is a boundary of the video shot.

· Motion: Utilizes object motion information in video. A sudden change in motion means that a change in video shot occurred.

The detection algorithm of video shots may be designed by combining these approaches. In addition, algorithms for object recognition such as Scale Invariant Feature Transform (SIFT) or Speed Up Robust Features (SURT) can be used for boundary detection. The boundary detection of video shots is the most important process for content-based video analysis and has a direct impact on the accuracy and efficiency of media recognition information retrieval.

In one embodiment of the present invention, a video shot boundary detection approach is to detect the boundaries of a video shot by comparing histogram similarities based on the HSV (Hue Saturation Value) color space of a video frame. The HSV technique separates luminance or image brightness from color information. After eliminating the value of intensity, HSV can be used to handle many lighting changes. In order to measure the similarity between frame_i and frame_ (i + 1) of successive frames, one can choose one of several methods to perform histogram matching such as correlation, chi-square, histogram intersection and Bhattacharyya. An example of one embodiment of the present invention is the Chi square distance.

FIG. 2 is a diagram illustrating boundary detection of a video shot in accordance with an embodiment of the present invention. This provides independence from changes in common video processing steps, including lossy compression, resizing, frame rate, or format conversion.

The representative frame selection unit 120 selects a predetermined frame among the frames of the video shot classified by the video shot classification unit 110 as a representative frame. In this case, the representative frame selecting unit 120 can select the first frame of each video shot as the representative frame.

Selecting the representative frame is to find the most representative frame in the video. By selecting a representative frame in each video segment, redundant data in the video can be removed. Selecting a representative frame can reduce a large amount of video data without sacrificing the comparison factor. The representative frame selection method can be considered simply to select an I frame from a video stream, but there may be some limitations in selecting an I frame as a representative frame. The original video can be re-encoded or edited by cropping or connecting a partial frame. Even though the original and edited video contents overlap, the two versions can have completely different binaries. In this case, the position of the I frame may be selected differently from the original version. Therefore, the I frame of the edited video may not be the representative frame since it may be different from the I frame of the original video.

In one embodiment of the present invention, a representative frame is selected based on the boundary of video shots classified by the video shot classification unit 110. The selection of the representative frame is mainly performed by combining the first frame, the intermediate frame and the last frame of each video shot or the first frame and the last frame, but in the embodiment of the present invention, only the first frame is selected as the representative frame .

FIG. 3 is a view for explaining selection of representative frames according to an embodiment of the present invention.

The fingerprint application unit 130 applies a video fingerprint technique to the representative frame selected by the representative frame selection unit 120 to extract video features. The fingerprint application unit 130 extracts a video feature by applying a perceptual image hashing technique to a representative frame selected by the representative frame selection unit 120 using a video fingerprint technique. The perceptual image hashing technique uses a DCT-based hash technique To obtain a low-frequency DCT coefficient related to the perception image.

The main function of the fingerprint application technique is to generate a video fingerprint and measure the performance based on the video fingerprint matching. Using a video fingerprint reduces storage complexity and speeds up comparison matching.

Fingerprinting techniques use cryptographic hash functions such as MD5, SHA-1, and SHA-256. A cryptographic hash function is a mathematical function that takes input data and returns a short binary string of fixed size. The value returned by the hash function is called a hash value, a message digest, or a digital fingerprint.

Cryptographic hashing techniques are commonly used to identify a particular entity (or chunk) in binary data, such as an executable file. Since cryptographic hashing is very sensitive to small changes, there are significant difficulties in applying cryptographic hashing to multimedia content with different presentation versions, such as format conversion and compression. That is, the multimedia file may experience various distortion phenomena such as resizing, formatting, and changing the frame rate. Therefore, it may be pointless to directly compare the data at the binary level among the multimedia files to grasp the change contents and the similarity.

Therefore, a robust video hashing technique for video fingerprinting is needed to identify or authenticate different versions of multimedia content. In an embodiment of the present invention, a perceptual image hashing technique is applied. There are various types of perceptual image hashing algorithms such as BMB (Block Mean Value Based) hash, DCT (DCT based) hash, MH (Marr-Hildreth operator based) hash and RADIAL (RADIAL variance based) hash.

In one embodiment of the present invention, a DCT hash algorithm is selected to focus on "accuracy" which is one of the system design goals for video fingerprint generation. This is because the low-frequency DCT coefficients based on the characteristics of the DCT can be the "perceptual entities" of the image. The steps of the algorithm used to generate the video fingerprint of the DCT hash are as follows.

The fingerprint application unit 130 converts the representative frame selected by the representative frame selecting unit 120 from RGB to YCbCr to remove unnecessary high frequencies. Then, the fingerprint application unit 130 uses a median filter for noise reduction of the Y matrix. Then, the fingerprint application unit 130 normalizes the image passed through the median filter to a predetermined size. For example, you can adjust the image size to 32 x 32. The fingerprint application unit 130 obtains a DCT coefficient by applying DCT to a Y matrix of a predetermined size, and generates a hash value for the representative frame by applying a hash to the obtained DCT coefficient.

Fingerprint based matching can eliminate the overhead of byte-to-byte comparison. Through boundary detection of the video shots, the video is divided into several chunks and representative frames are selected as the keys of each chunk. A video fingerprint is generated for each representative frame, and a video fingerprint is a hash value extracted from each representative frame as a hash function. To calculate the hash value of each chunk, a hash list is generated using Hash64, which is a 64-bit hash.

A diagram for explaining generation of a hash list according to an embodiment of the present invention is shown in FIG. The fingerprint-based matching technique according to the present invention is a small fingerprint, so there is less storage overhead and more efficient coherence, and there is a lower computational overhead for detecting similarity.

The similarity measuring unit 140 compares the video characteristic of the original multimedia file 10 extracted from the fingerprint application unit 130 with the video characteristic of the changed multimedia file 20 to measure the similarity.

The similarity measuring unit 140 checks the similarity and the redundancy using the hash list to proceed with forensic examination of the multimedia file. FIG. 5 illustrates an example of a comparison between a hash list of original multimedia generated according to an exemplary embodiment of the present invention and a modified multimedia hash list. The basic idea of fingerprint based matching is to find the similarity and redundancy by comparing the original multimedia and the changed fingerprint of the multimedia.

6 is a flowchart illustrating a multimedia file searching method according to an embodiment of the present invention.

 When the original multimedia file 10 is provided (S602), the video shot classification unit 110 collects consecutive similar frames from the original multimedia file 10 into one video shot. In this case, the video shot classification unit 110 detects the boundary of the video shot by comparing the histogram similarity based on the color space of color, saturation, and brightness (HSV) in detecting the video shot boundary for classifying the video shot, (S604).

The representative frame selection unit 120 selects a predetermined frame among the frames of the video shot classified by the video shot classification unit 110 as a representative frame (S606). In this case, the representative frame selecting unit 120 can select the first frame of each video shot as the representative frame.

The fingerprint application unit 130 extracts video features by applying a video fingerprint technique to the representative frame selected by the representative frame selection unit 120 (S608). The fingerprint application unit 130 extracts a video feature by applying a perceptual image hashing technique to a representative frame selected by the representative frame selection unit 120 using a video fingerprint technique. The perceptual image hashing technique uses a DCT-based hash technique To obtain a low-frequency DCT coefficient related to the perception image.

The fingerprint application unit 130 generates a hash resist using the extracted video fingerprint for each representative frame and stores the hash resist of the original multimedia file 10 in operation S610.

When the changed multimedia file 10 is provided (S612), the video shot classifying unit 110 classifies consecutive similar frames in the changed multimedia file 20 into one video shot. In this case, the video shot classification unit 110 detects the boundary of the video shot by comparing the histogram similarity based on the color space of color, saturation, and brightness (HSV) in detecting the video shot boundary for classifying the video shot, (S614).

The representative frame selection unit 120 selects a predetermined frame among the frames of the video shot classified by the video shot classification unit 110 as a representative frame (S616). In this case, the representative frame selecting unit 120 can select the first frame of each video shot as the representative frame.

The fingerprint application unit 130 extracts video features by applying a video fingerprint technique to the representative frame selected by the representative frame selection unit 120 (S618). The fingerprint application unit 130 extracts a video feature by applying a perceptual image hashing technique to a representative frame selected by the representative frame selection unit 120 using a video fingerprint technique. The perceptual image hashing technique uses a DCT-based hash technique To obtain a low-frequency DCT coefficient related to the perception image.

The fingerprint application unit 130 generates hash resists using the extracted video fingerprints for each representative frame and stores the hash resists of the changed multimedia file 20 in operation S620.

The similarity measurement unit 140 compares the hash resist file in which the video characteristic of the original multimedia file 10 extracted from the fingerprint application unit 130 is stored with the hash resist file in which the video characteristic of the changed multimedia file 20 is stored And the degree of similarity is measured (S622).

The scope of protection of the present invention should be interpreted according to the claims. It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention as defined by the appended claims. It should be interpreted that it is included in the scope of right.

10: original multimedia file 20: changed multimedia file
110: video shot classification unit 120: representative frame selection unit
130: finger print applying unit 140: similarity measuring unit

Claims (7)

A multimedia file similarity search system for comparing a multimedia file of an original and a multimedia file of the original multimedia file,
A video shot classifying unit for classifying consecutive similar frames in the original multimedia file or the modified multimedia file into one video shot;
A representative frame selection unit selecting a predetermined frame among the frames of the video shot classified by the video shot classification unit as a representative frame;
A fingerprint application unit for extracting video features by applying a video fingerprint technique to a representative frame selected by the representative frame selection unit; And
A similarity measuring unit for comparing the video feature extracted from the representative frame of the original multimedia file with the video feature extracted from the representative frame of the changed multimedia file in the fingerprint application unit in the fingerprint application unit, The multimedia file similarity retrieval system comprising: The method according to claim 1,
Wherein the video shot classification unit detects a video shot boundary by comparing histogram similarities based on a color space of hue, saturation, and brightness (HSV) in detecting a video shot boundary for classifying the video shots. File similarity search system. The method according to claim 1,
Wherein the representative frame selection unit selects a first frame of each video shot as a representative frame. 4. The method according to any one of claims 1 to 3,
Wherein the fingerprint application unit extracts a video feature by applying a perception image hashing technique to a representative frame selected by the representative frame selection unit using a video fingerprint technique. 5. The method of claim 4,
Wherein the perceptual image hashing technique obtains a low frequency DCT coefficient related to a perceptual image using a DCT based hash technique. A multimedia file similarity searching method for comparing a similarity between a multimedia file of an original and a multimedia file edited from the original multimedia file,
A video shot classification step of classifying consecutive similar frames from the original multimedia file or the modified multimedia file into one video shot;
A representative frame selecting step of selecting a predetermined one of the frames of the video shot classified in the video shot sorting step as a representative frame;
A fingerprint applying step of extracting a video feature by applying a video fingerprint technique to the representative frame selected in the representative frame selection step; And
A similarity measure step of comparing the video feature extracted from the representative frame of the original multimedia file with the video feature extracted from the representative frame of the changed multimedia file in the fingerprint application step in the fingerprint application step, Wherein the multimedia file similarity searching method comprises: The method according to claim 6,
Wherein the fingerprint applying step extracts a video feature by applying a perceptual image hashing technique to a representative frame selected in the representative frame selection step using a video fingerprint technique.

Images