CN103257992A - Method and system for retrieving similar videos - Google Patents

Abstract

The present invention discloses a similar video retrieval method and system, wherein the method includes: using a color space model to calculate the feature vector of each video key frame set in a video library; mapping the feature vector through a hash function, according to the mapping Construct an index in the interval of the vector value after each motion vector mapping; determine the serial number of the corresponding index according to the interval of each vector value mapped by each motion vector of the video to be retrieved, extract the video information corresponding to all the feature vectors under the corresponding index, and according to the video to be retrieved and the corresponding index The number of similar key frames in the extracted video is used to calculate the similarity; the video corresponding to the calculation result greater than the threshold is output as the retrieval result. The efficiency and accuracy of similar video retrieval are improved by adopting the method disclosed in the invention.

Description

A method and system for similar video retrieval

technical field

The invention relates to the field of computer application technology, in particular to a method and system for similar video retrieval.

Background technique

With the rapid development of network streaming media technology, more and more videos and multimedia data appear on the Internet. These videos grow geometrically and become the mainstream of information release and entertainment on the Internet. In the face of massive multimedia data, how to quickly find similar or even identical videos from massive videos has become a hot research issue.

Traditional text-based video retrieval methods have many shortcomings. It mainly completes the retrieval of similar videos by indexing the surrounding text of the video, but the video content is different from the text content, and there is a big difference between the video data and its semantic information, and the way of relying solely on the surrounding text cannot accurately describe the video Content also ignores multimedia features and information such as video visual perception. On the other hand, using a purely text-based method requires manual labeling of massive video data in the network, which is a huge workload. At the same time, labeling videos is somewhat subjective, and different people may have different understandings of the same video. As a result, the accuracy of similarity retrieval is very low, and the effect is difficult to be further improved.

Contents of the invention

The purpose of the present invention is to provide a method and system for similar video retrieval, which improves the efficiency and accuracy of similar video retrieval.

A method for similar video retrieval, the method comprising:

Using the color space model to calculate the feature vector of each video key frame set in the video library;

Mapping the feature vector through a hash function, and constructing an index according to the interval where the mapped vector value is located;

Determine the serial number of the corresponding index according to the range of each vector value mapped by each motion vector of the video to be retrieved, extract the video information corresponding to all feature vectors under the corresponding index, and extract similar key frames from the video to be retrieved and the video extracted under the corresponding index The number of similarity calculations;

Output the video corresponding to the calculation result greater than the threshold as the retrieval result.

A system for similar video retrieval, the system comprising:

Feature vector calculation module, for utilizing the color space model to calculate the feature vector of each video key frame set in the video storehouse;

An index construction module, configured to map the feature vector through a hash function, and construct an index according to the interval of the mapped vector value;

The similarity calculation module is used to determine the serial number of the corresponding index according to the range of each vector value mapped by each motion vector of the video to be retrieved, extract the video information corresponding to all feature vectors under the corresponding index, and extract the corresponding index according to the video to be retrieved and the corresponding index. The number of similar key frames in the output video is used to calculate the similarity;

The retrieval result output module is configured to output the video corresponding to the calculation result greater than the threshold as the retrieval result.

It can be seen from the above-mentioned technical solution provided by the present invention that the feature vectors are extracted through the content of the video, and these feature vectors are used to establish indexes, and based on this, similarity calculations are performed with the videos to be retrieved; Retrieval of similar videos in the library.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings on the premise of not paying creative efforts.

FIG. 1 is a flow chart of a method for similar video retrieval provided by Embodiment 1 of the present invention;

FIG. 2 is a flowchart of a clustering algorithm provided by Embodiment 1 of the present invention;

FIG. 3 is a schematic diagram of a similar video retrieval system provided by Embodiment 2 of the present invention;

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Embodiment one

FIG. 1 is a flowchart of a similar video retrieval method provided by Embodiment 1 of the present invention. As shown in Figure 1, it mainly includes the following steps:

Step 11, extract the key frame set of each video in the video library.

Since the difference between adjacent frames in the video is not large, if the method of extracting key frames at equal time intervals is used, the extraction frequency is too high, which will cause a huge amount of feature data and affect the feature extraction speed; and the difference between adjacent frames is not large, and the similarity is too large. However, if the extraction frequency is too low, although the amount of feature data can be reduced and the feature extraction speed can be accelerated, the extracted feature set cannot fully reflect the overall features of the video, thereby reducing the accuracy of retrieval.

Therefore, for each video data in the video library, the key frame is first extracted, for example, a clustering algorithm is used to complete the edge detection task of the video shot by judging whether the features between adjacent image frames change drastically, namely Use the most streamlined feature set to fully reflect the overall characteristics of the video.

As shown in Figure 2, the clustering algorithm mainly includes the following steps:

Step 111, start extracting video frames from the nth frame, and calculate the distance between each subsequent frame and its frame centering on this frame.

This step is based on the algorithm of the distance between frames to determine whether there is a sudden change in the content between adjacent frames. Considering that there is noise information in the first few frames of the video, it is filtered, and the video frame is extracted from the nth frame, and the frame Calculate the distance between each subsequent frame and the nth frame as the center.

Step 112 , judging whether the inter-frame distance between the n+i (i>0)th frame and the nth frame is greater than a threshold, if yes, go to step 113 ; otherwise, go to step 114 .

Step 113 , take the nth frame as a key frame, and turn to step 114 .

If the interframe distance between the n+i frame and the n frame is greater than the threshold, the n+i frame is a sudden change frame relative to the n frame, and the n frame to the n+i frame (n+i -1) As the same category, and select the nth frame as keyframe extraction.

Step 114, judging whether the number of remaining frames is less than S.

If the number of remaining frames of the current video is less than S (for example, 2), the key frame extraction operation is ended; otherwise, the above steps can be repeated until the number of remaining frames is less than S.

It can be seen from the above steps that the key frames selected in the clustering can better reflect the content of the video footage and have low computational complexity. It should be noted that the clustering algorithm described above is only an example, and users can also use other methods to select representative key frames from the video in actual work.

Step 12, using the color space model to calculate the feature vector of each video key frame set in the video library.

Exemplarily, the feature vector can be calculated through the HSV (hue-saturation-luminance) color space model, so as to fully reflect the ability of color space distribution and information changes, enhance the ability to distinguish between pictures, and improve retrieval performance.

After the image of the key frame is converted through the HSV color space model, the color of each pixel is represented by the value of hue h, saturation s, and brightness v.

In order to improve the accuracy of the color model, the colors similar to black and white in the image can be treated as the same color respectively, that is (1) black area: all colors with v<15% are classified as black, let h=0, s=0, v=0; (2) White area: all s<10% and v>80% colors are classified as white, let h=0, s=0, v=1; (3) Color area: located in The h, s, and v values of colors outside the black and white areas remain unchanged.

Further, in order to reduce the dimensionality of the histogram vector, the HSV space is properly quantized before the histogram is calculated to reduce the amount of calculation. For example, the three components of h, s, and v can be quantized at non-equal intervals according to human color perception, and a quantization standard table can be produced, and then the h, s, and v values of each pixel color in the image can be extracted separately, according to The quantization table obtains the quantized color h, s, and v values of each pixel, and calculates the HSV color histogram vector of the entire picture, that is, the feature vector of the key frame.

Step 13, building an index.

There are many video files in the database. Even if more redundant data is filtered out through the above two steps, the amount of calculation is still relatively large. Therefore, it is necessary to build an index to further reduce the amount of computation.

Exemplarily, a distributed LSH (Locality Sensitive Hash) structure based on memory constraints may be used for index construction.

The distributed LSH structure adopts the Master-Slaver (master-slave) architecture. The Master is responsible for maintaining the division strategy of the LSH table, and the Slaver (which can be used as a storage node) is responsible for maintaining the data in the LSH table.

Through the hash mapping function, the feature vector is mapped into a one-dimensional vector and stored in the hash table maintained by the corresponding slaver. Taking the feature vector b as an example, the master first divides the value range of the hash table maintained by each slaver, transforms the feature vector b through a hash function, and maps it to a one-dimensional space whose value is C, according to the hash table data maintained by each slaver node The range searches for the range to which C belongs, and the master assigns the vector and the mapped value to the hash bucket under the slaver to which it belongs. Each hash bucket has an independent number, and each hash bucket includes the video and frame numbers corresponding to each feature vector.

After each feature vector has passed the above operations, the construction of the distributed LSH structure index is completed.

In addition, the data information in the hash bucket can be maintained by multiple data servers and stored in memory, which overcomes the curse of dimensionality, reduces the complexity of retrieval, and is fast and reliable.

Step 14, retrieving similar videos.

The retrieval process mainly includes the following steps: 1) Process the video to be retrieved in the manner of step 11-step 12 to obtain the corresponding feature vector. 2) Each feature vector of the video to be retrieved is mapped through a hash function, and the master locates its slaver and hash bucket number according to the mapped value. 3) Extract the video information corresponding to all feature vectors in the hash bucket with a certain number, and count the number of key frames contained in each video. 4) According to the number of similar key frames in the video to be retrieved and the video extracted under the corresponding index and combined with the following formula, calculate the similarity between the video to be retrieved and the video to which the feature vector extracted in the hash bucket belongs:

$\mathrm{<span\; class="notranslate">\; sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\frac{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; KF</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&cap;</span>{\mathrm{<span\; class="notranslate">\; KF</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>}{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; KF</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; +</span>\frac{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; KF</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&cap;</span>{\mathrm{<span\; class="notranslate">\; KF</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>}{{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; KF</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; ;</span>$

Among them, sim(V _i ,V _j )∈[0,1]; |KF _i | is the number of key frames of the video V _i to be retrieved; |KF _j | is the video V to which the feature vector extracted from the hash bucket belongs The number of key frames of _j ; |KF _i ∩KF _j | indicates the number of key frames where V _i and V _j are similar.

It can be seen from the above that based on the distributed LSH index structure, data objects that are not similar and cannot become results can be filtered out, which can further reduce the amount of calculation.

Step 15, outputting the retrieval result.

According to the above formula, the correlation between the video to be retrieved and each video in the current hash bucket can be calculated. The larger the result, the higher the correlation. Therefore, the video corresponding to the calculation result greater than the threshold is output as the retrieval result.

The embodiment of the present invention uses the visual feature extraction technology, and based on the memory-constrained distributed hash structure to index the high-dimensional feature vector, and calculates the similarity of the video to be retrieved according to the returned similar key frame set, and realizes a large number of similar videos. quick search.

Through the above description of the implementation manners, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus a necessary general hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be CD-ROM, U disk, mobile hard disk, etc.), including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments of the present invention.

Embodiment two

Fig. 4 is a schematic diagram of a similar video retrieval system provided in Embodiment 2 of the present invention, as shown in Fig. 4, mainly including:

Feature vector calculation module 41, for utilizing the color space model to calculate the feature vector of each video key frame set in the video storehouse;

An index construction module 42, configured to map the feature vector through a hash function, and construct an index according to the interval where the mapped vector value is located;

The similarity calculation module 43 determines the serial number of the corresponding index according to the interval of each vector value after each motion vector mapping of the video to be retrieved, extracts the video information corresponding to all feature vectors under the corresponding index, and extracts the corresponding index according to the video to be retrieved and the corresponding index. The number of similar key frames in the video is used to calculate the similarity;

The retrieval result output module 44 is configured to output the video corresponding to the calculation result greater than the threshold as the retrieval result.

The feature vector calculation module 41 includes:

The key frame set extraction module 411 is used to extract each video key frame set in the video library by a clustering algorithm, and this module includes: a clustering module 412, which is used to extract video frames from the nth frame, and use the The frame is the center to calculate the distance between subsequent frames and their frames; when the distance between the n+i frame and the n frame is greater than the threshold, the nth frame to the n+i-1th frame form a cluster, and the nth frame frame as a keyframe.

The feature vector calculation module 41 includes:

The image color conversion module 412 is used to convert the image color in the key frame by using the hue-saturation-brightness HSV color model;

The image feature vector calculation module 413 is used to use the HSV space to perform non-equal interval quantization on the image after color conversion, and perform histogram calculation to obtain the feature vector of the image.

The index building module 42 includes:

The vector value acquisition module 421 is used to map the feature vector to a one-dimensional space through a hash function to obtain a corresponding vector value;

The feature vector distribution module 422 is used to determine the number of the storage node according to the interval where the vector value is located, and distribute the feature vector to the hash bucket corresponding to the storage node; the hash bucket contains each feature vector corresponding to Video and frame information.

Described similarity calculating module 43 comprises:

The information extraction module 431 is used to determine the corresponding storage node and the number of the corresponding hash bucket according to the interval of each vector value after each motion vector mapping of the video to be retrieved, and extract the corresponding video of all the feature vectors in the hash bucket of the corresponding number information;

The video similarity calculation module 432 is used to calculate the similarity between the video to be retrieved and the video to which the feature vector extracted from the hash bucket belongs, and its formula is:

$\mathrm{<span\; class="notranslate">\; sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\frac{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; KF</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&cap;</span>{\mathrm{<span\; class="notranslate">\; KF</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>}{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; KF</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; +</span>\frac{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; KF</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&cap;</span>{\mathrm{<span\; class="notranslate">\; KF</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>}{{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; KF</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; ;</span>$

Among them, |KF _i | is the number of key frames of the video V _i to be retrieved; |KF _j | is the number of key frames of the video V _j to which the feature vector extracted from the hash bucket belongs; |KF _i ∩KF _j | means The number of keyframes where V _i and V _j are similar.

It should be noted that, the specific implementation manners of the functions implemented by each processing unit included in the above apparatus have been described in detail in the foregoing embodiments, so details will not be repeated here.

The embodiment of the present invention uses the visual feature extraction technology, and classifies and indexes the high-dimensional feature vectors based on the memory-constrained distributed hash structure, and calculates the similarity of the videos to be retrieved according to the returned similar key frame set, realizing massive Quick retrieval of similar videos.

Those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional modules according to needs. The internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person familiar with the technical field can easily conceive of changes or changes within the technical scope disclosed in the present invention. Replacement should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (10)

1. the method for similar video retrieval is characterized in that this method comprises:

Utilize the color space model to calculate the eigenvector of each key frame of video set in the video library;

Shine upon described eigenvector by hash function, according to index building between the vector value location after the mapping;

According to the numbering of determining manipulative indexing between each vector value location after each motion vector mapping of video to be retrieved, extract the video information of all eigenvector correspondences under the manipulative indexing, carry out calculation of similarity degree according to the number of similar key frame in the video that extracts under video to be retrieved and the manipulative indexing;

To export as result for retrieval greater than the video of the result of calculation correspondence of threshold value.

2. method according to claim 1 is characterized in that, this method also comprises the step of extracting the key frame of video set by clustering algorithm, and its step comprises:

Extract frame of video since the n frame, and centered by this frame each frame of calculated for subsequent and its frame pitch from;

When the frame pitch of n+i frame and n frame from greater than threshold value the time, then n frame to the n+i-1 frame constitutes a cluster, and with the n frame as key frame.

3. method according to claim 1 is characterized in that, the described step of utilizing the color space model to calculate the eigenvector of each key frame of video set in the video library comprises:

Utilize tone saturation degree brightness hsv color model that the color of image in the key frame is changed;

Image after utilizing the HSV space to color conversion carries out unequal interval and quantizes, and carries out the eigenvector that histogram calculation obtains image.

4. method according to claim 1 is characterized in that, described step according to index building between the vector value location after the mapping comprises:

Eigenvector is mapped to the one-dimensional space by hash function, obtains corresponding vector value;

Determine affiliated memory node numbering according to the interval at vector value place, and eigenvector is assigned in the Hash bucket of this memory node correspondence; The information that comprises video and the frame of each eigenvector correspondence in the described Hash bucket.

5. method according to claim 4 is characterized in that, described step of carrying out similarity calculating comprises:

According to the memory node under determining between each vector value location after the mapping of each motion vector of video to be retrieved and the numbering of corresponding Hash bucket, and extract the information of the corresponding videos of all eigenvectors in the reference numeral Hash bucket;

The similarity of video under the eigenvector that calculates video to be retrieved and from the Hash bucket, extract, its formula is:

$\mathrm{sim}(V_i,V_j)=(\frac{|{\mathrm{KF}}_i\&cap;{\mathrm{KF}}_j|}{\left|{\mathrm{KF}}_i\right|}+\frac{|{\mathrm{KF}}_i\&cap;{\mathrm{KF}}_j|}{{|\mathrm{KF}}_j|})/2;$

Wherein, | KF _i| be video V to be retrieved _iQuantity of key frames; | KF _j| be video V under the eigenvector that extracts in the Hash bucket _jQuantity of key frames; | KF _i∩ KF _j| expression V _iAnd V _jThe number of similar key frame.

6. the system of similar video retrieval is characterized in that this system comprises:

The eigenvector computing module is used for utilizing the color space model to calculate the eigenvector of each key frame of video set of video library;

The index construct module is used for shining upon described eigenvector by hash function, according to index building between the vector value location after the mapping;

Similarity calculation module, be used for according to the numbering of determining manipulative indexing between each vector value location after each motion vector mapping of video to be retrieved, extract the video information of all eigenvector correspondences under the manipulative indexing, carry out calculation of similarity degree according to the number of similar key frame in the video that extracts under video to be retrieved and the manipulative indexing;

The result for retrieval output module is used for and will exports as result for retrieval greater than the video of the result of calculation correspondence of threshold value.

7. system according to claim 6 is characterized in that, described eigenvector computing module comprises:

Key frame set extraction module be used for extracting the key frame of video set by clustering algorithm, and this module comprises: the cluster module, be used for extracting frame of video since the n frame, and centered by this frame each frame of calculated for subsequent and its frame pitch from; When the frame pitch of n+i frame and n frame from greater than threshold value the time, then n frame to the n+i-1 frame constitutes a cluster, and with the n frame as key frame.

8. system according to claim 6 is characterized in that, described eigenvector computing module comprises:

The color of image modular converter is used for utilizing tone saturation degree brightness hsv color model that the color of image of key frame is changed;

Characteristics of image vector computing module, the image after being used for utilizing the HSV space to color conversion carries out unequal interval and quantizes, and carries out the eigenvector that histogram calculation obtains image.

9. system according to claim 6 is characterized in that, described index construct module comprises:

The vector value acquisition module is used for eigenvector is mapped to the one-dimensional space by hash function, obtains corresponding vector value;

The eigenvector distribution module is used for determining affiliated memory node numbering according to the interval at vector value place, and eigenvector is assigned in the Hash bucket of this memory node correspondence; The information that comprises video and the frame of each eigenvector correspondence in the described Hash bucket.

10. system according to claim 9 is characterized in that, described similarity calculation module comprises:

Information extraction modules is used for according to the memory node under determining between each vector value location after each motion vector mapping of video to be retrieved and the numbering of corresponding Hash bucket, and extracts the information of the corresponding video of all eigenvectors in the reference numeral Hash bucket;

The video similarity calculation module be used for to be calculated the similarity of video under video to be retrieved and the eigenvector that extracts from the Hash bucket, and its formula is:

$\mathrm{sim}(V_i,V_j)=(\frac{|{\mathrm{KF}}_i\&cap;{\mathrm{KF}}_j|}{\left|{\mathrm{KF}}_i\right|}+\frac{|{\mathrm{KF}}_i\&cap;{\mathrm{KF}}_j|}{{|\mathrm{KF}}_j|})/2;$

Wherein, | KF _i| be video V to be retrieved _iQuantity of key frames; | KF _j| be video V under the eigenvector that extracts in the Hash bucket _jQuantity of key frames; | KF _i∩ KF _j| expression V _iAnd V _jThe number of similar key frame.

Images