JP5116017B2 - Video search method and system - Google Patents

Description

The present invention relates to a moving image search method and system , and more particularly, to a moving image search method and system capable of performing a high-accuracy moving image search even for a moving image having no cut points or shots or a low-quality moving image in which it is difficult to identify cut points or shots. .

  As a video search method for searching for a required video from a large number of videos, feature quantities (image feature quantities) related to brightness, color, etc. are extracted from the search target videos, and the necessary videos are extracted based on these image feature quantities. A search method is known. Such moving image search methods can be broadly classified into two methods, for example, a strict search method that strictly searches for moving image content that has been illegally leaked on the network, and an ambiguous search method that searches for a moving image similar to a certain moving image. Here, a conventional moving image search method will be described using an ambiguous search method as an example.

  TREC Video Retrieval Workshop (TRECVID) is known as a workshop for evaluating video search technology. A search task is included in an experiment performed in TRECVID, and TRECVID search tasks are disclosed in Non-Patent Documents 1 and 2.

  In this search method, as a feature for searching for a shot corresponding to a search query from “shots” of a search target video, a medium level feature amount estimated from a low level feature amount such as color information, or the shot Text information (closed captions and speech recognition processing results) is used. However, in these search methods, the search target moving image is limited to individual shots that are relatively short.

  Non-Patent Documents 3, 4, 5, 6 and Patent Document 1 disclose similar search methods for moving images composed of a plurality of shots such as television programs.

  Non-Patent Document 3 discloses a moving image search technique in which a unit of moving image search is a section composed of a plurality of shots, and for example, the same topic is searched from many news programs. Here, by using a similarity calculation index called Earth Mover's Distance (EMD) and calculating the similarity between videos, the same topic is searched from news programs.

  In Non-Patent Document 4 and Patent Document 1, a moving image topic tracking system that automatically tracks the progress of the same topic of a news program by applying the technique of Non-Patent Document 3 is proposed. In these documents, a method of updating a user profile based on relevance feedback information from a user and improving the accuracy of topic tracking after that is introduced.

  Non-Patent Document 5 proposes a technique for searching for the same topic by using a visual feature amount extracted from a moving image in addition to text information given to a news program. Specifically, by using a technique that applies face detection processing to identify the person appearing in the search target video, using this identification result to associate the same topic, A technique for associating the same topic by applying a process for detecting the same news gathering VTR (a process for detecting whether or not the videos are jointly distributed) has been proposed.

The above-mentioned conventional technology is a search method that shows the effectiveness of professionally produced videos such as TV programs, but in recent years, “video sharing sites” represented by YouTube and the like have become very popular. boom. In such a moving image sharing site, a moving image taken by a general user can be uploaded to the site by the user himself and made available to other site viewers. When uploading a moving image, the user can give category information or “tag” representing the content of the moving image. When searching for a moving image that a viewer of the moving image sharing site wants to view, the search can be performed using this category information or tag information as a key.
SF. Chang et al: “Columbia University TRECVID 2005 Video Search and High-level Feature Extraction”, Proc of TRECVID 2005, 2005. AG Hauptmann et al: “CMU Informedia's TRECVID 2005 Skirmishes”, Proc of TRECVID 2005, 2005. Y. Peng et al .: “EMD-based video clip retrieval by many-to-many matching”, Proc CIVR 2005, pp. 71-81, 2005. M. Uddenfeldt et al .: “Adaptive video news story tracking based on Earth Mover's Distance”, Proc. ICME 2006, pp. 1029-1032, 2006. Y. Zhai et al: Tracking news stories across different sources, Proceedings of ACM Multimedia 2005, pp. 2-9, 2005. Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas, “The earth mover's distance as a metric for image retrieval,” Int. J. Comput. Vision, vol. 40, no. 2, pp. 99-121, 2000. Japanese Patent Application No. 2006-93841

  In each of the patent and non-patent documents described above, the search target moving image is limited to a moving image such as a television program produced by a professional. In a moving image produced by a professional, a large number of shots are often included in the moving image by editing, and even with the above-described conventional technology, it is a premise for the effectiveness of search that a shot can be extracted.

  Video shots can be identified based on cut points, but most of the videos uploaded to video sharing sites are low-quality videos taken by amateur users with mobile phones. When the conventional search methods are applied, the resulting shots may differ from the actual video content due to the deterioration of cut point detection accuracy caused by low quality, and each search method may not function effectively. Is expensive.

  In the first place, in the case of moving images taken by general users, it is considered that many shots do not exist. It is difficult to apply the conventional search method as it is to such a moving image. For example, in calculating similarity between videos using EMD, the distance between keyframes of shots extracted from the search target video and the length of each shot are used as feature quantities. Since there is no single shot, only one shot is extracted, so that the similarity between moving images is calculated only by the distance between the substantial key frames. Therefore, if the extracted key frame features are similar, even if the content other than the key frame has completely different properties, the similarity between videos will be calculated high, and the search accuracy will be high. Leading to deterioration. Of course, due to the above problem, the similarity between moving images having a high actual similarity is low, and there is a high possibility that a search result unintended for the user is obtained.

An object of the present invention shows the above-mentioned solves the problem of the prior art, and enables clustering at the cut point or shot without moving or cut point or shot identification difficult low quality, videos, video search precision It is an object of the present invention to provide a moving image search method and system that make it possible.

  In order to achieve the above object, the present invention is characterized in that the following measures are taken.

  The moving image search system of the present invention includes a frame feature amount extracting unit that extracts a feature amount in units of frames for each search target movie, and each frame of each search moving image is divided into a plurality of clusters based on the extracted frame feature amount. Clustering means for performing clustering, query video specifying means for specifying a query video, similarity calculation means for calculating the similarity between the search target video and the query video based on the clustering result of each video, and the similarity And a search result output means for outputting the calculation result.

  According to the present invention, the following effects are achieved.

  According to the moving image search method and system of the present invention, a search target moving image is analyzed in units of frames, feature amounts are extracted, and frames having similar feature amounts are classified into the same cluster for each moving image. Since the similarity is calculated by regarding each cluster as a shot, it is possible to perform a high-accuracy video search even for a video with no cut points or shots, or with a low-quality video in which it is difficult to identify cut points or shots.

  DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the best embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main part of the moving picture search system according to the present invention.

  The moving image database (DB) 11 stores a large number of moving images to be searched. The feature amount extraction module 12 analyzes each moving image stored in the moving image DB 11 for each predetermined unit and extracts the feature amount of the image. In this embodiment, each moving image is analyzed in units of frames, and color information of each frame is extracted. The clustering module 13 performs a clustering process for classifying highly related frames into the same group for each search target moving image based on the frame feature amount extracted for each frame in the feature amount extraction module 12. This clustering process is a process that takes into account the nature of an individual user-captured moving image that is considered to have few cut points. The clustering result is stored in the feature amount DB 14 for each moving image.

  In the query moving image specifying module 15, a user specifies a moving image to be a search query (hereinafter also referred to as a query moving image). The user may specify a query video from among the search target videos stored in the video DB 11, or may separately input a query video. In addition, when a query video is input separately, as shown in FIG. 2, the query video specification module 15 analyzes the query video input module 15a that inputs the query video externally, The query feature amount extraction module 15b that extracts the feature amount and the clustering module 15c that classifies highly related frames into the same group based on the query feature amount extracted for each frame of the query moving image need to be configured.

  As will be described in detail later, the similarity calculation module 16 calculates the similarity between the query moving image and all search target moving images based on the comparison result between the clusters of the moving images. The search result output module 17 presents the search result for the designated query video to the user of this system. Specifically, a search target video having a high similarity to the query video is displayed on the system GUI.

  Next, the operation of the present invention will be described in detail with reference to a flowchart. The operation of the present invention is based on a “search target video feature amount extraction process” for extracting a feature amount from a search target video in advance, a video similar to the query video based on the feature amount of the search target video and the feature amount of the query video Is broadly classified as “video search processing”.

  FIG. 3 is a flowchart showing the procedure of the “search target moving image feature amount extraction process”. In step S1, one of the search target moving images registered in the moving image DB 11 is taken into the feature amount extraction module 12. In step S2, feature extraction processing is executed in the feature amount extraction module 12, and an image feature amount is extracted for each frame from the current search target moving image. Image features extracted here are MPEG-7 Visual (ISO / IEC 15938-3, "Information technology --- Multimedia content description interface --- Part 8: Extraction and use of MPEG-7 descriptions," Dec 2002.)), and a color histogram representing a color distribution in a frame image.

  In step S3, it is determined whether or not feature amount extraction has been completed from the current search target moving image. If it has not been completed, the process returns to step S2 and the feature amount extraction processing is similarly performed for the next frame. . In the extraction process, if the image feature amount is extracted from all the frames of the search target moving image, the calculation amount becomes enormous. Therefore, a predetermined rule (for example, at equal intervals) from all n frames constituting the moving image, It is desirable to reduce the amount of calculation by selecting only m frames as feature amount extraction frames.

  When the feature amount extraction processing for the current search target video is completed, in step S4, frame clustering processing is executed in the clustering module 13, and each frame from which the image feature amount is extracted in the frame feature amount extraction processing is the feature. Based on quantity, it is classified into multiple classes. The purpose of this process is to create a pseudo “shot” by clustering frames that have similar features to a video shot by a general user, where there are few or no cut points. Is to build.

  FIG. 4 is a conceptual diagram of the frame clustering process. Of all the frames constituting the search target moving image, only the frame on which the feature extraction process has been performed includes a plurality of frame clusters C1, C2 based on the image feature amount. It is classified as one of the following. In the example shown in FIG. 4, three frames are classified into the frame cluster C1, and four frames are classified into the frame cluster C2. By performing clustering processing in this way, even if the search target video does not have a cut point, it is possible to extract detailed features according to the content of the video, which is used for calculating the similarity between subsequent videos. be able to.

  FIG. 5 is a flowchart showing the procedure of the clustering process executed in step S4. Here, a case where 15 frame feature amounts are extracted from the search target moving image will be described as an example.

  The clustering algorithm includes a bottom-up hierarchical clustering algorithm that integrates similar frames and clusters sequentially, and a top that performs clustering after determining the number of clusters in advance, such as the k-means method. There is a down type (non-hierarchical) clustering algorithm, but in video search, the dynamic number of clusters according to the content of the video can express the characteristics of the video itself more. Therefore, here, a bottom-up type will be described as an example.

  In step S401, one of the frame feature amounts is captured, and in step S402, a cluster having the frame feature amount as belonging data is generated. In step S403, a representative vector of the cluster is generated. In step S404, it is determined whether or not the generation of the cluster and the cluster representative vector has been completed for all the frame feature amounts of the current search target moving image. If not completed, the process returns to step S401, and the next frame feature amount is determined. Each process described above is repeated.

  When the above-described processing is completed for all 15 frame feature amounts and 15 clusters C1 to C15 and their cluster feature vectors are generated, in step S405, between the two clusters Ci and Cj for all clusters. A distance is calculated. In this embodiment, the Ward method, which is a representative hierarchical clustering method, is employed, and the distance d (Ci, Cj) between the two clusters Ci, Cj is first calculated by the following equation (1). However, E (Ci) represents the sum of the squares of the distances between all the belonging data of the cluster Ci and the representative vector (centroid) of the cluster Ci.

  In step S406, two clusters having the smallest inter-cluster distance d (Ci, Cj) are integrated into one cluster from all combinations of clusters at the present time. In step S407, a representative vector of the integrated cluster is generated.

  Here, when the cluster belonging data is a general numerical vector, the representative vector of the cluster can be obtained by calculating a centroid vector or the like, and the distance between the data can also be calculated by the Euclidean length or the like. However, when the color arrangement information is used as the frame feature amount, since the numerical value of the vector is a DCT coefficient, it is possible to obtain a theoretically strange result if each element value is simply added.

  Therefore, in this embodiment, the distance between color arrangements defined in Non-Patent Document 1 is adopted as a method for calculating the distance between data. The cluster representative vector adopts the mode value for each element of all data (vector) belonging to the cluster as the element value of the representative vector. When a plurality of mode values appear in the same element, for example, a method of adopting an element value of a frame having a small frame number is applied.

  Note that when an image feature quantity that can calculate a centroid vector or Euclidean distance, such as a color histogram, is used as the frame feature quantity, clustering may be performed based on these criteria.

  In step S408, it is determined whether or not the clusters are aggregated into one, and the above integration process is repeated until the clusters are aggregated into one. When the clusters are aggregated into one, the process proceeds to step S409, and processing for determining the number of clusters and extracting each cluster is executed.

  FIG. 6 schematically illustrates how 15 frame feature quantities are aggregated into one cluster in a stepwise manner by repeating cluster integration in step S406. The vertical axis represents the frame identifier, and the horizontal axis represents The inter-cluster distance [value on the left side of the above equation (1)] when each cluster is integrated is shown.

  In the present embodiment, in order to determine the number of clusters for each moving image, the distance at the time of cluster integration is set as a threshold, and only clusters integrated at a distance equal to or less than the threshold are extracted. In FIG. 6, when the vertical wavy line is a threshold value, 10 clusters integrated below this distance are clusters extracted from this moving image. In the example of FIG. 6, the belonging data is extracted as one cluster for each frame feature amount having the frame identifiers “1”, “8”, and “14”.

  Next, the procedure of the moving image search process will be described with reference to the flowchart of FIG. In step S31, it is determined whether or not a query moving image is specified in the query moving image specifying module 15. In the present embodiment, when any of the search target videos stored in the video DB 11 is designated as a query video, or when a query video is newly input by the user, it is determined that the query video has been designated.

  When a query moving image is newly input, the process proceeds to step S32, where the query moving image is analyzed and the frame feature amount is extracted in the same procedure as the procedure for extracting the frame feature amount from the search target moving image. In step S33, each frame is clustered in the same manner as described above based on the frame feature amount. Note that if any of the search target videos stored in the video DB 11 is designated as a query video, the clustering result of the query video is already registered in the feature amount DB 14, and thus the processes in steps S32 and S33 are omitted. The

  In steps S34 and S35, the similarity between the query video and each search target video is calculated by applying the similarity calculation method based on Earth Mover's Distance (EMD) disclosed in Non-Patent Document 6. However, in Non-Patent Document 6, the similarity (EMD) between videos is calculated based on two feature quantities, namely, key frame color information of each shot extracted from the search target video and the shot length. On the other hand, since shot extraction is not performed in the present invention, the method of Non-Patent Document 6 cannot be applied as it is. Specifically, it is necessary to extract each feature quantity corresponding to the color information and shot length of the key frame in the non-patent document from the cluster of each frame obtained as a result of the feature quantity extraction.

  Therefore, in the present invention, the representative vector of each cluster is used as the feature quantity that replaces the color information of the key frame of the shot, and the number of data belonging to each cluster is used as the feature quantity that replaces the shot length.

  In step S34, the feature vector of each cluster and the number of belonging data are captured for one of the search target moving images. In step S35, the search is based on the EMD disclosed in Non-Patent Document 6 using the feature vector and number of belonging data of each cluster of the search target moving image and the feature vector and number of belonging data of each cluster of the query moving image as parameters. The degree is calculated. That is, the similarity is calculated based on the distance between the representative vectors of each moving image and the distance between the numbers of belonging data.

  In step S36, it is determined whether or not the similarity calculation has been completed for all the search target videos. If not, the process returns to step S34, and the above-described processes are repeated while switching the search target videos.

  When the similarity calculation is completed for all the search target videos, the process proceeds to step S37, and the search target videos are sorted based on the similarity. In step S38, a plurality of search target videos with higher similarity are presented to the user as search results.

It is the functional block diagram which showed the structure of the moving image search system which concerns on this invention. It is the functional block diagram which showed the other structure of the moving image search system which concerns on this invention. It is the flowchart which showed the procedure of the search object moving image feature-value extraction process. It is a conceptual diagram of a frame clustering process. It is the flowchart which showed the procedure of the clustering process. It is the figure which showed a mode that a frame was clustered in steps. It is the flowchart which showed the procedure of the moving image search process.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 ... Movie DB, 12 ... Feature-value extraction module, 13 ... Clustering module, 14 ... Feature-value DB, 15 ... Query animation specification module, 16 ... Similarity calculation module, 17 ... Search result output module

Claims (8)

In a video search system that searches a number of search target videos for videos similar to the query video,
Frame feature amount extraction means for extracting feature amounts in units of frames for each search target video;
Clustering means for classifying each frame of each search video into a plurality of clusters based on the extracted frame feature amount;
Query video specification means for specifying a query video,
Similarity calculation means for calculating the similarity between the search target video and the query video based on the clustering result of each video,
And a search result output means for outputting the calculation result of the similarity. The moving image search system according to claim 1 , wherein the query moving image specifying unit specifies one of the search target moving images as a query moving image. The query video specifying means
A way to enter query videos,
Means for extracting a query feature value in frame units from the query video;
The moving image search system according to claim 1 , further comprising: a unit that classifies each frame of the query moving image into a plurality of clusters based on the extracted query feature amount. The similarity calculation means includes:
Means for obtaining the representative vector and the number of belonging data of each cluster of the search target video;
A means for obtaining the representative vector and the number of belonging data of each cluster of the query video,
Video retrieval system according to any one of claims 1 to 3, characterized in that it comprises a similarity between each video clusters, and means for calculating, based on the distance of the distance and belonging data number between between the representative vector . In a video search method that searches a number of search target videos for videos similar to the query video,
A procedure for extracting feature values in frame units for each search target video;
A procedure for classifying each frame of each search video into a plurality of clusters based on the extracted frame feature amount;
Steps to specify a query video,
A procedure for calculating the similarity between the search target video and the query video based on the clustering result of each video,
And a procedure for outputting the calculation result of the similarity. The moving image search method according to claim 5 , wherein in the procedure of specifying the query moving image, one of the search target moving images is specified as a query moving image. The procedure to specify the query video is:
How to enter a query video,
A procedure for extracting a query feature amount from the query video in frame units;
6. The moving image search method according to claim 5 , further comprising a step of classifying each frame of the query moving image into a plurality of clusters based on the extracted query feature amount. The procedure for calculating the similarity is:
The procedure for obtaining the representative vector and the number of data belonging to each cluster of the search target video,
The procedure to find the representative vector and the number of data belonging to each cluster of the query video,
The similarity between each video clusters, video searching method according to any one of claims 5 to 7, characterized in that it comprises a step of calculating, based on the distance of the distance and belonging data number between between the representative vector .

Images