JP2014099110A - Image search device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To more accurately search an image indicating an instance.SOLUTION: A feature summarization and tabulation part 12 creates a summarized feature DB 21 in which local features extracted from an instance image are summarized and an instance feature DB 22 in which the appearance frequency of each summarized feature in each instance image group is shown, and extracts the local features from each frame image. A feature collation and tabulation part 15 creates a frame image feature DB 23 of each video and a video feature DB 24 on the basis of the collation result of each local feature extracted from each frame image with each summarized feature in the summarized feature DB 21. An inverse frame image frequency calculation part 16 creates an inverse frame image frequency DB 25 indicating the IFF(Inverse Frame Frequency) of each summarized feature, and a search ranking part 17 refers to the instance feature DB 22, the inverse frame image frequency DB25 and the video feature DB 24, and calculates EBM(Extension Best Match) 25 under the consideration of the instance feature DB 22 and the IFF from BM(Best Match), and creates a search result based on the EBM 25.

Description

  The present invention relates to an image search apparatus, method, and program.

  Conventionally, an image including an instance is searched from a large-scale image database by using an image indicating an instance such as a specific person, thing, or place as a query. For example, based on the similarity of the appearance pattern of local features between the image group showing the instance and the video frame image group, a technology that searches the video including the instance, that is, the query image group and video similarity measure is used. A video search technique has been proposed (see Non-Patent Document 1, for example).

Cai-Zhi Zhu et al., "Large Vocabulary Quantization for Searching Instances from Videos.", In Proc. Of ICMR'12, 2012.

  An object of the present invention is to provide an image search apparatus, method, and program capable of searching for an image showing an instance with higher accuracy by an approach different from the prior art.

  In order to achieve the above object, the image search apparatus of the present invention is characterized by extracting a plurality of features from each frame image of an image group consisting of an instance image indicating an instance to be a query and a plurality of frame images to be searched. The instance in each of the plurality of instance images of the aggregated feature obtained by aggregating overlapping features from a plurality of features extracted from each of the plurality of instance images indicating each of the plurality of instances by the extracting unit and the feature extracting unit A feature aggregation totaling unit that aggregates the first appearance frequency for each, a collation between each of the aggregated features and a plurality of features extracted from the frame image, and for each of the aggregated features, a second in each frame image A feature collating and summing unit for summing up the appearance frequency and the third appearance frequency in the image group; On the basis of the second appearance frequency, reverse frame image frequency calculation means for calculating a reverse frame image frequency that increases as the appearance frequency of the aggregate feature in the image group decreases, and the inverse of each of the aggregate features The search target represented by the sum of the importance of each aggregated feature determined based on the frame image frequency, the first appearance frequency, the third appearance frequency, and the image group length related to the number of aggregated features included in the image group. Search result creating means for obtaining an evaluation value of each image group for the instance and creating a search result based on the evaluation value.

  According to the image search device of the present invention, the feature extraction unit extracts a plurality of features from an instance image indicating an instance that is a query. Here, a plurality of features are extracted from each of a plurality of instance images indicating each of the plurality of instances. Then, the feature aggregation and aggregation unit first appearance for each instance in the plurality of instance images of each of the aggregated features obtained by aggregating overlapping features among the plurality of features extracted from the plurality of instance images by the feature extraction unit Aggregate frequency.

  In addition, the feature extraction unit extracts a plurality of features from each frame image of an image group including a plurality of frame images to be searched. Then, the feature collating and summing unit collates each aggregated feature with a plurality of features extracted from the frame image, and for each aggregated feature, the second appearance frequency in each frame image and the third appearance frequency in the image group Are counted. Further, the reverse frame image frequency calculation means calculates the reverse frame image frequency that becomes higher as the appearance frequency of the aggregate feature in the image group is lower for each of the aggregate features based on the second appearance frequency.

  Then, the search result creating means determines each aggregate feature determined based on the image group length related to the reverse frame image frequency, the first appearance frequency, the third appearance frequency, and the number of aggregate features included in the image group. The evaluation value of each image group for the search target instance represented by the sum of the importance levels is obtained, and a search result based on the evaluation value is created.

  In this way, the search result is created based on the evaluation value in which the appearance frequency in the instance image of the aggregated feature extracted and aggregated from the instance image is considered as the importance of each aggregated feature. You can search for images.

  Further, the feature aggregation and aggregation unit is configured to calculate the first appearance frequency based on an additional feature that is not included in the aggregation feature among features extracted from an instance image indicating an instance newly added as a search target. Updating the result, the feature collating and summing unit collates the additional feature with a plurality of features extracted from the frame image, updates the second appearance frequency and the third appearance frequency, and the reverse frame image Frequency calculation means recalculates the reverse frame image frequency based on the updated second appearance frequency, and the search result creation means recalculates the reverse frame image frequency and the updated first appearance. A search result for the newly added instance can be created based on the frequency and the updated third appearance frequency. As a result, it is possible to search for an image showing an instance with high accuracy for the added instance only by performing processing for the added amount.

  The search result creating means may include a file name of the image group or a file name of the image group and an evaluation value of the image group in the search result. The search result may be created based on the evaluation value, and various forms of search results can be created.

  The image search method of the present invention is an image search method in an image search apparatus including a feature extraction unit, a feature aggregation totaling unit, a feature matching totaling unit, an inverse frame image frequency calculation unit, and a search result creation unit. The feature extraction means extracts a plurality of features from each frame image of an image group consisting of an instance image indicating an instance to be a query and a plurality of frame images to be searched, and the feature extraction means From the plurality of features extracted from each of a plurality of instance images indicating each of the instances, the first appearance frequency for each instance in each of the plurality of instance images of each of the aggregated features that aggregate overlapping features is aggregated, and A feature collating and summing unit collates each of the aggregated features with a plurality of features extracted from the frame image, and For each feature, the second appearance frequency in each frame image and the third appearance frequency in the image group are tabulated, and the inverse frame image frequency calculation means calculates the second appearance frequency for each of the aggregated features. Based on this, the inverse frame image frequency that increases as the appearance frequency of the aggregated feature in the image group becomes low, and the search result creation means calculates the inverse frame image frequency and the first appearance frequency of each of the aggregated features. , And the third appearance frequency, and the evaluation value of each image group for the search target instance represented by the sum of importance of each aggregated feature determined based on the image group length related to the number of aggregated features included in the image group This is a method of obtaining and creating a search result based on the evaluation value.

  The image search program of the present invention is a program for causing a computer to function as each means constituting the image search apparatus.

  As described above, according to the image search device, method, and program of the present invention, the evaluation value considering the appearance frequency of the aggregated feature extracted from the instance image in the instance image as the importance of each aggregated feature Since a search result is created based on the above, an effect that an image showing an instance can be searched with higher accuracy can be obtained.

It is the schematic which shows the structure of the video search device which concerns on 1st Embodiment. It is a figure which shows an example of the data structure of an instance image. It is a figure which shows an example of the data structure of a local feature. It is a figure which shows an example of the data structure of aggregation feature DB. It is a figure which shows an example of the data structure of instance feature DB. It is a figure which shows an example of the data structure of frame image characteristic DB. It is a figure which shows an example of the data structure of video characteristic DB. It is a figure which shows an example of the data structure of reverse frame image frequency DB. It is a figure which shows an example of the data structure of a search result. It is a flowchart which shows the content of the image | video search processing routine in 1st Embodiment. It is the schematic which shows the structure of the video search device which concerns on 2nd Embodiment. It is a figure which shows an example of the data structure of additional feature DB. It is a figure which shows an example of the data structure of updated aggregation feature DB. It is a figure which shows an example of the data structure of updated instance feature DB. It is a figure which shows an evaluation result.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Outline of each embodiment>
In each embodiment, the present invention is applied to a video search apparatus that receives an image (hereinafter referred to as “instance image”) indicating an instance (example), searches a video including the instance from a large-scale video database, and outputs a search result. A case in which the image search apparatus is applied will be described. The video search device according to each embodiment includes an inverse frame image frequency that is an index of the importance of local features of an instance image group, an appearance frequency of local features in the instance image group and the video group, and a local feature included in each video. By paying attention to the video length, which is the total number of images, an instance search system that performs high-precision video search including instances (hereinafter also referred to as “instance search”) is realized.

  In each embodiment, a ranking method called BM25 (Best Match 25), which is often used for Web page keyword search, is applied to instance search. When applying BM25, IDF (Inverse Document Frequency), which is an index indicating the importance of keywords in BM25, is regarded as the importance of local features related to instances, and is an index for searching for a video including many local features with high importance And In addition, since the query in the instance search is expressed as a large vector corresponding to the local feature amount obtained from one or more instance images, the BM 25 that assumes a small vector as a query is changed from a document search based on a keyword to a video search. The expanded BM 25 (referred to as “EBM 25” in each embodiment) is applied to the instance search.

<First Embodiment>
The video search apparatus 10 according to the first embodiment is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a video search processing routine described later. As shown in FIG. 1, this computer functionally includes an instance image feature extraction unit 11, a feature aggregation totaling unit 12, a frame image extraction unit 13, a frame image feature extraction unit 14, and a feature matching totalization unit. 15, a reverse frame image frequency calculation unit 16, a search ranking unit 17, and a search result output unit 18. In addition, the video search device 10 is provided with a predetermined storage area for storing an additional feature database (DB) 21, an instance feature DB 22, a frame image feature DB 23, a video feature DB 24, and an inverse frame image frequency DB 25.

  An instance image is input to the video search apparatus 10 as an instance search query. In the first embodiment, in order to create the instance feature DB 22, a plurality of instances are targeted, and an instance image group including a plurality of instance images is input for each instance. For example, when the number of instances is q and h instance images are prepared for each instance, q instance image groups and a total number of q × h instance images are input. However, one or more instance images may be input for one instance. Alternatively, an image indicating an instance may be input and each frame of the image may be input as an instance image group. Note that as the number of instance images included in the instance image group for one instance increases, the search accuracy of the instance search improves.

FIG. 2 shows an example of the data structure of the instance image. FIG. 2 shows an example in which the instance image is composed of n × m pixels and has RGB values as the pixel values of each pixel. The position (x _n , y _m ) and the RGB value (r _nm , g) for each pixel are shown. _nm , b _nm ).

  The instance image feature extraction unit 11 receives an instance image group input to the video search device 10, detects a feature point from each instance image, and extracts a feature vector describing a feature amount of the detected feature point as a local feature. . For example, the instance image feature extraction unit 11 detects a location where the luminance value changes drastically in the instance image using the Harris-Laplace method (“C. Harris et al.,“ A combined corner and edge detector. ”, 4th Alvey Vision Conf., 1988.)). And the feature-value of each detected feature point is described by Compact Color SIFT (refer to "K. Mikolajczyk et al.," Scale and affine invariant interest point detectors. ", IJCV, 2004.). Compact Color SIFT is a local feature amount obtained by adding a 64-dimensional vector representing chromaticity to a 128-dimensional SIFT feature amount relating to luminance.

  FIG. 3 shows an example of the data structure of local features. FIG. 3 is an example using the above-described Compact Color SIFT. The extracted identification numbers (feature 1, feature 2,..., Feature i) of each local feature and the 192-dimensional Compact Color SIFT feature (feature) Vector).

  The feature aggregation / aggregation unit 12 aggregates overlapping local features into one among the local features extracted by the instance image feature extraction unit 11. "Overlapping local features" means that the same local features are extracted when the same local features are extracted from almost the same instance image, such as when each frame of video is used as the instance image group. Is regarded as “duplication”. The feature aggregation / aggregation unit 12 creates an aggregation feature DB 21 in which the aggregated local features are aggregated features, and stores them in a predetermined storage area. FIG. 4 shows an example of the data structure of the aggregate feature DB 21. In the example of FIG. 4, the data structure has an identification number (feature 1, feature 2,..., Feature j) of each aggregate feature and a 192-dimensional Compact Color SIFT feature quantity (feature vector) associated with each other. Yes. This data structure is the same as the data structure of the local features (for example, FIG. 3) extracted by the instance image feature extraction unit 11, but the number of local features is aggregated from i to j by the feature aggregation / aggregation unit 12. It is shown that.

  In addition, the feature aggregation totaling unit 12 extracts the number of appearances of each aggregate feature in the instance image group indicating each instance based on the aggregate feature for each instance as an instance feature, creates the instance feature DB 22, Store in the storage area. FIG. 5 shows an example of the data structure of the instance feature DB 22. FIG. 5 is an example in which the number of instances is q and the total number of aggregate features is j. The identification number of each instance (instance 1,..., Instance q) and the number of appearances of each aggregate feature kf (keypoint) frequency) is associated with the data structure.

  The feature aggregation / aggregation unit 12 may further aggregate by clustering the aggregated local features to reduce the number of dimensions.

  The frame image extraction unit 13 receives a video group input to the video search device 10 and extracts a frame image from each video at a rate of, for example, 1 fps (one frame per second). Since the data structure of the frame image is the same as the data structure of the instance image (for example, FIG. 2), detailed description is omitted.

  The frame image feature extraction unit 14 differs from the instance image feature extraction unit 11 only in that the target for extracting local features is not the instance image but the frame image extracted by the frame image extraction unit 13. Since the data structure of the extracted local features is the same as the data structure of the local features extracted by the instance image feature extraction unit 11 (for example, FIG. 3), detailed description thereof is omitted.

  The feature matching totaling unit 15 compares each local feature extracted by the frame image feature extracting unit 14 with each aggregate feature stored in the aggregate feature DB 21. In the case of the above Compact Color SIFT, for example, the cosine similarity between 192-dimensional feature vectors (takes a value in the range of 0 to 1 and is 1 in the case of the same feature vector), and the cosine similarity is a predetermined value. The local features and the aggregate features described above (for example, 0.95) are determined as matching features. In the case where there are a plurality of aggregate features whose cosine similarity is equal to or greater than a predetermined value with respect to the local features extracted from the frame image, the aggregate features having the largest cosine similarity are sparse local features. Judged as matching aggregate features.

  Note that collation between local features is not limited to using cosine similarity, and any measure may be used as long as it is a scale for measuring the distance and similarity between feature vectors.

  Further, when local features are aggregated by clustering by the feature aggregation / aggregation unit 12, the feature matching aggregation unit 15 uses the centroid of each cluster in the aggregation feature DB 21 to determine the local feature extracted from the frame image. Verification can be performed.

  The feature matching totaling unit 15 creates the frame image feature DB 23 for each video based on the matching result between the local feature extracted from the frame image and the aggregate feature stored in the aggregate feature DB 21, and stores it in a predetermined storage area. Remember. FIG. 6 shows an example of the data structure of the frame image feature DB 23. In the example of FIG. 6, the identification number (frame image 1,..., Frame image N) of each frame image extracted from one video is associated with the appearance frequency Kf of each aggregated feature in each frame image. Yes.

  In addition, the feature matching totaling unit 15 creates the video feature DB 24 based on the matching result between the local feature extracted from the frame image and the local feature stored in the aggregate feature DB 21 and stores it in a predetermined storage area. . FIG. 7 shows an example of the data structure of the video feature DB 24. In the example of FIG. 7, the video identification number (video 1,..., Video v) is associated with the number of appearances KF of each aggregated feature in each video. The number of appearances KF can be obtained by summing up the number of appearances for each aggregated feature in the frame image feature DB 23 created for each video from the frame image 1 to the frame image N.

The inverse frame image frequency calculation unit 16 refers to the frame image feature DB 23 for each video, and calculates the inverse frame image frequency IFF _j (Inverse Frame Frequency) of the aggregate feature j based on the following equation (1).

Here, N is the number of all frame images in one video, n _j is the number of frame images including the aggregate feature j in one video, and the number of appearances Kf _j of the aggregate feature _j is 1 in the frame image feature DB 23. It can be obtained by counting the frame images as described above. When the aggregate feature j appears at a high frequency in the frame image feature DB 23, the aggregate feature j can be regarded as a local feature having a low identification ability for the instance, and therefore the IFF _j becomes small. Conversely, when the frequency of appearance of the aggregated feature j in the frame image feature DB 23 is low, the aggregated feature j can be regarded as an aggregated feature with a high identification capability for the instance, so IFF _j becomes large.

The inverse frame image frequency calculation unit 16 calculates IFF _j for all of the aggregate features (features 1,..., Feature j) stored in the aggregate feature DB 21 and creates a calculation result as the inverse frame image frequency DB 25. , Stored in a predetermined storage area. FIG. 8 shows an example of the data structure of the inverse frame image frequency DB 25. In the example of FIG. 8, the identification number (feature 1,..., Feature j) of the aggregate feature is associated with the calculated IFF _j . The reverse frame image frequency DB 25 is created for each video.

The search ranking unit 17 searches the input video group for video candidates that may contain the instance for each instance, and ranks the video candidates. Specifically, first, the search ranking unit 17 acquires the number of appearances kf ₁ ,..., Kf _j of each aggregate feature that is an instance feature of the instance q from the instance feature DB 22. Further, the search ranking unit 17 acquires the reverse frame image frequencies IFF ₁ ,..., IFF _j of each aggregate feature from the reverse frame image frequency DB 25 of the video v. Further, the search ranking unit 17 acquires the number of appearances KF ₁ ,..., KF _j of each aggregated feature associated with the video v from the video feature DB 24. Then, the search ranking unit 17 calculates the evaluation value EBM25 (q, v) of the video v for the instance q shown in the following equation (2).

Here, k ₁ , k ₂ , and b ₁ are setting parameters. For example, k ₁ = k ₂ = 1.2 and b ₁ = 0.75 can be set. Further, vl means video length, avvl means average video length, vl is the sum of the number of appearances KF of each aggregated feature associated with the video v, and avvl is in the video feature DB 24. It is the average of vl of each video. Σ _qj means the sum of the aggregate features 1,..., J of the instance q. The evaluation value EBM25 shown in the equation (2) is the BM25 shown in the following equation (3) (“SE Robertson et al.,“ Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval. ”, In Proc. Of SIGIR '04, 1994 "), as the importance of each aggregated feature, the term ((k ₂ +1) kf _j ) / (k ₂ + kf _j ) of the aggregated feature appearance frequency in the instance feature DB 22 and the reverse frame This further considers the image frequency IFF _j .

  Both the BM 25 and the EBM 25 give a high score to a video that includes many aggregate features that are highly important for the instance and that has a video length that is not too large. However, in the BM 25, the importance level of each aggregate feature is defined only by the appearance frequency of the aggregate feature in the frame image feature DB 23. However, in the EBM 25, the appearance frequency of the aggregate feature in the instance feature DB 22 is further considered. . The EBM 25 corresponds to an extension of the BM 25 when performing a search using a document as a query rather than a search using a keyword in the text search field.

  The search ranking unit 17 calculates the evaluation value EBM25 of each video (video 1,..., Video v) for the instance q, and creates a search result ranking the video in descending order of the evaluation value EBM25. A search result is created for each instance (instance 1,..., Instance q). FIG. 9 shows an example of the data structure of the search result. In FIG. 9, the identification number of each instance (instance 1,..., Instance q) and the video arranged in descending order of the evaluation value EBM 25 are associated with each other.

  The search result is not limited to the ranking format as described above, and only the video with the maximum evaluation value may be used as the search result, or videos with the evaluation value equal to or greater than the predetermined value are randomly arranged. It is good also as a search result. The search result may be a video file name, or a video file name and a value of the EBM 25. The search result can take various forms as long as it is based on the value of the EBM 25.

  The search result output unit 18 outputs the search result created by the search ranking unit 17.

  Next, the operation of the video search apparatus 10 according to the first embodiment will be described. When a plurality of instance image groups indicating a plurality of instances are input to the video search device 10, the video search processing routine shown in FIG.

  In step 100, the instance image feature extraction unit 11 receives the instance image group input to the video search device 10, detects a feature point from each instance image, and localizes a feature vector describing the feature amount of the detected feature point. Extract as a feature.

  Next, in step 102, the feature aggregation / aggregation unit 12 aggregates the overlapping local features into one from the local features extracted in step 100, and the aggregated feature DB 21 having the aggregated local features as an aggregate feature. Is created and stored in a predetermined storage area. Further, the feature aggregation / aggregation unit 12 creates an instance feature DB 22 indicating the number of appearances of each aggregated feature in each instance image group as a feature of each instance based on the aggregated feature, and stores it in a predetermined storage area.

  Next, in step 104, the frame image extraction unit 13 receives the video group input to the video search device 10, and extracts a frame image from each video. Next, in step 106, the frame image feature extraction unit 14 extracts local features from each frame image extracted in step 104.

  Next, in step 108, the feature collating and summing unit 15 calculates each local feature extracted from the frame image in step 106 and each aggregated feature in the aggregated feature DB 21 stored in step 102 between feature vectors. Match based on the similarity of. Then, the feature matching totaling unit 15 creates a frame image feature DB 23 indicating the appearance frequency Kf of each aggregated feature in each frame image based on the matching result, and stores it in a predetermined storage area. Also, the feature matching totaling unit 15 creates a video feature DB 24 indicating the number of appearances KF of each aggregated feature in each video based on the matching result, and stores it in a predetermined storage area.

Next, in step 110, the reverse frame image frequency calculation unit 16 refers to the frame image feature DB 23, and sets the reverse frame image frequency IFF _j of the aggregate feature _j to the aggregate feature (feature 1, feature 1). .., Feature j) are calculated, and an inverse frame image frequency DB 25 indicating the IFF of each aggregated feature is created and stored in a predetermined storage area. The reverse frame image frequency DB 25 is created for each video.

Next, in step 112, the search ranking unit 17 obtains the number of appearances kf ₁ ,..., Kf _j of each aggregate feature that is the instance feature of the instance q from the instance feature DB 22, and the inverse frame image frequency of each video. The inverse frame image frequency IFF ₁ ,..., IFF _j of each aggregated feature is acquired from the DB 25, and the appearance frequency KF ₁ ,..., KF _j of each aggregated feature associated with each video is obtained from the video feature DB 24. get. Then, the search ranking unit 17 calculates the evaluation value EBM25 of each video for the instance q, and creates a search result ranking the video in descending order of the evaluation value EBM25. A search result is created for each instance.

  Next, in step 114, the search result output unit 18 outputs the search result created in step 112, and ends the video search processing routine.

  As described above, according to the video search device according to the first embodiment, the EBM 25 that further considers the appearance frequency in the instance image group and the inverse frame image frequency as the importance of the aggregate feature is used as the evaluation value. Thus, it is possible to search for an image showing an instance with higher accuracy.

<Second Embodiment>
In the second embodiment, a case will be described in which a video is searched using a newly added instance image as a query in a state where a predetermined number of data has already been accumulated in the instance feature DB 22. Note that in the video search device according to the second embodiment, the same components as those of the video search device 10 according to the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  The video search apparatus 210 according to the second embodiment is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a video search processing routine described later. As shown in FIG. 11, this computer functionally includes an instance image feature extraction unit 11, a feature aggregation totaling unit 212, a frame image extraction unit 13, a frame image feature extraction unit 14, and a feature matching totalization unit. 215, the reverse frame image frequency calculation unit 16, the search ranking unit 17, and the search result output unit 18. In addition, the video search apparatus 10 is provided with a predetermined storage area for storing an aggregate feature DB 21, an instance feature DB 22, a frame image feature DB 23, a video feature DB 24, an inverse frame image frequency DB 25, and an additional feature DB 26.

  The feature aggregation / aggregation unit 212 receives the local feature of the new instance image extracted by the instance image feature extraction unit 11, refers to the aggregation feature DB 21, eliminates the overlapping local feature, and is newly added Extract features. Here, a case where an instance image group indicating the instance q + 1 is newly input and an additional feature j + 1 is added will be described. The feature aggregation totaling unit 212 stores the additional feature j + 1 in the additional feature DB 26 having the same data structure as the aggregate feature DB 21 and updates the aggregate feature DB 21 by the addition of the additional feature j + 1. FIG. 12 shows an example of the additional feature DB 26, and FIG. 13 shows an example of the updated aggregate feature DB 21.

In addition, the feature aggregation totaling unit 212 updates the instance feature DB 22 with the added instance q + 1 and the added feature j + 1. FIG. 14 shows an example of the updated instance feature DB 22. In the example of FIG. 14, the column of the additional feature j + 1 and the row of the instance q + 1 are added to the instance feature DB 22 (FIG. 5) before the update. Since the additional feature j + 1 is a newly added aggregate feature, the number of appearances kf _{j + 1} for the instances 1,..., Instance q is 0.

The feature matching totaling unit 215 updates the frame image feature DB 23 and the video feature DB 24 based on the matching result between the local feature extracted from the frame image and the additional feature stored in the additional feature DB 26. The matching method between the local feature and the additional feature is the same as the matching method between the local feature and the aggregated feature in the feature matching totaling unit 15 of the first embodiment. In the update of the frame image feature DB 23, the number of appearances Kf _{j + 1} of the additional feature j + 1 in each frame image is counted, and the column of the additional feature j + 1 is added to the frame image feature DB 23. In addition, in the update of the video feature DB 24, the number of appearances Kf _{j + 1} of the additional feature j + 1 in the updated frame image feature DB 23 for each video is totaled from the frame image 1 to the frame image N, so KF _{j + 1} is obtained, and a column of additional features j + 1 is added to the video feature DB 24.

  Next, the operation of the video search apparatus 210 according to the second embodiment will be described with respect to differences from the first embodiment.

  In step 102 of the video search processing routine of FIG. 10, the feature aggregation / aggregation unit 212 extracts additional features from the local features of the new instance image extracted in step 100 with reference to the aggregation feature DB 21, and stores them in the additional feature DB 26. In addition to storing, the aggregated feature DB 21 is updated by the amount of new added features. Further, the feature aggregation / aggregation unit 212 updates the instance feature DB 22 by the amount of new instances and additional features added.

  In step 108, the feature matching totaling unit 215 adds the additional feature to the frame based on the matching result between the local feature extracted in step 106 and the additional feature stored in the additional feature DB 26 in step 102. The image feature DB 23 and the video feature DB 24 are updated.

  In the subsequent processing, the database updated as described above is referred to, the evaluation value EBM 25 is calculated in the same manner as in the first embodiment, and a search result may be created.

  As described above, according to the video search device according to the second embodiment, when a predetermined number or more of data is accumulated in the instance feature DB, the newly added instance image is processed. As in the first embodiment, it is possible to search for an image showing an instance with higher accuracy.

<Evaluation results>
Here, the evaluation result of the search accuracy using the instance search task data set of TRECVID 2011, 2012 will be described. TRECVID is an annual competition in the field of video search and is hosted by the National Institute of Standards and Technology (NIST) in the United States. In the instance search task of TRECVID 2011, 25 instances are prepared, and an average of about 4 images is assigned to each instance. And about 20,000 video databases (overseas program videos) are to be searched. In the instance search task of TRECVID 2012, 21 instances are prepared, and an average of about 5 images is assigned to each instance. About 80,000 video databases (Consumer Generated Media (CGM) on the Web) are to be searched. The average number of correct answers for each instance of TRECVID 2011 is about 73, and about 59 for TRECVID2012. The accuracy of the search result ranking was evaluated by an index called Mean Average Precision (MAP). MAP is defined by the following equation (4).

Where | Q | is the total number of instances, | R _q | is the number of correct videos of instance q, j is the search result rank of video, and rel (q, j) is the video of rank j that is correct with respect to q. It is a function that returns 1 if the answer is incorrect. c (q, j) is the number of correct images existing from rank 1 to rank j. When correct videos are arranged in order from rank 1, that is, the highest search accuracy is MAP = 100 (%).

  FIG. 15 shows the evaluation result of the search accuracy. KF is a search ranking method using only the sum of appearance frequencies of local features in the video, and IFF is a search ranking method using only the sum of the reverse frame image frequencies of local features in the video. From the results of FIG. 15, it can be seen that the search accuracy is greatly improved by using IFF. Then, the ranking is improved by using the BM 25 that considers the appearance frequency of the local feature in the video and the normalization based on the video length, and the EBM 25 further considering the frequency of appearance of the local feature and the IFF in the instance image that is the query. It can be seen that the search accuracy of the method of the present embodiment gives results exceeding BM25. It has been confirmed that the difference in MAP is significant by a test at a risk level of 1%.

  In the future, video media is expected to increase explosively with the development of recording and recording devices, social networking services, etc., and robust video search technology that can cope with such situations is required. By using the method of the form, it becomes possible to realize a highly accurate instance search from a large-scale video database.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, although cases have been described with the above embodiments where the search target image is an image, the present invention can also be applied to a case where the search target image is a still image. In this case, for example, when 10,000 still images are input as search targets, the number of appearances KF of feature points in the video is set to the number of appearances KF ′ of feature points in the still image, and vl that is the sum of KF in the video is set. The above-described EBM 25 may be applied to vl ′, which is the total number of appearances of feature points in the still image, by replacing avdl, which is the average value of vl, with avdl ′, which is the average value of vl ′. The IFF is calculated by the equation (1) in a set of 10,000 still images.

  When the search target is a still image, search results in various forms based on the value of the EBM 25, such as a still image file name, a still image file name, and an EBM 25 value, are output as search results. Can do. Furthermore, when the search target is both a video and a still image, a search result may be output in which video and still images are mixed.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10,210 Image | video search apparatus 11 Instance image feature extraction part 12, 212 Feature aggregation total part 13 Frame image extraction part 14 Frame image feature extraction part 15, 215 Feature collation totaling part 16 Reverse frame image frequency calculation part 17 Search ranking part 18 Search Result output unit 21 Aggregated feature DB
22 Instance feature DB
23 Frame image feature DB
24 Video feature DB
25 Reverse frame image frequency DB
26 Additional features DB

Claims (5)

Feature extraction means for extracting a plurality of features from each frame image of an image group consisting of an instance image indicating an instance to be queried and a plurality of frame images to be searched;
The first for each instance in the plurality of instance images of the aggregated feature obtained by aggregating overlapping features from the plurality of features extracted from each of the plurality of instance images indicating each of the plurality of instances by the feature extraction unit. A feature aggregation and aggregation means for counting the appearance frequency;
Each of the aggregated features is compared with a plurality of features extracted from the frame image, and the second appearance frequency in each frame image and the third appearance frequency in the image group are totaled for each of the aggregated features. A feature matching and aggregation means;
For each of the aggregate features, based on the second appearance frequency, an inverse frame image frequency calculation unit that calculates a reverse frame image frequency that increases as the appearance frequency of the aggregate feature in the image group decreases;
The importance level of each aggregated feature determined based on the image frame length related to the reverse frame image frequency, the first appearance frequency, and the third appearance frequency of each aggregated feature, and the number of aggregated features included in the image group. A search result creating means for obtaining an evaluation value of each image group for a search target instance represented by a sum, and creating a search result based on the evaluation value;
Image search device including The feature aggregation and aggregation means calculates the aggregation result of the first appearance frequency based on an additional feature not included in the aggregation feature among features extracted from an instance image indicating an instance newly added as a search target. Updated,
The feature collation tabulation unit collates the additional feature with a plurality of features extracted from the frame image, updates the second appearance frequency and the third appearance frequency,
The inverse frame image frequency calculating means recalculates the inverse frame image frequency based on the updated second appearance frequency,
The search result creating means creates a search result for the newly added instance based on the recalculated reverse frame image frequency, the updated first appearance frequency, and the updated third appearance frequency. The image search device according to claim 1.   The image search apparatus according to claim 1, wherein the search result creation unit includes a file name of the image group or a file name of the image group and an evaluation value of the image group in the search result. An image search method in an image search apparatus including a feature extraction unit, a feature aggregation totaling unit, a feature matching totalization unit, an inverse frame image frequency calculation unit, and a search result creation unit,
The feature extraction means extracts a plurality of features from each frame image of an image group consisting of an instance image indicating an instance to be a query and a plurality of frame images to be searched;
The first for each instance in the plurality of instance images of the aggregated feature obtained by aggregating overlapping features from the plurality of features extracted from each of the plurality of instance images indicating each of the plurality of instances by the feature extraction unit. Total frequency of appearance,
The feature collating and summing unit collates each of the aggregated features with a plurality of features extracted from the frame image, and for each of the aggregated features, a second appearance frequency in each frame image, and in the image group Aggregate the third appearance frequency,
The inverse frame image frequency calculating means calculates, for each of the aggregate features, an inverse frame image frequency that increases as the appearance frequency of the aggregate feature in the image group decreases based on the second appearance frequency;
The search result creation means is determined based on an image group length related to the reverse frame image frequency, the first appearance frequency, and the third appearance frequency of each of the aggregate features, and the number of aggregate features included in the image group. An image search method for obtaining an evaluation value of each image group for a search target instance represented by a sum of importance of each aggregated feature and creating a search result based on the evaluation value.   The image search program for functioning a computer as each means which comprises the image search device of any one of Claims 1-3.

Images