JP5833499B2 - Retrieval device and program for retrieving content expressed by high-dimensional feature vector set with high accuracy - Google Patents

Description

  The present invention relates to a technique for accurately retrieving reference content similar to a query content (search key) similarly represented by a set of feature vectors from a set of reference content represented by a set of feature vectors. In particular, it is suitable for searching multimedia contents (for example, images) represented by a set of high-dimensional feature vectors.

  In recent years, not limited to online / offline, it has become possible to accumulate a large amount of content as the capacity of the storage increases. In addition, with the widespread use of information terminal devices typified by mobile phones and smartphones, digital content such as photograph data acquired by the user can be easily stored in a large amount in a database. Offline databases include storage devices such as HDD (Hard Disk Drive), DVD (Digital Versatile Disk), and Blu-ray disc. Online databases include social network services such as Flickr (registered trademark) and MySpace (registered trademark). According to these storage devices and services, a technique for searching for a large amount and various multimedia contents of individuals stored in a database becomes important.

  In order to search for multimedia contents, there is a technique for extracting a large number of feature vectors from these contents and outputting contents having a high degree of similarity between sets of feature vectors as search results. According to this technique, feature vectors of multimedia content are quantized and a histogram is created from the frequency of the quantized feature vectors. The similarity (distance) is calculated by the distance of the L1 norm or L2 norm between the histograms. The norm represents the distance between two points. The L1 norm means the sum of the absolute values of the dimensions of the two points, and the L2 norm means the sum of the squares of the values of the two points.

  There is also a technique for extracting a large amount of local feature vectors from image content, vector quantizing them, and calculating the similarity based on the number of local feature vectors vector-quantized to the same representative vector (for example, Non-Patent Document 1). reference).

  Furthermore, there is a technique for extracting a plurality of local invariant feature amounts from an image, making a histogram of the frequency of feature vectors, and calculating the similarity between the image and the category based on the overlapping ratio of the histograms (see, for example, Patent Document 1). . According to this technique, features (for example, background features) that are not necessary for pattern recognition of a subject can be removed based on the histogram. As a result, the feature of the object can be extracted without previously separating the object and the non-object from the image.

  Conventionally, a similar image search framework using local features is referred to as “Bag-of-Visual Words” (or Bag-of-Features, Bag-of-Keypoints) (see, for example, Non-Patent Document 1). According to this technique, a sentence retrieval method using a Bag-of-Words model and a transposed index is applied to retrieval of similar images. Bag-of-Words expresses a sentence as a feature vector defined by the frequency of one word, and uses IDF (Inverse Document Frequency) derived in advance based on the sentence set to determine the similarity between sentences. It is a framework to derive. On the other hand, Bag-of-Visual Words quantizes the local feature quantity of an image, regards the local feature quantity after quantization as a word, and expresses it as one feature vector similarly defined by the frequency. The same analogy method can be applied using the weighting used.

JP 2010-282581 A JP 2009-020769A

J. Sivic et al., "Video Google: A Text Retrieval Approach to Object Matching in Videos," in Proc. ICCV, 2003. H. Jegou, M. Douze, and C. Schmid, "Improving bag-offeatures for large scale image search," in IJCV, vol.87, no.3, pp.316-336, 2010. Y. Uchida, M. Agrawal, and S. Sakazawa, "Accurate Content-Based Video Copy Detection with Efficient Feature Indexing," in Proc. Of ICMR, 2011. D. G. Lowe, "Distinctive Image Features from Scale-InvariantKeypoints," International Journal of Computer Vision, vol. 60, no. 2, pp.91-110, 2004. H. Jegou, M. Douze, and C. Schmid, "Product quantization fornearest neighbor search," in IEEE Trans. On PAMI, vol. 33, no. 1, pp117-128, 2011. O. Boiman, E. Shechtman, and M. Irani, "In defense of nearest-neighbor based image classification," in Proc. Of CVPR, 2008.

  However, according to the existing Bag-of-Visual Words technique, IDF in sentence search is used when calculating a score of similarity between contents based on a feature vector. IDF represents an index for identifying how characteristic a specific word that appears in a sentence is for the purpose of text mining. In the case of IDF, it is designed on the premise that “each sentence is searched by a small number of words included in the sentence”, such as proper nouns. In other words, it is a premise that “each sentence is not searched by a word not included in the sentence”. Specifically, the IDF is calculated as the characteristic degree of the word in the corpus from the number of times a specific word appears in the sentence and the natural logarithm of the number of sentences including the sentence in the entire corpus. Is done.

  On the other hand, in the case of image search, it is necessary to extract a high-dimensional feature vector from a local invariant feature region. For example, according to a typical SIFT (Scale-Invariant Feature Transform) for extracting a feature vector used for object recognition, a feature region is divided into a plurality of blocks, and the direction of the luminance gradient from each block is used as a weighted histogram. Extract.

  Here, there are cases where the query content (content of the search key) contains a lot of feature vectors unrelated to the target with respect to the reference content (content to be searched). Specifically, this is a case where an image obtained by photographing the object with a camera is used as query content. In the reference content, for example, the background is white and only the search target object is shown, whereas in the query content, not only the target object but also various objects appear in the background. That is, various feature vectors irrelevant to the target object are detected in the background of the query content. This causes a decrease in search accuracy.

  Therefore, according to the present invention, when searching for a high-dimensional feature vector set, scoring is performed on the reference content in consideration that the feature vector of the query content includes an irrelevant feature vector. It is an object of the present invention to provide a search device and a program that can perform the above-mentioned.

According to the present invention, there is provided a search device for searching reference content similar to query content represented by a set of feature vectors from a set of reference content represented by a set of feature vectors,
Reference information storage means for storing a reference content identifier in association with each feature vector extracted from a plurality of reference contents R _j ;
Similar vector search means for searching for at least one feature vector set D of reference contents having similar feature vectors for each feature vector q _i of query contents using reference information storage means;
Using the mixed parameter λ, each feature vector q _i of the query content is generated from the probability λ · p (q _i | R _j ) generated from each searched reference content and a background model unrelated to the reference content. Based on the probability ratio with respect to the probability (1-λ) · p (q _i ), the score is added for each reference content R _j for all feature vectors q _i of the query content, and finally In particular, it has a voting means for outputting, as a search result, reference content R _j that has obtained a higher score above a predetermined threshold.

According to another embodiment of the search device of the present invention, the voting means includes the probability ratio of the number of appearances of feature vectors of all reference contents (D _s ) included in the set D of feature vectors of the searched reference contents. It is also preferable to calculate based on the ratio (n _j / D _s ) to the number of appearances of the feature vector (n _j ) of the reference content R _j .

According to another embodiment of the search device of the present invention, the voting means further calculates the probability ratio based on the ratio in the following equation (number of feature vectors of the reference content R _j included in the set D ( n _j ) ×
Number of feature vectors of all reference content (| R _all |)) /
(Number of feature vectors of all reference contents included in set D (D _s ) ×
Number of feature vectors of the reference content R _j (R _j ))
It is also preferable.

According to another embodiment of the search device of the present invention,
The similar vector search means classifies the set of feature vectors of the searched reference content into one or more subsets (clusters) D according to the average similarity to the feature vector q _i of the query content. Order subsets D up to the number m (≧ 1) of neighborhoods of
Preferably, the voting means calculates a score for each reference content for each subset D _t (1 ≦ t ≦ m), and uses a score in the subset D _t ′ that maximizes the score for voting.

According to another embodiment of the search device of the present invention,
The voting means calculates the score s _j of the reference content j with respect to the feature vector q _i of the query content with respect to the set D _t of feature vectors from the top to the t-th feature vector s _j = max _j log {λ / (1-λ) · (n _tj · | R _all |) / (Σ _{s = 1} ^t | D _s (q _i ) | · | R _j |) +1)}
n _tj : included in t-th subset D _t
Number of feature vectors for reference content j
Σ _{s = 1} ^t | D _s (q _i ) |: included in t-th subset D _t
Number of feature vectors of all reference content j
| R _all |: Number of all feature vectors in all reference contents
| R _j |: Number of all feature vectors in the reference content j
λ, 1-λ: Mixing parameters are also preferred.

According to another embodiment of the search device of the present invention,
The reference information storage means further stores an average representative vector of feature vectors in association with each subset (cluster) D,
It is also preferable that the similar vector search means searches the subset D by comparing each feature vector q _i of the query content with the representative vector of each subset D.

According to another embodiment of the search device of the present invention, the similar vector search means includes:
First means for searching for k subsets of feature vectors of reference content that are similar to each feature vector q _i of query content;
A second means for counting the number L of feature vectors of reference content included in the subset of the number of neighbors k ;
A third means for searching for upper m (m ≦ L) feature vectors similar to q _i among feature vectors of L reference contents included in a subset of k neighbors;
The number m of feature vectors in the third means is updated according to the number L of feature vectors counted by the second means ,
It is also preferable that D _t (qi) (1 ≦ t ≦ m) is composed of only the t-th feature vector.

According to another embodiment of the search device of the present invention, the number m of feature vectors in the third means is determined by αL (α ≦ 1) using the number L of feature vectors obtained by the second means . It is also preferable.

According to another embodiment of the search device of the present invention, the number m of feature vectors in the third means is determined by L ^α (α ≦ 1) using the number L of feature vectors obtained by the second means . It is also preferred that

According to another embodiment of the search device of the present invention,
It further has a feature vector set extraction means for extracting feature vectors from the reference content and the query content,
The feature vector set extraction means can output a plurality of feature vectors for each different type of algorithm,
The voting means calculates a score s for each of different types of feature vectors for each of the query content and the reference content, and uses a value obtained by weighted sum of the scores s of the different types of feature vectors for each reference content as a final score. It is also preferable.

According to another embodiment of the search device of the present invention,
The query content and the reference content are images,
It is also preferable that at least one instance (object, object) belonging to the same object or the same category is shown in the image as the reference content.

According to the present invention, there is provided a search program for causing a computer mounted in an apparatus for searching for reference content similar to query content represented by a set of feature vectors to function from a set of reference content represented by a set of feature vectors. There,
Reference information storage means for storing a reference content identifier in association with each feature vector extracted from a plurality of reference contents R _j ;
Similar vector search means for searching for at least one feature vector set D of reference contents having similar feature vectors for each feature vector q _i of query contents using reference information storage means;
Using the mixed parameter λ, each feature vector q _i of the query content is generated from the probability λ · p (q _i | R _j ) generated from each searched reference content and a background model unrelated to the reference content. Based on the probability ratio with respect to the probability (1-λ) · p (q _i ), the score is added for each reference content R _j for all feature vectors q _i of the query content, and finally In particular, the computer is caused to function as voting means for outputting the reference content R _j having a higher score equal to or higher than a predetermined threshold as a search result.

  According to the search device and the program of the present invention, when searching for a high-dimensional feature vector set, scoring for reference content is performed in consideration that the feature vector of query content includes an irrelevant feature vector. Can do.

It is a functional block diagram of the search device in this invention. It is explanatory drawing showing the processing content of a reference information generation part. It is explanatory drawing which votes from one set D of the feature vector of a reference content. FIG. 6 is an explanatory diagram for voting from a plurality of sets D of reference content feature vectors. It is explanatory drawing showing a hierarchical codebook. It is explanatory drawing which votes from the several feature vector of several reference content.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  According to the search device and program of the present invention, the reference content that is most similar to the query content (content of the search key) is searched from among a large number of reference contents (contents to be searched).

  FIG. 1 is a functional configuration diagram of a search device according to the present invention.

  According to FIG. 1, the search device 1 includes a reference information storage unit 10, a feature vector set extraction unit 11, a reference information generation unit 12, a similar vector search unit 13, and a voting unit 14. These functional components are realized by executing a program that causes a computer installed in the apparatus to function.

  The search device 1 inputs a large number of reference contents in advance and stores information related to the reference contents in the reference information storage unit 10. In addition, the search device 1 inputs query content serving as a search key at the time of search, and uses the reference information storage unit 10 to search for reference content most similar to the query content.

  The query content and the reference content are images, for example. In this case, the image as the reference content includes at least one instance (object, object) belonging to the same object or the same category.

[Feature vector set extraction unit 11]
The feature vector set extraction unit 11 extracts a set of feature vectors from one multimedia content. For example, when the multimedia content is an image, the feature vector is a local feature vector extracted from the local feature region of the image.

Specifically, the feature vector set extraction unit 11 extracts a set of feature vectors for each reference content R _j, and the set of feature vectors is output to the reference information storage unit 10. The feature vector set extraction unit 11 extracts a feature vector set Q (= {q _i }) from the query content, and the set of feature vectors is output to the similar vector search unit 13. The feature vector of the reference content and the feature vector of the query content have the same number of dimensions.

  For example, SIFT or SURF (Speeded Up Robust Features) is used as an algorithm for extracting a feature vector used for object recognition. For example, in the case of SIFT, a set of 128-dimensional feature vectors is extracted from one image (for example, see Non-Patent Document 4). SIFT is a technique for analyzing a characteristic local region using a scale space and describing a feature vector that is invariant to scale change and rotation. On the other hand, in the case of SURF, higher-speed processing is possible than SIFT, and a set of 64-dimensional feature vectors is extracted from one image.

[Reference information generator 12]
The reference information generation unit 12 executes the following process on the reference content feature vector set R _j and outputs a codebook to the reference information storage unit 10.

  FIG. 2 is an explanatory diagram illustrating processing contents of the reference information generation unit.

(S21) The set of reference content feature vectors is clustered into k clusters. For clustering, for example, k-means is used.
(S22) Next, a representative vector is derived for each cluster (average vector or median vector). This representative vector is also called "Visual Words".
(S23) A code book in which a unique ID n (= 1 to N) is assigned to each representative vector is generated.

For example, the representative vector f _n having the smallest distance from the input feature vector f is calculated.
Representative vector f _n = argmin _n || f−f _n || ²
Here, the code book is obtained by associating one or more reference content IDs (identifiers) belonging to the cluster for each representative vector f _n .

[Reference information storage unit 10]
Reference information storage unit 10 in association with the respective feature vectors extracted from a plurality of reference content R _j, and stores the reference content identifier. Here, the reference information storage unit 10 may store the code book (transposed index) output from the reference information generation unit 12. The code book is obtained by dividing feature vectors extracted from a plurality of reference contents R _j into a plurality of sets (clusters) D according to the degree of similarity. An average representative vector is assigned.

In the following embodiment, a case where a set of feature vectors R _j is extracted from a plurality of reference contents j will be described in detail. On the other hand, as in Non-Patent Document 6, for example, the query content can be classified into categories by setting the feature vector set R _j as a set of feature vectors based on a specific category. In this case, as will be described later, a score for each category can be calculated for the query content and classified into a plurality of top categories having the highest score, or a tag of a category having a score of a certain level or more can be added. .

[Similar vector search unit 13]
The similar vector search unit 13 uses the reference information storage unit 10 to search for at least one or more sets D of reference content feature vectors having similar feature vectors for each feature vector q _i of the query content. The shorter the distance between the query content feature vector q _i and the reference content feature vector, the higher the similarity. Specifically, a method using Cartesian product quantization (for example, see Non-Patent Document 5) or a method using Hamming Embedding (for example, see Non-Patent Document 2), which is one of the Approximate Nearest Neighbor algorithms. It is also preferable to use LSH (Locality-Sensitive Hashing). The reference content ID based on the searched set D of one or more feature vectors is output to the voting unit 14.

[Voting section 14]
Voting section 14, the adding the scores for each reference content R _j, performed for all feature vectors q _i of the query content, finally, reference content R _j to obtain a score of more than the upper predetermined threshold value Is output as a search result. According to the present invention, the voting unit 14 is voted by the calculation formula described in detail below, whereas the voting unit 14 has voted by the IDF according to the prior art.

According to the present invention, the reference content j ′ that is most likely to have generated the query content is derived. The following equation represents the posterior probability and represents the probability p that the query content would have been generated from the jth reference content.
j ′ = argmax _j p (R _j | Q)
Q: Set of feature vectors of query content R _j : Set of feature vectors of j-th reference content p (R _j | Q): From set Q of feature vectors of query content
A posteriori probability that a set of reference content feature vectors R _j is generated
argmax _j : means to derive j that maximizes the posterior probability of the right term

In general, the following posterior probability formula is established using Bayes' theorem. In this method, the posterior probability is calculated by multiplying the prior probability by the likelihood probability.
j ′ = argmax _j p (R _j | Q) = argmax _j p (Q | R _j ) p (R _j )
p (Q | R _j ): From the set R _j of reference content feature vectors,
Likelihood probability p (R _j ) that a set Q of query content feature vectors is generated: A set Rj of reference content feature vectors is searched
Prior probability
(The higher p (R _j ) means the higher the probability of being searched)

Here, it is assumed that the reference content to be searched is not biased and p (R _j ) is constant regardless of j. Then, p (R _j ) can be deleted, and is simply expressed as follows.
j ′ = argmax _j p (Q | R _j )

Here, it is assumed that the set Q of query content feature vectors is generated independently. “Independently generated” means that when a certain feature appears, there is no influence that a specific feature appears next, that is, it does not affect the previous result. In this case, it is the product of the probabilities based on the individual feature vectors q ₁ , q ₂ , q ₃ ,. In this case, it is represented by the following formula.
j ′ = argmax _j Π _{i = 1} ⁿ p (q _i | R _j )

Furthermore, the product of the probabilities can generally be expressed by the sum Σ of logs. This is because the magnitude relationship of the probabilities is maintained because it is a monotonically increasing function. In this case, it is represented by the following formula.
j ′ = argmax _j Π _{i = 1} ⁿ p (q _i | R _j ) = argmax _j Σ _{i = 1} ⁿ log p (q _i | R _j )

Here, each query feature vector is modeled as a linear combination of the probability generated from the feature vector set of the reference content and the probability generated from the background model unrelated to the reference content.
j ′ = argmax _j Σ _{i = 1} ⁿ logp (q _i | R _j )
= Argmax _j Σ _{i = 1} ⁿ log (λp (q _i | R _j ) + (1-λ) · p (q _i ))
= Argmax _j Σ _{i = 1} ⁿ (log (λp (q _i | R _j ) + (1-λ) · p (q _i )) − log (1-λ) · p (q _i ))
= Argmax _j Σ _{i = 1} ⁿ log {λ / (1-λ) · p (q _i | R _j ) / p (q _i ) +1}
i: ID of feature vector of query content
λ: Mixed parameter of linear combination p (q _i ): Probability generated from background model unrelated to reference content
(Based on the background image unrelated to the object in the query content)
λp (q _i | R _j ) + (1-λ) · p (q _i ):
The sum of the probability of p (q _i | R _j ) at λ and the probability of p (q _i ) at (1-λ) is
-Log (1-λ) · p (q _i ) which means the overall probability:
Even if an overall penalty is drawn for deformation, the order does not change.
For transformation of the formula described later.
λ / (1-λ) · p (q _i | R _j ) / p (q _i ) +1:
Based on "loga-logb = loga / b"

According to the present invention, the probability λ · p (q _i | R _j ) that each feature vector q _i of the query content is generated from each searched reference content using the mixed parameter λ, the reference content, The probability ratio with the probability (1-λ) · p (q _i ) generated from an irrelevant background model is used.

Here, it puts like the following formula.
s _ij = log {λ / (1-λ) · p (q _i | R _j ) / p (q _i ) +1}
i: ID of feature vector of query content
j: ID of the reference content
q _i : Feature vector of query content R _j : Reference content s _ij means that when query content feature vector q _i is observed, each q _i means a score for obtaining reference content j . That is, when q _i is observed, each means the likelihood generated from the reference content j.

Then, “s _ij ” is calculated for all of the feature vector i and the reference content j of the query content. Then, the reference content R _j having the maximum score Σ _{i = 1} ⁿ s _ij is selected as a search result.

However, since s _ij must be calculated for all i for each i, the amount of calculation becomes enormous when targeting a large-scale database.

Therefore, according to the present invention, an approximation method is applied to extract a feature vector set D (q _i ) similar to q _i from the feature vector set of the reference content for the query content feature vector q _i . Then, the calculation of s _ij is approximated as follows.
“S _ij ” is calculated only for the reference content R _j including the feature vector corresponding to D (q _i ). For other R _j , p (q _i | R _j ) = 0. _{Since ij} = log (1) = 0, the score of the reference content not including the feature vector corresponding to D (q _i ) is not increased or decreased.

Here, D (q _i ) is further composed of m (1 to M) disjoint sets.
D (q _i ) = D ₁ (q _i ) ∪D ₂ (q _i ) ∪ ... ∪D _m (q _i )
In the reference information storage unit 10, a plurality of representative vectors in a large number of reference contents are registered in a code book. Each representative vector is associated with the ID of the reference content. Here, the set of feature vectors of the reference content associated with each representative vector does not overlap each other among the m feature vectors. That is, it can be said that they are “prime”.

For Dt (q _i ) and Ds (q _i ), if t <s, the following inequality holds.
p (q _i | Dt (q _i ))> p (q _i | Ds (q _i ))
That is, when t <s, q _i has a higher probability of being generated from Dt (q _i ) than Ds (q _i ). For each of D ₁ (q _i )... D _m (q _i ), s _ij is calculated as follows.

For each Dt (q _i ), p (q _i | R _j ) and p (q _i ) are calculated by the k neighborhood density estimation method.
p (q _i | R _j ) = n _tj / (| R _j | · V _t )
p (q _i ) = Σ _{s = 1} ^t | Ds (q _i ) | / (| Rall | · V _t )
n _tj : D ₁ (q _i )... Dt (q _i ) number of feature vectors of R _j Rall: feature vector set of all reference contents V _t : distance between q _i and r′t as radius Volume of a supersphere (more than 3 dimensions)
(In addition, if 2D, it represents the area, 1D represents the length)
Rt ': Here farther feature vector from the most q _i in Dt (q _i), p (q _i) is, | Rall | of (N in k-nearest neighbor density estimation) number, Σ _{s = 1} ^t The probability that | Ds (q _i ) | (k in the k neighborhood density estimation method) falls is divided by the volume Vt.

Here, the case of calculating the V _t, required a great deal of computational complexity. Therefore, according to the present invention, the term of V _t is deleted by substituting it into the expression of s _ij , and calculation is performed according to the following expression.
s _ij = log {λ / (1-λ) · n _tj | Rall | / Σ _{s = 1} ^t (| Ds (q _i ) | · | R _j |) +1}
As for this score, only the largest score is added for each image.

That is, the voting unit 14 of the present invention calculates the score s _j of the reference content j with respect to the feature vector q _i of the query content for the set D _t of feature vectors from the top to the t-th by the following equation.
s _j = max _j log {λ / (1-λ) · (n _tj · | R _all |) / (Σ _{s = 1} ^t D _s (q _i ) · | R _j |) +1)}
n _tj : included in t-th subset D _t
Number of feature vectors for reference content j
Σ _{s = 1} ^t D _s (q _i ): included in t-th subset D _t
Number of feature vectors of all reference content j
| R _all |: Number of all feature vectors in all reference contents
| R _j |: Number of all feature vectors in the reference content j
λ, 1-λ: Mixing parameters

According to the above equation, the probability ratio “n _tj / Σ _{s = 1} ^t (| Ds (q _i ) |” is the value of all the reference contents Rj included in the set D of feature vectors of the searched reference contents. The appearance frequency is represented by the ratio of the number of appearances to the number of appearances of each reference content Rj.

[Specific Processing Contents in Similar Vector Search Unit 13 and Voting Unit 14]
Below, the specific processing content in the similar vector search part 13 and the voting part 14 in this invention is explained in full detail.

During the search, a set Q of feature vectors is extracted from the query content, and for each feature vector q _i , one or more sets D of reference content feature vectors linked to the representative vector are searched by vector quantization. The Then, vote for the corresponding reference content ID. After voting for all the feature vectors q _i of the query content, a reference content ID having a higher score is used as a search result.

  FIG. 3 is an explanatory diagram for voting from one set D of feature vectors of reference content.

It is also preferable that the similar vector search unit 13 searches for the top m sets D in the order similar to each feature vector q _i of the query content. According to FIG. 3, the top one set D (q _i ) is searched (m = 1, D (q _i ) = D ₁ (q _i )). Eight reference content IDs are registered in one set. However, note that the 8 IDs are not ordered. Among the eight reference content IDs, there are four (1, 4, 5, 6) unique IDs.

Reference content ID = 1: n ₁₁ = 3
score ₁ = score ₁ + log {λ / (1-λ) · (3 | Rall | / 8 | R ₁ |) +1}
(3 out of 8 means ID = 1)
Reference content ID = 4: n ₁₄ = 2
score ₄ = score ₄ + log {λ / (1-λ) · (2 | Rall | / 8 | R ₁ |) +1}
(2 out of 8 means ID = 4)
Reference content ID = 5: n ₁₅ = 1
score ₅ = score ₅ + log {λ / (1-λ) · (1 | Rall | / 8 | R ₁ |) +1}
(1 out of 8 means ID = 5)
Reference content ID = 6: n ₁₆ = 2
score ₅ = score ₅ + log {λ / (1-λ) · (2 | Rall | / 8 | R ₁ |) +1}
(2 out of 8 means ID = 6)
According to the present invention, the voting unit 14 calculates a score for each reference content for each subset D _t (1 ≦ t ≦ m), and uses the score in the subset D _t ′ that maximizes the score for voting. .

  FIG. 4 is an explanatory diagram for voting from a plurality of sets D of reference content feature vectors.

3 votes from the set D associated with the nearest representative vector, whereas FIG. 4 votes from the set D associated with the representative vector near k. The similar vector search unit 13 divides the set of feature vectors of the searched reference content into one or more subsets D according to the average similarity to the feature vector q _i of the query content, The subsets D up to several m (m ≧ 1) may be ordered.

According to FIG. 4, the similar vector search unit searches for three sets D (q _i ) of feature vectors of the reference content by k-neighbor search (m = 3) for one feature vector q _i of the query content. (m = 3, D (q _i ) = D ₁ (q _i ), D ₂ (q _i ), D ₃ (q _i )). According to FIG. 4, three reference content IDs are registered in the set of D ₁ (q _i ), and three reference content IDs are registered in the set of D ₂ (q _i ). Two reference content IDs are registered in the set of D ₃ (q _i ). According to FIG. 4, the order is in the order of D _1- > D _2- > D ₃ .

(First set t = 1, | D ₁ (q _i ) | = 3, Σ _{s = 1} ^t | D ₁ (q _i ) | = 3)
Reference content ID = 1: n ₁₁ = 2
score ₁ = score ₁ + log {λ / (1-λ) · (2 | Rall | / 3 | R ₁ |) +1}
Reference content ID = 4: n ₁₄ = 1
score ₄ = score ₄ + log {λ / (1-λ) · (1 | Rall | / 3 | R ₄ |) +1}
(Second set t = 2, | D ₂ (q _i ) | = 3, Σ _{s = 1} ^t | D ₂ (q _i ) | = 6)
Reference content ID = 5: n ₁₅ = 1
score ₅ = score ₅ + log {λ / (1-λ) · (1 | Rall | / 6 | R ₅ |) +1}
Reference content ID = 4: n ₁₄ = 1 × score ₄ = score ₄ + log {λ / (1-λ) · (2 | Rall | / 6 | R ₄ |) +1}
* Here, since it is the same as score ₄ obtained in D ₁ above, it is not adopted.
Reference content ID = 1: n ₁₁ = 1 piece xscore ₁ = score ₁ + log {λ / (1-λ) · (3 | Rall | / 6 | R ₁ |) +1}
★ Here, because it is smaller than score ₁ obtained with D ₁ above (2/3> 3/6),
Not adopted.
(Third set t = 3, | D ₃ (q _i ) | = 2, Σ _{s = 1} ^t | D ₃ (q _i ) | = 8)
Reference content ID = 6: n ₁₆ = 2
score ₆ = score ₆ + log {λ / (1-λ) · (2 | Rall | / 8 | R ₆ |) +1}

  FIG. 5 is an explanatory diagram showing a hierarchical code book.

  According to FIG. 5, the code book is hierarchically configured as compared with FIGS. 3 and 4 (see, for example, Non-Patent Document 7). Even in such a case, a score can be voted for each set of reference content feature vectors, as in FIGS. 3 and 4 described above.

  FIG. 6 is an explanatory diagram for voting from a plurality of feature vectors of a plurality of reference contents.

  According to FIG. 6, a plurality of feature vectors of a plurality of reference contents are not configured as a set D as shown in FIGS. From each feature vector of the query content, using the method of Cartesian product quantization, the method using Hamming Embedding (see Non-Patent Document 2, for example), or an algorithm such as LSH, the neighborhood feature of the reference content is simply used as an m neighborhood search. The vector has been searched.

(First feature vector t = 1, | D ₁ (q _i ) | = 1, Σ _{s = 1} ^t | D ₁ (q _i ) | = 1)
Reference content ID = 1: n ₁₁ = 1
score ₁ = score ₁ + log {λ / (1-λ) · (1 | Rall | / 1 | R ₁ |) +1}
(Second feature vector t = 2, | D ₂ (q _i ) | = 1, Σ _{s = 1} ^t | D ₂ (q _i ) | = 2)
Reference content ID = 4: n ₂₄ = 1
score ₄ = score ₄ + log {λ / (1-λ) · (1 | Rall | / 2 | R ₁ |) +1}
(Third feature vector t = 3, | D ₃ (q _i ) | = 1, Σ _{s = 1} ^t | D ₃ (q _i ) | = 3)
Reference content ID = 1: n ₃₁ = 1 piece x score ₁ = score ₁ + log {λ / (1-λ) · (2 | Rall | / 3 | R ₁ |) +1}
★ Here, because it is smaller than the score ₁ obtained in the previous D ₁ (1/1> 2/3),
Not adopted.
(Fourth feature vector t = 4, | D ₄ (q _i ) | = 1, Σ _{s = 1} ^t | D ₄ (q _i ) | = 4)
Reference content ID = 5: n ₄₅ = 1
score ₅ = score ₅ + log {λ / (1-λ) · (1 | Rall | / 4 | R ₁ |) +1}
(Fifth feature vector t = 5, | D ₅ (q _i ) | = 1, Σ _{s = 1} ^t | D ₅ (q _i ) | = 5)
Reference content ID = 4: n ₅₄ = 1
score ₄ = score ₄ + log {λ / (1-λ) · (2 | Rall | / 5 | R ₁ |) +1}
★ Here, because it is smaller than the score ₄ obtained in D ₂ above (1/2> 2/5),
Not adopted.
(Sixth feature vector t = 6, | D ₆ (q _i ) | = 1, Σ _{s = 1} ^t | D ₆ (q _i ) | = 6)
Reference content ID = 1: n ₆₁ = 1
score ₁ = score ₁ + log {λ / (1-λ) · (3 | Rall | / 6 | R ₁ |) +1}
★ Here, because it is smaller than the score ₁ obtained in D ₁ above (1/1> 3/6),
Not adopted.
(Seventh feature vector t = 7, | D ₇ (q _i ) | = 1, Σ _{s = 1} ^t | D ₇ (q _i ) | = 7)
Reference content ID = 6: n ₇₆ = 1
score ₆ = score ₆ + log {λ / (1-λ) · (1 | Rall | / 7 | R ₁ |) +1}
(Eighth feature vector t = 8, | D ₈ (q _i ) | = 1, Σ _{s = 1} ^t | D ₈ (q _i ) | = 8)
Reference content ID = 6: _n86 = 1
score ₆ = score ₆ + log {λ / (1-λ) · (2 | Rall | / 8 | R ₁ |) +1}
★ Here, because it is larger than the score ₆ obtained in D ₇ above (1/7 <2/8),
The score ₆ obtained in the previous D ₇ is not adopted.

[Method for Determining the Number m of Neighboring Feature Vectors in Similar Vector Search Unit 13]
(When using direct product quantization, Hamming Embedding, two-step m-neighbor search such as LSH)
It is also preferable that the similar vector search unit 13 updates (varies) the number m of neighboring feature vectors without making the fixed value a fixed value. In this case, the similar vector search unit 13 includes the following two steps (for example, see Non-Patent Documents 2 and 5).

(S1) Search for k subsets of feature vectors of reference content similar to each feature vector q _i of query content. That is, in S1, it narrows down by calculating | requiring a rough neighborhood set. For example, it is also preferable to derive a rough neighborhood set using vector quantization (see, for example, Non-Patent Documents 2 and 5).

(S2) Next, the number L of reference content feature vectors included in the k subsets of the number of neighbors is counted. This number L reflects the density of feature vectors around the feature vector of the query content. For example, for distance calculation in S2, a code obtained by binary encoding of a feature vector may be used (for example, refer to Non-Patent Document 2), or a feature vector may be encoded by direct product quantization. (For example, refer nonpatent literature 3).

(S3) Next, among the L reference content feature vectors included in the k subsets of neighborhoods, the top m (m ≦ L) feature vectors similar to q _i are searched. That is, an (approximate) distance between q _i and the feature vector of the reference content is further derived, and a stricter neighborhood set is derived. Then, the approximate feature vector number m in S1 is updated according to the number L of feature vectors counted in S2. For example, if the number L of feature vectors counted in S2 is greater than or equal to a predetermined threshold, the number m of neighboring feature vectors in S3 can be increased. Note that the t-th set D _t (qi) (1 ≦ t ≦ m) includes only the t-th feature vector.

Further, the number m of neighboring feature vectors in S1 may be determined by αL (α ≦ 1) using the number L of feature vectors obtained in S2. Further, it may be determined by L ^α (α ≦ 1).

  As another embodiment, the feature vector set extraction unit 11 preferably outputs a plurality of feature vectors for each different type of algorithm. Different types of algorithms are based, for example, on both SIFT and SURF. In this case, the voting unit 14 calculates a score s for each different type of feature vector for each of the query content and the reference content, and finally calculates a weighted sum of the scores s of the different types of feature vectors for each reference content. Score.

  As described above in detail, according to the search device and program of the present invention, when a high-dimensional feature vector set is searched, the feature vector of the query content includes an irrelevant feature vector. In consideration, the reference content can be scored.

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Search apparatus 10 Reference information storage part 11 Feature vector set extraction part 12 Reference information generation part 13 Similar vector search part 14 Voting part

Claims (12)

A search device for searching for reference content similar to query content represented by a set of feature vectors from a set of reference content represented by a set of feature vectors,
Reference information storage means for storing a reference content identifier in association with each feature vector extracted from a plurality of reference contents R _j ;
Using the reference information storage means, for each feature vector q _i of the query content, similar vector search means for searching for at least one set D of reference content feature vectors having similar feature vectors;
Using the mixed parameter λ, each feature vector q _i of the query content is generated from the probability λ · p (q _i | R _j ) generated from each searched reference content and a background model unrelated to the reference content. Based on the probability ratio with the generated probability (1-λ) · p (q _i ), adding a score for each reference content R _{j is} performed for all feature vectors q _i of the query content, A search apparatus, comprising: voting means for outputting, as a search result, reference content R _j that finally obtains a higher score above a predetermined threshold. The voting means uses the probability ratio for the appearance of feature vectors of the reference content R _j with respect to the number of appearances of feature vectors of all reference content (D _s ) included in the set D of feature vectors of the searched reference content. The search device according to claim 1, wherein calculation is performed based on a ratio (n _j / D _s ) to the number (n _j ). The voting means further calculates the probability ratio based on the ratio in the following formula (number of feature vectors of the reference content R _j included in the set D (n _j ) ×
Number of feature vectors of all reference content (| R _all |)) /
(Number of feature vectors of all reference contents included in set D (D _s ) ×
Number of feature vectors of the reference content R _j (R _j ))
The search device according to claim 2. The similarity vector search means classifies the set of feature vectors of the searched reference content into one or more subsets (clusters) D according to the average similarity to the feature vector q _i of the query content, Order subsets D up to the number m (≧ 1) of neighbors from
The voting means calculates a score for each reference content for each subset D _t (1 ≦ t ≦ m), and uses a score in the subset D _t ′ with the maximum score for voting. Item 4. The search device according to Item 2 or 3. The voting means calculates the score s _j of the reference content j with respect to the feature vector q _i of the query content for the set D _t of feature vectors from the top to the t th by the following formula: s _j = max _j log {λ / (1-λ) · (n _tj · | R _all |) / (Σ _{s = 1} ^t | D _s (q _i ) | · | R _j |) +1)}
n _tj : included in t-th subset D _t
Number of feature vectors for reference content j
Σ _{s = 1} ^t | D _s (q _i ) |: included in t-th subset D _t
Number of feature vectors of all reference content j
| R _all |: Number of all feature vectors in all reference contents
| R _j |: The number of all feature vectors in the reference content j λ, 1-λ: The mixed parameter according to claim 4. The reference information storage means further stores an average representative vector of feature vectors in association with each subset (cluster) D;
The search according to claim 4 or 5, wherein the similar vector search means searches the subset D by comparing each feature vector q _i of the query content with a representative vector of each subset D. apparatus. The similar vector search means includes:
First means for searching for k subsets of feature vectors of reference content that are similar to each feature vector q _i of query content;
A second means for counting the number L of feature vectors of reference content included in the subset of the number of neighbors k ;
A third means for searching for upper m (m ≦ L) feature vectors similar to q _i among feature vectors of L reference contents included in a subset of k neighbors;
The number m of feature vectors in the third means is updated according to the number L of feature vectors counted by the second means ,
7. The search device according to claim 4, wherein the D _t (qi) (1 ≦ t ≦ m) includes only a t-th feature vector. 8. The search apparatus according to claim 7, wherein the number m of feature vectors in the third means is determined by αL (α ≦ 1) using the number L of feature vectors obtained by the second means . . 8. The search according to claim 7, wherein the number m of feature vectors in the third means is determined by L ^α (α ≦ 1) using the number L of feature vectors obtained by the second means . apparatus. It further has a feature vector set extraction means for extracting feature vectors from the reference content and the query content,
The feature vector set extraction means can output a plurality of feature vectors for each different kind of algorithm,
The voting means calculates a score s for each of different types of feature vectors for each of the query content and the reference content, and uses a value obtained by weighting and summing the scores s of the different types of feature vectors for each reference content as a final score. The search device according to any one of claims 1 to 9, wherein: The query content and the reference content are images,
The search according to any one of claims 1 to 10, wherein the image as the reference content includes at least one instance (target object, object) belonging to the same object or the same category. apparatus. A search program for causing a computer mounted on an apparatus for searching reference content similar to query content represented by a set of feature vectors to function from a set of reference content represented by a set of feature vectors,
Reference information storage means for storing a reference content identifier in association with each feature vector extracted from a plurality of reference contents R _j ;
Using the reference information storage means, for each feature vector q _i of the query content, similar vector search means for searching for at least one set D of reference content feature vectors having similar feature vectors;
Using the mixed parameter λ, each feature vector q _i of the query content is generated from the probability λ · p (q _i | R _j ) generated from each searched reference content and a background model unrelated to the reference content. Based on the probability ratio with the generated probability (1-λ) · p (q _i ), adding a score for each reference content R _{j is} performed for all feature vectors q _i of the query content, A search program characterized by causing a computer to function as voting means for finally outputting the reference content R _j having a higher score above a predetermined threshold as a search result.

Images