JP5959446B2 - Retrieval device, program, and method for high-speed retrieval by expressing contents as a set of binary feature vectors - Google Patents

Description

  The present invention searches for a reference content similar to a query content (content serving as a search key) similarly represented by a set of feature vectors from a set of reference content (contents to be searched) represented by a set of feature vectors. Regarding technology. In particular, it is suitable for searching multimedia contents (for example, images) represented by a set of feature vectors.

  In recent years, not limited to online / offline, it has become possible to accumulate a large amount of content as the capacity of the storage increases. In addition, with the widespread use of information terminal devices typified by mobile phones and smartphones, digital content such as photograph data acquired by the user can be easily stored in a large amount in a database. Offline databases include storage devices such as HDD (Hard Disk Drive), DVD (Digital Versatile Disk), and Blu-ray disc. Online databases include social network services such as Flickr (registered trademark) and MySpace (registered trademark). According to these storage devices and services, a technique for searching for a large amount and various multimedia contents of individuals stored in a database becomes important.

  In order to search for multimedia contents, there is a technique for extracting a large number of feature vectors from these contents and outputting contents having a high degree of similarity between sets of feature vectors as search results. According to this technique, feature vectors of multimedia content are quantized and a histogram is created from the frequency of the quantized feature vectors. The similarity (distance) is calculated by the distance of the L1 norm or L2 norm between the histograms. The norm represents the distance between two points. The L1 norm means the sum of the absolute values of the dimensions of the two points, and the L2 norm means the sum of the squares of the values of the two points.

  There is also a technique for extracting a large amount of local feature vectors from image content, vector quantizing them, and calculating the similarity based on the number of local feature vectors vector-quantized to the same representative vector (for example, Non-Patent Document 1). reference).

  Furthermore, there is a technique for extracting a plurality of local invariant feature amounts from an image, making a histogram of the frequency of feature vectors, and calculating the similarity between the image and the category based on the overlapping ratio of the histograms (see, for example, Patent Document 1). . According to this technique, features (for example, background features) that are not necessary for pattern recognition of a subject can be removed based on the histogram. As a result, the feature of the object can be extracted without previously separating the object and the non-object from the image.

  Conventionally, a similar image search framework using local features is referred to as “Bag-of-Visual Words” (or Bag-of-Features, Bag-of-Keypoints) (see, for example, Non-Patent Document 1). According to this technique, a sentence retrieval method using a Bag-of-Words model and a transposed index is applied to retrieval of similar images. Bag-of-Words expresses a sentence as a feature vector defined by the frequency of one word, and uses IDF (Inverse Document Frequency) derived in advance based on the sentence set to determine the similarity between sentences. It is a framework to derive. On the other hand, Bag-of-Visual Words quantizes the local feature quantity of an image, regards the local feature quantity after quantization as a word, and expresses it as one feature vector similarly defined by the frequency. The same analogy method can be applied using the weighting used.

  Furthermore, in recent years, a technique based on the Fisher vector, which is an extension of the “Bag-of-Visual Words” framework, has attracted attention (see, for example, Non-Patent Document 3). According to the technique using the Fisher vector, the feature vector is modeled by a mixed Gaussian distribution, and the Fisher kernel related to the parameters of the mixed Gaussian distribution is explicitly mapped to the feature vector and used as the feature vector that represents the image. Can do. According to this technique, a set of feature vectors can be described by one fixed-length feature vector, and the Euclidean distance can be used as a distance measure between feature vectors.

  FIG. 1 is a functional configuration diagram of a conventional content search apparatus.

  According to the search device 1 of FIG. 1, in order to generate model parameters, a large number of training contents are input in advance and the model parameters are stored in advance. In addition, the search device 1 inputs in advance a large number of reference contents (contents to be searched) and stores in advance reference feature vectors converted to normalization using model parameters. Then, the search device 1 normalizes the query feature vector using the model parameter for the query content (the search key content), searches for the reference feature vector most similar to the query feature vector, and specifies the reference content To do.

  According to FIG. 1, the search device 1 includes a feature vector extraction unit 11, a model estimation unit 12, a model parameter storage unit 13, a feature vector conversion unit 14, a reference information storage unit 15, and a feature vector search unit 16. And have. These functional components are realized by executing a program that causes a computer installed in the apparatus to function.

  The feature vector extraction unit 11 extracts a set of feature vectors from each multimedia content. For example, when the multimedia content is an image, the feature vector is a local feature vector extracted from the local feature region of the image. The training content is converted into a set of feature vectors and output to the model estimation unit 12. Also, the reference content and the query content are each converted into a set of feature vectors and output to the feature vector conversion unit 14. For all these contents, feature vectors with the same number of dimensions (D dimensions) are extracted.

The feature vector extraction algorithm used for object recognition is, for example, SIFT (Scale-Invariant Feature Transform) or SURF (Speeded)
Up Robust Features) is used. For example, in the case of SIFT, a set of 128-dimensional feature vectors is extracted from one image. SIFT is a technique for analyzing a characteristic local region using a scale space and describing a feature vector that is invariant to scale change and rotation. On the other hand, in the case of SURF, higher-speed processing is possible than SIFT, and a set of 64-dimensional feature vectors is extracted from one image. While SIFT has a high processing cost and is difficult to perform real-time matching, SURF uses an integral image to speed up the processing.

  The model estimation unit 12 estimates model parameters of a Gaussian Mixture Model using the set of feature vectors of the training content output from the feature vector extraction unit 11, and outputs the model parameters. Bag-of-Features discrimination performance depends on the accuracy of probability density distribution modeling. A mixed Gaussian distribution is used because an arbitrary continuous density function can be expressed by adjusting the number of parameters and parameters.

  The model parameter storage unit 13 stores the model parameters output from the model estimation unit 12.

  The feature vector conversion unit 14 explicitly maps a set of feature vectors of the reference content and the query content to one fixed-length feature vector. For this mapping, a Fisher kernel based on the model parameters of the model parameter storage unit 13 is used. Specifically, the gradient vector of the log likelihood function of the model is obtained from the feature vector set, and is mapped to the feature vector by normalizing with a Fisher information matrix related to the model parameter. According to the technique described in Non-Patent Document 3, the Fisher information matrix is assumed to be a diagonal matrix. One transformed feature vector is called a Fisher vector. The feature vector conversion unit 14 outputs the Fisher vector mapped from the set of feature vectors of the reference content to the reference storage unit 15, and outputs the Fisher vector mapped from the set of feature vectors of the query content to the feature vector search unit 16. To do.

  The reference information accumulation unit 15 accumulates the normalized Fisher vector of the reference content output from the feature vector conversion unit 14.

  The feature vector search unit 16 uses the reference information storage unit 15 to search for the Fisher vector of the reference content that is most similar to the Fisher vector of the query content. Here, the Euclidean distance can be used, the Fisher vector of the reference content having a short distance from the Fisher vector of the query content is searched, and the reference content is specified.

JP 2010-282581 A

J. Sivic et al., "Video Google: A Text Retrieval Approach to Object Matching in Videos," in Proc. ICCV, 2003. D. G. Lowe, "Distinctive Image Features from Scale-InvariantKeypoints," International Journal of Computer Vision, vol. 60, no. 2, pp.91-110, 2004. F. Perronnin, J. Sanchez, and T. Mensink, "Improving the FisherKernel for Large-Scale Image Classification," in Proc. ECCV, 2010. E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, "ORB: Anefficient alternative to SIFT or SURF," in Proc. ICCV, 2011. A. Alahi, R. Ortiz, and P. Vandergheynst, "FREAK: Fast RetinaKeypoint," in Proc. CVPR, 2012. H. Jegou, M. Douze, and C. Schmid, "Product quantization fornearest neighbor search," in IEEE Trans. On PAMI, vol. 33, no. 1, pp117-128, 2011. Yohei Sanshin, “CVReadiing, ORB: an efficient alternative to SIFT or SURF”, [online], [Searched on December 5, 2012], Internet <URL: http://www.vision.cs.chubu.ac.jp /CV-R/jpdf/Rublee_iccv2011.pdf> Tatsuya Harada, “The Trend and Theory of General Object / Scene Recognition Using Large-Scale Data”, [online], [Search December 5, 2012], Internet <URL: https: //ipsj.ixsq.nii. ac.jp/ej/index.php?active_action=repository_view_main_item_detail&item_id=81096&item_no=1&page_id=13&block_id=8>

  However, with the widespread use of mobile terminals such as smartphones and tablet terminals, further memory saving and faster matching have been required for content search processing. In particular, according to the technical field of image recognition in the use of augmented reality (Augmented Reality), it is required to search for content at higher speed than SIFT or SURF in order to perform real-time processing.

  Therefore, an object of the present invention is to provide a search device, a program, and a method capable of searching for contents at a higher speed than SIFT and SURF.

According to the present invention, there is provided a search program that causes a computer installed in an apparatus to function to search for reference content similar to query content from a set of reference content using model parameters extracted from training content. And
For each of the training content, the reference content, and the query content, a feature vector extraction unit that extracts a set of D-dimensional binary feature vectors x _{1 to} x _T ,
From the set of binary feature vectors of the training content, the mixture ratio w _i for the i (1 ≦ i ≦ N) -th multivariate Bernoulli distribution and the d (1 ≦ d ≦ D) -th parameter of the i-th multivariate Bernoulli distribution model estimation means for calculating μ _id and the Fisher information amount f _id related to the parameter μ _id ;
Model parameter accumulating means for accumulating the mixing ratio w _i , parameter μ _id, and Fisher information amount f _id ;
From the set of binary feature vectors of the reference content or query content, one of the reference content or query content corresponding to the reference content or query content using the mixture ratio w _i , parameter μ _id, and Fisher information amount f _id stored in the model parameter storage means Feature vector conversion means for calculating a Fisher vector;
The computer is caused to function as a feature vector search unit that searches for the Fisher vector of the reference content that is most similar to the Fisher vector of the query content.

According to another embodiment of the search program of the present invention,
The feature vector extracting means preferably causes the computer to function to extract a set of binary feature vectors using ORB (Oriented FAST and Rotated BRIEF) or FRAK (Fast Retina Keypoint).

According to another embodiment of the search program of the present invention,
The model estimation means uses a set of binary feature vectors x _{1 to} x _T of the training content,
For the E (Expectation) step, estimate the expected value γ _t (i) of the latent variable i for each binary feature vector x _i ,
For the M (Maximization) step, update the mixture ratio w _i and parameter μ _i using the expected value γ _t (i),
By repeating these E step and M step until convergence, a parameter group λ of the mixture ratio w _i and parameter μ _i is calculated. Λ (w ₁ ,..., W _N and μ ₁₁ ,. _ND )
It is also preferable to make the computer function.

According to another embodiment of the search program of the present invention,
Model estimation means
Calculate a Fisher score s _{id in} which the log likelihood function of the parameter μ _id is defined by partial differentiation,
It is also preferable to cause the computer to function to calculate the Fisher information amount f _id as the variance of the Fisher score s _id .

According to another embodiment of the search program of the present invention,
Feature vector conversion means, for each set of binary feature vectors, calculates the Fischer score s _id using the parameter mu _id, they were calculated cumulative Fisher scores s' _id obtained by accumulating for each id,
It is also preferable to cause the computer to function to calculate a Fisher vector v _id obtained by dividing each accumulated Fisher score s ′ _id by the square root √f _id of the corresponding Fisher information amount f _id .

According to another embodiment of the search program of the present invention,
Model estimation means
Calculate a Fisher score s _{id in} which the log likelihood function of the parameter μ _id is defined by partial differentiation,
The principal component analysis is performed on the vector (s _{i1 to} s _iD ) for each mixed element i of the Fisher score s _id ,
As a result of the principal component analysis, K pieces having the largest eigenvalues are output as the Fisher information amount f _id ,
Further output K eigenvectors g _iK corresponding to the eigenvalues,
The model parameter storage means preferably causes the computer to function so as to further store the eigenvector g _iK .

According to another embodiment of the search program of the present invention,
Feature vector conversion means, for each set of binary feature vectors, calculates the Fischer score s _id using the parameter mu _id, they were calculated cumulative Fisher scores s' _id obtained by accumulating for each id,
A normalized vector v obtained by normalizing (projecting) a vector (s ′ _{i1 to} s _iD ) of each cumulative Fisher score s ′ _id using a corresponding eigenvector g _ik (g _{i1 to} g _iK ) for each mixed element i. 'Calculate _ik ,
It is also preferred to have the computer function to calculate a Fisher vector v _ik obtained by dividing the normalized vector v ′ _ik by the square root √g _ik of the corresponding eigenvector g _ik .

According to the present invention, a search device that searches for reference content similar to query content from a set of reference content using model parameters extracted from training content,
For each of the training content, the reference content, and the query content, a feature vector extraction unit that extracts a set of D-dimensional binary feature vectors x _{1 to} x _T ,
From the set of binary feature vectors of the training content, the mixture ratio w _i for the i (1 ≦ i ≦ N) -th multivariate Bernoulli distribution and the d (1 ≦ d ≦ D) -th parameter of the i-th multivariate Bernoulli distribution model estimation means for calculating μ _id and the Fisher information amount f _id related to the parameter μ _id ;
Model parameter accumulating means for accumulating the mixing ratio w _i , parameter μ _id, and Fisher information amount f _id ;
From a set of binary feature vectors of the reference content or query content, one Fisher vector corresponding to the reference content or query content is obtained using the mixture ratio w _i of the model parameter storage means, the parameter μ _id, and the Fisher information amount f _id. A feature vector conversion means for calculating;
And feature vector search means for searching for the Fisher vector of the reference content that is most similar to the Fisher vector of the query content.

According to the present invention, there is provided a search method for searching reference content similar to query content from a set of reference content using a model parameter extracted from training content using an apparatus,
As a first step of accumulating model parameters,
For each training content, a set of D-dimensional binary feature vectors x _{1 to} x _T is extracted,
From the set of binary feature vectors of the training content, the mixture ratio w _i for the i (1 ≦ i ≦ N) -th multivariate Bernoulli distribution and the d (1 ≦ d ≦ D) -th parameter of the i-th multivariate Bernoulli distribution Calculate μ _id and Fisher information amount f _id related to parameter μ _id ,
Accumulate the mixing ratio w _i , the parameter μ _id and the Fisher information amount f _id ,
As a second step of accumulating reference information,
For each reference content, extract a set of D-dimensional binary feature vectors,
From the set of binary feature vectors of each reference content, one Fisher vector is calculated using the mixture ratio w _i , the parameter μ _id, and the Fisher information amount f _id accumulated as model parameters.
Accumulate Fisher vector,
As a third step of searching reference content from query content,
From the set of binary feature vectors of each query content, one Fisher vector is calculated using the mixture ratio w _i , the parameter μ _id and the Fisher information amount f _id accumulated as model parameters,
It has a search for the Fisher vector of the reference content that is most similar to the Fisher vector of the query content.

  According to the search device, program, and method of the present invention, content can be searched at a higher speed than SIFT or SURF.

It is a functional block diagram of the content search apparatus in a prior art. It is a functional block diagram of the content search device in the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  According to the search device, program, and method of the present invention, reference content similar to query content is searched from a set of reference content using model parameters extracted from training content. Here, according to the present invention, extracting a binary feature vector from a feature vector extracted from content is a first feature. The second feature is that these binary feature vectors are modeled by a multivariate mixed Bernoulli distribution and a Fisher vector is extracted from the model parameters.

  FIG. 2 is a functional configuration diagram of the content search apparatus according to the present invention.

  The functional configuration of the search device of FIG. 2 is the same as the functional configuration of FIG. However, the processing contents in each functional component are different. Hereinafter, the feature vector extraction unit 11, the model estimation unit 12, the model parameter storage unit 13, the feature vector conversion unit 14, the reference information storage unit 15, and the feature vector search unit 16 in the search device 1 will be sequentially described. explain.

[Feature vector extraction unit 11]
The feature vector extraction unit 11 in the present invention extracts a set of D-dimensional binary feature vectors X = {x _{1 to} x _T } for each of training content, reference content, and query content. For example, when the multimedia content is an image, the feature vector is a local binary feature vector extracted from the local feature region of the image. A set of binary feature vectors extracted from the training content is output to the model estimation unit 12. Each set of binary feature vectors extracted from the reference content and the query content is output to the feature vector conversion unit 14.

  According to the present invention, an ORB (Oriented FAST and Rotated BRIEF) (for example, see Non-Patent Document 4 and Non-Patent Document 7) or FRAK (Fast Retina Keypoint) (for example, Non-Patent Document 5) is used as an algorithm for extracting binary feature vectors. Use. In the case of ORB, a set of 256-bit binary feature vectors is extracted from one content. For example, in order to execute matching at high speed, there is BRIEF (Binary Robust Independent Elementary Features) as a feature description by a binary code. According to the present invention, “ORB” is used which can describe a feature in which rotation invariance is introduced into BRIEF. In particular, according to the ORB, it is possible to maintain an accuracy equal to or higher than that of SIFT or SURF and realize a speed increase of several hundred times.

<About ORB>
The ORB is composed of two steps of “feature point detection processing” and “feature vector description processing”.

(Feature point detection processing)
According to the feature point detection process in the ORB, FAST (Features from Accelerated Segment Test) is used to detect key points at high speed. In addition, since FAST is not robust to scale changes, an image is converted into a plurality of sizes, and feature points are extracted from images of each size.

  Further, the existing FAST does not have an algorithm for calculating the key point orientation for obtaining rotation invariance. Therefore, ORB adopts Oriented FAST in order to obtain rotational invariance. By describing the features based on the orientation, even if the input image is rotated, the same key point can be detected as the same feature amount. Therefore, the direction vector of the center of the key point and the center of gravity of the brightness of the patch is used.

(Feature vector description processing)
Next, according to the feature vector description processing in the ORB, a binary feature vector is extracted for each detected feature point by using a BRIF feature vector descriptor. These are obtained from the magnitude relationship of the luminance of two pixels around the feature point.

  BRIEF can execute keypoint feature description by binary code. According to SIFT and SURF, high-dimensional real numbers are used for feature description. However, when a high-dimensional real number is used, there is an increase in memory capacity and similarity calculation. Therefore, by using BREF based on ORB, it is possible to save memory by describing features by binary code, and it is possible to reduce processing costs by using a Hamming distance for similarity calculation.

  According to BRIEF, a binary code is generated from the sign of the luminance difference between two points randomly selected in the patch. The pixels to be selected are randomly selected according to a Gaussian distribution centered on the key point position. Here, the ORB selects pixels using learning in order to perform matching with higher accuracy. The selected pixel position is used for feature description as a binary code with high feature description capability when the bit variance of the pair is large and the correlation of N pairs is low. N pairs are narrowed down using the Greedy algorithm.

[Model estimation unit 12]
The model estimation unit 12 calculates a mixture ratio w _i related to the i-th multivariate Bernoulli distribution and a d-th parameter μ _{id of} the i-th multivariate Bernoulli distribution from the set of binary feature vectors of the training content. These are calculated as model parameters λ.
λ (w ₁ ,..., w _N and μ ₁₁ ,..., μ _ND )
In addition, the model estimation unit 12 according to the present invention further calculates a Fisher information amount f _id related to the parameter μ _id .
f ₁₁ ,..., f _ND (N × D): Fisher information amount

N：混合数 <Calculation of parameters w _i and μ _id based on multivariate mixed Bernoulli distribution>
According to the present invention, a model parameter λ obtained by modeling a set of binary feature vectors with a “multivariate mixed Bernoulli distribution” is estimated. The Bernoulli distribution is a discrete probability distribution with a probability p of 1 and a probability q = 1−p of 0. If X is a random variable according to Bernoulli distribution, the mean of the random variable X is p, and the variance is pq = p (1-p). The “multivariate mixed Bernoulli distribution” expresses a probability p (x _t | λ) that a binary feature vector x _t is generated.

N: Number of mixtures

μ_id：i番目の多変量ベルヌーイ分布のd番目のパラメータ
x_t,d：バイナリ特徴ベクトルx_tのd番目のビット
D：バイナリ特徴ベクトルのビット長
p_i（x_t|λ）：バイナリ特徴ベクトルx_tがi番目の多変量ベルヌーイ分布から生成
された際に、d番目のビットが1となる確率 Since they are mixed distributions, different multivariate Bernoulli distributions from p ₁ to p _N are selected with the respective mixing ratios w _i to generate x _t . The probability that the binary feature vector x _t is generated from the i-th multivariate Bernoulli distribution is expressed by the following equation.

μ _id : d-th parameter of i-th multivariate Bernoulli distribution
x _{t, d} : d-th bit of binary feature vector x _t
D: Bit length of binary feature vector
p _i (x _t | λ): Binary feature vector x _t is generated from the i-th multivariate Bernoulli distribution
The probability that the dth bit will be 1

Specifically, these parameters are estimated from a set of binary feature vectors x _{1 to} x _T of the training content by an iterative process of an EM (Expectation-Maximization) algorithm. The EM algorithm is an estimation method for statistical parameters based on the maximum likelihood method, and is used when the probability model depends on a latent variable that cannot be observed.

γt(i)：t番目の訓練ベクトルがi番目の多変量ベルヌーイ分布から生成された確率 In the E (Expectation, expected value) step, the expected value γ _t (i) of the model likelihood is estimated for each binary feature vector x _i based on the distribution of the latent variable z _ti .

γt (i): Probability that the t-th training vector was generated from the i-th multivariate Bernoulli distribution

In the M (Maximization) step, the mixture ratio w _i and the parameter μ _i are updated in order to maximize the expected value γ _t (i) of the likelihood calculated in the E step. The parameter calculated in the M step is used to determine the distribution of latent variables in the next E step.

By repeating these E step and M step until convergence, the parameter group λ of the mixture ratio w _i and the parameter μ _i that maximizes the log likelihood is calculated λ (w ₁ ,..., W _N and μ ₁₁・ ・ ・ ・ ・ ・ μ _ND ）

<Calculation of Fisher information amount f _id >
Further, the model estimation unit 12 calculates “Fischer information amount f _id ” related to the parameter μ _id of the multivariate mixed Bernoulli distribution. The Fisher kernel is a generative approach.
approach) and a discriminative approach (see Non-Patent Document 8, for example). In the Fisher kernel, first, a gradient vector derived from a probability density distribution that generates a local descriptor is calculated, and this gradient vector normalized by a Fisher information matrix is used as one feature vector that represents an image. . Assuming that the Fisher information matrix is a diagonal matrix, normalization is equivalent to normalizing the gradient for each parameter with the amount of Fisher information. According to the Fisher kernel, a feature vector having a larger number of elements can be obtained even with a codebook of the same size as compared with Bag of Features. That is, since there is a lot of information expressed by feature vectors, there is no need to project to a high-dimensional space using a kernel method with high calculation cost, and sufficient performance can be obtained even with linear identification.

(First embodiment in which the Fisher information matrix is a diagonal matrix)
The model estimation unit 12
(S11) A Fisher score s _{id in} which a logarithmic likelihood function of the parameter μ _id is defined by partial differentiation is calculated,
(S12) The Fisher information amount f _id is calculated as the variance of the Fisher score s _id .

The Fisher information amount is defined by the second moment of the Fisher score. Fisher's score for μ _id is a log-likelihood function L (λ | X) = log P (when a binary feature vector set X = {x ₁ ,..., x _T } is observed for a multivariate mixed Bernoulli distribution. X | λ) is defined as a partial derivative with respect to μ _id .

Fisher relates mu _id score s _id is defined by the following equation.

Further, when the above-described γt (i) is used, the following equation is obtained.

Fisher information f _id relates mu _id is defined by the following equation.

とによって、フィッシャー情報量ｆ_idは、以下のようになる。

Conventionally (Non-Patent Document 3), the Fisher information amount is approximately calculated from the parameter λ. In the present invention, as shown below, the Fisher information amount is directly calculated from the sample, thereby calculating an accurate Fisher information amount that is not approximate and ensuring accuracy.
Equation 7 is the independence of x _t

Thus, the Fisher information amount f _id is as follows.

(Second embodiment using principal component analysis)
The model estimation unit 12
(S21) Calculate a Fisher score s _id defined by partial differentiation with respect to the parameter μ _id of the log likelihood function,
(S22) A principal component analysis is performed on the Fisher score s _id ,
(S23) As a result of the principal component analysis, K values having large eigenvalues are output as normalization parameters f _id ,
(S24) Further output K eigenvectors g _iK corresponding to the eigenvalues.

  Particularly in the case of an image, there is a correlation between the bits of the binary feature vector. For this reason, the assumption of the diagonal matrix as in the first embodiment is not necessarily established. Therefore, according to the second embodiment, decorrelation and normalization are performed using principal component analysis as follows. Principal component analysis refers to transforming an original observed value having a correlation between variables into a value called an uncorrelated principal component using orthogonal rotation.

For the i-th multivariate Bernoulli distribution multivariate mixed Bernoulli distribution, calculated Fisher scores s _i1, · · ·, a s _iD, binary feature vector set x _1, · · ·, relative to x _T, principal component analysis Execute. K from the largest eigenvalues of the result of the principal component analysis are set as f _i1 ,..., F _iK , the corresponding eigenvectors are set as g _{i1 ,.} Output.

[Model parameter storage unit 13]
The model parameter accumulating unit 13 uses the mixture ratio w _i (i = 1 to N) parameter μ _id (i = 1 to N, d = 1 to D) and the Fisher information amount as model parameters output from the model estimating unit 12. f _id (i = 1 to N, d = 1 to D) is stored. Further, according to the second embodiment, the model parameter accumulating unit 13 further accumulates eigenvectors g _ik (i = 1 to N, k = 1 to K).

[Feature vector conversion unit 14]
The feature vector conversion unit 14 uses the mixture ratio w _i , the parameter μ _id, and the Fisher information amount f _{id of} the model parameter storage unit 13 from the set of binary feature vectors x _{1 to} x _T of the reference content or the query content, One Fisher vector v corresponding to the reference content or query content is calculated.

(For the first embodiment of the model estimation unit 12)
The feature vector conversion unit 14
(S13) A Fisher score s _id (s _{11 to} s _ND ) is calculated for each set of binary feature vectors x _{1 to} x _T using the parameters w _i and μ _id and accumulated for each id. Calculate s' _id (s' ₁₁ ~ s' _ND )
(S14) A Fisher vector v _id is calculated by dividing each accumulated Fisher score s ′ _id by the square root √f _id of the corresponding Fisher information amount f _id .
v _id = s _id / √f _id
f ₁₁ ,..., f _ND (N × D): Fisher information amount

(For the second embodiment of the model estimation unit 12)
The feature vector conversion unit 14
(S25) For each set of binary feature vectors, the Fisher score s _id (s _{11 to} s _ND ) is calculated using the parameters w _i and μ _id , and the accumulated Fisher score s ′ _id (s _'11 ~s' _ND) is calculated,
(S26) For each mixed element i, calculate a normalized vector v ′ _id normalized (projected) using each cumulative Fisher score s ′ _id (s ′ _{i1 to} s ′ _iD ) and the corresponding eigenvector g _iK ,
(S27) the normalized vector v _'id, to calculate a Fischer vector v _id divided by the square root √G _id of the corresponding eigenvectors g _id.
v _id = v ' _id / √g _id

  The feature vector conversion unit 14 outputs the Fisher vector converted for the reference content to the reference information storage unit 15, and the Fisher vector converted for the query content is output to the feature vector conversion unit 16.

[Feature vector search unit 16]
The feature vector search unit 16 uses the reference information storage unit 15 to search for the Fisher vector v _R of the reference content that is most similar to the Fisher vector v _Q of the query content, as in FIG. Here, it is possible to use the Euclidean distance, v as the distance between _Q and v _R is shorter, the higher the degree of similarity of the reference content to the query content. Specifically, it is also preferable to use a method (for example, refer to Non-Patent Document 6) that uses direct product quantization, which is one of the Approximate Nearest Neighbor algorithms, or LSH (Locality-Sensitive Hashing).

  As described above in detail, according to the search device, program, and method of the present invention, content can be searched at a higher speed than SIFT or SURF.

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Search apparatus 11 Feature vector extraction part 12 Model estimation part 13 Model parameter storage part 14 Feature vector conversion part 15 Reference information storage part 16 Feature vector search part

Claims (9)

A search program that causes a computer mounted on a device to function to search for reference content similar to query content from a set of reference content using model parameters extracted from training content,
For each of the training content, the reference content, and the query content, a feature vector extraction unit that extracts a set of D-dimensional binary feature vectors x _{1 to} x _T ,
From the set of binary feature vectors of the training content, the mixture ratio w _i for the i (1 ≦ i ≦ N) -th multivariate Bernoulli distribution and the d (1 ≦ d ≦ D) -th parameter of the i-th multivariate Bernoulli distribution model estimation means for calculating μ _id and the Fisher information amount f _id related to the parameter μ _id ;
Model parameter accumulating means for accumulating the mixing ratio w _i , parameter μ _id, and Fisher information amount f _id ;
1 corresponding to the reference content or query content from the set of binary feature vectors of the reference content or query content using the mixture ratio w _i , parameter μ _id, and Fisher information amount f _id stored in the model parameter storage means. Feature vector conversion means for calculating two Fisher vectors;
A search program that causes a computer to function as a feature vector search unit that searches for a Fisher vector of a reference content that is most similar to a Fisher vector of a query content.   The said feature vector extraction means makes a computer function so that a set of binary feature vectors may be extracted using ORB (Oriented FAST and Rotated BRIEF) or FRAK (Fast Retina Keypoint). Search program. The model estimation means includes a set of binary feature vectors x _{1 to} x _{T of} training content,
For the E (Expectation) step, estimate the expected value γ _t (i) of the latent variable i for each binary feature vector x _i ,
For the M (Maximization) step, the mixture ratio w _i and the parameter μ _i are updated using the expected value γ _t (i),
By repeating these E step and M step until convergence, a parameter group λ of the mixture ratio w _i and parameter μ _i is calculated. Λ (w ₁ ,..., W _N and μ ₁₁ ,. _ND )
The search program according to claim 1 or 2, wherein the computer functions as described above. The model estimation means includes
Calculating a Fisher score s _{id in} which a logarithmic likelihood function of the parameter μ _id is defined by partial differentiation;
The search program according to any one of claims 1 to 3, wherein the computer is caused to calculate a Fisher information amount f _id as a variance of the Fisher score s _id . The feature vector conversion means calculates a Fisher score s _id using the parameter μ _id for each set of binary feature vectors, calculates a cumulative Fisher score s ′ _id obtained by accumulating these for each _id ,
5. The search program according to claim 4, wherein the computer is caused to calculate a Fisher vector v _id obtained by dividing each cumulative Fisher score s ′ _id by a square root √f _id of a corresponding Fisher information amount f _id . . The model estimation means includes
Calculating a Fisher score s _{id in} which a logarithmic likelihood function of the parameter μ _id is defined by partial differentiation;
A principal component analysis is performed on the vector (si1 to siD) for each mixed element i of the Fisher score s _id ,
As a result of the principal component analysis, K pieces having large eigenvalues are output as the Fisher information amount f _id ,
Further output K eigenvectors g _iK corresponding to the eigenvalues,
The search program according to any one of claims 1 to 3, wherein the model parameter storage means causes a computer to further store the eigenvector g _iK . The feature vector conversion means calculates a Fisher score s _id using the parameter μ _id for each set of binary feature vectors, calculates a cumulative Fisher score s ′ _id obtained by accumulating these for each _id ,
A normalized vector v obtained by normalizing (projecting) a vector (s ′ _{i1 to} s _iD ) of each cumulative Fisher score s ′ _id using a corresponding eigenvector g _ik (g _{i1 to} g _iK ) for each mixed element i. 'Calculate _ik ,
The search program according to claim 6, wherein the computer functions to calculate a Fisher vector v _ik obtained by dividing the normalized vector v ′ _ik by the square root √g _ik of the corresponding eigenvector g _ik . A search device for searching reference content similar to query content from a set of reference content using model parameters extracted from training content,
For each of the training content, the reference content, and the query content, a feature vector extraction unit that extracts a set of D-dimensional binary feature vectors x _{1 to} x _T ,
From the set of binary feature vectors of the training content, the mixture ratio w _i for the i (1 ≦ i ≦ N) -th multivariate Bernoulli distribution and the d (1 ≦ d ≦ D) -th parameter of the i-th multivariate Bernoulli distribution model estimation means for calculating μ _id and the Fisher information amount f _id related to the parameter μ _id ;
Model parameter accumulating means for accumulating the mixing ratio w _i , parameter μ _id, and Fisher information amount f _id ;
From a set of binary feature vectors of reference content or query content, one Fisher vector corresponding to the reference content or query content using the mixture ratio w _i , parameter μ _id and Fisher information amount f _{id of the} model parameter storage means. Feature vector conversion means for calculating
And a feature vector search unit that searches for a Fisher vector of reference content that is most similar to a Fisher vector of query content. A search method for searching reference content similar to query content from a set of reference content using a model parameter extracted from training content using an apparatus,
As a first step of accumulating model parameters,
For each training content, a set of D-dimensional binary feature vectors x _{1 to} x _T is extracted,
From the set of binary feature vectors of the training content, the mixture ratio w _i for the i (1 ≦ i ≦ N) -th multivariate Bernoulli distribution and the d (1 ≦ d ≦ D) -th parameter of the i-th multivariate Bernoulli distribution Calculate μ _id and Fisher information amount f _id related to parameter μ _id ,
Accumulate the mixing ratio w _i , the parameter μ _id and the Fisher information amount f _id ,
As a second step of accumulating reference information,
For each reference content, extract a set of D-dimensional binary feature vectors,
From the set of binary feature vectors of each reference content, one Fisher vector is calculated using the mixture ratio w _i , the parameter μ _id, and the Fisher information amount f _id accumulated as model parameters.
Accumulates the Fisher vector,
As a third step of searching reference content from query content,
From the set of binary feature vectors of each query content, one Fisher vector is calculated using the mixture ratio w _i , the parameter μ _id and the Fisher information amount f _id accumulated as model parameters,
A search method comprising searching for a Fisher vector of reference content that is most similar to a Fisher vector of query content.

Images