JP2015056077A - Image retrieval device, system, program, and method using image based binary feature vector - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image retrieval device, program, and method capable of minimizing the capacity of a reference information database and also reducing the amount of arithmetic operation performed during image retrieval.SOLUTION: The image retrieval device comprises: hash codebook storage means for storing a hash codebook for each hash value; bit number codebook storage means for storing a high-entropy bit number set for each hash value; local feature extraction means for extracting a binary feature vector set from an image; hash means for outputting a hash value most resembling the binary feature vector by using the hash codebook storage means; bit string generation means for generating a new bit string from a hash value bit number set by using the bit number codebook storage means; transposition index storage means for storing a reference image bit string for each hash value in correlation with each other; and retrieval means for retrieving a reference image from a query image bit string.

Description

  The present invention relates to a technique for retrieving an image using a binary feature vector.

  In recent years, image recognition and search techniques based on local feature points have attracted attention. As a feature vector extraction algorithm used for object recognition, for example, SIFT (Scale-Invariant Feature Transform) (see, for example, Non-Patent Document 1) or SURF (Speeded Up Robust Features), which is robust to changes in rotation and scale, is used. For example, in the case of SIFT, a set of 128-dimensional feature vectors is extracted from one image. SIFT is a technique for analyzing a characteristic local region using a scale space and describing a feature vector that is invariant to scale change and rotation. On the other hand, in the case of SURF, higher-speed processing is possible than SIFT, and a set of 64-dimensional feature vectors is extracted from one image. While SIFT has a high processing cost and is difficult to perform real-time matching, SURF uses an integral image to speed up the processing.

  On the other hand, while portable terminals such as smartphones and tablet terminals are widespread, further memory saving and faster matching have been required for content search processing. In particular, according to the technical field of image recognition in the use of augmented reality (Augmented Reality), it is required to search for content at higher speed than SIFT or SURF in order to perform real-time processing. For this reason, FAST (Features from Accelerated Segment Test) (for example, see Non-Patent Document 2) and FRAK (Fast Retina Keypoint) (for example, see Non-Patent Document 3), which are binary feature vector extraction algorithms, are attracting attention. This makes it possible to extract feature vectors at a higher speed than SIFT and SURF, and to further reduce the extracted feature vectors.

JP 2010-282581 A

J. Sivic et al., "Video Google: A Text Retrieval Approach to Object Matching in Videos," in Proc. ICCV, 2003. E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, "ORB: Anefficient alternative to SIFT or SURF," in Proc. ICCV, 2011. A. Alahi, R. Ortiz, and P. Vandergheynst, "FREAK: Fast RetinaKeypoint," in Proc. CVPR, 2012. “Correlation coefficient”, [online], [searched August 31, 2013], Internet <URL: http://en.wikipedia.org/wiki/%E7%9B%B8%E9%96%A2% E4% BF% 82% E6% 95% B0>

  According to a portable terminal such as a smartphone, there is an application that can immediately search for a reference image similar to a captured query image. On the other hand, according to the image search (recognition) technique, a large amount of reference information to be recognized needs to be stored in advance as a database. Therefore, the image search application itself installed in the smartphone needs to have a reference information database.

  However, the larger the number of reference images that are recognized, the larger the database becomes, and as a result, the size of the application itself increases. On the other hand, the user tends to feel resistance to installing a large-capacity application on a smartphone having a relatively small memory capacity. In addition, as the capacity of the database increases, the amount of computation processing (and computation time) required for retrieval by the image retrieval application increases.

  Of course, the image search application installed in the smartphone can also download only the database file sequentially via the network. However, the larger the capacity of the database file, the longer it takes to download it, which may result in a loss of usability.

  Accordingly, an object of the present invention is to provide an image search apparatus, program, and method that can reduce the reference information database as much as possible and reduce the amount of calculation processing during image search.

According to the present invention, an image search device for searching a reference image similar to a query image from a number of reference images,
Hash codebook storage means for storing a hash codebook used for quantization of a binary feature vector for each hash value;
Bit number codebook storage means for storing a set of bit numbers having a high amount of information (entropy) for each hash value;
Local feature extraction means for extracting a set of binary feature vectors from a query image and a reference image;
Hashing means for outputting a hash value most similar to a binary feature vector using a hash codebook storage means;
A bit string generation unit that selects a partial bit string from a binary feature vector with reference to a bit number set of the hash value in the bit number codebook storage unit;
For each hash value, transposed index storage means for storing a bit string of a reference image in association with each other,
Search means for searching for a reference image from a bit string of a query image using transposed index storage means.

According to another embodiment of the image search device of the present invention,
It may further comprise a hash codebook generating means for clustering a large number of binary feature vectors in a large number of reference images and storing the representative binary feature vector for each cluster and a hash value as an identifier thereof in a hash codebook storage means. preferable.

According to another embodiment of the image search device of the present invention,
It is also preferable that the hash codebook generating means uses a k-means method or a k-medoids method for clustering.

According to another embodiment of the image search device of the present invention,
For a number of binary feature vectors included in one cluster of the same hash value, a bit number sequence in which bit numbers with a high amount of information of the bit and a small correlation between the bit and other bits are arranged. It is also preferable to further include a bit number code book generating means for storing the bit number string for each hash value in the bit number code book storage means.

According to another embodiment of the image search device of the present invention,
The bit number code book generating means stores the bit number one by one in the bit number string.
For a large number of binary feature vectors included in one cluster of the same hash value, a temporary bit number sequence in which the average value of each bit (0/1) is arranged in order from the bit close to 0.5 (the bit with the highest amount of information) Generate and
If the absolute value of the correlation coefficient with other bits already accommodated in the bit number sequence is less than or equal to a predetermined threshold, the bit number of the upper bit is accommodated in the bit number sequence in order from the upper bit of the temporary bit number sequence It is also preferable to accommodate this until the bit number string reaches a predetermined number.

According to another embodiment of the image search device of the present invention,
Preferably, the transposed index storage means registers a transposed index in which a plurality of sets of reference image identifiers and bit strings are associated in a list format for each hash value LIDn for the reference image.

According to another embodiment of the image search device of the present invention,
Search means are
Enter multiple sets of hash values LIDn and the query image,
For each set of hash value and bit string of query image, obtain a list of transposed indexes corresponding to the hash value,
Calculate the dissimilarity between the query image bit string and the reference image bit string included in the acquired list,
Calculate the score value that increases as the dissimilarity is lower,
The score value is cumulatively added for each reference image,
It is also preferable to output a reference image having the highest score value as a search result.

According to another embodiment of the image search device of the present invention,
It is also preferable that the search means cumulatively add only the score value corresponding to the upper predetermined number (K) having a low dissimilarity to the reference image.

According to another embodiment of the image search device of the present invention,
For the search means, the upper predetermined number (K) is a preset fixed number or the number of dissimilarities whose dissimilarity is equal to or less than the threshold TH based on a preset threshold TH related to dissimilarity It is also preferable.

According to another embodiment of the image search device of the present invention,
In the search means, for the addition score corresponding to the i-th item in the top predetermined number (K), if the K-th dissimilarity is larger than the i-th dissimilarity, the value of the addition score is large It is also preferable that it is set.

According to another embodiment of the image search device of the present invention,
Regarding the search means, the addition score corresponding to the i-th item in the top predetermined number (K) is
(1) Whether the square of the i-th dissimilarity is subtracted from the square of the K-th dissimilarity,
(2) Whether 1 is subtracted from the ratio of the square of the Kth dissimilarity and the square of the ith dissimilarity,
(3) Subtract 1 from the square of the ratio of the Kth dissimilarity and the ith dissimilarity, or
(4) It is also preferable to set the square of the ratio of the Kth dissimilarity and the ith dissimilarity minus one.

According to another embodiment of the image search device of the present invention,
The local feature extraction means preferably extracts a binary feature vector based on ORB (Oriented FAST and Rotated BRIEF), FREK (Fast Retina Keypoint), or BRISK (Binary Robust Independent Elementary Features).

According to the present invention, there is provided a program for causing a computer mounted on an apparatus for searching a reference image similar to a query image to function from among a large number of reference images,
Hash codebook storage means for storing a hash codebook used for quantization of a binary feature vector for each hash value;
Bit number codebook storage means for storing a set of bit numbers having a high amount of information (entropy) for each hash value;
Local feature extraction means for extracting a set of binary feature vectors from a query image and a reference image;
Hashing means for outputting a hash value most similar to a binary feature vector using a hash codebook storage means;
A bit string generation unit that selects a partial bit string from a binary feature vector with reference to a bit number set of the hash value in the bit number codebook storage unit;
For each hash value, transposed index storage means for storing a bit string of a reference image in association with each other,
Using the transposed index storage means, the computer functions as search means for searching for a reference image from a bit string of a query image.

According to the present invention, a method for searching a reference image similar to a query image from among a large number of reference images using an apparatus,
The device is
For each hash value, a hash codebook storage unit that stores a hash codebook used for quantization of binary feature vectors;
A bit number codebook storage unit for storing a set of bit numbers having a high amount of information (entropy) for each hash value;
For each hash value, it has a transposed index storage unit that stores a bit string of a reference image in association with each other,
A first step of extracting a set of binary feature vectors from a query image and a reference image;
A second step of outputting a hash value most similar to the binary feature vector using the hash codebook storage unit;
A third step of selecting a partial bit string from a binary feature vector with reference to a set of bit numbers of the hash value in the bit number codebook storage unit;
And a fourth step of searching the reference image from the bit string of the query image using the transposed index storage unit.

  According to the image search apparatus, program, and method of the present invention, the reference information database can be made as small as possible, and the amount of calculation processing during image search can be reduced.

It is a functional block diagram of the image search device in the present invention. It is a function block diagram of the search information delivery server in this invention. It is a function block diagram of the terminal which can communicate with the search information delivery server of FIG. It is explanatory drawing showing the image search process in this invention. It is explanatory drawing showing the production | generation of the bit string in this invention. It is explanatory drawing showing the data structure of a transposed index registration part. It is explanatory drawing showing the process in a search part. It is a flowchart showing the process of the search part in this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a functional configuration diagram of an image search apparatus according to the present invention.
FIG. 2 is a functional configuration diagram of the search information distribution server in the present invention.
FIG. 3 is a functional configuration diagram of a terminal capable of communicating with the search information distribution server of FIG.

The image search apparatus 1 inputs a large amount of reference images (search target images) in advance, and generates and accumulates a database (codebook or index). Therefore, it can be divided into two embodiments as follows.
(1) Image Retrieval Device (Single Unit): As shown in FIG. 1, the image retrieval device 1 inputs a large number of reference images from a user, and generates and accumulates a database for the reference images. Then, the image search device 1 inputs a query image (search key image) from the user, searches a reference image for the query image using a database, and outputs a search result of the reference image to the user.

(2) Server-client system: As shown in FIG. 2, the search information distribution server 2 inputs a large amount of reference images, and generates and stores a database for the reference images.
The terminal 1 as a client downloads the database information from the search information distribution server 2 via the network. The database information is stored in a storage area (memory space or disk space) of the terminal 1.
Then, as illustrated in FIG. 3, the terminal 1 inputs a query image from the user, searches a reference image for the query image using a database, and outputs a search result of the reference image to the user.

  Although FIG. 1 differs from FIG. 2 and FIG. 3 in system configuration, each functional component is completely the same. The following description will be made with reference to FIG.

  According to FIG. 1, the image search apparatus 1 includes a hash codebook storage unit 101, a bit number codebook storage unit 102, a transposed index storage unit 103, a local feature extraction unit 11, a hash unit 12, and a bit string generation. Section 13, search section 14, hash codebook generation section 15, and bit number codebook generation section 16. These functional components are realized by executing an image search program that causes a computer installed in the apparatus to function. For example, when searching for an image with a smartphone, the image search device 1 may be an installable image search application. Further, the processing flow of these functional components can be understood as an image search method using the apparatus.

  FIG. 4 is an explanatory diagram showing image search processing in the present invention. Hereinafter, description will be made with reference to FIG. 4 as appropriate.

[Local feature extraction unit 11]
The local feature extraction unit 11 extracts a set of binary feature vectors of local features from the query image and the reference image. According to the present invention, ORB (Oriented FAST and Rotated BRIEF) or FRAK (Fast Retina Keypoint) is used as the binary feature vector extraction algorithm. In the case of ORB, a set of 256-bit binary feature vectors is extracted from one content. For example, in order to execute matching at high speed, there is BRIEF (Binary Robust Independent Elementary Features) as a feature description by a binary code. According to the present invention, “ORB” is used which can describe a feature in which rotation invariance is introduced into BRIEF. In particular, according to the ORB, it is possible to maintain an accuracy equal to or higher than that of SIFT or SURF and realize a speed increase of several hundred times.

  The ORB is composed of two steps of “feature point detection processing” and “feature vector description processing”.

(Feature point detection processing)
According to the ORB feature point detection process, FAST is used to detect key points at high speed. In addition, since FAST is not robust to scale changes, an image is converted into a plurality of sizes, and feature points are extracted from images of each size.
Further, the existing FAST does not have an algorithm for calculating the key point orientation for obtaining rotation invariance. Therefore, ORB adopts Oriented FAST in order to obtain rotational invariance. By describing the features based on the orientation, even if the input image is rotated, the same key point can be detected as the same feature amount. Therefore, the direction vector of the center of the key point and the center of gravity of the brightness of the patch is used.

(Feature vector description processing)
Next, according to the feature vector description processing in the ORB, a binary feature vector is extracted for each detected feature point using the BRIEF feature vector descriptor. These are obtained from the magnitude relationship of the luminance of two pixels around the feature point.
BRIEF can execute keypoint feature description by binary code. According to SIFT and SURF, high-dimensional real numbers are used for feature description. However, when a high-dimensional real number is used, there is an increase in memory capacity and similarity calculation. Therefore, by using BREF based on ORB, it is possible to save memory by describing features by binary code, and it is possible to reduce processing costs by using a Hamming distance for similarity calculation.
According to BRIEF, a binary code is generated from the sign of the luminance difference between two points randomly selected in the patch. The pixels to be selected are randomly selected according to a Gaussian distribution centered on the key point position. Here, the ORB selects pixels using learning in order to perform matching with higher accuracy. The selected pixel position is used for feature description as a binary code with high feature description capability when the bit variance of the pair is large and the correlation of N pairs is low. N pairs are narrowed down using the Greedy algorithm.

[Hash codebook storage unit 101]
The hash codebook storage unit 101 stores a hash codebook used for quantization of the binary feature vector for each hash value s. According to FIG. 4, the code book stores code vectors B _{1 to} B _S having the same bit length as the binary feature vector B. In the case of an ORB binary feature vector, it is, for example, 256 bits long.

[Hash codebook generator 15]
The hash codebook generation unit 15 clusters a large number of binary feature vectors in a large number of reference images. For clustering, the k-medoids method using the Hamming distance may be used, or the k-means method may be simply used. The hash codebook generation unit 15 uses a representative binary feature vector for each cluster as a code vector, associates a hash value as an identifier thereof, and stores the hash value in the hash codebook storage unit 101.

[Hash part 12]
Using the hash codebook storage unit 101, the hash unit 12 outputs the hash value LID of the code vector B _s most similar to the binary feature vector B output from the local feature extraction unit 11. If the binary feature vectors are similar, the hash unit 12 can return the same hash value with a high probability. Therefore, speeding up can be realized by matching only binary feature vectors having the same hash value.

“The most similar” may be, for example, the one having the shortest Hamming distance.
LID = argmin _s ham (B, B _s )
ham (x, y): Hamming distance between x and y The hash unit 12 uses the hash value LID _n (= s) of the n (n = 1 to N) -th binary feature vector of the query image as the bit string generation unit 13. Output to.

  The database search time is increased by matching only the most similar feature vector in consideration of the fact that much time is required for feature vector matching.

[Bit Number Codebook Storage Unit 102]
The bit number codebook storage unit 102 stores a set of bit numbers having a high information amount (entropy) for each hash value s. The number of bit number sets is 64, for example. According to FIG. 4, for each hash value, bit numbers representing the bit positions of the binary feature vectors are registered in descending order of information amount. Here, it is sufficient that a set of bit numbers having a high information amount is registered, and it is not always necessary that the information amount is in descending order.

[Bit string generation unit 13]
The bit string generation unit 13 refers to the bit number set C _s of the hash value s in the bit number codebook storage unit 102 and selects a partial bit string (for example, 64 bits) from the binary feature vector B. In this bit string, only the bit at the position of the bit number described in the bit number set C _s is extracted from the binary feature vector B and arranged. As a result, a bit string shorter than the input binary feature vector is generated. This means that only a bit string that contributes to discrimination between binary feature vectors is generated and the other bits are discarded. By storing only the bit string as a code book in the database, the capacity of the database itself can be reduced, and the amount of calculation processing at the time of image search can be reduced.

From the binary feature vector B, a set of a hash value LID and a bit string RC is obtained.
(LID, RC)
LID: log ₂ S bits (S is the number of codebooks in the hash codebook storage unit)
RC: L·log ₂ F bit (F is the number of codebooks in the bit number codebook storage unit)

Then, the bit string generation unit 13 outputs in the following two directions.
(1) For the binary feature vector of the query image, a plurality of sets of bit strings RC are output to the search unit 14 for each hash value LIDn.
(2) For the binary feature vector of the reference image, a plurality of sets of bit strings RC are output to the transposed index registration unit 103 for each hash value LIDn.
Note that n in LIDn represents the n (n = 1 to N) th binary feature vector of the input image.

[Bit Number Codebook Generation Unit 16]
The bit number code book generating unit 16 generates a bit number string for each hash value, and stores the bit number string in the bit number code book storage unit 102. Since the distribution of bit strings differs for each hash value, a code book of bit number strings suitable for those distributions is created. The “bit number sequence” is a bit number that has a high information amount of the bit and a small correlation between the bit and other bits for a number of binary feature vectors included in one cluster of the same hash value. Are arranged.

  FIG. 5 is an explanatory diagram showing generation of a bit string in the present invention.

For each hash value, a bit number string is generated by the following steps.
(S51) A correlation coefficient between bits is calculated for all the bits of the binary feature vector. For example, in the case of ORB, a 256-bit round-robin correlation coefficient is calculated as shown in FIG. 5 (see, for example, Non-Patent Document 4). The correlation coefficient takes a value of −1 to 1. In the following, for example, the correlation coefficient between the i-th bit and the j-th bit is calculated using eight vectors.
(0, 0, 0, 0, 1, 1, 1, 1)
↑ i-th bit of vector 1
↑ i-th bit of vector 2
↑ Bit i of vector 3
...
↑ i-th bit of vector 8 (1, 1, 1, 1, 0, 0, 0, 0)
↑ jth bit of vector 1
↑ jth bit of vector 2
↑ jth bit of vector 3
...
↑ jth bit of vector 8 In this case, the correlation coefficient is calculated as follows.
(0, 0, 0, 0, 1, 1, 1, 1) <-> (1, 1, 1, 1, 0, 0, 0, 0) Correlation coefficient = -1
Similarly, according to another example, the correlation coefficient is calculated as follows.
(0, 0, 0, 0, 1, 1, 1, 1) <-> (0, 0, 0, 1, 1, 1, 1, 0) Correlation coefficient = 0.5
(0, 0, 0, 0, 1, 1, 1, 1) <-> (0, 0, 1, 1, 1, 1, 0, 0) Correlation coefficient = 0

(S52) The average value for each bit number is calculated for a number of binary feature vectors included in one cluster of the same hash value. Here, the following relationship is assumed from the average value.
The closer the average value is to 0, the higher the probability that both bits are “0”.
The closer the average value is to 0.5, the higher the probability that both bits are inverted.
(When one bit is “1”, the other bit is likely to be “0”)
The closer the average value is to 1, the higher the probability that both bits are “1”.

(S53) A “provisional bit number sequence” is generated in which the average value of each bit (0/1) is arranged in order from the bits close to 0.5 (bits with a large amount of information).

(S54) If the absolute value of the correlation coefficient with other bits already accommodated in the bit number sequence is not more than a predetermined threshold in order from the upper bit of the temporary bit number sequence, the bit number of the upper bit is changed to the bit number. One bit is stored in the column. This means that bits having high correlation between bits are not selected. This is accommodated until the bit number string reaches a predetermined number (for example, 64 bits).

[Transposed index storage unit 103]
The transposed index storage unit 103 registers a set (RID, RC _n ) of an image identifier RID and a bit string RC as a list for each hash value LIDn.

  FIG. 6 is an explanatory diagram showing the data structure of the inverted index registration unit.

A plurality of sets of hash values LIDn and bit strings RCn are associated with the reference image RID.
RID-> (LID ₁ , RC ₁ ) (LID ₂ , RC ₂ ) ... (LID _n , RC _n ) ... (LID _N , RC _N )
For each pair (LID, RC), the pair (RID, RC) of the image identifier RID and the bit string RC is registered by being linked to the hash value LID in the transposed index.
LID _1- > (RID, RC) (RID, RC) (RID, RC) ...
LID _2- > (RID, RC) (RID, RC) (RID, RC) ...
...
LID _n- > (RID, RC) (RID, RC) (RID, RC) ...
...

[Search unit 14]
The search unit 14 uses the transposed index storage unit 103 to search for a reference image from the bit string of the query image. The input query image is composed of a plurality of sets of hash values LID _n and bit strings QC _n .
Query code-> (LID ₁ , QC ₁ ) (LID ₂ , QC ₂ ) ... (LID _n , QC _n ) ... (LID _N , QC _N )

FIG. 7 is an explanatory diagram illustrating processing in the search unit.
FIG. 8 is a flowchart showing the processing of the search unit in the present invention.

The search unit 14 executes the following processing steps for each reference image (S4), and finally outputs the reference image having the highest score value as a search result (S5).
score [] = 0 (S0)
for each i = 1 to N
Get LID _i- th list of inverted index (S1)
About pairs (RID ₁ , RC ₁ ) to (RID _M , RC _M ) in the list
The distance D _ij of the pair (QC _i , RC _j ) based on the query image is calculated,
A pair of distance and image identifier (D _ij , RID _j ) is created (S2)
Sort D _ij in ascending order (S3)
Select the top K-th set D _{i'j '}
for each k = 1 ~ K
For the k-th D _{i′j ′} , score [RC _{j ′} ] + = S (D _{i′j ′} , D)
end for
end for

(S80) First, as an initial setting, a variable score [] = 0 is set.

The search unit 14 repeatedly executes the following processes S1 to S3 for each set of hash values and bit strings (LID _i , QC _i ) based on the input query image (i = 1 to N).

(S81) For each hash value and bit string pair (LID _i , QC _i ) of the query image, a transposed index list (RID _j , RC _j )... Corresponding to the hash value LID _i is acquired.

(S82) The following processing is repeatedly executed for the list arranged in the hash value LID _i of the inverted index (j = 1 to M).
A distance D _ij between the bit string QC _i of the query image and each bit string RC _j of the reference image in the acquired list is calculated. Then, a set (D _ij , RID _j ) of the distance D _ij and the image identifier RID _j is created.

(S83) Next, the sets (D _ij , RID _j ) of the distance D _ij and the image identifier RID _j are sorted in ascending order of distance (ascending order).

  Then, only the upper predetermined number (K) having a short distance is selected. A short distance means a high degree of similarity. The upper predetermined number (K) may be a fixed number set in advance. Alternatively, the distance may be the number of distances where the distance is equal to or less than the threshold TH based on a preset threshold TH related to the distance.

Specifically, for a plurality of sets (D _ij , RID _j ) of the top K, a score value that increases as the distance is shorter is calculated, and the score values are cumulatively added. Specifically, using the kth distance D _{i′j ′} and the Kth distance D, the score value is calculated by either of the following.
S (D _{i′j ′} , D): Vote score value for image having k-th distance (1) The square of i-th distance is subtracted from the square of k-th distance S (D _{i 'j'} , D) = D ² -D _{i'j '} ²
(2) It is assumed that 1 is subtracted from the ratio of the square of the kth distance and the square of the ith distance. S (D _{i′j ′} , D) = D ² / D _{i′j ′} ² −1
(3) It is assumed that 1 is subtracted from the square of the ratio of the kth distance and the ith distance. S (D _{i′j ′} , D) = (D / D _{i′j ′} ) ² −1
(4) S (D _{i′j ′} , D) = (D / D _{i′j ′} −1) ^{2, which is the} square of 1 minus the ratio of the kth distance and the ith distance

(S84) The score values are cumulatively added for each reference image. S1 to S3 are repeated for the next reference image.

(S85) Finally, the reference image with the highest score value is output as the search result.

  As described above in detail, according to the image search apparatus, system, program, and method of the present invention, the reference information database can be made as small as possible and the amount of calculation processing during image search can be reduced. it can.

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Image search device, terminal 101 Hash code book storage unit 102 Bit number code book storage unit 103 Transposed index storage unit 11 Local feature extraction unit 12 Hash unit 13 Bit string generation unit 14 Search unit 15 Hash code book generation unit 16 Bit number code book Generation unit 2 Search information distribution server

Claims (14)

An image search device for searching a reference image similar to a query image from a large number of reference images,
Hash codebook storage means for storing a hash codebook used for quantization of a binary feature vector for each hash value;
Bit number codebook storage means for storing a set of bit numbers having a high amount of information (entropy) for each hash value;
Local feature extraction means for extracting a set of binary feature vectors from the query image and the reference image;
Hashing means for outputting the most similar hash value to the binary feature vector using the hash codebook storage means;
A bit string generation means for selecting a partial bit string from the binary feature vector with reference to a bit number set of the hash value in the bit number codebook storage means;
For each hash value, transposed index storage means for storing a bit string of the reference image in association with each other,
An image search apparatus comprising: search means for searching for a reference image from a bit string of the query image using the transposed index storage means.   A hash codebook generating means for clustering a large number of binary feature vectors in a large number of reference images and storing a representative binary feature vector for each cluster and a hash value as an identifier thereof in the hash codebook storage means; The image search device according to claim 1.   The image search apparatus according to claim 2, wherein the hash codebook generating unit uses a k-means method or a k-medoids method for clustering.   For a number of binary feature vectors included in one cluster of the same hash value, a bit number sequence in which bit numbers with a high amount of information of the bit and a small correlation between the bit and other bits are arranged. 4. The image search apparatus according to claim 2, further comprising a bit number code book generating unit that stores the bit number string for each hash value in the bit number code book storage unit. The bit number code book generating means stores the bit number string one bit number at a time in the bit number string.
For a large number of binary feature vectors included in one cluster of the same hash value, a temporary bit number sequence in which the average value of each bit (0/1) is arranged in order from the bit close to 0.5 (the bit with the highest amount of information) Generate and
If the absolute value of the correlation coefficient between other bits already contained in the bit number sequence in order from the upper bit of the temporary bit number sequence is less than or equal to a predetermined threshold, the bit number of the upper bit is changed to the bit number sequence. The image search apparatus according to claim 4, wherein the image search apparatus stores the data until the bit number string reaches a predetermined number. The transposed index storage unit registers a transposed index in which a plurality of pairs of a reference image identifier and the bit string are associated in a list format for each hash value LIDn with respect to a reference image. Item 6. The image search device according to any one of Items 1 to 5. The search means includes
Enter multiple sets of hash values LIDn and the query image,
For each set of hash value and bit string of the query image, obtain a list of transposed indexes corresponding to the hash value,
Calculating the dissimilarity between the query image bit string and the bit string of the reference image included in the acquired list;
Calculate a score value that increases as the dissimilarity is lower,
The score value is cumulatively added for each reference image,
An image search apparatus that outputs a reference image having the highest score value as a search result.   The image search apparatus according to claim 7, wherein the search means cumulatively adds only a score value corresponding to the upper predetermined number (K) having a low dissimilarity to the reference image.   For the search means, the upper predetermined number (K) is a preset fixed number, or a dissimilarity in which the dissimilarity is equal to or less than a threshold TH based on a preset threshold TH related to dissimilarity The image search device according to claim 7, wherein In the search means, when the K-th dissimilarity is larger than the i-th dissimilarity for the i-th added score corresponding to the i-th among the upper predetermined number (K), the value of the added score The image search device according to claim 7, wherein a large value is set. For the search means, the addition score corresponding to the i-th among the upper predetermined number (K) is:
(1) Whether the square of the i-th dissimilarity is subtracted from the square of the K-th dissimilarity,
(2) Whether 1 is subtracted from the ratio of the square of the Kth dissimilarity and the square of the ith dissimilarity,
(3) Subtract 1 from the square of the ratio of the Kth dissimilarity and the ith dissimilarity, or
(4) The image search apparatus according to claim 7, wherein the ratio is obtained by subtracting 1 from the ratio of the Kth dissimilarity and the ith dissimilarity to the square. The local feature extraction means extracts a binary feature vector based on ORB (Oriented FAST and Rotated BRIEF), FREK (Fast Retina Keypoint), or BRISK (Binary Robust Independent Elementary Features).
The image search apparatus according to claim 1, wherein the image search apparatus is an image search apparatus. A program for causing a computer mounted on a device for searching a reference image similar to a query image to function from among a large number of reference images,
Hash codebook storage means for storing a hash codebook used for quantization of a binary feature vector for each hash value;
Bit number codebook storage means for storing a set of bit numbers having a high amount of information (entropy) for each hash value;
Local feature extraction means for extracting a set of binary feature vectors from the query image and the reference image;
Hashing means for outputting the most similar hash value to the binary feature vector using the hash codebook storage means;
A bit string generation means for selecting a partial bit string from the binary feature vector with reference to a bit number set of the hash value in the bit number codebook storage means;
For each hash value, transposed index storage means for storing a bit string of the reference image in association with each other,
A program that causes a computer to function as search means for searching a reference image from a bit string of the query image using the transposed index storage means. A method of searching for a reference image similar to a query image from a number of reference images using an apparatus,
The device is
For each hash value, a hash codebook storage unit that stores a hash codebook used for quantization of binary feature vectors;
A bit number codebook storage unit for storing a set of bit numbers having a high amount of information (entropy) for each hash value;
A transposed index storage unit that stores a bit string of the reference image in association with each hash value;
A first step of extracting a set of binary feature vectors from the query image and the reference image;
A second step of outputting a hash value most similar to the binary feature vector using the hash codebook storage unit;
A third step of selecting a partial bit string from the binary feature vector by referring to a bit number set of the hash value in the bit number codebook storage unit;
And a fourth step of searching for a reference image from a bit string of the query image using the transposed index storage unit.

Images