JP5995916B2 - Image search apparatus, method, and program - Google Patents

Description

  The present invention relates to an image search apparatus, method, and program, and more particularly, to an image search apparatus, method, and program for searching for a reference image similar to a query image.

  Conventionally, as a technique to quickly analyze whether the same subject appears in the image, the local feature extracted from the image is classified into clusters created in advance, and the local feature of the same cluster is more There is a technique called “bag of visual words” that determines that the more an image including more, the higher the possibility that the same subject is captured (for example, Non-Patent Document 1). In this method, instead of comparing the local feature values one by one, only the local feature values classified into the same cluster can be processed and counted, so that the similarity of images can be calculated at high speed. Similar images can be found.

  There is also a method for verifying the consistency of the rotation angle of local feature and the ratio of enlargement / reduction. For example, in the technique described in Non-Patent Document 2, the query image 1 and the reference image shown in FIG. 14 have three rotation angles of 135 degrees and an enlargement ratio of 0.5. Two points from the reference image have a rotation angle of 90 degrees and an enlargement ratio of 2, but the other point differs in both the rotation angle and the enlargement ratio. Therefore, it can be determined that the query image 1 having many points with consistent rotation angles and enlargement rates has a higher degree of similarity than the query image 2.

Sivic, Josef, and Andrew Zisserman, “Video Google: A text retrieval approach to object matching in videos.”, Computer Vision, 2003, Proceedings, Ninth IEEE International Conference on, IEEE, 2003. Jegou, Herve, Matthijs Douze, and Cordelia Schmid, “Improving bag-of-features for large scale image search.”, International Journal of Computer Vision 87.3, 2010, p.316-336.

  However, in the method of Non-Patent Document 1, since the geometric information of each local feature amount is lost, it is determined that the local feature amount extracted from a non-corresponding portion corresponds, and similar image extraction is performed. Accuracy will be reduced. For example, the three local feature amounts extracted from the triangle of the reference image shown in FIG. 14 are determined not only to match the query image 1 but also to the query image 2 that is a different graphic, and the similarity is high. It will be judged.

  In the method of Non-Patent Document 2, since only one axis is considered in the rotation direction, it cannot cope with rotation in a three-dimensional direction. For example, as shown in FIG. 15, a query image 1 obtained by rotating a reference image in a two-dimensional plane can be determined to match the reference image, but is an image viewed from the vertical direction or the horizontal direction in a three-dimensional space. For the query image 2, rotation about those axes is not acquired as information accompanying the local feature amount, so geometric consistency cannot be verified, and even if the same object is high, There was a problem that it could not be judged. Therefore, it becomes a problem to be able to verify the geometric consistency in the three-dimensional space.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image search apparatus, method, and program that can accurately acquire a reference image similar to a query image.

  In order to achieve the above object, an image search apparatus according to the present invention is an image search apparatus for searching a reference image similar to a query image from a plurality of reference images, and for each of the plurality of reference images, Clustering is performed on a plurality of converted images generated by performing a plurality of conversions including a change in rotation or magnification on a reference image, and local feature amounts extracted from each of the reference images, and the local features A centroid storage unit storing a centroid feature amount obtained for each of a plurality of clusters into which each of the quantities is classified, and for each of the plurality of reference images, the plurality of converted images, and the For each local feature extracted from each reference image, the centro of the cluster to which the local feature belongs. And an index storage unit storing geometric features representing the transformation, and generating the plurality of transformed images by performing the plurality of transformations on the query image based on the input query image, From each of the image and each of the plurality of converted images, the local feature amount extraction unit that extracts the local feature amount and the geometric feature, and each of the local feature amount of the query image extracted by the local feature amount extraction unit, A similarity between each feature quantity of the centroid for each of the plurality of clusters stored in the centroid storage unit and a local feature quantity of the query image is calculated, and corresponds to the local feature quantity of the query image For each of the centroid acquisition unit for acquiring the centroid and the local feature amount of the query image extracted by the local feature amount extraction unit, The ratio or difference of the geometric feature with respect to the reference image stored in the index storage unit corresponding to the centroid acquired by the toroid acquisition unit is calculated, and the geometric feature of each of the plurality of reference images is calculated. A voting unit for voting to a ballot box for the reference image provided for a ratio or difference with respect to the calculated reference image among the ballot boxes provided for each ratio or difference. And, for each of the plurality of reference images, a score unit that calculates a score indicating a similarity between the query image and the reference image based on a vote result of the ballot box for the reference image, and the score Based on the score calculated by the section, similar to the query image A similar image output unit that outputs a reference image.

  The image search method according to the present invention includes a local feature amount extraction unit, a centroid acquisition unit, a voting unit, a score unit, and a similar image output unit, and each of a plurality of reference images is rotated or rotated with respect to the reference image. Each of the local feature values is classified by performing clustering on a plurality of converted images generated by performing a plurality of conversions including a change in magnification and a local feature value extracted from each of the reference images. A centroid storage unit that stores the centroid feature values obtained for each of the plurality of clusters, and each of the plurality of reference images is extracted from each of the plurality of converted images and the reference images. For each local feature amount, a centroid of a cluster to which the local feature amount belongs and a geometric feature representing the transformation are stored. An image search method in an image search apparatus for searching a reference image similar to a query image from a plurality of reference images using an index storage unit, wherein the local feature quantity extraction unit is based on the input query image Performing a plurality of transformations on the query image to generate a plurality of transformation images, and extracting the local feature amount and the geometric feature from each of the query image and the plurality of transformation images; For each of the local feature values of the query image extracted by the local feature value extraction unit, the centroid acquisition unit has characteristics of each centroid for each of the plurality of clusters stored in the centroid storage unit. A degree of similarity between the amount and the local feature amount of the query image, and the centro corresponding to the local feature amount of the query image The voting unit corresponding to the centroid acquired by the centroid acquisition unit for each of the local feature amounts of the query image extracted by the local feature amount extraction unit A ratio or difference between the reference image stored in the index storage unit and the geometric feature is calculated, and a ballot box provided for each of the geometric feature ratio or difference for each of the plurality of reference images. Voting in a ballot box for the reference image provided for a ratio or difference between the calculated geometric feature and the geometric feature, and the score unit for each of the plurality of reference images , Based on the vote result of the ballot box for the reference image, and the query image Calculating a score indicating the degree of similarity with the reference image, and outputting the reference image similar to the query image based on the score calculated by the score unit. , Including.

  The local feature amount extraction unit of the present invention can extract the enlargement ratio of the image and the rotation angle of each of the three axes of the image as the geometric feature.

  The score unit of the present invention, for each of the plurality of reference images, out of the number of votes of each of the ballot box for the reference image, the maximum number of votes, the similarity with the reference image It can be calculated as a score.

  The voting unit of the present invention relates to each of the local feature amounts of the query image extracted by the local feature amount extraction unit in the index storage unit corresponding to the centroid acquired by the centroid acquisition unit. A ratio or difference between the geometric feature and the stored reference image is calculated, and the calculation is performed in the ballot box provided for each of the geometric feature ratio or difference for each of the plurality of reference images. The query image or the reference corresponding to the centroid acquired by the centroid acquisition unit in a ballot box for the reference image provided for a ratio or difference between the geometric feature and the reference image. The number of votes corresponding to the number of local features of the image, by the centroid acquisition unit It is possible to vote the number of votes corresponding to the number of the reference images having the local feature amount corresponding to the obtained centroid, or the number of votes corresponding to the number of local feature amounts extracted from the reference image. .

  The program of the present invention is a program for causing a computer to function as each unit of the image search device of the present invention.

  According to the image search device, method, and program of the present invention, for each of a plurality of reference images, a plurality of converted images generated by performing a plurality of conversions including changes in rotation and magnification on the reference image. Then, clustering is performed on the local features extracted from each of the reference images, and the centroid features obtained for each of the plurality of clusters into which the local features are classified are stored and extracted. For each of the local feature amounts, a centroid of a cluster to which the local feature amount belongs and a geometric feature are stored, and a plurality of transformations are performed on the query image to generate a plurality of transformation images. A local feature and a geometric feature are extracted from each converted image, and each of the extracted local features of the query image is assigned to each of a plurality of clusters. Calculating the similarity between each feature quantity of the centroid and the local feature quantity of the query image, obtaining a centroid corresponding to the local feature quantity of the query image, and extracting each local feature quantity of the query image Voting provided for each of the geometric feature ratios or differences for each of the plurality of reference images. Of the boxes, vote for the ballot box for the reference image provided for the ratio or difference of the calculated geometric feature to the reference image, and based on the ballot box vote result for the reference image, the query image and the reference A score indicating the degree of similarity with the image is calculated, and a reference that is similar to the query image is calculated based on the calculated score By outputting the image, it is possible to obtain a reference image similar to the query image accuracy, the effect is obtained that.

It is a conceptual diagram which shows the process which rotates a reference image or changes an expansion rate. It is a block diagram which shows the example of 1 structure of the image search device which concerns on the 1st Embodiment of this invention. It is a conceptual diagram which shows the process of the affine transformation of a reference image. It is a conceptual diagram which shows an example of a centroid management table. It is a conceptual diagram which shows a centroid-reference feature vector relationship management table. It is a conceptual diagram which shows the process of the vote of the similarity degree regarding a geometric feature. It is a figure which shows an example of the geometric feature of a reference image and a query image. It is a figure which shows an example of the ratio or difference of the geometric feature of a reference image and a query image, and a voting process. It is a flowchart which shows the content of the learning process routine of the image search device which concerns on the 1st and 2nd embodiment of this invention. It is a flowchart which shows the content of the image search process routine of the image search apparatus which concerns on the 1st and 2nd embodiment of this invention. It is a flowchart which shows the content of the image search process routine of the image search apparatus which concerns on the 1st and 2nd embodiment of this invention. It is a block diagram which shows the example of 1 structure of the clustering apparatus concerning the 2nd Embodiment of this invention. It is a block diagram which shows the example of 1 structure of the image search device which concerns on the 2nd Embodiment of this invention. It is explanatory drawing for demonstrating a prior art. It is explanatory drawing for demonstrating a prior art.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Below, the outline | summary of embodiment of this invention is demonstrated first.

<Outline of Embodiment of the Present Invention>
In the embodiment of the present invention, first, a local feature amount is extracted from an image. Next, as shown in FIG. 1, the image is rotated in the vertical direction, the horizontal direction, and in the two-dimensional plane, and a plurality of converted images with different enlargement ratios are created. Extract it. Thereby, for example, the local feature amount extracted from the query image 2 in FIG. 15 matches much with the local feature amount extracted from the reference image 5 in FIG. 1, and the similarity between the query image 2 and the reference image is high. Can be determined.

  Specifically, in the embodiment of the present invention, as a pre-process, first, a local feature amount and a geometric feature of the local feature amount are extracted for each reference image registered in advance. In addition, the extracted local feature amounts are clustered. When the query image is input, the local feature amount and the geometric feature of the local feature amount are also extracted from the query image, and the geometric feature of the reference image belonging to the cluster corresponding to each local feature amount is extracted. Verify the consistency of ratios or differences. A reference image having a local feature amount with a high consistency in geometric ratio or difference with respect to the local feature amount of the query image is output as having a high degree of similarity with the query image.

[First Embodiment]
<Configuration of Image Search Device 100>
A configuration of the image search apparatus 100 according to the first embodiment will be described. In the first embodiment, a case will be described in which the present invention is applied to an image search apparatus that searches a reference image similar to a query image from a plurality of reference images. As shown in FIG. 2, the image search apparatus 100 according to the present invention includes a CPU, a RAM, and a ROM that stores a program and various data for executing a learning process routine and an image search process routine described later. Can be configured with a computer. Functionally, the image search apparatus 100 includes an input unit 1, a calculation unit 2, and an output unit 3, as shown in FIG.

  The input unit 1 receives input of a reference image set including a plurality of reference images and a query image.

  The calculation unit 2 includes a local feature amount extraction unit 20, a clustering unit 22, a centroid storage unit 24, an index storage unit 26, a centroid acquisition unit 28, a voting unit 30, a score unit 32, and a similar image. And an output unit 34.

  The local feature amount extraction unit 20 generates a plurality of converted images by performing a plurality of conversions including a change in the rotation or enlargement ratio on the reference image for each of the plurality of reference images input by the input unit 1. To do. Then, the local feature amount extraction unit 20 extracts a local feature amount and a geometric feature from each of the generated plurality of converted images and reference images.

  Further, the local feature quantity extraction unit 20 performs a plurality of conversions including a change in the rotation or enlargement ratio on the query image based on the query image input by the input unit 1, and performs a plurality of conversions on the query image. The converted image is generated. Then, the local feature amount extraction unit 20 extracts a local feature amount and a geometric feature from each of the query image and the plurality of converted images. In the embodiment of the present invention, the magnification of the image and the rotation angle of the triaxial image are extracted as geometric features.

  Specifically, when a reference image set is input, the local feature quantity extraction unit 20 first extracts a local feature quantity from each of a plurality of reference images. As the local feature amount, for example, SIFT (Scale Invariant Feature Transform) in the case where the rotation parameter and the scale parameter are not used can be used. The local feature amount is extracted not only from the reference image but also from the converted image in which the reference image is rotated in the vertical direction, the horizontal direction, and in the two-dimensional plane, and the enlargement ratio is changed. The local feature amount is expressed in a vector format. The converted image obtained by rotating the reference image can be generated, for example, by affine transformation of the reference image.

  By affine transformation, the point (x, y) in the image before transformation corresponds to the point (x ′, y ′) shown in the following equation (1) in the image after transformation. FIG. 3 is a conceptual diagram illustrating an example of processing for rotating a reference image by affine transformation.

  Here, s is the magnification of the query image and the reference image, and θ, φ, and ψ are the rotation angles of the three axes shown in FIG. The enlargement factor s and the rotation angles θ, φ, and ψ are examples of geometric features.

  In addition, when the query image is input, the local feature amount extraction unit 20 extracts the local feature amount and the corresponding geometric feature from the query image in the same manner as when the reference image set is input.

  If the local feature to be used is a local feature for which a parameter representing the rotation angle is calculated, a parameter representing the rotation angle of the local feature is not created as a value of ψ, without creating a reference image in which ψ is changed. May be used. Further, when the local feature to be used is a local feature for which a parameter representing the enlargement ratio (scale) is calculated, a reference image in which s is changed is not created, and the enlargement ratio of the local feature is used as the value of s. A parameter representing can be used.

  The clustering unit 22 performs clustering on the plurality of converted images generated for each of the plurality of reference images input by the input unit 1 and the local feature amount extracted from each of the reference images. Each is classified into a plurality of clusters. The number of clusters in clustering may be set to an arbitrary value in advance. Moreover, the clustering part 22 calculates | requires the centroid of the said cluster about each of several clusters. The centroid is, for example, a feature value obtained by obtaining the centroid of local feature values belonging to a cluster. Then, the clustering unit 22 stores the centroid feature amount obtained for each of the plurality of clusters as a clustering result in the centroid storage unit 24 described later.

  In addition, the clustering unit 22 has, for each of the plurality of reference images input by the input unit 1, the local feature amount of each of the plurality of converted images and the local feature amount extracted from each of the reference images. The centroid of the cluster to which it belongs and the geometric feature corresponding to the local feature amount are stored in the index storage unit 26 described later.

  The centroid storage unit 24 stores the centroid feature amount for each of the plurality of clusters obtained by the clustering unit 22. An example of the clustering result stored in the centroid storage unit 24 is shown in FIG. The clustering result shown in FIG. 4 is data indicating a combination of a centroid ID and a feature amount.

  The centroid ID is an ID for identifying the centroid of the cluster. The clustering result stored in the centroid storage unit 24 shown in FIG. 4 is an example, and the stored attributes and names may be different. Moreover, you may preserve | save in formats other than a table, such as a text file format.

  The index storage unit 26 stores, for each of a plurality of reference images, a centroid of a cluster to which the local feature amount extracted from the reference image belongs, and a geometric feature corresponding to the local feature amount. An example of data stored in the index storage unit 26 is shown in FIG. The data shown in FIG. 5 is data indicating combinations of the image ID, local feature ID, centroid ID, and geometric feature of the reference image. The image ID and the local feature amount ID are IDs for identifying the reference image and the local feature amount. The data stored in the index storage unit 26 shown in FIG. 5 is an example, and the stored attributes and names may be different. Moreover, you may preserve | save in formats other than a table, such as a text file format.

  The centroid acquisition unit 28 includes, for each local feature amount of the query image extracted by the local feature amount extraction unit 20, each feature amount of the centroid for each of the plurality of clusters stored in the centroid storage unit 24. The similarity with the local feature amount of the query image is calculated, and the centroid corresponding to the local feature amount of the query image is acquired.

  Specifically, the centroid acquisition unit 28 compares each local feature amount of the query image extracted by the local feature amount extraction unit 20 with the feature amount of each centroid stored in the centroid storage unit 24. The centroid having the highest similarity to the local feature is selected, and the local feature is assigned to the centroid ID corresponding to the centroid. As the similarity for assigning the local feature quantity to the centroid, for example, the distance between the local feature quantity and the centroid feature quantity may be used.

  The voting unit 30 stores the reference image stored in the index storage unit 26 corresponding to the centroid acquired by the centroid acquisition unit 28 for each local feature amount of the query image extracted by the local feature extraction unit 20. The ratio or difference with respect to the geometric feature is calculated. In the embodiment of the present invention, a ratio is calculated for an enlargement ratio among geometric features, and a difference is calculated for a rotation angle.

  Then, the voting unit 30 is provided for the ratio or difference between the calculated reference image and the geometric feature of the voting box provided for each of the geometric feature ratio or difference for each of the plurality of reference images. Vote in the ballot box for the given reference image.

  Specifically, when voting, the voting unit 30 first prepares a ballot box to be voted in advance. The ballot box is created by dividing a four-dimensional space of a magnification ratio, a θ difference, a φ difference, and a ψ difference in a grid pattern with respect to the image ID of each reference image. For example, as shown in FIG. 6, when the enlargement ratio is divided into 8 steps, θ is 6 steps, φ is 6 steps, and ψ is 6 steps, 8 × 6 × 6 × 6 = 1728 ballot boxes are assigned to each reference image. Create against.

  Voting is performed for each combination of local feature amounts of the reference image and the query image corresponding to the same centroid ID. For example, as shown in FIG. 7, if local feature amounts of R1, R2, R3, and R4 are extracted from the reference images of ID1 and ID2 and Q1 and Q2 are extracted from the query image, the voting targets are R1, Q1, and R3. A combination of Q1, R2 and Q2, and R4 and Q2. An example of the result of voting is shown in FIG.

  As shown in FIG. 8, in the ballot box A for the image ID1, two votes are voted based on the combination of R1 and Q1 and R2 and Q2. The ballot box B for the image ID 2 is voted for one vote from the combination of R3 and Q1, the ballot box C is voted for one vote from the combination of R4 and Q2, and the number of votes voted in other ballot boxes is 0. It is. The number of votes in the ballot box with the most votes in the ballot box for image ID1 is 2, and the number of votes in the ballot box with the most votes in the ballot box for image ID2 is 1. This is the score of each image ID. It becomes. Although the query image has two local feature quantities that have the same centroid ID for both the reference image ID1 and ID2, the reference image ID1 is more similar to the query image than ID2 as described above. Can be determined to be high.

In addition, the number of votes for one vote may be weighted by the number of local feature amounts corresponding to the same centroid in the query image or the reference image. For example, regarding the centroid corresponding to the local feature amount of the query image, when there are n _q local feature amounts corresponding to the centroid in the query image and n _{r in} the reference image, in voting for the centroid the number of votes w _v can be the value of the expression (2) below.

  By using the number of votes shown in the above formula (2), for example, when a large amount of local feature values corresponding to the same centroid are included as in a window of a building image, the image similarity is the number of windows. Thus, it is possible to obtain a similar image in consideration of local feature values other than the window.

Further, the centroid corresponding to the local feature amount of the query image, local feature amount corresponding to the centroid, if a n _q pieces to the query image, following the vote count w _v in voting for the centroid It can be set as the value of Formula (3).

Further, the centroid corresponding to the local feature amount of the query image, local feature amount corresponding to the centroid, if a n _r pieces to the reference image, the following voting number w _v in voting for the centroid It can be set as the value of Formula (4).

In addition, the number of votes for one vote may be weighted according to how many reference images the local feature amount corresponding to each centroid is included in the reference image set. For example, when the total number of images is N and the local feature amount corresponding to the centroid corresponding to the local feature amount of the query image has been extracted from n _f reference images, the centroid of votes w _v in voting may be a value of the expression (5) below.

  By using the number of votes shown in the above formula (5), it is determined that the degree of similarity is not determined by voting regarding centroids included in many images, and is determined to be more similar when an unusual centroid corresponds. It becomes possible to make it easier.

Further, the number of votes for one vote may be weighted by the number of local feature values in the reference image. For example, when the number of local feature values extracted from a certain reference image is N _{r, the} number of votes w _v in voting for the ballot box for the reference image can be a value of the following equation.

  By using the number of votes shown in the above equation (6), the similarity is likely to increase when a large amount of local feature amounts are extracted from the reference image, while similar when a small amount of local feature amounts are extracted. It is possible to calculate the similarity by considering the ratio of the centroid IDs of the local feature amounts common to the query image and the reference image.

The formula (2), weighting of the formula (5) and (6) can be used at the same time each, in the case of using three simultaneous weighting, Votes w _v in voting the following formula (7) Value.

  The score unit 32 calculates, for each of the plurality of reference images, a score indicating the degree of similarity between the query image and the reference image based on the vote result of the ballot box for the reference image. Specifically, the score unit 32 calculates the score of the reference image by collecting the voting results of a plurality of ballot boxes corresponding to the image ID of the reference image.

  For example, the score unit 32 calculates, for each of the plurality of reference images, the maximum number of votes among the number of votes of each ballot box for the reference image as the score of the reference image.

  The similar image output unit 34 outputs a reference image similar to the query image based on the score calculated by the score unit 32. Specifically, the similar image output unit 34 sorts the calculated reference image scores in descending order, and outputs the image IDs of the reference images having the highest scores.

  The output unit 3 outputs the image IDs of the reference images output by the similar image output unit 34 as a list.

<Operation of Image Retrieval Device 100>
Next, regarding the operation of the image search apparatus 100 according to the first embodiment, a learning processing routine for registering a reference image will be described first. When the input unit 1 receives an input of a reference image set including a plurality of reference images, the image search device 100 executes a learning process routine shown in FIG.

  First, in step S100, the reference image set received by the input unit 1 is acquired.

  Next, in step S102, one reference image is set from the reference image set acquired in step S100.

  Next, in step S104, the local feature quantity extraction unit 20 performs a plurality of conversions including a change in rotation or magnification on the reference image set in step S102 to generate a plurality of converted images.

  In step S106, the local feature amount extraction unit 20 extracts a local feature amount and a geometric feature from each of the plurality of converted images generated in step S104 and the reference image set in step S102.

  In step S108, it is determined whether or not the processing in steps S102 to S106 has been completed for all the reference images included in the reference image set acquired in step S100. If not completed, the process returns to step S102 for processing. If repeated, the process proceeds to step S110.

  In step S110, the clustering unit 22 performs clustering on the local feature amount extracted in step S106, and classifies each local feature amount into a plurality of clusters.

  In step S112, the clustering unit 22 obtains the centroid of the cluster for each of the plurality of clusters obtained in step S110. Then, the centroid feature amount obtained for each of the plurality of clusters is stored in the centroid storage unit 24 by the clustering unit 22.

  In step S114, the clustering unit 22 sets one local feature amount among the plurality of local feature amounts extracted in step S106.

  In step S116, for the local feature amount set in step S114 by the clustering unit 22, the image ID of the reference image corresponding to the local feature amount, the centroid ID of the cluster to which the local feature amount belongs, and the local feature The geometric feature (enlargement ratio s and rotation angles θ, φ, ψ) corresponding to the quantity is stored in the index storage unit 26.

  In step S118, it is determined whether or not the processing of step S114 to step S116 has been completed for all the local feature values extracted in step S106. If not, the process returns to step S114 to repeat the processing and end. If so, the process ends. When the learning process routine ends, registration of the reference image is completed.

  Next, an image search processing routine for inputting a query image and acquiring a reference image similar to the query image will be described. When the input unit 1 receives an input of a query image, the image search device 100 executes an image search processing routine shown in FIGS. 10 and 11.

  First, in step S200, a query image whose input is accepted by the input unit 1 is acquired.

  Next, in step S202, the local feature amount extraction unit 20 performs a plurality of conversions including a change in rotation or enlargement ratio on the query image acquired in step S200 to generate a plurality of converted images.

  In step S204, the local feature amount extraction unit 20 extracts a local feature amount and a geometric feature from each of the query image acquired in step S200 and the plurality of converted images generated in step S202.

  In step S206, one local feature amount is set out of each of the local feature amounts extracted in step S204.

  In step S208, the centroid for the local feature of the query image set in step S206 by the centroid acquisition unit 28 for each of the plurality of clusters stored in the centroid storage unit 24, and the above step S206. The degree of similarity with the local feature amount of the query image set in is calculated, and a centroid corresponding to the local feature amount of the query image set in step S206 is acquired. Then, the centroid acquisition unit 28 assigns a centroid ID corresponding to the acquired centroid to the local feature amount of the query image set in step S206.

  In step S210, the voting unit 30 sets one local feature amount of the reference image stored in the index storage unit 26 corresponding to the centroid ID assigned in step S208.

  In step S212, the voting unit 30 acquires the image ID and the geometric feature stored in the index storage unit 26 corresponding to the local feature amount ID of the local feature amount set in step S210.

  In step S214, the voting unit 30 calculates a ratio or difference between the geometric feature corresponding to the local feature amount set in step S210 and the geometric feature of the reference image corresponding to the image ID acquired in step S212. To do. And with respect to the calculated ratio or difference among the ballot boxes provided for each of the geometric feature ratio or difference for the reference image corresponding to the image ID acquired in step S212 by the voting unit 30. Vote in the ballot box for the provided reference image.

  In step S216, it is determined whether or not the processing in steps S212 to S214 has been completed for all local feature amounts corresponding to the centroid ID assigned in step S208. If not, the process proceeds to step S210. If it returns and repeats a process and is complete | finished, it will transfer to step S218.

  In step S218, it is determined whether or not the processing from step S206 to step S216 has been completed for all the local feature amounts of the query image extracted in step S204. If not, processing returns to step S206 and processing is performed. If the process is completed, the process proceeds to step S220.

  In step S220, one reference image is set from the reference image set.

  In step S222, the score unit 32 calculates the maximum number of votes among the number of votes in each ballot box for the reference image set in step S220 as the score of the reference image.

  In step S224, it is determined whether or not the processing in steps S220 to S222 has been completed for all the reference images included in the reference image set. If not completed, the process returns to step S220 to repeat the processing. If so, the process proceeds to step S226.

  In step S226, the similar image output unit 34 sorts the scores of the reference images calculated in step S222 in descending order, and outputs the image ID of the reference image having the highest score.

  In step S228, the reference image ID output by the similar image output unit 34 is output as a list by the output unit 3, and the process ends.

  As described above, according to the image search device 100 according to the first embodiment, for each of a plurality of reference images, a plurality of conversions including rotation or enlargement rate change are performed on the reference image. The centroids obtained for each of a plurality of clusters into which local feature amounts are classified by clustering local feature amounts extracted from each of the generated plurality of converted images and reference images. Stores feature quantities, and for each extracted local feature quantity, stores the centroid and geometric features of the cluster to which the local feature quantity belongs, and performs multiple transformations on the query image to generate multiple transformed images Then, a local feature and a geometric feature are extracted from each of the query image and the plurality of converted images, and a plurality of local features are extracted for each of the extracted query images. Calculates the similarity between each feature quantity of the centroid for each of the clusters and the local feature quantity of the query image, obtains the centroid corresponding to the local feature quantity of the query image, and extracts the local feature of the query image For each of the quantities, calculate a ratio or difference with respect to the stored reference image corresponding to the acquired centroid, and provide for each of the geometric feature ratio or difference for each of the plurality of reference images. Among the calculated ballot boxes, vote for the ballot box for the reference image provided for the ratio or difference with the geometric feature with respect to the calculated reference image, and query based on the ballot box vote result for the reference image A score indicating the similarity between the image and the reference image is calculated, and the query image and the reference image are calculated based on the calculated score. By outputting the reference image similar, it is possible to acquire the reference image similar to the query image accurately.

  Further, the query image can be acquired at high speed and with high accuracy from the reference images registered in advance, which are images of the same subject.

  Further, for each local feature amount of the image, based on the similarity of the local feature amount and the ratio or difference of the geometric features, the corresponding ballot box is voted and voted to a plurality of ballot boxes corresponding to the image ID of the reference image. It is possible to determine that the similarity is high for images with the same object but different viewpoint directions and magnification rates by determining that the similarity is high for the reference images for which more votes have been made. .

  Also, by extracting the local feature magnification rate and each rotation angle in the three-dimensional space as geometric features, not only in the two-dimensional plane but also when the viewpoint changes in the vertical and horizontal directions, It is possible to determine that the degree of similarity is high.

  In addition, for each reference image, a ballot box for each area is created by dividing the four-dimensional space of the enlargement ratio and the difference between the three rotation angles in a grid pattern, and the ballot boxes corresponding to the reference images are voted. By determining the maximum value in the result as the score of the reference image, it is possible to determine which viewpoint is most likely to be viewed in the three-dimensional space, and the geometrical characteristics of the local feature when viewed from that direction. It becomes possible to verify the consistency.

  Further, at least one of the number of local features corresponding to the same centroid in the query image or the reference image, how many images each centroid was included in the reference image set, and the total number of local features in the reference image By weighting the number of votes for voting using the above, when a large amount of local features corresponding to the same centroid are included in an image, the similarity is determined only by the local features corresponding to the centroid. Or a voting about centroids included in many images, or the similarity is high when a large amount of local features are extracted from the reference image, When a small amount of local feature amount is extracted, it is possible to avoid the similarity from being easily lowered.

[Second Embodiment]
Next, a second embodiment will be described. In the second embodiment, the clustering is performed on the local feature amount extracted from each of the reference images, the score indicating the similarity between the query image and the reference image is calculated, and based on the calculated score Thus, the point that the processing for outputting the reference image similar to the query image is performed by a separate device is different from the first embodiment.

  In the second embodiment, the clustering apparatus 200 executes a learning process routine, and the image search apparatus 210 executes an image search process routine. Note that portions having the same configuration as those of the image search apparatus 100 according to the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

<Configuration of Clustering Device 200>
As illustrated in FIG. 12, the clustering apparatus 200 according to the second embodiment includes an input unit 4 and a calculation unit 5.

  The input unit 4 receives a reference image set including a plurality of reference images.

  The calculation unit 5 includes a local feature quantity extraction unit 50, a clustering unit 22, a centroid storage unit 24, and an index storage unit 26.

  Similar to the local feature quantity extraction unit 20, the local feature quantity extraction unit 50 performs a plurality of conversions including a change in the rotation or enlargement ratio with respect to each of the reference images input by the input unit 1. To generate a plurality of converted images. Then, the local feature amount extraction unit 20 extracts a local feature amount and a geometric feature from each of the generated plurality of converted images and reference images.

<Configuration of Image Retrieval Device 210>
As illustrated in FIG. 13, the image search apparatus 210 according to the second embodiment includes an input unit 6, a calculation unit 7, and an output unit 3.

  The input unit 6 receives an input of a query image.

  The calculation unit 7 includes a local feature amount extraction unit 70, a centroid storage unit 74, an index storage unit 76, a centroid acquisition unit 28, a voting unit 30, a score unit 32, and a similar image output unit 34. I have.

  Similar to the local feature amount extraction unit 20, the local feature amount extraction unit 70 performs a plurality of conversions including a rotation or a change in the enlargement ratio on the query image based on the query image input by the input unit 1. A plurality of converted images are generated. Then, the local feature amount extraction unit 70 extracts a local feature amount and a geometric feature from each of the query image and the plurality of converted images.

  The centroid storage unit 74 stores the centroid feature amount for each of the plurality of clusters obtained by the clustering unit 22 of the clustering apparatus 200.

  The index storage unit 76 stores, for each of a plurality of reference images, a centroid of a cluster to which the local feature amount obtained by the clustering unit 22 of the clustering apparatus 200 belongs, and a geometric feature corresponding to the local feature amount. The

  Next, operations of the clustering device 200 and the image search device 210 according to the second embodiment will be described.

<Operation of Clustering Device 200>
First, the operation of the clustering apparatus 200 will be described. When the input unit 1 receives an input of a reference image set including a plurality of reference images, the clustering apparatus 200 executes the learning processing routine shown in FIG. 9, and the centroid storage unit 24 stores each of the plurality of clusters. Centroid feature quantities are stored, and the index storage unit 26 stores a centroid of a cluster to which the local feature quantity for each of the plurality of reference images belongs, and a geometric feature corresponding to the local feature quantity. The

<Operation of Image Retrieval Device 210>
Next, the operation of the image search device 210 will be described. First, when a centroid feature amount for each of a plurality of clusters stored in the centroid storage unit 24 of the clustering device 200 is input to the image search device 210, the centroid feature amount is stored in the centroid storage unit 74. Further, when a centroid of a cluster to which a local feature amount belongs to each of a plurality of reference images stored in the index storage unit 26 of the clustering apparatus 200 and a geometric feature corresponding to the local feature amount are input. And stored in the index storage unit 76. When the input unit 6 receives an input of a query image, the image search processing routine shown in FIGS. 10 and 11 is executed by the image search device 210, and an image ID of a reference image similar to the query image is output.

  Note that the other configurations and operations of the clustering apparatus 200 and the image search apparatus 210 according to the second embodiment are the same as those of the first embodiment, and thus description thereof is omitted.

  As described above, according to the clustering apparatus 200 and the image search apparatus 210 according to the second embodiment, a reference image similar to a query image can be acquired with high accuracy.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, although the image search device 100 according to the first embodiment and the clustering device 200 according to the second embodiment have been described as including the centroid storage unit 24 and the index storage unit 26, For example, at least one of the centroid storage unit 24 and the index storage unit 26 is provided in an external device of the image search device 100, and the image search device 100 communicates with the external device using a communication unit, so that the centroid storage unit 24 and at least one of the index storage unit 26 may be referred to.

  Moreover, although the case where the image search apparatus 210 according to the second embodiment includes the centroid storage unit 74 and the index storage unit 76 has been described, for example, the centroid storage unit 74 and the index storage unit 76 are provided. Are provided in an external device of the image search device 210, and the image search device 210 communicates with the external device using a communication means, so that at least one of the centroid storage unit 74 and the index storage unit 76 is provided. You may make it refer.

  Further, the image search device 100, the clustering device 200, and the image search device 210 described above have a computer system therein, but the “computer system” is a homepage if a WWW system is used. It also includes the provision environment (or display environment).

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

1, 4, 6 Input unit 2, 5, 7 Calculation unit 3 Output unit 20, 50, 70 Local feature extraction unit 22 Clustering unit 24, 74 Centroid storage unit 26, 76 Index storage unit 28 Centroid acquisition unit 32 Score Unit 30 Voting unit 34 Similar image output unit 100 Image search device 200 Clustering device 210 Image search device

Claims (9)

An image search device for searching a reference image similar to a query image from a plurality of reference images,
For each of the plurality of reference images, a plurality of converted images generated by performing a plurality of conversions including a change in rotation or magnification on the reference image, and local feature amounts extracted from each of the reference images A centroid storage unit that stores centroid feature values obtained for each of a plurality of clusters in which each of the local feature values is classified by performing clustering on
For each of the plurality of reference images, for each of the plurality of converted images and each of the local feature amounts extracted from each of the reference images, a centroid of a cluster to which the local feature amount belongs and a geometry representing the conversion An index storage that stores the features;
Based on the inputted query image, the plurality of transformations are performed on the query image to generate a plurality of transformed images, and from each of the query image and the plurality of transformed images, the local feature amount and the A local feature extraction unit for extracting geometric features;
For each of the local feature amounts of the query image extracted by the local feature amount extraction unit, each feature amount of the centroid for each of the plurality of clusters stored in the centroid storage unit, and A centroid acquisition unit that calculates a similarity with a local feature and acquires the centroid corresponding to the local feature of the query image;
For each of the local feature amounts of the query image extracted by the local feature amount extraction unit, the reference image stored in the index storage unit corresponding to the centroid acquired by the centroid acquisition unit A ratio or difference with a geometric feature is calculated, and for each of the plurality of reference images, out of the voting box provided for each of the geometric feature ratio or difference, the geometric feature with respect to the calculated reference image A voting unit for voting in a ballot box for the reference image provided for the ratio or difference with
For each of the plurality of reference images, a score unit that calculates a score indicating the degree of similarity between the query image and the reference image based on the voting result of the ballot box for the reference image;
A similar image output unit that outputs the reference image similar to the query image based on the score calculated by the score unit;
Image search device including The image search apparatus according to claim 1, wherein the local feature amount extraction unit extracts an image enlargement ratio and a rotation angle of each of the three axes of the image as the geometric feature. The score unit calculates, for each of the plurality of reference images, the maximum number of votes among the number of votes of each of the ballot boxes for the reference image as the score indicating the degree of similarity with the reference image. The image search device according to claim 1 or 2. The voting unit stores each local feature amount of the query image extracted by the local feature amount extraction unit in the index storage unit corresponding to the centroid acquired by the centroid acquisition unit. The ratio or difference of the geometric feature with respect to the reference image is calculated, and the calculated reference among the ballot boxes provided for each of the geometric feature ratio or difference for each of the plurality of reference images. The query image or the local area of the reference image corresponding to the centroid acquired by the centroid acquisition unit in a ballot box for the reference image provided for a ratio or difference between the geometric feature with respect to the image The number of votes corresponding to the number of features is acquired by the centroid acquisition unit. The number of votes according to the number of the reference images having the local feature amount corresponding to the centroid or the number of votes according to the number of local feature amounts extracted from the reference image is voted. The image search device according to claim 1. Including a local feature extraction unit, a centroid acquisition unit, a voting unit, a score unit, and a similar image output unit,
For each of a plurality of reference images, a plurality of converted images generated by performing a plurality of conversions including a change in rotation or magnification on the reference image, and a local feature amount extracted from each of the reference images A centroid storage unit that stores centroid feature values obtained for each of a plurality of clusters in which each of the local feature values is classified by performing clustering on each of the plurality of reference images For each of the local feature values extracted from each of the plurality of converted images and the reference image, a centroid of a cluster to which the local feature value belongs and an index storage unit that stores geometric features representing the conversion are used. The image search device searches for a reference image similar to the query image from a plurality of reference images. An image search method in,
The local feature quantity extraction unit performs the plurality of conversions on the query image based on the input query image to generate a plurality of conversion images, and each of the query image and the plurality of conversion images Extracting the local feature and the geometric feature from:
For each of the local feature values of the query image extracted by the local feature value extraction unit, the centroid acquisition unit has characteristics of each centroid for each of the plurality of clusters stored in the centroid storage unit. Calculating a similarity between a quantity and a local feature quantity of the query image, and obtaining the centroid corresponding to the local feature quantity of the query image;
The voting unit stores each of the local feature values of the query image extracted by the local feature value extracting unit in the index storage unit corresponding to the centroid acquired by the centroid acquisition unit. The ratio or difference of the geometric feature with respect to the reference image is calculated, and the calculated reference among the ballot boxes provided for each of the geometric feature ratio or difference for each of the plurality of reference images. Voting in a ballot box for the reference image provided for a ratio or difference of the geometric feature to an image;
The score unit calculates, for each of the plurality of reference images, a score indicating the similarity between the query image and the reference image based on the voting result of the ballot box for the reference image;
The similar image output unit outputting the reference image similar to the query image based on the score calculated by the score unit;
Image search method including The image search method according to claim 5, wherein the step of extracting the geometric feature by the local feature amount extraction unit extracts an enlargement ratio of an image and a rotation angle of each of three axes of the image as the geometric feature. The step of calculating the score by the score unit is similar to the reference image, with respect to each of the plurality of reference images, the maximum number of votes in each of the ballot boxes for the reference image. The image search method according to claim 6, wherein the image search method calculates the score indicating the degree. The step of voting by the voting unit corresponds to the index storage unit corresponding to the centroid acquired by the centroid acquisition unit for each local feature amount of the query image extracted by the local feature amount extraction unit. The ratio or difference of the geometric feature with respect to the reference image stored in the reference image is calculated, and the calculation is performed among the ballot boxes provided for the geometric feature ratio or difference for each of the plurality of reference images. The query image corresponding to the centroid acquired by the centroid acquisition unit in a ballot box for the reference image provided for a ratio or difference with the geometric feature relative to the reference image, Obtain the number of votes corresponding to the number of local features of the reference image and the centroid The number of votes corresponding to the number of the reference images having the local feature amount corresponding to the centroid obtained by voting or the number of votes corresponding to the number of local feature amounts extracted from the reference image is voted. Item 8. The image search method according to any one of Items 7 to 8.   The program for functioning a computer as each part of the image search device of any one of Claims 1-4.