JP5164222B2 - Image search method and system - Google Patents

Description

  The present invention relates to an image search method and system for searching an image similar to a query image from among a large number of search target images, and in particular, compares local feature amounts of each image to obtain corresponding points, and based on the number of the corresponding points The present invention relates to an image search method and system for determining similarity between images.

  Non-Patent Document 1 discloses SIFT (Scale Invariant Feature Transform) as one method for robustly detecting an image of a rigid body such as a building. In this SIFT, local feature amounts are extracted in advance from both the query image and the search target image, and the nearest neighbor search is executed based on the Euclidean distance L between the local feature amounts of each image. Then, local feature quantities having a short distance are used as corresponding point pairs, and finally, a search target image having many corresponding point pairs is used as a search result. At this time, not only the nearest distance L1 but also the second nearest distance L2 is obtained in order to improve the matching point matching accuracy of the nearest feature point, and the ratio (L1 / L2: ratio of distances) of both is predetermined. If it is less than or equal to the threshold value t, it is regarded as a corresponding point, but if it is larger than the threshold value t, it is not regarded as a corresponding point.

  In Non-Patent Document 1, t = 0.8 because a rapid increase in false correspondence points was confirmed at t = 0.85 in a preliminary experiment. However, if the threshold value t is set to a small value, the accuracy of corresponding points tends to increase, but the number of corresponding point pairs decreases, so the search accuracy cannot always be increased. That is, in Non-Patent Document 1, it is difficult to set the threshold value t to an appropriate value.

  In response to such a technical problem, Patent Document 1 discloses that a threshold t is increased stepwise if there are few corresponding point pairs and a threshold t is decreased stepwise if there are many, so that a predetermined number of corresponding point pairs can be obtained. As described above, a technique for adaptively changing the threshold value t according to the number of corresponding point pairs to be extracted is disclosed.

JP 2007-140613 A

David G. Lowe, "Distinctive image features from scale-invariant keypoints" International Journal of Computer Vision, 60, 2 (2004), pp.91-110.

  In the above-described conventional technique, when the number of corresponding point pairs obtained with the predetermined threshold value t is small, the threshold value t increases adaptively, so the number of corresponding point pairs increases. However, since the same object is not originally included in the image, the threshold value t is also raised for a pair of query image and search target image with few corresponding point pairs. There was a problem that even a pair of points would be a corresponding point pair.

  Further, in the above-described conventional technology, for each feature point, the corresponding point is determined based only on the similarity of the local feature amount. For example, like a window in a building image or a signboard of a store developing a chain When similar-shaped subjects are included in each of the query image and the search target image, there is a problem that many feature points with high similarity are erroneously extracted as corresponding points.

  The object of the present invention is to solve the above-described problems of the prior art and consider the similarity between the query image and the search target image, not only the similarity of the local feature amount of each feature point, but also whether or not the subject is the same. It is an object of the present invention to provide an image search method and system that can be determined.

  In order to achieve the above object, the present invention extracts a local feature amount from a query image and feature points of each search target image in an image search system that searches an image similar to a query image from a set of search target images. Compares local feature values extracted from each feature point of the query image and the search target image with the local feature value extraction means, and extracts the corresponding points with the highest similarity in the top N best for each feature point of the query image And a difference distribution calculation means for comparing the local feature amount of the N best corresponding point with the local feature amount of the corresponding feature point of the query image, and calculating a distribution regarding the difference between the local regions of the two, based on the difference distribution, Corresponding point candidate extracting means for extracting a plurality of corresponding point candidates having high likelihood as corresponding points, comparing the local feature amount of the corresponding point candidate with the local feature amount of the corresponding feature point of the query image, and each local region Normalizing means for normalizing the position coordinates, and comparing the normalized position coordinates of the corresponding point candidates with the position coordinates of the corresponding feature points of the query image, and calculating the distribution of the relative positional relationship of each position coordinate A relative position distribution calculating means, a corresponding point extracting means for extracting a plurality of corresponding points having a high likelihood as corresponding points based on the relative position distribution, and a query image based on the extracted corresponding points And a similar image determination means for determining the search target image.

  According to the present invention, when extracting corresponding points based on local feature amounts of feature points, a small number of corresponding points are extracted under strict extraction conditions based on individual similarities between feature points as in the past. Rather, the extraction condition is relaxed to extract a large number of corresponding point candidates, and then the corresponding points with high likelihood are extracted based on the relative relationship of the local feature amounts of the large number of corresponding point candidates. did. Therefore, even when comparing images that have many feature points with similar local features despite different subjects, such as a building image window or an image of a chained store sign, these features can be compared. A point is less likely to be erroneously extracted as a corresponding point.

1 is a block diagram of an image search system according to an embodiment of the present invention. It is the figure which showed the calculation method of difference distribution typically. It is the figure which showed an example of difference distribution. It is the figure which showed an example of the extraction method of a corresponding point candidate. It is the figure which showed typically the calculation method of normalization and a relative position distribution. It is the figure which showed an example of relative position distribution. It is the figure which showed an example of the extraction method of a corresponding point. It is the main flow which showed operation | movement of one Embodiment of this invention. It is the flowchart which showed the procedure of the corresponding point candidate extraction process. It is the flowchart which showed the procedure of the corresponding point extraction process.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a main part of an image search system according to an embodiment of the present invention. Here, illustrations of components unnecessary for the description of the present invention are omitted.

  The local feature quantity extraction unit 1 extracts local feature quantities fq (i) and fw (k, j) with reference to each feature point of the query image Iq and a large number of search target images Iw (k). Note that the symbol k is an identifier of the search target image. In this embodiment, feature points are extracted and associated using SIFT. That is, for specifying the local region, feature point extraction based on the extreme value in the scale space by Difference of Gaussian (DoG) disclosed in Non-Patent Document 1 is used. As a result of the feature point extraction, the position of the feature point and the range of the local region are calculated, and a luminance gradient direction histogram is used as the feature descriptor of the local region.

  Such a direction histogram calculates the luminance gradient of each pixel in the feature region, weights it to generate a histogram, and rotates the feature region in that direction based on the direction of the most bin region (orientation) The direction histogram in the luminance direction is generated again, the histogram is further divided into blocks, the direction histogram is calculated in each block, and this is normalized and vectorized. In this embodiment, since the luminance direction in the block is divided into 8 directions and the target area is divided into 16 areas, one feature descriptor has 8 * 16 = 128 dimensions.

  As feature of these feature descriptors, local region features are generated, so it is resistant to occlusion, scales are determined for feature points, and the image size is unchanged, and orientation is set in the image plane based on the luminance gradient. Since it is performed, it is invariable to rotation with respect to the image plane. Furthermore, since edge components are used, it is resistant to luminance changes. Such feature point detection is performed on all pixels of the image, but even if a certain feature point takes an extreme value, it is excluded from the feature region if it is inappropriate as a feature point.

  In the present embodiment, each local feature quantity fq (i) of the query image Iq is expressed by the following equation (1). pq (i) is the position of the feature point expressed in homogeneous coordinates, op (i) is the orientation of the feature point, σq (i) is the scale at which the feature point was found, dq (i) is the feature descriptor, A symbol i is a feature point identifier of the query image Iq.

        fq (i) = {pq (i), op (i), σq (i), dq (i)} (1)

  Similarly, the local feature quantity fw (k, j) of each search target image Iw (k) is expressed by the following equation (2). pw (k, j) is the position of the feature point expressed in homogeneous coordinates, ow (k, j) is the orientation of the feature point, σw (k, j) is the scale where the feature point was found, dw (k, j ) Is a feature descriptor, and symbol j is a feature point identifier of the search target image Iw (k).

        fw (k, j) = {pw (k, j), ow (k, j), σw (k, j), dw (k, j)} (2)

  The N best extraction unit 2 compares the local feature amount of each feature point of the query image Iq with the local feature amount of each feature point of all the search target images Iw (k), and relaxes the extraction condition than the conventional technique. For each feature point of the query image Iq, the top N feature points with high local feature similarity are extracted as N best corresponding points from each search target image Iw (k). Therefore, if m feature points are set in the query image Iq, (m × N) N best corresponding points are extracted from all the search target images Iw (k).

  In the present embodiment, for each feature point of the query image Iq, a local feature closest search is performed for each feature point of all the search target images Iw (k), and for each search target image, The similarity is calculated based on the distance. In this embodiment, the similarity between the local feature quantities fq (i) and fw (k, j) of each feature point is represented by the feature descriptors dq (i), dw (k, represented by Euclidean distance L between j).

        L = | dq (i) −dw (k, j) | (3)

  The difference distribution calculation unit 3 compares the local feature amount of the N best corresponding point with the local feature amount of the corresponding feature point of the query image, and calculates a distribution relating to the difference of each local region. In this embodiment, the distribution regarding the scale ratio (or difference) of the local region and the angular difference (or angle ratio) of the orientation is calculated. In this embodiment, as shown in an example in FIG. 2, for each N best corresponding point, the size ratio of the local region (scale) between the local feature amount and the corresponding local feature amount of the query image Iq, and the angle of orientation The difference distribution shown in the example of FIG. 3 is calculated by calculating and associating the differences, and performing and plotting the differences for all N best corresponding points.

  The corresponding point candidate extraction unit 4 extracts a plurality of corresponding point candidates having a high likelihood as corresponding points based on the difference distribution. FIG. 4 is a diagram showing an example of a corresponding point candidate extraction method. In this embodiment, N best corresponding points existing within a predetermined range centering on a position having the highest distribution density are set as corresponding point candidates. The other N best corresponding points are removed as noise.

  The normalization unit 5 normalizes the position coordinates of both of the search target images and the query image based on the local feature amount of the corresponding point candidate, in particular, the size of the local region, by scaling one of the search target images and the query image. . In the present embodiment, the difference between the size of the query image Iq and the size of the search target image Iw (k) is calculated based on the corresponding point at the position with the highest distribution density, and based on this, the corresponding point candidate is calculated. The position coordinates are converted.

  The relative position distribution calculation unit 6 calculates a relative positional relationship distribution between the normalized position coordinates of the corresponding point candidate and the position coordinates of the corresponding feature point of the query image.

  FIG. 5 is a diagram schematically showing an example of the normalization and distribution calculation. In the present embodiment, the normalized position coordinates of the corresponding point candidates are compared with the position coordinates of the corresponding feature points of the query image. Then, the relative position distribution of each position coordinate in the X direction and the Y direction is obtained, and this is performed for all corresponding point candidates and plotted, thereby calculating the relative position distribution shown in FIG.

  The corresponding point extraction unit 7 extracts a plurality of corresponding points having high likelihood as corresponding points based on the relative position distribution. FIG. 7 is a diagram showing an example of a method for extracting corresponding points. In this embodiment, corresponding point candidates existing within a predetermined range centering on a position having the highest distribution density are used as the corresponding points. The corresponding point candidates are removed as noise. Further, when a plurality of corresponding points are extracted for each feature point of the query image Iq, other than the most likely corresponding points are removed.

  The similar image determination unit 8 extracts the search target image Iw (k) having the most corresponding points, and outputs this as a search result. Alternatively, projective transformation is performed using the plurality of extracted corresponding points, and the search target image Iw (k) having the largest number of corresponding points that can be projectively transformed is used as the search result. good.

  Next, the operation of the embodiment of the present invention will be described in detail with reference to a flowchart. FIG. 8 is a main flow showing the operation of one embodiment of the present invention.

  In step S1, the local feature quantity extraction unit 1 extracts a local area feature quantity fq (i) based on each feature point from the query image Iq, and further extracts each feature point from all search target images Iw (k). The feature quantity fw (k, j) of the local region with reference to is extracted. In step S2, one of the search target images Iw (k) is selected as the current comparison control. In step S3, a corresponding point candidate extraction process is performed in which the feature points of the search target image Iw (k) whose local feature amounts match or similar to the feature points of the query image Iq are extracted as corresponding point candidates.

  FIG. 9 is a flowchart showing the procedure of the corresponding point candidate extraction process executed in step S3. In step S101, one local feature quantity fq (of many feature points extracted from the query image Iq is shown. i) is selected. In step S102, the similarity between the local feature quantity fw of all feature points of the current search target image Iw (k) and the current local feature quantity fq (i) of the query image Iq is determined by the distance between the feature quantities ( Euclidean distance). In step S103, the top N corresponding points (N best corresponding points) with a short local feature distance are extracted. In step S104, it is determined whether or not the extraction of the N best corresponding points has been completed for all the feature points of the query image Iq, and the process returns to step S101 and is repeated until completion.

  In step S105, a pair of one of the extracted N best corresponding points and the corresponding feature point of the query image Iq is acquired. In step S106, the acquired local feature values of the pair are compared with each other, and the size ratio and the angle difference between the orientations are calculated. In step S107, as described with reference to FIG. 2, the current N best corresponding points are plotted based on the size ratio and the orientation angle difference. In step S108, it is determined whether or not plotting has been completed for all N best corresponding points. If not completed, the process returns to step S105, and the same procedure is repeated for the remaining N best corresponding points.

  When plotting is completed for all N best corresponding points and the phase distribution of FIG. 3 is completed, the process proceeds to step S109, and the maximum position of the distribution density is detected. In step S110, as shown in FIG. 4, N best corresponding points within a predetermined range from the maximum position are taken as corresponding point candidates, and other N best corresponding points are discarded as noise.

  Returning to FIG. 8, in step S4, a corresponding point extraction process for extracting corresponding points from the corresponding point candidates is executed. FIG. 10 is a flowchart showing the procedure of the corresponding point extraction process.

  In step S201, the local feature amount at the maximum position of the distribution density is compared with the local feature amount of the feature point of the query image Iq corresponding to the maximum position, and the search target image for the query image Iq based on the scale of both. The scaling factor of Iw (k) is calculated. In step S202, in order to normalize the position coordinates of the search target image Iw (k) in accordance with the query image Iq, the position coordinates of the corresponding point candidates are converted based on the scaling factor.

  In step S203, attention is paid to one of the corresponding point candidates, and as described with reference to FIG. 5, the normalized position coordinates and the position coordinates of the characteristic points of the query image Iq corresponding thereto are calculated. A deviation amount is calculated for the X direction and the Y direction. In step S204, the amount of deviation in the X and Y directions is plotted. In step S205, it is determined whether plotting has been completed for all corresponding point candidates. If not completed, the process returns to step S203, and the same procedure is repeated for the remaining corresponding point candidates.

  When plotting is completed for all the corresponding point candidates and the relative position distribution shown in FIG. 6 is completed, the process proceeds to step S206. As shown in FIG. 7, a predetermined extraction is performed around the position where the distribution density is maximum. A range is set. In step S207, the corresponding point candidates in the extraction range are set as corresponding points, and the other corresponding point candidates are discarded as noise.

  Returning to FIG. 8, in step S <b> 5, it is determined whether or not the corresponding point extraction processing has been completed for all the search target images Iw (k). If not completed, the process returns to step S2, and the above processes are repeated while switching the target search target image Iw (k). In step S6, the similar image determination unit 8 sorts the search target images Iw (k) by the number of corresponding points, and is the only search target image having the largest number of corresponding points, or a plurality of searches with the highest number of corresponding points. The target image is output as a search result.

  Using the corresponding points extracted as described above, projective transformation from the query image Iq is performed for each search target image Iw (k), and the corresponding points that can actually be used for constructing the projective transformation matrix. A single or upper plurality of search target images having the largest number may be output as search results.

  DESCRIPTION OF SYMBOLS 1 ... Local feature-value extraction part, 2 ... N best extraction part, 3 ... Difference distribution calculation part, 4 ... Corresponding point candidate extraction part, 5 ... Normalization part, 6 ... Relative position distribution calculation part, 7 ... Corresponding point extraction part , 8 ... Similar image determination unit

Claims (5)

In an image search system that searches an image similar to a query image from a set of search target images,
A local feature amount extraction means for extracting a local feature amount from the query image and feature points of each search target image;
N best extraction means for comparing local feature amounts extracted from each feature point of the query image and the search target image, and extracting corresponding points of the highest N best similarity for each feature point of the query image;
Comparing the local feature amount of the N best corresponding point with the local feature amount of the corresponding feature point of the query image, and calculating a distribution regarding the difference between the local regions of both,
Corresponding point candidate extracting means for extracting a plurality of corresponding point candidates with high likelihood as corresponding points based on the difference distribution;
Normalizing means for comparing the local feature amount of the corresponding point candidate with the local feature amount of the corresponding feature point of the query image, and normalizing the position coordinates of each local region;
A relative position distribution calculating unit that compares the normalized position coordinates of the corresponding point candidate with the position coordinates of the corresponding feature point of the query image, and calculates a relative positional relationship distribution of each position coordinate;
Corresponding point extraction means for extracting a plurality of corresponding points having a high likelihood as corresponding points based on the relative position distribution;
An image search system comprising: similar image determination means for determining a search target image similar to a query image based on the extracted corresponding points.   The difference distribution calculating means compares the local feature amount of the N best corresponding point with the local feature amount of the corresponding feature point of the query image, and calculates a distribution related to the difference in scale and orientation of the local regions of both. The image search system according to claim 1.   The relative position distribution calculating means compares the normalized position coordinates of the corresponding point candidate with the position coordinates of the corresponding feature points of the query image, and calculates a distribution of position shifts in the X direction and Y direction of each position coordinate. The image search system according to claim 1, wherein:   The similar image determining means performs a projective transformation between the query image and each search target image based on the plurality of extracted corresponding points, and is similar to the query image based on the corresponding points used for the projective transformation. 4. The image search system according to claim 1, wherein a search target image is determined. In an image search method for searching an image similar to a query image from a set of search target images,
A procedure for extracting a local feature amount from a feature point of a query image and each search target image;
A procedure for comparing local feature amounts extracted from each feature point of the query image and the search target image, and extracting a corresponding point with the highest N similarity for each feature point of the query image,
Comparing the local feature amount of the N best corresponding point with the local feature amount of the corresponding feature point of the query image, and calculating a distribution regarding the difference between the local regions of the two,
A procedure for extracting a plurality of corresponding point candidates having a high likelihood as corresponding points based on the difference distribution;
Comparing the local feature amount of the corresponding point candidate with the local feature amount of the corresponding feature point of the query image, and normalizing the position coordinates of each local region;
Comparing the normalized position coordinates of the corresponding point candidate with the position coordinates of the corresponding feature points of the query image, and calculating a relative positional relationship distribution of the position coordinates;
A procedure for extracting a plurality of corresponding points having high likelihood as corresponding points based on the relative position distribution;
A method for determining an image to be searched similar to a query image based on the extracted corresponding points.

Images