JP5616310B2 - Image matching apparatus and image matching program - Google Patents

Description

  The present invention relates to an image matching apparatus and an image matching program that perform image matching based on feature correspondence.

  In recent years, research on image matching has been actively conducted in a wide range of fields such as image processing, computer vision, and pattern recognition. Image matching is the same as or similar to a query image from a group of images in the search target database when there is a single query image input by the user and a database having one or more images as search targets. This is a technique for obtaining images. Image matching can also be used to determine whether an image similar to or similar to the query image exists in the search target database by setting the threshold value.

  Image matching approaches are broadly divided into region-based and feature-based approaches. The region-based matching method performs matching based on the difference and similarity between two images. Specifically, the query image or the object part in the query image is used as a template image, and the template image is compared with the image in the search target database while gradually shifting, and this processing is performed on all the search target database images. By doing so, image matching is realized. At the time of comparison, for example, a method for obtaining an image or position where the difference between the query image and the image in the search target database is minimized using a technique such as SAD (Sum of Absolute Differences) or SSD (Sum of Squared Differences), or There is a method for obtaining an image having the maximum similarity between a query image and a search target database image and a position thereof using a method such as NCC (Normalized Cross Correlation) or POC (Phase-Only Correlation) (for example, non-patent literature) 1).

  However, these region-based approaches have problems such as an increase in calculation time required for image matching and weakness against large image deformation. In recent years, feature-based image matching techniques have been actively studied. In feature-based matching, feature points (feature points and feature areas) with large changes in pixel gray values, such as corners, are extracted from each image in the query image and the search target database image, and the local regions around them are extracted. On the other hand, this is realized by calculating local descriptors (features) and comparing the local descriptors. Specifically, for all feature locations acquired from the query image, the local descriptor is compared with all the feature locations included in the image in the search target database, and the nearest neighbor of each feature location in the query image Is calculated, and a vote is cast on the image in the search target database including the characteristic portion, and an image with the maximum number of votes obtained as a voting result is output as a matching result image with the query image (for example, see Non-Patent Document 2). ).

  Note that there are methods such as Harris-Affine region, Hessian-Affine region, Difference of Gaussians (DoG) region, and Maximum Stable Extremal Regions (MSER) for extracting feature points and feature regions, and SIFT (Scale-Invariant Feature) Transform) and GLOH (Gradient Location-Orientation Histogram) have been proposed. Further, as a method for improving SIFT, PCA (Principal Component Analysis) -SIFT, SURF (Supeeded-Up Robust Features), ASIFT (Affine-SIFT), and the like have been proposed (see Non-Patent Document 1). These feature-based matching methods are suitable for applications that match a large number of images because they require less computation time than region-based matching methods and are robust to image deformation.

Koichi Ito, Toru Takahashi, Takafumi Aoki, “Examination of high-precision image matching method”, 25th Signal Processing Symposium, No. C5-1, pp. 547-552, 2010. Takayuki Hondo and Koichi Kise, “Performance comparison of local features for large-scale image recognition”, Image Recognition and Understanding Symposium (MIRU2008) Proceedings, IS5-6, pp. 550-555, 2008.

  By the way, in the method described in Non-Patent Document 2, at the time of collation, the number of feature locations corresponding to the feature locations of the query image among the feature locations existing in the images in each search target database is calculated. The number of votes for the inner image is compared. That is, one equivalent vote is cast for all the images in the search target database that have feature locations corresponding to the feature locations of the query image.

  For this reason, in the image in the search target database, an erroneous association occurs in which a feature location of an image other than an image suitable as a result of image matching with the query image is calculated as the nearest neighborhood of the feature location of the query image. Sometimes. Mismatch is mainly caused by the fact that the correct corresponding location for the feature location in the query image does not exist due to a change in the shooting angle, is hidden by another object, etc., or the image feature changes due to a change in the shooting environment. . If many mis-associations occur, the number of votes of images suitable as a result of image matching decreases, and there is a problem that image matching cannot be performed correctly.

  The present invention has been made in view of such circumstances, and in order to solve the problem that image matching cannot be performed correctly because the number of image votes suitable as a result of image matching is reduced when a large number of mis-associations occur. Another object of the present invention is to provide an image matching apparatus and an image matching program that can reduce the influence of incorrect matching in feature-based image matching.

  The present invention provides a feature representation means for calculating a feature amount for each feature location extracted from each of a query image and a stored image stored in advance in the search target database, and between the query image and the stored image, An image matching apparatus comprising: a matching unit that calculates a score from the feature amount calculated by a feature expression unit, and performs image matching based on a score vote result for the stored image corresponding to the feature portion of the query image. Then, the matching unit votes a score indicating the likelihood of the association between the feature location between the query image and the stored image, performs the image matching based on the score voting result, and matches the query image. The stored image is obtained.

  The present invention is characterized in that the collating means performs the image matching based on a value obtained by normalizing the score voting result by the number of characteristic portions in the stored image in the search target database.

  The present invention is characterized in that the feature expression means extracts the feature location after normalizing the query image and the saved image in accordance with an image size.

  In the present invention, when the matching unit obtains the stored image that matches the query image, the Hessian value of the feature location corresponding to the feature location of the query image with respect to the saved image in the search target database. The image matching is performed by performing score voting based on the above.

  In the present invention, when the matching unit obtains the stored image that matches the query image, the matching unit is based on the inter-vector distance value between the stored image in the search target database and the feature location of the query image. The image matching is performed by performing score voting.

  According to the present invention, the matching unit includes the feature image of the saved image in the search target database including the feature location having the shortest inter-vector distance value from the feature location of the query image. The score is voted against.

  In the present invention, the matching unit includes a predetermined number of feature locations in ascending order of the inter-vector distance value from the feature location of the query image among the feature locations of the saved image in the search target database. A score is voted for each of the stored images.

  According to the present invention, the matching unit includes a feature location that satisfies a predetermined condition from the feature location of the saved image in the search target database in ascending order of the inter-vector distance value from the feature location of the query image. A score is voted for each of the stored images that are included.

  The present invention is characterized by causing a computer to function as the image matching device.

  According to the present invention, it is possible to achieve an effect of improving the accuracy of image matching.

It is a block diagram which shows the structure of the 1st Embodiment of this invention. It is a flowchart which shows the processing operation of the characteristic expression part 11 shown in FIG. It is a flowchart which shows the processing operation of the collation part 12 shown in FIG. 1 in the 1st Embodiment of this invention. It is a flowchart which shows the processing operation of the collation part 12 shown in FIG. 1 in the 2nd Embodiment of this invention. It is a flowchart which shows the processing operation of the collation part 12 shown in FIG. 1 in the 3rd Embodiment of this invention.

<First Embodiment>
Hereinafter, an image matching apparatus according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of this embodiment. The image matching device is configured by a computer device. In FIG. 1, reference numeral 1 denotes an image matching unit that performs image matching processing. Reference numeral 2 denotes an image input unit for inputting two-dimensional image data obtained by photographing with a camera or the like. Reference numeral 3 denotes a storage unit that stores data necessary for image matching processing. Reference numeral 4 denotes an input unit composed of a keyboard or the like. Reference numeral 5 denotes a display unit composed of a display device or the like.

  Reference numeral 11 denotes a feature expression unit that extracts a query image input via the image input unit 2 and a feature location of a search target database image prepared in advance in the storage unit 3 and calculates a local descriptor of the feature location. The normalization processing unit 111 and the feature extraction unit 112 are included. Reference numeral 12 denotes a local descriptor of each feature location calculated by the feature representation unit 11 between the feature location of the image input by the image input unit 2 and the image prepared in the storage unit 3, The collation unit outputs an image matching result, and includes a score weighting unit 121 and a normalization processing unit 122.

  Next, the processing operation of the image matching apparatus shown in FIG. 1 will be described with reference to FIG. Here, a description will be given assuming that a two-dimensional image is input as a query and one or a plurality of two-dimensional images are stored in advance in the storage unit 3 as a search target database. Here, a feature location is extracted from the query image input from the image input unit 2 by the feature expression unit 11 shown in FIG. 1 and the image in the search target database stored in advance in the storage unit 3, and local description is performed for each feature location. The operation of calculating and saving children will be described. FIG. 2 shows an operation of extracting a feature location from a query image input from the image input unit 2 and a search target database image stored in advance in the storage unit 3, and calculating and storing a local descriptor for each feature location. It is a flowchart.

  First, when the user operates the input unit 4 to instruct input of a two-dimensional image, the feature expression unit 11 inputs two-dimensional image data from the image input unit 2 (step S1). The input two-dimensional image data is a user input query image and an image in a search target database stored in the storage unit 3 in advance. If the image sizes in the search target database are different, the normalization processing unit 111 normalizes and inputs the image size. This normalization has the meaning of keeping the number of feature locations detected at the time of feature location detection fair according to the image size, and has the effect of improving the probability of this association when finally associating feature locations.

  Next, the feature extraction unit 112 extracts feature locations from the input image (step S2), and calculates local descriptors at all the extracted feature locations (step S3). Note that the feature extraction processing is executed using a method such as Harris-Affine region, Hessian-Affine region, Difference of Gaussians (DOG), or Maximum Stable Extremal Regions (MSER) described in Non-Patent Document 1. The local descriptor is executed by using a method such as SIFT, GLOH, PCA-SIFT, SURF, or ASIFT described in Non-Patent Document 1, for example. The feature extraction unit 112 performs feature location extraction processing and local descriptor calculation processing on all input two-dimensional image data. Finally, the image file name of the two-dimensional image data and the local descriptor of the feature location obtained from each image are associated and stored in the storage unit 3 as feature data (step S4).

  Next, referring to FIG. 3, the collation unit 12 shown in FIG. 1 uses the feature data stored in the storage unit 3 by the feature expression unit 11 to localize feature points between the query image and the search target database image. An operation of comparing descriptors and outputting an image matching result will be described. FIG. 3 compares the local descriptor of the feature location in the query image with the local descriptor of the feature location in the search target database image, and votes on the search target database image having the closest feature location. FIG. 5 is a flowchart showing an operation of specifying an image in the search target database having many feature locations that are closest to the feature location of the query image, and outputting the voting result ranking of this image and each search target database image as an image matching result. is there.

  First, the collation unit 12 reads out the feature data in which the local descriptor of the feature location extracted from each of the query image and the search target database image is stored from the storage unit 3 (step S11). And the collation part 12 compares the local descriptor group of a query image with the local descriptor group of the image in a search object database among the read feature data. In the comparison process, first, the local descriptor of the feature portion of the query image is expressed as a vector to obtain a query feature vector. This process is performed for all the feature portions of the query image, and a query feature vector group is calculated. Similarly, the local descriptors of all feature locations of the image in the search target database are expressed as vectors, and a search target feature vector group is calculated.

Next, an inter-vector distance with each vector of the search target feature vector group is calculated with respect to the feature vector of the m-th feature location of the query image (step S12), and the search target with the smallest inter-vector distance is calculated. A feature vector is searched, and a vote is performed for the image in the search target database having the search target feature vector (step S13). The inter-vector distance represents the likelihood of the similarity of the feature portion, and the two target n-order vectors x = (x ₁ , x ₂ ,..., X _n ) ^T , y = (y ₁ , y ₂ ,..., y _n ) The Euclidean distance d _{Em of} ^T can be obtained by equation (1).

As a method for calculating the inter-vector distance, other known methods such as Manhattan distance may be used. At the time of voting, the score weighting unit 121 uses values such as the inter-vector distance obtained above and the feature strength (Hessian value) of the feature location calculated by the feature expression unit as an index of the probability of the feature location association. It may be used to calculate the score value. This certainty is inversely proportional to the distance between vectors and proportional to the Hessian value. For example, at the m-th feature location in the query image, the reciprocal of the value obtained by adding the inter-vector distance value and the offset value as in equation (2) is voted as the score value S _m . The offset value is an adjustment value that is set because there is no reciprocal when the inter-vector distance value is 0. Therefore, it is preferable to set a small value that does not affect voting. For example, 0.0001 is set. When the Hessian value is used as an index, the Hessian value itself of the characteristic portion in the corresponding image in the search target database may be used as the score value. This process is performed on the characteristic portions of all query images.

  Finally, as the vote counting process, the normalization processing unit 122 normalizes the total score value in all the search target database images according to the total number of feature locations of each image. This normalization is to find out how many feature locations in the image in each search target database correspond to the feature locations in the query image. There is an effect of improving the certainty of this association. Subsequently, an image having the maximum total score value is obtained or ranked by the total score value and output as a matching result (step S14).

  As described above, at the time of matching, after matching the query image and the feature portion of the image group in the search target database, the weighting at the time of voting is performed by performing voting with the probability of matching as a weight. Compared with the method, it is possible to reduce the influence when a mishandling occurs.

<Second Embodiment>
Next, an image matching device according to a second embodiment of the present invention will be described with reference to FIG. In the first embodiment, in the voting process at the time of matching, the nearest feature point for each feature vector of the query image is obtained from the search target feature vectors, and the score value is calculated from the inter-vector distance value between the corresponding vectors. Voted. In the second embodiment or later, for each feature vector of the query image in the voting process, it is possible to obtain a plurality of similar images by using not only the nearest neighbor but also vector distance information after the second neighborhood, An image matching apparatus capable of improving accuracy will be described.

First, in the voting process at the time of matching after the feature data is read (step S21), each vector of the search target feature vector group and the distance between the vectors are calculated with respect to the feature vector of the feature portion of the query image (step S22). ), The ranking of search target feature vectors is generated in ascending order of the distance between the vectors. Next, in this ranking, voting is performed on images in the search target database having search target feature vectors corresponding to the preset number K of voting targets (step S23). For example, when the number of votes K is set to 5, for each feature location of the query image, a vote is given to the feature location from the nearest neighbor to the fifth neighborhood among the feature locations of the image in the search subject database. At the time of voting, if the distance between vectors in the k-th neighborhood (1 ≦ k ≦ 5) in the m-th feature location in the query image is d _Emk , the score value S _mk is the distance value between vectors as shown in equation (3). Are calculated as the reciprocal of the value obtained by adding the offset value and the offset value, and each of the images in the search target database is voted. Also in the second embodiment, the score weighting unit 121 uses the distance between vectors calculated above and the feature strength of the feature location calculated by the feature representation unit (Hessian) as an index of the probability of the feature location association. The score value may be calculated using a value such as (value). This voting process is performed on the characteristic portions of all the query images.

  Finally, as the vote counting process, the normalization processing unit 122 normalizes the total score value in all the search target database images according to the total number of feature locations of each image. Thereafter, an image having the maximum total score value is obtained or ranked by the total score value and output as a matching result (step S24).

  As described above, at the time of matching, after associating the feature location of the image group in the search target database corresponding to the feature location of the query image up to the vicinity of K, the inter-vector distance value is weighted for each feature location of the query image. By implementing up to the vicinity of K, the accuracy can be improved as compared with the first embodiment. In particular, when the search target database contains many feature parts similar to the feature parts of the query image, voting scores can be given so that there is no difference in weighting between the neighborhoods where there is no significant difference between vectors. The accuracy can be improved.

<Third Embodiment>
Next, an image matching apparatus according to a third embodiment of the present invention will be described with reference to FIG. In the second embodiment, the number of neighbors K to be voted for is designated in advance and voted. In the third embodiment, an image matching apparatus that realizes high-speed and high-accuracy image matching by automatically quitting voting when a significant difference occurs in voting scores between neighbors will be described.

First, in the voting process at the time of matching after the feature data is read (step S31), each vector of the search target feature vector group and the inter-vector distance are calculated with respect to the feature vector of the mth feature location of the query image. (Step S32), rankings of images in the search target database are generated in ascending order of the vector distance. Next, the voting process is performed. When the threshold coefficient set in advance is R, whether or not the voting process is performed depends on whether or not the voting in the vicinity of l satisfies the expression (4) as a discriminant (step S33). decide. For example, R is set to 0.5. Note that the larger the value of R, the smaller the number of voting objects, and the smaller the value of R, the larger the number of voting objects. If this discriminant is not satisfied (= 0), a voting process is performed, and the discriminant in the neighborhood of (l + 1), which is the next neighborhood, is applied. If the discriminant is satisfied, the voting process is not performed, and the process proceeds to the process for the feature vector of the (m + 1) th feature location of the query image.

  In this way, while changing the number of votes for each feature location of the query image, voting is performed on the images in the search target database corresponding to all the feature locations of the query image (step S34). The calculation method of each vote score is the same as that of the second embodiment.

  Finally, as in the case of the second embodiment, as a vote counting process, the normalization processing unit 122 normalizes the total score value of all the images in the search target database according to the total number of feature locations of each image. Thereafter, an image having the maximum total score value is obtained or ranked by the total score value and output as a matching result (step S35).

  As described above, at the time of matching, after associating the query image with the feature portion of the image group in the search target database, the voting is performed on a plurality of candidates by using the distances between the vectors as the weights. Compared with the embodiment, especially when the search target database includes many feature parts similar to the feature parts of the query image, it is possible to make a voting score so that there is no difference in weighting between the neighborhoods having no significant difference. The accuracy can be improved. Furthermore, since the number of votes to be voted is automatically assigned and executed while confirming the difference between the neighborhoods for each feature portion of the query image without fixing the number of votes, the second embodiment can prevent useless votes. Compared with, high-speed and high-precision image matching can be realized.

  In all of the embodiments, normalization according to the image size in the feature representation unit in FIG. 1 and normalization according to the number of feature locations of the total score value of each search target database image in the matching unit are performed. However, it is not necessary to carry out.

  As described above, the voting score value is weighted at the time of image matching, and in order to improve the certainty of the voting score value, the feature extraction unit 11 performs the normalization process according to the image size and performs feature extraction. To do. In addition, the total score value is normalized according to the number of feature portions of each image group in the search target database after the weighting in the matching unit 12. As a result, the accuracy of image matching can be improved.

  1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed to execute image matching processing. You may go. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  Although the present invention has been specifically described above based on the embodiments, the description of the above embodiments is for explaining the present invention, and limits the invention described in the claims, or It should not be construed as reducing the scope. Further, the configuration of each part of the present invention is not limited to the above-described embodiment, and various modifications can be made within the technical scope described in the claims.

  In order to solve the problem that image matching cannot be performed correctly due to a decrease in the number of votes of images suitable as a result of image matching when a large number of misassociations occur, the influence of misassociation is reduced in feature-based image matching. Can be applied to indispensable uses.

  DESCRIPTION OF SYMBOLS 1 ... Image matching part, 11 ... Feature expression part, 111 ... Normalization process part, 112 ... Feature extraction part, 12 ... Collation part, 121 ... Score weighting part, 122 ... Normalization processing unit, 2 ... Image input unit, 3 ... Storage unit, 4 ... Input unit, 5 ... Display unit

Claims (6)

A feature representation means for calculating a feature amount for each feature location extracted from each of a query image and a stored image stored in advance in the search target database;
A score is calculated from the feature amount calculated by the feature expression means between the query image and the saved image, and image matching is performed based on a score vote result for the saved image corresponding to the feature location of the query image. An image matching device comprising a matching means for performing,
The verification means includes
The query image calculated by calculating the inter-vector distance between the vector generated from the feature quantity of the feature location of the query image and the vector generated from the feature quantity of the feature location of the stored image, and arranged in ascending order of the inter-vector distance. A list of the stored images for each feature point of
The inter-vector distance between the vector generated from the feature quantity of the feature location of the query image and the vector generated from the feature quantity of the feature location of the i-th stored image in the list (i is a natural number), and the query image Up to the smallest i-th occurrence of a significant difference between the vector generated from the feature quantity of the feature location and the vector distance between the vector generated from the feature quantity of the feature location of the (i + 1) -th stored image in the list Identify saved images of
Vote for a score indicating the probability of correspondence between the feature location of the query image and the feature location of the saved image with respect to the smallest i-th saved image specified for each feature location of the query image, and the score An image matching apparatus that performs the image matching based on a voting result to obtain the stored image that matches the query image.   The image matching apparatus according to claim 1, wherein the matching unit performs the image matching based on a value obtained by normalizing the score voting result by the number of feature locations in the stored image in the search target database. .   The image matching apparatus according to claim 1, wherein the feature expression unit extracts the feature portion after normalizing the query image and the saved image according to an image size. Said score indicating the likelihood of association used in collating means, with respect to the stored image of the search target database is a score based on the Hessian value of feature points corresponding to the feature points of the query image image matching device according to any one of claims 1, wherein 3. The score indicating the probability of correspondence used in the matching unit is the distance between vectors between a vector generated from the feature quantity of the feature location of the query image and a vector generated from the feature quantity of the feature location of the saved image. image matching device according to any one of claims 1 to 3, characterized in that the score based. Image matching program for functioning as an image matching device according to the computer in any one of claims 1-5.

Images