JP2013069149A - Image similarity determination device, image similarity determination method and image similarity determination program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To determine similarity by collectively determining geometrical correspondence between images using a series of a plurality of feature point groups calculating local feature quantity and making the images correspond to each other with higher calculation efficiency and with higher accuracy.SOLUTION: The image similarity determination device for determining similarity of a plurality of images includes: feature point detection means for detecting a feature point group from each of the plurality of images; feature point sampling means for performing prescribed sampling from the feature point group detected from each of the plurality of images to prepare and output series information of feature points provided with an order relation for each of the plurality of images; feature vector generation means for feature-vectorizing the series information of the feature points provided with the order relation and outputting the series information for each of the plurality of images; linear transformation means for performing linear transformation for each feature vector of the plurality of images and outputting each feature vector; and similarity determination means for determining the similarity of the plurality of images using the linear transformation result.

Description

  The present invention relates to an image similarity determination device, an image similarity determination method, and an image similarity determination program.

  An image is an array pattern of pixels, and determination of image similarity is a problem of pattern classification. The main prior arts of pattern classification methods are roughly classified into the following two in Non-Patent Document 1. The first method is a template matching method, in which a model stored in advance with respect to a known pattern and an input pattern are directly compared on an image, and the most consistent pattern is selected. is there. The second method is pattern classification in a feature space, in which a certain amount or feature is extracted from the pattern, and the pattern is classified based on these.

  Here, for each of the first method and the second method, it is considered to detect correspondence between images when an object included in the image rotates, moves, or deforms. In the case of the first method, (1) preparing a template including all variations, (2) rotating / moving / deforming the template and comparing it with the target image may be considered. However, the method (1) cannot be used unless it is known what kind of deformation is possible. Further, the method (2) requires a large amount of calculation.

  On the other hand, the second method is a method for extracting local features as compared with the first method, and conventionally used features include a gradient histogram feature and a moment feature. SIFT (Scale-Invariant Feature Transform), which is an algorithm that performs matching based on the gradient histogram feature and the direction histogram of the luminance gradient, can take correspondence between images regardless of rotation, movement, and enlargement / reduction.

  However, deformation other than enlargement / reduction cannot be handled except when the degree of deformation is very small. The moment feature is also a feature quantity that is invariant to rotation, movement, and enlargement / reduction with respect to the geometric feature of the shape, but deformation other than enlargement / reduction cannot be handled except when the degree of deformation is very small. However, since these are local features, the feature amount of a plurality of local regions can be obtained, and if it is possible to determine which local regions correspond to each other between the reference image and the target image, image rotation, movement, and scaling Can be estimated.

  However, when estimating such a transformation matrix, only the coordinate values of the corresponding feature point group are used. The feature amount of each feature point is only used for associating the feature points, and is not reflected when obtaining a geometrical relationship such as rotation, movement, and enlargement / reduction of an image. Therefore, even when an error occurs at the stage of associating feature points, there is no choice but to obtain a geometric relationship based on the error.

  Conventionally, DP matching (dynamic programming) and HMM (Hidden Markov Model) are well known as methods for associating feature quantity sequences with each other. If these methods are used to estimate the correspondence between sequences using both the feature value and the coordinate sequences as inputs, the accuracy of the correspondence may be improved rather than obtaining a geometrical relationship only from the coordinate values. However, even in that case, a conversion matrix for rotation, movement, and enlargement / reduction of the image must be obtained separately.

Image Recognition and Image Understanding Keigaku Publishing (Chapter 2) p. 16-p. 17 Hiroyoshi Fujiyoshi. "Gradient-based feature extraction-SIFT and HOG-", Information Processing Society of Japan Research Report CVIM 160, pp. 211-224, 2007

  As described above, according to the prior art, when an image including the same object photographed by a different method is detected or an image including a similar object is detected, the similarity between images is determined. There is a problem that it is difficult to do. This problem is a problem that occurs with the rotation, movement, and deformation of an object included in an image. In addition, there is a problem that occurs due to uncertainties in the feature values obtained from individual local regions in the image. Even if the similarity between images is determined according to the first method described above, the similarity between the object included in the image and the reference pattern can be obtained, but the relative shape of the objects included in the image can be calculated. There has been no method for calculating the degree of similarity having a correlation with the similarity.

  When the similarity is calculated by the second method described above, first, a plurality of corresponding points are detected between images using local feature amounts, and a geometric relationship between the target image and the reference pattern ( The rotation amount, the parallel movement amount, the transformation matrix of deformation, etc.) need to be obtained separately using only the coordinates of the corresponding points. That is, the prior art performs a two-step process of performing geometric correspondence between images based on the coordinates of a plurality of feature points associated with each other, after associating local regions based on local feature amounts. It is necessary to perform processing.

  The present invention has been made in view of such circumstances, and obtains a geometric correspondence between images collectively using a series of a plurality of feature point groups for which local feature amounts have been calculated, and is further calculated. An object of the present invention is to provide an image similarity determination device, an image similarity determination method, and an image similarity determination program that are efficient and that can perform similarity determination by associating images with higher accuracy.

  The present invention is an image similarity determination device that determines the similarity of a plurality of images, and includes feature point detection means for detecting a feature point group from each of the plurality of images, and feature points detected from each of the plurality of images. A feature point sampling unit that performs predetermined sampling from the group and generates and outputs sequence information of feature points having an order relationship for each of the plurality of images, and the order relationship for each of the plurality of images. Feature vector generation means for outputting the feature point series information obtained by converting it into feature vectors, linear conversion means for performing linear conversion on each of the feature vectors of the plurality of images, and using the result of the linear conversion And similarity determining means for determining the similarity of the plurality of images.

  According to the present invention, the feature point series information is information having an order relationship with respect to a feature point on the contour of an object included in the image, a feature point inside the object, or a combination thereof. Features.

  The present invention is characterized in that the feature vector generation means uses the position information obtained for the sampled feature points, the color information of an object included in the image, or a combination thereof as a feature vector.

  According to the present invention, the position information includes coordinates of feature points in Euclidean space, polar coordinates of feature points, coordinates in a complex plane, difference information of position vectors between feature points, or contours of objects included in the image. It is characterized by using a tangent vector or a combination thereof.

  The present invention is characterized in that the linear conversion means performs linear conversion using Procrustes analysis.

  The present invention is characterized in that the similarity determination means determines the similarity by using a Procrutests distance, or a combination of the Procruces distance and other feature quantities.

  The present invention is an image similarity determination method for determining the similarity of a plurality of images, a feature point detecting step for detecting a feature point group from each of the plurality of images, and a feature point detected from each of the plurality of images A feature point sampling step of performing a predetermined sampling from the group and generating and outputting sequence information of feature points having an order relationship for each of the plurality of images, and the order relationship for each of the plurality of images. A feature vector generation step for outputting the feature point series information obtained by converting the feature point into a feature vector, a linear conversion step for performing a linear conversion on each of the feature vectors of the plurality of images, and a result of the linear conversion And a similarity determination step for determining the similarity of the plurality of images.

  The present invention is characterized by causing a computer to function as an image similarity determination device.

  According to the present invention, by taking correspondence between images based on correspondence of a series of feature points included in an image, an object included in the image is rotated, moved, deformed, or a feature included in the image. Even in the case where the positional relationship between the points is deviated, the degree of similarity between the objects included in the image or between the object included in the image and the reference pattern prepared in advance is calculated, and based on this, the image including the similar object It is possible to search for images and to arrange images in a similar order.

It is a block diagram which shows the structure of one Embodiment of this invention. It is explanatory drawing which shows an example of a target image. It is explanatory drawing which shows an example of an image processing result. It is explanatory drawing which shows an example of the output format of feature point information. It is explanatory drawing which shows the result of having visualized the similarity of an image.

  Hereinafter, an image similarity determination apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numeral 1 denotes an image similarity determination device constituted by a computer device. Reference numeral 11 indicates that two images for which the degree of similarity between them is to be determined, or one image that is a reference image and one or more images for which the degree of similarity is to be determined are input from a camera or the like. It is an image input part which inputs from. Reference numeral 12 denotes a feature point detection unit that detects feature points from the image input by the image input unit 11 and outputs candidate information of feature points. Reference numeral 13 denotes a feature point sampling unit that performs sampling according to a predetermined method from feature point candidate information, and generates and outputs a sequence having an order relationship based on the predetermined order relationship. It is.

  Reference numeral 14 denotes a feature vector generation unit that extracts a predetermined feature amount from a sampled sequence of feature points and outputs it as a feature amount vector (sequence). Reference numeral 15 denotes a linear conversion unit that performs linear conversion on the feature vector. With the processing so far, the feature vectors of each of the two images that are input in the image input unit 11 and for which the similarity between them is to be determined, or one or more images for which the similarity between the reference image and the similarity is to be determined are determined. As a result, linear transformation using them is performed. Reference numeral 16 denotes a similarity determination unit that outputs the result of linear transformation as a numerical value of similarity.

  Next, the image input unit 11 shown in FIG. 1 will be described. The images input by the image input unit 11 are two images for which the degree of similarity between them is to be determined, or one or more images for which the degree of similarity between one image to be a reference image and the target image is to be determined. Although what kind of thing may be sufficient, it demonstrates as what is the image which image | photographed the object with a certain amount of deformation | transformation here. As an example of such an image, a product image on a website can be applied as shown in FIG. The size of the image and the size of the contained products may be different as shown in FIG.

  Next, the feature point detection unit 12 shown in FIG. 1 will be described. The feature point detection unit 12 detects an edge or an outline or corner of an object based on the edge as the target feature point. For corner detection, there are existing methods such as a Harris detector and a method using eigenvalues, any of which may be used. Also, other generally known feature detection methods may be used, and the SIFT features mentioned above in the prior art can also be used. In the case of the image of FIG. 2A, when edge detection is performed, an outline of an object (clothes) is obtained as shown in FIG. The feature point detection unit 12 outputs the coordinates of the points on the contour or the coordinates and the feature amount to the feature point sampling unit together with the input image. Further, corner detection or the like may be performed as a feature point, and its coordinates may be output together. It can be expected that the accuracy of the subsequent similarity determination will be improved if a point with a lot of information is detected as a feature point.

  Next, the feature point sampling unit 13 shown in FIG. 1 will be described. The feature point sampling unit 13 samples the feature point group obtained by the feature point detection unit 12 with order. When the example of FIG. 3 (A) is input, for example, sampling is performed as shown in FIG. 3 (B) so that the distances on the contour are close to equal intervals. At this time, the ordering method is determined in advance, for example, by providing ordering in the counterclockwise direction from the upper left of the image. Further, when not only the contour but also the corner is output from the feature point detection unit 12, these are also sampled and combined with the feature points sampled from the contour, and the coordinates, or the coordinates and the feature amount are input to the image input unit 11 Are output to the feature vector generation unit 14 together with the image input at. At this time, priority can be given to the point with more information as a feature point, and it can be expected that accuracy of similarity determination in the subsequent stage is improved. In the case of the example in FIG. 3B, the sampled points are indicated by circles on the contour.

  Next, the feature vector generation unit 14 shown in FIG. 1 will be described. The feature vector generation unit 14 converts the sampled and ordered feature point information output from the feature point sampling unit 13 into a feature vector. The row component of the feature vector is a feature amount output from the feature point sampling unit 13 or a feature amount detected again from the feature point coordinates and image output from the feature vector generation unit 14, and is n-dimensional. If there is a feature amount, there are n rows. As this feature amount, various feature amounts conventionally used can be used in combination, and the moment feature described above in comparison with the conventional method can be applied.

  Also, in the position information obtained for the feature points, the coordinates of the feature points in the Euclidean space, the polar coordinates of the feature points, the coordinates in the complex plane, the difference information of the position vectors between the feature points, or the objects included in the input image A feature amount using a tangent vector of the contours of the two or a combination thereof may be used. For example, if only x and y coordinates are used, n = 2. When using coordinate values and color (r, g, b) components, n = 2 + 3 = 5. The column components of the feature vector are m columns if there are m points sampled in order. The output format is, for example, text data containing numerical values as shown in FIG.

  Next, the linear conversion unit 15 shown in FIG. 1 will be described. Procrustes analysis is a method of comparing time series data by performing linear transformation. As described above, methods for comparing time series data include HMM and DP matching. Procrustes analysis does not require a model like HMM. Like DP matching, it is close to matching that takes into account the expansion and contraction between time series, but unlike DP matching, there is a matrix of linear transformations that says “What kind of linear transformation can be best matched as a result?” Is output. For this reason, by ordering the feature quantities in the image and considering this as a time series, matching is performed from the linear transformation matrix obtained as a result of the matching, not only the similarity between the images, but also one of the images. It is possible to know how the image is deformed and corresponds to the other image, its rotation, movement, scale and linear deformation.

Now, it is assumed that two feature vectors of X and Y are obtained from two images whose similarity is to be calculated. However, it is assumed that the position is normalized by performing movement around the center of gravity of each element.

によって求めることができる。 At this time, the Procruces distance when one feature vector is rotated and scaled so that the two feature vectors overlap most is

Can be obtained.

となる。 Hereinafter, how to obtain this distance will be described. The distance between X and Y feature vectors is

It becomes.

となる。 When Y is rotated by the rotation matrix R

It becomes.

R that minimizes this is obtained by maximizing the last term.

である。 Also, when Y is scaled by σ, σ that minimizes Δ ² (X, σYR) is

It is.

  In the case of such R and σ, the expression (3) becomes the expression (2).

  In the processing of the feature vector generation unit 14 and the linear conversion unit 15, the coordinate expression constituting the feature vector and the distance between the feature vectors are expressed by polar coordinate expression, direction data (angle data and chain code), or complex expression thereof. It may be used. The Procrustes analysis is described in the document "Online character recognition method using Procrustes analysis and vector correlation, Miho Kikuchi, Kazutoshi Tagai, Teppei Takano, Hiroshi Kaneko, Journal of the Institute of Image Electronics Engineers of Japan, Vol. 36, No. 5, 2007." ing.

  Next, the similarity determination unit 16 shown in FIG. 1 will be described. The similarity determination unit 16 calculates the similarity by integrating the value after linear conversion output from the linear conversion unit 15 and other feature amounts based on a predetermined formula, and outputs this To do.

  FIG. 5 is a result of visualizing the similarity of product images on a plane having two similarities of x and y as the similarity between two patterns with different parameters of the Procrutes distance. In order to show the effectiveness of the present invention, the result of mapping on the same plane using two similarities with two patterns with different moment feature parameters as two axes is shown. Since the distance scales are different, it does not make sense to compare absolute distances on a plane in the two methods. However, as the order relation, positions on the plane of products that are considered to be intuitively similar in the moment feature are mixed, but in the present invention, shapes that are considered to be intuitively similar are arranged in order.

  As described above, the feature point sampling unit 13 can apply the linear transformation in the linear transformation unit 15 by appropriately grouping the feature point groups included in the image. It is possible to determine the similarity of images so as to collectively obtain a linear transformation matrix that performs such correspondence in between.

  As described above, when obtaining the similarity between two images, assuming that the image has rotation and enlargement / reduction, a feature vector (two-dimensional matrix) is obtained from each of the two images. The feature vector is rotated and scaled to obtain the most similar transformation (rotation and scale), and the similarity between feature vectors based on the most similar transformation is the similarity between the two images. Thereby, when determining the similarity of the image, even if the object in the image has rotation, movement, deformation, or a combination thereof, by determining the similarity of the object based on the shape of the object, Search for images that contain the same or similar objects, arrange many product images in a similar order, and identify images taken from the same scene from different directions using the objects in the images as clues become.

  The program for realizing the function of the processing unit in FIG. 1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer system and executed to determine the image similarity. Processing may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  By using a series of multiple feature point groups for which local feature quantities have been calculated, geometrical correspondence between images is obtained in a lump, and images are associated with each other more efficiently and more accurately. It can be applied to applications where it is indispensable to perform similarity determination.

  DESCRIPTION OF SYMBOLS 1 ... Image similarity determination apparatus, 11 ... Image input part, 12 ... Feature point detection part, 13 ... Feature point sampling part, 14 ... Feature vector generation part, 15 ... Linear Conversion unit, 16 ... similarity determination unit

Claims (8)

An image similarity determination device for determining the similarity of a plurality of images,
Feature point detecting means for detecting a feature point group from each of a plurality of images;
Feature point sampling means for performing predetermined sampling from the feature point groups detected from each of the plurality of images and generating and outputting series information of the feature points having an order relationship for each of the plurality of images;
For each of the plurality of images, feature vector generation means for converting the sequence information of the feature points having the order relation into a feature vector and outputting the feature vector;
Linear conversion means for performing linear conversion on each of the feature vectors of the plurality of images and outputting;
An image similarity determination device comprising: similarity determination means for determining the similarity of the plurality of images using a result of the linear transformation.   The feature point series information is information having an order relationship with respect to a feature point on an outline of an object included in the image, a feature point inside the object, or a combination thereof. Item 2. The image similarity determination device according to Item 1.   The image according to claim 1, wherein the feature vector generation unit uses the position information obtained for the sampled feature points, the color information of an object included in the image, or a combination thereof as a feature vector. Similarity determination device.   The position information includes coordinates of feature points in Euclidean space, polar coordinates of feature points, coordinates in a complex plane, difference information of position vectors between feature points, or contact vectors of contours of objects included in the image, or The image similarity determination apparatus according to claim 3, wherein a combination thereof is used.   The image similarity determination apparatus according to claim 1, wherein the linear conversion unit performs linear conversion by using Procruses analysis.   The image similarity determination apparatus according to claim 1, wherein the similarity determination unit determines the similarity using a Procrutests distance, or a combination of the Procruces distance and other feature quantities. An image similarity determination method for determining the similarity of a plurality of images,
A feature point detection step of detecting a feature point group from each of a plurality of images;
A feature point sampling step of performing a predetermined sampling from the feature point group detected from each of the plurality of images and generating and outputting series information of the feature points having an order relationship for each of the plurality of images;
For each of the plurality of images, a feature vector generation step for outputting the sequence information of the feature points having the order relation as a feature vector; and
A linear transformation step of performing a linear transformation on each of the feature vectors of the plurality of images and outputting;
A similarity determination step of determining a similarity of the plurality of images using a result of the linear transformation.   An image similarity determination program for causing a computer to function as the image similarity determination apparatus according to claim 1.

Images