JP2003109009A - Method and instrument for measuring similarity of image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To develop a method for generally, accurately and efficiently measuring a similarity of various images including a color image and further to rank similarities by developing a method for quantitatively measuring the similarities. SOLUTION: A method and device is provided for measuring a sample image and a measurement target image similar to the sample image with two-dimensional dynamic programming algorithm by using characteristic quantity in each image. Dijkstra's algorithm is also adopted to realize fast measuring.

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention relates to a technique for quantitatively comparing two or more images. In particular, it is a technique relating to a method of quantitatively measuring how similar a certain sample image and at least one measurement target image have.

[0002]

2. Description of the Related Art As computer image processing has become commonplace, one problem is how to measure the similarity of matrix-displayed images. From the comparison that the technique is similar in visual impression, various applications are conceivable such as strict image comparison, that is, detection when some change occurs in the image. Especially in the former comparison, quantitative measurement by a computer is difficult, and the conventional technique does not provide an appropriate method.

For example, in the comparison of color images, there is a method of displaying the features of the images in a histogram and comparing information regarding color histograms. However, in this method, since the color arrangement information is lost, even if the images themselves are completely different, if the overall colors are similar, they are detected as similar images.

In order to solve this, it is possible to use color information and color arrangement information, but the amount of information becomes enormous in the conventional method, and it is impossible to perform practical measurement. Furthermore, even if the time cost is ignored, there is no room for applying information other than color in the conventional method, and it is difficult to say that this is a general-purpose method for measuring the similarity between images from various angles.

[0005]

SUMMARY OF THE INVENTION Therefore, the present invention is intended to solve the problems of the above-mentioned prior art, and is versatile, accurate, and efficient for various images including color images. To develop a method of measuring similarities. Furthermore, the development of a method for quantitatively measuring the similarity enables even ranking of the similarity.

[0006]

In order to achieve the above object, the image similarity measuring method of the present invention has the following features. That is, a method of quantitatively measuring the similarity between images displayed in a matrix according to the present invention, and first, for a sample image and at least one measurement target image, the sample image and all measurement target images Is divided into a plurality of blocks based on at least either the number of pixels or the area.

Then, for at least one of the blocks, the feature amount of each block is determined by a predetermined method, and the sample image and each measurement target image are determined from the feature amount using a two-dimensional dynamic programming matching method. Quantitatively calculate the similarity with. The present invention is characterized in that the dynamic programming conventionally specified for matching one-dimensional data such as a voice signal is further extended to two dimensions and applied to the above blocks.

Further, in the above two-dimensional dynamic programming matching method, the Dijkstra algorithm can be used to solve the shortest path problem of the matching distance in order to further reduce the calculation cost.

In particular, in the configuration in which the sample image and each measurement target image are color images, the feature amount is the chromaticity of color, and the chromaticity most frequently used in each block is the feature of the block. It may be determined as the amount.

The present invention can also be provided as an apparatus for realizing the above method. That is, the following means are provided. (1) Image input means for inputting a certain sample image and at least one measurement target image, (2) the sample image and all measurement target images are divided into a plurality of blocks based on at least either the number of pixels or the area. Image block forming means for dividing, (3) a feature amount determining means for determining a feature amount of each block in at least one of the blocks by a predetermined method, (4) a two-dimensional dynamic programming matching method from the feature amount make use of,
The similarity calculation means quantitatively calculates the similarity between the sample image and each measurement target image, and (5) each of the similarity output means that outputs the similarity of each measurement target image.

[0011]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described based on examples shown in the drawings. The embodiment can be appropriately modified without departing from the gist of the present invention. An example of measuring the similarity of a plurality of images in a color image will be shown below, but as described above, various elements can be used as the feature amount in the present invention, so the present invention is not necessarily limited to the case where chromaticity is the feature amount. .

First, a two-dimensional dynamic programming (DP) matching algorithm will be described. Dynamic programming is generally used for matching 1D data such as audio signals. Furthermore, conventional D
A two-dimensional DP matching technique that extends P matching is also known. The present invention is characterized in that this is used to measure the similarity of images. Specifically, with respect to the matrix for which the feature amount is obtained, the similarity between the matrices is calculated.

The target image is an image that can be displayed in a matrix, and is, for example, image data for a computer that is composed of a large number of pixels. First, the image is divided into one block for each pixel or each of a plurality of pixels. The term "block" is used here because not only one pixel is treated as a unit but also a plurality of pixels may be treated as a matrix in a two-dimensional DP matching algorithm to be described below. Therefore, one pixel is treated as a matrix. Does not prevent

In the case of a color image, each pixel is given a chromaticity represented by a numerical value. These are data representing each of red, green, and blue color components in 256 gradations. When a plurality of pixels is used as one block, for example, the chromaticity that is most frequently used among the pixels forming the block can be given as the representative chromaticity of the block. In this way, the chromaticity that is the characteristic amount is determined for each block.

Now consider a matrix of size n × n. The matrix is shown in FIG. Then, S is a sample parameter matrix called a sample matrix, and I is an object feature parameter matrix called an object matrix. d (I, S) is
Let us denote the matching distance between the object and the sample matrix. Matching distance d (I, S) (1
1 ...) to obtain matrices I (12) and S
(13) is divided into n column vectors. Iα = (Iα ₁ a column vector of the object matrix I, Iα _2, ···, I
α _n ), and the column vector of the sample matrix S is Sβ
= (Sβ ₁ , Sβ ₂ , ..., Sβ _n ).

Equation 1 defines the element similarity distance between Iα _i and Sβ _j .

[Formula 1] Where i = 1, 2, ..., n and j = 1, 2, ..., n. I
For all pairs of element similarity distances that belong to α and Sβ, the element similarity distance matrix (1
0) can be obtained. d (Iα _i , Sβ _j ) can also be written as d (i, j) (11 ...) In the matrix of FIG.

When calculating the column vector matching distance d _row (α, β) between I α (12) and S β (13),
d (i, j) can be regarded as a vertex. Directed arc (2
0) (21) (22) is used, d (i-1, j) (23) to d (i, j) (24) are changed to d (i-1, j) as shown in FIG.
-1) (25) to d (i, j) (24), and d (i, j-
1) From (26) to d (i, j) (24), 2 vertices each
Link one. Here, the three arcs are multiplied by the weighting values of w ₁ , w ₂ and w ₃ .

Each d (i, j) is arc-linked by the process described above when -R ≤ ij ≤ R. Here, R is the band of the matching window. As shown in FIG. 3, the initial vertex is d (1, 1) (31) and the terminal vertex is d (n, n) (3).
The directed graph G (30) of 2) can be drawn. d
The length of the shortest path from (1, 1) (31) to d (n, n) (32) is represented by Pmin. This can be calculated using the Dijkstra algorithm of graph theory.

The Dijkstra algorithm is known as an algorithm for finding the shortest path. The column vector matching distance d _row (α, β) is defined by Equation 2.

[Formula 2] The column vector matching distance is a basic formula for calculating a matching distance between two feature amount matrices.

Next, the matching distance in the feature amount matrix is calculated. A two-dimensional DP matching algorithm is realized in two stages. First, the column vector of the object matrix is compared with each column vector of the sample matrix to obtain similar column vectors. Next, the matching distance between the two characteristic parameter matrices is calculated using the similar column vector and the standard column vector of the sample matrix.

The column vector Iα of the object matrix I is compared with each column vector Sβ of the sample matrix S by DP matching. Where α = 1, 2, ・ ・ ・,
m. The minimum column vector matching distance dmin (α, k) is determined from [d _row (α, β) | β = 1, 2, ..., M]. d
When min (α, k) is smaller than the threshold Tr, Iα is considered to be similar to the column vector Sk.

Using the vector SV [1..m] shown in equation 3,
Record the similar column vector number for Iα.

[Formula 3] For vector SV, if α ₁ and α ₂ are α ₁ ≠ α ₂ and SV [α ₁ ] ≠ 0

[Formula 4]

In other words, the object matrix I
Of two different column vectors in the same matrix are not similar to the same column vector in the sample matrix S. FIG. 4 shows a method of obtaining the similar column vector SV = (0, 0, 1, 2) (40) as an example. In this example, the column vectors (42a) (42b) (42c) of the object matrix (41)
There are four (42d) and sample matrices (4
3) Each column vector (44a) (44b) (44c)
(44d).

By calculating the column vector matching distance, the column vector (1, 2, 1, 1, 8) (42c) becomes the first column vector (1, 1, 2, 2 of the sample matrix (43).
Similar to (1,8) (44a), the column vector (5, 6, 7, 7, 7)
(42d) is the second column vector (5,
It can be seen that it is similar to 6, 7, 8, 6) (44b).
The results are represented as similar column vectors (45) SV [3] = 1 (46c) and SV [4] = 2 (46d).

The column vectors (3, 4, 6, 4, 3) (42a) and
(4, 3, 0, 3, 4) (42b) is dissimilar to any column vector of the sample matrix (43). Therefore, the similar column vector (45), SV [1] = 0 (46a)
And SV [2] = 0 (46b).

In this way, the object matrix I (41) can be represented by the similar column vector SV (45). If the number "0" is greater than the percentage Tp in the similar column vector SV (45), the object matrix I
It can be concluded that (45) is not similar to the sample matrix S (43), ie g (I, S) = 1.

In the present invention, an object matrix I is formed for each image whose similarity is to be measured, and a sample matrix is formed for the sample image, and a similar column vector is calculated for the image to be measured. Further, the matching distance is calculated using the DP matching method as follows.

The feature matrix matching distance is a vector when the number "0" is smaller than the defined percentage Tp.
(SV [1], SV [2], ・ ・ ・, SV [m]) and (S [1], S [2], ・ ・ ・, S
It is obtained by calculating the vector matching distance between [m]). Where S [v] is the sample matrix S
It is the vth column vector. FIG. 5 is a diagram showing this approach method.

Using the DP matching method described above, the SV
From (S [1], S [2], ..., S [m]), such column vector matching, that is, a directed graph can be drawn. The element matching distance d (SV [u], S [v]) is defined by Equation 5.

[Formula 5] Where u = 1, 2, ..., m and v = 1, 2, ..., m. Object matrix I and sample matrix S
The matching distance g (I, S) between the two can be obtained by calculating the shortest path length.

The outline of the two-dimensional DP matching and Dijkstra algorithm in the present invention has been described above. Here, a method for realizing color content similarity search of images using the above two-dimensional DP matching algorithm will be shown. We propose a new color content expression and similarity search method for color images (color images) based on this expression.

There are various distinguishable colors in the color image, but in most color images,
A few colors, which can be called characteristic colors, dominate.
These characteristic colors are more effective in considering similarities than other colors. Clustering (stratification method) (see References 1 and 2) and threshold method (References 3 and 3).
See 4, 5, 6. ) Etc., several automatic color classification methods have been developed. (Reference 1) Mohan S. Kankanhalli, Babu M. Mehtre
and Jian Kang Wu, "Cluster-based color matching f
or image retrieval ", Pattern Recognition, Vol.29,
No.4, pp.701-708, 1996 (Reference 2) M. Celenk, "A color clustering tec
hnique for image segmentation ", Computer Vision,
Graphics, and Image Processing, Vol.52, pp.145-17
0, 1990 (Reference 3) Y.Liu, S.Guo, S.Oda and S.Ozawa, "A
new bilevel thresholding method based on morpholo
gy and fourth central moment ", The Journalof the
Institute of Image Electronics Engineers of Japan,
Vol.28, N0.4, pp.380-386, 1999 (Reference 4) JR Smith and S. Chang, '(Single co
lor extraction and image query ", IEEE Internation
al Conference on Image Processing, Vol.3, pp.528-5
31, 1995 (Reference 5) H. Yang and S. Ozawa, '(Extraction o
f bibliography information based on the image of b
ook cover '), IEICE Trans. on Information and Syst
em, Vol.E82-D, N0.7, pp.1109-1116, 1999 (Reference 6) Yuehu Liu. A Study on Color Image
Retrieval Method, Ph.D. thesis, University of Kei
o, 2000

The characteristic colors can be obtained by using these methods to construct the characteristic color set f. f is represented by the following Expression 6.

[Formula 6] Ck is a characteristic color here.

To handle the location of the colors, a characteristic color matrix is created that represents the locations of the characteristic colors in the color image. Color image f (x, y) in block m
Divide into × n pieces. B _ij represents a block. Where i =
1, 2, ..., M and j = 1, 2 ,. Each block B _ij can be examined to determine the proportion λCk (i, j) of pixels having the characteristic color Ck. The definition of λCk (i, j) is as in Expression 7.

[Formula 7] Here, s _x × s _y is the size of each block.

After calculating the ratio of each block in the image, the characteristic color matrix F = [FC _k | k = 1, 2, ...,
The μ value of μ] is obtained. FC _k represents a characteristic color matrix with a characteristic color C _k . The element value FC _k [i, j] of the characteristic color matrix is defined by Equation 8.

[Formula 8] Here, T _pre is the set threshold. A color image is represented by using the characteristic color matrix μ thus obtained.

In the present invention, two color images of the sample image and the target image whose similarity is to be measured are represented by S and I, and two factors, that is, color similarity and spatial similarity, are taken into consideration. Measure the similarity between I. The spatial similarity between S and I is expressed based on its characteristic color matrix. These matrices provide color chromaticity and position information. The characteristic color matrix of the sample image S is represented by SCk, and the characteristic color matrix of the object image I is represented by ICk. Where k = 1, 2, ・ ・
・, Μ.

Suppose g (IGk, SGk) is the matching distance between two characteristic color matrices with characteristic color Ck. The spatial similarity SS (I, S) is defined by equation 9.

[Formula 9] Here w _k is a weighting value, which determines the relative importance of the characteristic color C _k . Σμ / k = 1wk = 1. g (ICk,
SCk) can be evaluated using a two-dimensional DP matching algorithm. The matching distance is a degree of dissimilarity. Therefore, the spatial similarity between the object and the sample image can be evaluated by the difference.

The similarity search technique using the two-dimensional DP matching algorithm according to the present invention has been described above. Here, the above-mentioned series of flowcharts are shown in FIG. First, as the first step (50), the column vector matching distance drow (α, β) between Iα and Sβ is calculated.
As the second step (51), the minimum column vector matching distance dmin (α, β) is determined from drow (α, β). In the third step (52), SV [1..m] is used to record the similar column vector number of each Iα. In the fourth stage (53), an element similarity distance matrix (10) of size n × n is obtained for all pairs of element similarity distances belonging to Iα and Sβ. In the fifth step (54), the matching distance g (I, S) between the object matrix I and the sample matrix S
Calculates the shortest path length using the Dijkstra algorithm. After each of these steps, the similarity is measured by the matching distance g (I, S).

In order to improve the speed and accuracy of the similarity image search proposed above, this embodiment is further illustrated in FIG.
A high-speed search is realized by dividing into two steps shown in.
First, the candidate image group (63) is selected from the intersecting state of the color histogram (61) of the object image I (60) and the color histogram (61) of the sample image S (62). Although a database of a plurality of object images I (60) may include many dissimilar images, it is possible to exclude these dissimilar images by evaluating the matching distance between the characteristic color matrices. I can. Then, the process proceeds to the two-dimensional DP matching method (64) according to the present invention.

In order to demonstrate the effect of the present invention, an experiment for measuring image similarity is conducted. This is an experiment to check whether the search result matches well with the similarity judgment made by human.
A multifaceted test was carried out on an experimental database consisting of 120 various color images. The content of the image is a sense of nature, animals, flowers and people.

As an example, the flower image shown in FIG. 8 was used as a sample image. This sample image is 1
Divide into 6 × 12 blocks, and use characteristic color C1 and
C2 was selected. Here, C1 and C2 are 60 to 80 and 182 to 212, respectively, when expressed by pixel hue component values. An image of a flower is represented by using two characteristic color matrices each having a size of 16 × 12. Four color images are searched by the color image similarity search method proposed in the present invention. The experimental results are shown in FIGS. 9 and 10. In FIG. 9, the measured values of similarity (similarity) calculated by the two-dimensional DP matching algorithm are listed. In FIG. 10, Flower104
(110), Flower107 (111), animal101 (11
2), each image of Flower 106 (113) is shown. Since the image is actually a color image, the similarity is as shown in FIG. Tr = 0.45, Tp = 40% and matching window band R = 3 are adopted as main parameters.

In order to evaluate the performance of the proposed method of measuring the color content similarity of images, the success rate of image retrieval is defined by Equation 10.

[Formula 10] SR = D / L Here, SR is the success rate, D is the number of similar images among the selected images, and L is the number of preferentially selected images. When searching for similar images, the user first picks up the first L of the preferred images and performs an image search to determine if it was successful. We conducted several experiments to evaluate the performance of the proposed retrieval method. The results are shown in Fig. 11.

In FIG. 11, the retrieval rate is first (100)
Is the average success rate of the target images determined to have the highest similarity (rank 1st) as a result of measurement, within 5th rank (101)
Is the average success rate within the top 5 and the 10th within the top (102) is the average success rate within the top 10. The average rank (103) is the average rank of the search images that were successful after the image similarity search, and the bottom rank (1
04) is the order when the search image is the lowest in all the searches. These experiments show that the proposed method yielded good search results and that the two-phase search strategy reduced the search time.

As described above, the conventional DP matching is expanded to 2
A dimensional DP matching algorithm was used to determine a method for evaluating two matching distances of the feature matrix. It was found that this method is a novel method for measuring the similarity of images, and not only high-speed measurement is possible, but also the measurement result close to human sense is obtained. By using the Dijkstra algorithm,
The speed was increased.

[0044]

Since the image similarity measuring method and apparatus of the present invention have the above-mentioned configuration, they have the following effects. That is, by adopting a two-dimensional dynamic programming algorithm, high-speed, accurate, and efficient measurement of similarity can be performed, and not only chromaticity but any parameter can be used as a feature amount. It is possible to provide a measuring method having a property. Further, the Dijkstra algorithm can efficiently calculate the similarity between vectors as the shortest path distance, which leads to high-speed measurement.

[Brief description of drawings]

FIG. 1 is an element similarity distance matrix of size n × n.

FIG. 2 shows three directed arcs connecting d (i, j) and a weighting value.

FIG. 3 is a directed graph G.

FIG. 4 is a method of calculating a similar column vector SV.

FIG. 5 is a method of calculating a matching distance g (I, S).

FIG. 6 is a flowchart of two-dimensional DP matching according to the present invention.

FIG. 7 is a method of measuring the degree of similarity in the present embodiment.

FIG. 8 is a sample image image.

FIG. 9 is a table showing measured values of similarity.

FIG. 10 is a list of images showing experimental results.

FIG. 11 is a result of an evaluation experiment for evaluating the performance of the search method.

[Explanation of symbols]

50 First stage (d
Calculation of _row 51 Same, second step (determination of d _min ) 52 Same, third step (record similar column vector number) 53 Same, fourth step (create element similarity distance matrix) 54 Same, fifth step (Calculation of matching distance)

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Liu Yuehu             4-2-1 Kanaikitamachi, Koganei City, Tokyo Independent             Communications Research Institute F-term (reference) 5B050 BA10 EA09 EA18 GA08                 5B056 BB00 HH03                 5L096 AA02 AA06 BA08 DA01 GA19                       GA59 HA08 JA03 JA11 JA20

Claims (6)

[Claims] 1. A method for quantitatively measuring the similarity between images displayed in a matrix, wherein a sample image and at least one measurement target image are pixel-sampled. The block is divided into a plurality of blocks based on at least one of the number and the area, the feature amount of each block is determined by a predetermined method for at least one of the blocks, and a two-dimensional dynamic programming matching method is determined from the feature amount. An image similarity measuring method in which the similarity between the sample image and each measurement target image is quantitatively calculated by using the image to measure the similarity. 2. The image similarity measuring method according to claim 1, wherein in the two-dimensional dynamic programming matching method, a Dijkstra algorithm is used to solve the shortest path problem of the matching distance. 3. In the configuration in which the sample image and each measurement target image are color images, the feature amount is a color chromaticity, and the chromaticity most frequently used in the block is the chromaticity of the block for each block. The image similarity measuring method according to claim 1, wherein the image similarity is determined as a feature amount. 4. An apparatus for quantitatively measuring the similarity between images displayed in a matrix, comprising image input means for inputting a sample image and at least one image to be measured, the sample image and all of them. Image blocking means for dividing the measurement target image into a plurality of blocks based on at least one of the number of pixels and the area, and a feature for determining a feature amount of each block by a predetermined method for at least one of the blocks. Quantity determination means, similarity calculation means for quantitatively calculating the similarity between the sample image and each measurement target image from the feature quantity using a two-dimensional dynamic programming matching method, and similarity between each measurement target image An image similarity measuring apparatus, comprising: a similarity output unit that outputs a property. 5. The image similarity measuring apparatus according to claim 4, wherein the similarity calculation means uses a Dijkstra algorithm for solving the shortest path problem of the matching distance in the two-dimensional dynamic programming matching method. 6. In a configuration in which the sample image and each measurement target image are color images, the feature amount determining means sets the feature amount as color chromaticity, and is used most frequently in each block for each block. 6. The image similarity measuring device according to claim 4, wherein the chromaticity is determined as the feature amount of the block.