JP2018205937A - Image retrieval device and program - Google Patents

Abstract

To provide an image retrieval device and a program capable of precisely retrieving an image of a subject which is the same as or highly similar to a subject in a query image from images in an image database of a retrieval object or on a network.SOLUTION: This image retrieval device 1 includes: an image feature calculation part 111, 121 for calculating an image feature vector obtained by combining an overall feature vector and a detailed feature vector to each of a retrieval object image and a query image; an image similarity calculation part 122 for acquiring first similarity by comparing respective overall feature vectors between the retrieval object image and the query image, acquiring second similarity by comparing respective detailed feature vectors between the retrieval object image and the query image, and calculating similarity obtained by considering both; and a similar image extraction part 123 for retrieving and extracting an image of a subject being the same as or highly similar to a subject in the query image by evaluating the similarity between the retrieval object image and the query image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image search technique using a computer and a database.

  One of the mainstream methods of image search that is currently in practical use is to perform a search using a word as a query for a keyword that is manually or automatically assigned to an image in advance. However, the types and number of keywords that can be assigned are limited, and there are images for which appropriate keywords cannot be given, and therefore it is not always possible to provide the search results desired by the user.

  Therefore, in recent years, a similar image retrieval (Content-Based Image Retrieval, CBIR) technique for retrieving an “appearance-like” image based on comparison of image characteristics between images without using a keyword is known.

  The main operation of the similar image search is as follows. For example, image features such as colors, textures, and shades are extracted in advance from images registered in the image database. Similarly, an image feature is extracted from an image given as a query, compared with the image feature of the image in the image database, and a similarity is calculated. Then, an image having a high similarity is automatically selected and the search result is displayed on the screen.

  In such a similar image retrieval technique, several techniques for retrieving an image similar in appearance to a query image are disclosed (see, for example, Patent Documents 1 and 2).

  In addition, a technique is disclosed in which, after performing region division based on the symmetry axis, the positions of the feature points of the query image and the comparison image are compared (see, for example, Patent Document 3).

JP 2003-233800 A JP 2003-216649 A JP 2010-272092 A

  Although the techniques disclosed in Patent Documents 1 and 2 can search for a roughly similar image with respect to a query image, the image features used in these techniques are the details of the subject in the image. It is difficult to reflect the positional information of the structure and local features. Therefore, there is a problem that it is difficult to search for an image of a subject that is the same as or similar to the subject in the query image with high accuracy.

  In addition, the technique disclosed in Patent Document 3 is a technique for comparing the positions of feature points between a query image and a comparison image after performing region segmentation based on the symmetry axis, and is concerned with the processing efficiency. Although it is effective, there may be a lack of contrasting local features due to the region segmentation, so there is room for improvement with regard to searching for an image of a subject that is the same or similar to the subject in the query image with high accuracy. There is.

  In view of the above-described problems, an object of the present invention is to perform an image search that enables a high-accuracy search for an image of a subject that is the same as or similar to the subject in a query image from an image database or network image to be searched. To provide an apparatus and a program.

  An image search apparatus according to the present invention is an image search apparatus that searches an image of a subject that is the same as or similar to a subject in a query image from an image database to be searched or an image on a network. Image feature calculation means for calculating an image feature vector that combines an overall feature vector and a detailed feature vector for each of the images, and a comparison between the overall feature vector in the search target image and the overall feature vector in the query image 1 similarity is obtained, and a second similarity is obtained by comparing the detailed feature vector in the search target image with the detailed feature vector in the query image, and both the first similarity and the second similarity are obtained. Image similarity calculation means for calculating the similarity taking into account the search target image and the previous image based on the calculated similarity It characterized in that it and a similar image extracting means for extracting search the image of the subject and the same or highly similar object in the query image by evaluating the similarity between the query image.

  Further, in the image search device of the present invention, the image feature calculation means is characterized in that the density of the image changes in two directions by a predetermined value or more with respect to each of the search target image and the query image, One or more pieces of color, texture, and gradient information in a peripheral region within a predetermined image size of the feature point are obtained as vectorized local features, and a predetermined representative for each of the entire search target image and the query image The local feature is vector-quantized using a vector, and a histogram obtained by normalizing the appearance frequency of the local feature subjected to vector quantization is obtained as the overall feature vector.

  In the image search device of the present invention, the image feature calculation means enumerates a set of feature points whose distances in the feature space of the local feature are close to each of the search target image and the query image, and the same symmetry Perform grouping for each set of feature points connected by sex to extract symmetry groups, and a symmetry group in which the number of feature points exceeds a predetermined threshold, or a fixed number of symmetry groups in order of the number of belonging feature points And using the predetermined representative vector, the local feature is vector-quantized for each symmetry group in the search target image, and a histogram obtained by normalizing the appearance frequency of the vector-quantized local feature is described in detail above. It is obtained as a feature vector.

  Further, in the image search device of the present invention, the image similarity calculation means calculates the second similarity obtained by comparing the detailed feature vector in the search target image and the detailed feature vector in the query image for each symmetry group. It is characterized by calculating | requiring by the average value, addition value, or weighted average value of similarity.

  In the image search device according to the present invention, the image similarity calculation means may calculate the search target image and the query image by weighted average as a similarity considering both the first similarity and the second similarity. It is characterized by evaluating the similarity.

  In the image search device of the present invention, the similar image extraction means evaluates the similarity with the query image with respect to a predetermined number of the search target images, and selects a predetermined number of search target images from those having a high degree of similarity. It is characterized by automatic selection and output to the outside.

  In the image search device of the present invention, the image feature calculation means allows extraction of a plurality of symmetry groups, and uses one or more of line symmetry, rotational symmetry, and translational symmetry to determine symmetry groups. It is characterized by extracting.

  Furthermore, the program of the present invention is configured as a program for causing the image search apparatus of the present invention to function as a computer.

  According to the present invention, an image of a subject that is the same as or similar to the subject of the query image can be searched with high accuracy. In particular, it is effective for a subject having symmetry, and since images often include symmetry, the search accuracy can be improved over a wide range over the conventional technique.

It is a block diagram which shows schematic structure of the image search device of one Embodiment by this invention. It is a flowchart which shows the process of the image feature calculation part in the image search device of one Embodiment by this invention. It is a flowchart which shows the process of the image similarity calculation part in the image search device of one Embodiment by this invention. (A), (b) is explanatory drawing which concerns on the process of the image similarity calculation part in the image search device of one Embodiment by this invention, respectively. (A), (b) is a figure which compares numerically the search result in the image search device of one Embodiment by this invention, and the search result of a typical conventional technique, respectively.

  Hereinafter, an image search apparatus 1 according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of an image search apparatus according to an embodiment of the present invention.

〔Device configuration〕
The image search apparatus 1 of this embodiment includes a preprocessing unit 11 and a search processing unit 12. The preprocessing unit 11 includes an image feature calculation unit 111 and an image feature vector storage unit 112. The search processing unit 12 includes an image feature calculation unit 121, an image similarity calculation unit 122, and a similar image extraction unit 123.

  The image search apparatus 1 according to the present embodiment can be configured as a computer that can be connected to a network such as the Internet, and searches for an image of a subject having the same or high similarity based on a query image. It is a device that searches for a predetermined image database to be connected or an image on a network that can be browsed via a network.

In the example described below, in order to improve the understanding of the present invention, N search target image groups I ₁ and I stored in the search target image database 2 specified in advance as a predetermined image database connected to the network. _2, ..., with respect to I _N, describes an example of searching an image of the query image I _Q of the subject and the same or highly similar object with high accuracy.

Here, the image retrieval apparatus 1 of the present embodiment is for performing a search of high subject and identity or similarity of the query image I _Q "subject", the image size and angle of the object in the search target image, Even if there is a difference in color, texture, shading, etc., if the “subject” has the same or high similarity, the search is successful, and this is performed with higher accuracy than in the conventional technique.

First, the image feature calculation unit 111 in the pre-processing unit 11 performs image feature vectors V [I for N search target image groups I ₁ , I ₂ ,..., I _N stored in the search target image database 2. _{1], V [I 2]} , ..., and calculates the V _{[I N],} each search target image group _{I 1, I 2, ...,} and stores the image feature vector storage unit 112 in association with _{I N.}

On the other hand, the image feature calculation unit 121 in the search processing unit 12 calculates an image feature vector V [I _Q ] for the query image I _Q given by the user and outputs it to the image similarity calculation unit 122. The processing of the image feature calculation unit 111 in the preprocessing unit 11 and the image feature calculation unit 121 in the search processing unit 12 are the same, and details thereof will be described later.

Image similarity calculation unit 122, the query image and the image feature vector V _{[I Q]} about _{I Q,} the N frames stored in the image feature vector storage unit 112 searched images _{I 1, I 2, ...,} I N Are compared with respective image feature vectors V [I ₁ ], V [I ₂ ],..., V [I _N ], and similarities S (I _Q , I ₁ ), S (I _Q , I ₂ ),..., S (I _Q , I _N ) are calculated, and similar image extraction is performed in association with the _N search target image groups I ₁ , I ₂ ,. Output to the unit 123.

The similar image extraction unit 123 includes the similarity S (I _Q , I ₁ ), S (I _Q , I ₂ ),..., S (I _Q , I _N ) for the _N search target image groups. A predetermined number of images with high similarity (in this example, the top R images) I _d1 , I _d2 ,..., I _dR are automatically selected and output as search results to the outside (for example, a display device).

[Device operation]
(Image feature calculation processing)
FIG. 2 is a flowchart showing processing of the image feature calculation unit 111 in the image search apparatus 1 according to the embodiment of the present invention. Note that the processing of the image feature calculation unit 121 is the same as that in FIG.

The image feature calculation unit 111 performs image feature vectors V [I ₁ ] and V [I ₂ for N search target image groups I ₁ , I ₂ ,..., I _N stored in the search target image database 2, respectively. ],..., V [I _N ] is calculated. That is, obtains the image feature vector V [I _1] to the search target image I _1, is the same for other retrieval target images, in the following, the search target image I ₁ image feature vector V [I _1] to The required processing will be described.

First, the image feature calculating section 111 detects the feature points to the search target image _{I 1} (step S1). The feature point here is a point where the shading of the image changes in two directions or more by a predetermined value, and corresponds to a corner, a cross, or a point close to them. As a feature point detection method, for example, a method using a Gaussian pyramid used for SIFT (Scale-Invariant Feature Transform) detection can be used (for example, “DG Lowe,“ Object recognition from local scale-invariant features, ” Proc. Of IEEE International Conference on Computer Vision (ICCV), pp. 1150-1157, 1999).

Subsequently, image feature calculating section 111 calculates the local feature of the peripheral region of each feature point detected to the search target image I ₁ (predetermined image size) (step S2). The local feature mentioned here is a vector obtained by vectorizing any one or more of the color, texture, and gradient of the surrounding area of the feature point. As an example of vectorizing the gradient feature, MIFT considering mirror inversion (For example, “X. Guo and X. Cao,“ MIFT: A framework for feature descriptors to be mirror reflection invariant, ”Image and Vision Computing, vol. 30, no. 8, pp. 546 -556, 2012).

Then, the image feature calculating section 111, the processing of steps S1, S2, search target image group _I 1 of N _sheets, I 2, ..., performed respectively to _{I N.}

In the following, processing is divided into overall feature calculation processing and detailed feature calculation processing.
First, the overall feature calculation process will be described.

Image feature calculating section 111, using the representative vector, vector quantizes the local features for the whole of the search target image I ₁ (step S3). In this example, the representative vector is obtained by clustering all or part of local features extracted from all images of the _N search target image groups I ₁ , I ₂ ,. Determine as a vector. The number of clusters is appropriately determined according to the size of the database. This representative vector may be previously selected from an unspecified number of images.

Subsequently, image feature calculating section 111, histogram the frequency of the resultant vector quantized local features to the search target image I ₁ at step S3 (step S4). That is, determine the frequency of occurrence of local features with representative vectors in the search target image I within _1.

Subsequently, the image feature calculation unit 111 normalizes the histogram obtained in step S4 (step S5). As a normalization method, a general method such as L2 normalization using tf-idf can be used. That is, based on the appearance frequency of local features that appear in the search target image I ₁ , a feature amount that predicts the whole is defined.

Subsequently, the image feature calculation unit 111 sets the normalized histogram obtained in step S5 as an overall feature vector V _G [I ₁ ] for the search target image I ₁ (step S6).

Then, the image feature calculating section 111, the processing of steps S3 to S6, the search target image group _I 1 of N _sheets, I 2, ..., go respectively to _{I N,} the entire feature vector _V G _{[I 1} ], V _G [I ₂ ],..., V _G [I _N ]. That is, for a certain search target image _{I n,} the image feature calculating section 111 obtains the entire feature vector _V G _{[I n].}

  Next, detailed feature calculation processing will be described.

Image feature calculating section 111, the search target image I ₁ to enumerate a set of distances is near feature points in the feature space of local features, performs grouping for each set of feature points to be bound by the same symmetry (step S7). For example, when paying attention to line symmetry, a set of feature points connected by the same target axis is set as one group. This group is called a symmetry group. As a method of grouping a set of feature points, line symmetry, rotational symmetry, or translational symmetry can be used. In this example, an example using line symmetry will be described.

  For example, “Z. Tang, P. Monasse, and J. Morel,“ Reflexive Symmetry Detection in Single Image, ”Proc. Of the 8th International Conference Curves and Surfaces, pp. 452-460. , 2014 ”or“ G. Loy and J. Eklundh, “Detecting symmetry and symmetric constellations of features,” ECCV, 2006 ”.

  Further, as a grouping process based on rotational symmetry, for example, the technique disclosed in “G. Loy and J. Eklundh,“ Detecting symmetry and symmetric constellations of features ”,“ ECCV, 2006 ”can be used.

  As a grouping process based on translational symmetry, for example, “M. Park, K. Brocklehurst, RT Collins, and Y. Liu,“ Deformed lattice detection in real-world images using mean-shift belief propagation, ”IEEE Transaction on Pattern Analysis. and Machine Intelligence (TPAMI), vol. 31, No. 10 ”can be used.

Here, the image feature calculating section 111, among the group of search configured to target image I _1, the feature point number of feature points belonging to the each group to employ only those exceeding a predetermined threshold value, or belongs By adopting only a certain number of symmetry groups in descending order, the reliability of the detected symmetry groups can be increased. The number of symmetry groups finally adopted is M ₁ (M ₁ is an integer of 0 or more).

Subsequently, image feature calculating section 111, using the representative vector obtained in step S3, for each symmetric group of search target image I _1, is vector-quantized local features (step S8).

Subsequently, image feature calculating section 111, for each symmetric group of search target image I _1, to produce a histogram of the frequency of the vector quantized local features (step S9). Therefore, M ₁ histograms are obtained.

Subsequently, the image feature calculation unit 111 normalizes the histogram for each symmetry group obtained in step S9 (step S10). The normalization method is the same as in step S5. That is, based on the appearance frequency of local features appearing in each symmetry group in the search target image I ₁ , a feature quantity that predictively represents each symmetry group is defined.

Subsequently, the image feature calculation unit 111 uses the detailed feature vectors V _Sym [I ₁ , 1] and V _Sym [I ₁ ] for the search target image I ₁ obtained by normalizing the histogram obtained in step S10 for each symmetry group. , 2],..., V _Sym [I ₁ , M ₁ ] (step S11).

Then, the image feature calculating section 111, the processing of steps S7 to S11, the search target image group _I 1 of N _sheets, I 2, ..., go respectively to _{I N,} N pieces of search target images I _{1, I} 2, ..., for a certain search target image _{I n} of _{I n,} detailed feature vector _{V Sym [I n, 1]} , V Sym [I n, 2], ..., V Sym [I n, M _n ] is obtained.

Finally, the image feature calculation unit 111 puts together the entire feature vector obtained in the above step S6 and the detailed feature vector obtained in the above step S11 into N search target image groups I ₁ and I. _2, ..., with respect to _{I N,} each image feature vector _{V [I 1], V [} I 2], ..., obtain V _{[I N].} That is, for a certain search target image _{I n,} the image feature calculating section 111, the image feature vector _{V [I n] = {V} G [I n], V Sym [In, 1], V Sym [I n, 2],..., V _Sym [I _n , M _n ]}.

However, the search target image group _{I 1,} I 2 of N sheets, ..., with respect to certain search target image _{I n} of _{I N,} the _M n = 0 (symmetric group has not obtained) case , Image feature vector V [I _n ] = {V _G [I _n ], V _Sym [I _n , 1]}. Here, V _Sym [I _n , 1] when the symmetry group is not obtained is determined so that it can be identified (for example, in the image similarity calculation process described later with reference to FIG. It is determined as a 0 vector so that the image similarity is always 0).

The processing of the image feature calculation unit 121 is the same as that in FIG. 2, and the image feature calculation unit 121 performs image feature vector V [I _Q ] = {V _G [I _Q ], V _Sym for the query image I _Q. [I _Q , 1], V _Sym [I _Q , 2],..., V _Sym [I _Q , M _Q ]} are obtained.

In this manner, the image feature calculating section 111 and 121, for example, FIG. 4 the entire feature vector composed of many 〇 frame within the represented feature points (a), the total for the query image I _Q wherein as a vector _V G _{[I Q],} about certain search target image _{I n} the entire feature vector _V G _{[I n],} is comparably calculated one another.

Further, the image feature calculation units 111 and 121 are represented in a large number of frames within a single-dot dashed line O frame indicating the first symmetry group having the first target axis Sym1 in common shown in FIG. 4B, for example. and the detailed feature vectors constructed from the feature points, the query image _I as a detailed feature vector _V Sym _[I Q, 1] for _Q, one search target image detail feature vector _V Sym for _{I n [I} n, 1 ] So that they can be compared with each other.

Furthermore, the image feature calculation units 111 and 121 are represented in a large number of frames within a single-dot dashed line O frame indicating a second symmetry group having the second target axis Sym2 in common shown in FIG. 4B, for example. and the detailed feature vectors constructed from the feature points, the query image _I as a detailed feature vector _V Sym _[I Q, 2] for _Q, one search target image detail feature vector _V Sym for _{I n [I} n, 2 ] So that they can be compared with each other.

(Image similarity calculation processing)
FIG. 3 is a flowchart showing the processing of the image similarity calculation unit 122 in the image search device 1 according to the embodiment of the present invention.

Image similarity calculation unit 122, the query image and the image feature vector V _{[I Q]} about _{I Q,} the N frames stored in the image feature vector storage unit 112 searched images _{I 1, I 2, ...,} I N Are compared with respective image feature vectors V [I ₁ ], V [I ₂ ],..., V [I _N ], and similarities S (I _Q , I ₁ ), S (I _Q , I ₂ ), ..., S (I _Q , I _N ) is calculated. For the calculation of the similarity, a general one such as an inner product of vectors or a value obtained by normalizing the Euclidean distance to 0 to 1 is used.

First, the image similarity calculation unit 122 selects an image feature vector V [I _Q ] related to the query image I _Q and a certain search target image from the _N search target image groups I ₁ , I ₂ ,. I _n relates inputs the image feature vector V _{[I n]} (step S21).

Subsequently, the image similarity calculation unit 122 calculates the total image similarity S _G (I _Q , I _n ) and the detailed image similarity S _Sym (I _Q , I _n ) as an average value of the similarity for each symmetry group. Individually.

First, the image similarity calculation unit 122, a query image _{I Q} and the image feature vector V entire feature vector _V G in _{[I Q] [I Q]} about the image feature vector V [I related to the search target image _{I n} The overall feature vector V _G [I _Q ] at _n ] is compared to obtain the overall image similarity S _G (I _Q , I _n ) (step S22).

Second, the image similarity calculation unit 122 obtains the detailed image similarity S _Sym (I _Q , I _n ) according to the following procedure (step S23).

First, the subscript set of the symmetry group of the query image I _Q is ID _Q = {1, 2,..., M _Q }, and the subscript set of the symmetry group of the search target image I _n is ID _n = {1, 2, ..., M _n }. _Here, M Q, _{M n} is the number of detailed feature vectors of each query image _{I Q} search target image _{I n.}

  First, the initial value of m is set to 1.

(Processing A) The image similarity calculation unit 122 selects i and j that maximize the similarity between V _sym [I _Q , i] and V _sym [I _n , j] from ID _Q and ID _n , respectively. The similarity is S _Sym (I _Q , I _n ) [m]. The detailed image similarity is calculated in the same manner as the overall image similarity.

(Process B) Next, the image similarity calculator _122, S _Sym checks whether the _{(I Q, I n) [} m] = 0, S Sym (I Q, I n) [m] If = 0, the process proceeds to “Process E”.

(Processing C) Next, when S _Sym (I _Q , I _n ) [m] = 0 is not _satisfied , the image similarity calculation unit 122 excludes i and j from ID _Q and ID _n , and sets m = m + 1. .

(Process D) image similarity calculation unit _122, ID Q, until ID _n is an empty set, repeating the above (process A) ~ a (process _C), when the ID Q, is ID _n becomes an empty set, Move to “Process E”.

(Process E) image similarity calculator _{122, S Sym (I Q, I} n) _{a, S Sym (I Q, I} n)) [1], ..., S Sym (I Q, I n) [m -1] as an average value. However, when S _Sym (I _Q , I _n ) [1] = 0, it is assumed that S _Sym (I _Q , I _n ) = 0.

In this way, the image similarity calculation unit 122 obtains the detailed image similarity S _Sym (I _Q , I _n ) as an average value of similarities for each symmetry group.

Finally, the image similarity calculation unit 122, a query image _{I Q} and the image feature vector V _{[I Q]} regarding the image similarity S by comparison with the image feature vector V about the search target image _{I n [I n]} (I _Q , I _n ) is weighted to the total image similarity S _G (I _Q , I _n ) and the detailed image similarity S _Sym (I _Q , I _n ) as an average value of the similarity for each symmetry group The average is obtained by the following equation (step S24).
S (I _Q , I _n ) = (1−a) S _G (I _Q , I _n ) + aS _Sym (I _Q , I _n )

  Here, a is a parameter that determines the contribution of the target axis (0 <a <1).

That is, in the example shown in FIG. 4B, M _Q = M _n = 2 and the similarity S _Sym (I _Q , I _n ) [1] of the detailed feature vectors in the first symmetry group is V _Sym [ The similarity between I _Q , 1] and V _Sym [I _n , 1], and the detailed feature vector S _Sym (I _Q , I _n ) [2] in the second symmetry group is expressed as V _Sym [I _Q , 2] The similarity is V _Sym [I _n , 2]. For this reason, the detailed image similarity S _Sym (I _Q , I _n ) is expressed as S _Sym (I _Q , I _n ) = (S _Sym (I _Q , I _n )) as an average value of similarities for each symmetry group. [1] + S _Sym (I _Q , I _n ) [2]) / 2.

  In the example shown in FIG. 4B, in order to enhance the understanding of the present invention, the plurality of symmetry groups do not overlap each other, but actually the image search device 1 of the present embodiment extracts them. The overlapping of symmetry groups is allowed, thereby preventing the loss of local features to be compared and improving the accuracy.

The image similarity calculation unit 122 instead of _{obtaining the} detailed image similarity S _Sym (I _Q , I _n ) as the average value of the similarity for each symmetry group, that is, the sum of the similarities for each symmetry group, _{, S Sym (I Q, I} n) = S Sym (I Q, I n) [1], ..., S Sym may be determined as the sum of _{(I Q, I n) [} m-1].

Alternatively, instead of _{obtaining the} detailed image similarity S _Sym (I _Q , I _n ) as the average value of the similarity for each symmetry group, the image similarity calculation unit 122 weights the similarity average for each symmetry group. That is, S _Sym (I _Q , I _n ) = S _Sym (I _Q , I _n ) [1],..., S _Sym (I _Q , I _n ) [m−1] may be obtained as a weighted average value. Good. As the weight at this time, when a value serving as an index of reliability of the symmetric axis is obtained as a result of the algorithm for detecting the symmetric axis, it can be used.

As described above, by _{obtaining the} detailed image similarity S _Sym (I _Q , I _n ) as the average value, the addition value, or the weighted average value of the similarity for each symmetry group, the similarity is increased as more symmetry axes are detected. The situation where the degree becomes low can be avoided.

As described above, the image similarity calculation unit 122 has the image feature vectors V [I ₁ for the N search target image groups I ₁ , I ₂ ,..., I _N stored in the image feature vector storage unit 112. ], V [I ₂ ],..., V [I _N ] are compared with the image feature vector V [I _Q ] related to the query image I _Q , thereby obtaining the respective similarities S (N) for the N search target image groups. I _Q , I ₁ ), S (I _Q , I ₂ ),..., S (I _Q , I _N ) are calculated. As a result, the similar image extraction unit 123 automatically selects a predetermined number of images (higher R images in this example) I _d1 , I _{d2 ,.} It can be selected and output to the outside (for example, a display device) as a search result.

  5 (a) and 5 (b) respectively show a search result in the image search apparatus 1 according to an embodiment of the present invention, and a search result of a typical conventional technique (that is, similarity evaluation of only the entire feature vector). It is a figure which compares with numerical value. Each of the data sets used in FIGS. 5 (a) and 5 (b) is a data set relating to various constructions including various butterflies and overpasses which are generally disclosed, and one image (FIG. 5) is used. A certain butterfly for (a) and a certain crossover for FIG. 5 (b) were evaluated as query images, and the other images were evaluated as search target images. For symmetry, considering line symmetry, Z. Tang, P. Monasse, and J. Morel, “Reflexive Symmetry Detection in Single Image,” Proc. Of the 8th International Conference Curves and Surfaces, pp. 452- 460, 2014 ”is used.

  In the butterfly data set shown in FIG. 5A, in the comparative example corresponding to the conventional technique, a feature vector in consideration of symmetry is not used (parameter a = 0), and in the embodiment according to the present invention, symmetry is considered. The feature vector was used (parameter a = 0.7).

  Further, in the crossover data set shown in FIG. 5B, in the comparative example corresponding to the conventional technique, a feature vector considering symmetry is not used (parameter a = 0), and in the embodiment according to the present invention, symmetry is used. Is used (parameter a = 0.5).

The average relevance ratio AP is calculated by the following equation, where P _i is the relevance ratio (= number of correct answers / total number) up to the i-th correct answer. N is the total number of correct images. 0 ≦ AP ≦ 1, and the closer to 1, the better the search accuracy.

  As can be seen from the comparison shown in FIGS. 5A and 5B, an image of a subject that is the same as or similar to the subject of the query image can be searched with high accuracy. In particular, the search results for butterflies are significantly better than those of conventional techniques over crossovers. This is effective for subjects with symmetry, and images often contain symmetry. The search accuracy can be improved over the technique. Further, in FIGS. 5A and 5B, the parameter a = 0.7 or 0.5 is set as an embodiment according to the present invention, but the value set by searching for an optimal numerical value as each search result. However, it was also confirmed that general versatility can be obtained by setting the parameter a in the range of 0.5 to 0.7.

  In the comparison shown in FIGS. 5A and 5B, in order to exclude subjectivity, evaluation was performed with respect to searching for an image of the “same subject” with the subject in the query image with high accuracy. And more effective results can be expected for the purpose of searching with high accuracy images of “subjects with the same or high similarity”.

  In summary, in the image search device 1 according to the present embodiment, the image feature calculation units 111 and 121 firstly combine the entire feature vector and the detailed feature vector for each of the search target image and the query image. Calculate

  Here, the image feature calculation units 111 and 121 in the image search apparatus 1 according to the present embodiment have a feature point that the shade of the image changes in two directions by a predetermined value or more with respect to each of the search target image and the query image. , Obtaining one or more pieces of color, texture, and gradient information in a peripheral region within a predetermined image size of each feature point as a local feature obtained by vectorization, and calculating a predetermined value for each of the search target image and the query image The local feature is subjected to vector quantization using the representative vector, and a histogram obtained by normalizing the frequency of appearance of the local feature obtained by vector quantization is obtained as an overall feature vector.

  In addition, the image feature calculation units 111 and 121 in the image search device 1 of the present embodiment list a set of feature points that are close to each other in the feature space of the local feature for each of the search target image and the query image. Performs grouping for each set of feature points connected by symmetry, extracts symmetry groups, symmetry groups whose number of feature points exceeds a predetermined threshold, or a certain number of symmetry points in descending order of belonging feature points Extract a group, vector-quantize the local feature for each symmetry group in the search target image using the predetermined representative vector, and detail the histogram that normalized the frequency of appearance of the vector-quantized local feature Ask as a vector.

  Then, in the image search device 1 of the present embodiment, the image similarity calculation unit 122 obtains the first similarity by comparing the overall feature vector in the search target image with the overall feature vector in the query image, A second similarity is obtained by comparing the detailed feature vector with the detailed feature vector in the query image, and a similarity considering both the first similarity and the second similarity (weighting based on parameter a in this example) Average), the similarity between the search target image and the query image is evaluated.

  In particular, the image similarity calculation unit 122 calculates the second similarity based on the comparison between the detailed feature vector in the search target image and the detailed feature vector in the query image, the average value of the similarity for each symmetry group, the added value, Or it calculates | requires by a weighted average value.

  Then, the image search apparatus 1 according to the present embodiment uses the similar image extraction unit 123 to evaluate the similarity with the query image with respect to the predetermined number of search target images, and selects the predetermined number of search target images from those having high similarity. Automatically select and output to the outside.

  As described above, the image search apparatus 1 according to the present invention obtains the first similarity based on the comparison of the entire feature vectors and the second similarity based on the comparison of the detailed feature vectors, and the first similarity and the second Since the similarity between the search target image and the query image is evaluated based on the similarity (in this example, the weighted average based on the parameter a) considering both of the similarities, for example, it is disclosed in Patent Document 3 Compared to other conventional techniques such as the technique, the difference in the detailed structure and positional relationship of the subject of the image can be reflected in the similarity.

  In addition, the technique disclosed in Patent Document 3 is a technique for comparing the positions of feature points between a query image and a comparison image after performing region division based on the symmetry axis. However, the symmetry group extracted by the image search apparatus 1 according to the present invention allows the overlap, thereby preventing the loss of the local feature to be compared with accuracy. Can be increased.

  Further, the symmetry used in the technique disclosed in Patent Document 3 is limited to line symmetry, but the image search apparatus 1 according to the present invention is also applicable to cases where a plurality of symmetries exist in one image. The symmetry group can be extracted using one or more of line symmetry, rotational symmetry, and translational symmetry. As a result, it is possible in principle to construct image features that reflect information on the detailed structure and positional relationship of the subject as compared to the image features that have been used in the past, and it is possible to increase the accuracy of similar image searches. It becomes.

  The image search apparatus 1 according to the present invention is configured as a computer, and a program for causing the computer to realize each component according to the present invention is stored in a memory (not shown) provided inside or outside the computer. Is done. An image search apparatus according to the present invention is appropriately read from a memory by reading a program describing processing contents for realizing the function of each component under the control of a central processing unit (CPU) provided in the computer. The function of each component of 1 can be realized by a computer.

The present invention is not limited to the above-described exemplary embodiments, but is limited only by the description of the scope of claims. For example, in the example of the above-described embodiment, N search target image groups I ₁ , I ₂ ,..., I _N stored in the search target image database 2 specified in advance as a predetermined image database connected to the network. contrast, the image of the query image I _Q of the subject and the same or highly similar object has been described an example of searching with high accuracy, it can be searched for images on browsable network via the network. For example, when images on a network that can be browsed are to be searched, randomly collected images may be stored in the search target image database 2.

  According to the present invention, it is possible to search for an image of a subject that is the same as or similar to the subject of the query image with high accuracy, and is particularly effective for a subject having symmetry. Since the accuracy can be improved, it is useful for a material search in an archive, a material search in video production, or an image search engine on the Web.

DESCRIPTION OF SYMBOLS 1 Image search apparatus 2 Search object image database 11 Pre-processing part 12 Search processing part 111 Image feature calculation part 112 Image feature vector storage part 121 Image feature calculation part 122 Image similarity calculation part 123 Similar image extraction part

Claims (8)

An image search apparatus for searching for an image of a subject that is the same as or similar to a subject in a query image from an image database to be searched or an image on a network,
Image feature calculation means for calculating an image feature vector that combines the overall feature vector and the detailed feature vector for each of the search target image and the query image;
A first similarity is obtained by comparing the global feature vector in the search target image with the global feature vector in the query image, and the first similarity is obtained by comparing the detailed feature vector in the search target image with the detailed feature vector in the query image. Image similarity calculation means for obtaining a similarity of 2 and calculating a similarity considering both the first similarity and the second similarity;
Similar image extraction that searches for and extracts images of subjects that are the same as or similar to the subject in the query image by evaluating the similarity between the search target image and the query image based on the calculated similarity Means,
An image search apparatus comprising:   The image feature calculation means has a feature point in which the shade of the image changes in two directions by a predetermined value or more with respect to each of the search target image and the query image, and a peripheral region within a predetermined image size of each feature point As a local feature obtained by vectorizing any one or more of color, texture, and gradient in the image, the local feature is vector quantized using a predetermined representative vector for each of the search target image and the query image. The image search apparatus according to claim 1, wherein a histogram obtained by normalizing the appearance frequency of the local features obtained by normalization and vector quantization is obtained as the overall feature vector.   The image feature calculation means enumerates a set of feature points whose distance in the feature space of the local feature is close to each of the search target image and the query image, and for each set of feature points connected by the same symmetry Perform a grouping to extract a symmetry group, extract a symmetry group in which the number of feature points exceeds a predetermined threshold, or a certain number of symmetry groups in order of the number of belonging feature points. Using, for each symmetry group in the search target image, vector quantization of the local features, and obtaining a histogram that normalizes the appearance frequency of the vector quantized local features as the detailed feature vector, The image search device according to claim 1.   The image similarity calculation means calculates the second similarity by comparing the detailed feature vector in the search target image and the detailed feature vector in the query image, an average value of the similarity for each symmetry group, an added value, Or it calculates | requires by a weighted average value, The image search device as described in any one of Claim 1 to 3 characterized by the above-mentioned.   The image similarity calculation means evaluates the similarity between the search target image and the query image based on a weighted average as a similarity considering both the first similarity and the second similarity. The image search device according to any one of claims 1 to 4.   The similar image extracting means evaluates similarity with the query image for a predetermined number of the search target images, automatically selects a predetermined number of search target images from those having a high degree of similarity, and outputs them to the outside. The image search device according to claim 1, wherein the image search device is a feature.   The image feature calculation means allows extraction of a plurality of symmetry groups, and extracts symmetry groups using one or more of line symmetry, rotational symmetry, and translational symmetry. The image search device according to any one of 1 to 6.   The program for functioning the image search device as described in any one of Claim 1 to 7 as a computer.