JP6966875B2 - Image search device and program - Google Patents

Description

The present invention relates to an image retrieval technique using a computer and a database.

One of the mainstream methods of image search currently in practical use is to search by using a word as a query for a keyword given to an image in advance manually or automatically. However, since the types and number of keywords that can be given are limited, and some images cannot be given appropriate keywords, it is not always possible to provide the search results desired by the user.

Therefore, in recent years, a similar image retrieval (Content-Based Image Retrieval, CBIR) technique for searching for "similar in appearance" images based on comparison of image features between images without using keywords has been known.

The main operations of similar image search are as follows. For example, image features such as color, texture, and shading are extracted in advance from an image registered in an image database. Image features are similarly extracted from the image given as a query, compared with the image features of the images in the image database, and the degree of similarity is calculated. Then, an image having a high degree of similarity is automatically selected and the search result is displayed on the screen.

In such a similar image search technique, some techniques for searching for an image having a roughly similar appearance to a query image are disclosed (see, for example, Patent Documents 1 and 2).

Further, a technique for comparing the positions of feature points between a query image and a comparison image after performing region division based on the axis of symmetry is disclosed (see, for example, Patent Document 3).

Japanese Patent Application Laid-Open No. 2003-233800 Japanese Patent Application Laid-Open No. 2003-216649 Japanese Unexamined Patent Publication No. 2010-272092

The techniques disclosed in Patent Documents 1 and 2 can search for images that are roughly similar in appearance to the query image, but the image features used in these techniques are detailed of the subject in the image. It is difficult to reflect information on the positional relationship of structures and local features. Therefore, there is a problem that it is difficult to search for an image of a subject having the same or high similarity as the subject in the query image with high accuracy.

Further, the technique disclosed in Patent Document 3 is a technique of comparing the positions of feature points between the query image and the comparison image after performing region division based on the axis of symmetry, and the point of processing efficiency thereof. However, there is room for improvement in searching for images of subjects that are the same as or highly similar to the subject in the query image with high accuracy, because the local features to be compared may be lost due to the region division. There is.

In view of the above problems, an object of the present invention is an image search that enables highly accurate search of an image of a subject in a query image that is the same as or highly similar to the subject from an image database or an image on a network to be searched. To provide equipment and programs.

The image search device of the present invention is an image search device that searches for an image of a subject that is the same as or highly similar to the subject in the query image from an image database to be searched or an image on a network, and is a search target image and a query. For each of the images, the image feature calculation means for calculating the image feature vector by combining the overall feature vector and the detailed feature vector, and the overall feature vector in the search target image and the overall feature vector in the query image are compared. The similarity of 1 is obtained, and the second similarity is obtained by comparing the detailed feature vector in the search target image with the detailed feature vector in the query image, and both the first similarity and the second similarity are obtained. The image similarity calculation means for calculating the similarity in consideration of the above, and the same or similar to the subject in the query image by evaluating the similarity between the search target image and the query image based on the calculated similarity. The image feature calculation means includes a similar image extraction means for searching and extracting an image of a highly high-quality subject, and the image feature calculation means changes the shade of the image in two directions by a predetermined value or more with respect to each of the search target image and the query image. The point to be searched is used as a feature point, and one or more information of any one or more of a color, a texture, and a gradient in a peripheral area within a predetermined image size of each feature point is obtained as a vectorized local feature, and the search target image and the query are obtained. and vector quantization the local features using a predetermined representative vectors for the entire of each image, a histogram obtained by normalizing the frequency of the vector quantized local features, and wherein Rukoto determined as the whole feature vector ..

Further, in the image search device of the present invention, the image feature calculation means enumerates a set of feature points whose distances in the feature space of the local feature are close to each of the search target image and the query image, and has the same symmetry. A symmetry group is extracted by grouping each set of feature points linked by gender, and a symmetry group in which the number of feature points exceeds a predetermined threshold, or a fixed number of symmetry groups in descending order of the number of feature points to which they belong. Is extracted, the local feature is vector-quantized for each symmetry group in the search target image using the predetermined representative vector, and a histogram obtained by normalizing the appearance frequency of the vector-quantified local feature is obtained in the above-mentioned details. It is characterized by being obtained as a feature vector.

Further, in the image search device of the present invention, the image similarity calculation means determines the second similarity by comparing the detailed feature vector in the search target image with the detailed feature vector in the query image for each symmetry group. It is characterized in that it is obtained by the average value, the addition value, or the weighted average value of the similarity of.

Further, in the image search device of the present invention, the image similarity calculation means uses a weighted average as a similarity considering both the first similarity and the second similarity to obtain the search target image and the query image. It is characterized by evaluating the similarity with.

Further, in the image search device of the present invention, the similar image extraction means evaluates the similarity of a predetermined number of the search target images with the query image, and selects a predetermined number of search target images from the one with the highest degree of similarity. The feature is that it is automatically selected and output to the outside.

Further, in the image search device of the present invention, the image feature calculation means allows the extraction of a plurality of symmetry groups at the time of extracting the symmetry group, and is one of line symmetry, rotational symmetry, and translational symmetry. Using the above, grouping is performed for each set of feature points connected by the same symmetry, and the plurality of symmetry groups are extracted.

Further, the program of the present invention is configured as a program for operating the image search device of the present invention as a computer.

According to the present invention, it is possible to search for an image of a subject that is the same as or highly similar to the subject of the query image with high accuracy. In particular, it is effective for a subject having symmetry, and since the image often contains symmetry, it is possible to improve the search accuracy in a wide range as compared with the conventional technique.

It is a block diagram which shows the schematic structure of the image search apparatus of one Embodiment by this invention. It is a flowchart which shows the processing of the image feature calculation unit in the image search apparatus of one Embodiment by this invention. It is a flowchart which shows the processing of the image similarity calculation unit in the image search apparatus of one Embodiment by this invention. (A) and (b) are explanatory views relating to the processing of the image similarity calculation unit in the image search apparatus of one embodiment according to the present invention, respectively. (A) and (b) are diagrams for numerically comparing the search result in the image search device of one embodiment according to the present invention with the search result of a typical conventional technique.

Hereinafter, the image retrieval device 1 according to the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of an image retrieval device according to an embodiment of the present invention.

〔Device configuration〕
The image search device 1 of the present embodiment includes a preprocessing unit 11 and a search processing unit 12. The preprocessing unit 11 includes an image feature calculation unit 111 and an image feature vector storage unit 112. Further, the search processing unit 12 includes an image feature calculation unit 121, an image similarity calculation unit 122, and a similar image extraction unit 123.

The image search device 1 of the present embodiment can be configured as a computer that can be connected to a network such as the Internet to search for an image of a subject having the same or high similarity based on the query image. A device that searches for images on a predetermined image database to be connected or a network that can be viewed via a network.

In the example described below, in order to enhance the understanding of the present invention, N search target image groups I ₁ and I stored in the search target image database 2 previously specified as a predetermined image database connected to the network. _2, ..., with respect to I _N, describes an example of searching an image of the query image I _Q of the subject and the same or highly similar object with high accuracy.

Here, the image retrieval apparatus 1 of the present embodiment is for performing a search of high subject and identity or similarity of the query image I _Q "subject", the image size and angle of the object in the search target image, Even if there are differences in color, texture, shading, etc., if the "subject" is the same or has high similarity, the search is successful, and this is performed with higher accuracy than the conventional technique.

First, the image feature calculating section 111 in the pretreatment unit 11, the search target image database of N frames stored in the 2 search target image group _{I 1,} I 2, ..., with respect to _{I N,} respectively image feature vector V [I _{1], V [I 2]} , ..., and calculates the V _{[I N],} each search target image group _{I 1, I 2, ...,} and stores the image feature vector storage unit 112 in association with _{I N.}

On the other hand, the image feature calculating section 121 in the search processing unit 12, with respect to the query image I _Q given from the user, calculates the image feature vector V [I _Q], and outputs the image similarity calculation unit 122. The processing of the image feature calculation unit 111 in the preprocessing unit 11 and the image feature calculation unit 121 in the search processing unit 12 are the same, and the details thereof will be described later.

Image similarity calculation unit 122, the query image and the image feature vector V _{[I Q]} about _{I Q,} the N frames stored in the image feature vector storage unit 112 searched images _{I 1, I 2, ...,} I N Compare the respective image feature vectors V [I ₁ ], V [I ₂ ], ..., V [ _IN ] with respect to their respective similarity degrees S (I _Q , I ₁ ), S (I _Q , I). ₂ ), ..., S (I _Q , _IN ) is calculated, and similar images are extracted in association with the _{N search target image groups I 1} , I ₂ , ..., IN stored in the search target image database 2. Output to unit 123.

The similar image extraction unit 123 has the similarity S (I _Q , I ₁ ), S (I _Q , I ₂ ), ..., S (I _Q , _IN ) with respect to the N search target image groups, respectively. A predetermined number of images with high similarity (in this example, the top R images) _Id1 , _Id2 , ..., _IdR are automatically selected and output to the outside (for example, a display device) as a search result.

[Device operation]
(Image feature calculation processing)
FIG. 2 is a flowchart showing the processing of the image feature calculation unit 111 in the image search device 1 of the embodiment according to the present invention. The processing of the image feature calculation unit 121 is the same as that of FIG.

Image feature calculating section 111, the search target image is stored in the database 2 of the N sheets searched image group _{I 1,} I 2, ..., with respect to _{I N,} respectively image feature vector _{V [I 1], V [} I 2 ], ..., V [ _IN ] is calculated. That is, obtains the image feature vector V [I _1] to the search target image I _1, is the same for other retrieval target images, in the following, the search target image I ₁ image feature vector V [I _1] to The required process will be explained.

First, the image feature calculation unit 111 detects feature points for the _{search target image I 1 (step S1).} The feature point referred to here is a point where the shading of the image changes by a predetermined value or more in two directions, and corresponds to a corner, a cross, or a point close to them. As a method for detecting feature points, for example, a method using the Gaussian pyramid used for detecting SIFT (Scale-Invariant Feature Transform) can be used (for example, "DG Lowe," Object recognition from local scale-invariant features, "". See Proc. Of IEEE International Conference on Computer Vision (ICCV), pp. 1150-1157, 1999).

Subsequently, the image feature calculation unit 111 calculates the local features of the peripheral region (within a predetermined image size) of each feature point detected _{for the search target image I 1 (step S2).} The local feature here is a vector of information on any one or more of the color, texture, and gradient of the area around the feature point, and is an example of vectorizing the gradient feature with MIFT considering mirror surface inversion. The so-called method is available (eg, “X. Guo and X. Cao,“ MIFT: A framework for feature descriptors to be mirror reflection invariant, ”Image and Vision Computing, vol. 30, no. 8, pp. 546. -556, 2012 ”).

Then, the image feature calculation unit 111 performs the processing of the above steps S1 and S2 for the N image search target image groups I ₁ , I ₂ , ..., _IN , respectively.

In the following, the processing is divided into the overall feature calculation process and the detailed feature calculation process.
First, the overall feature calculation process will be described.

The image feature calculation unit 111 vector-quantizes the local features with respect to the entire _{search target image I 1 using the representative vector (step S3).} In this example, the representative vector clusters all or part of the local features extracted from all the images of the _{N search target image groups I 1} , I ₂ , ..., IN, and represents the vector at the center of the cluster. Defined as a vector. The number of clusters is appropriately determined according to the size of the database. This representative vector may be selected in advance from an unspecified number of images.

Subsequently, the image feature calculation unit 111 creates a histogram of the appearance frequency of the vector-quantized local features obtained in step S3 with respect to the _{search target image I 1 (step S4).} That is, the appearance frequency of the local feature having the representative vector in the search target image I _{1 is examined.}

Subsequently, the image feature calculation unit 111 normalizes the histogram obtained in step S4 (step S5). As the normalization method, a general method such as L2 normalization using tf-idf can be used. That is, based on the frequency of occurrence of local feature appearing to the search target image I _1, Teigizukeru the feature quantity predicting express its entirety.

Subsequently, image feature calculating section 111 is the entire feature vector _V G _{[I 1]} of the normalized histogram obtained in step S5 to the search target image _{I 1} (step S6).

Then, the image feature calculating section 111, the processing of steps S3 to S6, the search target image group _I 1 of N _sheets, I 2, ..., go respectively to _{I N,} the entire feature vector _V G _{[I 1 ], V G [I 2]} , ..., determine the V G _{[I N].} That is, for a certain search target image _{I n,} the image feature calculating section 111 obtains the entire feature vector _V G _{[I n].}

Next, the detailed feature calculation process will be described.

The image feature calculation unit 111 _{enumerates a set of feature points that are close in distance to the search target image I 1 in} the feature space of the local feature, and groups the set of feature points connected by the same symmetry (step). S7). For example, when focusing on line symmetry, a set of feature points connected by the same target axis is regarded as one group. This group is called a symmetry group. As a method of grouping a set of feature points, line symmetry, rotational symmetry, or translational symmetry can be used, but in this example, an example using line symmetry will be described.

As a grouping process by line symmetry, for example, "Z. Tang, P. Monasse, and J. Morel," Reflexive Symmetry Detection in Single Image, "Proc. Of the 8th International Conference Curves and Surfaces, pp. 452-460. , 2014 ”or“ G. Loy and J. Eklundh, “Detecting symmetry and symmetric constellations of features,” ECCV, 2006 ”.

Further, as the grouping process by rotational symmetry, for example, the technique disclosed in the above-mentioned "G. Loy and J. Eklundh," Detecting symmetry and symmetric constellations of features, "ECCV, 2006" can be used.

In addition, as grouping processing by translational symmetry, for example, "M. Park, K. Brocklehurst, RT Collins, and Y. Liu," Deformed lattice detection in real-world images using mean-shift belief propagation, "IEEE Transaction on Pattern Analysis. The techniques disclosed in and Machine Intelligence (TPAMI), vol. 31, No. 10 ”are available.

Here, the image feature calculation unit 111 _{adopts only those groups configured for the search target image I 1} in which the number of feature points belonging to each group exceeds a predetermined threshold value, or the feature points to which the image feature points belong are. By adopting only a certain number of symmetry groups in descending order, the reliability of the detected symmetry groups can be increased. The number of symmetry groups finally adopted is M ₁ (M ₁ is an integer of 0 or more).

Subsequently, image feature calculating section 111, using the representative vector obtained in step S3, for each symmetric group of search target image I _1, is vector-quantized local features (step S8).

Subsequently, the image feature calculation unit 111 generates a histogram of the appearance frequency of vector-quantized local features for each symmetry group in the _{search target image I 1 (step S9).} Thus, _{M 1} single histogram is obtained.

Subsequently, the image feature calculation unit 111 normalizes the histogram for each symmetry group obtained in step S9 (step S10). The normalization method is the same as in step S5. That is, based on the frequency of occurrence of local feature appearing in each symmetry group in the search target image I _1, Teigizukeru the feature quantity predicting representing the respective symmetry group.

Subsequently, the image feature calculation unit 111 uses the detailed feature vector V _Sym [I ₁ , 1], V _Sym [I _{1 ] for the search target image I 1} to obtain the histogram normalized for each symmetry group obtained in step S10. , 2], ..., V _Sym [I ₁ , M ₁ ] (step S11).

Then, the image feature calculating section 111, the processing of steps S7 to S11, the search target image group _I 1 of N _sheets, I 2, ..., go respectively to _{I N,} N pieces of search target images I _{1, I} 2, ..., for a certain search target image _{I n} of _{I n,} detailed feature vector _{V Sym [I n, 1]} , V Sym [I n, 2], ..., V Sym [I n, M _n ] is obtained.

Finally, the image feature calculation unit 111 puts together the overall feature vector obtained in step S6 and the detailed feature vector obtained in step S11, and N images of the search target image groups I ₁ and I _2, ..., with respect to _{I N,} each image feature vector _{V [I 1], V [} I 2], ..., obtain V _{[I N].} That is, for a certain search target image _{I n,} the image feature calculating section 111, the image feature vector _{V [I n] = {V} G [I n], V Sym [In, 1], V Sym [I n, _2], obtained _..., V _Sym [I n, the _M n]}.

However, the search target image group _{I 1,} I 2 of N sheets, ..., with respect to certain search target image _{I n} of _{I N,} the _M n = 0 (symmetric group has not obtained) case the image feature vector _{V [I n] = {V} G [I n], V Sym [I n, 1]} defined as. Here, V _Sym when symmetry group is not obtained _{[I n, 1]} is detailed in the image similarity calculation process that effect will be described below with reference to the determined (e.g. FIG. 3 can be identified It is defined as a 0 vector so that the image similarity is always 0).

Processing of the image feature calculating section 121 is also similar to the processing of FIG. 2, the image feature calculating section 121, with respect to the query image _{I Q,} the image feature vector _{V [I Q] = {V} G [I Q], V Sym _{[I Q, 1], V} Sym [I Q, 2], obtained _{..., V Sym [I Q,} M Q] a}.

In this way, the image feature calculation units 111 and 121, for example, provide an overall feature vector composed of feature points represented within a large number of frames shown in FIG. 4A, and the query image _IQ shows the overall feature. as a vector _V G _{[I Q],} about certain search target image _{I n} the entire feature vector _V G _{[I n],} is comparably calculated one another.

Further, the image feature calculation units 111 and 121 are represented by, for example, a large number of 〇 frames in the one-dot broken line 〇 frame indicating the first symmetry group having the first target axis Sym1 in common as shown in FIG. 4 (b). and the detailed feature vectors constructed from the feature points, the query image _I as a detailed feature vector _V Sym _[I Q, 1] for _Q, one search target image detail feature vector _V Sym for _{I n [I} n, 1 ], Calculated so that they can be compared with each other.

Further, the image feature calculation units 111 and 121 are represented by, for example, a large number of 〇 frames within the one-dot broken line 〇 frame indicating the second symmetry group having the second target axis Sym2 in common as shown in FIG. 4 (b). and the detailed feature vectors constructed from the feature points, the query image _I as a detailed feature vector _V Sym _[I Q, 2] for _Q, one search target image detail feature vector _V Sym for _{I n [I} n, 2 ], Calculated so that they can be compared with each other.

(Image similarity calculation processing)
FIG. 3 is a flowchart showing the processing of the image similarity calculation unit 122 in the image retrieval device 1 of the embodiment according to the present invention.

Image similarity calculation unit 122, the query image and the image feature vector V _{[I Q]} about _{I Q,} the N frames stored in the image feature vector storage unit 112 searched images _{I 1, I 2, ...,} I N Compare the respective image feature vectors V [I ₁ ], V [I ₂ ], ..., V [ _IN ] with respect to their respective similarity degrees S (I _Q , I ₁ ), S (I _Q , I). ₂ ), ..., S (I _Q , _IN ) is calculated. For the calculation of the degree of similarity, a general one such as an inner product of vectors or a normalization of the Euclidean distance to 0 to 1 is used.

First, the image similarity calculation unit 122, an image feature vector V _{[I Q]} about the query image _{I Q,} search target image group _I 1 of N _sheets, I 2, ..., one search target image among the _{I N} I _n relates inputs the image feature vector V _{[I n]} (step S21).

Then, the image similarity calculation unit 122, the entire image similarity _{S G (I Q, I n} ) and an average detail image similarity as value _S Sym of similarity for each symmetry group _{(I Q,} I n) Is individually calculated.

First, the image similarity calculation unit 122, a query image _{I Q} and the image feature vector V entire feature vector _V G in _{[I Q] [I Q]} about the image feature vector V [I related to the search target image _{I n} by comparing the entire feature vector _V G _{[I Q]} in _n], the entire image similarity _{S G (I Q, I n} ) Request (step S22).

Second, the image similarity calculation unit 122 obtains detailed image similarity _{S Sym (I Q, I n} ) to the following procedure (Step S23).

First, the query image _ID subscripts set of symmetry groups of _{I Q Q = {1,2, ...} , M Q}, the search target image _{I ID} subscripts set of symmetry groups of _n n = {1, 2, ..., M _n }. _Here, M Q, _{M n} is the number of detailed feature vectors of each query image _{I Q} search target image _{I n.}

First, the initial value of m is set to 1.

(Process A) an image similarity calculation unit _{122, V sym [I Q, i} ] and _{V sym [I} n, j] similarity is maximum between i and j, respectively _ID Q, selected from ID _n , the degree of similarity _S Sym and _{(I Q, I n) [} m]. The calculation of the detailed image similarity is the same as the calculation of the overall image similarity.

(Process B) Next, the image similarity calculator _122, S _Sym checks whether the _{(I Q, I n) [} m] = 0, S Sym (I Q, I n) [m] If = 0, the process proceeds to "Process E".

(Process C) Next, the image similarity calculator _122, S _Sym not equal _{(I Q, I n) [} m] = 0, except for the i, j from the ID Q, ID _n, and m = m + 1 ..

(Process D) image similarity calculation unit _122, ID Q, until ID _n is an empty set, repeating the above (process A) ~ a (process _C), when the ID Q, is ID _n becomes an empty set, Move to "Process E".

(Process E) image similarity calculator _{122, S Sym (I Q, I} n) _{a, S Sym (I Q, I} n)) [1], ..., S Sym (I Q, I n) [m -1] is calculated as the average value. _However, S _Sym case of _{(I Q, I n) [} 1] = 0, S Sym (I Q, I n) and = 0.

In this manner, the image similarity calculator 122, detailed image similarity as the average value of the similarity of each symmetry group _{S Sym (I Q, I n} ) is determined.

Finally, the image similarity calculation unit 122, a query image _{I Q} and the image feature vector V _{[I Q]} regarding the image similarity S by comparison with the image feature vector V about the search target image _{I n [I n] (I} Q, _{I n)} of the entire image similarity _{S G (I Q, I n} ) and, detailed image similarity as the average value of the similarity of each symmetry group _{S Sym (I Q, I n} ) weighted of As an average, it is calculated by the following equation (step S24).
_{S (I Q, I n)} = (1-a) S G (I Q, I n) + aS Sym (I Q, I n)

However, a is a parameter that determines the contribution of the target axis (0 <a <1).

That is, in the example shown in FIG. 4 _(b), an _M Q ₌ M n ₌ 2, the similarity _S Sym detailed feature vectors in the first symmetry group _{(I Q, I n) [} 1] is _{V Sym} [ I _Q, 1] and _V Sym _[I n, 1] of similarity, _S Sym detailed feature vectors in the second symmetry group _{(I Q, I n) [} 2] and _V Sym _[I Q, 2] and it has a degree of similarity of the _{V Sym [I n, 2]} . Therefore, detailed image similarity _{S Sym (I Q, I n} ) is the average value of the similarity of each symmetry _{group, S Sym (I Q, I} n) = (S Sym (I Q, I n) _{[1] + S Sym (I} Q, n) [2]) / 2 to become.

In the example shown in FIG. 4B, in order to enhance the understanding of the present invention, the plurality of symmetry groups do not overlap each other, but in reality, the image search device 1 of the present embodiment extracts. Overlapping symmetry groups are allowed, which can prevent loss of contrasting local features and improve accuracy.

The image similarity calculation unit 122, detailed image similarity _{S Sym (I Q, I n} ) as the average value of the similarity of each symmetry group instead of obtaining the sum of the similarity of each symmetry group, i.e. _{, S Sym (I Q, I} n) = S Sym (I Q, I n) [1], ..., S Sym may be determined as the sum of _{(I Q, I n) [} m-1].

Alternatively, the image similarity calculation unit 122, detailed image similarity _{S Sym (I Q, I n} ) as the average value of the similarity of each symmetry group instead of obtaining the degree of similarity weighted average for each symmetric group , _{i.e., S Sym (I Q, I} n) = S Sym (I Q, I n) [1], ..., S Sym be calculated as a weighted average of _{(I Q, I n) [} m-1] good. As the weight at this time, if a value that is an index of the reliability of the axis of symmetry is obtained as a result of the algorithm for detecting the axis of symmetry, it can be used.

Thus, the similarity of the average value of each symmetry group, the additional value, or weighted average detail image similarity as value _{S Sym (I Q, I n} ) by calculating the similarity as the axis of symmetry is detected many It is possible to avoid the situation where the degree is low.

As described above, the image similarity calculation unit 122 has the image feature vector V [I _{1] for each of the N search target image groups I 1} , I ₂ , ..., IN stored in the image feature vector storage unit 112. _{], V [I 2],} ..., V [ for _{I N],} the query image _I by comparing an image feature vector V _{[I Q]} regarding _Q, respectively for N pieces of retrieval target image group similarity S ( _IQ , IQ ₁ ), S (I _Q , I ₂ ), ..., S (I _Q , _IN ) are calculated. _{As a result, the similar image extraction unit 123 automatically performs I d1} , I _d2 , ..., I _dR of a predetermined number of images having a high degree of similarity (in this example, the upper R images) among the N search target image groups. It can be selected and output as a search result to the outside (for example, a display device).

5 (a) and 5 (b) show the search results of the image search device 1 of the embodiment according to the present invention, and the search results of a typical conventional technique (that is, the similarity evaluation of only the overall feature vector). It is a figure which compares numerically. The datasets used in FIGS. 5 (a) and 5 (b) are images of one of them (FIG. 5) using the datasets for various structures including various butterflies and overpasses, which are generally disclosed, respectively. A butterfly for (a) and an overpass for FIG. 5 (b) were evaluated as query images, and other images were evaluated as search target images. Regarding symmetry, considering line symmetry, the above "Z. Tang, P. Monasse, and J. Morel," Reflexive Symmetry Detection in Single Image, "Proc. Of the 8th International Conference Curves and Surfaces, pp. 452- The method based on "460, 2014" is used.

In the butterfly data set shown in FIG. 5A, it is assumed that the feature vector considering symmetry is not used in the comparative example corresponding to the conventional technique (parameter a = 0), and the symmetry is considered in the embodiment according to the present invention. It was assumed that the feature vector was used (parameter a = 0.7).

Further, in the overpass data set shown in FIG. 5B, it is assumed that the feature vector considering symmetry is not used in the comparative example corresponding to the conventional technique (parameter a = 0), and the symmetry in the embodiment according to the present invention. It was assumed that the feature vector in consideration of the above was used (parameter a = 0.5).

Average precision AP fits rate up i th correct answer (= number of correct answers / total) as P _i, is calculated by the following equation. However, N is the total number of correct images. 0 ≦ AP ≦ 1, and the closer it is to 1, the better the search accuracy is.

As can be seen from the comparisons shown in FIGS. 5A and 5B, it is possible to search for an image of a subject having the same or high similarity as the subject of the query image with high accuracy. In particular, it is effective for subjects with symmetry, as the search results for butterflies are significantly better than conventional techniques over overpasses, and images often contain symmetry, so they are traditional in a wide range. The search accuracy can be improved more than the technique. Further, in FIGS. 5 (a) and 5 (b), the parameter a = 0.7 or 0.5 is set as an embodiment according to the present invention, respectively, but the values set by searching for the optimum numerical value as each search result. However, it was also confirmed that versatility can be obtained by setting the parameter a in the range of 0.5 to 0.7.

In the comparison shown in FIGS. 5A and 5B, in order to eliminate the subjectivity, the subject in the query image and the image of the "same subject" were evaluated with high accuracy, but the subject in the query image was evaluated. More effective results can be expected for the purpose of searching for images of "subjects with high same or high similarity" with high accuracy.

Inclusively, in the image search device 1 of the present embodiment, the image feature calculation units 111 and 121 first use an image feature vector in which an overall feature vector and a detailed feature vector are combined for each of the search target image and the query image. To calculate.

Here, the image feature calculation units 111 and 121 in the image search device 1 of the present embodiment have a feature point that the shading of the image changes in two directions by a predetermined value or more with respect to each of the search target image and the query image. , Information on any one or more of color, texture, and gradient in the peripheral region within the predetermined image size of each feature point is obtained as a vectorized local feature, and is predetermined for each of the search target image and the query image. The local feature is vector-quantified using a representative vector, and a histogram obtained by normalizing the appearance frequency of the vector-quantified local feature is obtained as an overall feature vector.

Further, the image feature calculation units 111 and 121 in the image search device 1 of the present embodiment enumerate a set of feature points whose distances in the feature space of the local feature are close to each of the search target image and the query image, and are the same. A symmetry group is extracted by grouping each set of feature points connected by symmetry, and a symmetry group in which the number of feature points exceeds a predetermined threshold, or a fixed number of symmetries in descending order of the number of feature points to which they belong. A detailed feature is a histogram in which a group is extracted, the local feature is vector-quantized for each symmetry group in the search target image using the predetermined representative vector, and the appearance frequency of the vector-quantified local feature is normalized. Obtained as a vector.

Then, in the image search device 1 of the present embodiment, the image similarity calculation unit 122 obtains the first similarity by comparing the overall feature vector in the search target image with the overall feature vector in the query image, and obtains the first similarity in the search target image. The second similarity is obtained by comparing the detailed feature vector with the detailed feature vector in the query image, and the similarity considering both the first similarity and the second similarity (weighting based on the parameter a in this example). The similarity between the search target image and the query image is evaluated by (average).

In particular, the image similarity calculation unit 122 determines the second similarity by comparing the detailed feature vector in the search target image with the detailed feature vector in the query image, and obtains the average value and the addition value of the similarity for each symmetry group. Alternatively, it is obtained by a weighted average value.

Then, the image search device 1 of the present embodiment evaluates the similarity of a predetermined number of search target images with the query image by the similar image extraction unit 123, and selects a predetermined number of search target images from the one with the highest degree of similarity. Automatically select and output to the outside.

As described above, the image search device 1 according to the present invention obtains the first similarity by comparing the overall feature vectors and the second similarity by comparing the detailed feature vectors, and obtains the first similarity and the second similarity. Since the similarity between the search target image and the query image is evaluated by the similarity considering both of the similarity of the above (in this example, the weighted average based on the parameter a), it is disclosed in, for example, Patent Document 3. It is possible to reflect the difference in the detailed structure and positional relationship of the subject in the image in the degree of similarity as compared with other conventional techniques such as the technique.

Further, the technique disclosed in Patent Document 3 is a technique of comparing the positions of feature points between the query image and the comparison image after performing region division based on the axis of symmetry, which is caused by the region division. However, the symmetry group extracted by the image search device 1 according to the present invention allows the duplication, thereby preventing the loss of the contrasting local features and the accuracy. Can be enhanced.

Further, the symmetry used in the technique disclosed in Patent Document 3 is limited to line symmetry, but the image search device 1 according to the present invention is also applied when a plurality of symmetries exist in one image. And one or more of line symmetry, rotational symmetry, and translational symmetry can be used to extract symmetry groups. As a result, in principle, it is possible to construct an image feature that reflects information on the detailed structure and positional relationship of the subject as compared with the image feature that has been conventionally used, and it is possible to improve the accuracy of similar image search. It becomes.

The image search device 1 according to the present invention is configured as a computer, and a program for realizing each component according to the present invention in the computer is stored in a memory (not shown) provided inside or outside the computer. Will be done. An image search device according to the present invention, appropriately reading a program in which processing contents for realizing the functions of each component are described from a memory under the control of a central processing unit (CPU) provided in the computer. The function of each component of 1 can be realized in a computer.

The present invention is not limited to the examples of the above-described embodiments, but is limited only by the description of the scope of claims. For example, in the example of embodiment described above, the N frames stored in the search image database 2 that is pre-specified as a predetermined image database to be connected to a network search target image group I _1, I 2, _..., a I _N contrast, the image of the query image I _Q of the subject and the same or highly similar object has been described an example of searching with high accuracy, it can be searched for images on browsable network via the network. For example, when searching for images on a network that can be browsed, randomly collected images may be stored in the search target image database 2.

According to the present invention, it is possible to search for an image of a subject that is the same as or highly similar to the subject of the query image with high accuracy, which is particularly effective for a subject having symmetry, and searches in a wider range than the conventional technique. Since the accuracy can be improved, it is useful in a material search in archives, a material search in video production, or an image search engine on the Web.

1 Image search device 2 Search target image database 11 Preprocessing unit 12 Search processing unit 111 Image feature calculation unit 112 Image feature vector storage unit 121 Image feature calculation unit 122 Image similarity calculation unit 123 Similar image extraction unit

Claims (7)

An image search device that searches for images of subjects that are the same as or highly similar to the subjects in the query image from the image database to be searched or images on the network.
An image feature calculation means that calculates an image feature vector that combines an overall feature vector and a detailed feature vector for each of the search target image and the query image.
The first similarity is obtained by comparing the overall feature vector in the search target image with the overall feature vector in the query image, and the detailed feature vector in the search target image is compared with the detailed feature vector in the query image. An image similarity calculation means for obtaining the similarity of 2 and calculating the similarity in consideration of both the first similarity and the second similarity.
Similar image extraction to search and extract images of subjects that are the same as or highly similar to the subject in the query image by evaluating the similarity between the search target image and the query image based on the calculated similarity. With means ,
The image feature calculation means has a feature point in which the shade of the image changes by a predetermined value or more in two directions with respect to each of the search target image and the query image, and the peripheral region within the predetermined image size of each feature point. Information on any one or more of the color, texture, and gradient in the above is obtained as a vectorized local feature, and the local feature is vector-quantized using a predetermined representative vector for each of the search target image and the query image. However, the image retrieval apparatus according to claim Rukoto a histogram obtained by normalizing the frequency of the vector quantized local features, as the whole feature vector. The image feature calculation means enumerates a set of feature points whose distances in the feature space of the local feature are close to each of the search target image and the query image, and for each set of feature points linked by the same symmetry. Grouping is performed to extract symmetry groups, a symmetry group in which the number of feature points exceeds a predetermined threshold, or a certain number of symmetry groups in descending order of the number of feature points to which they belong is extracted, and the predetermined representative vector is obtained. It is characterized in that the local feature is vector-quantified for each symmetry group in the search target image, and a histogram obtained by normalizing the appearance frequency of the vector-quantified local feature is obtained as the detailed feature vector. The image search device according to claim 1. The image similarity calculation means obtains the second similarity by comparing the detailed feature vector in the search target image with the detailed feature vector in the query image, and obtains the average value and the addition value of the similarity for each symmetry group. The image search device according to claim 1 or 2 , wherein the image search device is obtained by a weighted average value. The image similarity calculation means is characterized in that the similarity between the search target image and the query image is evaluated by a weighted average as the similarity considering both the first similarity and the second similarity. The image search device according to any one of claims 1 to 3. The similar image extraction means evaluates the similarity of a predetermined number of search target images with the query image, automatically selects a predetermined number of search target images from those having a high degree of similarity, and outputs the image to the outside. The image search device according to any one of claims 1 to 4, which is characterized. The image feature calculation means allows the extraction of a plurality of symmetry groups at the time of extracting the symmetry groups, and is connected by the same symmetry using one or more of line symmetry, rotational symmetry, and translational symmetry. The invention according to claim 2, wherein the grouping is performed for each set of feature points and the plurality of symmetry groups are extracted, or claim 3 to claim 2 is cited . Image search device. A program for operating the image search device according to any one of claims 1 to 6 as a computer.

Images