JP4374902B2 - Similar image search device, similar image search method, and similar image search program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a technical field of similar image search in which, when a plurality of query images are inputted, a similar image is searched from search target images in a database for each query image.
Specifically, in order to prevent unauthorized use, such as photos and characters of popular idols, or original images with copyrights such as company logos posted on the Internet as imitation images synthesized with other images, An object is to search for an image similar to the original image from images existing on the Internet and detect a candidate for an illegally used image. In addition, when query images from a plurality of clients are generated as search requests at the same time, it is applied to the field of similar image search in the server client method, such as a case where similar image search of a plurality of query images is performed on the server side. .
[0002]
[Prior art]
Many similar image search techniques for searching for similar images from search target images with respect to an inquiry image have been disclosed (for example, Patent Document 1). These conventional techniques are effective techniques for searching similar images for a single query image, but for searching similar images for multiple query images, each query image It is necessary to repeat similar image search independently, and the search time is very long and the cost effectiveness is remarkably deteriorated.
[0003]
In this way, when multiple query images exist and similar image search is performed on each query image, the search results of other query images are also used instead of repeatedly performing each query independently. No method has been proposed to improve the overall search efficiency.
[0004]
[Patent Document 1]
JP 2000-29885 A (pages 2 to 3)
[0005]
[Problems to be solved by the invention]
As described above, in order to search each similar image with respect to a plurality of query images using the conventional technique for performing similar image search with respect to a single query image, similar images are independently used for each query image. The search needs to be repeated, so that the time required for the search takes a time substantially proportional to the number of inquiry images, and when the number of inquiry images increases, a long time is required for execution.
[0006]
Accordingly, an object of the present invention is to create an index for a plurality of query images and use the index of the query images to speed up and improve the efficiency of similar image search for a plurality of query images.
[0007]
[Means for Solving the Problems]
A first invention is a similar image retrieval device for retrieving similar images in response to input of a plurality of inquiry images, and a target image for extracting a feature amount from a target image in an image database in which the retrieval images are stored in advance. Clustering is performed to divide images having similar image feature quantities into the same cluster using the feature quantity extraction means and the image feature quantity extracted by the target image feature quantity extraction means, and the image features in the cluster for each cluster and the target index generation means for generating an average vector of an amount as an index, the query image feature extraction means for extracting an image feature quantity of each image from the plurality of query images set by the user input, in the query image feature extraction means clustering is performed to divide an image between the image characteristic amount is similar to the same cluster using the extracted image feature amount, the cluster To a query index generation means for generating an average vector of the image feature amount in the cluster as an index, in units of each of the clusters, the similarity between the query was Lee Ndekusu said pair Zoi Ndekusu When the similarity exceeds a preset index similarity threshold, the similarity between the target image and the query image is calculated, and the preset image similarity threshold is determined to determine the similarity image The present invention relates to a similar image search apparatus comprising: a similar image search means for searching for a search result; and a search result output means for outputting a result obtained by the similar image search means.
[0008]
That is, according to the first invention, by adopting the above-described configuration, a plurality of images that become user query images are subjected to similar image search using an index that is associated with an image feature amount. As compared with the case of searching by comparison, the number of images to be searched can be greatly reduced. As a result, the time required for searching is shortened and the cost effectiveness is remarkably improved.
[0009]
Further, the similar image retrieval means, characterized by limiting the search range of the similar image by calculating a similarity index index and the target image of the query image. That is, by comparing the query index and the target index each other by calculating the degree of similarity, it is possible to limit the search range. As a result, the number of direct comparisons between the images is reduced, and a higher-speed similar image search can be realized.
[0010]
The second invention relates to the similar image retrieval apparatus according to the first invention, wherein the target index generation means or the query index generation means generates an index hierarchically.
That is, according to the second invention, the target index generating means or the query index generating means generates the index hierarchically.
With this configuration, an index for the search target image or an index for the query image is generated hierarchically, and by using such a hierarchical index, it is possible to reduce the search area in logarithmic order, and a faster similarity Image search is realized.
[0011]
As described above, in the present invention, an index necessary for speeding up the search is generated from a plurality of query images, and an index necessary for speeding up the search is generated from the search target image. By using these indexes, the number of executions of similarity calculation at the time of similar image search is greatly reduced.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 shows a basic system configuration according to an embodiment of the present invention. In the similar image search system of the present invention, an image database 2 as an external storage device and a search device 1 having a query image database 2 ′ to be temporarily stored as a buffer in an internal memory and a user terminal 3 are connected via the Internet 4. A similar search is performed on various query images from the user.
[0013]
Further, the search device 1 generates a target index using the target image feature amount extraction unit 11 that extracts the feature amount of the target image from the image database 2 in which a large amount of search images are stored in advance, and the target image feature amount. Target index generation means 12, query image feature quantity extraction means 13 that extracts the query image feature quantity in response to input of the query image by the user, and query index generation means that generates a query index using the feature quantity of the query image 14. It is comprised by the similar image search means 15 which searches the similar image with respect to the inquiry image from a user from the image database 2, and the search result output means 16 which outputs the searched result.
[0014]
Further, in the image database 2, a target image that is a search target collected in advance from the Internet, an image feature amount of the target image extracted by the target image feature amount extraction unit 11, and a target index generation unit 12 are generated. Stores the target index. In the query image database 2 ′, the query image from the user, the image feature amount extracted by the query image feature amount extraction unit 13, and the query image generated by the query index generation unit 14 are temporarily stored in the internal memory. Stored.
[0015]
Here, the search device 1 is a computer, and a pre-installed program is executed on the computer, and the target image feature amount extraction unit 11, the target index generation unit 12, the query image feature amount extraction unit 13, and the query index generation unit. 14, similar image search means 15 and search result output means 16 are realized. The program is recorded on a computer-readable recording medium such as a floppy disk, a compact disk, or a CD-ROM. Although not shown in the drawing, the program is installed in a built-in or externally connected medium reader. It is good also as an executable state.
[0016]
Next, each means for performing a similar image search in the search device 1 of the present invention will be described in detail with reference to FIGS.
<Target Image Feature Extraction Unit 11>
The target image feature amount extraction unit 11 first selects an image to be searched for similar image search, and extracts predetermined image feature amounts such as color features (hue, brightness, saturation), shape, size, and the like from each image. To do. There are basically a plurality of images to be searched, and an image file saved in a computer, an image file collected from an intranet or the Internet, and the like can be used as a search target image.
[0017]
For example, a color histogram or the like is applied as the image feature amount. The color histogram is expressed in vector form as a value obtained by counting the number of pixels belonging to each partial space of a color space obtained by dividing a color space expressed in RGB (red, green, blue), etc. by an appropriate range. Is.
<Target index generation means 12>
The target index generation unit 12 creates an index of the search target image using the image feature amount of the target image extracted by the target image feature amount extraction unit 11.
[0018]
As a method of creating an index, for example, a method of performing clustering that divides images having similar image feature quantities into the same cluster, and holding a representative value of each cluster, for example, an average vector of image feature quantities in the cluster, as an index There is. Clustering is to classify given data into a plurality of groups (clusters) according to a certain evaluation criterion. As a representative technique for the clustering, there is a K-means method.
[0019]
The algorithm is as follows.
1) K initial cluster centers are appropriately determined.
2) Classify all images into the nearest cluster-centered cluster.
3) The center of gravity of the newly created cluster is set as the cluster center.
4) If all new cluster centers are the same as before, finish. Otherwise, return to 2).
[0020]
In addition to the above K-means method, a method of performing clustering by generating indexes such as hierarchical indexes, multidimensional indexes, and genre-specific indexes can be considered.
In the hierarchical index method, based on the index generation result obtained by the K-means method, a hierarchical index structure is formed by repeating several steps of clustering, enabling faster image retrieval. And
[0021]
In addition, the multi-dimensional index method is a color representation using BGR (blue, green, red) or HSI (hue, saturation, luminosity), compared to a one-dimensional method of comparing similarity with scalar values such as words and numerical values. ), A multidimensional index such as a color histogram with the color feature as an axis is generated, thereby enabling a high-precision and high-speed similar image search.
[0022]
Furthermore, the genre-based indexing method is a method of creating genre-specific indexes such as “talent”, “business person”, “corporate logo”, and “advertisement”. This is effective when performing similar operations and enables efficient similar image retrieval.
<Inquiry Image Feature Extraction Unit 13>
The inquiry image feature amount extraction unit 13 sets a plurality of inquiry images in response to input from the user, and extracts the feature amount of each image used for similar image search. Through the subsequent processing, an image similar in feature amount to the image set as the inquiry image is searched from the search target images and output. A similar image search result is output for each set inquiry image.
[0023]
FIG. 2 shows a schematic diagram of inquiry image setting according to the embodiment of the present invention. In the inquiry image setting screen in the figure, the displayed image is set as an inquiry image by pressing a “set” button displayed below. In this example, a case where a plurality of inquiry images are set in a batch is shown, but it is of course possible to set each inquiry image individually if necessary.
[0024]
The inquiry image feature amount extraction means 13 extracts a predetermined image feature amount from each of the set inquiry images. A color histogram or the like is applied as the image feature amount.
<Inquiry index generation means 14>
The inquiry index generation means 14 creates an index using the image feature amount of the inquiry image extracted by the inquiry image feature amount extraction means 13. By referring to this index, it is possible to determine the area to be searched, and since the search for unnecessary areas is omitted, the efficiency is greatly improved.
[0025]
FIG. 3 is a schematic diagram of query index generation according to the embodiment of the present invention. Similar images are included in the same cluster, and dissimilar images are included in different clusters, and an index is created for each cluster.
The method for creating the index is the same as the example described in the target index generation means 12.
<Similar image search means 15>
The similar image search means 15 searches for a search target image whose image feature quantity is similar to the query image. In order to perform the similar image search, it is necessary to calculate the similarity between the image feature amount of the inquiry image and the image feature amount of the search target image. The processing for calculating the similarity for each of the inquiry image and the search target image increases the calculation cost as the number of images increases.
[0026]
Therefore, in the present invention, it is possible to reduce the number of times of calculating the similarity by using the indexes generated in the query index generation unit and the target index generation unit. In similar image search, by using each created index, the cluster containing similar images is examined, the similarity with the images in the cluster is calculated, and the similarity with the images in the cluster containing dissimilar images The degree is not calculated.
[0027]
Hereinafter, the similarity calculation according to the present invention will be described in comparison with the prior art.
FIG. 4 shows a similar image search for a plurality of conventional query images. This figure shows an example of four inquiry images and six search target images. In the conventional method that does not generate an index, when the images are combined, it is necessary to execute the process of calculating the similarity 4 × 6 = 24 times.
[0028]
On the other hand, FIG. 5 shows an example of similar image search for a plurality of query images using an index according to the embodiment of the present invention. In this figure, four query images and six search target images are taken as an example. The query image is composed of clusters A and B each composed of two images, and each search target image is composed of three images. This shows a case where the cluster is composed of clusters α and β and two query indexes and two target indexes are generated.
[0029]
Consider a case in which an inquiry image belonging to cluster A is searched.
1) If the similarity between the index of cluster A and the two target indexes has not been calculated, the similarity S is calculated and stored in the image database 2. If the similarity is calculated when searching for another query image, the similarity is not required to be calculated by using the stored result.
2) When the similarity S satisfies the condition, the cluster corresponding to the target index is set as the search area. However, when the condition is not satisfied, the cluster corresponding to the target index is excluded from the search area.
3) The similarity S between the search target image in the search area and the query image is calculated. The obtained similarity is stored in the image database 2.
4) When the similarity S satisfies the condition, the search target image is output as a result of the similar image search.
[0030]
Thus, by using the query index and the target index, it is not necessary to calculate the similarity for all combinations (24 times in FIG. 4) of the query image and the search target image. In the example of FIG. 5, the number of similar calculations can be reduced to, for example, 2 × 2 (indexes) + 2 × 3 × 2 (images) = 16.
<Search result output means 16>
The search result output means 16 outputs a similar image for each inquiry image searched by the similar image search means.
[0031]
FIG. 6 is a schematic diagram of search result output according to the embodiment of the present invention. A plurality of query images set on the left side of the figure are displayed, and search result images of similar image searches for the respective query images are displayed on the right side.
In the search result output described above, similar results are sequentially output when the search result for the query image is completed. This makes it possible to use the results of the query images at an early stage in the order in which the search is completed without waiting until search results for all the query images are obtained. The similar image search result can be used for another search by storing it in the database.
[0032]
Further, as a method for limiting the range of similar images to be output for each inquiry, the following can be mentioned.
1) Set the number of similar images to be output for each inquiry image.
2) A threshold is set for the similarity for each inquiry image, and a similar image within the threshold is output.
3) The total number of similar images to be output is set for all inquiry images.
[0033]
Such a method is conceivable.
As a similar image output format, a method of displaying each query image in a viewable format as an image or a method of outputting a similar image file name in a text format can be considered. And it is also possible to display the similarity of each similar image as a result as needed.
[0034]
FIG. 7 shows a similarity threshold table according to the embodiment of the present invention. FIG. 7A shows an index similarity threshold table, and FIG. 7B shows an image similarity threshold table. Both the index threshold and the image similarity threshold indicate similarity values of 0.5, 0.7, etc., depending on each case.
The similar image search process using each means in the search apparatus 1 will be described below with reference to the flowcharts of FIGS.
[0035]
FIG. 8 shows a flowchart of search object image storage according to the embodiment of the present invention. First, in step S11, the target image feature amount extraction unit 11 selects an image to be searched from the image database 2 stored in advance, and extracts an image feature amount. Next, in step S12, the target index generation means 12 generates a target index using the image feature amount extracted in the previous step, and stores the image feature amount and the target index data in the image database 2 in step S13. To do.
[0036]
FIG. 9 shows a flowchart of the similar image search according to the embodiment of the present invention. First, in step S21, the inquiry image feature amount extraction unit 13 receives data of a plurality of inquiry images input by the user, and extracts the feature amount of each image.
In step S22, a query index is generated using the query image feature quantity. In step S23, the target index of the image database 2 is referred to. In step S24, the similarity between the query index and the target index is calculated. In step S25, with reference to the index similarity value table defined in FIG.
[0037]
If it exceeds the index similarity threshold, the similarity between the inquiry image and the target image is calculated in step S26. In step S27, the image similarity threshold value table defined in FIG. 7B is referred to and it is determined whether or not the threshold regulation conditions match. If the specified conditions are met, a satisfactory similar image is output as a search result. If the conditions are not met, the process proceeds to the next calculation of the similarity of the target image until all the processes are completed.
[0038]
If it is determined in step S25 that the index similarity is not met, the process skips to step S30 and proceeds to the calculation of the similarity of the next target index until all the target indexes are completed.
As described above, as an example of introducing a query index and a target index as a system for performing a similar search efficiently and efficiently with respect to a plurality of query images, an index for clustering the target images is not generated, and individual images are not generated. Needless to say, it is possible to increase the search speed even if only the query index is applied without retaining the feature amount as it is and providing the target index on the database side.
[0039]
(Supplementary Note 1) A similar image search device for searching for similar images for input of a plurality of inquiry images,
A target image feature amount extraction means for selecting a target image from an image database in which search images are stored in advance and extracting the feature amount;
Target index generation means for generating an index for associating a plurality of target images using the image feature quantity extracted by the target feature quantity extraction means;
Query image feature quantity extraction means for extracting image feature quantities from a plurality of query images set and input by the user;
Query index generation means for generating an index for associating the query image using the image feature quantity extracted by the query image feature quantity extraction means;
Similar image search means for searching for similar images from search target images by comparing each index generated for the target image and the query image;
A similar image search apparatus comprising: a search result output means for outputting a result obtained by the similar image search means.
[0040]
(Supplementary Note 2) The similar image search according to supplementary note 1, wherein the similar image search means limits a search range of similar images by calculating a similarity between the index of the inquiry image and the index of the target image. apparatus.
(Supplementary Note 3) The similar image search device according to Supplementary Note 1 or 2, wherein the target index generation unit or the query index generation unit generates indexes hierarchically.
[0041]
(Additional remark 4) The said similar index search apparatus of Additional remark 1 characterized by the said target index production | generation means not producing | generating the index which clusters a search object image, but holding | maintaining the image feature-value of each image as it is.
(Supplementary note 5) The similar image search device according to supplementary note 1, wherein the target index generation means generates an index using a multidimensional index.
[0042]
(Additional remark 6) The said similar image search apparatus of Additional remark 1 characterized by the said target image feature-value extraction stage dividing a search target image into a some genre, and extracting.
(Supplementary Note 7) A similar image search method for searching for similar images with respect to input of a plurality of inquiry images,
A target image feature amount extraction step of selecting a target image from an image database in which search images are stored in advance and extracting the feature amount;
A target index generation step of generating an index for associating a plurality of target images using the image feature amount extracted in the target image feature amount extraction step;
An inquiry image feature extraction step for extracting image features from a plurality of inquiry images set and input by the user;
A query index generation step for generating an index for associating the query image using the image feature amount extracted by the query image feature amount extraction step;
A similar image search step of searching for similar images from search target images by comparing each index generated for the target image and the query image;
And a search result output step of outputting a result obtained by the similar image search step.
[0043]
(Supplementary Note 8) A similar image search program for searching for similar images in response to input of a plurality of inquiry images,
On the computer,
A target image feature amount extraction step of selecting a target image from an image database in which search images are stored in advance and extracting the feature amount;
A target index generation step of generating an index for associating a plurality of target images using the image feature amount extracted in the target image feature amount extraction step;
An inquiry image feature extraction step for extracting image features from a plurality of inquiry images set and input by the user;
A query index generation step for generating an index for associating the query image using the image feature amount extracted by the query image feature amount extraction step;
A similar image search step of searching for similar images from search target images by comparing each index generated for the target image and the query image;
A similar image search program for executing a search result output step for outputting a result obtained by the similar image search step.
[0044]
【The invention's effect】
As described above, in the present invention, by using the index of the query image and the index of the image to be searched, the similarity can be calculated and compared, thereby greatly narrowing the search range. A high-speed similar image search is realized for a plurality of input query images. In addition, by speeding up similar image search for a plurality of query images in this way, even if the number of query images is increased, a rapid increase in execution time required for the search is suppressed, and a large number of query images can be searched. Similar image search is realized in real time.
[Brief description of the drawings]
FIG. 1 is a diagram showing a basic system configuration according to an embodiment of the present invention.
FIG. 2 is a diagram showing a schematic diagram of inquiry image setting according to the embodiment of the present invention.
FIG. 3 is a diagram showing a schematic diagram of query index generation according to the embodiment of the present invention.
FIG. 4 is a diagram illustrating a similar image search for a plurality of conventional query images.
FIG. 5 is a diagram showing an example of similar image search for a plurality of query images using an index according to the embodiment of the present invention.
FIG. 6 is a diagram showing a schematic diagram of search result output according to the embodiment of the present invention.
FIG. 7 is a diagram showing a similarity threshold table according to the embodiment of the present invention.
FIG. 8 is a diagram showing a flowchart of storing a search target image according to the embodiment of the present invention.
FIG. 9 is a flowchart of a similar image search according to the embodiment of the present invention.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Search apparatus 2 Image database 2 'Query image database 3 User terminal 4 Internet 11 Target image feature-value extraction means 12 Target index generation means 13 Query image feature-value extraction means 14 Query index generation means 15 Similar image search means 16 Search result output means

Claims (4)

A similar image search device for searching for similar images in response to input of a plurality of inquiry images,
A target image feature amount extracting means for extracting a feature amount from a target image in an image database in which search images are stored in advance;
Clustering is performed to divide images having similar image feature quantities into the same cluster using the image feature quantities extracted by the target image feature quantity extraction unit , and an average vector of the image feature quantities in the cluster is indexed for each cluster. and the target index generation means for generating a,
Query image feature quantity extraction means for extracting the image feature quantity of each image from a plurality of query images set and input by the user;
Clustering is performed to divide an image between the image characteristic amount is similar with the image characteristic amount extracted by the query image feature extraction means in the same cluster, each cluster, an index of the average vector of the image feature amount in the cluster A query index generating means for generating
Each of said cluster as a unit, when it exceeds the index similarity threshold similarity calculated, the similarity is set in advance between the pair Zoi Ndekusu and the query was Lee Ndekusu, the subject Similar image search means for calculating a similarity between an image and an inquiry image, determining a preset image similarity threshold, and searching for a similar image;
A similar image search apparatus comprising: a search result output means for outputting a result obtained by the similar image search means.   The similar image retrieval apparatus according to claim 1, wherein the target index generation unit or the query index generation unit generates the index hierarchically. A similar image retrieval method for retrieving similar images in response to input of a plurality of inquiry images,
A target image feature amount extraction step for extracting a feature amount from a target image in an image database in which search images are stored in advance;
Clustering is performed to divide images having similar image feature quantities into the same cluster using the image feature quantities extracted in the target image feature quantity extraction step , and an average vector of the image feature quantities in the cluster is indexed for each cluster. and the target index generation step of generating as,
An inquiry image feature amount extraction step for extracting an image feature amount of each image from a plurality of inquiry images set and input by the user;
Clustering is performed to divide an image between the image characteristic amount is similar with the image characteristic amount extracted by the query image feature amount extraction step to the same cluster, each cluster, an index of the average vector of the image feature amount in the cluster a query index generation step of generating as,
Each of said cluster as a unit, when it exceeds the index similarity threshold similarity calculated, the similarity is set in advance between the pair Zoi Ndekusu and the query was Lee Ndekusu, the subject A similarity image search step of calculating a similarity between the image and the inquiry image, determining a preset image similarity threshold, and searching for a similar image;
And a search result output step of outputting a result obtained by the similar image search step. A similar image search program for searching for similar images in response to input of a plurality of inquiry images,
On the computer,
A target image feature amount extraction step for extracting a feature amount from a target image in an image database in which search images are stored in advance;
Clustering is performed to divide images having similar image feature quantities into the same cluster using the image feature quantities extracted in the target image feature quantity extraction step , and an average vector of the image feature quantities in the cluster is indexed for each cluster. and the target index generation step of generating as,
An inquiry image feature amount extraction step for extracting an image feature amount of each image from a plurality of inquiry images set and input by the user;
Clustering is performed to divide an image between the image characteristic amount is similar with the image characteristic amount extracted by the query image feature amount extraction step to the same cluster, each cluster, an index of the average vector of the image feature amount in the cluster a query index generation step of generating as,
Each of said cluster as a unit, when it exceeds the index similarity threshold similarity calculated, the similarity is set in advance between the pair Zoi Ndekusu and the query was Lee Ndekusu, the subject A similarity image search step of calculating a similarity between the image and the inquiry image, determining a preset image similarity threshold, and searching for a similar image;
A similar image search program for executing a search result output step for outputting a result obtained by the similar image search step.

Images