JP2011070244A - Device, method and program for retrieving image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate ranking of similar image retrieval considering a retrieval intention of a user based on a query image. <P>SOLUTION: In an image retrieval device, a similarity integration part 50 performs weighting to similarity by similar object retrieval and similarity by similar area retrieval by use of the number of characteristic areas detected from the query image by a characteristic vector generation part 30c of a similar object retrieval part 30 and a division number of dividing the query image by a characteristic vector generation part 40c of a similar area retrieval part 40, and ranks respective retrieval target images by integrated similarity. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image search apparatus, an image search method, and a program for searching for similar images based on feature values of images.

  By inputting an image as a search key and comparing image feature amounts (represented by quantifying image features such as color scheme, texture, and shape), an image that is a search key (hereinafter referred to as a “query image”) A technique for searching for an image similar to is known. When the user inputs a query image, similar images are searched from the image database storing the search target images, and search results ranked by similarity are obtained.

  As the search intention of the user when performing the similar image search, a plurality of intentions are assumed depending on the target image. First, for example, the search intention is to search for an image including an object similar to an object (an object such as a person, a thing, or a character) in a query image. In this case, the image search is performed based on the feature amount extracted by paying attention to the object in the image.

  In addition, there is a search intention to search for an image having a configuration similar to that of the entire query image. In this case, an image search is performed based on the feature amount of each area constituting the image.

  In this manner, when there are a plurality of search intentions of the user, it is necessary to switch the image search algorithm according to the search intention. However, since it is difficult to determine the user's search intention from only the query image, there is a technique for performing an image search using a plurality of search algorithms and integrating the similarities (scores) obtained by the search to obtain a search result. It has been devised (for example, see Patent Document 1).

Japanese Patent Laid-Open No. 2001-160057

  By the way, when integrating similarities based on a plurality of search algorithms as in the technique of Patent Document 1 described above, the similarity for each search algorithm is considered evenly unless the user designates the search intention. The similarity of the other side affects the ranking by the algorithm.

  For example, in the case of a query image including an object, it is desirable that the image including the object is ranked higher, but if the similarity focused on the region is integrated into this ranking, the same object as the query image is not included. Images with similar color schemes and patterns of the entire image are ranked higher.

  As described above, when the similarities based on a plurality of search algorithms are simply integrated, each search algorithm affects the ranking, so it is difficult to obtain a search result with a ranking that considers the user's search intention. .

  In some cases, a combination of a plurality of images is specified as a query. In this case, similar image searches are performed for each query image using a plurality of search algorithms, and the search results are integrated, but the integration of the search results is complicated.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to generate a ranking of similar image searches in consideration of a user's search intention based on a query image.

In order to achieve the above object, the first invention provides:
In an image search apparatus for calculating a similarity between a search target image stored in an image database and a query image and outputting a search result based on the similarity,
Feature region detecting means for detecting a feature region indicating the presence of an object from the query image;
Object vector generation means for generating a feature vector of the query image based on the detected feature region;
Object similarity calculation means for calculating the similarity between the query image and the image to be searched based on the feature vector generated by the object feature generation means;
Area dividing means for dividing the query image into areas;
Area vector generation means for generating a feature vector of the query image based on each of the divided image areas;
Area similarity calculating means for calculating the similarity between the query image and the search target image based on the feature vector generated by the area vector generating means;
The similarity calculated by the object similarity calculating unit and the similarity calculated by the region similarity calculating unit, the number of feature regions detected by the feature region detecting unit, and the region by the region dividing unit A similarity integration unit that performs weighting based on the number of divisions and integrates the similarity for each image to be searched;
It is characterized by having.

  According to the first invention, the similarity based on the feature vector generated based on the feature region based on the number of feature regions detected from the query image and the number of region divisions of the query image, and the region-divided image The similarity based on the feature vector generated based on the region is weighted. That is, when the number of feature areas indicating the presence of an object in the query image is large, weighting on the similarity by the feature vector generated based on the feature area is increased, and when there are few feature areas, the image area The weight of similarity based on the feature vector generated based on is increased.

  For this reason, it is possible to weight each similarity based on the query image to indicate whether the search intention is focused on the presence of the object or the search intention focused on the area configuration of the entire image. Therefore, it is possible to generate a ranking of similar image searches in consideration of the user's search intention based on the query images.

In the second invention, the area dividing means includes
The number of region divisions is determined based on the number of feature regions detected by the feature region detection means.

  For this reason, since the number of region divisions of the query image is determined based on the number of feature regions detected from the query image, the region is divided finely when a large number of feature regions are detected, and roughly divided when the number of feature regions is small. The accuracy of area division can be changed according to the number of feature areas.

In the third invention,
When the query image is an image generated by cutting out a part of the image,
The similarity integration unit includes:
Normalizing the feature region detected by the feature region detection unit from the query image based on the number of feature regions detected from the image before clipping, and integrating the similarity using the normalized number of feature regions; It is a feature.

  For this reason, by normalizing the number of feature regions of the query image based on the number of feature regions of the original image before cutting out the query image, weighting can be performed as the probability that the query image includes an object. it can.

In addition, the fourth invention is
In an image search method in which a computer calculates a similarity between a query image and a query image stored in an image database, and outputs a search result based on the similarity.
A feature region detection step of detecting a feature region indicating the presence of an object from the query image;
An object vector generation step of generating a feature vector of the query image based on the detected feature region;
An object similarity calculation step of calculating a similarity between the query image and the search target image based on the feature vector generated in the object feature generation step;
Area dividing means for dividing the query image into areas;
A region vector generation step of generating a feature vector of the query image based on each of the divided image regions;
A region similarity calculation step of calculating a similarity between the query image and the search target image based on the feature vector generated in the region vector generation step;
The similarity calculated in the object similarity calculation step, the similarity calculated in the region similarity calculation step, the number of feature regions detected in the feature region detection step, and the region in the region division step A similarity integration step of integrating the similarities for each image to be searched by performing weighting based on the number of divided divisions;
Is performed by the computer.

  According to a fifth aspect of the present invention, there is provided an image search program that causes a computer to execute the image search method according to the fourth aspect of the present invention.

  According to the fourth and fifth inventions, as in the first invention, the weighting indicating whether the search intention is focused on the presence of an object or the search intention focused on the area configuration of the entire image is based on the query image. Therefore, it is possible to generate a ranking of similar image search considering the user's search intention based on the query image.

  According to the present invention, it is possible to generate a ranking of similar image search in consideration of a user's search intention based on a query image.

The block diagram which shows the function structure of the image search device which concerns on this invention. The flowchart which shows the processing flow of the indexing process in a similar object search. The flowchart which shows the processing flow of the indexing process in a similar area | region search. The figure for demonstrating the outline | summary of the indexing by a visual keyword. The flowchart which shows the processing flow of a search process (an object similarity calculation process, an area similarity calculation process, a similarity integration process). The figure which shows the example of extraction of the characteristic area | region with respect to a query image, and the example of area division. The approximation graph about the probability that an object will be included in a query image.

[Configuration of image search device]
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a functional block diagram of an image search apparatus 1 to which the present invention is applied. A client terminal (not shown) is connected to the image search apparatus 1 via a communication network (such as the Internet or a telephone line network) so that data communication can be performed.

  The image search apparatus 1 receives a query image transmitted from a client terminal such as a personal computer or a mobile terminal connected via a communication network as a search request. Then, a similar image search is performed according to the search request, and search results ranked in the order of similarity are returned to the client terminal.

  The image search apparatus 1 according to the present embodiment uses an image search method by indexing image areas based on visual keywords. An image search using visual keywords represents an image as a set of a plurality of image areas, and an image index (feature) based on a feature amount obtained from image areas (hereinafter referred to as “partial images” as appropriate) constituting each image. Vector), which can be said to be an application of text search technology for obtaining feature values of sentences from keywords in text.

For this reason, in image search using visual keywords, text search technology (transposition index, vector space model, word appearance frequency, etc.) is applied to image region search by treating the image region in the image as a keyword. Scale and high speed can be realized. As reference technical literature on image search using visual keywords,
Sivic and Zisserman: “Efficient visual search for objects in videos”, Proceedings of the IEEE, Vol.96, No.4, pp.548-566, Apr 2008.
・ Yang and Hauptmann: “A text categorization approach to video scene classification using keypoint features”, Carnegie Mellon University Technical Report, pp. 25, Oct 2006.
・ Jiang and Ngo: “Bag-of-visual-words expansion using visual relatedness for video indexing”, Proc. 31 ^st ACM SIGIR Conf., Pp.769-770, Jul 2008.
・ Jiang, Ngo, and Yang: “Towards optimal bag-of-features for object categorization and semantic video retrieval”, Proc. 6th ACM CIVR Conf., Pp.494-501, Jul. 2007.
・ Yang, Jiang, Hauptmann, and Ngo: “Evaluating bag-of-visual-words representations in scene classification”, Proc. 15 ^th ACM MM Conf., Workshop on MMIR, pp.197-206, Sep. 2007.
Etc.

  Also, by expressing a single image as a set of multiple region images, unlike general similar image search, search using a partial image obtained by cutting out a part of an image with an arbitrary size or position as a query image is possible. It becomes possible. For this reason, in order to obtain a desired search result, the user can express a query more directly and accurately by an operation such as designating a part of an image.

In the present embodiment, the following two image searches are used as an application of the image region search technique using visual keywords.
First, a similar object search is performed for searching for a similar image including an object similar to the object included in the query image. An object is an object having a shape such as a person or an object. In this search, a specific object needs to be included in the query image.
The other is a similar region search that searches for a similar image that includes the same image region as the query image or an image region that is visually similar and has similar overall characteristics. In this search, the query image does not need to include a specific object, and may be a combination of a plurality of images.

  The image search apparatus 1 includes search algorithms corresponding to the similar object search and the similar area search, performs each search process on the query image, and outputs a search result.

  As shown in FIG. 1, the image search device 1 includes a query image reception unit 10, an image DB 20, a similar object search unit 30, a similar region search unit 40, a similarity integration unit 50, and a search result output unit 60. The These functional units are configured by so-called computers, and are associated with a CPU (Central Processing Unit) as an arithmetic / control device, a RAM (Random Access Memory) and a ROM (Read Only Memory) as a storage medium, a communication interface, and the like. It is realized with.

  The query image receiving unit 10 receives and receives a query image that is a search key for similar image search transmitted from a client terminal.

  The image DB 20 is a database that stores image data to be searched for images similar to the query image (hereinafter referred to as “search target image” as appropriate). For example, an image ID, which is identification information for identifying an image, and image data are stored in association with each other.

  The similar object search unit 30 calculates and outputs, for each search target image, the degree of similarity of the query image and the search target image, focusing on the object included in the image. Although details will be described later, a feature region indicating the presence of an object in an image is extracted, and a feature vector is generated based on a feature amount of the image region including the feature region. Then, the similarity between the query image and the search target image is calculated based on the feature vector for each image.

  Referring to FIG. 1, the similar object search unit 30 includes an indexing unit 30a, an index DB 30b, a feature vector generation unit 30c, an object similarity calculation unit 30d, and a cluster DB 30e.

  The similar object search unit 30 generates a feature vector of image data stored in the image DB 20 and indexes it in advance, and compares the feature vector generated from the query image with the feature vector stored in the index DB 30b. Thus, a similarity calculation step for calculating the similarity is performed. The generation of feature vectors performed by the indexing unit 30a and the generation of feature vectors performed by the feature vector generation unit 30c use the same algorithm, and details of each processing step will be described later.

  The similar area search unit 40 calculates and outputs, for each search target image, the degree of similarity for each image area in the image for the query image and the search target image. Briefly, an image is divided into N × M areas, and an image feature vector is generated based on the feature amount of the image area. Then, the similarity between the query image and the search target image is calculated based on the feature vector for each image.

  According to FIG. 1, the similar region search unit 40 includes an indexing unit 40a, an index DB 40b, a feature vector generation unit 40c, a region similarity calculation unit 40d, and a cluster DB 40e. Similar to the similar object search unit 30, the similar region search unit 40 performs an index processing step and a similarity calculation step. Note that the generation of feature vectors performed by the indexing unit 40a and the generation of feature vectors performed by the feature vector generation unit 40c can be realized using the same algorithm.

  The similarity integration unit 50 calculates the similarity for each search target image calculated by the similar object search unit 30 and the similarity for each search target image calculated by the similar region search unit 40 for each search target image. Perform operations to be integrated.

  The search result output unit 60 generates data obtained by ranking the search target images based on the similarities integrated by the similarity integrating unit 50. The data output by the search result output unit 60 is, for example, data obtained by sorting the image IDs of search target images based on the similarity. An address (URL) for accessing the image DB 20 may be added to the image ID.

[Indexing process in similar object search]
Next, the indexing process in the similar object search will be described with reference to a flowchart showing the process flow of the indexing process step in FIG. 2 and a schematic diagram of generating feature vectors in FIG.

First, the indexing unit 30a reads the image data stored in the image DB 20, and detects a feature region focused on an object in each search target image (step S11). As a detection method of this feature region,
・ Harris-affine
・ Hessian-affine
・ Maximally stable extremal regions (MSER)
・ Difference of Gaussians (DoG)
・ Laplacian of Gaussian (LoG)
・ Determinant of Hessian (DoH)
Etc.
As a feature region detection method, a method of selecting an image region having a random size and position in an image as a feature region is also known.

  The feature region detection technology is published in “Local Invariant Feature Detectors: A Survey” (Foundations and Trends in Computer Graphics and Vision, Vol.3, No.3, pp.177-280, 2007.). Any known technique can be adopted as appropriate.

  The indexing unit 30a detects a feature region (an ellipse or a rectangle) having a predetermined shape centered on a feature point in the image as an “region image”.

  FIG. 4 shows an example in which a feature point in the image is detected for the image ID “0001”, and regions T1 to T6 centered on the feature point are cut out. Further, an example is shown in which regions T11 to T14 are extracted for the image ID “0002” and regions T21 to T26 are extracted for the image ID “0003”.

  Next, the indexing unit 30a extracts a feature amount from the region image (step S12). In the indexing in the similar object search, local feature amounts that are resistant to affine transformation such as scale change, rotation, and angle change are extracted. As an example of the local feature amount, for example, the following can be cited.

・ SIFT
・ Gradient location and orientation histogram
・ Shape context
・ PCA-SIFT
・ Spin images
・ Steerable filters
・ Differential inverters
・ Complex filters
・ Moment inviteants

  The extraction of local features is published in “A performance evaluation of local descriptors” (IEEE Trans. Pattern Analysis and Machine Intelligence, Vol.27, No.10, pp.1615-1630, 2005.) A known technique can be adopted as appropriate.

  Then, the set of area images is clustered based on the feature amount (step S13). The indexing unit 30a determines a cluster to which the region image belongs based on the distribution of images clustered in advance in the cluster DB 30e and the distance between the region image and the feature amount space.

  The cluster DB 30e is a database in which images are clustered in advance based on feature values, and the images and clusters are stored in association with each other. The image to be clustered in the cluster DB 30e may be a feature region (region image) detected from an image stored in the image DB 20, or a feature region (region image) detected from an image prepared in advance for learning. There may be.

  By clustering, region images having similar feature amounts belong to the same cluster. For example, if all the region images are arranged in the feature amount space with the feature of the image as an axis, and a region in which a set of region images of a predetermined ratio or more with respect to the total number of region images is obtained is regarded as a cluster. Good. Hereinafter, an ID for identifying each cluster will be described with “#” attached.

As a standard method of clustering, k-means, Hierarchical Agglomerative Clustering (HAC) or the like is used. As a reference,
-Jain, Murty, Flynn: "Data clustering: a review", ACM Computing Surveys, Vol. 31, Issue 3, pp. 264-323, 1999.
There is.

  In FIG. 4, the extracted images of the regions T1, T3 to T6 are the cluster # 1, the image of T2 is the cluster # 2, the images of the regions T12 to T14 are the cluster # 3, the images of the regions T11 and T21 are the cluster # 4, the region The images of T24 to T26 are classified as cluster # 5, and the images of regions T22 and T23 are classified as cluster # 6.

  When the area images are clustered, the indexing unit 30a acquires the cluster to which each area image detected from each search target image belongs and the appearance frequency of the area image in each cluster (step S14). For example, no. Regarding the 0001 image, five area images appear in cluster # 1 and one area image in # 2.

  Then, the indexing unit 30a generates a feature vector based on the cluster to which the region image of each search target image belongs and the appearance frequency (step S15). A feature vector generated based on an example of the clustering feature vector shown in FIG. 4 is as follows.

  The indexing unit 30a stores the feature vector generated for each search target image as an index in the index DB 30b in association with the image ID. With the above processing, the indexing unit 30a generates a feature vector in which an object in the image is focused on the image stored in the image DB 20, and stores it in the index DB 30b in advance.

[Indexing process in similar area search]
Next, the indexing process in the similar area search will be described with reference to the flowchart showing the process flow of the indexing process in FIG. 3 and the outline of the feature vector generation in FIG.

  First, the indexing unit 40a of the similar region search unit 40 reads the image data stored in the image DB 20, and divides each search target image into predetermined regions (step S21). Here, as a method of dividing the image, for example, a method of dividing the image into predetermined M × N blocks or a method of dividing the divided blocks so that the size of the divided blocks becomes predetermined m × n pixels. is there. For example, when an image is divided into 10 × 10 blocks, if the size of the image is 640 × 480 pixels, the size of one block is 64 × 48 pixels.

  In the example of FIG. The 0001 image is divided into 7 × 6 blocks. No. For images of 0002, 5 × 7 blocks, No. The 0003 image is divided into 6 × 6 blocks.

  Next, the indexing unit 40a extracts a feature amount from each block-shaped image obtained by dividing the search target image (step S22). Then, the set of divided images is clustered based on the feature amount (step S23). The feature amount extracted in the indexing of the similar area search is a numerical expression of the feature of each image such as the color scheme, texture, and shape of the image. FIG. 4 illustrates a case where the region images cut out based on the detection points are clustered as feature regions, and illustration of clustering of divided images is omitted.

References about image features are:
“Content-based image retrieval at the end of the early years”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.22, No.12, pp.1349-1380.Dec 2000.
It has been known.

  Further, the indexing unit 40a determines a cluster to which the divided image belongs based on the distribution of the images clustered in advance in the cluster DB 40e and the distance in the feature amount space between the divided images. The cluster DB 40e is a database in which images are clustered based on feature amounts in advance and the images and clusters are stored in association with each other. The image to be clustered in the cluster DB 40e may be a divided image obtained by dividing an image stored in the image DB 20 into predetermined regions, or may be a divided image obtained by dividing an image prepared in advance for learning.

  When the divided images are clustered, the indexing unit 40a acquires a cluster to which the divided images divided from each search target image belong and the appearance frequency of the divided images in each cluster (step S24). Then, the indexing unit 40a generates a feature vector based on the cluster to which the divided images constituting each search target image belong and the appearance frequency (step S25).

  The indexing unit 40a stores the feature vector generated for each search target image as an index in the index DB 40b in association with the image ID. Through the above processing, the indexing unit 40a generates a feature vector focusing on the area constituting each image for the image stored in the image DB 20, and stores it in the index DB 40b in advance.

  As described above, the feature vector generated by the indexing unit 30a of the similar object search unit 30 is generated from a feature region where the object is highly likely to exist, and thus is effective as an index indicating the feature of the object.

  Further, since the feature vector generated by the indexing unit 40a of the similar region search unit 40 is generated from each divided image constituting the image, it is effective as an index indicating the overall configuration of the image.

[Search processing of image search device]
Next, a search process for searching for an image similar to the query image will be described with reference to the flowchart of FIG. 5 and the outline of the similarity calculation in FIG.

  When the query image receiving unit 10 receives a query image from the client terminal, the search processing shown in FIG. 5 is performed. This search processing is roughly divided into object similarity calculation processing, region similarity calculation processing, and similarity integration. It is divided into processing.

[Object similarity calculation processing]
First, the object similarity calculation process will be described.
The feature vector generation unit 30c of the similar object search unit 30 acquires a query image from the query image reception unit 10, and detects a feature region from the query image in the same manner as the indexing process described above (step S31).

  Then, a feature amount is extracted from the detected feature region (region image) (step S32). The feature vector generation unit 30c clusters each region image based on the feature amount extracted from each region image and the cluster information stored in the cluster DB 30e, and identifies the cluster to which each region image belongs (step S1). S33).

  Clustering of the region image detected from the query image is performed on a cluster that focuses on the feature of the object in the cluster DB 30e. Based on the cluster to which each region image belongs and the appearance frequency of the region image in the cluster, a feature vector is generated in the same manner as described above (step S34).

  Based on the feature vector of the query image and the feature vector of each search target image stored in the index DB 30b, the object similarity calculation unit 30d of the similar object search unit 30 determines the similarity (hereinafter “object”) for each search target image. (Referred to as “similarity”) (step S35). As a method for calculating the similarity, a known technique such as a cosine distance or a Bhattacharyya distance is used.

Here, for example, it is assumed that the next feature vector is generated from the query image by indexing by similar object search.

  At this time, when the cosine distances with the feature vectors generated from the images with the image IDs “0001” to “0003” illustrated in FIG.

  That is, based on a feature vector focused on an object included in each of the query image and the search target image, the search target image with the image ID “0003” is most similar to the query image, and the similarity is Calculated.

[Area similarity calculation processing]
Next, the region similarity calculation process will be described.
First, the feature vector generation unit 40c of the similar region search unit 40 acquires a query image from the query image reception unit 10, and divides the query image into predetermined regions in the same manner as the indexing process described above (step S41). Then, feature amounts are extracted from the image divided into predetermined regions (step S42).

  The feature vector generation unit 40c clusters each region image based on the feature amount extracted from each divided image and the cluster information stored in the cluster DB 40e, and specifies the cluster to which each region image belongs (step S43). ). Clustering of the divided images obtained from the query image is performed on clusters that focus on each area constituting the image generated by the indexing unit 40a based on the feature amount of the search target image.

  The feature vector generation unit 40c generates a feature vector in the same manner as described above based on the cluster to which each divided image belongs and the frequency of appearance of the divided image in the cluster (step S44).

  The region similarity calculation unit 40d of the similar region search unit 40 uses a similarity for each search target image (hereinafter, “region” based on the feature vector of the query image and the feature vector of each search target image stored in the index DB 40b. (Referred to as “similarity”) (step S45). In other words, the area similarity calculation unit 40d calculates the similarity based on the feature vector focused on each area constituting the entire image.

[Similarity integration processing]
Next, the similarity integration process will be described.
The similarity integration unit 50 is obtained by dividing the number of feature regions extracted from the query image by the feature vector generation unit 30c of the similar object search unit 30 and the query image by the feature vector generation unit 40c of the similar region search unit 40. The division number of the obtained area image and the division number are acquired (step S51).

  Then, the object similarity calculated by the object similarity calculating unit 30d and the region similarity calculated by the region similarity calculating unit 40d for each search target image are weighted based on the number of feature regions and the number of divisions. Similarities are integrated for each search target image (step S52). The similarity integrated for each search target image is hereinafter referred to as “integrated similarity” and is calculated by the following equation.

Integrated similarity = object similarity × (number of feature areas / (number of feature areas + number of divisions))
+ Area similarity x (number of divisions / (number of feature areas + number of divisions))

  In other words, the integrated similarity calculated by the above formula increases the weight for the object similarity when the number of feature regions indicating the features of the object included in the query image is large, and the query image has a feature in the entire image. Are integrated with each other so that the weight for the region similarity is increased.

  For example, it is assumed that the query image Q1 as shown in FIG. At this time, it is assumed that the similar object search unit 30 extracts feature areas in the query image Q1, and extracts 170 feature areas indicated by black dots in FIG. 6B from the query image Q1. In the similar area search unit 40, it is assumed that the query image Q1 is divided into 80 blocks of 8 × 10 as shown in FIG.

  Based on the number of feature areas and the number of divisions, the weight multiplied by the object similarity is 170 / (170 + 80) = 0.68, and the weight multiplied by the area similarity is 80 / (170 + 80) = 0.32.

  Therefore, when the object similarity and the region similarity of the query image Q1 and each search target image are integrated, the object is included in the query image Q1, and thus the object similarity specialized for the object is higher. A weighted integrated similarity is calculated.

  Further, it is assumed that the query image Q3 as shown in FIG. At this time, it is assumed that the feature region in the query image Q3 is extracted by the similar object search unit 30, and 20 feature regions indicated by black dots in FIG. 6 (f) are extracted from the query image Q3. In the similar area search unit 40, it is assumed that the query image Q3 is divided into 80 blocks of 8 × 10 as shown in FIG.

  Based on the number of feature regions and the number of divisions, the weight multiplied by the object similarity is 20 / (20 + 80) = 0.2, and the weight multiplied by the region similarity is 80 / (20 + 80) = 0.8.

  Therefore, when the object similarity and the region similarity of the query image Q3 and each search target image are integrated, the occupancy rate of the objects included in the query image Q3 is low, and the overall configuration such as color scheme is characteristic. Therefore, the integrated similarity that is weighted higher in the area similarity specialized for the image area is calculated.

  As described above, according to the present embodiment, using the number of feature regions detected from the query image by the similar object search and the number of divisions for dividing the query image by the similar region search, Then, weighting is performed on the similarity by the similar region search, and ranking is performed by the integrated similarity of each search target image.

  As a result, if there are many feature regions indicating the presence of an object in the query image input by the user at the time of search, the ranking based on the similarity by similar object search is regarded as important. In addition, when there are few feature areas indicating the presence of an object in the query image, the ranking of the entire query image based on the similar area search is regarded as important. Therefore, it is possible to generate a ranking of similar image searches in consideration of the user's search intention based on the query images.

  In addition, this invention is not limited by embodiment mentioned above, It can change in the range which does not deviate from the summary.

[Modification 1]
For example, the user may input a ratio for reflecting the similarity by the similar object search and the similarity by the similar area search when integrating the similarities, and the ratio may be added to the weighting. In this case, the ratio α is set and input by the user at the time of inputting the query image or in advance setting processing.

Calculation of the integrated similarity using the ratio α is performed by the following equation.
Integrated similarity = α × object similarity × (number of feature areas / (number of feature areas + number of divisions))
+ (1-α) × region similarity × (number of divisions / (number of feature regions + number of divisions))
Α is a value in the range of 0-1.

  Thereby, the weight according to a user's setting input can be added to the similarity at the time of integrating object similarity and area | region similarity. Therefore, the user can appropriately set which of the object and the image configuration of the query image is important when included in the query image.

[Modification 2]
Further, when the query image is an image obtained by cutting out a part of an image (original image), the weight value may be normalized based on the number of divisions of the query image and the number of divisions of the original image. Good.

When normalization is performed by the number of divisions, the calculation of the integrated similarity is performed by the following equation.
Number of normal feature areas = number of feature areas of query image / number of feature areas of original image Normal number of divisions = number of divisions of query image / number of divisions of original image Integrated similarity = object similarity x (number of normal feature areas / (normal feature Number of areas + number of regular divisions))
+ Area similarity x (Normal division number / (Normal feature area number + Normal division number))
However, the number of feature areas of the original image ≧ the number of feature areas of the query image, the number of divisions of the original image ≧ the number of divisions of the query image.

  For example, when the image Q1 shown in FIG. 6A is an original image and an image obtained by cutting out a part of this image is a query image Q1a, the feature region and the number of divisions of the image Q1, and the feature of the query image Q1a Normalization is performed based on the number of regions and the number of divisions.

  Accordingly, the number of feature regions and the number of divisions of the query image are normalized based on the number of feature regions detected from the original image cut out of the query image and the number of divisions divided by the indexing unit 40a when the original image is indexed. Therefore, it is possible to suppress variation in integrated similarity due to individual query images.

  In the second modification, as in the first modification, the object similarity and the region similarity may be weighted based on the ratio α input by the user.

[Modification 3]
Further, the number of divisions by indexing in the similar area search may be a value based on the number of feature areas detected from the image in the indexing in the similar object search.

  For example, when the number of feature regions detected from the image by the indexing unit 30a and the feature vector generation unit 30c is K, the number of divisions when the indexing unit 40a and the feature vector generation unit 40c divide the image is set to K. In the case of division into N × N blocks, the total number of regions is N × N = (square root of K) × (square root of K) = K, that is, the same as the number of feature regions.

  Thus, by setting the number of divisions based on the number of feature regions detected from the image, for example, the division accuracy is lowered when the number of feature regions is small, and the division accuracy is high when the number of feature regions is large. Become.

  For this reason, the division accuracy of the image can be changed according to the number of feature regions detected from the image as the number of divisions in the indexing of the similar region search. Also, normalization when calculating the integrated similarity is not necessary.

[Modification 4]
Further, the integration of object similarity and region similarity may be calculated based on the number of feature regions detected by indexing of similar object search. The calculation of the integrated similarity using only the number of feature areas is performed by the following equation.

  Integrated similarity = object similarity × f (number of feature regions) + region similarity × (1−f (number of feature regions))

Note that f (n) approximates the probability that an object is included in the query image, and is given by Equation 1 or Equation 2 as an example. F ₁ , F ₂ , N ₁ and N ₂ are set experimentally. FIG. 7A shows a graph of f (n) according to Equation 1, and FIG. 7B shows a graph of f (n) according to Equation 2.

According to the above formula, when the probability that an object is included in the query image is increased, an integrated similarity with importance placed on the object similarity is calculated. When the integrated similarity is calculated based only on the number of feature areas, only the similarity based on the object similarity search is obtained as the integrated similarity when the number of feature areas is N ₂ or more.

  In addition, the image search apparatus 1 performs feature vector generation, similarity calculation, and similarity integration using the functional blocks illustrated in FIG. 1, but the functional blocks illustrated in FIG. It is not a thing.

  For example, in the above description, the indexing unit and the feature vector generation unit are illustrated and described as separate functional blocks. However, the preprocessing (indexing) for the image stored in the image DB 20 and the processing for the query image (feature vector Generation) are described individually for convenience of explanation, and may be realized by one functional block.

  That is, the indexing unit generates a feature vector from the image stored in the image DB 20 and stores it in the index DB, and also generates a feature vector from the query image input from the query image receiving unit 20 to generate an object similarity calculation unit. Or may be output to the region similarity calculation unit.

[Modification 5]
In the above description, the case where one query image is received has been described as an example, but a plurality of query images may be received. In this case, feature vectors are generated for each query image by similar object search or similar region search. Then, the feature vector generation unit adds a plurality of feature vectors generated from each query image to compose one feature vector, and the object similarity calculation unit and the region similarity calculation unit compare with the feature vector in the index DB. The similarity is calculated.

  In this way, the addition of a plurality of feature vectors enables synthesis into one feature vector is based on a technique unique to a visual keyword that represents an image as a set of a plurality of image regions. That is, when there are a plurality of query images, a set of partial images (region images and divided images) detected from the plurality of query images can be regarded as a set of partial images for one query (search request). Therefore, it is possible to synthesize feature vectors.

  The similarity integration unit performs weighting based on the total number of feature regions detected from each query image by the feature vector generation unit and the total number of divisions obtained by dividing each query image into regions, and the object similarity and Integration with region similarity.

  As a result, even when there are a plurality of query images input by the user at the time of search, the weighting of importance of each similarity by similar object search and similar region search can be calculated from the entire query image, and the search intention of the user can be determined. It is possible to generate a ranking of similar image searches in consideration.

  In addition, even when a plurality of query images are input, the similarity of the search result by the similar object search and the similarity of the search result by the similar region search can be integrated, and the search according to the search intention of the user The ranking of results can be obtained with high accuracy.

[Modification 6]
In the above description, the feature vector is generated based on the appearance frequency of the region image and the divided image. However, weighting is performed by TF / IDF (term frequency-inverse document frequency) which is a word weighting method in the text search. Thus, a feature vector may be generated.

For reference materials on TF / IDF,
CDManning, P. Raghavan and H. Schutze: "Introduction to Information Retrieval", Cambridge University Press. 2008.
It has been known.

Hereinafter, a method of weighting by TF / IDF will be briefly described.
The appearance frequency (k, i) obtained by weighting the appearance frequency in each cluster k with the image ID i by TF / IDF is calculated by the following equation.

Appearance frequency (k, i) = TF (k, i) / T (i) * IDF (k)
IDF (k) = log (N / DF (k))
still,
TF (k, i) is the number of occurrences of partial images for image i in cluster k (appearance frequency)
T (i) is the number of partial images detected from the image i, DF (k) is the number of images that appear in the cluster k, and N is the total number of images in the image DB 20.

  Based on the above example, when N = 3, the feature vector based on the appearance frequency weighted by TF / IDF is as follows.

  In this way, the document in TF / IDF is regarded as an image, the word in the document is regarded as a partial image, and weighting is performed, so that the importance of the partial image appearing in each image is reduced, and the specific image appears prominently. The appearance frequency can be weighted so as to increase the importance of the characteristic partial image.

  In the above-described embodiment, the number of partial images detected from the similarity calculated by the image search algorithm specialized for the object and the similarity calculated by the image search algorithm specialized for the image area configuration, respectively. The similarity has been described as being integrated based on weighting based on the above, but the combination of the algorithms can be changed as appropriate.

  For example, a feature area specialized for a face image, a similarity degree searched using a feature quantity specialized for a face image, and a similarity degree obtained by an algorithm used for a feature quantity specialized for a landscape image, and a landscape The weighting described above may be performed according to the number of feature regions specialized for an image. In this case, since it is possible to perform weighting according to whether the query image is a face image or a landscape image, it is possible to obtain a ranking of search results according to the user's search intention.

  Moreover, although it demonstrated as integrating the similarity by two search algorithms, similarity can be integrated with respect to a some search algorithm. That is, the similarity can be weighted by using the similarity based on each search algorithm and the number of partial images detected from the query image by each search algorithm.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1 Image Search Device 10 Query Image Accepting Unit 20 Image DB
30 Similar Object Search Unit 30a Indexing Unit 30b Index DB
30c Feature vector generation unit 30d Object similarity calculation unit 40 Similar region search unit 40a Indexing unit 40b Index DB
40c Feature vector generation unit 40d Region similarity calculation unit 50 Similarity integration unit 60 Search result output unit Q1 Query image

Claims (5)

In an image search apparatus for calculating a similarity between a search target image stored in an image database and a query image and outputting a search result based on the similarity,
Feature region detecting means for detecting a feature region indicating the presence of an object from the query image;
Object vector generation means for generating a feature vector of the query image based on the detected feature region;
Object similarity calculation means for calculating the similarity between the query image and the image to be searched based on the feature vector generated by the object feature generation means;
Area dividing means for dividing the query image into areas;
Area vector generation means for generating a feature vector of the query image based on each of the divided image areas;
Area similarity calculating means for calculating the similarity between the query image and the search target image based on the feature vector generated by the area vector generating means;
The similarity calculated by the object similarity calculating unit and the similarity calculated by the region similarity calculating unit, the number of feature regions detected by the feature region detecting unit, and the region by the region dividing unit A similarity integration unit that performs weighting based on the number of divisions and integrates the similarity for each image to be searched;
An image search apparatus comprising: The region dividing means includes
The image search apparatus according to claim 1, wherein the number of region divisions is determined based on the number of feature regions detected by the feature region detection unit. When the query image is an image generated by cutting out a part of the image,
The similarity integration unit includes:
Normalizing the feature region detected by the feature region detection unit from the query image based on the number of feature regions detected from the image before clipping, and integrating the similarity using the normalized number of feature regions; The image search apparatus according to claim 1, wherein the image search apparatus is an image search apparatus. In an image search method in which a computer calculates a similarity between a query image and a query image stored in an image database, and outputs a search result based on the similarity.
A feature region detection step of detecting a feature region indicating the presence of an object from the query image;
An object vector generation step of generating a feature vector of the query image based on the detected feature region;
An object similarity calculation step of calculating a similarity between the query image and the search target image based on the feature vector generated in the object feature generation step;
Area dividing means for dividing the query image into areas;
A region vector generation step of generating a feature vector of the query image based on each of the divided image regions;
A region similarity calculation step of calculating a similarity between the query image and the search target image based on the feature vector generated in the region vector generation step;
The similarity calculated in the object similarity calculation step, the similarity calculated in the region similarity calculation step, the number of feature regions detected in the feature region detection step, and the region in the region division step A similarity integration step of integrating the similarities for each image to be searched by performing weighting based on the number of divided divisions;
The image search method characterized in that the computer performs the above.   An image search program for causing a computer to execute the image search method according to claim 4.

Images