JP2010061285A - Image retrieval device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device and method for improving the retrieving precision of a similar image using a query image. <P>SOLUTION: A query accepting part 1 accepts a basic query image for image retrieval. A query extension part 2 creates a pseudo query image by performing image processing to the basic query image. A retrieval part 3 obtains a temporary retrieval result by retrieving a similar image from a plurality of target images about both the basic query image and the pseudo query image. An integration part 4 obtains a retrieval result for output by integrating the temporary retrieval results, and performing overall ranking. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique for searching for a similar image using a query image.

  As a technique for searching for an image similar to a certain image (so-called input image), for example, there is a technique described in Patent Document 1 below.

  In the technique of this document, a feature amount is extracted from an input image. This feature amount is compared with the feature amount of the image registered in the database. A registered image in which the distance between feature amounts is within a predetermined range can be acquired as a similar image.

  Incidentally, the feature amount is generally expressed as multidimensional vector data. Increasing the number of data dimensions can be expected to improve the search accuracy, but the calculation cost increases. Therefore, in practice, the number of dimensions of the vector data is limited. For this reason, in the actual similar image search, there is a problem that noise increases and the search accuracy tends to be low.

On the other hand, in the following Patent Document 1, the search accuracy is improved by allowing the user to select a plurality of query images.
Japanese Patent Laid-Open No. 11-238078

  However, the method of selecting a plurality of query images imposes an operation burden on the user. For this reason, it is actually difficult to select many query images. In addition, since the similar image search is performed based on features that are mechanically extracted from the image such as color, shape, and texture, the search result greatly depends on the query image selected by the user. Therefore, the conventional technique has room for improvement in terms of improving the search accuracy.

  The present invention has been made in view of the above situation. One of the objects of the present invention is to provide an apparatus or method capable of improving the search accuracy of similar images using query images.

  The present invention is configured as described in any of the following items.

(Item 1)
A query reception unit, a query expansion unit, a search unit, and an integration unit,
The query reception unit is configured to receive a basic query image,
The query expansion unit is configured to generate a pseudo query image by performing image processing on the basic query image.
The search unit is configured to obtain a temporary search result by searching similar images from a plurality of target images for both the basic query image and the pseudo query image,
The image search apparatus according to claim 1, wherein the integration unit is configured to obtain the search results for output by integrating the temporary search results and performing overall ranking.

  In the present invention, in addition to the temporary search result corresponding to the basic query image which is the original query image, a pseudo query image is further generated, and the temporary search result based on this is acquired. These temporary search results can be integrated to obtain a final search result. Therefore, a highly accurate search result can be obtained while keeping the burden on the operator low.

(Item 2)
The image search device according to item 1, further comprising an image registration unit, wherein the plurality of target images are recorded in the image registration unit.

  In general, it is preferable that an image to be searched is registered in advance in some database in order to shorten the search time.

(Item 3)
The image search device according to item 1 or 2, wherein the integration unit is configured to integrate the temporary search results using the ranks of the temporary search results in each of the basic query image and the pseudo query image. .

  Results of similar image search based on the basic query image can be ranked using, for example, similarity. Similarly, similar image search results based on pseudo-query images can also be ranked using, for example, similarity. Temporary search results can be integrated using the ranking in ranking.

(Item 4)
The integration unit is configured to make the weighting of the rank in the temporary search result corresponding to the basic query image larger than the weighting of the rank in the temporary search result corresponding to the pseudo query image. The image search device described in 1.

  In general, a temporary search result based on a basic query image is considered to be more reliable than a temporary search result based on a pseudo query image. Therefore, a higher weight is given to the rank in the temporary search result based on the basic query image than the rank in the temporary search result based on the pseudo query image. Thereby, the reliability of the integrated search result can be improved.

(Item 5)
The image search according to item 3, wherein the integration unit is configured to adjust the weighting of the rank in the temporary search result corresponding to the pseudo query image according to a property of the basic query image or a characteristic of the search unit. apparatus.

  For example, if the basic query image is a person photo, changing the color increases the possibility of searching for an image other than the person. Thus, it is considered that the reliability of the pseudo query image changes depending on the nature of the basic query image. Similarly, it is considered that the reliability of the pseudo query image changes depending on the characteristics of the search unit (so-called search engine). Therefore, it is preferable to adjust the weighting of the rank in the temporary search result corresponding to the pseudo query image according to the property of the basic query image or the characteristics of the search unit.

(Item 6)
The image search device according to item 1 or 2, wherein the integration unit is configured to integrate the temporary search results using a similarity between the temporary search results and the basic query image or the pseudo query image. .

  It is possible to integrate the temporary search results by using the similarity between the temporary search results and the basic query image or the pseudo query image. An image having a high degree of similarity is generally considered to have high reliability as a search result. In the invention of this item, an image having a high similarity can be output as a final search result. Moreover, it is possible to perform different weighting with respect to the similarity of a search result for every pseudo query image.

(Item 7)
Receiving a basic query image;
Generating a pseudo query image by performing image processing on the basic query image;
Obtaining a temporary search result by searching similar images from a plurality of target images for both the basic query image and the pseudo query image;
An image search method comprising: obtaining the output search results by integrating the provisional search results and performing overall ranking.

(Item 8)
A computer program for causing a computer to execute the steps according to item 7.

  This computer program is stored in an appropriate recording medium (for example, an optical recording medium such as a CD-ROM or a DVD disk, a magnetic recording medium such as a hard disk or a flexible disk, or a magneto-optical recording medium such as an MO disk). Can be stored. This computer program can be transmitted via a communication line such as the Internet.

  ADVANTAGE OF THE INVENTION According to this invention, the apparatus thru | or technique which can improve the search precision of the similar image using a query image can be provided.

(Apparatus configuration of the first embodiment)
An image search apparatus according to a first embodiment of the present invention will be described with reference to the accompanying drawings.

  As shown in FIG. 1, the image search apparatus according to the first embodiment includes a query reception unit 1, a query expansion unit 2, a search unit 3, an integration unit 4, a display unit 5, and an image registration unit. 6 as a basic configuration.

  The query receiving unit 1 is configured to receive a basic query image 7. The basic query image 7 is generally an image designated by the user for image search. An image serving as a key for image search is generally called a query image, but in this specification, it is called a basic query image in order to distinguish it from a pseudo query image described later.

  The query expansion unit 2 is configured to generate a pseudo query image by performing image processing on the basic query image 7.

  The search unit 3 is configured to obtain a temporary search result by searching similar images from a plurality of target images for both the basic query image 7 and the pseudo query image.

  The integration unit 4 is configured to obtain output search results by integrating the temporary search results and performing overall ranking.

  The display unit 5 displays a list 8 as an output search result. The display unit 5 may be any display unit that can display the list 8 to the outside, such as various displays and printers.

  The image registration unit 6 records a plurality of target images. The image registration unit 6 includes an image DB 61, an index unit 62, and an index DB 63.

  The image DB 61 is for registering images to be searched. Here, the image DB 61 may store feature amounts or vector data linked to images. Such data is also referred to herein as an image. The index unit 62 attaches an index in order to facilitate the search for images registered in the image DB 61. The index DB 63 stores the created index.

  The operation of the image search apparatus in this embodiment will be described in detail below.

(Image search method according to the first embodiment)
The operation of the image search apparatus according to the first embodiment will be described with further reference to FIGS. FIG. 2 is a flowchart showing the overall flow of the image search method. FIG. 3 is an explanatory diagram schematically showing the expansion operation of the query image.

(Step SA-1 in FIG. 2)
First, a registered image (search target image) is prepared and registered in the image registration unit 6. The image registration unit 6 performs necessary indexing and records the index in the index DB 63. For example, an image acquired from a website can be registered in the image registration unit 6 via the Internet.

(Step SA-2 in FIG. 2)
Next, the user inputs the query image 7 from a user terminal (not shown). This query image 7 is called a basic query image in this specification.

  The query reception unit 1 receives the input basic query image 7 via, for example, the Internet.

(Step SA-3 in FIG. 2)
Next, the expansion unit 2 generates a pseudo query image 9 based on the basic query image 7. The number of pseudo query images 9 may be one or a plurality of two or more. A larger number of pseudo-query images 9 can be considered to improve the search accuracy. FIG. 3 shows an example in which three pseudo query images 9 are generated.

  An example of generating a pseudo query image will be described in more detail with reference to FIG. The pseudo query image is generated by performing image processing on the basic query image 7. Here, in this specification, image processing is to obtain different images by manipulating certain parameters without losing the essential characteristics of the original image.

  In the example of FIG. 4, examples of image processing (so-called deformation) by rotation, blurring, embossing, intermediate value, equalization, and threshold processing are shown. Of course, image processing techniques are not limited to these.

(Step SA-4 in FIG. 2)
Next, a similar image is searched from the image registration unit 6 for each query image (basic query image and pseudo-query image). As a similar image search method, a conventionally used method can be used, and therefore, detailed description thereof is omitted.

  As a result of this search, a search result corresponding to each query image (basic query image and pseudo-query image) can be obtained. This search result is referred to as a temporary search result 10 in this specification (see FIG. 3). This is to distinguish from the final (ie, output) search results.

  Further, the search unit 3 ranks the temporary search results 10 according to the degree of similarity with each query image. As a method for calculating the similarity, a conventionally used method can be used. For example, low-level visual feature quantities (for example, color, shape, texture, etc.) can be calculated as multidimensional vector data, and the similarity can be expressed by the distance between the multidimensional vector data. Then, the search unit 3 calculates the similarity between the multidimensional vector of the image registered in the image registration unit 6 and the multidimensional vector data corresponding to the query image, and displays the search results in descending order of similarity. Output.

(Step SA-5 in FIG. 2)
Next, the integration unit 4 integrates the temporary search results 10 as described below, and generates an output search result 11 (see FIG. 3).

  An example of the temporary search result 10 is shown in FIG. Here, in addition to the temporary search results based on the basic query image, temporary search results based on two pseudo query images are shown. In FIG. 6, the data obtained as the temporary search results are respectively labeled as I-XX. XX is a number given corresponding to the image.

  In FIG. 6, the ranking (ranking) of the temporary search results based on the basic query image 7 is denoted by reference symbol R−1, and the ranking (ranking) of the temporary search results based on the two pseudo query images is denoted by the symbol R−. 2 and R-3 are attached respectively.

  First, the symbols are defined as follows.

  At this time, the score for each search result can be calculated by the following equation.

  Here, Rank (r, i) is set to the square to reduce the score value when the ranking is low (that is, the number of rankings is large). Increasing the index value can increase the influence of ranking. The value can be determined experimentally. The absolute value of R indicates the number of rankings, and the division is performed for normalization.

  In this example, different weights are adopted for each ranking. For example, the weight for the ranking of the temporary search result based on the basic query image (Weight (R-1) = 0.8 in the following calculation example) can be set higher than other rankings. The temporary search result based on the basic query image selected by the user as the image search query is considered to have higher reliability than the other results, and thus the search finally obtained by weighting in this way The reliability of the result can be improved.

  A specific calculation example for the score is shown below.

  As shown above, the search result with the symbol I-09 appears three times in all rankings, and the score is 0.157. Similar calculations can be made for other images.

  Next, the integration unit 4 sorts the images based on the score of each image. Then, based on the sorted rank, a predetermined rank is sent to the display unit 5 as a final search result. Here, the default order may be the order specified by the user or the administrator, or the order dynamically generated by the system side.

  The display unit 5 can present the search result based on the query image to the user.

  In the present embodiment, in addition to the temporary search result corresponding to the basic query image 7 which is the original query image, a pseudo query image 9 is further generated, and a temporary search result based on this is acquired. Then, these temporary search results can be integrated to obtain a final search result. Therefore, a highly accurate search result can be obtained while keeping the burden on the operator low.

  In the embodiment, the weight on the temporary search result based on the basic query image is increased. The weighting of the ranking is not limited to this, and can be adjusted according to the nature of the basic query image or the characteristics of the search unit 3, for example.

  For example, if the basic query image 7 is a person photo, changing the color increases the possibility of searching for an image other than a person. Further, when the basic query image 7 is a landscape photograph, the use of the rotated image for the search increases the possibility of searching for noise.

  For this reason, for example, when the basic query image 7 is a person image, by reducing the weighting of the rank in the temporary search result corresponding to the pseudo query image whose color has been processed, the search result by the color processing can be reduced. The impact can be reduced.

  When the basic query image 7 is a landscape photograph, a feature amount (rotation-invariant feature, for example, a color histogram) that does not change the feature amount even when the image is rotated may be important when searching for similar images. It is common. For this reason, it is possible to prevent noise from being included in the search result by lowering the weighting of the rank in the temporary search result corresponding to the pseudo query image by image rotation.

  In addition, in the case of a system that searches for product images, the basic query image may include items that are not related to the product (for example, background) in addition to the product, so when generating a pseudo query image, It is preferable to perform image processing such as deleting the background. In this case, the information that the user is interested in (the product itself) is expressed more accurately in the pseudo query image excluding the background and the like than the basic query image. Is made larger than the weight of the temporary search result of the basic query image, a more accurate search result can be obtained. As described above, in the case of product image search, it is preferable that the weight of the temporary search result corresponding to the pseudo query image from which the background is deleted is larger than the weight of the temporary search result corresponding to the basic query image. it is conceivable that.

  Further, in the case of searching for product images, a method of changing the color of the product after removing the background from the basic query image is also effective. For example, when searching for an image including a car, each pseudo query image is generated by further changing the color of the image obtained by removing the background from the basic query image. At this time, a method of increasing the weight of a pseudo-query image of a common color (black, white, red, etc.) as a car color and reducing the weight of a pseudo-query image of a color uncommon as a car color (pink, etc.) By using, it can suppress that many product images other than a car are included.

  As described above, it is considered that the reliability of the pseudo query image changes depending on the nature of the basic query image. Similarly, depending on the characteristics of the image registration unit and the search unit (so-called search engine), such as the type of image (product, person, landscape, etc.) registered in the image DB 61 and the type of feature amount considered important at the time of search. However, the reliability of the pseudo query image is considered to change. Therefore, it is possible to adjust the weighting of the rank in the temporary search result corresponding to the pseudo query image according to the property of the basic query image or the characteristics of the search unit.

(Second Embodiment)
Next, an image search device according to a second embodiment of the present invention will be described. In the description of the second embodiment, the same reference numerals are used for elements that are basically common to the apparatus of the first embodiment described above, thereby avoiding complicated description.

  In the first embodiment described above, the ranking of the temporary search results is performed using the ranking in the temporary search results for each query image. On the other hand, the integration unit 4 according to the second embodiment is configured to integrate the temporary search results using the similarity between the temporary search results and the basic query image or the pseudo query image.

  A specific calculation example will be described below.

  First, the meaning of the symbols to be used is defined as follows.

  At this time, the score for each search result can be calculated by the following equation. Using the scores calculated in this manner, the temporary search results can be ranked in the same manner as in the first embodiment, and the final search results can be obtained.

  In the present embodiment, the temporary search results can be integrated using the similarity between the temporary search results and the basic query image or the pseudo query image. An image having a high degree of similarity is generally considered to have high reliability as a search result. Therefore, it is considered that an image having a high degree of similarity can also be output as a final search result in the method of the second embodiment.

  In the second embodiment, as in the first embodiment, the weighting on the similarity can be changed for each ranking corresponding to the query image. This is clear from the above equation. Thereby, for example, the score can be changed according to the nature of the query image, and high-accuracy image search is possible.

  Other configurations and advantages of the second embodiment are basically the same as those of the first embodiment described above, and thus detailed description thereof is omitted.

  The operations of the above-described embodiments can be implemented by incorporating appropriate computer software into the computer.

  The contents of the present invention are not limited to the above embodiment. In the present invention, various modifications can be made to the specific configuration within the scope of the claims.

  For example, each component described above may exist as a functional block, and may not exist as independent hardware. As a mounting method, hardware or computer software may be used. Furthermore, one functional element in the present invention may be realized by a set of a plurality of functional elements, and a plurality of functional elements in the present invention may be realized by one functional element.

  Moreover, the functional element may be arrange | positioned in the position physically separated. In this case, the functional elements may be connected by a network. It is also possible to realize functions or configure functional elements by grid computing.

1 is a block diagram illustrating a basic configuration of an image search device according to a first embodiment of the present invention. It is a flowchart for demonstrating schematically the whole flow of the image search method which concerns on 1st Embodiment of this invention. It is explanatory drawing for demonstrating typically the image search method which concerns on 1st Embodiment of this invention. It is explanatory drawing which shows the example of a pseudo query image. 5 is a flowchart for explaining a procedure for integrating temporary search results in the first embodiment. In 1st Embodiment, it is explanatory drawing for demonstrating the procedure which integrates a temporary search result.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Query reception part 2 Query expansion part 3 Search part 4 Integration part 5 Display part 6 Image registration part 7 Query image (basic query image)
8 Search result list 9 Pseudo query image 10 Temporary search result 11 Output search result

Claims (8)

A query reception unit, a query expansion unit, a search unit, and an integration unit,
The query reception unit is configured to receive a basic query image,
The query expansion unit is configured to generate a pseudo query image by performing image processing on the basic query image.
The search unit is configured to obtain a temporary search result by searching similar images from a plurality of target images for both the basic query image and the pseudo query image,
The image search apparatus according to claim 1, wherein the integration unit is configured to obtain the search results for output by integrating the temporary search results and performing overall ranking. The image search device according to claim 1, further comprising an image registration unit, wherein the plurality of target images are recorded in the image registration unit. The image search according to claim 1, wherein the integration unit is configured to integrate the temporary search results using ranks of the temporary search results in each of the basic query image and the pseudo query image. apparatus. The integration unit is configured such that a weight on a rank in the temporary search result corresponding to the basic query image is larger than a weight on a rank in the temporary search result corresponding to the pseudo query image. 4. The image search device according to 3. The image according to claim 3, wherein the integration unit is configured to adjust weighting to a rank in the temporary search result corresponding to the pseudo query image according to a property of a basic query image or a characteristic of the search unit. Search device. The image search according to claim 1, wherein the integration unit is configured to integrate the temporary search results using a similarity between the temporary search results and the basic query image or the pseudo query image. apparatus. Receiving a basic query image;
Generating a pseudo query image by performing image processing on the basic query image;
Obtaining a temporary search result by searching similar images from a plurality of target images for both the basic query image and the pseudo query image;
An image search method comprising: obtaining the output search results by integrating the provisional search results and performing overall ranking. A computer program for causing a computer to execute the steps according to claim 7.