JP2011203776A - Similar image retrieval device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent deterioration in retrieving efficiency in retrieving a similar image on the basis of an image.SOLUTION: The similar image retrieval device is configured to store image feature values and text feature values based on a text relating to an image in a feature storage means for each image, to receive a request instruction for retrieving a similar image together with an image designated by a user, to retrieve a similar image on the basis of the similarity of the image feature values corresponding to the image designated when the request instruction is received and the similarity of the text feature values corresponding to the text associated with the image designated when the request instruction is received, by referring to the feature storage means for each image to be retrieved, and to display the retrieval result of the similar image.

Description

  The present invention relates to a similar image search using an image as a key.

  Conventionally, as a technique for searching various information disclosed on the Internet, in addition to a keyword search using a specified keyword as a search key (query), a specified image is used as a search key ("query image"). A so-called similar image search that searches for similar images is known (see, for example, Non-Patent Document 1). In the similar image search, images similar to the query image are searched for and displayed by comparing information called feature amounts obtained by converting features such as color schemes, contour lines, patterns, and patterns for each image into data. .

  Further, although not in the field of similar image search, as a proposal related to a user interface for presenting searched information to a user, in Patent Document 1, when a plurality of contents to be searched are displayed on the screen, related keywords are similar. Connect each other with a link line. In this way, it is possible to easily grasp the similar relationship between a large number of displayed contents, and search for desired contents efficiently. Similar to this, if similar images are also displayed as links according to the similarity of images, similar images can be easily searched by following the links.

Google Inc., “Google Similar Images”, [online], [Search March 17, 2010], Internet <URL: http://similar-images.googlelabs.com/>

JP 2008-129942 A

  However, if a similar image is searched based only on visual features such as the color and shape of the image, there is a problem that the search accuracy is lowered because the image is far from the original image.

  In view of the above problems, an object of the present invention is to prevent a decrease in search accuracy when a similar image search is performed using an image as a key.

  Based on the above object, the similar image search apparatus according to one aspect (1) of the present invention includes, for each image, feature storage means for storing an image feature amount and a text feature amount based on text associated with the image. Receiving means for receiving a request instruction for similar image search with image designation from the user, and by referring to the feature storage means for each image that is a candidate to be searched, an image feature for the image specified in the request instruction Similar image search means for searching for a similar image based on the similarity of the amount and the similarity of the text feature amount with respect to the text associated with the image specified in the request instruction, and a similar image search result are displayed. And a result display means.

  According to another aspect (5) of the present invention, the above aspect is captured by a method category, and a feature storage unit that stores, for each image, an image feature amount and a text feature amount based on text associated with the image. A similar image search method in which a computer having a similar image search performs a reception step in which the computer receives a request instruction for a similar image search accompanied by an image designation from a user, and each image for which the computer is a candidate to be searched By referring to the feature storage means, the similarity of the image feature amount to the image specified in the request instruction, the similarity of the text feature amount to the text associated with the image specified in the request instruction, A similar image search step for searching for similar images based on the results, and a result display step for the computer to display search results for similar images. Characterized in that it comprises Tsu and up, the.

  Another aspect (6) of the present invention is a similar image retrieval program for causing the computer to perform a similar image retrieval, in which the above aspect is captured in the category of a computer program. , A feature storage means for storing a text feature quantity based on text related to an image is realized in a computer, and the computer is caused to accept a request for a similar image search accompanied by an image designation from a user, and the computer is searched. By referring to the feature storage means for each image that is a candidate, the similarity of the image feature amount with respect to the image specified in the request instruction, and the text feature for the text associated with the image specified in the request instruction Search for similar images based on the amount of similarity and display the search results for similar images on a computer Characterized in that to.

  As described above, according to the present invention, when a similar image is searched based on a specified image, not only the similarity of the image feature amount with the specified base image but also the specified base image is associated. By using the similar image narrowing down based on the degree of similarity related to the text feature amount with respect to the text, it is possible to prevent a decrease in search accuracy when performing a similar image search using the image as a key.

  In another aspect (2) of the present invention, in any one of the above aspects, the accepting unit accepts designation of an image to be used as a query for similar image retrieval from images retrieved based on designated text. The similar image search means uses the designated text as the text associated with the designated image.

  In this way, by accepting the specification of the query image from the search image based on the specified text and using the specified text as the basis of the text feature amount on the query side, the text itself specified by the user is similar. Since it can be reflected in the image search, the prevention of a decrease in search accuracy is further ensured.

  According to another aspect (3) of the present invention, in any of the above aspects, when the similarity of the text feature amount is calculated using a plurality of texts associated with the designated image, those used for the calculation are used. It has a weighting means for performing a predetermined weighting between a plurality of texts.

  In this way, when calculating the similarity of the text feature amount using a plurality of texts associated with the image, for example, a predetermined weight such as relatively increasing the weight of the text directly designated by the user by type input or the like. By performing the weighting, a highly accurate search result suitable for the user's intention can be realized.

  According to another aspect (4) of the present invention, in any of the above aspects, the retrieved image is associated with the presence / absence or similarity of the image feature amount with respect to the designated image and the designated image. It is characterized by having a similarity display means for displaying at least one of the similarity of the text feature quantity with respect to the text and the similarity.

  As described above, it is easy to grasp the search result by indicating whether the obtained search result is suitable for the image or the text.

  It should be noted that a category (method for the apparatus, program for the method, etc.) different from each of the above-described modes and more specific modes described below are also included in the present invention. For different categories, “means” shall be appropriately read as “step”.

  According to the present invention, it is possible to prevent a decrease in search accuracy when a similar image search is performed using an image as a key.

The functional block diagram which shows the structure of embodiment of this invention. The figure which illustrates the information (data) used by embodiment of this invention. The flowchart which shows the process sequence in embodiment of this invention. The figure which shows the example of a screen for searching an image from a text in embodiment of this invention. The figure which shows the example of a screen which establishes an image for the similar image search in embodiment of this invention. The conceptual diagram which shows an example of the weighting in embodiment of this invention. The conceptual diagram which shows an example of the weighting in embodiment of this invention. The figure which shows the example of a screen display of the similar image search result in embodiment of this invention. The figure which shows the example of a screen display of the similar image search result in embodiment of this invention. The figure which shows the example of a screen display of the similar image search result in embodiment of this invention.

  Next, modes for carrying out the present invention (referred to as “embodiments”) will be described with reference to the drawings. It should be noted that assumptions common to those already described in the background art and problems are omitted as appropriate.

[1. Constitution〕
In this embodiment, as shown in FIG. 1, a similar image search device 1 (hereinafter referred to as “this device 1” or “this device”) that provides a similar image search service in response to a request from a terminal T via a communication network N. Called). This apparatus 1 has at least an arithmetic control unit 6 such as a CPU, a storage device 7 such as an external storage device (HDD or the like) and a main memory, and a communication network N (Internet, mobile phone network, Communication means 8 (LAN adapter etc.). A terminal T is an information processing apparatus used by a user, such as a personal computer (PC), a smart phone, or a mobile phone terminal device. FIG. Assumption.

  In the apparatus 1, a predetermined computer program (not shown) stored (installed) in advance in the storage device 7 controls the arithmetic control unit 6, so that elements (11, 12, 13, etc.) shown in FIG. Etc.). Among these elements, the information storage means can be realized in the storage device 7 in any format such as various files and databases (also referred to as “DB”), variables such as arrays, various stacks and registers, and system setting values.

  Among such storage means, as illustrated in FIG. 2, the feature storage means 11 is compared with an image to be a query (also referred to as “query image”) and searched for an image (in the example of FIG. 2, Each feature is identified by identification information such as an image file name, etc. (hereinafter referred to as “candidate image”), and an image feature amount and a text feature amount based on a text (character string) related to the image are stored. It is means to do. Here, the example of the image feature amount is a simplified example for explanation based on the shape and color included in the image, and actual elements and expression formats are free. Further, the text feature amount is obtained by associating, with respect to one or more texts related to the image, a weight representing the depth of the relationship between the text and the image.

  In this way, text related to an image is a keyword such as an object, a scene, a theme, or a time zone (hereinafter also referred to as a “tag”) in the image, and the typical source (acquisition source) is the image. Is extracted from a web page on which is posted, or is associated with the image on such a web page. For example, if one or more tags are given to an image on a web page, a text feature amount can be extracted in the same manner as an image feature amount by a vector space model, which is a general method for similar text search.

  By storing such a text feature amount together with the image feature amount in the feature storage unit 11, in this embodiment, a similar image search can be performed in consideration of the text feature amount. However, since the source of the text feature amount is free and is not limited to extraction from the web page as described above, for example, an administrator or a user may input in advance for each image. Each means other than the storage means is a processing means for realizing and executing the following information processing functions and operations.

[2. Action)
In the apparatus 1 configured as described above, a processing procedure related to a similar image search is shown in a flowchart of FIG. This processing procedure includes receiving a query image, searching for a similar image, and displaying a search result.

[2-1. (Reception of query image)
That is, first, the accepting unit 12 accepts a request instruction for a similar image search accompanied by a query image specification from the user who uses the terminal T via the terminal T and the communication network N (step S31). Here, an example of the screen transition leading to the designation of the query image is shown. For example, on the screen illustrated in FIG. 4, as a result of inputting the desired text X1 in the character string input field F (so-called text box) and operating the search button B1 with the mouse pointer P, the input text (“designation” Assume that the image search results via the Web image search API or the like are displayed as a list as illustrated in FIG.

  In this list display (FIG. 5), a predetermined similarity search button B2 is displayed for each of a plurality of query images G hit in the search, and the accepting means 12 corresponds to the similarity search button B2 operated with the mouse pointer P or the like. A request instruction is accepted using an image as a query image. The configuration of the user interface that accepts the designation of the image is free, and it is not essential that the search starts from the designated text as in the examples of FIGS. 4 and 5, for example, a recommended image collection or a topic image corner. It is also possible to simply accept the specification of the query image from the list of images.

[2-2. (Search for similar images)
When a query image is designated as described above (step S31), subsequent to weight setting processing (steps S32 to S35), which will be described later, the similar image search means 13 stores feature information for many candidate images to be searched. By referring to the means 11, the similarity of the image feature quantity (also referred to as “image similarity”) with respect to the query image specified in the request instruction, and text such as a tag associated with the query image (“reference A similar image is searched based on the similarity of the text feature amount (also referred to as “text similarity”) with respect to the text (also referred to as “text”) (step S36).

  At this time, when a query image is specified from an image group searched based on the specified text (for example, FIGS. 4 and 5), the similar image search means 13 uses the reference text associated with the specified query image as the reference text. If at least the designated text is used and there is a tag associated with the image, the tag and the designated text are combined as a reference text. Note that when a similar image search is performed by selecting a query image without performing an image search from the designated text, only text such as a tag associated with the selected query image may be used as the reference text.

  In the search for similar images as described above, the criteria and algorithm for calculating the image similarity and the text similarity can be freely determined. For example, the image feature amounts and the text feature amounts can be set as multi-dimensional vector data, respectively. It is considered common to calculate and compare these. More specifically, for example, between the query image and each candidate image, a distance function indicating the image similarity (a function related to distance calculation in the feature amount space; hereinafter the same) and a distance function indicating the text similarity are: If it is the same distance formula, for example, Euclidean distance, high-speed search is possible with a general multidimensional space index, but if it is not a different metric space or Euclidean distance, a metric space index represented by a graph index is used. You may search using.

  The similar image search means 13 comprehensively evaluates the similarity of the image feature amount and the similarity of the text feature by addition, multiplication, etc., so that the image similarity and the text are determined according to the setting of the calculation standard in the comprehensive evaluation. With regard to the similarity, at least one of the candidate images that is superior or the candidate image that is neither significantly inferior is prioritized as the top of the similar search results.

[2-3. Weighting in similar image search)
In addition, when calculating the similarity of a text feature amount using a plurality of texts associated with a designated query image or candidate image, the weighting unit 14 performs a predetermined weighting between the plurality of texts used for the calculation. . Here, the description will be divided into the setting of the weight for the text and the calculation of the similarity using the set weight. As for the setting of the weight, setting of the weight (for example, FIG. 2) for each candidate image and setting of the weight for the reference text side of the query image can be considered. An example is shown.

  The setting of the weight on the candidate image side is performed by determining the content of the image, the content of the web page that includes the image, the content of the caption associated with the image on the web page, the XML of the web page, for example Based on the priority order between tags associated with the image by description or the like, it is stored in advance in the feature storage unit 11. The setting of the weight for the reference text may be performed when the similarity calculation is repeated for each candidate image. Here, an example is shown in which the calculation is performed prior to the calculation.

  For example, first, when the specified text exists, the specified text and the tag attached to the image searched with the specified text do not always match. In some cases, the assigned image may be searched, or some of a plurality of tags assigned to the image included in the search result may match the specified text.

  Therefore, in order to more reflect the intention of the searcher appearing in the designated text, the following weight is given in relation to the designated text. First, as illustrated in FIG. 6 (1), the tag matches the text X1 in the tag (for example, X2) given to the query image G specified from the search result based on the specified text X1 (for example, “mountain”). When there is nothing to do, the weighting means 14 adds the designated text X1 to the reference text in addition to the tag of the query image G as illustrated in FIG. For example, a predetermined weight (for example, “5”) that is relatively larger than “1”) is used.

  On the other hand, as illustrated in FIG. 7A, the text X1 is included in the tags (for example, X2 and X3) given to the query image G specified from the search result based on the specified text X1 (for example, “mountain”). If there is a match (related tag X3 “mountain”), as shown in FIG. 7 (2), the weight of the matching tag (for example, “1”) has a large predetermined weight as described above. (For example, “5”) is added to, for example, “6”. Note that the weight of the matching tag may be replaced with such a large predetermined weight (for example, “1” → “5”).

  In order to realize the weighting of the reference text based on the specified text as described above, in the processing procedure of FIG. 3, if there is a specified text (step S32: “YES”), if there is no specified text in the tag of the query image (step S33: “NO”) In addition to the tag of the query image, the designated text is used as the reference text (step S34), and a predetermined value (“5” in the above example) is added to the weight of the designated text in the reference text ( Step S35).

  As described above, the weight set on both the text side of the candidate image and the reference text side of the query image is reflected in the calculation of the similarity in the similar image search (step S36). Candidate images with matching text will hit with high priority. For example, the similarity of the text feature amount is highest when the reference text having a large weight on the image query side matches the text having a large weight on the candidate image side.

[2-4. (Display of search results)
Then, the result display unit 15 displays a search result of similar images reflecting the weights as described above (step S37). At this time, the similarity display means 16 determines whether or not the image feature amount is similar or similar to the query image and whether or not the text feature amount is similar to the reference text associated with the query image. And at least one of them is displayed. Here, the image similarity and the text similarity are displayed, and an example thereof is shown in FIG. This example is a display screen example of a similar image search result. The upper part of the screen is a search condition display field J, and the lower part is a display field K of similar image search results based on the condition. In the similarity display field R1 for each image of the search result, the image similarity and the text similarity are numerically displayed.

  As another example of the similarity display, if the image similarity and the text similarity are indicated by the length of the black portion of the bar graph for each image as in the similarity display column R2 shown in FIG. There is an advantage that can be grasped visually. As another example, in the similarity display field R3 shown in FIG. 10, the height of each of the image similarity and the text similarity is relatively shown by a band graph. Can be easily grasped in a simple display form.

[3. effect〕
In the present embodiment, as described above, when a similar image is searched based on the specified image, not only the similarity of the image feature amount with the specified base image but also the specified base image is associated. By using the similar image narrowing based on the similarity based on the text feature amount (for example, FIG. 2) for the text that has been added (step S36 in FIG. 3), the search accuracy when performing a similar image search using the image as a key is reduced. Can be prevented.

  In particular, in the present embodiment described above, the specification of a query image is received from a search image based on the specified text (for example, FIG. 4), and the specified text is used as the basis of the text feature amount on the query side (see FIG. Steps S32 to S34 of FIG. 3, FIGS. 6 (1) and (2)), the text itself consciously specified can be reflected in the similar image search, so that the prevention of a decrease in search accuracy is further ensured.

  Further, in the above embodiment, when the similarity of text feature amounts is calculated using a plurality of texts associated with an image, for example, the weight of the text directly designated by the user by type input or the like is relatively set. By performing predetermined weighting such as enlarging (step S35 in FIG. 3) (FIGS. 6 (1) and (2)), a highly accurate search result suitable for the user's intention can be realized.

  Further, in the present embodiment described above, the search result obtained can be easily grasped by indicating whether the obtained search result is suitable for an image or text (for example, FIGS. 8 to 10).

[4. Other embodiments]
In addition, said each embodiment is only an illustration, and this invention includes what is illustrated below and other embodiment other than that. For example, the designated text is not limited to type input from a keyboard or the like, but may be designated by selecting from a list of keywords of interest, for example. Further, although the above embodiment is based on the client-server system (FIG. 1) including the similar image search device 1 and the terminal T, this device may be realized as a stand-alone device.

  In addition, each element such as means may be realized by other information processing mechanisms such as an electronic circuit based on a wired logic or the like without being limited to an arithmetic control unit of a computer. Further, each configuration diagram, data diagram, flowchart diagram, and the like are merely examples, and the presence / absence of each element, its order, specific contents, and the like can be changed as appropriate. For example, the apparatus of the present invention may be realized by using a plurality of apparatuses such as servers, and a configuration in which each storage unit is realized by a separate and independent server apparatus or system is also common. Depending on the function, the configuration can be flexibly changed, for example, by calling an external platform or the like with an API (application program interface) or network computing (so-called cloud or the like).

DESCRIPTION OF SYMBOLS 1 Similar image search device 6 Arithmetic control part 7 Storage device 8 Communication means 11 Feature memory means 12 Acceptance means 13 Similar image search means 14 Weighting means 15 Result display means 16 Similarity display means B1 Search button B2 Similar search button F Character string input Field G Query image J, K Condition display field N Communication network P Mouse pointer R1, R2, R3 Similarity display field T Terminal

Claims (6)

Feature storage means for storing, for each image, an image feature and a text feature based on text associated with the image;
Accepting means for accepting a request instruction for a similar image search with image designation from a user;
By referring to the feature storage unit for each image that is a candidate to be searched, the similarity of the image feature amount with respect to the image specified in the request instruction and the text associated with the image specified in the request instruction Similar image search means for searching for a similar image based on the similarity of the text feature amount,
A result display means for displaying a search result of similar images;
A similar image retrieval apparatus comprising: The accepting means accepts designation of an image as a query for a similar image search from images retrieved based on designated text,
The similar image search device according to claim 1, wherein the similar image search unit uses the specified text as text associated with the specified image. When calculating the similarity of the text feature amount using a plurality of texts associated with the specified image, there is provided a weighting means for performing a predetermined weighting between the plurality of texts used for the calculation. The similar image search device according to claim 1 or 2.   For the retrieved image, at least one of the similarity or similarity of the image feature amount with respect to the specified image and the similarity of the text feature amount with respect to the text associated with the specified image 4. The similar image search apparatus according to claim 1, further comprising a similarity display unit configured to display the image. A similar image search method in which a computer having feature storage means for storing an image feature amount and a text feature amount based on text associated with an image performs a similar image search for each image,
An accepting step in which a computer accepts a request instruction for a similar image search accompanied by designation of an image from a user;
By referring to the feature storage means for each image that is a candidate to be searched, the computer associates the image feature quantity similarity with the image specified in the request instruction and the image specified in the request instruction. A similar image search step for searching for a similar image based on the similarity of the text feature quantity to the selected text;
A result display step in which a computer displays a search result of similar images;
A similar image search method comprising: A similar image search program for causing a computer to perform a similar image search,
For each image, the computer realizes feature storage means for storing an image feature amount and a text feature amount based on text associated with the image,
Let the computer accept a request for a similar image search with image designation from the user,
By referring to the feature storage unit for each image that is a candidate to be searched for by the computer, the similarity of the image feature amount with respect to the image specified in the request instruction and the image specified in the request instruction are associated with each other. Search for similar images based on the similarity of the text feature to the text
A similar image search program for displaying a similar image search result on a computer.

Images