JP5121872B2 - Image search device - Google Patents

Description

  The present invention relates to a technique for retrieving image data related to a query.

  In a similar image search known as an image search technique, feature amounts for colors, shapes, patterns, and the like are extracted from images, and image searches are performed based on the similarity of feature amounts between images. The user can search for visually similar images by designating images as queries. However, since the feature amount extracted by the computer is used, the accuracy of searching for an image that matches the query image in terms of content is not necessarily good.

  On the other hand, in tag image search, which is one of image search techniques, text information called a tag is assigned to an image, and image search is performed based on the tag. In general, words representing image characteristics are used for this tag. By comparing the tag specified as the query with the tag attached to the image, an image related to the content of the tag can be searched.

  As a technique combining the above-described similar image search and tag image search, there is a technique in which a similar image is searched based on image characteristics, and tag image search is performed using incidental information (tag) attached to the searched image. It is known (see Patent Document 1).

JP 2007-172195 A

  As described above, the tag image search is performed based on a tag that is a word representing the feature of the image. Therefore, search noise can be expected to be reduced by manually attaching the tag. However, when a tag is manually attached, there is a tendency that the tag content itself tends to vary. For example, various words such as “automobile”, “four-wheel drive vehicle”, “specific vehicle type name”, and the like may be used as tags for an automobile image. Further, it is artificially expensive for the user to confirm the contents of the image one by one and to give the tags comprehensively.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to improve the accuracy of tag image search by giving a tag to an image while suppressing artificial costs.

In order to solve the above-described problem, the first aspect of the present invention provides:
Storage means for storing an image, tag information, and a feature amount of the image in association with each other;
Query tag search means for searching for an image stored in the storage means based on tag information input as a query;
Similar image search means for searching for a similar image from the storage means based on the feature amount of the searched image;
Semantically related tag extraction means for extracting tag information semantically related to the tag information input as the query from the tag information associated with the similar image;
A tag registration unit that searches the storage unit for an image associated with the extracted tag information, associates the input tag information with the image, and additionally stores the tag information in the storage unit;
It is characterized by having.

  According to the first aspect, tag information semantically related to the query is collected from similar images similar to the search result obtained based on the tag information (query tag) input as the query, and the tag information In addition, a query is registered for the image retrieved in step (1). For this reason, the search range is expanded with respect to the query, and a tag is automatically assigned to a semantically related image. Therefore, the tag image search accuracy can be improved by adding a tag to the image while suppressing artificial costs.

  Further, in the second aspect, the tag registration means calculates a similarity between the image associated with the semantically related tag information and the image searched by the tag image search means, The tag information is additionally stored based on the degree of similarity.

  According to the second aspect, the tag is attached based on the similarity to the image searched for the query. For this reason, even in an image that is semantically related to a query, it is possible to narrow down the tag assignment range based on the visual similarity. Therefore, the tagging accuracy can be increased.

  In the third aspect, the storage unit further stores a weight value indicating a degree of association between the image and the tag information in association with each other, and the tag registration unit performs the input based on the calculated similarity degree. It is characterized in that the weight value of the tag information is set.

  According to the third aspect, since the calculated similarity is stored as the degree of association between the image and the tag information, it can be used as the degree of association of the search result when performing an image search based on the tag. The accuracy of tag image search can be improved.

  Further, in the fourth invention, there is further provided search result output means for outputting the image searched by the tag registration means and the image searched by the tag image search means as a search result for the input tag information. It is characterized by providing.

  According to the fourth aspect of the invention, tagging is sequentially performed by a query input by the user to obtain an image search result. For this reason, indexing of tag information can be promoted by a user's search act.

  In the fifth aspect, the image processing device further comprises tag image search means for searching for an image from the storage means based on the input tag information, wherein the query tag search means uses the tag information input by a user as the tag image. The image is searched by inputting to the search means, and the tag registration means searches the image associated with the tag information by inputting the extracted tag information to the tag image search means. It is characterized by that.

  According to the fifth aspect, the query tag search means and the tag registration means can obtain an image search result by sharing one tag image search means.

  According to the present invention, it is possible to improve the accuracy of tag image search by adding a tag to an image while suppressing artificial costs.

The block diagram which shows an example of the block of an image search device. The figure which shows the data structural example of image DB, a tag index, and a feature-value index. 6 is a flowchart illustrating an example of processing operation of the image search apparatus according to the first embodiment. The figure which shows the specific example of the image search and tag provision in 1st Embodiment. 10 is a flowchart illustrating an example of processing operation of the image search apparatus according to the second embodiment. The figure which shows the specific example of the image search and tag assignment in 2nd Embodiment. The figure which shows the data structural example of the tag index in other embodiment.

[First Embodiment]
First, an image search apparatus according to a first embodiment to which the present invention is applied will be described with reference to the accompanying drawings. FIG. 1 is a block diagram illustrating an example of a functional configuration of the image search apparatus 1 according to the present embodiment. The image search device 1 is connected to a user terminal T through a communication network such as the Internet so that they can communicate with each other.

  The user terminal T is a terminal provided with an input function for a user to input a query for image search, and an output function for displaying and outputting a search result corresponding to the query, and includes a CPU, an input device, a display device, and the like. This is realized by a personal computer, a portable terminal or the like.

  The image search device 1 performs an image search based on the query input at the user terminal T, and returns the search result to the user terminal T. As shown in FIG. 1, the image search device 1 includes a query receiving unit 10, a search unit 20, an output unit 40, an image DB unit 50, and a thesaurus dictionary 90.

  The query receiving unit 10 receives and receives a query for image search from the user terminal T. A query that a user inputs for image search includes a keyword and an image. The query receiving unit 10 generates a query tag based on the keyword input as a query. The query tag is tag information (text information), and an example of the generation method is given below.

・ A method that uses the keyword entered by the user as a query tag ・ A method that searches for a tag that has a relationship with the keyword entered by the user and uses that tag as a query tag To select and use this as a query tag

The keyword can be input at the user terminal T. As a keyword input method,
A method in which the user inputs text. There is a method in which the user selects from among options presented to the user.

  The search unit 20 is a functional unit that searches for an image stored in the image DB unit 50 based on a query, and is similar to a tag image search unit 22 that performs an image search using a tag as a query, and an image search that uses an image as a query. The image search unit 24 includes a semantic related tag extraction unit 26 and a tag addition unit 28.

  The image DB unit 50 includes an image DB 60, a tag index 70, and a feature amount index 80. The image DB 60 is a database that stores an image ID (for example, a file name) that uniquely identifies an image and image data in association with each other as shown in FIG. When the image is managed as a file, the file name is associated with the header information of the image data, and the image data and the file name are managed integrally.

  The tag index 70 is a database that stores an image ID stored in the image DB 60 and one or a plurality of tag information in association with each other as shown in FIG. It becomes an index. An association is made between the image data stored in the image DB 60 and the tag information stored in the tag index 70 by the image ID.

  The tag information is a character string or a symbol string having some meaning, and of course can be constituted by a single character or symbol. For example, in the case of kanji, even a single character is meaningful and can function as a tag.

  The image and tag information stored in the image DB unit 50 may be collected from the Internet by a crawler based on so-called robot search. At this time, the tag information may be set by dividing text information described in the vicinity of the image into words by morphological analysis or the like on the web page on which the image is posted. As tag information, an image file name, URL, or ALT tag (anchor text) may be set.

  The tag image search unit 22 is input by performing a tag image search that searches the tag index 70 for an image ID associated with tag information that matches the input tag information (or approximately matches forward match or the like). Obtain image search results for tag information. As input of tag information, there are an input from the query receiving unit 10 and an input from the tag providing unit 28.

  The feature amount index 80 is a database that stores an image ID and an image feature amount in association with each other as shown in FIG. 2C, and the feature amount is an index for search. Here, examples of the feature amount of the image include color, shape, texture, and the like, and all the features can be represented by multidimensional feature vectors. When calculating the similarity (or distance) of images, the similarity between feature vectors is calculated. The image ID is associated with the image data stored in the image DB 60 and the feature amount stored in the feature amount index 80.

  The similar image search unit 24 calculates and extracts a feature amount from a query image (query image), and calculates a similarity between the feature amount and the feature amount stored in the feature amount index 80. To obtain an image search result for the query image.

  In the image search device 1 of the present embodiment, first, a query tag generated by the query receiving unit 10 is input to the tag image search unit 22 and a tag image search (first tag image search) is performed based on the query tag. The similar image search is performed using the search result image as a query image. This provides an image that is visually similar to the image associated with the query tag.

  The semantic related tag extraction unit 26 collects tag information given to the similar image obtained by the similar image search, and extracts tag information semantically related to the query tag from the tag information. The thesaurus dictionary 90 is used to collect semantically related tag information.

  The thesaurus dictionary 90 is a dictionary DB that records relevance between words, and stores synonyms that are semantically similar in association with each other. The semantic related tag extraction unit 26 reads a synonym for the query tag from the thesaurus dictionary 90, and extracts tag information that approximately matches the synonym from the tag information collected as the semantic related tag.

  As a method for extracting semantic related tags, the following method may be used in addition to the thesaurus dictionary 90. That is, the attributes (categories, etc.) set for the words included in the tag information are mapped to the vector space and expressed as feature vectors, and the distance between the feature vectors (for example, cosine distance) is calculated, It is good also as measuring a closeness (relevance) and extracting a semantically relevant tag. In addition, as a method for calculating the closeness of words, a known technique based on language processing may be adopted as appropriate.

  The tag assigning unit 28 causes the tag image search unit 22 to perform a second tag image search by inputting the semantic related tag extracted by the semantic related tag extraction unit 26 to the tag image search unit 22. Get the search results. Thereby, in addition to the search result based on the query tag generated by the query receiving unit 10, the search result by the semantically related tag is obtained.

  In addition, the tag assigning unit 28 uses the tag information generated by the query receiving unit 10 as tag information for an image obtained as a result of a second tag image search, that is, a tag image search using a semantically related tag. Are additionally registered in the tag index 70. Thereby, the query tag used in the first tag image search can be given to the search result of the second tag image search.

  As described above, the query tag search unit of the present invention is realized by the association between the query receiving unit 10 and the tag image search unit 22, and the tag registration unit is realized by the association between the tag addition unit 28 and the tag image search unit 22. Is done.

  As described above, the query tag is generated based on a keyword or the like input by the user. The tag assigning unit 28 may additionally register the keyword as the generation source in the tag index 70 as tag information.

  The output unit 40 is for outputting the search result obtained by the search unit 20 to the user terminal T. Specifically, the output unit 40 generates display data (for example, web page data) accessible to the image data of the search result. And it transmits to the user terminal T.

  The output unit 40 performs ranking using the degree of association between the query tag and the tag information of the tag index 70 as a list of images searched by the tag image search unit 22 of the search unit 20, and displays the list of images. Data to generate. For example, the result of the first tag image search is ranked higher, and the result of the second tag image search is ranked lower.

[Image search method]
Next, an example of the operation of the image search apparatus 1 will be described with reference mainly to FIGS.

  First, the user inputs a query using the user terminal T. The input query is sent to the query receiving unit 10 via the Internet. The query receiving unit 10 acquires a query tag based on the query as described above (step S11). For example, when a keyword is input as a query, a query tag is acquired from the keyword by the following method. Of course, the user can directly input the tag itself.

(1) A method of identifying a tag close to a query keyword as a character string from among tags stored in the tag index 70 by using similar character string matching.
(2) A method of identifying a tag semantically close to a query keyword from the tags stored in the tag index 70 using the thesaurus dictionary 90.

  The tag image search unit 22 obtains an image having a tag corresponding to the query tag as a search result by performing a first tag image search based on the query tag (step S12). As shown in FIG. 4A, when the tag 1 is acquired as a query tag, two images G1 and G2 associated with the tag 1 are searched (see FIG. 4B).

  Next, the similar image search unit 24 performs a similar image search using the image of the first tag image search result as a query image, and acquires the search result (step S13). That is, here, the image is searched using the feature amount of the image. As a result, images G11 to G13 visually similar to the image G1 and images G21 to G23 similar to the image G2 are searched as shown in FIG.

  Then, the semantic related tag extraction unit 26 collects tags attached to similar images (step S14). Tags collected in this way are referred to as related tags. In FIG. 4D, the tags 3, 4, 5, and 7 assigned to the images G11 to G13 and G21 to G23 are collected, and only tags directly related to the query specified by the user are collected. The tag can be acquired in a wider range.

  The semantic related tag extraction unit 26 extracts a tag (semantic related tag) that is semantically related to the query tag from the collected related tags (step S15). In FIG. 4 (e), three tags of tags 3 to 5 are extracted as semantically related tags.

  For example, the related tag “white box” when the query tag is “car” and the related tag given to the images collected by similar image search is “car” “car” “white box” Although there is a possibility that the tag is not semantically related to the query tag, in step S15, those "car" and "car" that are semantically related to the query tag are extracted as semantically related tags. However, tags other than the query tag can be acquired, and it can be said that the tag is expanded in that range.

  Then, the tag image search unit 22 performs a second tag image search based on the semantic related tag (step S16). In FIG. 4F, three images (G31 to G33, G41 to G43, G51 to G53) are searched for each of the tags 3 to 5. Since the image obtained in this manner is obtained by a search using a tag that is semantically related to the query tag, there is a high possibility that the image is related to the query specified by the user.

  The tag assigning unit 28 additionally stores the query tag as tag information in the tag index 70 for the image of the tag image search result (step S17). That is, the query tag is associated with the image ID of the search result image and stored in the tag index 70.

  As a result, as shown in FIG. 4G, semantically related query tags are automatically assigned to images searched by the second tag image search. Therefore, labor for tagging the image can be saved.

  The output unit 40 returns the document data including the images of the search results in step S12 and step S16 to the user terminal T.

  As described above, according to the present embodiment, the image of the search result is expanded by performing a similar image search from the result of the tag image search performed based on the query tag, and the query tag is semantically converted from the expanded image. By collecting near tags and further performing a tag image search, the search result is prevented from moving away from the meaning of the query tag.

  Moreover, tagging in consideration of the semantic content of the tag can be automatically performed by adding a tag semantically close to the query tag to the final search result image. Therefore, the tag image search accuracy can be improved by adding a tag to the image while suppressing artificial costs.

[Second Embodiment]
Next, a second embodiment will be described with reference to FIGS. Note that functional blocks and processing steps implemented in the same functional configuration as those of the above-described embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  FIG. 5 is a flowchart showing the operation of the image search apparatus 1 in the second embodiment. Also in the second embodiment, the image search apparatus 1 performs the processing from steps S11 to S16 and performs a tag image search based on the semantically related tags, as in the first embodiment. That is, in the example of FIG. 4, a search result in the state (f) is obtained.

  In the second embodiment, between the image obtained by the second tag image search (step S16) of the tag image search unit 22 and the image obtained by the first tag image search (step S12) for the query tag. Is calculated (step S18). This calculation of the degree of similarity can be realized by the function of the similar image search unit 24.

  For example, as shown in FIG. 6, it is assumed that an image G1 is obtained as a tag image search result for the query tag. It is assumed that images G31 to G33, G41 to G43, and G51 to G53 are obtained in the second tag image search. In this case, the similarity between the image G1 and each of the images obtained by the second tag image search is calculated (see FIG. 6H).

  When calculating the similarity, if there are a plurality of search results for the query tag (first tag image search) (for example, images G1 and G2), the average value of the feature values of the plurality of images (image G1 and image G1). An average value of G2 feature values) may be calculated, and tagging may be performed based on the similarity between the average value and the second tag image search result. In this case, it is possible to save the trouble of calculating the similarity for all combinations between the first tag image search result and the second tag image search result.

  In addition, a similarity between a plurality of images (for example, images G1 and G2) as a result of the first tag image search and an individual image (for example, image G31) as a result of the second tag image search is calculated. An average value (an average value of the similarity between the image G1 and the image G31 and a similarity between the image G2 and the image G31) may be calculated, and a tag may be assigned based on the average value. In this case, it is possible to give a tag to each image by comprehensively considering the result of the first tag image search.

  The tag assigning unit 28 assigns and registers a query tag to the image obtained by the second tag image search based on the calculated similarity (step S19). That is, when the similarity is equal to or higher than a predetermined value (for example, when the similarity is 0.5 in the similarity range of 0 to 1 or when the similarity is relatively higher than a certain level), a query tag is assigned.

  For example, if a tag image search result as shown in FIG. 6 is obtained, the tag 1 as a query tag is assigned to the images G31 to G33, G42, G43, and G52 whose similarity is determined to be equal to or higher than a predetermined value. (See FIG. 6I).

  In this way, the tag is assigned using the similarity between the tag image search result for the query tag and the second tag image search result. As a result, the same tag as the query tag can be assigned to an image that is visually similar to the image to which the query tag is assigned, that is, an image that is highly relevant to the query. Can do.

  In the second embodiment, tagging is performed based on the calculated similarity, but the similarity may be set as a weight value in tag image search and further registered.

  For example, an index in which a score (weight value) is associated with the correspondence between the tag information and the image ID is configured like a tag index 72 shown in FIG. This score is a weight value serving as an index indicating the degree of association between tag information and an image.

  When a query tag is registered for an image in step S19 described above, the similarity is added to the score associated with the query tag and the image. In FIG. 6, the image G31 is registered with the tag 1 added thereto, but the similarity is added to the score associated with the tag 1 and the image G31.

  When the tag image search unit 22 searches the tag index 72 for an image associated with the query tag, the tag image search unit 22 outputs an image whose score is a predetermined value or more as a search result. For this reason, an image highly relevant to the query tag is searched, and the accuracy of the tag image search can be further improved.

  The operation of each embodiment described above can be implemented by incorporating appropriate computer software into the computer. The contents of the present invention are not limited to the above embodiment. In the present invention, various modifications can be made to the specific configuration within the scope of the claims.

  For example, each component described above may exist as a functional block, and may not exist as independent hardware. As a mounting method, hardware or computer software may be used. Furthermore, one functional element in the present invention may be realized by a set of a plurality of functional elements, and a plurality of functional elements in the present invention may be realized by one functional element.

  Moreover, the functional element may be arrange | positioned in the position physically separated. In this case, the functional elements may be connected by a network. It is also possible to realize functions or configure functional elements by grid computing.

DESCRIPTION OF SYMBOLS 1 Image search apparatus 10 Query reception part 20 Search part 22 Tag image search part 24 Similar image search part 26 Semantic related tag extraction part 28 Tag provision part 40 Output part 50 Image DB part 60 Image DB
70, 72 Tag index 80 Feature value index 90 Thesaurus dictionary

Claims (7)

Storage means for storing an image, tag information, and a feature amount of the image in association with each other;
Query tag search means for searching for an image stored in the storage means based on tag information input as a query;
Similar image search means for searching for a similar image from the storage means based on the feature amount of the searched image;
Semantically related tag extraction means for extracting tag information semantically related to the tag information input as the query from the tag information associated with the similar image;
A tag registration unit that searches the storage unit for an image associated with the extracted tag information, associates the input tag information with the image, and additionally stores the tag information in the storage unit;
An image search apparatus comprising: The tag registration means includes
Calculating the similarity between the image associated with the semantically related tag information and the image searched by the query tag search means, and additionally storing the tag information based on the similarity The image search device according to claim 1. The storage means further stores a weight value indicating a degree of association between the image and the tag information in association with each other,
The tag registration means includes
The image search apparatus according to claim 2, wherein a weight value of the input tag information is set based on the calculated similarity.   The search result output means for outputting the image searched by the tag registration means and the image searched by the query tag search means as a search result for the input tag information. The image search device according to any one of? Tag image search means for searching for an image from the storage means based on the input tag information;
The query tag search means includes:
By inputting the tag information input by the user into the tag image search means, the image is searched,
The tag registration means includes
5. The image search according to claim 1, wherein an image associated with the tag information is searched by inputting the extracted tag information to the tag image search unit. apparatus. In an image search method in which a computer searches for an image from a storage unit that stores an image, tag information, and a feature amount of the image in association with each other,
A query tag search step of searching for an image stored in the storage unit based on tag information input as a query;
A similar image search step of searching for a similar image from the storage means based on a feature amount of the searched image;
A semantically related tag extraction step of extracting tag information that is semantically related to the tag information input as the query from the tag information associated with the similar image;
A tag registration step of searching the storage unit for an image associated with the extracted tag information, and additionally storing the input tag information in association with the image;
The image search method characterized in that the computer performs the above.   The program for making a computer perform each process of the image search method of Claim 6.

Images