JP2011138420A - Search system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To search a plurality of images similar to or related to a query image at high speed. <P>SOLUTION: A search system 10 searches an image by the following procedure. (1) A query image and an area code are inputted (S505). (2) Each feature amount (color, contour, and composition) of the query image is extracted (S510). (3) Search conditions are set for extracting an image corresponding to the area code inputted in (1) and having the highest degree of similarity to each feature amount extracted in (2) (S515). (4) An image ID of a similar image satisfying the search conditions set in (3) is extracted from image feature information of a search DB 11 (S520). (5) Image IDs of surrounding images corresponding to the image ID of the similar image extracted in (4) are extracted from surrounding-image information of the search DB 11 (S525). (6) The image ID of the similar image extracted in (4) is related to the arrangement position on the plane together with the image IDs of the surrounding images extracted in (5) so as to output them together (S530). <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image retrieval system, and more particularly to a process for retrieving an image similar to or related to a query image.

<Prior art 1>
A system is known that searches for similar images based on the similarity of feature amounts extracted from product images (see, for example, Non-Patent Document 1).
In this system, in order to improve the accuracy of the search, the background portion of the product image is removed in advance, and the feature amount is extracted only from the target product portion. The feature amount extracted from the product image is color, texture (pattern), shape, and category.
A product recommendation system using the above system is also known (see Patent Document 1).

<Prior Art 2>
A similar image search system that displays product images as search results in a grid pattern (mosaic pattern) on a search result page is known (see, for example, Non-Patent Documents 1 and 2).
In the system, the clicked product image is arranged at the center of the display area, and the similar image is arranged around it.

<Prior Art 3>
2. Description of the Related Art A website image search system that searches a website from images related to products or services is known (see, for example, Patent Document 2).
When the server (1) constituting the system receives an image taken by the camera-equipped mobile phone (4) by e-mail (S11 to S12 in FIG. 3), it extracts a feature amount from the received image (FIG. 3). S13), searching for a website having a similarity greater than or equal to a predetermined value compared with the feature quantity of the website stored in the database, and sending the URL of the search result to the mobile phone (4) by e-mail A reply is made (S14 in FIG. 3).

JP 2009-251850 A JP 2003-271650 A

ALBERT Co., Ltd., “Similar image search system“ SUDACHI ”” [online], Internet <URL: http://www.albert2005.co.jp/technology/sudachi.html> ALBERT Inc., “Find.jp Help”, [online], Internet <URL: http://mitsukaru.jp/html/info/faq.htm>

The problem to be solved by the present invention is that a plurality of images similar to or related to a query image are searched at high speed.
The problem to be solved by the present invention is that the load required for the search process is reduced as compared with the case where a plurality of similar images and related images are individually searched.

  In order to solve the above problem, the search system of the present invention has search data storage means for storing the feature amount of the image in correspondence with the identification information of the image in which the surrounding image is set in advance. A search request input means for inputting a query image or identification information thereof, a feature amount of the query image is extracted, and a search condition for extracting an image similar to the query image is set using the extracted feature amount. Search condition setting means, similar image search means for extracting identification information of similar images satisfying the set search conditions from the search data storage means, and identification information of the extracted similar images as images around the similar images And a search result output means for outputting together with the identification information.

The search data storage means further stores identification information of images around the image in correspondence with the identification information of the image, and the search result output means stores identification information of the images around the similar image. May be extracted from the search data storage means.
The identification information of the image corresponds to an arrangement position on a plane, and the search result output unit specifies identification information of an image around the similar image by calculation using the identification information of the similar image. May be.
The search data storage means further stores an image URL related to the image in correspondence with the identification information of the image, and the search result output means stores each identification information of the similar image and surrounding images. Corresponding image URLs may be extracted from the search data storage means, and each image URL may be output in association with the arrangement position on the plane.

The search request input means receives the query image or its identification information from the user terminal, and the search result output means obtains each image URL of the similar image and its surrounding images. A search result page including an image reference as a reference destination may be generated and transmitted to the user terminal.
The search result output means sets a link for transmitting the corresponding image identification information to each image reference, and the search request input means outputs the image identification information via the search result page. You may receive from the said user terminal.

The search data storage means further stores geographical information corresponding to the subject of the image in association with the identification information of the image, and the search request input means includes the geographical information together with the query image or its identification information. The search condition setting means may set the search condition further using the received geographical information.
The image identification information is an image URL corresponding to an arrangement position on a plane, and the search result output means specifies the URL of an image around the similar image by calculation using the URL of the similar image. May be.

  In order to solve the above-described problem, the search method of the present invention includes search data storage means for storing the feature amount of an image corresponding to the image identification information set in advance for the surrounding image. A search request inputting step for inputting a query image or identification information thereof, a search for extracting a feature amount of the query image, and extracting an image similar to the query image using the extracted feature amount A search condition setting step for setting conditions, a similar image search step for extracting identification information of similar images satisfying the set search conditions from the search data storage means, and the identification information of the extracted similar images And a search result output step for outputting together with identification information of the surrounding image.

  In order to solve the above-described problem, the search program of the present invention has search data storage means for storing the feature amount of the image in correspondence with the identification information of the image in which the surrounding image is set in advance. A search request input step for inputting a query image or identification information thereof to a computer system, a feature amount of the query image is extracted, and a search for extracting an image similar to the query image using the extracted feature amount A search condition setting step for setting conditions, a similar image search step for extracting identification information of similar images satisfying the set search conditions from the search data storage means, and the identification information of the extracted similar images And a search result output step for outputting together with the identification information of the surrounding image.

The search system of the present invention has search data storage means for storing the feature amount of the image corresponding to the identification information of the image in which the surrounding image is set in advance, and features of the input query image The search condition for extracting an image similar to the query image is set using the extracted feature quantity, the identification information of the similar image satisfying the search condition is extracted from the search data storage means, and the extracted similarity The identification information of the image is output together with the identification information of the surrounding image.
For example, if images around each image are set based on the similarity / relationship of the feature amount, a plurality of similar images / related images can be searched in a time for searching for one similar image similar to the query image. Can be output.
Therefore, the search system of the present invention can output identification information of a plurality of images similar or related to the input query image at high speed. Further, if the search system of the present invention is used, the load required for the search process is reduced as compared with the case where a plurality of similar images and related images are individually searched.

It is explanatory drawing of the outline | summary of the system of embodiment. (Embodiment) It is a block diagram which shows a system configuration. (Embodiment) It is explanatory drawing of the data item of a database. (Embodiment) It is a flowchart which shows a search procedure. (Embodiment) It is a flowchart which shows the procedure of a similar image search process. (Embodiment) It is a display example of a search result page. (Embodiment) It is explanatory drawing of the data item etc. of a database. (Modification 1) It is a flowchart which shows the procedure of a similar image search process. (Modification 1)

<Embodiment>
[1. Outline of Embodiment]
[(A) Premise]
In the present embodiment, the positional relationship on the plane of all the images to be searched is set based on a specific viewpoint (for example, the similarity / relevance of the feature amount, the location of the subject, etc.). In other words, surrounding images when images are arranged on a plane are preset for each image.
FIG. 1A illustrates the positional relationship of images. In the present embodiment, as shown in FIG. 1A, it is assumed that surrounding images are specified for each image when the images are arranged in a grid pattern (mosaic shape).

[(B) Features]
In the system of this embodiment, identification information of an image similar to a query image (hereinafter referred to as “similar image”) is combined with identification information of an image (hereinafter referred to as “surrounding image”) to be arranged around the image. Output.
FIG. 1B illustrates the positional relationship between the similar image and the surrounding image. When the system of this embodiment searches for one similar image (110) similar to the query image, the similar image (110) and a plurality of surrounding images to be arranged around the similar image (110) The identification information of m × n (sheets) images (for example, 3 × 3 = 9 (sheets) images included in the range of 120) is output as a search result.

[(C) Effect]
As described above, the system according to the present embodiment can search a plurality of similar images and related images including the similar image and its surrounding images in a time for searching for one similar image similar to the query image.
Therefore, the system of this embodiment can search a plurality of images similar to or related to the query image at high speed. Further, if the system of the present embodiment is used, the load required for the search process is reduced as compared with a case where a plurality of similar images and related images are individually searched.

[2. System configuration]
FIG. 2 shows the configuration of the system of this embodiment.
As shown in FIG. 2, the system of the present embodiment is a client / server system in which the search system 10 and the user terminal 20 can be connected via a communication network (the Internet 30 or the like in the present embodiment).
The search system 10 has a search DB 11 and an image DB 12.

[2-1. Search system]
In FIG. 2, a search system 10 is a server group that provides an image search service.
The search system 10 includes, for example, a Web server having a Web page generation / transmission function, a DB server that manages a database, a search server that executes a similar image search process, and other necessary server devices.

[2-2. Search DB]
In FIG. 2, a search DB 11 is a database that stores information used to provide an image search service.
In the present embodiment, it is assumed that a plurality of pieces of image feature information, image URL information, and surrounding image information are stored.

FIG. 3A shows main items of the image feature information.
As shown in FIG. 3A, one piece of image feature information includes “image ID”, “region code” corresponding to the location of the subject of the image, and “feature amount” extracted from the image. .
In the present embodiment, it is assumed that three types of color, contour, and composition extracted from each image are stored as “features”. Note that the method for extracting various feature amounts from the image may be the same as the conventional example (for example, the method of <Prior Art 1>).

FIG. 3B shows main items of the image URL information.
As shown in FIG. 3B, one piece of image URL information is obtained by associating an “image ID” with an “image URL” of the image.

FIG. 3C shows main items of the surrounding image information.
As shown in FIG. 3C, one piece of surrounding image information includes an “image ID” and an image ID of the surrounding image related to the image (“ambient image ID”).
In the present embodiment, “peripheral image IDs” related to eight surrounding images (upper left image, upper image, upper right image, left image, right image, lower left image, lower image, lower right image) are stored. To do. This data structure is suitable for outputting 3 × 3 = 9 (sheets) images as search results.
The number of “ambient image IDs” to be stored can be changed as appropriate. By storing at least “upper or lower” and “left or right” “peripheral image ID” for each image, all images are arranged on a plane according to a predetermined positional relationship. Is possible.

[2-3. Image DB]
In FIG. 2, an image DB 12 is a database that stores images to be searched in the image search service.
Assume that each image is stored in a predetermined file format (for example, JPEG, GIF, etc.). Also, it is assumed that each image has an appropriate file name and can be specified by the corresponding image URL.

[2-4. User terminal]
In FIG. 2, a user terminal 20 is a terminal used by a user of an image search service.
The user terminal 20 has a web browser, and can read the web page received from the search system 10 and display it on the display.
In the present embodiment, an existing information processing terminal (for example, a mobile phone) having a communication function and an imaging function is suitable as the user terminal 20.

[3. Search process]
[3-1. Search process steps]
FIG. 4 shows a procedure of search processing by the search system 10.
As a premise, it is assumed that the user operates the user terminal 20 and designates a query image and a region on a predetermined search request input page (not shown). The query image is, for example, a still image (for example, JPEG format) captured using the imaging function of the user terminal 20.
[11] A search request is received from the user terminal 20 (S405). In this embodiment, the image data of the designated query image and the region code corresponding to the designated region are received. Note that identification information (for example, an image ID, an image URL, etc.) of the query image may be received. Moreover, the structure which does not receive an area code may be sufficient.

[12] A similar image search process is executed (S410, FIG. 5).
[13] A search result page is generated (S415) and transmitted to the user terminal 20 (S420). Specifically, the image URL corresponding to each image ID output by the similar image search process is extracted from the image URL information (FIG. 3B) of the search DB 11, and each image URL is referred to as the reference destination. Generate a search results page with references.
[14] When a request for a reference image specifying a URL is received from the user terminal 20 (S425), a reference image corresponding to the URL is extracted from the image DB 12 (S430) and transmitted to the user terminal 20 (S435). Note that the procedure [14] is repeated a plurality of times.

[3-2. Similar image search process]
FIG. 5 shows a procedure of similar image search processing by the search system 10.
[21] A query image and a region code are input (S505). When the identification information of the query image is received, the corresponding query image is acquired.
[22] The feature amount of the query image is extracted (S510). In this embodiment, color, outline, and composition feature quantities are extracted. Note that the method for extracting various feature amounts from the image may be the same as the conventional example (for example, the method of <Prior Art 1>).
[23] Search conditions are set (S515). In the present embodiment, a search condition is set for extracting an image corresponding to the region code input in [21] and having the highest similarity with each feature amount extracted in [22].

[24] Image IDs of similar images satisfying the search conditions set in [23] are extracted from the image feature information (FIG. 3A) of the search DB 11 (S520).
[25] The image ID of the surrounding image corresponding to the image ID of the similar image extracted in [24] is extracted from the surrounding image information (FIG. 3C) in the search DB 11 (S525).
[26] The image ID of the similar image extracted in [24] is output in association with the arrangement position on the plane together with the image ID of the surrounding image extracted in [25] (S530).

[3-3. Search results page]
FIG. 6 shows a display example of the search result page.
The search result page 600 is a display example of a Web page transmitted to the user terminal 20 when a search request is received from the user terminal 20.
As shown in FIG. 6, the following information is displayed on the search result page 600.
-Posted query image (610).
-The designated area (620).
Search result (630). In the display example of FIG. 6, a total of nine images (3 rows and 3 columns) of an image (similar image) having the highest degree of similarity with the query image (610) and images (surrounding images) to be arranged around the image are similar. It is displayed. All nine image subjects (accommodation facilities) are located in the designated area (“Aomori Prefecture”).

It is assumed that each image displayed in the search result (630) column is provided with a link for transmitting a search request designating an image ID related to the image to the search system 10.
When the user operates the user terminal 20 and clicks one of the images displayed on the search result page, a search request including the image ID of the image (query image) is transmitted to the search system 10. When receiving a search request from the user terminal 20, the search system 10 executes a search process (FIG. 4).

[4. Modified example]
[4-1. The image ID corresponds to the arrangement position (Modification 1)]
[(A) Overview]
In the above-described embodiment, the surrounding image information (FIG. 3C) is stored in the search DB 11, and therefore the image ID assigned to each image may be an arbitrary character string. In this respect, the mechanism of the embodiment can flexibly cope with a change in positional relationship and the like, and has excellent expandability.
On the other hand, if the ID of each image corresponds to the arrangement position on the plane, it is not necessary to store the surrounding image information in the search DB 11. Therefore, the mechanism of the first modification can suppress the data storage capacity.

[(B) Modified example of data item etc.]
FIG. 7A illustrates how to assign an image ID to each image.
As shown in FIG. 7A, in the first modification, each arrangement position on the plane corresponds to “[row number] [column number]”, and the row number and column number of the position to be arranged in each image. An image ID including is attached. For example, the image ID may be “(row number) _ (column number)” “(row number) − (column number)” or the like. The “(row number)” and “(column number)” are both half-width numeric strings with an appropriate number of digits.

FIG. 7B shows the main items of the image feature information (similar to the image feature information of FIG. 3A).
As shown in FIG. 7B, one piece of image feature information includes “image ID”, “region code” corresponding to the location of the subject of the image, and “feature” extracted from the image. .
Also in the first modification, it is assumed that three types of color, contour, and composition extracted from each image are stored as “features”.

FIG. 7C shows main items of the image URL information (similar to the image URL information of FIG. 3B).
As shown in FIG. 7C, one piece of image URL information associates an “image ID” with the “image URL” of the image.

[(C) Modified Example of Similar Image Search Processing]
FIG. 8 shows a procedure of similar image search processing by the search system 10.
In FIG. 8, the processes denoted by the same reference numerals as those in FIG. 5 are the same processes as those in FIG.
[31] A query image and a region code are input (S505). When the identification information of the query image is received, the corresponding query image is acquired.
[32] The feature amount of the query image is extracted (S510). In the first modification, feature quantities of color, contour and composition are extracted.

[33] A search condition is set (S515). In the first modification, a search condition for extracting an image corresponding to the region code input in [31] and having the highest similarity with each feature amount extracted in [32] is set.
[34] Image IDs of similar images satisfying the search conditions set in [33] are extracted from the image feature information (FIG. 7B) in the search DB 11 (S520).
[35] Using the image ID of the similar image extracted in [34], the image ID of the surrounding image is specified by calculation (S825). For example, if the image ID of the similar image is [m] [n], the image ID of the surrounding image can be specified by simple calculation (see FIG. 7A).
[36] The image ID of the similar image extracted in [34] is output in association with the arrangement position on the plane together with the image ID of the surrounding image specified in [35] (S830).

[4-2. The image URL is output in association with the arrangement position (Modification 2)]
In the above-described embodiment, the image ID is output in the similar image search process (S530 in FIG. 5), and the image URL corresponding to the image ID is extracted from the search DB 11 in the search result page generation step of the search process (FIG. 5). 4 S415). The same applies to the first modification (S830 in FIG. 8 and S415 in FIG. 4).
In contrast, in the similar image search process, the image URL corresponding to each image ID is extracted from the image URL information (FIG. 3B or FIG. 7C) in the search DB 11, and each image URL is arranged on a plane. You may output in relation to a position. The effects obtained are the same as those of the above-described embodiment or modification 1 only by changing the processing sharing in the search system 10.

[4-3. The image URL corresponds to the arrangement position (Modification 3)]
[(A) Overview]
In the first modification described above, since the ID of each image corresponds to the arrangement position on the plane, it is not necessary to store the surrounding image information in the search DB 11. Therefore, the mechanism of the first modification can suppress the data storage capacity.
On the other hand, if the file name of each image (part of the image URL) is an image ID and each file name corresponds to the arrangement position on the plane, the image URL information does not have to be stored in the search DB 11. . Therefore, the mechanism of the modification 3 can further suppress the data storage capacity as compared with the mechanism of the modification 1.

10 Search system 11 Search DB
12 Image DB
20 User terminal 30 Internet 600 Search result page

Claims (10)

Search data storage means for storing the feature amount of the image corresponding to the image identification information set in advance for the surrounding image;
Search request input means for inputting a query image or identification information thereof;
Search condition setting means for extracting a feature amount of the query image and setting a search condition for extracting an image similar to the query image using the extracted feature amount;
Similar image search means for extracting identification information of similar images satisfying the set search conditions from the search data storage means;
And a search result output means for outputting the extracted identification information of the similar image together with the identification information of the surrounding image of the similar image. The search system according to claim 1,
The search data storage means further stores identification information of images around the image in correspondence with the identification information of the image,
The search system is characterized in that the search result output means extracts identification information of images around the similar image from the search data storage means. The search system according to claim 1,
The identification information of the image corresponds to the arrangement position on the plane,
The search result output means specifies identification information of an image around the similar image by calculation using the identification information of the similar image. The search system according to any one of claims 1 to 3,
The search data storage means further stores an image URL related to the image in association with the identification information of the image,
The search result output unit extracts an image URL corresponding to each identification information of the similar image and the surrounding image from the search data storage unit, and outputs the image URL in association with an arrangement position on a plane. Search system characterized by The search system according to claim 4, wherein
It can be connected to the user terminal,
The search request input means receives the query image or identification information thereof from the user terminal,
The search system according to claim 1, wherein the search result output unit generates a search result page including an image reference using each image URL of the similar image and the surrounding image as a reference destination, and transmits the search result page to the user terminal. The search system according to claim 5, wherein
The search result output means sets a link for transmitting identification information of a corresponding image to each image reference,
The search system is characterized in that the search request input means receives image identification information from the user terminal via the search result page. The search system according to claim 5 or 6,
The search data storage means further stores geographical information corresponding to the subject of the image in association with the identification information of the image,
The search request input means receives geographical information from the user terminal together with the query image or identification information thereof,
The search condition setting means further sets the search condition using the received geographical information. The search system according to claim 1,
The image identification information is an image URL corresponding to an arrangement position on a plane,
The search result output means specifies the URL of an image around the similar image by calculation using the URL of the similar image. A computer system having search data storage means for storing a feature amount of an image in association with surrounding image identification information set in advance.
A search request input step for inputting a query image or identification information thereof;
A search condition setting step of extracting a feature amount of the query image and setting a search condition for extracting an image similar to the query image using the extracted feature amount;
A similar image search step of extracting identification information of similar images satisfying the set search condition from the search data storage means;
A search result output step of outputting the extracted identification information of similar images together with the identification information of images around the similar images. In a computer system having search data storage means for storing the feature amount of the image corresponding to the image identification information set in advance for the surrounding image,
A search request input step for inputting a query image or identification information thereof;
A search condition setting step of extracting a feature amount of the query image and setting a search condition for extracting an image similar to the query image using the extracted feature amount;
A similar image search step of extracting identification information of similar images satisfying the set search condition from the search data storage means;
A search result output step of outputting the extracted identification information of similar images together with the identification information of images around the similar images.