JP2010250529A - Device, and method for retrieving image, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology for improving the convenience of a user interface in presenting the retrieval result of similar image retrieval. <P>SOLUTION: A image retrieval device includes: an image database; a retrieval request acceptance means for accepting a query image; a similar image retrieval means for extracting a similar image which is similar to the query image from images registered in the image database; and a retrieval result presentation means for arranging the similar image around the query image, and for presenting the retrieval result of the connection display of the query image and the similar image at a display means. Then, when the new query image is accepted by the retrieval request accepting means, the retrieval result presentation means holds the connection display of the retrieval results presented by the display means in the past, and adds the retrieval result based on the query image to the retrieval result, and presents the retrieval result at the display means. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image search apparatus, an image search method, and a program for searching for similar images based on an image designated by a user, and in particular, a user interface (UI) for presenting search results of similar images. User Interface).

2. Description of the Related Art Conventionally, keyword search using a keyword as a search key (query) is known as a method for searching various information (for example, web pages, image contents, etc.) published on the Internet. Furthermore, when image content is a search target, by using the image as a search key, comparing image feature amounts (representing the image features such as color schemes, contour lines, patterns, and patterns as numerical values) A technique for searching for an image similar to an image that is a search key (hereinafter referred to as a query image) has been proposed (for example, Patent Document 1).
In the similar image presentation system described in Patent Literature 1, when a product image is specified by a client terminal that has accessed via a network, an image similar to the product image is searched from other product images. A list of thumbnails is presented to the client terminal. Further, a similar image search can be further performed using a similar image presented as a search result as a search key.

  Patent Document 2 discloses a technique related to a user interface when presenting content to a user. Specifically, when a plurality of contents to be searched are presented on the display screen, contents having similar characteristics are connected by a link line. As a result, it is possible to easily grasp the similarity relationship between a large number of contents presented on the display screen, so that desired contents can be efficiently searched from them.

Japanese Patent No. 4058288 JP 2008-129942 A

In the similar image search, the query image specified by the user reflects the user's search intention to know an image similar to this image. In particular, when a query image is selected from the search results and the search is continuously executed, the search history should reflect the search intention of the user.
However, in the conventional similar image search, when a search key (query image) is newly designated from similar images presented as search results, only images similar to the designated query image are presented as search results. Thus, the search history by the query image specified in the past is not reflected.
In addition, even when the technique described in Patent Document 2 is applied when presenting the result of similar image search, the searched similar images are simply connected by link lines based on the individual similarities. When a search key is newly specified from similar images presented as a result, only images similar to the query image are presented as search results.

  As described above, in the conventional similar image search, the user's search intention is not sufficiently reflected in the search result, so that it is difficult to efficiently search for an image desired by the user. Although it is conceivable to display search results in consideration of the history of query images specified by the user (so-called refined search), it is necessary to perform various analyzes using a query log or the like to read the user's search intention. Because there are various problems inherent, there are no practical applications.

  The present invention provides a technique capable of improving the convenience of a user interface when presenting a search result of a similar image search.

In order to achieve the above object, the invention described in claim 1 is an image search device for searching for an image similar to a query image in response to a search request by a query image specified by a user.
An image database in which image feature values are registered for each image;
Search request accepting means for accepting, as a query image, an image designated by the user from the images in the image database displayed on the display means;
Similar image search means for extracting a similar image similar to the query image from images registered in the image database;
Search result presenting means for arranging the similar image around the query image and presenting a search result in which the query image and the similar image are connected and displayed on the display means,
When a new query image is accepted by the search request accepting unit, the search result presenting unit retains a linked display of the search results presented in the past on the display unit, and adds the new search result to the search result. The search result based on the query image is added and presented on the display means.

  According to a second aspect of the present invention, in the image search device according to the first aspect, the search result presenting means erases the connected display of images that are separated from the latest query image by a predetermined distance or more. To do.

The invention according to claim 3 is the image search device according to claim 2, further comprising connection information recording means for recording connection information relating to an image from which the connection display has been deleted,
The search result presenting means redisplays the deleted linked display when a similar image searched for the query image is recorded in the linked information recording means.

  According to a fourth aspect of the present invention, in the image search device according to any one of the first to third aspects, the search result presentation means is based on a similarity between the query image and a similar image similar thereto. The length of the connected display for connecting these is set.

  According to a fifth aspect of the present invention, in the image search device according to the fourth aspect, the search result presenting means is configured to display the plurality of similar images when the searched similar images are common to similar images for other query images. A ratio of lengths of the respective linked displays is set based on the similarity between each of the query images and the common similar image.

  According to a sixth aspect of the present invention, in the image search device according to any one of the first to fifth aspects, the search result presenting means includes a similar image similar to the new query image as a past search result. If it is included, the similar image is arranged in the middle of a plurality of query images similar to the similar image, and the similar image and a new query image are connected and displayed.

  According to a seventh aspect of the present invention, in the image search device according to any one of the first to sixth aspects, the search result presenting means is configured to display a query image and a similar image included in the search result different from each other It is characterized by doing.

  The invention according to claim 8 is the image search device according to any one of claims 1 to 7, wherein the search result presenting means is a similarity degree among the similar images extracted by the similar image search means. A predetermined number is arranged in descending order.

According to the ninth aspect of the present invention, in response to a search request based on a query image designated by a user, a computer of an image search apparatus having an image database in which image feature amounts are registered for each image. A method for searching for an image similar to a query image,
Receiving an image designated by the user from the images in the image database displayed on the display means as a query image;
Extracting a similar image having a feature amount similar to the feature amount of the query image from images registered in the image database;
Disposing the similar image around the query image and presenting the search unit with a search result in which the query image and the similar image are connected and displayed on the display unit;
When a new query image is received, the search result based on the new query image is added to the search result while the linked display of the search results presented in the past is held on the display means and presented to the display means. And a step.

The invention according to claim 10 includes an image database in which image feature amounts are registered for each image, and in response to a search request by a query image specified by a user, an image similar to the query image is obtained. In the computer of the image search device to search,
Receiving an image designated by the user from the images in the image database displayed on the display means as a query image;
Extracting a similar image having a feature amount similar to the feature amount of the query image from images registered in the image database;
Disposing the similar image around the query image and presenting the search unit with a search result in which the query image and the similar image are connected and displayed on the display unit;
When a new query image is received, the search result based on the new query image is added to the search result while the linked display of the search results presented in the past is held on the display means and presented to the display means. Is a program for realizing the steps.

According to the present invention, when an image similar to the latest query image is presented, a search result based on a past query image is included (for example, a connection mode by a link line is maintained). Can be expressed accurately. Further, since the search is repeated, the images that meet the user's search intention are gathered at the center, so that the user can efficiently search for the desired image.
Therefore, the convenience of the user interface when presenting the search result in the similar image search is greatly improved.

It is the schematic shown about an example of the network structure to which the image search apparatus which concerns on this invention is applied. It is a block diagram which shows the outline of the hardware constitutions of a web server. It is explanatory drawing which shows an example of a data structure of an image database. It is explanatory drawing which shows an example of a data structure of the table for image displays. It is a flowchart shown about an example of the process sequence in a web server and a client terminal when a user utilizes a similar image search service. It is explanatory drawing shown about an example of the transition aspect of the search result screen which concerns on a similar image search. It is explanatory drawing shown about another example of the transition aspect of the search result screen which concerns on a similar image search. It is explanatory drawing shown about another example of the transition aspect of the search result screen which concerns on a similar image search.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a schematic diagram showing an example of a network configuration to which an image search apparatus according to the present invention is applied.
As shown in FIG. 1, a client terminal 2 such as a personal computer 2a or a portable terminal 2b is connected to a web server 1 via a communication network N so that data communication can be performed with each other. The communication network N is a telephone line network such as the Internet or a telecommunication carrier, a mobile phone communication network, or the like.

  In FIG. 1, a web server 1 provides a similar image search service for searching for images published on the Internet using a query image to the client terminal 2. When a search request based on a query image is made by the user at the client terminal 2, an image similar to the query image (similar image) is extracted in response to this, and the search result is presented to the client terminal 2. That is, in this embodiment, the web server 1 constitutes an image search device according to the present invention.

FIG. 2 is a block diagram showing an outline of the hardware configuration of the web server 1.
As shown in FIG. 2, the web server 1 is a general computer including a control unit 11, a storage unit (auxiliary storage) 12, an input / output unit 13, a communication unit 14, an image database 15, and the like. Each block is electrically connected by a bus line 16.

The control unit 11 includes a CPU (Central Processing Unit) 111 as an arithmetic / control device, a RAM (Random Access Memory) 112 and a ROM (Read Only Memory) 113 as a main storage device.
The ROM 113 stores a program called BIOS (Basic Input Output System) and basic setting data. The RAM 112 is loaded with programs such as an OS and various applications read from the storage unit 12.
The CPU 111 executes various processes in accordance with programs developed in the RAM 112 and controls each block.

The storage unit 12 includes, for example, a hard disk and stores an OS, various application programs (for example, a server program and a web application program), various data, and the like.
The input / output unit 13 constitutes a user interface for use by the administrator of the web server 1.
The communication unit 14 performs processing according to a communication protocol such as TCP / IP, and transmits / receives data to / from the client terminal 2 via the communication network N.

  The image database 15 is referred to when searching for a similar image in response to a search request using a query image, and data relating to images published on the Internet is registered. In the web server 1, for example, a program that collects (crawls) images published on the Internet is always executed, and when an image published on the Internet is updated (increased or decreased), an image database is associated therewith. The contents of 15 are updated.

FIG. 3 is an explanatory diagram showing an example of the data configuration of the image database 15.
As shown in FIG. 3, data (image data) related to the collected image and a feature amount representing a visual feature of the image are registered in the image database 15 for the image ID. Here, the image data includes the location (URL) of the image, tagged text information, thumbnails, and the like. In addition to a feature amount representing a visual feature, a frequency or a histogram obtained from an image can be used as the feature amount.
In the present embodiment, the actual data is analyzed when the image is collected, thereby calculating the color C, the contour S, the pattern, and the pattern T, which are constituent elements of the image, and registering these as feature amounts. When determining the similarity between a plurality of images, the similarity between the images is calculated based on the feature amount of each image. For example, when the similarity is equal to or greater than a threshold value, both are determined to be similar. The A known technique can be used for calculating the feature amount and the similarity.

In the image database 15, an image display table 151 for generating a search result screen for similar image search is temporarily registered. In this image display table 151, information related to display of images (query image and similar image) included in the search result of similar image search is registered.
The image display table 151 is generated, for example, when the similar image search service is used in the client terminal 2, and is discarded when the use of the similar image search service ends (logout).

FIG. 4 is an explanatory diagram showing an example of the data configuration of the image display table 151. As shown in FIG. 4, the image display table 151 includes a display object table (FIG. 4A) for arranging a query image and a similar image (a corresponding thumbnail) at a predetermined position on the search result screen, and a query. It is composed of a link table (FIG. 4B) for connecting images and similar images with link lines.
Although the thumbnails corresponding to the images are arranged on the search result screen, they may be described as arranging the images by simplifying the expression.

In the display object table shown in FIG. 4A, an image ID of an image included in the search result, an image type (query image / similar image), and display coordinates for arranging thumbnails corresponding to the respective images are registered. FIG. 4A shows a case where (x1, y1) is set as the display coordinates of the query image “# 1” and (x2, y2) is set as the display coordinates of the similar image “# 2”. . On the search result screen, thumbnails corresponding to the respective images are displayed according to the display coordinates set in the display object table.
If the image type is “query image” in the display object table, a similar image search using the image has already been executed. That is, by referring to the display object table when executing the similar image search, it is possible to determine whether or not the similar image search using the designated image is necessary. The display object table also functions as a log for accumulating query images selected by the user as search keys. That is, an image whose image type is “query image” indicates that the user has selected it as a query image.

  In the link table shown in FIG. 4B, the image ID of the extracted similar image is registered corresponding to the image ID of the query image. FIG. 4B shows a case where similar images “# 2” and “# 3” are linked to the query image “# 1”. In the search result screen, thumbnails corresponding to linked images (images “# 1” and “# 2” and images “# 1” and “# 3” in FIG. 4B) are connected by a link line. The Rukoto.

The control unit 11 of the web server 1 provides a similar image search service to the client terminal 2 by executing server software. In the similar image search service, when a search request using a query image is made from the client terminal 2, a similar image search is executed in response to the search request. Then, a similar image similar to the query image is extracted and presented to the client terminal 2 as a search result.
In this embodiment, the user interface which comprises the search result screen shown on the client terminal 2 is characteristic. Specifically, when a query image is newly specified from the images displayed on the display unit of the client terminal 2, a search result based on the specified query image is not newly presented. The search results (search results including the current query image as a similar image) are presented in association with each other. Thereby, the user's search intention can be accurately expressed on the search result screen.

Since the hardware configuration of the client terminal 2 shown in FIG. 1 is almost the same as that of the web server 1, the illustration is omitted. The client terminal 2 includes a control unit, a storage unit, an input unit, a display unit, a communication unit, and the like.
The configuration of the control unit of the client terminal 2 is the same as the configuration of the control unit 11 of the web server 1.
The storage unit is configured by, for example, a hard disk, and stores an OS, various application programs (for example, a web browser program), various data, and the like.
The input unit includes, for example, a keyboard and a mouse, and is used when a user inputs information. The input unit inputs a search key when making a search request to the web server 1, for example.
The display unit is configured with, for example, a liquid crystal display or the like, and displays a predetermined screen based on a display control command from the control unit. For example, a GUI (Graphical User Interface) of a web browser is displayed on the display unit.
The communication unit performs processing according to a communication protocol such as TCP / IP, and transmits / receives data to / from the web server 1 via the communication network N.

  The control unit of the client terminal 2 outputs a similar image search service screen provided by the web server 1 via the Internet to the display unit by executing the web browser program. Further, in the similar image search service, when a user instructs a search using a query image, this search request is transmitted to the web server 1. And the search result transmitted from the web server 1 is acquired, and a search result screen is output to a display part.

FIG. 5 is a flowchart illustrating an example of a processing procedure in the web server 1 and the client terminal 2 when the user uses the similar image search service. This process is executed by the control unit 11 of the web server 1 and the control unit of the client terminal 2.
In FIG. 5, for example, the user performs an access operation (for example, input of URL) to the similar image search service on the web browser of the client terminal 2, and the use request of the similar image search service is requested from the web server 1. It shows about the processing procedure after is transmitted.

  In step S101 in FIG. 5, the web server 1 transmits an HTML file of the top page of the similar image search service in response to the use request from the client terminal 2. A control program (for example, JAVA (registered trademark) program) for dynamically generating and providing a web page according to a user operation in the client terminal 2 is embedded in the HTML file transmitted from the web server 1. ing.

In step S201, the client terminal 2 analyzes the HTML file transmitted from the web server 1 and displays the top page of the similar image search service on the web browser.
In the similar image search service of this embodiment, an image group that can be used as a search key (query image) is presented on the top page, and the user is allowed to select a query image from the image group. For example, an image with a high frequency specified as a query image within a predetermined period is presented, or an image registered in the image database 15 is randomly selected and presented. Further, for example, images registered in the image database 15 may be classified in advance, and an image representing each category (for example, an image reflecting the characteristics of the category) may be presented.

  In step S202, it is determined whether a query image as a search key is designated by the user. For example, when a search button is clicked in a state where one image is selected from images presented on the top page or the search result screen, it is determined that a search request using a query image has been made. If it is determined that a search request based on the query image is made by the user, the process proceeds to step S203, and the search request based on the query image is transmitted to the web server 1.

  In step S <b> 102, the web server 1 acquires a query image included in the search request from the client terminal 2. That is, an image designated by the user is accepted as a query image. Then, information about the acquired query image is registered in the image display table 151.

Here, in the case of a newly requested similar image search, the image ID of the acquired query image is registered and “query image” is set as the image type in the display object table of FIG. Also, the image ID of the acquired query image is registered as the query image ID of the link table in FIG.
On the other hand, in the case of a similar image search in which a query image is designated from the similar images presented on the search result screen, that is, when information about the acquired query image has already been registered in the image display table 151, FIG. In the display object table of 4 (a), the image type set corresponding to the image ID of the acquired query image is changed from “similar image” to “query image”. Further, the image ID of the acquired query image is set as the query image ID of the link table in FIG. That is, the image ID registered as the similar image ID in the link table of FIG. 4B is also registered as the query image ID.

In step S103, a similar image similar to the query image is extracted from the images registered in the image database 15. For example, referring to the image database 15, the feature amount of the acquired query image is compared with the feature amount of other registered images to calculate the similarity, and an image whose similarity is equal to or higher than a threshold, or a predetermined higher priority A number of images are extracted as similar images. And the information regarding the extracted similar image is registered into the image display table 151 shown in FIG.
In the present embodiment, among the similar images extracted by the similar image search means, only a predetermined number (for example, six) images having higher similarity are registered as similar images. That is, the number of images presented on the search result screen is predetermined. Thereby, when many similar images are extracted, the search result screen can be prevented from becoming too complicated.

Here, in the case of a newly requested similar image search, the image ID of the extracted similar image is registered and “similar image” is set as the image type in the display object table of FIG. Further, the image ID of the extracted similar image is registered in association with the query image ID as the similar image ID in the link table of FIG.
On the other hand, in the case of a similar image search in which a query image is designated from similar images presented on the search result screen, the image ID of the newly extracted similar image is registered in the display object table of FIG. (Similar images already registered are not newly registered), and “similar image” is set as the image type. Further, the image ID of the extracted similar image is registered in association with the query image ID as the similar image ID in the link table of FIG.

  In step S104, based on the acquired query image and the extracted similar image, a display coordinate serving as an arrangement position of the image (the corresponding thumbnail) on the search result screen is set. At this time, it is desirable to set the display coordinates of the image so that the display area can be used effectively (with as little margin as possible) while ensuring the visibility of the image presented on the search result screen.

In this embodiment, in principle, the display coordinates of a query image are set, and the display coordinates of a similar image similar to the query image are set around the display coordinates.
However, when a new query image is specified from similar images included in the search result, similar images similar to the surrounding image are retained around the new query image while maintaining the connection mode by the link line in the search result. Each display coordinate is set so that can be connected by a link line. When similar images similar to the new query image are included in the past search results, the display coordinates are changed so that the similar image is arranged in the middle of the plurality of query images similar to the similar image. Will be. At this time, it is desirable that similar images are arranged on a line that is the shortest distance between the plurality of query images. However, if the images overlap due to the arrangement with other images, the arrangement position may be shifted to the extent that the overlap is avoided.
In other words, while maintaining the display mode of past search results as much as possible, each image is displayed according to the number of images to be displayed, the size of the display area, and the link relationship between images (complexity of drawn link lines). Coordinates are set.

  In step S105, each image is arranged at the set display coordinates, and the data (HTML file) of the search result screen in which the images having the link relationship are connected by the link line is generated and transmitted to the client terminal 2. .

In step S204, the client terminal 2 analyzes the HTML file on the search result screen and displays the search result on the web browser. The search result screen allows the user to know a similar image similar to the query image.
Furthermore, the similar image presented on the search result screen can be designated as a new query image. In this case, the subsequent processing is executed in step S202.

FIG. 6 is an explanatory diagram illustrating an example of a transition mode of a search result screen related to similar image search.
As shown in FIG. 6A, on the top page of the similar image search service, for example, a plurality of images that can be used as search keys are arranged in a matrix. Note that the arrangement of the images may be a random arrangement. In FIG. 6A, the image A is selected as the query image from the presented image group. As described above, when the image A is designated as the query image, the search result shown in FIG. 6B is presented.

As shown in FIG. 6B, in the search result screen using the query image A, six similar images similar to the query image A are arranged around the query image A, and a link line is formed between the query image A and the similar images. L is connected. That is, the display coordinates of each image are set so that the search result screen is rendered in this way.
In FIG. 6B, the image B is selected as the next query image (new query image) from the similar images presented as the search results. As described above, when the image B is designated as the query image, the search result shown in FIG. 6C is presented.

As shown in FIG. 6C, in the search result screen by the query image B, the connection form by the link line in the search result shown in FIG. Six similar images (including the query image A) are arranged, and the query image B and the similar images are connected by a link line L. That is, the display coordinates of each image are set so that the search result screen is rendered in this way. Specifically, in order to arrange similar images around the query image B, the display position of the query image B is moved upward from the position shown in FIG.
In FIG. 6C, the image C is selected as the next query image (new query image) from the similar images presented as the search results. Thus, when the image C is designated as the query image, the search result shown in FIG. 6D is presented.

As shown in FIG. 6D, on the search result screen by the query image C, the connection form by the link line in the search result shown in FIG. Six similar images (including the query image A and the similar image X) are arranged, and the query image C and the similar images are connected by a link line L.
In addition, since the similar image X similar to the new query image C is included in the past search results (search results based on the query image A), the similarity is between the two query images A and C similar to the similar image X. An image X is arranged, and a similar image X and a new query image C are connected by a link line L.
That is, the display coordinates of each image are set so that the search result screen is rendered in this way. Specifically, in order to arrange a similar image around the query image C and connect the query image C and the similar image X by the link line L, the display position of the query image C is the position shown in FIG. The display position of the similar image X is moved upward from the position shown in FIG.
In FIG. 6D, the image D is selected as the next query image (new query image) from the similar images presented as the search results. Thus, when the image D is designated as the query image, the search result shown in FIG. 6E is presented.

As shown in FIG. 6 (e), the search result screen by the query image D retains the connection mode by the link line in the search result shown in FIG. Six similar images (including the query image C and the similar image Y) are arranged, and the query image D and the similar images are connected by a link line L.
In addition, since the similar image Y similar to the new query image D is included in the past search results (search results based on the query image B), this similarity is intermediate between the two query images B and D similar to the similar image Y. An image Y is arranged, and a similar image Y and a new query image D are connected by a link line L.
That is, the display coordinates of each image are set so that the search result screen is rendered in this way. Specifically, in order to place a similar image around the query image D and connect the query image D and the similar image Y by the link line L, the display position of the query image D is the position shown in FIG. It has been moved upwards. Since the adjustment can be made only with the display area of the query image D, the display position of the similar image Y does not change from the position shown in FIG.
In this way, when the search is repeated, the graph structure as the search result dynamically changes, and an image that meets the user's search intention, that is, a similar image common to a plurality of query images specified by the user, is displayed centering on the query image. Since they gather in the center of the space, the user can efficiently search for a desired image.

  Further, as shown in FIG. 6, the query image is displayed in a shaded manner and the similar image is displayed in white, so that the query image and the similar image are displayed differently on the search result screen. Actually, the query image and the similar image can be distinguished by changing the thickness and color of the image frame. Thereby, since the user's search history (which image is the query image) can be visually grasped, the query image once designated can be prevented from being designated again, and the convenience is improved.

As described above, the web server 1 constitutes an image search device that searches for an image similar to a query image in response to a search request by a query image specified by a user.
Then, for each image, an image database (15) in which a feature amount representing a visual feature of the image is registered, and an image in the image database (15) displayed on the display means (display unit of the client terminal 2). Search request accepting means (control unit 11, step S102 in FIG. 5) for accepting an image designated by the user as a query image, and a similar image similar to the query image among images registered in the image database (15). Similar image search means to extract (control unit 11, step S103 in FIG. 5), arranges similar images around the query image, and displays the search result in which the query image and the similar image are connected and displayed (connected by a link line). The search result presenting means (control unit 11, steps S104 and S105 in FIG. 5) to be presented to the means is provided.
When a new query image is received by the search request receiving means, the search result presenting means (control unit 11, steps S104 and S105 in FIG. 5) holds a linked display of the search results presented in the past on the display means. However, the search result by the new query image is added to the search result and presented on the display means.
When a similar image similar to the new query image is included in the past search results, the similar image is arranged in the middle of a plurality of query images similar to the similar image, and the similar image and the new query image are arranged. Will be displayed concatenated.

As a result, when an image similar to the latest query image is presented, search results based on past query images are included (for example, the connection mode by link lines is maintained), so that the user's search intention is accurately expressed. it can. Further, since the search is repeated, the images that meet the user's search intention are gathered at the center, so that the user can efficiently search for the desired image.
Therefore, the convenience of the user interface when presenting the search result in the similar image search is greatly improved.

By the way, if the image search by the query image is repeated, it is conceivable that the similar images included in the search result increase and the display area is insufficient. In this case, this can be dealt with by deleting the connected display of images that are separated from the latest query image by a predetermined distance or more.
FIG. 7 is an explanatory diagram illustrating another example of the transition mode of the search result screen related to the similar image search.
As shown in FIG. 7A, when similar image searches are repeated a plurality of times and search results based on the latest query image a are added, the six images b to g in the upper left of the screen are separated from the query image a by a predetermined distance R. They will be separated from each other. That is, since it can be determined that these images b to g are less relevant to the latest query image a, these linked displays are deleted.
That is, the search result screen shown in FIG. 7B is presented as a result of a similar image search using the latest query image a. In FIG. 7B, the images b to g are erased from the screen, but these images b to g are displayed in a part of the display area simply by erasing the connected display (link by the link line). You may make it reduce-display collectively.

In this way, when linking display of some images is deleted, the link information recorded in the link table (see FIG. 4B) is not deleted. When a new query image is specified and an image deleted from the screen is included in the similar image of the query image, the link information held in the link table is referred to, and the link related to the image is connected. Reset the display.
For example, when the search result shown in FIG. 7B is that the image i is designated as the query image in the plane, and the similar image of the image i includes the image g from which the linked display with the image h is deleted, FIG. The search result screen shown in (c) will be presented. That is, similar images (including the similar image g) are arranged around the new query image i and displayed in a connected manner, and the connected display with the query image h is displayed again for the similar image g.

As described above, the web server 1 as the image search device deletes the linked display of images that are separated from the latest query image by a predetermined distance (R) or more. Thereby, the display area can be effectively used without impairing search results useful for the user such as an image highly relevant to the latest query image.
In addition, a connection information recording unit (link table) for recording connection information (link information) related to an image from which the connection display has been deleted is provided, and a similar image searched for the query image is recorded in the connection information recording unit. Then, the deleted linked display is displayed again. As a result, even if an image has been deleted from the connection display, the connection display between the images is formed by using the past search results as much as possible, so that the similarity between the images can be efficiently presented on the search result screen. .

  Note that the latest search result can be easily grasped by setting the display position of the latest query image as the center of the search result. Further, when the number of similar images included in the search result increases and the display area becomes insufficient, the display area may be scrolled and presented, or may be displayed in a reduced scale as a whole.

As mentioned above, although the invention made by this inventor was concretely demonstrated based on embodiment, this invention is not limited to the said embodiment, It can change in the range which does not deviate from the summary.
For example, in the above embodiment, the display coordinates of the image are set so as to effectively use the display area while ensuring the visibility of the image presented on the search result screen, but the similarity between the query image and a similar image similar thereto is set. Based on the degree, the length of the connected display that connects them may be set.
Further, for example, when the similar image searched is common to similar images with respect to other query images, the ratio of the lengths of the respective linked displays based on the similarity of the similar images common to each of the plurality of query images. May be set.

Specifically, the similarity between images can be reflected in the length of the link line by using a technique of simulating the images presented on the search result screen as a combination of springs (spring model). The reaction force F by the spring is expressed as follows according to Hooke's law.
F = −k (x−L)
x: length of spring, L: length of spring when no load is applied, k: spring constant spring model, simulation is performed so that F of each spring expressed in this way is balanced. Therefore, if the value representing the similarity between images is S,
F = −k (x−S · L)
Thus, by changing the length of the spring, it can be reflected in the length of the link line. Also,
F = −s · k (x−L)
As described above, the length of the link line can be reflected by changing the spring constant. In this case, a difference in length appears only when an image is connected to a plurality of images by link lines.
Here, the value S representing similarity is a value obtained by normalizing the degree of similarity between images 0 to 1 (similar to 0 as the similarity is similar) or the order (rank) similar to the query image (= (R−1). ) / N, R: rank, N: number of registrations or number of searches).
In this way, by reflecting the similarity between the query image and the similar image in the length of the link line, the search result screen is represented as a feature amount space. Will be able to grasp.

In the embodiment described above, an image group that can be used as a search key is presented on the top page of the similar image search service. However, when a user uploads an image from the client terminal 2, a query image that serves as a search key is displayed. It may be possible to specify. Further, prior to the similar image search, an image search using a keyword as a search key may be performed so that the first query image can be designated from the image group presented on the search result screen.
Even when a new query image is specified in a state where the search result screen is displayed, the query image is not specified by the user from the displayed image group but by the user uploading the image from the client terminal 2. May be specified.

Further, as shown in FIGS. 8A and 8B, once the query image (parent image) Q1 is designated by the user, a similar image (child image) similar to the query image is extracted, and Such similar images (child images) may be used as query images Q2 to Q7, and similar images (grandchild images) similar to these may be extracted, and similar images may be automatically searched and presented step by step.
As a result, a wide range of graphs are presented as search results only when the user designates a query image only once, so that the search can be performed more efficiently.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1 Web server (image search device)
2 Client terminal N Communication network 11 Control unit (search request receiving means, similar image search means, search result presentation means)
12 storage unit 13 input / output unit 14 communication unit 15 image database 16 bus line

Claims (10)

An image search device for searching for an image similar to a query image in response to a search request by a query image specified by a user,
An image database in which image feature values are registered for each image;
Search request accepting means for accepting, as a query image, an image designated by the user from the images in the image database displayed on the display means;
Similar image search means for extracting a similar image similar to the query image from images registered in the image database;
Search result presenting means for arranging the similar image around the query image and presenting a search result in which the query image and the similar image are connected and displayed on the display means,
When a new query image is accepted by the search request accepting unit, the search result presenting unit retains a linked display of the search results presented in the past on the display unit, and adds the new search result to the search result. An image search apparatus characterized in that the search result based on a query image is added and presented on the display means.   The image search apparatus according to claim 1, wherein the search result presenting unit deletes a connected display of images that are separated from the latest query image by a predetermined distance or more. Comprising connection information recording means for recording connection information relating to an image from which the connection display has been erased;
The search result presenting means redisplays the deleted linked display when a similar image searched for the query image is recorded in the linked information recording means. The image search apparatus described.   4. The search result presenting means sets the length of a linked display that connects the query image and a similar image similar to the query image based on the similarity between the query image and the similar image. The image search device according to item.   When the searched similar image is common to similar images with respect to other query images, the search result presenting means is configured to link each of the plurality of query images and the similar similar image based on the similarity. The image search device according to claim 4, wherein a ratio of display lengths is set.   When the similar image similar to the new query image is included in the past search results, the search result presenting unit arranges the similar image between a plurality of query images similar to the similar image, and The image search apparatus according to claim 1, wherein the similar image and a new query image are displayed in a concatenated manner.   The image search device according to any one of claims 1 to 6, wherein the search result presentation unit displays a query image and a similar image included in the search result differently.   The image according to any one of claims 1 to 7, wherein the search result presenting unit arranges a predetermined number of the similar images extracted by the similar image searching unit in descending order of similarity. Search device. For each image, a computer of an image search apparatus having an image database in which image feature amounts are registered responds to a search request by a query image specified by a user and searches for an image similar to the query image. A method,
Receiving an image designated by the user from the images in the image database displayed on the display means as a query image;
Extracting a similar image having a feature amount similar to the feature amount of the query image from images registered in the image database;
Disposing the similar image around the query image and presenting the search unit with a search result in which the query image and the similar image are connected and displayed on the display unit;
When a new query image is received, the search result based on the new query image is added to the search result while the linked display of the search results presented in the past is held on the display means and presented to the display means. And a method for retrieving an image. For each image, an image database in which image feature amounts are registered is provided, and in response to a search request by a query image designated by a user, a computer of an image search apparatus that searches for an image similar to the query image,
Receiving an image designated by the user from the images in the image database displayed on the display means as a query image;
Extracting a similar image having a feature amount similar to the feature amount of the query image from images registered in the image database;
Disposing the similar image around the query image and presenting the search unit with a search result in which the query image and the similar image are connected and displayed on the display unit;
When a new query image is received, the search result based on the new query image is added to the search result while the linked display of the search results presented in the past is held on the display means and presented to the display means. Program for realizing the steps.

Images