JP4949354B2 - Image search apparatus, image search method and program - Google Patents

Description

  The present invention relates to an image search apparatus, an image search method, and a program for searching for images published on the Internet.

2. Description of the Related Art Conventionally, in an image search service for searching images published on the Internet, when a user inputs a keyword (query) at a client terminal, an image assigned with the keyword as a tag is searched. ing. Here, the tag is a keyword indicating the subject attached to the image and the situation at the time of shooting, and is assigned by the image provider. Instead of the tag or in combination with the tag, an image file name or a character string arranged around the image may be used.
When it is desired to narrow down the image group obtained as a search result, a keyword is further added and the search is performed again.

However, in the conventional image search, a search result varies depending on a keyword such as a tag related to an image or a keyword input by the user, and thus a desired image cannot always be accurately searched using the keyword input by the user.
For example, a user with low search skill may not be able to specify an appropriate keyword for searching for a desired image, such as when it is difficult to accurately express a subject included in an image with a keyword.
When searching for products offered (sold) on the Internet, it is easy to specify a product category as a keyword, but it is not easy to specify a keyword to further narrow down the products. For example, it may be difficult to express these as keywords for features such as ties and suits that are based on color, pattern, and shape characteristics.

Therefore, in recent years, an image is input as a search key, and it is similar to the image that is the search key by comparing the feature amount of the image (a numerical expression of image features such as color scheme, texture, and shape). A technique for searching for an image has been proposed (for example, Patent Document 1).
More specifically, in Patent Document 1, an image processing server compares an image transmitted as a search key from a user terminal with a standard image (standard pattern) of a product prepared in advance, and performs pattern matching processing. The product is specified by performing the process, and the desired information is acquired by inquiring the merchant server with the product recognition information (keyword indicating the feature of the product) of the specified product.
Further, in Patent Document 1, when narrowing down the search results, a partial image (such as an automobile emblem or headlight) indicating a partial feature of a product is further acquired from the user terminal, and pattern matching processing is performed on the partial image. It is disclosed.
JP 2006-31200 A

In the technique described in Patent Document 1 described above, when searching for an image (including a partial image) transmitted from a user terminal using a search key, pattern matching processing is performed on this image. Prepare in advance on the processing server side. For this reason, the system provider of the image processing server requires a complicated operation of appropriately taking and processing a standard image.
Even if an image of a product is transmitted from the user terminal to the image processing server, an appropriate search result cannot be obtained unless a standard image corresponding to this image is prepared on the image processing server side.
Further, when the search results are narrowed down by partial images, it is necessary to acquire partial images by taking a picture with a camera at the user terminal, so that it is difficult to quickly narrow down the search results.

  The present invention relates to an image search device, an image search method, and a program for searching for an image published on the Internet, and the user's operability related to image search is facilitated and an image desired by the user It is an object to provide a technology capable of efficiently obtaining the information.

In order to achieve the above object, the invention according to claim 1 is an image search device for searching for an image in response to a search request by a keyword,
Representative partial image generation means for generating a representative partial image corresponding to the index word based on an image associated with each index word;
Representative partial image database registration means for registering the representative partial image generated by the representative partial image generation means in the representative partial image database for each index word;
Classification means for classifying the image used for generating the representative partial image in association with the representative partial image generated based on the image;
In response to the search request, representative partial image presentation means for searching and presenting a representative partial image associated with an index word from the representative partial image database;
Search result presenting means for retrieving and presenting the classified image by the representative partial image selected by the user operation among the representative partial images presented by the representative partial image presenting means, To do.

  According to a second aspect of the present invention, in the image search device according to the first aspect, the representative partial image generation means divides the image associated with each index word into predetermined blocks, respectively. Each partial image is clustered based on the feature amount, and the representative partial image is selected from the partial images belonging to each cluster.

  According to a third aspect of the present invention, in the image search device according to the second aspect, the classifying unit extracts, for each representative partial image, an image that is a division source of the partial image belonging to the cluster in which the representative partial image is selected. It is classified into the following.

  According to a fourth aspect of the present invention, in the image search device according to any one of the first to third aspects, the representative partial image presenting means includes the index word in response to a search request by the index word. The associated image is presented together with the representative partial image.

  According to a fifth aspect of the present invention, in the image search device according to any one of the first to fourth aspects, the representative partial image generating means is distinguished from a background from an image associated with each index word. An object can be detected, and when an object is detected from the image, an image indicating the feature of the object is used as a representative partial image.

  According to a sixth aspect of the present invention, in the image search device according to any one of the first to fifth aspects, the search result presentation means is selected from among images associated with the index word. An image related to the representative partial image is retrieved and presented.

  According to a seventh aspect of the present invention, in the image search apparatus according to any one of the first to sixth aspects, the search result presenting means can select a plurality of representative partial images, and the selected representative portion It is characterized by searching for images related to all images.

According to an eighth aspect of the present invention, in the image search apparatus according to any one of the first to seventh aspects, an acquisition unit that acquires an image published on the Internet and an index word related to the image. Further comprising
The representative partial image generation means generates the representative partial image based on the image acquired by the acquisition means.

The invention according to claim 9 is a method of searching for an image in response to a search request by a keyword,
Generating a representative partial image corresponding to the index word based on an image associated with each index word;
Registering the generated representative partial image in the representative partial image database for each index word;
Classifying the image used for generating the representative partial image in association with the representative partial image generated based on the image;
In response to the search request, searching and presenting the representative partial image associated with the index word from the representative partial image database;
Searching and presenting the classified image by the representative partial image selected by the user operation from the presented representative partial images;
It is characterized by providing.

The invention according to claim 10 is provided in a computer.
Generating a representative partial image corresponding to the index word based on an image associated with each index word;
Registering the generated representative partial image in the representative partial image database for each index word;
Classifying the image used for generating the representative partial image in association with the representative partial image generated based on the image;
In response to the search request, searching and presenting the representative partial image associated with the index word from the representative partial image database;
Searching and presenting the classified image by the representative partial image selected by the user operation from the presented representative partial images;
It is a program for realizing.

According to the present invention, when an image search using a keyword as a search key is performed, a representative partial image related to the keyword can be presented and used as a search key. An image search can be performed. Therefore, the operability of the user related to the image search is facilitated.
Further, since the image search is performed using both the keyword and the representative partial image as search keys, a desired image can be efficiently searched.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a schematic diagram showing a network configuration to which an image search apparatus of the present invention is applied. FIG. 1 shows a case where content (for example, an image search service) is provided from the web server 1 to the client terminal 2 via the communication network N and the content is used at the client terminal 2.

  As shown in FIG. 1, a client terminal 2 is connected to a web server 1 via a communication network N so that data communication can be performed with each other. The communication network N is a telephone line network such as the Internet or a telecommunication carrier, a mobile phone communication network, or the like.

The web server 1 provides the client terminal 2 with an image search service for searching for images published on the Internet using keywords.
The client terminal 2 is, for example, a personal computer 2a or a mobile terminal 2b such as a mobile phone. In the image search service provided by the web server 1, when a keyword is transmitted as a search key from the client terminal 2, an image search is performed based on this keyword.

  That is, in the present embodiment, the web server 1 is connected to a client terminal 2 that accepts a user operation via a communication network N, and is published on the Internet in response to a search request by a keyword from the client terminal 2. It becomes an image search device for searching for an image.

FIG. 2 is a block diagram showing an outline of the hardware configuration of the web server 1.
As shown in FIG. 2, the web server 1 is a general computer including a control unit 11, a storage unit (auxiliary storage) 12, an input / output unit 13, a communication unit 14, an image database 15, a representative partial image database 17, and the like. It is. Each block is electrically connected by a bus line 16.

The control unit 11 includes a CPU (Central Processing Unit) 111 as an arithmetic / control device, a RAM (Random Access Memory) 112 and a ROM (Read Only Memory) 113 as a main storage device.
The ROM 113 stores a program called BIOS (Basic Input Output System) and basic setting data. The RAM 112 is loaded with programs such as an OS and various applications read from the storage unit 12.
The CPU 111 executes various processes in accordance with programs developed in the RAM 112 and controls each block.

The storage unit 12 includes, for example, a hard disk and stores an OS, various application programs (for example, a server program and a web application program), various data, and the like.
The input / output unit 13 constitutes a user interface for use by the administrator of the web server 1.
The communication unit 14 performs processing according to a communication protocol such as TCP / IP, and transmits / receives data to / from the client terminal 2 via the communication network N.

The image database 15 is referred to when searching for an image in response to a keyword search request, and stores an image for each keyword related to the image. Here, the keyword related to the image includes a tag attached to the image, a file name of the image, a character string arranged around the image, and the like. A keyword registered as an index in the image database 15 and a representative image database 17 described later may be referred to as an “index word”.
The image database 15 stores an image for each representative partial image in order to make it possible to narrow down a search result by a keyword by a representative partial image described later.

FIG. 4 is an explanatory diagram showing an example of the data configuration of the image database 15.
As shown in FIG. 4, the image database 15 includes an index part IDX1a, an index part IDX1b, and a database part DB1.

  In the index part IDX1a, an image ID is registered for each index word serving as a search key. In FIG. 4A, for example, image IDs “# 1, # 2,...” Are registered for the index word “sakura”, and image IDs “# 11, # 12,.・ ・ ”Indicates that it is registered.

In the index part IDX1b, an image ID is registered for each representative partial image serving as a search key. In FIG. 4B, for example, the image ID “# 1, # 2,...” Is registered for the representative partial image R1 (for example, an image showing cherry blossom petals), and the representative partial image R2 (for example, empty sky) is registered. This indicates that the image ID “# 1,...” Is registered for the image shown in FIG. Further, the image ID “# 11,...” Is registered for the representative partial image R11 (for example, an image showing a polka dot pattern), and the image ID “#” for the representative partial image R12 (for example, an image showing a stripe pattern). # 12,... ”Is registered.
Here, the representative partial image ** (for example, the representative partial image R1) indicates that a representative partial image whose later-described representative partial image ID is “**”. An image ** (for example, image # 1) described in the database unit DB1 indicates that the image ID is “**”.

  In the database unit DB1, for example, actual image data, a title attached to the image, and a URL of a web page that publishes the image are registered for each image ID. The data included in the database unit DB1 is not limited to this, and the content to be presented to the client terminal 2 as a search result is registered as appropriate. In FIG. 4C, for example, as the content of the image ID “# 1”, actual data (image # 1) of an image in which cherry blossoms are photographed as a subject, a title “landscape with cherry blossoms”, and a URL are registered. It shows that.

  The index part IDX1a shown in FIG. 4 (a) and the database part DB1 shown in FIG. 4 (c) have the same configuration as the database of the conventional web server (search server), and the index part shown in FIG. 4 (b). IDX1b is a structure peculiar to this embodiment. By providing the index part IDX1b, an image search using the representative partial image as a search key is realized.

  In FIG. 2, the representative partial image database 17 is referred to in order to present a representative partial image related to the keyword to the client terminal 2 when a search request by the keyword is made from the client terminal 2. A representative partial image is stored for each keyword (index word).

FIG. 5 is an explanatory diagram showing an example of the data configuration of the representative partial image database 17.
As shown in FIG. 5, the representative partial image database 17 includes an index part IDX2 and a database part DB2.
In the index part IDX2, a representative partial image ID is registered for each index word. In FIG. 5A, for example, the representative partial image ID “R1, R2,...” Is registered for the index word “sakura”, and the representative partial image ID “R11, R12,.・ ・ ”Indicates that it is registered.
In the database unit DB2, an image is registered for each representative partial image ID. In FIG. 5B, for example, the representative partial image R1 (image with the representative partial image ID “R1”) is an image showing cherry blossom petals, the representative partial image R2 is an image showing sky, and the representative partial image R11 is polka dots. In the image indicating the pattern, the representative partial image R12 indicates that the image indicates the stripe pattern.

The CPU 111 of the web server 1 performs processing according to a user operation on the client terminal 2 by executing server software and a web application.
For example, when a search request with a keyword is made by a user operation on the client terminal 2, the image database 15 is referred to and a search result that matches the keyword is presented to the client terminal 2. At this time, the representative partial image database 17 is referred to and the representative partial image related to the keyword related to the search request is presented to the client terminal 2.
Then, when one or more representative partial images are selected from the presented representative partial images at the client terminal 2, the search results are narrowed down using the selected representative partial images as search keys, and the results are narrowed down to the client terminal 2 To present.

FIG. 3 is a block diagram showing an outline of the hardware configuration of the client terminal 2.
As shown in FIG. 3, the client terminal 2 is a general personal computer including a control unit 21, a storage unit 22, an input unit 23, a display unit 24, a communication unit 25, and the like. Each part is electrically connected by a bus line 26.

The configuration of the control unit 21 is the same as the configuration of the control unit 11 of the web server 1.
The storage unit 22 includes, for example, a hard disk, and stores an OS, various application programs (for example, a web browser program), various data, and the like.
The input unit 23 includes, for example, a keyboard and a mouse, and is used when a user inputs information.
The display unit 24 is configured by, for example, a liquid crystal display or the like, and displays a predetermined screen based on a display control command from the control unit 21. For example, a GUI (Graphical User Interface) of a web browser is displayed on the display unit 24.
The communication unit 25 performs processing according to a communication protocol such as TCP / IP, and transmits / receives data to / from the web server 1 via the communication network N.

For example, when the user inputs the URL of the image search service using the input unit 23 on the activated web browser, the CPU 211 of the client terminal 2 requests the web server 1 for a web page specified by the URL. Then, an HTML file (for example, an image search service screen) transmitted from the web server 1 is analyzed, and a web page is displayed on the web browser.
Further, when a search key (query) such as a keyword is input by a user operation on the image search service screen, a search request using the search key is made to the web server 1. When the search result is presented by the web server 1, a search result screen including the search result is displayed.

  FIG. 6 is a flowchart illustrating an example of database registration processing in the web server 1. This database registration process is always executed by the search program. Therefore, when an image published on the Internet is updated (increased or decreased), the contents of the image database 15 and the representative partial image database 17 are updated accordingly.

  In step S101, images published on the Internet are collected (crawler). At this time, index words (for example, a tag attached to the image, a file name of the image, a character string arranged around the image, etc.) related to the image are also collected together with the image.

  In step S102, the collected image is registered in the image database 15 for each index word related to the image (indexing of the image with respect to the keyword). By this processing, data is registered in the index part IDX1a and the database part DB1 shown in FIG.

  In step S103, the image registered for each index word is divided into blocks each having a predetermined size. Hereinafter, an image of each divided block is referred to as a partial image. By this process, a plurality of partial images are generated for one keyword.

Here, as a method of dividing the image, for example, a method of dividing the image into predetermined M × N blocks or a method of dividing the divided blocks so that the size of the divided blocks becomes predetermined m × n pixels. is there. For example, when an image is divided into 10 × 10 blocks, if the size of the image is 640 × 480 pixels, the size of one block is 64 × 48 pixels.
In addition, if the object that is the subject and the background can be clearly distinguished as in the product image, the block to be divided should be set so that a partial image showing the object characteristics (product color and pattern) can be obtained. Also good. In this case, it is desirable to determine the block size according to the size of the object. This can be realized by making it possible to detect an object (product) that is distinguished from the background from the product image.

In step S104, the partial images divided in step S103 are collected for each index word, and the partial images for each index word are classified into clusters based on the feature amount (clustering). That is, partial images having similar feature amounts belong to the same cluster. Here, the feature amount is a numerical representation of image features such as image color scheme, texture, and shape.
For example, as shown in FIG. 7, all partial images are arranged in a feature amount space with the image features (for example, color scheme and texture) as XY axes, and the partial images exceed a predetermined ratio with respect to the total number of partial images. The region from which the set is obtained may be regarded as a cluster.

  In step S105, a representative partial image is selected from the partial images belonging to each cluster for each index word. For example, a partial image located at the center of the cluster in the feature amount space shown in FIG. 7, that is, a partial image that most accurately represents the cluster feature is selected as the representative partial image.

  In step S106, one or more selected representative partial images are registered in association with each index word in the representative partial image database 17 (indexing of the representative partial image with respect to the index word). Further, in the image database 15, for each selected representative partial image, an image that is a source of the partial image classified into the cluster to which the representative partial image belongs is registered (indexing of the image with respect to the representative partial image).

By the database registration process shown in FIG. 6, various data are registered in the image database 15 shown in FIG. 4 and the representative partial image database shown in FIG.
That is, when the image # 1 with the tag “sakura” is collected in step S101, as shown in FIG. 4, the actual image data (image image) corresponding to the image ID “# 1” in the database unit DB1. # 1) The title and URL are registered, and the image ID “# 1” corresponding to the index word “sakura” is registered in the index part IDX1a (step S102).

Further, when the representative partial image R1 is selected for a cluster (a cluster indicating the characteristics of the cherry blossom petals) consisting of a set of partial images showing the cherry petals in steps S103 to S105, as shown in FIG. The actual data (representative partial image R1) of the representative partial image is registered in the database portion DB2 corresponding to the representative partial image ID “R1”, and the representative partial image ID corresponding to the index word “sakura” is registered in the index portion IDX2. “R1” is registered (step S106).
Further, since the images # 1 and # 2 are the source of the partial image classified into the cluster to which the representative partial image R1 belongs (cluster indicating the characteristics of the cherry blossom petals), the representative image is represented in the index part IDX1b of the image database 15. Image IDs “# 1, # 2” are registered corresponding to the partial image R1.

8 and 9 are explanatory diagrams visually showing an algorithm for selecting a representative partial image. 8 and 9, for the sake of simplicity, when an image is divided into 4 × 3 blocks, partial images having similar feature amounts are schematically shown with the same pattern.
In FIG. 8, an image A is an image to which two tags “mountain” and “sea” are assigned. It is composed of elements corresponding to the sea).
The image B is an image to which two tags “mountain” and “forest” are assigned. The pattern P1, the pattern P4 (element corresponding to the tree trunk), the pattern P5 (element corresponding to the leaf), and the pattern P6. It shall be composed of (elements corresponding to soil).
Therefore, the image A is registered for the index word “sea”, the image B is registered for the index word “forest”, and the images A and B are registered for the index word “mountain”.

As shown in FIG. 9, the image A registered for the index word “sea” is divided into 4 × 3 blocks by the process shown in step S <b> 103 of FIG. 6, and partial images A <b> 1 to A <b> 12 are generated. .
Next, the partial images A2 to A4, A7, A8, and A12 corresponding to the mountains are classified into cluster 1 and the partial images A1, A6, and A11 corresponding to the sandy beach are classified into cluster 2 by the process shown in step S104 of FIG. Then, partial images A5, A9, and A10 corresponding to the sea are classified into cluster 3. In FIG. 9, it is assumed that the clustered partial images are arranged based on the feature amount. That is, with the feature amount of the partial image located at the center of the array as a reference, the difference in the feature amount increases as the image is located at the end of the array.
6, the partial image A4 positioned at the center of the cluster 1 is selected as the representative partial image indicating the mountain, and the partial image A6 positioned at the center of the cluster 2 is the representative partial image indicating the sandy beach. The partial image A9 located at the center of the cluster 3 is selected as a representative partial image showing the sea.

Similarly, the image B registered for the index word “forest” is divided into 4 × 3 blocks by the process shown in step S103 of FIG. 6, and partial images B1 to B12 are generated.
Next, by the process shown in step S104 of FIG. 6, partial images B1, B5, and B9 corresponding to the trunk of the tree are classified into cluster 4, partial images B2, B3 corresponding to the mountains are classified into cluster 1, and Corresponding partial images B6, B10, and B11 are classified as cluster 5, and partial images B4, B7, B8, and B12 corresponding to soil are classified as cluster 6.
6, the partial image B5 positioned at the center of the cluster 4 is selected as the representative partial image indicating the trunk of the tree, and the partial image B2 positioned at the center of the cluster 1 is the representative indicating the mountain. The partial image B10 selected as the partial image, the partial image B10 positioned at the center of the cluster 5 is selected as the representative partial image indicating the leaves of the tree, and the partial image B7 positioned at the center of the cluster 6 is selected as the representative partial image indicating the soil.

  Further, since the images A and B are registered for the index word “mountain”, this is a combination of the processes performed for the index words “sea” and “forest”. In this case, the partial images belonging to the cluster 1 indicating the feature of the mountain are the partial images A2 to A4, A7, A8, A12 and the partial images B2 and B3, so the central partial image A8 is selected as the representative partial image. Will be. For the other clusters 2 to 6, the same representative partial image as that selected for the index word “sea” or “forest” is obtained.

As described above, the web server 1 as the image search device generates representative partial images (representative partial images) corresponding to the index words based on the images associated with the index words. (Steps S103 to S105 in FIG. 6), representative partial image database registration means (Step S106 in FIG. 6) for registering the generated representative partial image in the representative partial image database 17 for each index word, and generation of the representative partial image Classification means (step S106 in FIG. 6) for classifying the image used in the above in association with the representative partial image generated based on the image.
Furthermore, an acquisition unit (step S101 in FIG. 6) for acquiring an image published on the Internet and an index word related to the image is provided.

Specifically, the representative partial image generating means divides the image associated with each index word into predetermined blocks (step S103 in FIG. 6), and uses the divided individual partial images as their feature amounts. Based on the clustering (step S104 in FIG. 6), a representative partial image is selected from the partial images belonging to each cluster (step S105 in FIG. 6).
Further, the classifying unit classifies the image that is a division source of the partial image belonging to the cluster for which the representative partial image is selected for each representative partial image and registers the image in the image database 15.
As a result, the web server 1 automatically generates a representative partial image based on the collected image without requiring a complicated operation of creating a representative partial image by appropriately capturing and processing the system provider. Is done.

  Further, the web server 1 includes image database registration means (step S102 in FIG. 6) for registering the acquired image in the image database 15 for each index word related to the image. Thereby, it is possible to easily realize an image search using a keyword as a search key and an image search using a representative partial image as a search key.

Further, the representative partial image generation means is configured to be able to detect an object distinguished from the background from the image registered for each index word, and when an object is detected from the image, the feature (color or pattern) of this object The image showing is a representative partial image.
As a result, for example, tie, suit, etc., for which color, pattern, and shape features are the criteria for selecting a product, even if it is difficult to express these as keywords, it is more efficient for the representative partial image. Images can be searched.

FIG. 10 is a flowchart illustrating a processing procedure in the web server 1 and the client terminal 2 when performing an image search in the client terminal 2.
The processing procedure illustrated in FIG. 10 is started, for example, when a web browser is activated in the client terminal 2 and an image search service provided by the web server 1 is accessed.
In the following description, it is assumed that the image database 15 of the web server 1 has the data configuration shown in FIG. 4, and the representative partial image database 17 has the data configuration shown in FIG.

  In step S201, the HTML file of the image search service screen is transmitted from the web server 1. A control program (for example, a JAVA (registered trademark) program) for transmitting a keyword or a representative partial image as a search key based on a user operation is embedded in the HTML file and an HTML file on a search result screen described later. .

In step S202, the web browser of the client terminal 2 analyzes the HTML file of the image search service screen, and the image search service screen is displayed on the display unit 24. The user can obtain information (including an image, a title, a URL, and the like) regarding a desired image by inputting a keyword in a search window arranged in the image search service screen and instructing a search.
For example, if you want to search for an image with “sakura” as the subject, you can enter “sakura” in the search window and give a search instruction, or you want to search for an image with “tie” as the subject. In such a case (such as when searching for a product in an online shop), a search instruction may be given by entering “tie” in the search window.

In step S203, when a keyword is input by a user operation and a search request is instructed on the image search service screen, a search request using this keyword as a search key is transmitted to the web server 1.
For example, when the user inputs the keyword “sakura” and performs a search instruction, a search request is made using the keyword “sakura” as a search key. When the keyword “tie” is input and a search instruction is issued, the keyword “tie” is entered. A search request is made using as a search key.

In step S <b> 204, the web server 1 refers to the image database 15, and compares the keyword used as a search key with an index word registered in the image database 15, thereby executing an image search using the keyword.
For example, in the case of a search request with the keyword “cherry blossom”, image # 1 (scenery with cherry blossoms), image # 2 (cherry blossom petals),... Are specified as images that match the search key. In the case of a search request using the keyword “tie”, image # 11 (polka dot tie), image # 12 (stripe tie),... Are specified as images that match the search key.

In step S205, the representative partial image database 17 is referred to, and the keyword used as the search key is compared with the index word registered in the representative partial image database 17, so that the representative partial image corresponding to the keyword (index word) is obtained. Extracted.
For example, in the case of a search request using the keyword “cherry blossom”, the representative partial image R1 (cherry blossom petals), the representative partial image R2 (empty),... Are extracted. In the case of a search request using the keyword “tie”, the representative partial image R11 (polka dot pattern), the representative partial image R12 (striped pattern),... Are extracted.

In step S <b> 206, information related to an image that matches the keyword set as the search key is transmitted to the web server 1 as a search result. At this time, the representative partial image extracted in step S205 is presented together with the search result. That is, an HTML file of a search result screen including a search result based on a keyword set as a search key and a representative partial image extracted based on the keyword is generated and transmitted to the client terminal 2.
For example, in the case of a search request using the keyword “cherry blossom”, the search result screen includes a search result including images # 1, # 2,... And information attached thereto (title, URL, etc.) and representative partial image R1, A representative partial image group consisting of R2,. In the case of a search request using the keyword “tie”, the search result screen includes a search result including images # 11, # 12,... And information (title, URL, etc.) associated therewith, and representative partial images R11, A representative partial image group consisting of R12,... Is included.

At this time, when presenting a plurality of representative partial images, it is desirable to arrange and present the partial images classified into clusters to which the representative partial images belong. For example, when the partial images are clustered as shown in FIG. 9, the representative partial image for the index word “mountain” has the largest number of partial images classified into the cluster 1. Make sure they are arranged first.
Or you may make it arrange | position and present in order with many images which become the origin of the partial image classified into the cluster to which a representative partial image belongs. For example, when the partial images are clustered as shown in FIG. 8, the representative partial images for the index word “mountain” are the two images A and B that are the basis of the partial images classified as cluster 1. Since the original image of the partial images classified into other clusters is either the image A or the image B, the representative partial image A8 of the cluster 1 is arranged first.
In this way, the convenience of the GUI can be improved by presenting the representative partial images in the order in which the user is likely to be used when the search results are narrowed down by the representative partial images.

In step S207, the HTML file of the search result screen transmitted from the web server 1 is analyzed in the client terminal 2, and the search result screen is displayed on the display unit 24.
The user can narrow down the search result by the representative partial image by selecting one or a plurality of representative partial images and instructing the search on the search result screen on which the representative partial image is presented.
For example, in the search result screen using the keyword “sakura”, when it is desired to narrow down to images including “sakura petals” as a subject, the representative partial image R1 may be selected and a search instruction may be given. If it is desired to narrow down to images included (such as landscape images with cherry blossoms), the representative partial image R2 may be selected and a search instruction may be issued. On the search result screen using the keyword “tie”, if it is desired to narrow down to images whose tie pattern is “polka dot pattern”, the representative partial image R11 may be selected and a search instruction may be given. If it is desired to narrow down to the “striped” image, the representative partial image R12 may be selected and a search instruction may be issued.

In step S208, when one or a plurality of representative partial images are selected by a user operation and a search instruction is issued on the search result screen on which the representative partial images are presented, a search request using the selected representative partial images as a search key. Is transmitted to the web server 1.
For example, when a representative partial image R2 is selected and a search instruction is issued on the search result screen by the keyword “sakura”, a search request is made using the representative partial image R2 as a search key. On the search result screen using the keyword “tie”, when the representative partial image R11 is selected and a search instruction is issued by the user, a search request is made using the representative partial image R11 as a search key.

In step S209, the web server 1 refers to the image database 15 to execute an image search using the representative partial image serving as the search key.
For example, in the case of a search request by the representative partial image R2 on the search result screen with the keyword “sakura”, the image # 1 (scenery with cherry blossoms),... Is specified as an image that matches the search key. That is, the image # 2 is excluded because it does not have a partial image classified as a cluster (cluster indicating an empty feature) to which the representative partial image R2 belongs.
In the case of a search request by the representative partial image R11 on the search result screen with the keyword “tie”, the image # 11 (polka dot tie),... Is specified as an image that matches the search key. That is, image # 12 is excluded because it does not have a partial image classified as a cluster to which the representative partial image R11 belongs (cluster indicating the characteristics of the polka dot pattern).

In step S <b> 210, an HTML file of a search result screen (a search result screen) including a search result narrowed down by the representative partial image serving as the search key is generated and transmitted to the client terminal 2.
For example, in the case of a search request by the representative partial image R2 on the search result screen with the keyword “sakura”, the search result screen includes a search result including the images # 1,... And information (title, URL, etc.) attached thereto. Will be included. In the case of a search request by the representative partial image R11 on the search result screen with the keyword “tie”, the search result screen includes a search result including the image # 11,... And information (title, URL, etc.) attached thereto. Will be included.

  In step S <b> 211, the HTML file of the search result screen transmitted from the web server 1 is analyzed in the client terminal 2, and the search result screen (narrowing result screen) is displayed on the display unit 24.

FIG. 11 is an explanatory diagram illustrating an example of a search result screen using the keyword “suit”.
In the search result screen D shown in FIG. 11, a search window SW for a user to input a keyword and a search button SB for instructing search are arranged at the top of the window. These are also arranged on the initial image search service screen. The keyword (for example, suit) input on the image search service screen is also displayed in the search window SW as it is on the search result screen D.

As shown in FIG. 11, for example, when a search request using the keyword “suit” is made, the search result SR including an image matching the keyword “suit” and information associated therewith, and the keyword “suit” are extracted. The representative partial image RP is presented on the search result screen D.
The search result SR and the representative partial image RP displayed in FIG. 11 are a part of the result obtained by the search request using the keyword “suit”, and the scroll bar of the web browser or the page transition operation is operated. By doing so, you can browse all the results.

In FIG. 11, a representative partial image indicating the fabric (pattern) of the suit is presented as the representative partial image RP corresponding to the keyword “suit”. Therefore, the user can obtain a search result narrowed down to an image in which the suit of the selected fabric is a subject by selecting a representative partial image of the desired fabric.
For example, when the user selects a representative partial image RP1 with vertical stripes and performs a search instruction, an image (an image not displayed on the window shown in FIG. 11) in which a suit with vertical stripes is the subject as in the image SR1. Will be presented as a narrowing result.
Thus, when searching for a product characterized by a color or a pattern, the user can visually specify a search key (representative partial image), which is very convenient.

Here, as a technique for realizing the search instruction by the representative partial image, for example, as a button for instructing each representative partial image constituting the representative partial image RP to perform a search using the representative partial image as a search key. What is necessary is just to add a function. That is, when the user moves the mouse cursor to one representative partial image and performs a click operation, the representative partial image is selected and a search instruction is issued.
Further, a radio button is provided adjacent to each representative partial image, and a search instruction using the representative partial image is realized by performing a click operation on the search button SB in a state where the representative partial image is selected by the radio button. It may be. In this case, it is possible to perform aperture search using a plurality of representative partial images. When a search instruction using a plurality of representative partial images is performed, an image search is performed by a preset AND search or OR search. In the case of AND search, images related to all selected representative partial images are searched.

  As described above, the web server 1 as the image search device responds to a search request by a keyword from the client terminal 2 and presents a representative partial image presenting a representative partial image associated with this keyword (index word). Means (steps S205 and S206 in FIG. 10) and search result presenting means (FIG. 10) for presenting, as a search result, an image related to the representative partial image selected by the user operation in the client terminal 2 among the presented representative partial images. 10 steps S209, S210).

  According to the web server 1, when an image search using a keyword as a search key is performed, a representative partial image related to the keyword can be presented and used as a search key. An image search can be performed. Therefore, the operability of the user related to the image search is facilitated.

Further, the representative partial image presenting means presents an image associated with the keyword (index word) together with the representative partial image in response to the search request by the keyword from the client terminal 2 (steps S204 and S206 in FIG. 9). ).
As a result, the result of the image search based on the keyword can be obtained, so that it is possible to determine whether to use the representative partial image as a search key according to the result, and the degree of freedom related to the image search is increased.

Further, the search result presenting means retrieves (extracts) and presents an image related to the selected representative partial image from images associated with the index word (images retrieved by keywords).
As a result, an image search is performed using both the keyword and the representative partial image corresponding to the keyword as a search key, so that a desired image can be efficiently searched.

As mentioned above, although the invention made by this inventor was concretely demonstrated based on embodiment, this invention is not limited to the said embodiment, It can change in the range which does not deviate from the summary.
For example, in the above embodiment, the representative partial image is presented together with the search result by the keyword in response to the search request by the keyword from the client terminal 2, but only the representative partial image may be presented. .
Thereby, since the display area for presenting the representative partial image can be enlarged in the client terminal 2, the selection operation when using the representative partial image as a search key is facilitated. This is effective when it is assumed that an image search is performed using a representative partial image, that is, when a search result based only on a keyword is not required.

In the above embodiment, the representative partial image and the image are registered in the image database 15 in association with each other to realize the image search by the representative partial image. However, the similar image search is performed based on the feature amount of the representative partial image. You may make it perform. That is, an image related to the selected representative partial image may be searched based on the similarity between the images.
In this case, the index part IDX1b in the image database 15 is not necessary. Instead, the feature quantity of each image is stored in the database part DB1 of the image database 15, and the feature quantity comparison process can be executed.

The present invention relates to an image search apparatus that searches for an image in response to a search request using a keyword, but also when searching for an image in response to a search request using an image as a search key (so-called similar image search). May be applicable.
In this case, a search result is presented by a similar image search using the image as a search key, and a representative partial image similar to the image used as the search key is presented.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

It is the schematic shown about the network structure of the goods search system which concerns on this embodiment. It is a block diagram which shows the outline of the hardware constitutions of a web server. It is a block diagram which shows the outline of the hardware constitutions of a client terminal. It is explanatory drawing which shows an example of a data structure of an image database. It is explanatory drawing which shows an example of a data structure of a representative partial image database. It is a flowchart shown about an example of the database registration process in a web server. It is explanatory drawing which shows the feature-value space for clustering the partial image of the image linked | related with the index word "sakura". It is explanatory drawing which shows the algorithm at the time of selecting a representative partial image visually. It is explanatory drawing which shows the algorithm at the time of selecting a representative partial image visually. It is a flowchart shown about the process sequence in a web server and a client terminal when performing an image search in a client terminal. It is explanatory drawing shown about an example of the search result screen by keyword "suit".

Explanation of symbols

1 Web server (image search device)
2 Client terminal N Communication network

Claims (10)

An image search device for searching for an image in response to a search request by a keyword,
Representative partial image generation means for generating a representative partial image corresponding to the index word based on an image associated with each index word;
Representative partial image database registration means for registering the representative partial image generated by the representative partial image generation means in the representative partial image database for each index word;
Classification means for classifying the image used for generating the representative partial image in association with the representative partial image generated based on the image;
In response to the search request, representative partial image presentation means for searching and presenting a representative partial image associated with an index word from the representative partial image database;
Search result presenting means for retrieving and presenting the classified image by the representative partial image selected by the user operation among the representative partial images presented by the representative partial image presenting means;
An image search apparatus comprising:   The representative partial image generation means divides an image associated with each index word into predetermined blocks, clusters the divided partial images based on their feature values, and generates a portion belonging to each cluster. The image search apparatus according to claim 1, wherein the representative partial image is selected from images.   The image search apparatus according to claim 2, wherein the classifying unit classifies the image that is a division source of the partial image belonging to the cluster for which the representative partial image is selected for each representative partial image.   4. The representative partial image presenting means presents an image associated with the index word together with the representative partial image in response to a search request by the index word. The image search device according to item.   The representative partial image generation means is configured to be able to detect an object distinguished from the background from an image associated with each index word, and when an object is detected from the image, an image indicating a feature of the object is displayed. The image search device according to claim 1, wherein the image search device is a representative partial image.   6. The search result presenting means retrieves and presents an image related to the selected representative partial image from images associated with the index word. The image search device according to one item.   7. The search result presentation unit is capable of selecting a plurality of representative partial images, and searches for images related to all the selected representative partial images. 8. Image search device. The image processing apparatus further includes an acquisition unit that acquires an image published on the Internet and an index word related to the image,
The image search device according to claim 1, wherein the representative partial image generation unit generates the representative partial image based on the image acquired by the acquisition unit. A method of searching for an image in response to a keyword search request,
Generating a representative partial image corresponding to the index word based on an image associated with each index word;
Registering the generated representative partial image in the representative partial image database for each index word;
Classifying the image used for generating the representative partial image in association with the representative partial image generated based on the image;
In response to the search request, searching and presenting the representative partial image associated with the index word from the representative partial image database;
Searching and presenting the classified image by the representative partial image selected by the user operation from the presented representative partial images;
An image search method comprising: On the computer,
Generating a representative partial image corresponding to the index word based on an image associated with each index word;
Registering the generated representative partial image in the representative partial image database for each index word;
Classifying the image used for generating the representative partial image in association with the representative partial image generated based on the image;
In response to the search request, searching and presenting the representative partial image associated with the index word from the representative partial image database;
Searching and presenting the classified image by the representative partial image selected by the user operation from the presented representative partial images;
A program to realize

Images