JP2010218247A - Server, terminal device, program, information storage medium and image retrieval method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a server, a terminal device or the like, which make effective use of vagueness of image retrieval. <P>SOLUTION: A server includes: a reception processing part 204 for performing processing to receive transmission data about a retrieval object image through a network 10; an image recognition part 206 for performing the image recognition processing of the received retrieval object image, and for retrieving a plurality of similar images similar to retrieval object images from among registered images registered in an image database 220; a tag processing part 210 for acquiring tags associated with the respective similar images in the plurality of retrieved similar images; and a data creation part 214 for creating similarity retrieval result data including the plurality of tags acquired from the plurality of similar images as retrieval result tags. The tag processing part 210 acquires other tags which are the tags associated with the respective similar images in the plurality of similar images and other than the tag for a first re-retrieval key, based on first response data concerning the tag for the first re-retrieval key selected from the retrieval result tags. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a server, a terminal device, a program, an information storage medium, an image search method, and the like.

  2. Description of the Related Art Conventionally, there has been known an image search system that searches for products or the like taken by a camera using image recognition processing. For example, Patent Literature 1 discloses a system for searching for a store where a product can be purchased by photographing a product desired by a user using a mobile phone with a camera and transmitting the captured image via the Internet. Has been.

  However, in the image recognition processing, there is a problem that it is difficult to accurately specify the corresponding product from the captured image. For this reason, in Patent Document 1, in order to improve the image recognition rate, various devices such as adjustment at the time of shooting are performed.

  As described above, in the image search system using the image recognition processing so far, the focus is on how to accurately identify the photographed subject, and the ambiguity of the image search is reversely used. There was no such idea.

JP 2003-122757 A

  According to some aspects of the present invention, it is possible to provide a server, a terminal device, a program, an information storage medium, an image search method, and the like that effectively use the ambiguity of image search.

One aspect of the present invention is a server that performs a search process for a similar image of a search target image,
A reception processing unit for performing processing for receiving transmission data about the search target image via a network;
An image recognition unit that performs image recognition processing of the received search target image and searches for a plurality of similar images similar to the search target image from registered images registered in an image database;
A tag processing unit for acquiring a tag associated with each similar image of the plurality of similar images searched;
A data creation unit for creating similar search result data including a plurality of tags acquired from the plurality of similar images as search result tags;
A transmission processing unit that performs processing of transmitting the similar search result data via the network,
The reception processing unit
Performing a process of receiving first reply data about the tag for the first re-search key selected from the search result tags via the network;
The tag processing unit
Based on the first reply data, obtain other tags associated with each similar image of a plurality of similar images, excluding the tag for the first re-search key,
The data creation unit
A first re-search result data including a plurality of other tags acquired from the plurality of similar images based on the first reply data as a first re-search result tag is generated. To do.

  According to one aspect of the present invention, when transmission data for a search target image is received, an image recognition process is performed to search for a plurality of similar images similar to the search target image. Then, similar search result data including a plurality of tags (search result tags) acquired from a plurality of similar images is created. Further, when the first reply data for the tag for the first re-search key selected from the search result tags is received, it is another tag associated with each similar image of the plurality of similar images. First re-search result data including other tags (first re-search result tag) excluding the tag for one re-search key is created. Therefore, it is possible to present a plurality of tags (search result tags) acquired from a plurality of similar images and the first re-search result tag, so that it is possible to realize information presentation that effectively utilizes the ambiguity of image search.

In one aspect of the present invention, the tag processing unit includes:
The plurality of other tags may be acquired from a plurality of similar images including the tag for the first re-search key among the plurality of similar images searched.

  In this way, a plurality of tags (search result tags) acquired from a plurality of similar images are narrowed down to other tags (first re-search result tags) using the first re-search key tag. be able to.

In one aspect of the present invention, the transmission processing unit includes:
A process of transmitting the first re-search result data via the network may be performed,
The reception processing unit
A process of receiving second reply data about the tag for the second re-search key selected from the first re-search result tags via the network may be performed,
The tag processing unit
Based on the second reply data, another tag associated with each similar image of a plurality of similar images, the tag for the first re-search key and the tag for the second re-search key You may get other tags except
The data creation unit
Second re-search result data including a plurality of other tags acquired from the plurality of similar images based on the second reply data as a second re-search result tag may be created.

  In this way, when the second reply data for the tag for the second re-search key selected from the first re-search result tags is received, it is associated with each similar image of the plurality of similar images. Second re-search result data including other tags (second re-search result tag) other than the first re-search key tag and the second re-search key tag. Created. Therefore, since the second re-search result tag can be presented, it is possible to realize information presentation that effectively utilizes the ambiguity of the image search.

In one aspect of the present invention, the transmission processing unit includes:
A process of transmitting the first re-search result data via the network may be performed,
The reception processing unit
A process of receiving second reply data about the tag for the second re-search key selected from the first re-search result tags via the network may be performed,
The tag processing unit
Based on the second reply data, another tag associated with each similar image of a plurality of similar images, other than the tag for the second re-search key, may be acquired,
The data creation unit
Second re-search result data including a plurality of other tags acquired from the plurality of similar images based on the second reply data as a second re-search result tag may be created.

  In this way, when the second reply data for the tag for the second re-search key selected from the first re-search result tags is received, it is associated with each similar image of the plurality of similar images. Second re-search result data including other tags (second re-search result tags) other than the tag for the second re-search key is created. Therefore, since the second re-search result tag can be presented, it is possible to realize information presentation that effectively utilizes the ambiguity of the image search.

In one aspect of the present invention, the tag processing unit includes:
The plurality of tags acquired from the plurality of similar images may be sorted,
The data creation unit
Similar search result data including the plurality of sorted tags as the search result tags may be created.

  In this way, similar search result data sorted in an order based on a plurality of tags acquired from a plurality of similar images is created, so that a highly convenient image search system can be realized.

In one aspect of the present invention, the tag processing unit includes:
The number of appearances of each of the plurality of tags acquired from the plurality of similar images may be aggregated, and the plurality of tags may be sorted based on the aggregation result.

  In this way, since a plurality of tags are sorted according to the result of counting the number of appearances of tags, the order of a plurality of tags acquired from a plurality of similar images is determined according to the number of appearances of tags. It becomes possible to convert to an order.

In one aspect of the present invention, the tag processing unit includes:
A weighting coefficient may be obtained based on the similarity of the plurality of similar images with respect to the search target image, and tag aggregation processing may be performed based on the obtained weighting coefficient.

  In this way, tag totaling processing that reflects the similarity can be realized.

In one aspect of the present invention, the data creation unit includes:
Similar search result data further including the plurality of similar images may be created.

  In this way, since a plurality of similar images similar to the search target image can be presented, it is possible to realize information presentation that effectively uses the ambiguity of the image search.

In one aspect of the present invention, the data creation unit includes:
The first mode or the second mode is set so that the similar search result data includes the plurality of similar images in the first mode and includes the plurality of tags as the search result tags in the second mode. Switchable similarity search result data may be created.

  In this way, it is possible to switch between the presentation of similar images or the presentation of tags, so that it is possible to realize information presentation in accordance with the user's will.

In one aspect of the present invention, the data creation unit includes:
First hub screen information for presenting a plurality of link destinations corresponding to the first similar image is associated with the first similar image of the plurality of similar images, and the plurality of similar images For the second similar image of the images, similar search result data associated with second hub screen information for presenting a plurality of link destinations corresponding to the second similar image is created. Also good.

  In this way, the first hub image information for presenting the corresponding link destination is associated with the first similar image, and the corresponding link is associated with the second similar image. Second hub screen information for presenting the destination is associated. Accordingly, since it is possible to display a hub screen in which different link destinations are presented for each similar image of a plurality of similar images of the search target image, it is possible to realize information presentation that effectively uses the ambiguity of the image search.

In one aspect of the present invention, the data creation unit includes:
Similar search result data further including a plurality of related tags acquired from a related database based on the plurality of tags acquired from the plurality of similar images may be created.

  In this way, similar search result data including a plurality of related tags acquired from a plurality of tags acquired from a plurality of similar images via the related tag database is created. Accordingly, since related tags can be presented, information presentation that effectively utilizes the ambiguity of image search can be realized.

According to another aspect of the present invention, an image acquisition unit that acquires a search target image;
A transmission processing unit that performs processing of transmitting transmission data about the acquired search target image to a server via a network;
A reception processing unit that performs processing for receiving similar search result data of the search target image via the network;
A display control unit that performs display control of the display unit based on the received similar search result data,
The display control unit
Performing a control to display a similar search result screen on which a plurality of tags acquired from a plurality of similar images similar to the search target image are displayed as search result tags on the display unit;
The transmission processing unit
Performing a process of transmitting reply data about the tag for the re-search key selected from the search result tags via the network;
The receiving unit performs processing for receiving re-search result data created based on the reply data via the network,
The display control unit
Based on the received re-search result data, a plurality of other tags acquired from a plurality of similar images, excluding the tag for the re-search key, are displayed as re-search result tags. The present invention relates to a terminal device that controls to display a re-search result screen on the display unit. Another aspect of the present invention relates to a program that causes a computer to function as each of the above-described units, or a computer-readable information storage medium that stores the program.

  According to another aspect of the present invention, when transmission data about a search target image is transmitted to a server and similar search result data of the search target image is received, a plurality of images acquired from a plurality of similar images similar to the search target image Tag (search result tag) will be displayed. In addition, when the reply data about the tag for the re-search key selected from the search result tags is transmitted via the network and the re-search result data created based on the reply data is received, a plurality of similar images are received. A plurality of other tags (re-search result tags) other than the tag for the re-search key that are acquired are displayed. In this way, it is possible to realize information presentation that effectively utilizes the ambiguity of image search.

In another aspect of the invention, the display control unit includes:
You may perform control which displays the similar search result screen on which the said some similar image is further displayed on the said display part.

  In this way, since a plurality of similar images similar to the search target image can be presented, it is possible to realize information presentation that effectively uses the ambiguity of the image search.

In another aspect of the invention, the display control unit includes:
Control for displaying on the display unit a similar search result screen capable of switching between a first mode in which the plurality of similar images are displayed or a second mode in which the plurality of tags are displayed as the search result tags. May be performed.

  In this way, it is possible to switch between the presentation of similar images or the presentation of tags, so that it is possible to realize information presentation in accordance with the user's will.

In another aspect of the invention, the display control unit includes:
You may perform control which displays the similar search result screen in which the said some tag was sorted according to the appearance number of each tag on the said display part.

  In this way, since a plurality of tags are sorted according to the result of counting the number of appearances of tags, the order of a plurality of tags acquired from a plurality of similar images is determined according to the number of appearances of tags. It becomes possible to convert to an order.

In another aspect of the invention, the display control unit includes:
You may perform control which displays the similar search result screen on which the some related tag acquired from the related tag database based on the some tag acquired from the some similar image further is displayed on the said display part.

  In this way, many types of related tags can be presented.

Another aspect of the present invention is an image search method for performing a search process for similar images of a search target image,
Receiving transmission data about the search target image via a network;
The image recognition process of the received search target image is performed, and a plurality of similar images similar to the search target image are searched from registered images registered in an image database,
Obtaining a tag associated with each similar image of the plurality of similar images searched;
Creating similar search result data including a plurality of tags acquired from the plurality of similar images as search result tags;
Sending the similar search result data via the network;
Based on a tag for a re-search key selected from among the search result tags, other tags associated with each similar image of a plurality of similar images, excluding the tag for the first re-search key Get other tags,
The present invention relates to an image search method characterized by creating re-search result data including a plurality of other tags acquired from the plurality of similar images as re-search result tags based on the tag for the re-search key.

In another aspect of the present invention, the similar search result data is received via the network,
Based on the received similar search result data, control to display a similar search result screen on which the search result tag is displayed on the display unit,
Sending the tag for the re-search key via the network;
Receiving the re-search result data via the network;
Based on the received re-search result data, a re-search result screen on which the re-search result tag is displayed may be controlled to be displayed on the display unit.

1 is a configuration example of an image search system, a terminal device, and a server realized by the present embodiment. Explanatory drawing of the method of reversely utilizing the ambiguity of image search. Explanatory drawing of the method of reversely utilizing the ambiguity of image search. 4A and 4B are diagrams for explaining an outline of a procedure for creating similar search result data. FIG. 5A and FIG. 5B are diagrams for explaining an outline of a procedure for creating first re-search result data. FIGS. 6A and 6B are diagrams for explaining an outline of a procedure for creating second re-search result data. FIGS. 7A and 7B are conceptual diagrams illustrating ranges of a plurality of similar images. FIG. 8A and FIG. 8B are diagrams for explaining a modified example of the outline of the procedure for creating the second re-search result data. 9A and 9B show examples of similar search result screens. The flowchart for demonstrating the operation | movement of this embodiment. The flowchart for demonstrating the operation | movement of this embodiment. The flowchart for demonstrating the operation | movement of this embodiment. The flowchart for demonstrating the operation | movement of this embodiment. The flowchart for demonstrating the operation | movement of this embodiment. The flowchart for demonstrating the operation | movement of this embodiment. The figure for demonstrating operation | movement of this embodiment. An example of the data structure of an image database. 18A and 18B show examples of the data structure of the related tag database. 19A and 19B show examples of the data structure of similar search result data. 20A and 20B are other examples of the similar search result screen. 21A and 21B show examples of the hub screen. The other example of the data structure of similar search result data. The flowchart for demonstrating the tabulation process of a tag. FIG. 24A and FIG. 24B are diagrams for explaining tag counting processing. The figure for demonstrating the modification of this embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. System Configuration FIG. 1 shows a configuration example of an image search system, server, and terminal device realized by this embodiment. The image search system includes a terminal device 100 (electronic device, communication terminal, portable terminal device) and a server 200 that is connected to the terminal device 100 via a network 10. Note that other external servers and terminal devices can also be connected to the network 10 for communication.

  In the present embodiment, the server 200 searches for a plurality of similar images similar to the search target image acquired (captured) by the terminal device 100, for example. And the user interface etc. in an image search system are implement | achieved using the tag linked | related with each similar image of these some similar images.

  In other words, image retrieval using image recognition processing is inherently ambiguous. In the present embodiment, the ambiguity of such image search is used in reverse to create a spread in the user's preference / hobby / interest, user's behavior, e-commerce, and the like.

  For example, in A1 of FIG. 2, the user is photographing a wine label using the terminal device 100 such as a digital camera or a camera-equipped mobile phone. Then, the photographed image is transmitted to the server 200 via a network such as the Internet, and an image search is requested.

  In this case, the conventional image search systems focus on identifying the photographed wine based on the photographed image and introducing a store or the like that sells the wine. Therefore, how to accurately identify the wine from the photographed image has been a technical problem.

  On the other hand, in this embodiment, the ambiguity of the image search is reversed. For example, the search target image captured in A1 is transmitted from the terminal device 100 to the server 200, and a plurality of similar images similar to the search target image are searched.

  For example, as shown in A2, a liquor image similar to A1 wine is retrieved as a similar image and presented to the user. When a similar image is retrieved, information about the similar liquor is presented to the user as related information. In this case, the related information is presented using a hub screen as described later, for example. That is, a hub screen is displayed using the hub screen information associated with the similar image, and information about liquor similar to A2 is presented on the hub screen. Specifically, a first hub screen on which a link destination of a site related to A2 liquor is presented is displayed.

  Furthermore, as shown in A3 and A4, similar images and related information about liquor that is slightly similar to or completely different from A1 wine are also presented. Specifically, when the user selects a similar image of A3, the second hub screen on which the link destination of the site related to the alcohol of A3 is presented is displayed. When the user selects a similar image of A4, a third hub screen on which link destinations of a plurality of sites related to A4 liquor are presented is displayed. These first, second, and third hub screens are set so that at least some of the presented link destinations are different. By doing so, it is possible to create a spread of user's tastes such as unexpected discovery of taste, for example.

  Further, when the same wine image as A1 is searched by the similar search, as shown in A5, information such as the wine producing region, the sommelier's voice and the producer's voice is presented to the user. Furthermore, as shown in A6, information such as wine glasses and snacks is also presented to create the spread of e-commerce.

  In addition, for example, a tag (for example, liquor) of a similar image of A2 or A3 is presented. Information similar to a tag of a similar image (for example, alcohol) that the user may want based on the search target image is presented. In addition, related tags (for example, banquets) associated with tags of similar images (for example, liquor) and related information (for example, banquets) related links (for example, banquets) 4 hub screens) can also be presented. Speaking of liquor, it also presents information that the user may be interested in, as if associated with a banquet. By doing so, it is possible to create help for information that the user wants to know, such as discovery of a matter that the user wanted to find.

  In addition, when the user shoots a DVD work package as shown in B1 of FIG. 3, as shown in B2, B3, and B4, the relevance of similar works or other works searched as similar images. Present information to create a wide range of user interests.

  For example, when a similar image of the work of B2 is selected, the first hub screen on which the link destination related to the work of B2 is presented is displayed. Similarly, when similar images of the works B3 and B4 are selected, the second and third hub screens on which link destinations related to the works B3 and B4 are presented are displayed. Or as shown to B5 and B6, the relevant information of the work is shown to a user, and the spread of e-commerce is created. In addition, a tag for a similar image (for example, a DVD), a related tag associated with a tag for a similar image (for example, a DVD) (for example, a DVD mail order), and related information about the related tag (for example, a DVD The link destination of the site related to mail-order sales) is presented, and help for information that the user wants to know is created.

  As described above, in the image search system according to the present embodiment, the ambiguity of the similarity search is reversely used in the image search system of the present embodiment, which has been converged in a specific direction such as a product by image search in the past. By directing the vector in the direction, the user's tastes, hobbies, interests, behaviors, e-commerce spread, help of information that the user wants to know, etc. are realized.

  In order to realize such an idea image search, the present embodiment employs a system having a configuration as shown in FIG. Hereinafter, specific configurations of the image search system, the server 200, and the terminal device 100 according to the present embodiment will be described in detail.

  The network 10 (communication network) can include a mobile communication network, a public network such as the Internet network, or a fixed telephone network. The mobile communication network for communication connection with the terminal device 100 is a mobile telephone network such as a CDMA (Code Division Multiple Access) method or a PDC (Personal Digital Cellular) method, or mobile data such as a PDC-P (PDC-Packet) method. A communication network can be included. The network 10 may be wired or wireless.

  The server 200 (search server, management server) performs a search process for similar images of the search target image, and is connected to the terminal device 100 or other external server (not shown) via the network 10. The function of the server 200 can be realized by hardware such as an information processing apparatus (rental server, shared server) and a program such as a CGI program.

  The terminal device 100 is, for example, a digital camera (digital still camera or digital video camera) possessed by a user, a mobile phone with a camera, an information processing terminal, and the like. Etc. have a function to communicate with.

  The server 200 that is a Web server includes a processing unit 202, a storage unit 219, an image database 220, a related tag database 230, a communication unit 238, and an operation unit 240. The server 200 according to the present embodiment is not limited to the configuration shown in FIG. 1, and various components such as omitting some of these components (for example, the image database 220 and the related tag database 230) and adding other components. Can be implemented.

  The processing unit 202 performs various processing such as control processing, image search processing, and data creation processing of the entire server, and functions thereof are hardware such as various processors (CPU, etc.) and PHP (PHP: Hypertext Preprocessor). ) Or Perl, and can be realized by a CGI program written in a language such as Perl.

  The storage unit 219 serves as a work area for the processing unit 202, the communication unit 238, and the like, and its functions can be realized by a memory such as a RAM or an HDD (hard disk drive). The storage unit 219 includes a transmission data storage unit 221, a similar image ID storage unit 222, a tag processing result data storage unit 223, a similar search result data storage unit 224, a reply data storage unit 225, and a re-search result data storage unit 226. .

  The image database 220 is a relational database realized by, for example, MySQL or the like, and stores registered image data, metadata associated with the registered image, and the like. The image database 220 may store URL information of the hub screen.

  The related tag database 230 is a relational database realized by, for example, MySQL, and stores tags and related tags. The related tag database 230 may store URL information of related tag hub screens.

  The communication unit 238 is for communicating information with the terminal device 100 and other external servers via the network 10. The operation unit 240 is used by the operator of the server 200 to input various information for server management and the like.

  The processing unit 202 includes a reception processing unit 204, an image recognition unit 206, a tag processing unit 210, a data creation unit 214, and a transmission processing unit 218. Various modifications may be made such as omitting some of these components or adding other components.

  The reception processing unit 204 performs data reception processing. Specifically, a process for receiving transmission data regarding the search target image via the network 10 is performed, and for example, transmission data from the terminal device 100 is received. Here, the transmission data for the search target image may be the search target image data itself, or may be recognition data for the search target image (image recognition data). As the recognition data, for example, feature amount data (feature data) of the search target image, data specifying the search target image, or the like can be assumed. The reception processing unit 204 can receive reply data for the tag for the re-search key. The reception of the reply data will be described later.

  The image recognition unit 206 performs image recognition processing. Specifically, an image recognition process for the search target image is performed, and a similar image similar to the search target image is searched from registered images registered in the image database 220. For example, the search target image and each registered image in the image database 220 are collated, and a predetermined upper number (J) of registered images similar to the search target image are searched (extracted) as a plurality of similar images. )

  The image recognition unit 206 can include a recognition data creation unit 208. The recognition data creation unit 208 creates recognition data necessary for image recognition from the search target image data. For example, feature data as described below is created as the recognition data. Then, the image recognition unit 206 performs a collation process that compares the feature amount data that is the recognition data of the search target image with the feature amount data that is the recognition data of the registered image in the image database 220, thereby performing the search target image. And whether the registered images are similar to each other. Note that the form of the recognition data is not limited as long as the data can be distinguished from other images. Further, when the reception processing unit 204 receives recognition data such as feature amount data instead of the search target image data itself as transmission data from the terminal device 100 or the like, the configuration of the recognition data creation unit 208 is configured as follows. Can be omitted.

  The tag processing unit 210 performs various processes related to tags. Specifically, a tag (character information) associated with each similar image of a plurality of similar images (registered images) searched by the image recognition unit 206 is acquired. Then, for example, sorting processing of a plurality of tags acquired from a plurality of similar images is performed.

  That is, the image database 220 stores a corresponding tag associated with each registered image. This tag association can be realized, for example, by registering the corresponding tag when the user registers his / her captured image in the image database 220. Alternatively, it can be realized by a trader or server operator registering a corresponding tag in association with an image when registering the image. When the image recognition unit 206 searches for a plurality of similar images similar to the search target image, the tag processing unit 210 acquires tags associated with the plurality of similar images (registered images) from the image database 220.

  The tag in the present embodiment is information associated with an image, and is provided so as to be acquired together with the image. For example, the tag can be one or more genre information into which the image is classified. Specifically, for example, genre information such as “flower”, “sunflower”, “summer”, and “gogh” is associated with the sunflower flower image.

  If the selection items (options) classified by this tag are presented to the user using, for example, a selection menu displayed on the display unit of the mobile terminal device, the user can select any genre information by selecting the selection item. (See FIG. 2 and FIG. 3). In metaphor, if an image is a “starting point”, it can be said that a tag is a “direction vector” indicating directionality.

  The tag processing unit 210 performs a totaling process for the acquired tags. Specifically, the tag processing unit 210 includes a tag totaling unit 212, and the tag totaling unit 212 performs a totaling process of totaling the number of appearances for each tag of a plurality of tags acquired from a plurality of similar images. Then, the tag processing unit 210 performs a sorting process for a plurality of tags (search result tags) based on the counting result. Specifically, a plurality of tags are rearranged in descending order of appearance numbers. Note that the tag processing unit 210 selects only a part of the plurality of tags (for example, a part of the tags corresponding to the higher number of appearances among the plurality of tags rearranged in the descending order of the number of appearances). May be.

  The tag counting process is not limited to the process of simply counting the number of appearances of tags. For example, various modifications such as obtaining a weighting coefficient based on the similarity of similar images and performing a tabulation process using the weighting coefficient are possible. For example, even if the number of appearances is small, if the similarity of a similar image is high, the total value of the tags of the similar image may be increased.

  In addition, the tag processing unit 210 can acquire other tags (re-search result tags) excluding the tag for the re-search key based on the reply data. Regarding acquisition of other tags (re-search result tags), It will be described later.

  The data creation unit 214 creates various data such as similar search result data and re-search result data. For example, based on similar search result data for presenting multiple similar images, similar search result data for presenting multiple tags (search result tags) of multiple similar images, and multiple tags of multiple similar images Similar search result data for presenting a plurality of related tags acquired from the related database 230, similar search result data for displaying a hub screen, and other tags (re-search results) except for a tag for a re-search key Re-search result data etc. for presenting (tag) is created.

  Specifically, the data creation unit 214 creates similar search result data including a plurality of tags (search result tags) of a plurality of similar images. For example, when the image recognition unit 206 searches for a plurality of similar images of the search target image, the tag processing unit 210 acquires a plurality of tags of the plurality of similar images and performs aggregation processing. In this way, information presentation is realized by reversely using the ambiguity of image search. The data creation unit 214 can create re-search result data including a plurality of other tags (re-search result tags) acquired from a plurality of similar images based on the reply data. Will be described later.

  The data creation unit 214 can create similar search result data further including a plurality of related tags corresponding to a plurality of tags (search result tags) of a plurality of similar images. For example, the data creation unit 214 acquires a plurality of related tags from the related database 230 using the plurality of tags after the aggregation processing, and generates similar search result data including the acquired related tags. By presenting the tag and the related tag to the user, it is possible to more reliably create help for the information that the user wants to know.

  The data creation unit 214 provides hub screen information (hub screen setting information) for presenting a plurality of link destinations corresponding to each related tag for each related tag included in the similar search result data. Similar search result data may be created in association with.

  The data creation unit 214 can create similar search result data further including a plurality of similar images. For example, the data creation unit 214 creates similar search result data further including a plurality of similar images (registered images) searched (extracted) by the image recognition unit 206.

  The data creation unit 214 obtains a similarity search result in which hub screen information (hub screen setting information) for presenting a plurality of link destinations corresponding to each similar image is associated with each similar image of the plurality of similar images. Data may be created. For example, when the image recognition unit 206 searches for a plurality of similar images of the search target image, the data creation unit 214 determines that the first similar image is the first similar image among the plurality of similar images. The similar search result data associated with the first hub screen information for presenting a plurality of link destinations corresponding to is created. On the other hand, for the second similar image among the plurality of similar images, similar search result data associated with second hub screen information for presenting a plurality of link destinations corresponding to the second similar image. Create

  That is, in order to realize information presentation that reversely uses the ambiguity of the image search as described in FIGS. 2 and 3, a hub screen in which different link destinations are set for each similar image is displayed. . For example, when the user selects the first similar image, the first hub screen in which the first link destination is set is displayed. When the user selects the second similar image, the first link is displayed. Similar search result data is created so that a second hub screen in which a second link destination different from the destination is set is displayed. Specifically, at least some of the link destinations set in the first hub screen information are different from the plurality of link destinations set in the second hub screen information. . More specifically, a plurality of link destinations corresponding to tags associated with the first similar image are set in the first hub screen information, and the second similar image is set in the second hub screen information. Similar search result data is created so that a plurality of link destinations corresponding to the tag associated with is set. That is, a link destination where information corresponding to a tag associated with each similar image is presented is set on each hub screen.

  The data creation unit 214 is similar in which related tag hub screen information (related tag hub screen setting information) for presenting a plurality of link destinations corresponding to each related tag is associated with each related tag of the plurality of related tags. Search result data may be created. For example, when a plurality of tags of a plurality of similar images are acquired and totalized by the tag processing unit 210, the data creation unit 214 determines the first related tag among the plurality of related tags. Similar search result data associated with first related tag hub screen information for presenting a plurality of link destinations corresponding to one related tag is created. On the other hand, for the second related tag among the plurality of related tags, a similar search result in which second related tag hub screen information for presenting a plurality of link destinations corresponding to the second related tag is associated. Create data.

  The data creation unit 214 may use, for example, a plurality of tags sorted by the tag processing unit 210 as the aggregation process. For example, a data structure in which a plurality of tags are rearranged in descending order of the number of appearances is used. Alternatively, each tag of a plurality of sorted tags may be used. For example, similar search result data having a data structure in which one or a plurality of related tags corresponding to a plurality of tags sorted in descending order of appearance is associated is created.

  Note that when the data creation unit 214 selects some tags (selected tags) from among the plurality of tags, the data creation unit 214 selects a plurality of tags from the related tag database 230 based on the selected some tags (selected tags). Similar search result data in which one or a plurality of related tags among a plurality of related tags are associated with each tag (each selected tag) is obtained. . In this case, the data creation unit 214 can reduce the capacity of the similar search result data. In addition, the data creation unit 214 may create similar search result data in which each tag of each selected tag (each selected tag) is different from each related tag of a plurality of related tags.

  When the data creation unit 214 creates similar search result data including a plurality of similar images (registered images), the plurality of similar images included in the similar search result data may be rearranged in descending order of similarity. . The data creating unit 214 sets a part of similar images (registered images) searched by the image recognizing unit 206 (for example, the higher similarity among a plurality of similarities sorted in descending order of similarity). Only some corresponding similar images) may be selected.

  It should be noted that the data creation unit 214 includes search result data (the first search result data including the similar image data to which none of the plurality of tags included in the similar search result data is associated among the similar images having higher similarity to the search target image. 2 similar search result data). That is, among the predetermined number of similar images having higher similarity, a similar image that is not associated with any of a plurality of tags included in the similar search result data is searched, and the data about the similar image is searched as search result data. Included.

  Here, the data on the similar image may be the data of the similar image itself or data specifying the similar image. The data included in the similar search result data may be at least one of data on a similar image and a tag associated with the similar image.

  The transmission processing unit 218 performs data transmission processing. Specifically, processing for transmitting similar search result data and re-search result data via the network 10 is performed. In this case, similar search result data (search result data) and re-search result data created by the data creation unit 214 may be transmitted via the network 10 to the terminal device 100 that acquired the search target image. Then, the similar search result data may be transmitted to another external server or another terminal device.

  The terminal device 100 such as a digital camera, a mobile phone, a portable information terminal, or an information processing apparatus includes a processing unit 102, a storage unit 120, a communication unit 138, an operation unit 140, an imaging unit 150, and a display unit 160. The terminal device 100 according to the present embodiment is not limited to the configuration shown in FIG. 1, and various modifications such as omitting some of these components or adding other components are possible.

  The processing unit 102 performs various processes based on operation information from the operation unit 140, information received by the communication unit 138, and the like. The function of the processing unit 102 can be realized by hardware such as various processors (CPU and the like) and ASIC (gate array and the like), a program stored in the information storage medium 130, and the like.

  The storage unit 120 serves as a work area for the processing unit 102, the communication unit 138, and the like, and functions thereof can be realized by a memory such as a RAM or an HDD (hard disk drive). The storage unit 120 includes a transmission data storage unit 122, a similarity search result data storage unit 124, a reply data storage unit 126, and a re-search result data storage unit 128.

  The information storage medium 130 (a computer-readable medium) stores programs, data, and the like, and its function can be realized by an optical disk (CD, DVD), an IC card (memory card), or the like. The processing unit 102 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 130. That is, in the information storage medium 130, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit). Is memorized.

  The communication unit 138 is for communicating information with the server 200 and other terminal devices via the network 10. The operation unit 140 is used by the user to perform various operations of the terminal device 100. For example, the operation unit 140 can be realized by a button for selecting a selection item in the display screen of the display unit 160, a shutter button for image capturing, or the like.

  The imaging unit 150 is for imaging (photographing) a subject, and can be realized by, for example, a CCD or a CMOS sensor.

  The display unit 160 is for displaying various display screens, and can be realized by, for example, a liquid crystal display, a display using a light emitting element such as an organic EL, an electrophoretic display, or the like. The display unit 160 displays a similar search result screen, a re-search result screen, a hub screen (including a related tag hub screen), and the like.

  The processing unit 102 includes an image acquisition unit 104, a transmission processing unit 106, a reception processing unit 108, and a display control unit 110. Various modifications may be made such as omitting some of these components or adding other components.

  The image acquisition unit 104 acquires a search target image. For example, a search target image (subject image) obtained by imaging a subject with the imaging unit 150 is captured. For example, a search target image stored in a memory card or the like may be captured without actually capturing the search target image.

  The transmission processing unit 106 performs data transmission processing. For example, a process of transmitting transmission data for the search target image acquired by the image acquisition unit 104 to the server 200 via the network 10 is performed. That is, the acquired search target image data is converted into a format to be transmitted to the server 200 to create transmission data. The generated transmission data is transmitted to the server 200. The transmission data in this case may be the search target image data itself or may be data for recognizing the search target image. In addition, the transmission processing unit 106 transmits reply data for a tag for a re-search key selected from a plurality of tags (search result tags) acquired from a plurality of similar images similar to the search target image. However, transmission of the reply data will be described later.

  The reception processing unit 108 performs data reception processing. For example, processing for receiving similar search result data, re-search result data, and the like of the search target image via the network 10 is performed.

  The display control unit 110 performs display control of the display unit 160. For example, display control of the display unit 160 is performed based on the received similar search result data and re-search result data. Specifically, display control for displaying various screens such as a similar search result screen, a re-search result screen, a hub screen, and a selection screen on the display unit 160 is performed.

  For example, the display control unit 110 displays a similar search result screen on which a plurality of tags (search result tags) acquired from a plurality of similar images similar to the search target image are displayed based on the received similar search result data. Control to be displayed on the unit 160 is performed. The display control unit 110 controls the display unit 160 to display a similar search result screen on which a plurality of similar images similar to the search target image are displayed. The display control unit 110 may perform control to display a plurality of similar images or a plurality of tags (search result tags) on the display unit 160, and display both a plurality of similar images and a plurality of tags (search result tags). You may perform control displayed on the display part 160. FIG. Further, the display control unit 110 displays a plurality of related tags acquired from the related tag database 230 based on a plurality of tags (search result tags) acquired from a plurality of similar images based on the received similar search result data. The similar search result screen may be displayed on the display unit 160.

  The display control unit 110 also includes a plurality of other tags (re-search result tags) acquired from a plurality of similar images based on the received re-search result data, excluding tags for the re-search key. Control is performed to display a re-search result screen on which other tags (re-search result tags) are displayed on the display unit 160.

2. Outline of Operation FIG. 4 shows an outline of a procedure for creating similar search result data. A search target image as indicated by C1 in FIG. 4A is transmitted from the terminal device 100 to the server 200 as transmission data. A plurality of similar images as indicated by C2 in FIG. 4A are searched based on search target images (transmission data). In the server 200, a plurality of tags (search result tags) associated with a plurality of similar images are acquired. A plurality of tags (search result tags) are transmitted from the server 200 to the terminal device 100 as similar search result data. As an example of the search result tag, “flower” and “clothes” are indicated by C3 in FIG.

  For the search target image as shown by C1 in FIG. 4A, the user is interested in the tag "flower", interested in the tag "clothing", and interested in other tags. There are cases. For example, when the user is interested in the tag “flower”, the user tags the first re-search key from among the search result tags (“flower”, “clothes”) via the operation unit 150, for example. ("Flower") can be selected.

  FIG. 5 shows an outline of a procedure for creating the first re-search result data. A tag for the first re-search key (“flower”) as indicated by D1 in FIG. 5A is transmitted from the terminal device 100 to the server 200 as first reply data. A plurality of similar images as indicated by D2 in FIG. 5A are included in the search target image indicated by C1 in FIG. 4A and the tag (first reply data) for the first re-search key. Specified based on. In the server 200, a plurality of other tags (first re-search result tags) associated with a plurality of similar images are acquired. A plurality of other tags (first re-search result tags) are transmitted from the server 200 to the terminal device 100 as first re-search result data. As an example of the first re-search result tag, “cherry blossom” and “sunflower” are indicated by D3 in FIG. Note that each similar image of a plurality of similar images as indicated by D2 in FIG. 5A includes a first re-search key tag ("flower") and other tags (for example, "sakura", " The first re-search result tag does not include the tag for the first re-search key (“flower”).

  For the search target image as indicated by C1 in FIG. 4A, the user is interested in the tag “sakura”, the user is interested in the tag “sunflower”, and is interested in other tags. There are cases. For example, when the user is interested in the tag “sakura”, the user performs the second re-search from the first re-search result tags (“sakura”, “sunflower”) via the operation unit 150, for example. The tag for the key ("Sakura") can be selected.

  FIG. 6 shows an outline of the procedure for creating the second re-search result data. A tag for the second re-search key (“sakura”) as indicated by E1 in FIG. 6A is transmitted from the terminal device 100 to the server 200 as second reply data. A plurality of similar images as indicated by E2 in FIG. 6A includes a search target image indicated by C1 in FIG. 4A, a first re-search key tag (first reply data), and a first image. The second re-search key tag (second reply data) is designated. In the server 200, a plurality of other tags (second re-search result tags) associated with a plurality of similar images are acquired. A plurality of other tags (second re-search result tags) are transmitted from the server 200 to the terminal device 100 as second re-search result data. As an example of the second re-search result tag, “Edohigan” and “Yamazakura” are indicated by E3 in FIG. 6B. It should be noted that each similar image of a plurality of similar images as indicated by E2 in FIG. 6A includes a first re-search key tag ("flower") and a second re-search key tag (" Cherry ”) and other tags (for example,“ Edohigan ”,“ Yamazakura ”), and the second re-search result tag includes a tag for the first re-search key (“ flower ”) and the second The tag for the re-search key ("Sakura") is not included.

  FIG. 7 is a conceptual diagram illustrating the range of a plurality of similar images. A square area indicated by C2 in FIGS. 7A and 7B conceptually corresponds to a plurality of similar images indicated by C2 in FIG. A plurality of similar images corresponding to the quadrangular area indicated by C2 is searched based on the search target image from registered images registered in the image database 220, for example. The hatched area indicated by D2 in FIG. 7A conceptually corresponds to a plurality of similar images indicated by D2 in FIG. The plurality of similar images corresponding to the hatched area indicated by D2 are a plurality of similar images corresponding to the rectangular area indicated by C2, and are for the first re-search key indicated by D1 in FIG. A plurality of similar images having the tag (“flower”). A hatched area indicated by E2 in FIG. 7B conceptually corresponds to a plurality of similar images indicated by E2 in FIG. The plurality of similar images corresponding to the hatched area indicated by E2 are a plurality of similar images corresponding to the rectangular area indicated by C2, and are for the first re-search key indicated by D1 in FIG. And a second re-search key tag (“cherry blossom”) indicated by E1 in FIG. 6A.

  When the search result tag acquired from a plurality of similar images corresponding to the rectangular area indicated by C2 is a tag with a high abstraction level such as “flower” or “clothes”, the tag for the first re-search key (“ Using “flowers”), it is possible to acquire specific tags (first re-search result tags) such as “sakura” and “sunflower”. Furthermore, more specific tags (second re-search result tags) such as “Edhigan” and “Yamazakura” can be acquired using the second re-search key tag (“sakura”). As described above, the selection of the tag for the re-search key is repeated, so that the user can reach the target information.

  By the way, a plurality of similar images indicated by E2 in FIG. 6 (A) includes a first re-search key tag (“flower”) indicated by D1 in FIG. 5 (A) and FIG. 6 (A). It is necessary to have a tag ("sakura") for the second re-search key indicated by E1 in the middle. However, in this case, the user may not be able to reach the target information by repeatedly selecting the tag for the re-search key. Therefore, a plurality of similar images indicated by E2 in FIG. 6A have a tag for the second re-search key (“cherry tree”), and a tag for the first re-search key (“flower”). ) May be changed to a plurality of similar images irrelevant to the presence or absence. That is, in all or a part of the searches after the re-search, at least the tag for the previous search key is not used as the search condition. In other words, as described above, the conditions are not narrowed down as the search is repeated, but the same level of search is performed by changing the conditions one after another.

  FIG. 8 shows a modified example of the outline of the procedure for creating the second re-search result data. A tag for the second re-search key (“sakura”) as indicated by E1 in FIG. 8A is transmitted from the terminal device 100 to the server 200 as second reply data. A plurality of similar images as indicated by E2 'in FIGS. 8A and 8B are designated based on the second re-search key tag (second reply data). In the server 200, a plurality of other tags (second re-search result tags) associated with a plurality of similar images corresponding to the hatched area indicated by E2 'in FIG. It should be noted that a second re-search key tag (“sakura”) and other tags (“sakura”) are included in each similar image of a plurality of similar images as indicated by E2 ′ in FIG. 8A and FIG. 8B. For example, the second re-search result tag does not include a tag for the second re-search key (“cherry tree”). The second re-search result tag may include a tag for the first re-search key (“flower”) and a search result tag (for example, “clothes”).

  In the previous example, the search result D2 in FIG. 7 (A) is further narrowed down to E2 in FIG. 7 (B), but in this example, the search result D2 in FIG. 7 (A) is the search result in FIG. 8 (B). Only E2 'in the result is replaced, and no narrowing is performed. In spite of this, in both examples, the search result C2 of FIG. The hatched area indicated by E2 'in FIG. 8B is wider than the hatched area indicated by E2 in FIG. 7B. By repeatedly selecting the tag for the re-search key so as to obtain the hatched area indicated by E2 'in FIG. 8B, the user may easily reach the target information.

  FIG. 9 shows an example of a similar search result screen. As shown in FIG. 9A, the display control unit 110 may realize a list display mode (first mode) in which a plurality of similar images are displayed, and a plurality of tags (search result tags) are displayed. The tag display mode (second mode) may be realized. As shown in FIG. 9B, the tag display mode includes not only a plurality of tags (flowers, clothes) but also a plurality of related tags (dumplings, gardening, T-shirts, bargains) acquired based on the plurality of tags. ) May be displayed. The data creation unit 214 includes a list display mode or a tag so that the similar search result data includes a plurality of similar images in the list display mode and includes a plurality of tags (search result tags) and a plurality of related tags in the tag display mode. Similar search result data capable of switching display modes can be created.

  Then, in the similar search result screen as shown in FIG. 9A, when the first similar image among a plurality of similar images similar to the search target image is selected, it corresponds to the first similar image. The first hub screen on which a plurality of link destinations are presented is displayed. On the other hand, when the second similar image is selected from the plurality of similar images on the similar search result screen, the second hub screen on which a plurality of link destinations corresponding to the second similar image are presented. Is displayed. In this case, the plurality of link destinations presented on the first hub screen and the plurality of link destinations presented on the second hub screen are different from each other in at least some of the link destinations. Perform display control. Further, a plurality of link destinations corresponding to tags associated with the first similar image are presented on the first hub screen, and a plurality of links corresponding to tags associated with the second similar image are presented on the second hub screen. The link destination of is to be presented.

  In the similar search result screen as shown in FIG. 9B, when a first related tag (for example, dumpling) is selected from among a plurality of related tags, a plurality of items corresponding to the first related tag are selected. The first related tag hub screen on which the link destination is presented is displayed. On the other hand, in the similar search result screen, when a second related tag (for example, horticulture) is selected from among a plurality of related tags, a plurality of link destinations corresponding to the second related tag are presented. The second related tag hub screen is displayed. In this case, the plurality of link destinations presented on the first related tag hub screen and the plurality of link destinations presented on the second related tag hub screen are related so that at least some of the link destinations are different. Controls the display of the tag hub screen. A plurality of link destinations corresponding to the first related tag are presented on the first related tag hub screen, and a plurality of link destinations corresponding to the second related tag are presented on the second related tag hub screen. To.

  Further, the display control unit 110 may perform control to display a similar search result screen on which a plurality of tags acquired from a plurality of similar images similar to the search target image are sorted and used on the display unit 160. For example, a similar search result screen that uses the result of the sorting procedure in which a plurality of tags are sorted according to the number of appearances of each tag is displayed. Specifically, for each tag of a plurality of tags (search result tags), a corresponding one or a plurality of related tags among a plurality of related tags are displayed in association with each other.

  As shown in FIG. 9A, in the case where a plurality of similar images similar to the search target image are presented to the user, the ambiguity of the image search can be used. However, the range of similar images is limited to liquor that is slightly similar to or completely different from the wine of A1, as shown by A3 and A4 in FIG. In other words, for example, there is a limit in helping the user wants to know information. Therefore, as shown in FIG. 9B, a plurality of tags (search result tags) of a plurality of similar images and a plurality of related tags associated with a plurality of tags of the plurality of similar images are presented to the user. , Help for information that users want to know can be created. A plurality of related tags are acquired from the related database 230 using a plurality of tags of a plurality of similar images.

3. Detailed Operation Next, the operation of the present embodiment will be described with reference to the flowcharts of FIGS. 10, 11, 12, 13, 14, and 15. 10, 12, 13, and 14 are flowcharts for explaining the operation on the terminal device 100 side, and FIGS. 11 and 15 are flowcharts for explaining the operation on the server 200 side.

  As illustrated in FIG. 10, the terminal device 100 acquires a search target image (step S1). For example, an image obtained by photographing the subject with the imaging unit 150 is acquired as a search target image. Then, transmission data is created based on the acquired search target image (step S2). For example, the transmission data is created by converting into a server format. Then, the created transmission data is transmitted to the server (step S3).

  As shown in FIG. 11, the server 200 receives transmission data from the terminal device 100 (step S11). Then, the search target image is converted into recognition data (step S12). For example, the search target image data is converted into feature amount data.

  Next, the recognition data for the search target image is collated with the recognition data for the registered image in the image database 220 to obtain the image ID and similarity of the registered image similar to the search target image as shown in FIG. (Step S13). As shown in FIG. 16, the image IDs and the degrees of similarity of a plurality of (predetermined number) of similar images acquired are sent from the image recognition unit 206 to the tag processing unit 210. Note that the image recognition unit 206 can create a processing ID corresponding to the search target image and send the processing ID to the tag processing unit 210.

  Next, using the acquired image ID, a tag of the corresponding registered image is acquired from the image database 220, and tag counting processing is performed (step S14). That is, the tag processing unit 210 (tag totaling unit 212) uses the image IDs of a plurality of similar images sent from the image recognition unit 206 to retrieve a plurality of tags for these plurality of similar images from the image database 220. The tag is acquired and a tag counting process as described in detail later is performed. As shown in FIG. 16, the acquired tags and the image IDs of the corresponding similar images are sent from the tag processing unit 210 to the data creation unit 214. Note that the tag processing unit 210 can send the processing ID to the data creation unit 214.

  Next, a part of tags (selected tags) among the plurality of tags is selected based on the counting result (step S15). That is, some tags (selected tags) of the plurality of tags acquired through the tag counting process are sent from the tag processing unit 210 to the data creation unit 214. Further, a part of image IDs (selected image IDs) of the image IDs of a plurality (a predetermined number) of similar images from the image recognition unit 206 are sent from the tag processing unit 210 to the data creation unit 214. Note that the number of selected tags and the number of selected image IDs are determined according to the size of the similar search result screen, for example.

  Next, related tags are acquired from some tags (selected tags) via the related tag database 230 (step S16). That is, the data creation unit 214 acquires a related tag using a part of tags (selected tags) from the tag processing unit 210. The number of related tags is determined according to the size of the similar search result screen, for example. In addition, the data creation unit 214 can acquire not only related tags but also related tag hub screen information corresponding to the related tags via the related tag database 230 from some tags (selected tags).

  Next, similar search result data including some tags (selected tags), related tags, and similar images is created, and the similar search result data includes some tags (selected tags) as search result tags (step S17). ). That is, the data creation unit 214 acquires a similar image from the image database 220 by using a partial image ID (selected image ID) of the image ID sent from the tag processing unit 210. The data creation unit 214 can acquire not only similar images but also hub screen information corresponding to similar images from a part of the image IDs (selected image IDs) via the image database 220. Then, using the obtained related tags, similar images, hub screen information (including related tag hub screen information), and some tags (selection tags, search result tags) sent from the tag processing unit 210, similar Similar search result data (see FIGS. 19A and 19B described later) for displaying the search result screen and the hub screen is created. Then, the created similar search result data is transmitted to the terminal device 100 (step S18). The data creation unit 214 can create similar search result data including a process ID.

  As shown in FIG. 12, the terminal device 100 receives the similar search result data from the server (step S21). Then, for example, a similar search result screen (see FIG. 9A) on which similar images are displayed is displayed (step S22). A plurality of similar images are displayed on the screen of the terminal device 100 in the order of similarity, for example. Moreover, you may display a some similar image on the screen of the terminal device 100 corresponding to each tag of a some tag as shown in FIG.9 (B).

  Next, it is determined whether or not the user has selected a tag display mode (for example, a tab indicated by F1 in FIG. 9A) on the similar search result screen (step S23). A similar search result screen as shown in FIG. 9B is displayed (step S31).

  If the tag display mode is not selected in step S23, it is determined whether or not the user has selected a similar image (icon) on the similar search result screen (step S24). The hub screen (see FIGS. 21A and 21B described later) is displayed (step S25). With this hub screen, the link destination corresponding to the similar image is presented to the user.

  Next, it is determined whether or not the user has selected the link destination of the hub screen (step S26). If selected, the communication connection is established with the external server corresponding to the link destination (step S27). That is, the connection destination of the terminal device 100 is set to a link destination URL (Uniform Resource Locator). Further, it is determined whether or not the user has selected display of the similar search result screen (step S28). If selected, the process returns to step S22, and the display is switched from the hub screen to the similar search result screen to return to the original.

  When the tag display mode is selected in step S23, as shown in FIG. 9B, for example, a plurality of tags (search result tags) and a plurality of related tags corresponding to each tag of the plurality of tags are included. The displayed similar search result screen is displayed (step S31, FIG. 13).

  Next, it is determined whether or not the user has selected a related tag (hyperlink) on the similar search result screen (step S32). If selected, a related tag hub screen corresponding to the related tag is displayed. (Step S33). On the related tag hub screen, a link destination corresponding to the related tag is presented to the user.

  Next, it is determined whether or not the user has selected the link destination of the related tag hub screen (step S34). If selected, the communication connection is established with the external server corresponding to the link destination (step S35). That is, the connection destination of the terminal device 100 is set to the link destination URL. Further, it is determined whether or not the user has selected display of the similar search result screen (step S36). If selected, the process returns to step S31, and the display is switched from the related tag hub screen to the similar search result screen to return to the original. .

  If the related tag (hyperlink) on the similar search result screen is not selected in step S32, as shown in FIG. 14, has the user selected the tag for the re-search key from the plurality of tags on the similar search result screen? In step S41, if it is selected, the reply data for the tag for the re-search key is transmitted from the terminal device 100 to the server 200 (step S42). If the tag for the re-search key is not selected in step S41, the process returns to step S32 in FIG. Note that step S32 and step S41 may be performed simultaneously, and it may be determined whether a tag for a re-search key or a related tag has been selected. Note that the transmission processing unit 106 can transmit reply data including a processing ID. The transmission processing unit 106 may transmit reply data including the search target image instead of the processing ID.

  In step S41 of FIG. 14, it may be determined whether or not the user has selected a tag for the re-search key using the radio button indicated by F3 in FIG. 9B, as shown in FIG. 4B. Alternatively, the user may determine whether or not the user has selected a tag for the re-search key by using a click on the tag for the re-search key itself. Further, in step S42 in FIG. 14, the user may transmit a tag for a re-search key by using the submit button indicated by F4 in FIG. 9B. The tag for the re-search key may be transmitted using a click on the tag for the search key itself.

  As shown in FIG. 15, the server 200 receives return data from the terminal device 100 for the tag for the re-search key (step S51). The reception processing unit 204 sends the tag for the re-search key included in the reply data to the tag processing unit 210. The reception processing unit 204 can receive reply data including a process ID. When the reply data includes the process ID, the reception processing unit 204 does not need to receive reply data including the search target image. Therefore, the image recognition unit 206 does not need to extract a plurality of similar images similar to the search target image again, and specifically does not need to extract the image IDs and similarities of the plurality of similar images again. When the reply data is Web data, the reception processing unit 204 may receive a session ID instead of the processing ID.

  The tag processing unit 210 identifies the search target image corresponding to the tag for the re-search key, the already acquired image ID, and the tag of the corresponding registered image, and tag counting processing (tag counting for re-searching) Is performed (step S52). That is, the tag processing unit 210 identifies other tags that are associated with each similar image of a plurality of similar images, except for the tag for the re-search key (the tag for the first re-search key). Then, the tag counting process is performed. Further, the tag processing unit 210 designates a plurality of similar images corresponding to other tags. The plurality of similar images corresponding to other tags are conceptually, for example, a plurality of similar images selected from a plurality of similar images indicated by C2 in FIG. 7A. For example, FIG. It corresponds to a plurality of similar images corresponding to the hatched area indicated by D2 in A).

  Next, some other tags (selected tags) among a plurality of other tags are selected based on the counting result (step S53). When the number of other tags (selected tags) of some of the plurality of other tags acquired through the tag counting process (tag search process for re-searching) is small, the tag processing unit 210 All of a plurality of other tags may be sent to the data creation unit 214. Further, a part of image IDs (selected image IDs) of image IDs of a plurality of similar images designated based on other tags are sent from the tag processing unit 210 to the data creating unit 214.

  Next, related tags corresponding to other tags are acquired from some other tags (selected tags) via the related tag database 230 (step S54). In addition, the data creation unit 214 can acquire related tag hub screen information corresponding to some other tags (selected tags).

  Next, re-search result data including some other tags (selected tags), related tags, and similar images is created, and the re-search result data includes some other tags (selected tags) as re-search result tags. (Step S55). The data creation unit 214 can acquire not only similar images but also hub screen information corresponding to similar images from a part of the image IDs (selected image IDs) via the image database 220.

  Then, the created re-search result data is transmitted to the terminal device 100 (step S53). Thereafter, a step such as step S21 shown in FIG. 12 is performed. That is, the terminal device 100 receives the re-search result data instead of the similar search result data. Then, for example, as shown in FIG. 5B, a re-search result screen on which the selected tag (re-search result tag) is displayed is displayed.

  Thereafter, when the step such as step S42 shown in FIG. 14 is performed, that is, the terminal device 100 transmits the second reply data for the tag for the second re-search key selected from the re-search result tags. If so, step S51 shown in FIG. 15 is performed. However, in step S52, the tag processing unit 210 is another tag associated with each similar image of a plurality of similar images, the first re-search key tag and the second re-search key tag. The other tags except for (conceptually other tags corresponding to the hatched area indicated by E2 in FIG. 7B) are specified, and the tag counting process is performed. Alternatively, in step S52, the tag processing unit 210 is another tag associated with each similar image of a plurality of similar images, except for the second re-search key tag (conceptually illustrated in FIG. The other tag corresponding to the hatched area indicated by E2 ′ in FIG.

  Then, the created re-search result data (second re-search result data) is transmitted to the terminal device 100. Then, for example, as shown in FIG. 6B, a re-search result screen (second re-search result screen) on which the selection tag (second re-search result tag) is displayed is displayed.

  When the terminal device 100 receives similar search result data (or re-search result data) in step S21 in FIG. 12, the list display mode as shown in FIG. 9A is performed (step S22). You may implement tag display modes as shown to (B) (step S31). In this case, whether or not the user has selected a list display mode (for example, a tab indicated by F2 in FIG. 9B) on the similar search result screen (or the re-search result screen) between step S31 and step S32. If it is selected, a similar search result screen (or re-search result screen) as shown in FIG. 9A may be displayed.

  FIG. 17 shows an example of the data structure of image data registered in the image database 220. Image data designated by each image ID is converted into recognition data (feature data) and registered. Therefore, image recognition can be realized by collating the recognition data for the search target image with the recognition data for the registered image.

  As shown in FIG. 17, tags (character information, text data) such as “flowers” and “sakura” are associated with each image. Therefore, the tag of the similar image of the search target image can be read from the image database 220 by specifying the image ID.

  Further, hub screen information (hub page data) is associated with each image (each similar image). The hub screen information is information for presenting a link destination corresponding to each image. That is, in the hub screen information, a plurality of different link destinations are set for each image. For example, the first hub screen information in which a plurality of link destinations (URLs) corresponding to the image ID = 001 is set is associated. Further, the second hub screen information in which a plurality of link destinations corresponding to the image is set is associated with the image with the image ID = 002. As shown in FIG. 17, at least some of the link destinations associated with the image with the image ID = 001 and at least some of the link destinations associated with the image with the image ID = 002 are different. Is set to

  FIG. 18 shows an example of the data structure of related tag data registered in the related tag database 230. As shown in FIG. 18A, tags such as “flower” and “sakura” are related to each other. In other words, you can associate cherry blossoms, dumplings, and roses from flowers, and you can associate flowers, entrance ceremonies, and cherry-blossom viewing from cherry blossoms. As shown in FIGS. 18A and 18B, one or a plurality of corresponding tags (related tags) are associated with each tag of the plurality of tags.

  When the data creation unit 214 receives a plurality of tags such as “flowers” and “cherry blossoms” from the tag processing unit 210, the data creation unit 214 uses the tag “flowers” to retrieve, for example, “dumpling” from the related database 230. , “Horticultural”, “rose”, “sunflower”, “sakura” can be read out. The data creation unit 214 further reads out a plurality of related tags such as “flower”, “cherry blossom viewing”, “spring”, “entrance ceremony”, and “sakura mochi” from the related database 230 using the tag “sakura”. Can do. In this case, since the tag and the related tag overlap, the data creation unit 214 can ignore “spring” and “sakura” as the related tag. Further, the data creation unit 214 creates similar search result data (or re-search result data) by using some of the related tags (for example, “dumpling” and “horticulture”) among the plurality of related tags. be able to.

  FIG. 19 shows an example of the data structure of similar search result data (reply data from the server 200) created by the data creation unit 214. FIG. 19 shows an example of an XML format data structure described in XML (Extensible Markup Language). In FIG. 19, the description of the switching mode as shown in FIG. 4 is omitted, and FIG. 19A corresponds to the list display only mode as shown in FIG. FIG. 19B corresponds to a tag display only mode as shown in FIG.

  The portions indicated by G1 to G2 in FIG. 19A are similar search result data. A plurality of items about similar images are listed and arranged. For example, the items shown in G3 to G4 are the hub screen for presenting the URL information (similar image information in a broad sense) of a similar image as shown in v5 and the link destination for this similar image as shown in G6. URL information (hub screen information in a broad sense). That is, G5 is URL information indicating the storage location of the similar image, and G6 is URL information indicating the storage location of the hub screen. Similarly, the items indicated by G7 to G8 include URL information of other similar images and URL information of other hub screens corresponding to the similar images.

  The portions indicated by G9 to G10 in FIG. 19B are other similar search result data. The portions indicated by G11 to G12 are the blocks of the first tag, and the portions indicated by G13 to G14 are the blocks of the second tag. The first block is the “flower” tag block as shown in G15, the second block is the “clothing” tag block as shown in G16, and follows the “flower” tag in G15. A plurality of items about the “flower” tag are listed and arranged. For example, the items shown in G17 to G18 present related tag information (character information, text data) associated with the tag “flower” as shown in G19, and link destinations for this related tag as shown in G20. URL tag information (related tag hub screen information in a broad sense) of the related tag hub screen. That is, G19 is related tag information, and G20 is URL information indicating the storage destination of the related tag hub screen. Similarly, items shown in G21 to G22 also include related tag information associated with the tag “flower” and URL information of the related tag hub screen corresponding to the related tag. A portion indicated by G23 is a reply button for a tag for a re-search key selected from a plurality of tags.

  Further, following the “clothing” tag of G16, a plurality of items about the “clothing” tag are listed and arranged. Each of these items includes related tag information associated with the tag “clothes” and URL information of the related tag hub screen corresponding to the related tag, as in the case of the tag “flower”.

  FIG. 20A shows an example of a similar search result screen displayed on the display unit 160 of the terminal device 100. C <b> 1 in FIG. 4A is a search target image taken by the terminal device 100, and the server 200 receives transmission data regarding the search target image from the terminal 100. The server 200 performs image recognition of the search target image of C1 and searches for a plurality of similar images. Then, similar search result data as shown in FIG. 19A is created and transmitted to the terminal device 100.

  The terminal device 100 receives the similar search result data in FIG. 19A and displays a similar search result screen as shown in FIG. In this case, a plurality of similar images (including H1 to H4) are displayed, for example, in descending order of similarity.

  When the user selects (clicks) the H1 similar image in FIG. 20A, for example, a hub screen as shown in FIG. 21A is displayed. This hub screen is a central screen for jumping to a plurality of link destinations. In the hub screen of FIG. 21A, a selected similar image is displayed as indicated by I1, and indicated by I2. As described above, a plurality of link destinations corresponding to the similar image are displayed. For example, when the user selects (clicks) one of the plurality of link destinations, the user is connected to an external server that is the link destination, and various types of information are provided to the user. For example, in I2 of FIG. 21A, when a link destination “if you go to see cherry blossoms” is clicked, for example, the site moves to a site where various information about cherry blossom viewing is presented, and the information on that site is transferred to the terminal device 100. Is displayed.

  When the user selects a similar image indicated by H3 in FIG. 20A, for example, a hub screen as shown in FIG. 21B is displayed. In the hub screen of FIG. 21B, the selected similar image is displayed as indicated by I3, and a plurality of link destinations corresponding to the similar image are displayed as indicated by I4. For example, in I4 of FIG. 21 (B), clicking on the link “If you buy floral clothes”, you will be taken to a site where various information such as store information that stocks floral clothes is presented. , Information on the site is displayed on the terminal device 100.

  The similar search result screen displayed according to the present embodiment is not limited to the display mode as shown in FIG. 20A, and various modifications can be made. For example, although tags such as H5 and H6 are not displayed in FIG. 20A, tags may be displayed as in the similar search result screen of FIG. Also, a plurality of similar images displayed as shown in FIG. 20A may be selected by using a tag counting process.

  As described above, in the present embodiment, in the similar search result data in FIG. 19A, hub screen information for presenting a plurality of link destinations corresponding to each similar image is associated with each similar image. Yes. Specifically, first hub screen information for presenting a plurality of link destinations corresponding to the first similar image is associated with the first similar image, and for the second similar image, Second hub screen information for presenting a plurality of link destinations corresponding to the second similar image is associated. For example, the first similar image of G5 in FIG. 19A is associated with first hub screen information (URL information) for presenting a plurality of link destinations corresponding to the similar image as shown in G6. ing. Similarly, second hub screen information (URL information) for presenting a plurality of link destinations corresponding to the second similar image is associated with the second similar image (not shown).

  As a result, on the terminal device 100 side, a hub screen as shown in FIGS. 21A and 21B is displayed. For example, when the user selects a similar image (first similar image) shown in H1 on the similar search result screen in FIG. 20A, a plurality of images corresponding to the similar image are displayed as shown in I2 in FIG. A hub screen (first hub screen) on which the link destination is presented is displayed. When the user selects a similar image (second similar image) shown in H3 on the similar search result screen of FIG. 20A, a plurality of images corresponding to the similar image are displayed as indicated by I4 in FIG. A hub screen (second hub screen) on which the link destination is presented is displayed.

  FIG. 20B illustrates another example of the similar search result screen displayed on the display unit 160 of the terminal device 100. C <b> 1 in FIG. 4A is a search target image taken by the terminal device 100, and the server 200 receives transmission data regarding the search target image from the terminal 100. Then, the server 200 performs image recognition of the search target image of C1, and searches for a plurality of similar images.

  As shown in FIG. 17, a tag is associated with each similar image registered in the image database 220, and the server 200 acquires a plurality of tags associated with the plurality of similar images, and will be described in detail later. Performs tabulation processing for such tags. Then, similar search result data as shown in FIG. 19B is created and transmitted to the terminal device 100.

  The terminal device 100 receives the similar search result data of FIG. 19B and displays a similar search result screen as shown in FIG. In this case, the tag (character information) “flower” having the highest number of appearances in the tabulation process in the server 200 is displayed at the highest position on the similar search result screen as indicated by H5 in FIG. To do. Further, the tag “clothes” having the next highest number of appearances is displayed at a lower position of “flower” as indicated by H6. Further, as shown in H7 and H8, the related tag acquired from the related tag database 230 using the tag “flower” is displayed as a hyperlink (item) next to the tag “flower”.

  Similarly, as indicated by H9 and H10, other related tags acquired from the related tag database 230 using the tag “clothes” are displayed as hyperlinks next to the tag “clothes”.

  For example, when the user selects (clicks) the related tag of H7 in FIG. 20A, a related tag hub screen (not shown) similar to FIG. 21A or 21B is displayed. The related tab hub screen is a central screen for jumping to a plurality of link destinations, and the selected related tag is displayed at a position corresponding to I1 of the hub screen in FIG. A plurality of link destinations corresponding to the related tag are displayed at corresponding positions. For example, when the user selects (clicks) one of the plurality of link destinations, the user is connected to an external server that is the link destination, and various types of information are provided to the user. For example, when a link destination of “If you buy dumplings” is clicked, for example, the site shifts to a site where various information about dumplings is presented, and information on the sites is displayed on the terminal device 100.

  Note that the similar search result screen displayed according to this embodiment is not limited to the display mode as shown in FIG. 20B, and various modifications can be made. For example, in FIG. 20B, related tags are displayed as shown in H7, H8, H9, and H10, but the related tags may not be displayed as in the similar search result screen of FIG. .

  As described above, in the present embodiment, in the similar search result data of FIG. 19B, related tag hub screen information for presenting a plurality of link destinations corresponding to each related tag is associated with each related tag. ing. Specifically, first related tag hub screen information for presenting a plurality of link destinations corresponding to the first related tag is associated with the first related tag, and for the second related tag, Second related tag hub screen information for presenting a plurality of link destinations corresponding to the second related tag is associated. For example, as shown in G20, first hub screen information (URL information) for presenting a plurality of link destinations corresponding to the related tag is associated with the first related tag of G19 in FIG. 19B. ing. Similarly, second hub screen information (URL information) for presenting a plurality of link destinations corresponding to the second related tag is associated with a second related tag (not shown).

  In the present embodiment, the similar search result data in FIG. 19B created in the server 200 includes a plurality of tags such as “flowers” and “clothes” acquired from similar images, as shown in G15 and G16. (Search result tag) is included. These plural tags are sorted according to the number of appearances, for example. Accordingly, on the similar search result screen on the terminal device 100 side in FIG. 20B, a plurality of tags acquired from a plurality of similar images similar to the search target image are sorted and displayed. Specifically, as shown in H5 and H6, tags such as “flower” and “clothes” are displayed in descending order of the number of appearances. Tags may be displayed in an order different from the number of appearances.

  In the similar search result data in FIG. 19B, information of one or a plurality of related tags corresponding to a plurality of related tags is associated with each tag. For example, as shown in G19, a “flower” tag is associated with a related tag corresponding to the tag.

  Accordingly, in the similar search result screen on the terminal device 100 side in FIG. 20B, corresponding one or a plurality of related tags (hyperlinks) among a plurality of related tags are displayed in association with each tag. The For example, for the “flower” tag shown in H5, a plurality of related tags corresponding to the “flower” tag are displayed in association with each other, as shown in H7 and H8. Further, as shown in H9 and H10, a plurality of related tags corresponding to the “clothing” tag are displayed in association with the “clothing” tag shown in H6.

  As described above, in the present embodiment, the search result of the similar image using the photographed search target image as a key is information on the searched similar image as shown in the data structure of the similar search result in FIG. Are grouped by tags shown in G15 and G16 representing each class. For example, the portions displayed as “flowers” and “clothes” in H5 and H6 in FIG. 20B correspond to tags representing classes. The related tags shown in H7 to H10 are selected in the data creation unit 214 so as to be different from the tags displayed on the left side thereof. As described above, in the present embodiment, the similar images ordered by the similarity are converted into the order of the tags and displayed. When each related tag is selected, a related tag hub screen on which a different link destination is displayed for each related tag is displayed.

  The user can select a radio button (H11) for a tag on the similar search result screen in FIG. 20B or a hyperlink for a related tag. When the reply button indicated by H12 is selected in a state where the tag (for example, “flower”) on the similar search result screen is selected by the radio button, reply data regarding the tag for the re-search key is transmitted to the server 200. Thereafter, a re-search result screen is displayed. When the related tag hyperlink in the similar search result screen of FIG. 20B is selected, a related tag hub screen is displayed. Instead of selecting the radio button (H11), the re-search result screen may be displayed by selecting the tag itself of the similar search result screen in FIG.

  For example, as a first comparative example of the present embodiment, when the subject is “flower”, the user first selects the item “flower” on the screen of the terminal device 100, and then photographs the flower, A technique for executing image retrieval is conceivable.

  However, in the method of the first comparative example, in a situation where the subject is missed if it is “now”, the subject is missed and information about the subject cannot be obtained after all. There is a risk that.

  In this regard, in the method of the present embodiment, the subject is first photographed, and a method of retrieving a plurality of similar images from the photographed image is adopted, so that the subject cannot be photographed and the information cannot be obtained. The situation can be prevented. Further, it is not necessary to select an item such as “flower” before photographing the subject, and the user can obtain information on the subject with a simple operation. Of course, in order to facilitate the narrowing down of information, it is possible to employ a technique in which a subject is photographed after the user selects a desired genre and title, and an image search according to the technique of this embodiment is executed.

  Further, as a second comparative example of the present embodiment, a hub screen in which a link destination of the same theme is displayed both when “dumpling” is selected in FIG. 20B and when “horticulture” is selected. The method of displaying can be considered. For example, if “Dango” is selected, such as a link to a store related to “Dango” or “horticulture”, a mail-order link, a map information link, or a detailed information link, "" Is selected, the same format hub screen is displayed.

  However, this second comparative method is suitable for a system that aims to uniquely identify information from a captured image, but it does not obscure image retrieval as described in FIGS. It is not suitable for a system intended to use and spread the taste and hobbies of users.

  In this regard, according to the method of the present embodiment, the related tag hub screen on which the link destinations of different themes and types are displayed is displayed according to the similar image (hyperlink, item) selected by the user. For example, if the selected related tag is “Dango”, the link that displays the link destination of the dumpling purchase location, the link to the hit song related to the dumpling, or the link destination of the dumpling dictionary site Display the tag hub screen. On the other hand, if the selected related tag is “horticulture”, a related tag hub screen on which the link destination of the fruit growing method information and the link destination of the vegetable growing information is displayed. In this way, by displaying the link destinations of various setting patterns on the related tag hub screen, the ambiguity of the image search can be used in reverse, helping the user to find information that he wanted to find, etc. Can be realized.

  The data structure of the similar search result data and the display mode of the similar search result screen (or the re-search result screen) and the hub screen are shown in FIGS. 4 (B), 5 (B), 9 (A), and 9. (B), it is not limited to what was demonstrated in FIG. 19 (A)-FIG. 21 (B), Various deformation | transformation implementation is possible.

  For example, FIG. 22 shows another data structure example of the similar search result data. In FIG. 22, J1 and J2 are set with character information of tags of “flower” and “clothes”, and J3 and J5 are set with character information of titles of related tab hub screens. Further, the link destination when the title of J3 is selected is set in J4, and the link destination when the title of J5 is selected is set in J6. That is, in the data structure example of FIG. 19B, the URL information of the related tag hub screen is set, and when the terminal device 100 accesses the URL, the related hub screen is displayed. On the other hand, in FIG. 22, character information, screen configuration, and the like constituting the related tag hub screen are created by the data creation unit 214 and transmitted to the terminal device 100. As described above, as the data structure of the similar search result data, various forms can be adopted according to the system configuration and specifications of the image search.

4). Counting Process Next, a detailed example of the tag counting process will be described with reference to the flowchart of FIG.

  First, N = 1 is set (step S61). And the tag group linked | related with the Nth similar image obtained by image recognition is acquired (step S62). Specifically, the image ID of the Nth similar image is designated in the image database 220 of FIG. 17, and a tag group associated with the similar image is acquired. For example, in FIG. 24A, when the image ID of the similar image K1 is designated, a tag group including “flower” and “sakura” is acquired. In this case, for example, one or more tags are associated with one image ID, and duplicate tags are not associated with each other. Preferably, a plurality of tags are associated with one image ID.

  Next, M = 1 is set (step S63). Then, the Mth tag in the tag group associated with the Nth similar image is acquired (step S64). And the count value of the appearance number about the acquired tag is incremented (step S65). For example, in FIG. 24A, when the tag “flower” is acquired from the tag group consisting of “flower” and “sakura”, the count value of the number of appearances of the tag “flower” is incremented by one.

  Instead of simply counting up the numerical value, for example, when a tag is acquired, an image ID is added to the array corresponding to the tag, and the number of elements in the array is used as the count value. Good.

In addition, when the tag counting process is performed using the weighting coefficient obtained based on the similarity of the similar images, the similarity of the Nth similar image in step S62 is acquired from the image recognition unit 206, for example. Then, based on the obtained similarity, a weighting coefficient corresponding to the similarity is obtained by performing a predetermined calculation or using a conversion table having the similarity as an input and a weighting coefficient as an output. For example, the higher the similarity is, the higher the weighting coefficient is obtained. In step S65, the count value may be incremented by performing arithmetic processing such as multiplying the obtained weighting coefficient by the appearance number of the tag. In this way, it is possible to realize tag counting processing that reflects the similarity of similar images.
Next, it is determined whether or not the aggregation for all tags of the tag group has been completed (step S66). If not completed, M is incremented by 1 (step S67), and the process returns to step S64. Get the next tag. For example, the tag of “clothes” next to “flower” is acquired.

  When the aggregation for all the tags in the tag group is completed, it is determined whether or not the tag group has been acquired for all the similar images searched (a predetermined number of similar images with higher similarity) (step S68). If not completed, N is incremented by 1 (step S49), and the process returns to step S62. On the other hand, if completed, a sorting process is executed based on the number of appearances of each tag (step S70).

  For example, in FIG. 24B, the count values of the number of appearances of “flower”, “clothes”, and “cherry blossom” are 5, 4, and 3, respectively. Therefore, the tags are rearranged so that “flower” is the highest rank, followed by “clothes” and “cherry blossom”. Accordingly, tag display as shown at H5 and H6 in FIG. Thereby, the order of the images ordered by the similarity of the images by the image recognition can be converted into the order of the tags ordered by the number of appearances of the tags.

  For example, in order to know the name of “flower” in front of the user, the user photographs the “flower” and transmits the photographed image to the server 200. At this time, as a result of the image search, for example, in the top 20 similar images, instead of the image of the “flower” itself, an image of a handkerchief with a pattern similar to the “flower” or a person holding the flower May come to the top.

  When a similar search result screen showing the order of the images is displayed to the user who wants to know the name of “flower”, instead of the “flower” image itself, If an image of a person holding a flower comes to the top, it takes time for the user to reach the desired “flower” image, and many unexpected images will be accessed.

  On the other hand, even in such a case, it is generally expected that many of the top 20 similar images are images associated with the tag “flower”. Therefore, by sorting the tags using the number of appearances of the tags, the user can access a desired image (and its information) with less effort.

  By the way, in FIG. 24B, the specific count value of the number of appearances of the tag “sakura” compared to the tag “flower” is the number of appearances of abstract tags such as “flower” and “clothes”. In some cases, the tag “sakura” is not included in the search result tag of the similar search result image. Depending on the size of the similar search result image, the tag “sakura” may not be included in the search result tag of the similar search result image. In such a case, by selecting a tag for the re-search key from among the search result tags, or by repeatedly selecting a tag for the re-search key, the user can select a specific item such as “Sakura”. You can get to the tag. In the following, the tag counting process for a plurality of similar images specified by the tag for the re-search key will be described.

  The N similar images conceptually correspond to the rectangular region indicated by C2 in FIG. 7A, and P (P <N) designated using the tag for the first re-search key. Conceptually corresponds to a hatched area indicated by D2 in FIG. In the tag counting process for the P similar images, whether or not the tag group associated with the Nth similar image includes the tag for the first re-search key between step S62 and step S63 in FIG. Determine whether. If the tag group associated with the Nth similar image does not include the tag for the first re-search key, step S69 is performed without performing step S63. That is, for example, when the tag for the first re-search key is “flower”, similar images that do not include the tag “flower” are excluded from the target of the tag counting process.

  When the tag group associated with the Nth similar image includes a tag for the first re-search key, the M-th tag is a tag for the first re-search key between step S64 and step S65. It is determined whether or not there is. If the Mth tag is a tag for the first re-search key, step S67 is performed without performing step S65. That is, for example, when the tag for the first re-search key is “flower”, step S65 is performed only for other tags excluding the tag “flower”.

  The tag counting process for a plurality of similar images that are designated using the tag for the second re-search key and conceptually correspond to the hatched area indicated by E2 in FIG. It is determined between S62 and S63 whether or not the tag group associated with the Nth similar image includes a tag for the first re-search key and a tag for the second re-search key. Further, when the tag group associated with the Nth similar image includes the tag for the first re-search key and the tag for the second re-search key, between the step S64 and the step S65, the Mth It is determined whether the tag is a tag for the first re-search key or a tag for the second re-search key. If the M-th tag is a tag for the first re-search key or a tag for the second re-search key, step S67 is performed without performing step S65. That is, for example, when the tag for the first re-search key is “flower” and the tag for the second re-search key is “sakura”, other tags except for the tag “flower” and the tag “sakura” Step S65 is performed only for the tag.

  The tag totaling process for a plurality of similar images that are designated using the tag for the second re-search key and conceptually correspond to the hatched area indicated by E2 ′ in FIG. It is determined between step S62 and step S63 whether or not the tag group associated with the Nth similar image includes a tag for the second re-search key. Further, when the tag group associated with the Nth similar image includes the tag for the second re-search key, the M-th tag is for the second re-search key between step S64 and step S65. It is determined whether or not it is a tag. If the Mth tag is a tag for the second re-search key, step S67 is performed without performing step S65. That is, for example, when the tag for the second re-search key is “sakura”, step S65 is performed only for other tags excluding the tag “sakura”.

5). Image Recognition Next, a detailed example of the image recognition method will be described. As an image recognition method for searching for similar images, for example, there are a recognition method using feature points and a recognition method using features of the entire image.

5.1 Recognition Method Using Feature Points 1.1 Extraction of Feature Points First, blur processing is performed on the original image step by step to create a blur image. Then, the difference between the blur images is taken to create a difference image, and the pixel having the extreme value in the difference image (the coordinate whose value is larger or smaller than the surroundings) is used as the feature point. Usually, a plurality of feature points are extracted from one image.

1.2 Calculation of Feature Quantity The feature quantity of each feature point is expressed by a 128-dimensional vector, which is called a feature vector. This feature vector is obtained by adding a differential value of a blur image in a certain range (hereinafter referred to as a pickup range) centered on the feature point obtained in 1.1 above to the corresponding dimension.

  The difference value to be added is weighted so that the value becomes larger as it is closer to the feature point. The index of the dimension to be added is obtained from the coordinates obtained by rounding the coordinates of each pixel in the pickup range (hereinafter referred to as pickup coordinates) to 4 × 4 and the orientation obtained by rounding the differential orientation of the pickup coordinates in eight directions. And this feature-value is calculated | required about all the several feature points calculated | required by said 1.1.

1.3 Image Matching The above processing 1.1 and 1.2 is performed on the registered image and the query image (search target image) to extract feature points and feature amounts. Then, the feature amount closest to the feature amount of the feature point extracted from the question image is searched from the registered image. Here, the closeness of the feature amount is expressed by the distance of the feature vector.

  In this way, the feature points of the registered image corresponding to the feature points extracted from the search target image are obtained, and the registered image having the most corresponding feature points is returned as a correct image (similar image). When a plurality of images are returned as correct images, the top J images (J is an integer of 2 or more) are returned as correct images (similar images) in order from the image having many corresponding feature points.

5.2 Recognition Method Using Features of Entire Image 2.1 Calculation of Feature Amount First, the entire image is converted into a gray scale and converted to a luminance image. The luminance image is reduced to 16 × 16 pixels, and for example, a 256-dimensional vector is used as a composition feature.

  The hue of the entire pixel is divided into 10 sections, for example, and the histogram is used as a color feature of the entire image and expressed as a 10-dimensional vector. Further, the entire image is divided into 512 colors, and the top 30 colors of the occupied area are used as representative color features.

2.2 Image matching The above-described 2.1 processing is performed on each of the registered image and the question image to calculate a feature amount. Then, a registered image having a feature closest to the question image is returned as a correct image (similar image). When returning a plurality of images as correct images, the top J images are returned in order from the image closest to the question image. The closeness of the feature is expressed by the distance of each feature vector. Also, the weighting is changed depending on whether the composition feature or the color feature is important.

6). Modified Examples Next, various modified examples of the present embodiment will be described.

  In this embodiment, the similar search result data created by the server 200 is transmitted to the terminal device 100 that has photographed the subject, and the similar search result screen, hub screen, for example, as shown in FIGS. 20 (A) to 21 (B). Although the case of displaying (including the related tag hub screen) has been described, the present embodiment is not limited to this. For example, the created similar search result data (or re-search result data) may be transmitted to another external server or the like so that it can be used to create various information and screens.

  For example, FIG. 25 shows an example in which similar search result data is transmitted to a blog server to create a blog screen. A captured image (search target image) of the user is displayed at L1 in FIG. In L2, a tag included in the similar search result data is displayed. That is, tags such as “flowers” and “cherry blossoms” extracted from the plurality of similar images of the photographed image by the method of this embodiment are displayed. Also, the shooting date and time of the shot image, the name of the photographer, etc. are displayed. In addition, a plurality of related tags associated with a plurality of tags are displayed in L3.

  Further, in L4 of FIG. 25, the blog server creates a click screen for a link destination for mail order of flower seeds based on the tag “flower”. Further, at L5, the link destination of the related tag hub screen of FIG. 20B is displayed.

  In addition, the blog screen is displayed so that the re-search result tag is displayed when the blog screen viewer selects one tag (tag for the re-search key) among the plurality of tags indicated by L2 in FIG. You may control. Similarly, the blog screen may be controlled so that the related tag corresponding to the re-search result tag is displayed.

  As described above, the similar search result data (or re-search result data) created according to the present embodiment is transferred not only to the terminal device 100 that captured the image but also to a server or the like of another system in various modes. It is possible to use. The data creation unit 214 may create similar search result data by adding other information related to shooting or transmission in addition to similar image and tag information. For example, information such as the shooting date / time and the transmission date / time detected on the terminal 100 side may be added.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described together with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term anywhere in the specification or the drawings. Further, the configurations and operations of the terminal device and the server are not limited to those described in the present embodiment, and various modifications can be made.

10 network, 100 terminal device, 102 processing unit, 104 image acquisition unit,
106 transmission processing unit, 108 reception processing unit, 110 display control unit, 120 storage unit,
122 transmission data storage unit, 124 similar search result data storage unit,
126 reply data storage unit, 128 re-search result data storage unit, 130 information storage medium,
138 communication unit, 140 operation unit, 150 imaging unit, 160 display unit,
200 server, 202 processing unit, 204 reception processing unit, 206 image recognition unit,
208 data generation unit for recognition, 210 tag processing unit, 212 tag totaling unit,
214 data creation unit, 218 transmission processing unit, 219 storage unit,
220 image database, 221 transmission data storage unit, 222 similar image ID storage unit,
223 tag processing result data storage unit, 224 similarity search result data storage unit,
225 reply data storage unit, 226 re-search result data storage unit,
230 related tag database, 238 communication unit, 240 operation unit

Claims (20)

A server that performs a search process for similar images of search target images,
A reception processing unit for performing processing for receiving transmission data about the search target image via a network;
An image recognition unit that performs image recognition processing of the received search target image and searches for a plurality of similar images similar to the search target image from registered images registered in an image database;
A tag processing unit for acquiring a tag associated with each similar image of the plurality of similar images searched;
A data creation unit for creating similar search result data including a plurality of tags acquired from the plurality of similar images as search result tags;
A transmission processing unit that performs processing of transmitting the similar search result data via the network,
The reception processing unit
Performing a process of receiving first reply data about the tag for the first re-search key selected from the search result tags via the network;
The tag processing unit
Based on the first reply data, obtain other tags associated with each similar image of a plurality of similar images, excluding the tag for the first re-search key,
The data creation unit
A server that creates first re-search result data including a plurality of other tags acquired from the plurality of similar images based on the first reply data as a first re-search result tag. In claim 1,
The tag processing unit
The server, wherein the plurality of other tags are acquired from a plurality of similar images including a tag for the first re-search key among the plurality of similar images searched. In claim 1 or 2,
The transmission processing unit
Performing a process of transmitting the first re-search result data via the network;
The reception processing unit
Performing a process of receiving second reply data about the tag for the second re-search key selected from the first re-search result tags via the network;
The tag processing unit
Based on the second reply data, another tag associated with each similar image of a plurality of similar images, the tag for the first re-search key and the tag for the second re-search key Get other tags except
The data creation unit
A server that creates second re-search result data including a plurality of other tags acquired from the plurality of similar images as second re-search result tags based on the second reply data. In claim 1 or 2,
The transmission processing unit
Performing a process of transmitting the first re-search result data via the network;
The reception processing unit
Performing a process of receiving second reply data about the tag for the second re-search key selected from the first re-search result tags via the network;
The tag processing unit
Based on the second reply data, another tag associated with each similar image of a plurality of similar images is acquired except for the tag for the second re-search key,
The data creation unit
A server that creates second re-search result data including a plurality of other tags acquired from the plurality of similar images as second re-search result tags based on the second reply data. In any one of Claims 1 thru | or 4,
The tag processing unit
Performing a sorting process of the plurality of tags acquired from the plurality of similar images;
The data creation unit
A similar search result data including the plurality of sorted tags as the search result tag is created. In any one of Claims 1 thru | or 4,
The tag processing unit
The server, wherein the number of appearances of each of the plurality of tags acquired from the plurality of similar images is totaled, and the plurality of tags are sorted based on the totaling result. In claim 6,
The tag processing unit
A server characterized in that a weighting coefficient is obtained based on the similarity of the plurality of similar images with respect to the search target image, and a tag counting process is performed based on the obtained weighting coefficient. In any one of Claims 1 thru | or 7,
The data creation unit
A server that creates similar search result data further including the plurality of similar images. In claim 8,
The data creation unit
The first mode or the second mode is set so that the similar search result data includes the plurality of similar images in the first mode and includes the plurality of tags as the search result tags in the second mode. A server characterized by creating switchable similarity search result data. In claim 8 or 9,
The data creation unit
First hub screen information for presenting a plurality of link destinations corresponding to the first similar image is associated with the first similar image of the plurality of similar images, and the plurality of similar images For the second similar image of the images, similar search result data associated with second hub screen information for presenting a plurality of link destinations corresponding to the second similar image is created. A server characterized by In any one of Claims 1 thru | or 10.
The data creation unit
A server that creates similar search result data further including a plurality of related tags acquired from a related database based on the plurality of tags acquired from the plurality of similar images. An image acquisition unit for acquiring search target images;
A transmission processing unit that performs processing of transmitting transmission data about the acquired search target image to a server via a network;
A reception processing unit that performs processing for receiving similar search result data of the search target image via the network;
A display control unit that performs display control of the display unit based on the received similar search result data,
The display control unit
Performing a control to display a similar search result screen on which a plurality of tags acquired from a plurality of similar images similar to the search target image are displayed as search result tags on the display unit;
The transmission processing unit
Performing a process of transmitting reply data about the tag for the re-search key selected from the search result tags via the network;
The receiving unit performs processing for receiving re-search result data created based on the reply data via the network,
The display control unit
Based on the received re-search result data, a plurality of other tags acquired from a plurality of similar images, excluding the tag for the re-search key, are displayed as re-search result tags. The terminal device is characterized by performing control to display the re-search result screen on the display unit. In claim 12,
The display control unit
A terminal device that controls to display a similar search result screen on which the plurality of similar images are further displayed on the display unit. In claim 13,
The display control unit
Control for displaying on the display unit a similar search result screen capable of switching between a first mode in which the plurality of similar images are displayed or a second mode in which the plurality of tags are displayed as the search result tags. The terminal device characterized by performing. In any of claims 12 to 14,
The display control unit
The terminal device characterized by performing control which displays on the said display part the similar search result screen by which the said some tag was sorted according to the appearance number of each tag. In any of claims 12 to 15,
The display control unit
Performing a control to display a similar search result screen on which a plurality of related tags acquired from a related tag database based on the plurality of tags acquired from the plurality of similar images is further displayed on the display unit. Terminal device to do. An image acquisition unit for acquiring search target images;
A transmission processing unit that performs processing of transmitting transmission data about the acquired search target image to a server via a network;
A reception processing unit that performs processing for receiving similar search result data of the search target image via the network;
As a display control unit that performs display control of the display unit based on the received similar search result data,
Make the computer work,
The display control unit
Performing a control to display a similar search result screen on which a plurality of tags acquired from a plurality of similar images similar to the search target image are displayed as search result tags on the display unit;
The transmission processing unit
Performing a process of transmitting reply data about the tag for the re-search key selected from the search result tags via the network;
The receiving unit performs processing for receiving re-search result data created based on the reply data via the network,
The display control unit
Based on the received re-search result data, a plurality of other tags acquired from a plurality of similar images, excluding the tag for the re-search key, are displayed as re-search result tags. A program for performing control to display a re-search result screen on the display unit.   A computer-readable information storage medium, wherein the program according to claim 17 is stored. An image search method for performing a search process for a similar image of a search target image,
Receiving transmission data about the search target image via a network;
The image recognition process of the received search target image is performed, and a plurality of similar images similar to the search target image are searched from registered images registered in an image database,
Obtaining a tag associated with each similar image of the plurality of similar images searched;
Creating similar search result data including a plurality of tags acquired from the plurality of similar images as search result tags;
Sending the similar search result data via the network;
Based on a tag for a re-search key selected from among the search result tags, other tags associated with each similar image of a plurality of similar images, excluding the tag for the first re-search key Get other tags,
A re-search result data including a plurality of other tags acquired from the plurality of similar images as re-search result tags based on the re-search key tag. In claim 19,
Receiving the similar search result data via the network;
Based on the received similar search result data, control to display a similar search result screen on which the search result tag is displayed on the display unit,
Sending the tag for the re-search key via the network;
Receiving the re-search result data via the network;
An image search method comprising: controlling a display unit to display a re-search result screen on which the re-search result tag is displayed based on the received re-search result data.

Images