JP2015032248A - Image search device, system and program for searching for data - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce accuracy deterioration of an image search because a feature point of a key image is wrongly associated with a feature point of a retrieved image having nothing to do with the feature point of the key image when searching for a similarity image similar to a key image included in a query image from retrieved images.SOLUTION: An image search server is configured to: associate a feature point of a query image with a feature point of a retrieved image on the basis of a degree of similarity of an amount of feature about the feature points thereof; identify the retrieved image in which a ratio of the feature point associated with the feature point of the query image is more than a threshold value as a candidate image; identify an area of a circumscribed rectangle including the feature point associated with the candidate image as to each of the candidate images inside as a key image; associate a feature point of the key image with the feature point of the candidate image on the basis of a degree of similarity of feature about the feature points thereof; and identify the candidate image in which a ratio of the feature point associated with the feature point of the key image is more than the threshold value as a similarity image.

Description

  The present invention relates to an image search device, a data search system, and a program.

  There is an image retrieval technique for retrieving an image similar to a certain image from a plurality of registered images. For example, Patent Document 1 proposes an image search system that searches for an image similar to a query image from a plurality of search target images.

  In the image search system proposed in Patent Document 1, first, for each of a plurality of search target images, pairing of the feature points of the search target image and the query image is performed. Subsequently, in each of the search target image and the query image, a comparison area is specified based on the paired feature points. Subsequently, the feature points of the comparison area of the search target image and the comparison area of the query image are paired. Then, based on the number of feature points paired in the comparison area, a search target image similar to the query image is output as a search result.

JP 2010-204908 A

  An object of the present invention is to specify a search target image whose key image feature points are irrelevant when a similar image similar to a key image that is a search target image included in a query image is specified from a plurality of search target images. It is to reduce a decrease in the accuracy of image search caused by being erroneously associated with a feature point.

  The present invention relates to each of a plurality of search target images, a first feature amount data acquisition unit that acquires first feature amount data indicating each feature amount of a plurality of feature points of the search target image, and a query image Second feature amount data acquisition means for acquiring second feature amount data indicating each feature amount of the plurality of feature points, a feature amount indicated by the first feature amount data, and the second feature amount A first association that associates one or more feature points of the plurality of feature points of the query image with one of the feature points of the plurality of search target images based on a result of comparison with the feature amount indicated by the data. And one or more searched images satisfying a condition in which the number of feature points associated with the feature points of the query image by the first association unit among the plurality of searched images satisfies a predetermined condition. Candidate image specification to specify as candidate images of similar images A plurality of query images based on a comparison result between the feature amount indicated by the first feature amount data of the step and the feature amount of the one or more candidate images and the feature amount indicated by the second feature amount data. Second associating means for associating one or more feature points of the one or more candidate images with one or more feature points of the feature points, and the second association among the one or more candidate images. Proposed as a first aspect is an image search apparatus comprising similar image specifying means for specifying candidate images satisfying a predetermined number of feature points associated with feature points of the query image by means as a similar image To do.

  In the image search device according to the first aspect, the present invention relates to each of the one or more candidate images, the query image associated with the feature point of the candidate image by the first association unit. Key image specifying means for specifying, as a key image, one or more areas of the query image that include the feature points of the image in a predetermined condition, wherein the second association means relates to each of the one or more key images. , Based on the result of comparison between the feature quantity indicated by the first feature quantity data of the feature point of the one or more candidate images and the feature quantity indicated by the second feature quantity data of the feature point of the key image, A configuration in which one or more feature points of the one or more candidate images are associated with one or more feature points of the plurality of feature points of the key image is proposed as a second aspect.

  In the image search device according to the first aspect of the present invention, the second association unit relates to each of the one or more key images, and is one or more of a plurality of feature points of the key image. A configuration in which the key image specifying unit associates any one of the feature points of the candidate image used for specifying the key image with each of the feature points is proposed as a third aspect.

  Further, the present invention provides the image search device according to any one of the first to third aspects, wherein each of the plurality of searched images is included in any of a plurality of pages, and the first feature amount data The acquisition unit acquires the first feature amount data grouped for each page including the search target image, and the candidate image specifying unit performs the feature points of the query image by the first association unit. Identifying the number of feature points associated with each group to which the first feature value data of the feature points belongs, and identifying the candidate image based on the number of feature points for each of the identified groups. This configuration is proposed as a fourth aspect.

  Further, the present invention provides the image search device according to any one of the first to fourth aspects, wherein each of the plurality of searched images is included in any of a plurality of pages, and the first feature amount data The acquisition unit acquires the first feature amount data grouped for each page including the search target image, and the similar image specifying unit uses the second association unit to perform feature points of the query image. Specifying the number of feature points associated with each group to which the first feature value data of the feature points belongs, and specifying the similar image based on the number of feature points for each specified group. This configuration is proposed as a fifth aspect.

  According to the present invention, in the image search device according to any one of the first to fifth aspects, a feature point of the query image required under conditions used when the similar image specifying unit specifies a similar image. The number of feature points of the image to be searched that is associated with the feature point of the image to be searched that is associated with the feature point of the query image that is requested under the conditions used when the candidate image specifying unit specifies the candidate image. A configuration that is larger than the number of is proposed as a sixth aspect.

  In the image search device according to any one of the first to fifth aspects, the present invention provides an image data acquisition unit that acquires image data representing a query image, and a plurality of query images represented by the image data. A feature amount data generating unit that specifies feature points and generates feature amount data indicating the feature amounts of each of the plurality of feature points, and the second feature amount data acquisition unit includes the feature amount data generation unit. As a seventh aspect, a configuration in which the feature amount data generated by the above is acquired as the second feature amount data is proposed.

  Further, the present invention includes a terminal device and an image search device, and the terminal device acquires image data acquisition means for acquiring image data representing a query image, and transmission means for transmitting the image data to the image search device. Receiving means for receiving data transmitted from the image search apparatus as a response to the image data transmitted by the transmission means, wherein the image search apparatus relates to each of a plurality of searched images. Feature amount data acquisition means for acquiring first feature amount data indicating each feature amount of a plurality of feature points of the search image, reception means for receiving the image data from the terminal device, and the image data Feature quantity data generating means for specifying a plurality of feature points of a query image and generating second feature quantity data indicating the feature quantities of each of the plurality of feature points; Based on the result of the comparison between the feature quantity indicated by the collection amount data and the feature quantity indicated by the second feature quantity data, the plurality of search target points are included in one or more feature points of the plurality of feature points of the query image. A first associating means for associating any one of the feature points of the image; and the number of feature points associated with the feature points of the query image by the first associating means among the plurality of searched images. Candidate image specifying means for specifying one or more searched images that satisfy a predetermined condition as candidate images of similar images, and feature amounts indicated by the first feature amount data of feature points of the one or more candidate images; Based on the result of comparison with the feature amount indicated by the second feature amount data, one or more feature points of the plurality of feature points of the query image may be any one of the feature points of the one or more candidate images. Second associating means for associating Among the one or more candidate images, a similar image that identifies candidate images that satisfy a predetermined condition for the number of feature points associated with the feature points of the query image by the second association unit as a similar image A data search system comprising specifying means and transmitting means for transmitting data corresponding to the similar image specified by the similar image specifying means to the terminal device is proposed as an eighth aspect.

  The present invention further includes a terminal device and an image search device, wherein the terminal device includes image data acquisition means for acquiring image data representing a query image, and a plurality of feature points of the query image represented by the image data. A feature amount data generation unit that identifies and generates feature amount data indicating the feature amount of each of the plurality of feature points, a transmission unit that transmits the feature amount data to the image search device, and a transmission unit that transmits the feature amount data. Receiving means for receiving data transmitted from the image search device as a response to the feature amount data, the image search device relating to each of the plurality of search target images with a plurality of features of the search target image. Feature amount data acquisition means for acquiring feature amount data indicating each feature amount of a point as first feature amount data, and the feature amount data transmitted from the terminal device Based on the result of comparison between the receiving means for receiving the second feature quantity data and the feature quantity indicated by the first feature quantity data and the feature quantity indicated by the second feature quantity data, First association means for associating one or more feature points of the plurality of feature points with one or more feature points of the plurality of search target images, and among the plurality of search target images, the first association unit The candidate image specifying means for specifying one or more searched images as the candidate images of similar images, wherein the number of feature points associated with the feature points of the query image satisfies a predetermined condition, and the one or more candidates One or more of the plurality of feature points of the query image based on the result of comparison between the feature amount indicated by the first feature amount data and the feature amount indicated by the second feature amount data. One or more candidates for the feature point A second associating means for associating one of the feature points of the image, and the number of feature points associated with the feature points of the query image by the second associating means among the one or more candidate images. A data search comprising: a similar image specifying unit that specifies a candidate image that satisfies a predetermined condition as a similar image; and a transmission unit that transmits data corresponding to the similar image specified by the similar image specifying unit to the terminal device The system is proposed as a ninth aspect.

  In addition, the present invention relates to a process for acquiring first feature amount data indicating each feature amount of a plurality of feature points of the search target image for each of a plurality of search target images, and a plurality of query images. A process of obtaining second feature amount data indicating the feature amount of each of the feature points, a comparison between the feature amount indicated by the first feature amount data and the feature amount indicated by the second feature amount data. Based on the result, a process of associating one or more feature points of the plurality of feature points of the query image with one of the feature points of the plurality of search target images, and the query among the plurality of search target images. A process for specifying one or more search target images that satisfy a predetermined number of feature points associated with image feature points as candidate images of similar images, and the first of the feature points of the one or more candidate images. The feature amount indicated by the feature amount data of 1; A process of associating one or more feature points of the plurality of feature points of the query image with one of the feature points of the one or more candidate images based on a result of comparison with the feature amount indicated by the feature amount data of 2 And a program for executing, as a similar image, a candidate image that satisfies a predetermined condition for the number of feature points associated with the feature points of the query image among the one or more candidate images. Proposed as a tenth aspect.

  According to said 1st aspect, compared with the case where it is based on the image search device which does not specify a candidate image, a similar image can be searched with high precision.

  According to said 2nd aspect, a similar image can be searched with high precision compared with the case by the image search apparatus which does not specify a key image.

  According to the third aspect, similar images can be identified at a higher speed than in the case of using an image search device that associates one of the feature points of all candidate images with the feature point of the key image. .

  According to the fourth aspect, a similar image can be searched with higher accuracy than in the case of an image search device that does not use page information in specifying a candidate image.

  According to said 5th aspect, compared with the case where the image search apparatus which does not use the information of a page in specification of a similar image can search for a similar image with high precision.

  According to said 6th aspect, the risk that the similar image which should be searched is excluded in the specific step of a candidate image is reduced.

  According to said 7th aspect, generation | occurrence | production of the feature-value data of the query image in the apparatus which requests | requires the search of a similar image with respect to an image search apparatus is not required.

  According to said 8th aspect, the user of a terminal device can receive the data according to the similar image of a query image by transmitting a query image to an image search device using a terminal device.

  According to said 9th aspect, the user of a terminal device receives the data according to the similar image of a query image by transmitting the feature-value data of the query image produced | generated in the terminal device to an image search device. Can do.

  According to the tenth aspect described above, an image search device that can search for a similar image with higher accuracy than the case of using an image search device that does not specify a candidate image is realized by a computer.

1 is a diagram illustrating an overall configuration of an image search system according to an embodiment. It is a figure explaining the outline | summary of EMM of one Embodiment. It is a block diagram which shows the hardware constitutions of the user terminal of one Embodiment. It is a block diagram which shows the hardware constitutions of the image search server of one Embodiment. It is a figure which shows the structural example of feature-value DB of one Embodiment. It is a figure which shows the structural example of content DB of one Embodiment. It is a block diagram which shows the function structure of the user terminal and image search server of one Embodiment. It is a figure which shows the structural example of the feature-value table of one Embodiment. It is a figure which shows the outline | summary of operation | movement of the image search system of one Embodiment. It is a flowchart of the process which the image search system of one Embodiment performs. It is a flowchart of the process which the image search system of one Embodiment performs. It is a figure which shows the state with which the feature point of the query image and the feature point of the to-be-searched image were matched in one Embodiment. It is the figure which illustrated the key image specified in one embodiment. It is a figure which shows the state with which the feature point of the key image and the feature point of the candidate image were matched in one Embodiment. It is the figure which showed the content browsing screen displayed on the user terminal of one Embodiment. It is a figure which shows the structural example of feature-value DB231 of one modification. It is a block diagram which shows the function structure of the user terminal and image search server of a modification. It is a flowchart of the process which the image search system of one modification is performed. It is the figure which illustrated the key image specified in one modification.

Embodiments of the present invention will be described with reference to the drawings.
[Embodiment]
FIG. 1 is a diagram showing an overall configuration of an image search system 1 of the present embodiment. As shown in FIG. 1, the image search system 1 includes a user terminal 10 and an image search server 20 and is a system for performing an image search. Although only one user terminal 10 is illustrated in FIG. 1, the number of user terminals 10 varies depending on the number of users who use the image search system 1 in practice. The user terminal 10 and the image search server 20 communicate with each other via the communication line 100. The communication line 100 is a communication line including the Internet, for example, but the type of the communication line is not particularly limited.

  The user terminal 10 is a communication terminal having a photographing function (camera function), photographs the printed matter 40 shown in FIG. 1, and sends image data representing a query image including a key image that is a retrieval target image to the image retrieval server 20. Send. The printed material 40 is, for example, an advertisement, a brochure, or a magazine, and is a printed material on which image elements such as characters and photographs are printed on a medium (for example, a paper medium). On the printed matter 40 used in the image search system 1, a marker 41 called an embedded media marker (EMM) is printed so as to overlap these image elements.

  The marker 41 is a mark that the digital content is associated with the image element of the printed matter 40. The marker 41 is printed in a color and density that do not hinder the user from visually recognizing the image element of the printed matter 40. The image search server 20 provides digital content to the user terminal 10 in accordance with the result of the image search performed using the query image.

    The user terminal 10 is a smartphone here, but is another communication terminal having a photographing function, such as a mobile phone terminal, a tablet terminal, a notebook computer, a PDA (Personal Digital Assistant), a portable game machine, or the like. Also good.

  FIG. 2 is a diagram for explaining the outline of the EMM. A schematic diagram of the marker 41 is shown in FIG. As shown in FIG. 2A, the marker 41 includes a feature boundary 411, an anchor point 412, and a media icon 413. The feature boundary 411 is represented here by a circular line image, and indicates a region including an image element (or image element group) constituting a key image among image elements printed on the printed matter 40 to the user. As shown in FIG. 2B, the user of the user terminal 10 shoots the printed matter 40 so that the entire marker 41 is included in the shooting range (hereinafter, expressed as “shoot the marker 41”). Digital content can be displayed on the user terminal 10. The anchor point 412 indicates a position where information associated with digital content is described in the printed matter 40.

  The media icon 413 is an icon that indicates the type of digital content to the user. A media icon 413 illustrated in FIG. 2 is an icon imitating a video camera, and means that the associated digital content is moving image content. In this case, when the user photographs the marker 41 using the user terminal 10, the user terminal 10 reproduces the moving image content provided from the image search server 20, as shown in FIG. The digital content associated with the marker 41 is not limited to a moving image, and may be any type such as a still image or a web page.

  FIG. 3 is a block diagram illustrating a hardware configuration of the user terminal 10. As illustrated in FIG. 3, the user terminal 10 includes a control unit 11, a UI (User Interface) unit 12, a communication unit 13, a photographing unit 14, and a storage unit 15.

  The control unit 11 includes a microcomputer having a central processing unit (CPU), a read only memory (ROM), and a random access memory (RAM). The CPU controls each unit of the user terminal 10 by reading the program stored in the ROM or the storage unit 15 into the RAM and executing it. The UI unit 12 is an operation display unit that provides a graphical user interface (GUI), and includes a display panel that displays an image on a display surface, and a touch screen that is provided on the display surface and is operated by a user. Note that the user terminal 10 may have other operation means for accepting an operation such as a physical key, or may have a function for accepting an operation by voice.

  The communication unit 13 has an interface for connecting to the communication line 100 for communication. The imaging unit 14 is an imaging device having an imaging element such as a CCD (Charge Coupled Device), and generates image data representing a captured image. Here, the photographing lens (not shown) of the photographing unit 14 is provided on the back side of the user terminal 10. The storage unit 15 includes a storage device such as an EEPROM (Electronically Erasable and Programmable ROM) and a flash memory, and stores an OS (Operating System) executed by the control unit 11 and other programs.

  FIG. 4 is a block diagram illustrating a hardware configuration of the image search server 20. As shown in FIG. 4, the image search server 20 includes a control unit 21, a communication unit 22, and a storage unit 23. The control unit 21 is a control device having an arithmetic device including a CPU and an ASIC (Application Specific Integrated Circuit) and a memory, and controls each unit of the image search server 20. The communication unit 22 is an interface for connecting to the communication line 100 for communication. The storage unit 23 is a storage device having a hard disk device, for example, and stores a program executed by the control unit 21. Further, the storage unit 23 includes a feature amount DB (Data Base) 231 and a content DB 232.

  The feature amount DB 231 is a database in which feature amounts of feature points of the search target image are registered for each of the plurality of search target images. FIG. 5 is a diagram illustrating a configuration example of the feature amount DB 231. The feature amount DB 231 has data records corresponding to each of the searched images, and each data record has, as a data field, an “image identifier” that identifies the searched image and a “feature point identifier” that identifies a feature point of the searched image. ", A" feature amount "which is a local feature amount for each feature point. In FIG. 5, the image shown to the left of the feature point identifier is a searched image, and the portion indicated by ○ is the feature point of the searched image. In FIG. 5, the search target image is illustrated for convenience in the description of the operation of the image search system 1 described later. However, the feature amount DB 231 does not need to store image data of the search target image. . In the following description, a search target image identified by an image identifier “I00001” is referred to as “search target image“ I0001 ””.

  As shown in FIG. 5, one featured image usually has a plurality of feature points, and one data record in the feature amount DB 231 has a feature corresponding to each of a plurality of feature points identified by a feature point identifier. The amount is registered.

  In the present embodiment, a SIFT (Scale Invariant Feature Transform) feature quantity is used as a feature quantity, but a feature quantity according to another method may be employed instead of the SIFT feature quantity. The SIFT feature value is a local feature value that creates a brightness gradient histogram based on the representative brightness gradient direction of a pixel and expresses the brightness gradient with a multidimensional vector (128-dimensional vector). SIFT feature values are described in, for example, “Nagahashi, Fujiyoshi, Kinde“ Object identification using SIFT features based on region segmentation ”, Institute of Electrical Engineers of Japan, Systems and Control Study Group, pp39-44, January 2007). As it is well known, detailed description thereof is omitted.

  The content DB 232 is a database in which content data representing digital content corresponding to the searched image is registered for each of the plurality of searched images. FIG. 6 is a diagram illustrating a configuration example of the content DB 232. The content DB 232 also has a data record corresponding to each searched image, and each data record has “image identifier” for identifying the searched image and “content data” corresponding to the searched image as data fields. doing. The image identifier of the content DB 232 is common to the image identifier of the feature amount DB 231. The data record of the feature amount DB 231 and the data record of the content DB 232 are associated with each other by the image identifier. The content data is data representing digital content provided from the image search server 20 to the user terminal 10. In this embodiment, it is assumed that content data is registered in the content DB 232, but data specifying content data such as a URI (Uniform Resource Identifier) indicating a storage location of the content data is used instead of the content data. It may be registered in the DB 232.

  In the present embodiment, the image search server 20 includes the feature amount DB 231 and the content DB 232, but one or both of the feature amount DB 231 and the content DB 232 are managed by a device external to the image search server 20, and the image A configuration in which the search server 20 accesses the feature amount DB 231 and the content DB 232 via a communication line or the like may be employed.

  FIG. 7 is a block diagram illustrating functional configurations of the user terminal 10 and the image search server 20. The user terminal 10 functions as a device including a query image acquisition unit 101, a transmission unit 102, and a reception unit 103 by executing a program. The query image acquisition unit 101 generates and acquires image data representing a query image by photographing by the photographing unit 14. The image data generated by the query image acquisition unit 101 is temporarily stored in the storage unit 15 of the user terminal 10.

  The transmission unit 102 transmits the image data acquired by the query image acquisition unit 101 to the image search server 20. The receiving unit 103 receives content data transmitted from the image search server 20 as a response to the image data transmitted by the transmitting unit 102.

  The image search server 20 executes a program, thereby receiving a reception unit 201, a feature amount data generation unit 202, a storage unit 203, a first feature amount data acquisition unit 204, and a second feature amount data acquisition unit. 205, a first association unit 206, a candidate image specification unit 207, a key image specification unit 208, a second association unit 209, a similar image specification unit 210, a content data output unit 211, and a transmission It functions as a device including the unit 212.

  The receiving unit 201 receives and acquires image data representing a query image transmitted from the user terminal 10. The feature amount data generation unit 202 specifies a plurality of feature points of the query image represented by the image data received by the reception unit 201, and generates feature amount data representing each feature amount of the specified plurality of feature points. Since a known method is used as the feature point extraction and feature amount data generation method performed by the feature amount data generation unit 202, description thereof is omitted.

  FIG. 8 is a diagram illustrating a configuration example of a feature amount table including a plurality of feature amount data generated by the feature amount data generation unit 202. The feature amount table includes a data record corresponding to each of a plurality of feature points of the query image. Each data record has, as data fields, a “feature point identifier” for identifying a feature point, a “position” of the feature point in the query image, and a “feature amount” of the feature point. The feature amount data generation unit 202 temporarily stores the generated feature amount table in the storage unit 203 until the image search process is completed.

  The storage unit 203 stores the feature amount DB 231 and the content DB 232, and temporarily stores the feature amount table generated by the feature amount data generation unit 202. The storage unit 203 temporarily stores data generated by other components of the image search server 20.

  The first feature amount data acquisition unit 204 reads and acquires feature amount data stored in the feature amount DB 231 from the storage unit 203 as first feature amount data. The second feature amount data acquisition unit 205 reads out and acquires feature amount data stored in the feature amount table from the storage unit 203 as second feature amount data.

  The first association unit 206 includes a feature amount indicated by the first feature amount data read from the feature amount DB 231 by the first feature amount data acquisition unit 204, and a feature amount table by the second feature amount data acquisition unit 205. The feature amounts indicated by the second feature amount data read out from are compared, and based on the result of the comparison, for each of one or more feature points of the plurality of feature points of the query image, Match one of the feature points. More specifically, the first association unit 206 determines the similarity between the feature amount indicated by the feature amount data registered in the feature amount DB 231 and the feature amount indicated by the feature amount data included in the feature amount table. The distance between feature points as an index is sequentially calculated according to a known calculation formula. Thereafter, the first associating unit 206 associates the feature points of the search target image that cause the distance between the feature points equal to or less than a predetermined threshold with respect to each of the feature points of the query image. At this time, when there are a plurality of feature points of the image to be searched that cause the distance between the feature points equal to or less than the threshold, for example, the feature point of the image to be searched that causes the minimum distance between feature points is associated with the feature point of the query image. In addition, when there are two or more feature points of a query image that minimize the distance between feature points with a feature point of one search target image, the feature points of the query image indicating a smaller distance between feature points are the target points of the search image. Corresponding to the feature point of the search image.

  The candidate image specifying unit 207 resembles a searched image that satisfies the condition in which the number of feature points associated with the feature points of the query image by the first association unit 206 among a plurality of searched images satisfies a predetermined condition. Specify as a candidate image. More specifically, the candidate image specifying unit 207 relates, for each of the plurality of search target images, the number of feature points associated with any feature point of the query image with respect to the number of feature points of the search target image. Ratio (hereinafter referred to as “corresponding ratio”). Then, a search target image whose calculated correspondence ratio is equal to or greater than a predetermined threshold (for example, 70%) is specified as a candidate image of a similar image.

  For each of one or more candidate images specified by the candidate image specifying unit 207, the key image specifying unit 208 determines the feature points of the query image associated with the feature points of the candidate image by the first associating unit 206. An area of the query image included under a predetermined condition is specified as a key image. More specifically, the key image specifying unit 208 specifies, for each of one or more candidate images specified by the candidate image specifying unit 207, the feature points of the query image associated with the feature points of the candidate images. . Subsequently, the key image specifying unit 208 specifies the positions of the specified feature points in the query image from the feature point table, and specifies a circumscribed rectangle that includes all the feature points inside. The key image specifying unit 208 specifies the specified circumscribed rectangular area from the query image as a key image. The key image specifying unit 208 specifies key images as many as the number of candidate images.

  The second associating unit 209 relates to each of one or more key images specified by the key image specifying unit 208, and sets the feature points of candidate images read from the feature amount DB 231 by the first feature amount data acquiring unit 204. The feature amount indicated by one feature amount data is compared with the feature amount indicated by the second feature amount data of the feature point of the key image read from the feature amount table by the second feature amount data acquisition unit 205, and the comparison Based on the result, each of one or more feature points of the plurality of feature points of the key image is associated with one of the feature points of one or more candidate images. The specific process performed by the second associating unit 209 is the process performed by the first associating unit 206 except that the feature amount used for comparison is limited to the feature points of the candidate image and the feature points of the key image. This is a common process.

  The similar image specifying unit 210 uses the second associating unit 209 to select one or more candidate images specified by the candidate image specifying unit 207 for each of the one or more key images specified by the key image specifying unit 208. A candidate image that satisfies the condition that the number of feature points associated with the feature points of the key image is determined is specified as a similar image. More specifically, for each of one or more key images, the similar image specifying unit 210 first specifies a candidate image having the largest number of feature points associated with the feature points of the key image. Subsequently, the similar image specifying unit 210 calculates a ratio (correspondence ratio) of the number of feature points associated with any of the feature points of the key image to the number of feature points of the specified candidate image. If the calculated correspondence ratio is equal to or greater than a predetermined threshold (for example, 80%), the candidate image is specified as a similar image of the key image.

  The content data output unit 211 reads content data corresponding to the image identifier of the similar image from the content DB 232 (FIG. 6) regarding the similar image specified for each of the key images by the similar image specifying unit 210, and sends it to the transmission unit 212. Output. The content data output from the content data output unit 211 to the transmission unit 212 indicates a representative position (for example, the center position of the key image) in the query image of the key image determined to be similar to the similar image corresponding to the content data. Position data is added.

  The transmission unit 212 transmits the content data output from the content data output unit 211 to the user terminal 10 that is the transmission source of the image data received by the reception unit 201. When no similar image is specified by the similar image specifying unit 210, the content data output unit 211 outputs message data notifying the transmission unit 212 of the failure to search for a similar image, and the transmission unit 212 outputs the message. Data is transmitted to the user terminal 10 instead of the content data.

  The user terminal 10 displays the digital content represented by the content data received by the receiving unit 103 on the UI unit 12. At that time, the user terminal 10 displays the query image on the UI unit 12 in accordance with the image data temporarily stored in the storage unit 15, and the position on the query image indicated by the position data added to each content data. Display digital content.

  The above is the configuration of the image search system 1. Next, the operation of the image search system 1 will be described. In the following description, it is assumed that the data illustrated in FIG. 5 is stored in the feature amount DB 231 and the data illustrated in FIG. 6 is stored in the content DB 232. FIG. 9 is a diagram showing an outline of the operation of the image search system 1 described below. 10A and 10B (hereinafter, FIG. 10A and FIG. 10B are collectively referred to as FIG. 10) are flowcharts showing a sequence of processing performed by the image search system 1.

  First, the user starts an application for using the image search system 1 by operating the user terminal 10, and then photographs the printed matter 40 using the camera function (step S101). The query image acquisition unit 101 of the user terminal 10 generates and acquires image data representing the captured query image. The image data generated by the query image acquisition unit 101 is temporarily stored in the storage unit 15.

  In the example of FIG. 9, two markers 41 are printed on the printed matter 40, and a diamond figure (hereinafter simply referred to as “diamond”) and a clover figure (hereinafter referred to as “diamond”) are included in the feature boundary 411 of these markers 41. Is simply printed). The diamond is the same image as the searched image “I0001” in the feature amount DB 231 (FIG. 5), and the clover is the same image as the searched image “I0002”.

  The user can individually photograph each of the two markers 41, but may photograph the two markers 41 at a time. Here, it is assumed that the user captures two markers 41 printed on the printed matter 40 at a time. The query image photographed by the user includes a triangle mark (hereinafter referred to as “triangle”) in addition to a diamond and a clover. This triangle is not any searched image.

  The transmission unit 102 of the user terminal 10 transmits image data representing the query image to the image search server 20 (step S102). The receiving unit 201 of the image search server 20 receives image data from the user terminal 10 (step S103), and delivers it to the feature amount data generating unit 202. The receiving unit 201 may temporarily store the image data in the storage unit 203, and the feature amount data generation unit 202 may read the image data from the storage unit 203. In the following description, when a configuration unit of the image search server 20 delivers data to another configuration unit, the data is directly delivered as in the case of delivery of image data from the reception unit 201 to the feature amount data generation unit 202. Instead, data may be delivered via the storage unit 203.

  The feature amount data generation unit 202 extracts feature points of the query image represented by the image data, and specifies the position and feature amount of each of the extracted feature points. The feature amount data generation unit 202 generates a feature amount table (FIG. 8) that stores the positions and feature amounts of the feature points of the query image (step S104) and stores them in the storage unit 203. Here, it is assumed that n (n is a natural number) feature points are extracted from the query image as shown in FIG.

  Subsequently, the second feature quantity data acquisition unit 205 reads the feature quantity data of the first data record (data record of the feature point identifier “p1”) from the feature quantity table as second feature quantity data (step S105). , And hand it over to the first association unit 206. When the second feature quantity data acquisition unit 205 delivers the feature quantity data to the first association unit 206, if the feature quantity data is the last feature quantity data stored in the feature quantity table, this is indicated. Append last notification data. In this case, since the feature amount data handed over by the second feature amount data acquisition unit 205 to the first association unit 206 is not the last, the last notification data is not added.

  Subsequently, the first feature amount data acquisition unit 204 adds the first feature point identifier (“p1”) included in the first data record (data record of the image identifier “I0001”) from the feature amount DB 231 (FIG. 5). The corresponding feature amount data is read as first feature amount data (step S106), and is transferred to the first association unit 206. Similar to the second feature amount data acquisition unit 205, the first feature amount data acquisition unit 204 stores the feature amount data in the feature amount DB 231 when passing the feature amount data to the first association unit 206. If it is the last feature amount data, last notification data indicating that is added. In this case, since the feature amount data handed over by the first feature amount data acquisition unit 204 to the first association unit 206 is not the last, the last notification data is not added.

  The first association unit 206 includes the feature amount indicated by the second feature amount data delivered from the second feature amount data acquisition unit 205 and the first feature amount delivered from the first feature amount data acquisition unit 204. The feature point distance is calculated using the feature amount indicated by the feature amount data, and the feature point distance data indicating the calculation result is generated. The first associating unit 206 identifies the distance data between the feature points and data for identifying the two feature points compared (the feature point identifier for the second feature amount data, the image identifier for the first feature amount data, and (Feature point identifier) is temporarily stored in the storage unit 203 (step S107).

  The first association unit 206 determines whether or not the last notification data has been added to the first feature value data delivered from the first feature value data acquisition unit 204 in step S106 (step S108). In this case, since the last notification data is not added to the first feature data (step S108; No), the first association unit 206 sends the next feature data to the first feature data acquisition unit 204. Request. In response to this request, the first feature value data acquisition unit 204 reads the next feature value data (in this case, feature value data corresponding to the feature point identifier “p2” of the image identifier “I0001”) from the feature value DB 231. Then, the first association unit 206 is handed over (step S106).

  Subsequently, the first associating unit 206 receives the second feature amount data already transferred from the second feature amount data acquiring unit 205 in step S105, and the first feature amount data acquiring unit in a new step S106. Using the first feature value data delivered from 204, the process of step S107 is performed. Thereafter, the processing in steps S106 to S108 is repeated until the first association unit 206 performs the processing in step S107 using the first feature data with the last notification data.

  When the first association unit 206 completes the process of step S107 using the first feature amount data with the last notification data (step S108; Yes), the second association amount data acquisition unit 205 in step S105. It is determined whether the last notification data has been added to the delivered second feature data (step S109). In this case, since the last notification data is not added to the second feature amount data (step S109; No), the first association unit 206 sends the next feature amount data to the second feature amount data acquisition unit 205. Request. The first association unit 206 requests the first feature amount data acquisition unit 204 for the first feature amount data in the feature amount DB 231.

  In response to the request of the first association unit 206, the second feature amount data acquisition unit 205 obtains the next feature amount data (in this case, feature amount data corresponding to the feature point identifier “p2”) from the feature amount table. Read and hand over to the first association unit 206 (step S105). In addition, the first feature value data acquisition unit 204 reads again the first feature value data (the feature value data corresponding to the feature point identifier “p1” of the image identifier “I0001”) from the feature value DB 231 and performs the first association. Delivered to the unit 206 (step S106).

  Subsequently, the first association unit 206 receives the second feature amount data delivered from the second feature amount data acquisition unit 205 in the new step S105, and the first feature amount data acquisition unit in the new step S106. Using the first feature value data delivered from 204, the process of step S107 is performed. Thereafter, steps S106 to S109 are performed until the first associating unit 206 performs the process of step S107 using the first feature value data with the last notification data and the second feature value data with the last notification data. The process is repeated.

  When the first association unit 206 completes the process of step S107 using the first feature value data with the last notification data and the second feature value data with the last notification data (step S109; Yes), The storage unit 203 stores feature point distance data indicating distances between feature points of combinations of all feature points of the query image and all feature points of all the searched images. Become.

  Subsequently, the first association unit 206 associates the feature points of the query image with the feature points of the search target image based on the distance data between the feature points temporarily stored in the storage unit 203 in step S107. (Step S110). FIG. 11 is a diagram illustrating a state in which the feature points of the query image and the feature points of the search target image are associated by the first association unit 206. In FIG. 11, feature points that have been associated are indicated by ●, and feature points that have not been associated are indicated by ○ (the same applies to FIG. 12 and the like).

  Subsequently, the candidate image specifying unit 207 determines that the ratio (correspondence ratio) of the feature points associated with the feature points of the query image in the search target image is equal to or greater than a predetermined threshold (70%). The search image is specified as a candidate image for a similar image (step S111). In the case of the example of FIG. 11, since 7 of the 8 feature points of the searched image “I0001” are associated with the feature points of the query image, the correspondence ratio is 87.5%. Similarly, the correspondence ratio of the searched image “I0002” is approximately 77.8%, the corresponding ratio of the searched image “I0003” is approximately 16.7%, and the corresponding ratio of the searched image “I0004” is approximately 22.2%. It becomes. Therefore, in this case, the candidate image specifying unit 207 specifies the searched image “I0001” and the searched image “I0002” as candidate images.

  Subsequently, for each of the candidate images specified by the candidate image specifying unit 207, the key image specifying unit 208 is a circumscribed rectangular region that includes all of the feature points of the query image associated with the feature points of the candidate image on the inside. Is specified as a key image (step S112). FIG. 12 is a diagram illustrating a key image specified by the key image specifying unit 208. In FIG. 12, a key image “K0001” is a key image corresponding to a search target image “I0001” that is a candidate image, and a key image “K0002” is a key image corresponding to a search target image “I0002” that is a candidate image. is there.

  Subsequently, the second association unit 209 associates the feature point of the key image with the feature point of the candidate image for each of the key images specified in step S112 (steps S113 to S118). The process of associating feature points by the second associating unit 209 is compared with the process of associating the feature points of the query image and the feature points of the search target image by the first associating unit 206 (steps S105 to S109). However, they are common except that the feature points to be matched are different.

  That is, the second association unit 209 first sets the feature amount data of the feature points of the first key image (in this case, the key image “K0001”) one by one by the second feature amount data acquisition unit 205. The data is read and delivered to the second association unit 209 as second feature amount data (step S113). In addition, the first feature value data acquisition unit 204 reads feature value data of feature points of the candidate images (in this case, the searched image “I0001” and the searched image “I0002”) one by one and reads the first feature value. The data is transferred to the second association unit 209 as data (step S114). The second association unit 209 generates the distance data between the feature points using the second feature amount data and the first feature amount data, and temporarily stores them in the storage unit 203 (step S115).

  Processing in steps S113 to S115 is performed based on the presence / absence of last notification data until calculation of the distance between feature points of all feature points of all key images and all feature points of all candidate images is completed (step It repeats by S116-S118).

  Subsequently, the second association unit 209 associates the feature points of the key image with the feature points of the candidate image based on the distance data between the feature points temporarily stored in the storage unit 203 in step S118 (step S118). S119). FIG. 13 is a diagram illustrating a state in which the feature points of two key images and the feature points of two candidate images are associated by the second association unit 209.

  Subsequently, the similar image specifying unit 210 has a candidate image in which the ratio (correspondence ratio) of the feature points associated with the feature points of the key image is equal to or greater than a predetermined threshold (90%) among the candidate images. Are identified as similar images (step S120). In the case of the example in FIG. 13, eight of the eight feature points of the search target image “I0001”, which is a candidate image, are associated with the feature points of the query image, so the correspondence ratio is 100%. Similarly, the correspondence ratio of the search target image “I0002” that is a candidate image is also 100%. Therefore, in this case, the similar image specifying unit 210 specifies the searched image “I0001” and the searched image “I0002” as similar images.

  Subsequently, for each similar image, the content data output unit 211 specifies the key image having the largest number of feature points associated with the feature points of the similar image as the key image corresponding to the similar image (step S121). . In this case, the key image “K0001” is specified as the key image corresponding to the searched image “I0001” that is a similar image, and the key image “K0002” is specified as the key image corresponding to the searched image “I0002” that is a similar image. Identified. In step S121, the correspondence between the similar image and the key image may be specified based on the correspondence between the candidate image and the key image specified in step S112.

  Subsequently, for each similar image, the content data output unit 211 reads the content data corresponding to the image identifier of the similar image from the content DB 232 (FIG. 6), and represents the representative position (center of the corresponding key image identified in step S121. Position data indicating the position) is added and output to the transmission unit 212. The transmission unit 212 transmits the content data added with the position data output from the content data output unit 211 to the user terminal 10 (step S122) (FIG. 9).

  The receiving unit 103 of the user terminal 10 receives the content data transmitted from the image search server 20 as a response to the transmission of the query image (step S102) (step S123). The user terminal 10 displays the digital content represented by the received content data on the UI unit 12 (step S124). At that time, the user terminal 10 displays the query image as the background image using the image data of the query image temporarily stored in the storage unit 15 previously.

  FIG. 14 is a diagram illustrating a content browsing screen displayed on the user terminal 10 in step S124. On the content browsing screen, corresponding digital content (or a button corresponding to the digital content) is displayed at the center position of the key image in the query image displayed as the background image. In the example of FIG. 14, both the digital content “C0001” and the digital content “C0002” are moving image content, and a button that prompts a reproduction operation is displayed instead of the digital content. The user can play back each moving picture content by operating these buttons. The operation of the image search system 1 has been described above.

  As described above, in the image search system 1, identification of candidate images according to the number of feature points of the searched image associated with the feature points of the query image by the first association unit 206, and The corresponding key image is specified. Further, the second association unit 209 identifies similar images according to the number of feature points of the candidate image associated with the feature points of the key image. In the image search system according to the conventional technique, the same association as the association performed by the first association unit 206, that is, the association between the feature point of the query image and the feature point of the searched image is performed. The search target image having the largest number of feature points is specified as a similar image. Therefore, according to the prior art, for example, the searched image “I0001” illustrated in FIG. 11 is specified as a similar image, and the searched image “I0002” is not specified as a similar image.

  Further, in the related art, when the condition that the corresponding ratio is 90% or more is specified as a similar image, both the searched image “I0001” and the searched image “I0002” have a corresponding ratio of 90%. Therefore, it is not specified as a similar image. That is, the search fails. Therefore, if the conditions for specifying as a similar image are relaxed and the corresponding ratio is 80% or more, for example, the searched image “I0001” is specified as a similar image, but the searched image “I0002” "Is not specified as a similar image. Furthermore, if the conditions for specifying as similar images are relaxed and the corresponding ratio is 70% or more, for example, both the searched image “I0001” and the searched image “I0002” are specified as similar images. However, if the threshold of the corresponding ratio when specifying as a similar image is too low, there is an increased risk that a searched image unrelated to the key image is erroneously specified as a similar image.

  On the other hand, according to the image search system 1, the search target image that is estimated to be unrelated to the query image by specifying the candidate image is excluded from the candidates for specifying the similar image. Therefore, the probability that the feature point of the query image that is erroneously associated with the irrelevant image to be searched is associated with the correct feature point of the image to be searched when the similar image is specified increases. Furthermore, according to the image search system 1, a region that is estimated to contain the same image as the image to be searched is extracted as a key image from the query image, and the second correspondence is applied to feature points of other regions. It is excluded from the association by the attaching unit 209. Therefore, the probability that a feature point of an image unrelated to the searched image such as a background image of the query image is erroneously associated with the feature point of the searched image is reduced. As a result, the feature points of the query image and the feature points of the image to be searched are associated with higher accuracy than in the case of the conventional technique.

[Modification]
The above-described embodiment is an example of the present invention, and the present invention is not limited to this. Various modifications can be made within the scope of the technical idea. Examples of these modifications are shown below. Note that two or more of the embodiments described above and the modifications exemplified below may be combined.

[First Modification]
In the first modified example, the search target images are grouped for each page, and the number of feature points of the search target image associated with the feature points of the query image is counted for each page in specifying candidate images. The candidate image is specified from the searched images of the page having the maximum number.

  FIG. 15 is a diagram illustrating a data configuration example of the feature amount DB 231 used in the first modification. As shown in FIG. 15, in the first modification, the data record of the feature value DB 231 is provided with a data field “page number”. This page number indicates on which page the searched image identified by the image identifier is printed in the printed matter 40 having a plurality of pages.

  In the first modification, in step S111 (FIG. 10), the candidate image specifying unit 207 first determines the number of feature points of the search target image associated with the feature points of the query image for each page number of the search target image. The search target image having the largest page number is selected. Thereafter, the candidate image specifying unit 207 specifies a candidate image based on the correspondence ratio among the search target images selected based on the page numbers.

  According to the first modified example, the page captured by the user terminal 10 is estimated based on the number of feature points of the searched image associated with the feature point of the query image, and printed on the page estimated to be captured. Since the candidate image is specified from the searched images that have been searched, the risk that the searched image printed on a page that has not been shot is erroneously specified as a candidate image is reduced. As a result, when the printed matter 40 has a plurality of pages, similar images are specified with higher accuracy than in the above-described embodiment.

[Second Modification]
In the second modified example, similarly to the first modified example, the searched images are grouped for each page (FIG. 15). However, in the second modified example, in specifying a similar image, the number of feature points of the candidate image associated with the feature point of the key image is counted for each page, and the number of the feature images of the page with the largest number is counted. A similar image is identified.

  In the second modification, the similar image specifying unit 210 first counts the number of feature points of the candidate images associated with the feature points of the key image for each page number of the candidate images in step S120 (FIG. 10). Then, the candidate image of the page number with the largest number is selected. Thereafter, the similar image specifying unit 210 specifies the similar image based on the correspondence ratio from the candidate images selected based on the page number.

  According to the second modification, the page captured by the user terminal 10 is estimated based on the number of feature points of the candidate image associated with the feature point of the key image, and printed on the page estimated to be captured. Since a similar image is identified from among the candidate images, the risk that a candidate image printed on a page that has not been shot is erroneously identified as a similar image is reduced. As a result, when the printed matter 40 has a plurality of pages, a similar image is specified with higher accuracy than in the above-described embodiment.

[Third Modification]
In the third modification, the key image is not specified, and the second association unit 209 associates the feature point of the query image with the feature point of the candidate image. Therefore, in the third modification, the image search server 20 does not include the key image specifying unit 208. In step S113 (FIG. 10), the feature amount data transferred from the second feature amount data acquisition unit 205 to the second association unit 209 is all feature amount data of the feature points of the query image. Therefore, the loop processing according to the determination in step S118 is not necessary.

  According to the third modification, since the key image is not specified, there is a possibility that the feature points of the region other than the key image in the query image are erroneously associated with the feature points of the candidate image. Therefore, according to the third modification, the accuracy of the image search is reduced as compared with the case in the above-described embodiment. However, since the candidate image is specified, the image search is performed with higher accuracy than in the case of the conventional technique.

[Fourth Modification]
In the fourth modified example, the second associating unit 209 uses the key image based on the correspondence between the candidate image and the key image specified in step S112 in the feature point association in step S119 (FIG. 10). And the feature points of the candidate image corresponding to the key image are associated with each other.

  For example, in the example shown in FIG. 12, the key image “K0001” is specified as a circumscribed rectangular area of the feature point of the query image associated with the feature point of the search target image “I0001” that is the candidate image. It is. Therefore, the candidate image corresponding to the key image “K0001” is the searched image “I0001”. Similarly, the candidate image corresponding to the key image “K0002” is the searched image “I0002”.

  In step S115, the second associating unit 209 calculates the feature value data (second feature value data) of the feature point of the key image “K0001” and the feature value data (first value of the feature point of the searched image “I0001”). The distance between feature points is calculated using the feature amount data. In step S119, the second association unit 209 sets the feature point of the key image “K0001” as one of the feature points of the searched image “I0001” that is a candidate image corresponding to the key image “K0001”. Associate. Similarly, in step S115, the second associating unit 209 determines the feature amount data (second feature amount data) of the feature point of the key image “K0002” and the feature amount data of the feature point of the searched image “I0002”. The distance between feature points is calculated using (first feature value data). In step S119, the second association unit 209 sets the feature point of the key image “K0002” as one of the feature points of the searched image “I0002” that is a candidate image corresponding to the key image “K0002”. Associate.

  According to the fourth modification, the processing amount in steps S113 to S119 is reduced compared to the case in the above-described embodiment, and a similar image is quickly identified.

[Fifth Modification]
In the fifth modification, the user terminal 10 extracts feature points of a query image and generates feature amount data. FIG. 16 is a diagram illustrating a functional configuration of the user terminal 10 and the image search server 20 in the fifth modified example. The image search server 20 in the fifth modified example does not include the feature amount data generation unit 202, and instead the user terminal 10 includes the feature amount data generation unit 104.

  FIG. 17 is a flowchart showing a part of a sequence of processing performed by the image search system 1 in the fifth modification. When the query image acquisition unit 101 generates image data of the query image by photographing (step S101), the feature amount data generation unit 104 generates a feature amount table (FIG. 8) using the image data of the query image (step S201). . Note that the process performed by the feature data generation unit 104 in step S204 is a process common to the process performed by the feature data generation unit 202 in the above-described embodiment.

  Subsequently, the transmission unit 102 transmits the feature amount table generated by the feature amount data generation unit 104 to the image search server 20 (step 202). The receiving unit 201 of the image search server 20 receives the feature amount table (step S203) and temporarily stores it in the storage unit 203. Thereafter, the processing after step S105 in the above-described embodiment is performed.

  According to the fifth modification, the feature amount data of the query image is generated by the user terminal 10, so that the processing load on the image search server 20 is reduced compared to the case in the above-described embodiment. Further, since the data amount of the feature amount table is generally smaller than the data amount of the image data, according to the fifth modification, communication is performed via the communication line 100 as compared with the case of the above-described embodiment. Data volume is reduced.

[Other variations]
(1) In the above-described embodiment, the SIFT feature amount is used as the feature amount. However, other local feature amounts such as SURF (Speeded Up Robust Features) feature amount may be used. The local feature amount of the present invention is not limited to the feature of the luminance gradient, and may represent the position (coordinate) of the feature point, for example.

(2) In the above-described embodiment, the image search server 20 transmits content data corresponding to the searched image similar to the key image to the user terminal 10, but the information to be transmitted is not limited to the content data. For example, the image search server 20 may transmit image data representing a similar image to the user terminal 10.

(3) The arrangement of the constituent units included in the user terminal 10 and the constituent units included in the image search server 20 can be variously changed. For example, the user terminal 10 may have all the functions of the image search server 20 and the image search server 20 may not include the image search server 20.

(4) In the above-described embodiment, a circumscribed rectangular region of the feature point of the query image associated with the feature point of the candidate image in specifying the key image is specified as the key image. The shape and size for specifying the key image are not limited to the circumscribed rectangle, but may be other shapes such as a circle, or other shapes such as a rectangle obtained by enlarging the circumscribed rectangle with a predetermined magnification. May be the size.

(5) In the embodiment described above, a region including all the feature points of the query image associated with the feature points of the candidate image in specifying the key image is specified as the key image. The area of the key image does not necessarily include all the feature points inside. The area of the key image may be specified according to other conditions as long as the area includes the feature points under a predetermined condition. Good. For example, a key image is identified by including an area including a feature point of a query image associated with a feature point of a candidate image at a predetermined density threshold or more, an area inside a line having a density gradient as a threshold, and the like. A configuration specified as a key image may be employed. When the key image is specified based on the density of feature points in this way, in the case illustrated in FIG. 13 in the above-described embodiment, the key image is specified as shown in FIG. As a result, compared to the case of the above-described embodiment, the probability that the key image includes a feature point that is not a feature point of the image to be searched (the same image as the image to be searched) is reduced, so that the accuracy of the image search is increased.

(6) In the above-described embodiment, based on the ratio (correspondence ratio) of the number of feature points associated with the feature points of the query image among the feature points of the image to be searched in the specification of candidate images and the specification of similar images. , It is configured to specify that. As long as it is based on the number of feature points associated with the feature points of the query image among the feature points of the image to be searched for, the candidate image and the similar image are identified by an index other than the correspondence ratio (for example, An absolute number of feature points, a number obtained by multiplying by weighting according to specification of feature points, or the like) may be used.

(7) Each function realized by the user terminal 10 or the image search server 20 according to the above-described embodiment or its modification may be realized by one or a plurality of hardware circuits, or one or a plurality of programs may be calculated by an arithmetic unit. May be realized by executing or a combination thereof. When the functions of the user terminal 10 and the image search server 20 are realized using a program, the program includes a magnetic recording medium (magnetic tape, magnetic disk (HDD (Hard Disk Drive), FD (Flexible Disk), etc.)), It may be provided in a state stored in a computer-readable recording medium such as an optical recording medium (such as an optical disk), a magneto-optical recording medium, or a semiconductor memory, or may be distributed via a communication line. The present invention can also be understood as an image search method performed by a computer.

DESCRIPTION OF SYMBOLS 1 ... Image search system, 10 ... User terminal, 11 ... Control part, 12 ... UI part, 13 ... Communication part, 14 ... Imaging | photography part, 15 ... Memory | storage part, 20 ... Image search server, 21 ... Control part, 22 ... Communication , 23 ... storage unit, 40 ... printed matter, 41 ... marker, 100 ... communication line, 101 ... query image acquisition unit, 102 ... transmission unit, 103 ... reception unit, 104 ... feature quantity data generation unit, 201 ... reception unit, 202 ... feature value data generation unit, 203 ... storage unit, 204 ... first feature value data acquisition unit, 205 ... second feature value data acquisition unit, 206 ... first association unit, 207 ... candidate image specification unit , 208 ... Key image specifying part, 209 ... Second association part, 210 ... Similar image specifying part, 211 ... Content data output part, 212 ... Transmitting part, 231 ... Feature value DB, 232 ... Content DB, 411 ... Feature boundary 412 ... anchor points, 413 ... media icon

Claims (10)

First feature quantity data acquisition means for acquiring first feature quantity data indicating each feature quantity of a plurality of feature points of the searched image for each of the plurality of searched images;
Second feature quantity data acquisition means for acquiring second feature quantity data indicating each feature quantity of the plurality of feature points of the query image;
Based on the result of the comparison between the feature amount indicated by the first feature amount data and the feature amount indicated by the second feature amount data, one or more feature points of the plurality of feature points of the query image are First associating means for associating any of the feature points of the plurality of images to be searched;
Among the plurality of searched images, one or more searched images satisfying a condition in which the number of feature points associated with the feature points of the query image by the first association unit satisfies a predetermined condition. Candidate image specifying means for specifying as a candidate image;
A plurality of feature points of the query image based on a result of comparison between the feature amount indicated by the first feature amount data and the feature amount indicated by the second feature amount data of the one or more candidate image feature points. Second association means for associating one or more feature points of the one or more candidate images with one or more feature points of
Of the one or more candidate images, a similar image that identifies candidate images that satisfy a predetermined condition for the number of feature points associated with the feature points of the query image by the second association unit as similar images An image search device comprising: a specifying unit. For each of the one or more candidate images, one or more of the query images including the feature points of the query image associated with the feature points of the candidate image by the first association unit under a predetermined condition. A key image specifying means for specifying an area as a key image;
The second association means relates to each of the one or more key images, the feature amount indicated by the first feature amount data of the feature point of the one or more candidate images, and the feature point of the key image. Based on the result of comparison with the feature amount indicated by the second feature amount data, one or more feature points of the plurality of feature points of the key image are associated with one of the feature points of the one or more candidate images. The image search device according to claim 1. The second association means relates to each of the one or more key images, and for each of one or more feature points of the plurality of feature points of the key image, the key image specifying means includes The image search device according to claim 2, wherein any one of the feature points of the candidate image used for identification is associated. Each of the plurality of searched images is included in any of a plurality of pages, and the first feature amount data acquisition unit is configured to group the first feature amounts grouped for each page including the searched image. Get the data,
The candidate image specifying unit specifies the number of feature points associated with the feature points of the query image by the first association unit for each group to which the first feature amount data of the feature points belongs. The image search device according to claim 1, wherein the candidate image is specified based on the number of feature points for each specified group. Each of the plurality of searched images is included in any of a plurality of pages, and the first feature amount data acquisition unit is configured to group the first feature amounts grouped for each page including the searched image. Get the data,
The similar image specifying unit specifies the number of feature points associated with the feature point of the query image by the second association unit for each group to which the first feature amount data of the feature point belongs. The image search device according to any one of claims 1 to 4, wherein the similar image is specified based on the number of feature points for each specified group. The number of feature points of the searched image associated with the feature points of the query image required under the conditions used when the similar image specifying unit specifies a similar image is determined by the candidate image specifying unit. 6. The image search device according to claim 1, wherein the image search device is larger than the number of feature points of the search target image associated with the feature points of the query image required under the conditions used when performing the search. Image data acquisition means for acquiring image data representing a query image;
A plurality of feature points of the query image represented by the image data, and feature amount data generating means for generating feature amount data indicating each feature amount of the plurality of feature points, and
The image search according to any one of claims 1 to 6, wherein the second feature amount data acquisition unit acquires the feature amount data generated by the feature amount data generation unit as the second feature amount data. apparatus. A terminal device and an image search device;
The terminal device
Image data acquisition means for acquiring image data representing a query image;
Transmitting means for transmitting the image data to the image search device;
Receiving means for receiving data transmitted from the image search device as a response to the image data transmitted by the transmitting means;
The image search device includes:
For each of the plurality of search target images, feature amount data acquisition means for acquiring first feature amount data indicating the feature amounts of the plurality of feature points of the search target image;
Receiving means for receiving the image data from the terminal device;
Feature quantity data generating means for specifying a plurality of feature points of the query image represented by the image data and generating second feature quantity data indicating each feature quantity of the plurality of feature points;
Based on a result of comparison between the feature quantity indicated by the first feature quantity data and the feature quantity indicated by the second feature quantity data, the one or more feature points of the plurality of feature points of the query image are First associating means for associating one of feature points of a plurality of search target images;
Among the plurality of searched images, one or more searched images satisfying a condition in which the number of feature points associated with the feature points of the query image by the first association unit satisfies a predetermined condition. Candidate image specifying means for specifying as a candidate image;
A plurality of feature points of the query image based on a result of comparison between the feature amount indicated by the first feature amount data and the feature amount indicated by the second feature amount data of the one or more candidate image feature points. Second association means for associating one or more feature points of the one or more candidate images with one or more feature points of
Among the one or more candidate images, a similar image that identifies candidate images that satisfy a predetermined condition for the number of feature points associated with the feature points of the query image by the second association unit as a similar image Specific means,
A data search system comprising: transmission means for transmitting data corresponding to the similar image specified by the similar image specifying means to the terminal device. A terminal device and an image search device;
The terminal device
Image data acquisition means for acquiring image data representing a query image;
Feature quantity data generating means for identifying a plurality of feature points of the query image represented by the image data and generating feature quantity data indicating each of the feature quantities of the plurality of feature points;
Transmitting means for transmitting the feature amount data to the image search device;
Receiving means for receiving data transmitted from the image search device as a response to the feature amount data transmitted by the transmitting means;
The image search device includes:
With respect to each of the plurality of search target images, feature amount data acquisition means for acquiring feature amount data indicating the feature amount of each of the plurality of feature points of the search target image as first feature amount data;
Receiving means for receiving the feature quantity data transmitted from the terminal device as second feature quantity data;
Based on a result of comparison between the feature quantity indicated by the first feature quantity data and the feature quantity indicated by the second feature quantity data, the one or more feature points of the plurality of feature points of the query image are First associating means for associating one of feature points of a plurality of search target images;
Among the plurality of searched images, one or more searched images satisfying a condition in which the number of feature points associated with the feature points of the query image by the first association unit satisfies a predetermined condition. Candidate image specifying means for specifying as a candidate image;
A plurality of feature points of the query image based on a result of comparison between the feature amount indicated by the first feature amount data and the feature amount indicated by the second feature amount data of the one or more candidate image feature points. Second association means for associating one or more feature points of the one or more candidate images with one or more feature points of
Of the one or more candidate images, a similar image that identifies candidate images that satisfy a predetermined condition for the number of feature points associated with the feature points of the query image by the second association unit as similar images Specific means,
A data search system comprising: transmission means for transmitting data corresponding to the similar image specified by the similar image specifying means to the terminal device. On the computer,
For each of the plurality of search target images, a process of obtaining first feature amount data indicating each feature amount of the plurality of feature points of the search target image;
Processing for obtaining second feature amount data indicating each feature amount of the plurality of feature points of the query image;
Based on the result of the comparison between the feature amount indicated by the first feature amount data and the feature amount indicated by the second feature amount data, one or more feature points of the plurality of feature points of the query image are A process of associating any of the feature points of the plurality of search target images;
A process of specifying one or more searched images satisfying a predetermined number of feature points associated with feature points of the query image among the plurality of searched images as candidate images of similar images;
A plurality of feature points of the query image based on a result of comparison between the feature amount indicated by the first feature amount data and the feature amount indicated by the second feature amount data of the one or more candidate image feature points. A process of associating one or more feature points of the one or more candidate images with one or more feature points of
A program for executing, as a similar image, a candidate image in which the number of feature points associated with the feature points of the query image satisfies a predetermined condition among the one or more candidate images.

Images