JP5946315B2 - Image search system - Google Patents

Description

The present invention relates to an image search system, an image search apparatus, and a computer program that extract an original image corresponding to an image displayed by a camera function of a portable terminal from a database and provide information related to the original image.

The mobile terminal is usually provided with a camera function, and when the mobile terminal is moved over the object to be photographed, the range where the camera lens is displayed is displayed in a predetermined area of the mobile terminal. If information related to the projected image can be accessed in real time, it can be applied to various presentations and education.
From this point of view, there is an “information processing apparatus and portable terminal” disclosed in Patent Document 1, for example, as a system for extracting an image similar to an image taken with a mobile phone from a database.

JP 2010-205121 A

In the above-mentioned Patent Document 1, the user designates a region of interest in an image, and tries to improve the accuracy of search by extracting and collating feature points with emphasis on that region. However, this has the following disadvantages. In order to designate a region from one image, it is necessary to classify the region by human eyes, which is not suitable for an application in which a retrieval result is instantaneously obtained from a captured image. If you want to search slowly, you can specify the area, but it is difficult to use it in a busy situation such as searching for products at a business location. It may be difficult to specify an area. For example, it is a case where a pamphlet or the like in which characters are printed on one side of a sheet is taken.
Further, Patent Document 1 does not clearly indicate whether or not “image” includes a moving image. There is no mention of what kind of processing is performed even if a moving image is included. If the function is to be expanded so that moving images are also included in the search target, the point that a person designates an area becomes a bottleneck.
In view of the above, an object of the present invention is to provide a moving image search service that targets not only a still image but also a moving image without reducing the accuracy of a search even if the entire captured image is used as a search key.

To achieve the above object, the present invention provides an image search system comprising:
Input means for accepting user input;
Output means for displaying a still image or a moving image (hereinafter referred to as “captured image”) obtained by the imaging means;
Feature point extraction means for extracting feature points of the displayed captured image;
An original image feature point database in which feature points of pre-collected image groups are stored;
A feature point collating unit that collates the feature points of the extracted captured image with the feature points extracted from the original image feature point database, and extracts information that specifies an image that satisfies a condition (hereinafter referred to as “winning image”);
Image-related information acquisition means for acquiring information related to the winning image based on the information specifying the winning image;
The feature point extraction means takes a plurality of still images, extracts feature points of each still image, and collects a series of feature points of a still image when a moving image is a feature point extraction target. And feature points of moving images .
Here, the “imaging means” is assumed to be a camera function built in the portable terminal. An object of the invention having such a configuration is to search for an image similar to the image displayed on the mobile terminal from the registered original image group, and various applications are conceivable. The sales representative described in the second embodiment makes a presentation using this system at the customer's site as one of application examples.
“Information specifying a hit image” refers to an image ID or a hash character string.
In the present invention, when the search target is a moving image, a plurality of still images are taken and the feature points of each still image are collected and used as the feature points of the moving image. As a result, the present invention can cope with not only still images but also moving images. For example, it is possible to shoot a TV screen and determine what program screen of which TV station is based on this feature point. .

In order to achieve the above object, the image search system of the present invention can take the following modes. In other words, the server and the mobile terminal are connected via a communication network,
The portable terminal is
Input means for accepting user input;
Output means for displaying a photographed image obtained by the imaging means;
Feature point extraction means for extracting feature points of the displayed captured image;
Image search request transmission means for transmitting the feature points of the extracted captured image to the server and requesting the identification of the hit image;
An image search result receiving means for receiving a search result, an image related information acquiring means for acquiring information stored in association with the winning image, and
The feature point extraction means takes a plurality of still images, extracts the feature points of each still image, collects the feature points of a series of still images, As feature points of the image,
The server
An original image feature point database storing feature points of the original image;
A feature point collating unit that collates a feature point received from the image search device with a feature point extracted from the original image feature point database, and extracts a hit image;
It is characterized by providing.
As described above, since the server performs the matching processing of the feature point which requires a processing load, a large amount of image data can be processed, and the search accuracy is improved.

In order to achieve the above object, the present invention preferably includes a feature point creating means for extracting feature points of the original image and registering them in the original image feature point database.
That is, since a function for automatically creating a feature point set is provided in advance, real-time feature point matching can be performed when operating the mobile terminal.

In order to achieve the above object, the feature point matching unit extracts, as candidate images, original images having many feature points that are close to Euclidean distances from the feature points of the captured image, and the positional relationship between the extracted candidate images. It is preferable to perform a storage determination process and regard a candidate image determined to have a stored positional relationship as a winning image.
If the determination processing for storing the positional relationship is used in combination, the accuracy of the extraction result can be improved. This is because the extraction result may include noise only by comparing the distance between the feature point of the captured image and the feature point of the original image.
“Determining whether to save the positional relationship” means taking out one corresponding feature point from each of the set of feature points of the photographed image determined to be close in distance and the set of feature points on the candidate image side, Based on the vector of the center of gravity and the extracted feature points, the other feature points are examined on the left or right side. If the corresponding feature points extracted from each set are on the same side, the points are scored. If the value is greater than or equal to the value, it is determined that the positional relationship is stored.

Since the process from image capture to hit image search is processed in a short time, it is possible to access relevant information as soon as any image is captured by the mobile terminal.
Therefore, the present invention is effective when a salesperson uses it as a sales support tool to search for products at a customer and obtain related information. For example, a sales representative makes a presentation using this system at a customer site. Business materials that are a mixture of conventional images and related information become thick catalogs that cannot be easily carried around, and are apt to cause confusion and omissions. Therefore, the information is divided into one's own mobile terminal and a simple pamphlet. By using both of these in a mutually complementary manner, business efficiency can be improved.
In the present invention, the database stores not the image itself but its feature points. As a result, the copyright problem can be avoided and the business related to privacy can be handled.

It is a figure which shows the system configuration | structure and functional block of 1st Embodiment. It is a flowchart which shows the process outline | summary of 1st Embodiment. It is a figure which shows the data structure stored in the original image feature point database of 1st Embodiment. It is a flowchart explaining the feature point collation process of 1st Embodiment. It is a figure which shows the system configuration | structure and functional block of 2nd Embodiment. It is a flowchart which shows the process outline | summary of 2nd Embodiment. It is a figure for demonstrating the moving image search process which is an application example of 2nd Embodiment. It is a figure which shows the example of storage data of the original image relevant-information storage means referred by the moving image search process which is an application example of 2nd Embodiment.

<< First Embodiment >>
Hereinafter, a system according to an embodiment of the present invention (hereinafter, “the present system”) will be described with reference to the drawings.

As shown in FIG. 1, this system includes a mobile terminal 1 used by a user, a server 2 that provides an image search service, and an appropriate external Web server 3, each of which is connected via a communication network such as the Internet N. Connected.

The portable terminal 1 is a portable information processing device such as a smartphone.
The portable terminal 1 includes an input unit 4, an output unit 5, an imaging unit 6, a storage unit 7, a processing unit 8, and a communication interface unit (not shown).

The input unit 4 includes a touch panel arranged on the screen of the output unit 5. Instructions for starting and ending the feature point extraction program, access to the Web server 3 and the like are performed through the input unit 4.
A display screen is essential for the output means 5, and a speaker is also included as appropriate.
The image pickup means 6 is a camera lens and an image pickup device , and such an image shooting function is indispensable for a portable terminal used in this system.

The storage means 7 stores a computer program that implements various processes by the processing means 8, parameters necessary for executing these programs, intermediate results of the processes, and the like. The portable terminal used in this system must have a memory necessary for executing a program for extracting feature points of a captured image.

The processing unit 8 includes a feature point extraction unit 9, an image search request transmission unit 10, an image search result reception unit 11, and an image related information acquisition unit 12.
The feature point extraction unit 9 extracts feature points from the captured image displayed on the screen 5 by the imaging unit 6.
The image search request transmission unit 10 transmits the feature points of the extracted captured image to the server 2. This feature point serves as a search key for image search. The search key is a feature point, not the captured image itself. This is because if an image is transmitted directly, there is a risk of infringement of privacy if a copyright problem occurs or if a person or person's house is shown.
If the feature point determined to correspond to the search key in the server 2 is extracted, the image search result receiving unit 11 is related to information related to the image corresponding to the feature point, that is, an image taken by the user. As possible information is transmitted, it is received. For example, if an entrance of a certain store is photographed, the URL of the store's website may be received.
The image related information acquisition unit 12 is a unit for accessing information related to an image taken by the user based on information transmitted from the server 2. For example, if a URL is transmitted from the server 2, the corresponding Web site is accessed based on the URL.

The classification of the means 9 to 12 included in the processing means 8 is for convenience of explanation, and the means are not clearly separated. These means are realized by the portable terminal 1 mounting a predetermined program. That is, it is assumed that this system is provided to the user as application software (application) for portable terminals, for example, in the form of an APK file.

The server 2 is an information processing apparatus having a storage unit 13, a processing unit 14, and an input / output unit and a communication interface unit (not shown).

The storage unit 13 includes an original image information storage unit 15 that stores information to be searched, a memory (not shown) that stores intermediate results of various processes, a computer program storage unit, and the like.
The original image information storage means 15 includes an original image feature point database (hereinafter referred to as “feature point DB”) 16, a feature point index database (hereinafter referred to as “index DB”) 17, and an original image related information database (hereinafter referred to as “related information”). DB ") 18. These databases will be described later.

The processing unit 14 of the server 2 includes a feature point creating unit 19, a search key receiving unit 20, a feature point collating unit 21, and a hit image information transmitting unit 22.
The feature point creation means 19 creates the feature points of the original image and the index for search during matching, and registers them in the feature point DB 16 and the index DB 17.
The search key receiving unit 20 receives feature point information that is an image search key from the mobile terminal 1.
The feature point collating means 21 collates the received search key with the feature points of the original image registered in the feature point DB 16 in advance, and sets the original image having the most closest feature points as the hit image.
The winning image information transmitting unit 22 extracts information related to the winning image, for example, a URL from the related information DB 18 and transmits the information to the mobile terminal 1.

Next, the operation of this system will be described with reference to FIG.
On the server 2 side, a feature point of the original image and an index for searching during collation are created (step S1). This process is performed independently of the processes in and after step S2, is performed before the system operation, and is appropriately updated after the system operation is started.
For the feature point extraction, for example, a known ORB ( Oriented FAST and Rotated BRIEF ) algorithm is used.
(See http://www.willowgarage.com/papers/orb-efficient-alternative-sift-or-surf for details)
The feature points of thousands to tens of thousands of original images are taken and stored in the feature point DB 16 and the index DB 17. Since feature points are extracted in this way, feature point matching processing can be speeded up.

Feature point information is stored in the feature point DB 16 by the number of original images. As shown in FIG. 3, the data structure includes, for each original image, an image ID to which the feature point belongs (int or hash character string of the original image), the number of feature points acquired from the original image (there is a width depending on the image), It has the value of a feature point vector (32 int values).
Furthermore, it also has the number of feature points and feature point vector values acquired from the reduced image, and the number of feature points and feature point vector values acquired from the enlarged image.
It should be noted here that the original image itself has no data. There are 16 feature point vectors, each having a set of X and Y coordinate values, so there are a total of 32 feature point vector values for each feature point. The speed and accuracy of the search can be improved by taking the vectors as fine as 16 directions and by setting the value type of the feature point vector to int.
This embodiment uses an ORB algorithm, but this algorithm has the disadvantage of being vulnerable to size changes. In order to compensate for this defect and maintain accuracy, not only the original image but also the feature points of the reduced image and the enlarged image are stored in the database at the same time. The feature points of the reduced image and the enlarged image are also necessary because the number of feature points and the extracted feature points differ depending on the resolution even if the subject is the same.

  For the creation of the index DB 17, a flann (Fast Library for Proximity Nearest Neighbors) algorithm is used. The flann algorithm is a high-speed approximate calculation method for K-neighbor search related to high-dimensional features, and based on this, an index tree is created, and matching is executed along this tree. Specifically, OpenCV (Open Source Computer Vision Library) is used, but since it is a publicly known function usage level, its details are omitted.

The related information DB 18 stores information related to the image in association with the image ID (or hash character string) to which the feature point belongs.

Next, the portable terminal 1 extracts feature points from the image displayed on the screen with the camera lens held up (step S2). The number of feature points per image may be several hundred. The extraction algorithm is exactly the same as in step S1.
The extracted feature points are transmitted to the server 2 as image search keys (step S3).

The server 2 collates the received feature point with the registered feature point of the original image (step S4).
The feature point matching process will be described with reference to FIG.
The number of original images is J, and a loop variable j is initialized to 1 in step S101. Next, the feature point data of the original image is extracted from the feature point DB 16 (step S102), the number of feature points having the Euclidean distance closest to each feature point of the search key is scored, and stored in the storage means 13 as an intermediate processing result. (Step S103). The information to be stored is which feature point of the search key corresponds to which feature point of which original image, and is referred to in the determination process of the positional relationship storage. In step S104, it is determined whether j = J. If j = J is not satisfied, 1 is added to the loop variable j (step S105), and the process returns to step S102 and is repeated in the same manner.
If j = J, K original images having the highest score are extracted as candidate images from the intermediate processing result (step S106). So far, noise removal is not considered. That is, since all data is processed as signals (= original data), more candidate images are employed.

The following processing focuses on removing noise. There are various methods for removing noise. In the present invention, it is performed by determining whether or not the positional relationship is stored.
This process can be said to be a process for checking the correspondence relationship more globally. Since it is thought that there is no global law for noise, images that are adopted because of noise will be eliminated at this stage.
First, in step S107, the loop variable k is initialized to 1. This k is counted up in the subsequent stage until the number of adoption candidate images set in the previous stage = K. At the same time, in step S107, the counter variable i is initialized to zero.
In step S108, the extracted original images are checked for the storage of positional relationships in order of highest score. That is, the feature point of the photographed image photographed by the mobile terminal 1 is associated with the feature point in the specific candidate image on a one-to-one basis. In the feature point set associated here, the feature point set on the photographed image side is set as subset A, and the feature point set on the candidate image side is set as subset B. The subset A is a set of feature points of the search key stored as an intermediate processing result in step S103, and the subset B is a set of feature points of a certain original image.
For each of subset A and subset B, centroids Ga and Gb are calculated from the positions of the feature points.
Next, two points are taken from the subset A side to be point 1A and point 2A. Points in subset B corresponding to point 1A and point 2A are point 1B and point 2B. It is checked whether the left and right positions of the point 2A with respect to the center of gravity Ga-> point 1A and the left and right positions of the point 2B with respect to the center of gravity Gb-> point 1B are the same. This is examined for all combinations of two points, and the ratio of the number of matching numbers to the whole is calculated. If this ratio is equal to or greater than a preset threshold value, it is determined that the positional relationship is stored. That is, the candidate image has also passed the noise check.

If the positional relationship is stored (Yes in step S108), the counter variable i is incremented, and this original image is stored as an intermediate processing result (step S109). Subsequently, it is determined whether k = K (step S110). If k = K is not satisfied, 1 is added to the loop variable k (step S111), and the process returns to step S108 and the same process is repeated.
If k = K (Yes in step S110), it is determined whether or not there is an original image that has passed the positional relationship storage check (step S112). If there is a passed original image (Yes in step S112), the i original images having the highest matching rate in step S108 are determined as hit images (step S113).
If there is no passed original image (No in step S112), it may be considered that there is no winning image, but the winning image may be presented to the user as follows (step S114). That is, from K pieces up to a predetermined upper limit value K2 is regarded as a hit image (K>K2> = 1). Rules are determined in advance for the criteria for selecting K2. Whether importance is attached to the score obtained in step S103, or whether the matching rate obtained in step S108 is important.

The server 2 extracts information related to the hit image from the related information DB 18 and transmits the information to the portable terminal 1 (step S5).
The portable terminal 1 receives provision of information related to the photographed image based on the information received from the server 2 (step S6). For example, if the provided information is a URL, the Web server 3 is accessed based on the URL, a Web page is acquired and displayed on the screen.
In this way, this system can acquire and display related information on the spot for images taken with the camera of a mobile terminal, so it is expected to be used in various situations such as business, education, and entertainment. .

<< Second Embodiment >>
The system according to the second embodiment of the present invention will be described below.
Compared with the first embodiment, this system performs shooting of an image, extraction of feature points, collation with feature points of a registered original image, and extraction of a hit image ID in a portable terminal. It differs from the first embodiment in that only information inquiry is performed.
The differences from the first embodiment will be mainly described below with reference to the drawings. In the figure, components having the same functions as those of the first embodiment are denoted by the same reference numerals.

<< 1. Configuration of this system >>
As shown in FIG. 5, the system includes a mobile terminal 101 used by a user, a server 102 that provides an information providing service using image search, and an external Web server 3 as appropriate. Connected via a communication network.

The portable terminal 101 is a portable information processing device such as a smartphone.
The portable terminal 101 includes an input unit 4, an output unit 5, an imaging unit 6, a storage unit 103, a processing unit 104, and a communication interface unit (not shown).

The storage unit 103 includes an original image information storage unit 105 that stores information to be searched, a memory (not shown) that stores intermediate results of various processes, a storage unit for computer programs, and the like. The portable terminal used in this system must have a memory necessary for executing a program for extracting feature points of a captured image and a program for matching feature points.
The original image information storage unit 105 includes an original image feature point database (hereinafter, “feature point DB”) 106. The feature point DB 106 will be described later.

The processing unit 104 includes a feature point extracting unit 107, a feature point collating unit 108, a hit image information acquiring unit 109, and an image related information acquiring unit 110.
The feature point extraction unit 107 extracts feature points from the photographed image displayed on the screen 5 by the imaging unit 6.
The feature point collating means 108 collates the feature points extracted from the photographed image with the feature points of the original image registered in advance in the feature point DB 106, and sets the original image having the most closest feature points as the hit image.
The winning image information acquisition unit 109 requests and receives information related to the winning image from the server 102.
The image related information acquisition unit 110 is a unit that accesses information related to an image captured by the user based on information transmitted from the server 102. For example, if a URL is transmitted from the server 102, the corresponding Web server 3 is accessed based on the URL, and the acquired Web page is displayed on the screen 5.

The classification of the respective means 107 to 110 included in the processing means 104 is for convenience of explanation, and the respective means are not clearly separated. These means are realized by the portable terminal 101 mounting a program provided in the form of an APK file. That is, it is assumed that this system is provided to the user as application software (application) for portable terminals. However, unlike the first embodiment, the feature point data of the original image is also provided with the application.

The server 102 is an information processing apparatus having a storage unit 111, a processing unit 112, and an input / output unit and a communication interface unit (not shown).

The storage unit 111 includes an original image related information storage unit 113 that stores information related to the original image.
The processing unit 112 includes a hit image information transmitting unit 114 that transmits information related to the hit image requested from the mobile terminal 101.

Next, the operation of this system will be described with reference to FIG.
The feature point DB 106 storing the feature points of the original image is stored in the storage unit 103 together with the application (step S201). The feature point DB 106 is stored independently of the processes in and after step S202, and is performed when the application is installed, and can be updated as appropriate. Thus, since the feature point set is prepared in advance, the feature points can be collated in real time.

The algorithm for feature point extraction is the same as in the first embodiment.
However, considering the memory capacity of the mobile terminal 101 and the CPU capacity, the number of original images is appropriately about 50-60.
In this embodiment, unlike the first embodiment, the index DB is not created in advance. Since the number of original images is as small as about 50, when collating with feature points of a captured image, an index is created and stored in the memory when feature point data of the original image is read.

The feature point DB 106 stores feature point information for the number of original images. The data structure is the same as in the first embodiment, and as shown in FIG. 3, for each original image, the image ID to which the feature point belongs (int or the hash character string of the original image), and the feature points acquired from the original image And feature point vector values (32 integer values). Furthermore, it also has the number of feature points and feature point vector values acquired from the reduced image, and the number of feature points and feature point vector values acquired from the enlarged image.

Feature points are extracted from the captured image projected on the screen with the camera lens held up (step S202). The extraction algorithm is exactly the same as in step S201, and the number of feature points is about several hundred.
The feature point of the captured image is collated with the registered feature point of the original image (step S203).
After extracting original image data having a number of closest feature points with respect to each feature point as a candidate image, determination of saving of positional relationship is performed for each candidate image, and a winning image is determined (step S204). This is the same as the first embodiment.

The ID of the hit image is transmitted to the server 102 side (step S205).
The server 102 retrieves the information of the hit image corresponding to the received ID from the original image related information storage unit 113 and transmits it (step S206).
The content is displayed according to the information of the hit image received from the server 102 (step S207).

As an application example of the second embodiment, a case where an employee in charge of sales has the portable terminal of this embodiment will be described.
Normally, sales representatives bring pamphlets with product information to customers, but these pamphlets vary from customer to customer. Otherwise it will be a thick catalog. It also changes over time (price, availability, URL of information site, etc.). You can't create a new brochure each time. Therefore, the present invention is utilized.
First, a feature point DB 106 is created by photographing products (about 50) that are likely to be relevant to the customer and stored in the portable terminal 101 of the customer. Of course, you can use what you have already created,
You may make it compact. Alternatively, it may be created immediately before visiting the customer in a flexible manner. Thus, when the system of the present invention is mounted on a portable terminal, it becomes an extremely handy sales support tool. Note that when capturing an original image, three types of images, a normal image, a reduced image, and an enlarged image are taken in consideration of later image matching. You can also print it to create a temporary brochure for the customer. The product pamphlet is made the original image because it is the least likely to change (the pamphlet will not be updated as often as new models are added). Changes in related information (e.g., website URL) may be absorbed only by one's own portable terminal regardless of the pamphlet.
The sales representative brings the pamphlet and mobile terminal to the customer. When the customer explains the brochure, the user takes a picture while holding the camera lens of the portable terminal 101 over the brochure. The processing means 104 of the portable terminal 101 extracts feature points and collates with stored feature points, and recognizes which pamphlet has been photographed. Display more detailed information related to the recognized brochure on the screen.
In this way, the sales representative can give an appropriate and detailed explanation or answer the customer's questions even if he / she only brings the portable terminal 101 and a pamphlet of about 50 products at most. Become.

Another utilization example of the second embodiment is a search for moving images.
Here, a moving image including still images F1, F2, F3, and F4 that appear in the order shown on the left side of FIG. 7 will be described as an example. In the case of a moving image, the feature point data of each still image is stored in the feature point DB 106. The feature point DB 106 stores feature point data of these four images, and each image is associated with a unique image ID (or hash character string). Note that the feature point DB 106 is not set with information for distinguishing between still images and moving images. If the imaging means 6 of the portable terminal 101 is held over this moving image, similar still images U1 to U4 that are not identical to the still images F1 to F4 can be obtained, and feature points extracted from these still images are stored in the feature point DB 106. Match with the feature point data of. As a result, the winning images are extracted in the order of F1, F2, F3, and F4. The portable terminal 101 transmits the image IDs of these hit images to the server 102.
The server 102 refers to the original image related information storage unit 113 as shown in the data storage example in FIG. 8, extracts the related information of the moving images corresponding to F1 to F4, and transmits it to the mobile terminal 101.
As shown in FIG. 8, the moving image (D1) is displayed on the screen of the mobile terminal 101. When the related information is received from the server 102, the screen display is changed to the related information (D2). The mobile terminal 101 can collect information related to the moving image by accessing the Web server 3 based on the URL displayed on the screen.
In FIG. 8, one ID is registered in the image ID column when the original image is a still image. In each process of feature point extraction and feature point matching in the portable terminal 101, it is not possible to distinguish whether the captured image is a still image or a moving image, and whether the still image is a moving image or not can be distinguished based on the image ID received on the server 102 side.

The processing flows and database structures of the first and second embodiments described above are merely examples, and are not limited thereto. For example, the ORB is used as the feature point extraction algorithm, but the present invention is not limited to this.
In the second embodiment, the server is inquired about information related to the hit image. However, the related information is also stored in the portable terminal in association with the original image. It may be displayed on the screen with reference.
Furthermore, the moving image search function of the present invention can be used, for example, to identify a television program currently being broadcast. On the server side, feature points are sequentially extracted from the images of the programs of each television station and stored in a queue. On the other hand, a user watching a certain TV program takes a picture with the camera function of the mobile terminal, extracts a feature point, and transmits it to the server. The server collates the received feature point with the feature point data stored in the queue, and identifies which TV station is the program currently being broadcast. This function of specifying a television program can be considered to be used in various fields including audience rating estimation.

Since information related to images taken by the camera function of the mobile terminal can be displayed on the screen in real time, it is particularly effective as a sales support tool at customers. Moreover, the possibility of a portable terminal can be further expanded by giving the portable terminal a role like an information magazine.

1: mobile terminal, 2: server 2, 3: Web server,
4: input means, 5: output means 6: imaging means, 7: storage means, 8: processing means,
9: Feature point extracting means, 10: Image search request transmitting means, 11: Image search result receiving means,
12: Image related information acquisition means, 13: Storage means, 14: Processing means,
15: Original image information storage means,
16: Original image feature point database (feature point DB),
17: Feature point index database (index DB),
18: Original image related information database (related information DB),
19: feature point creating means, 20: search key receiving means, 21: feature point collating means,
22: Hit image information transmission means 101: Mobile terminal, 102: Server, 103: Storage means, 104: Processing means,
105: Original image information storage means, 106: Original image feature point database (feature point DB),
107: feature point extraction means, 108: feature point collation means, 109: hit image information acquisition means, 110: image related information acquisition means, 111: storage means, 112: processing means,
113: Original image related information storage means, 114: Hit image information transmission means,
N: Internet

Claims (2)

Input means for accepting user input;
Output means for displaying a still image or a moving image (hereinafter referred to as “captured image”) obtained by the imaging means;
Feature point extraction means for extracting feature points of the displayed captured image;
An original image feature point database in which feature points of pre-collected image groups are stored;
A feature point collating unit that collates the feature points of the extracted captured image with the feature points extracted from the original image feature point database, and extracts information that specifies an image that satisfies a condition (hereinafter referred to as “winning image”);
Image-related information acquisition means for acquiring information related to the winning image based on the information specifying the winning image ,
When the moving image is a feature point extraction target, the feature point extracting unit takes a plurality of still images, extracts the feature points of each still image, collects the feature points of a series of still images, and collects the feature points of the moving image. An image search system characterized by a point . The server and mobile device are connected via a communication network,
The portable terminal is
Input means for accepting user input;
Output means for displaying a photographed image obtained by the imaging means;
Feature point extraction means for extracting feature points of the displayed captured image;
Image search request transmission means for transmitting the feature points of the extracted captured image to the server and requesting the identification of the hit image;
An image search result receiving unit for receiving a search result, an image related information acquiring unit for acquiring related information based on information for identifying a hit image,
The feature point extraction means takes a plurality of still images, extracts the feature points of each still image, collects the feature points of a series of still images, As feature points of the image,
The server
An original image feature point database storing feature points of the original image;
A feature point matching unit that compares feature points received from the portable terminal with feature points extracted from the original image feature point database, and extracts information for identifying a hit image;
An image search system comprising:

Images