JP6974222B2 - Search device, search method and search program - Google Patents

Description

The present invention relates to a search device, a search method, and a search program.

Conventionally, there has been known a technique for searching information similar to a search query by comparing the feature amount of the information to be a search query with the feature amount of the information to be searched. As an example of such a technique, in addition to the similarity between a plurality of local features extracted from an image to be a search query and a plurality of features of the image to be searched, the range in which the features are extracted. A technique for searching an image similar to an image used as a search query based on the degree of similarity of coordinate relationships is known.

Japanese Unexamined Patent Publication No. 2014-157443

However, in the above-mentioned conventional technique, it may not be possible to say that the similar search is efficiently performed.

For example, in the above-mentioned conventional technique, characteristic feature points are selected from a search query or an image to be searched, and a feature amount is extracted based on the information around the extracted feature points. However, even if the same image pickup target is included in the search query or the image to be searched, the value of the position and feature amount selected as the feature point changes depending on the shooting conditions such as the shooting direction and angle. Resulting in. As a result, it may be determined that the search query containing the same imaging target and the search target are not similar, or the search query containing different shooting targets and the search target may be determined to be similar. In addition, as the number of feature quantities extracted from search queries and search targets increases, the search cost required for similar searches increases.

The present application has been made in view of the above, and an object thereof is to improve the efficiency of similar search.

The search device according to the present application has an extraction unit for extracting useful feature quantities useful for searching similar information among feature quantities extracted from query information or information to be searched, and a usefulness extracted by the extraction unit. It is characterized by having a search unit for searching information similar to query information using a feature amount.

According to one aspect of the embodiment, the efficiency of similar search can be improved.

FIG. 1 is a diagram showing an example of a search process executed by the information providing device according to the embodiment. FIG. 2 is a diagram showing the relationship of processing executed by the information providing device according to the embodiment. FIG. 3 is a diagram showing a configuration example of the information providing device according to the embodiment. FIG. 4 is a diagram showing an example of information registered in the search target database according to the embodiment. FIG. 5 is a flowchart showing an example of the flow of the search process executed by the information providing device according to the embodiment. FIG. 6 is a diagram showing an example of a hardware configuration.

Hereinafter, a search device, a search method, and a mode for carrying out the search program (hereinafter, referred to as “embodiment”) according to the present application will be described in detail with reference to the drawings. Note that this embodiment does not limit the search device, search method, and search program according to the present application. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description will be omitted.

[Embodiment]
[1-1. About an example of information providing device]
First, with reference to FIG. 1, an example of a search process executed by an information providing device, which is an example of a search device, will be described. FIG. 1 is a diagram showing an example of a search process executed by the information providing device according to the embodiment.

In FIG. 1, the terminal device 100 and the information providing device 10 used by each user can communicate with each other via a predetermined network N (see, for example, FIG. 3) such as the Internet. Further, it is assumed that the information providing device 10 can communicate with various external servers other than the terminal device 100. Further, the information providing device 10 may be able to communicate with any number of terminal devices 100.

The terminal device 100 is a smart device such as a smartphone or tablet, and is a mobile terminal device capable of communicating with an arbitrary server device via a wireless communication network such as 3G (3rd Generation) or LTE (Long Term Evolution). be. The terminal device 100 may be not only a smart device but also an information processing device such as a desktop PC (Personal Computer) or a notebook PC.

The information providing device 10 is an information processing device that executes a search process, and is realized by, for example, a server device, a cloud system, or the like. For example, when the information providing device 10 receives an image to be a search query from the terminal device 100, the information providing device 10 searches for an image having contents similar to the received query image, and the searched image or information indicating the searched image (for example, a URL). (Uniform Resource Locator), thumbnails, etc.) are provided as search results to the terminal device 100, and a search process is executed.

To give a more specific example, the information providing device 10 holds an image showing a transaction target in an electronic shopping district. For example, the terminal device 100 holds an image of a product to be traded and an image related to a service to be traded (hereinafter, collectively referred to as “image of transaction target”). Such an image of a transaction target is, for example, an image registered by a seller who sells the transaction target in an electronic shopping district.

For example, when the information providing device 10 receives the query image from the terminal device 100 as a search query for searching the transaction target exhibited in the electronic shopping district, the information providing device 10 searches for an image of the transaction target similar to the received query image. .. Then, the information providing device 10 uses the terminal device 100 as a search result that summarizes the content explaining the transaction target corresponding to the searched image (that is, the transaction target captured in the image) and the link to the content for purchasing the transaction target. Send to.

[1-2. Search using features]
Hereinafter, as the search process executed by the information providing device 10, the flow of the process of searching for an image similar to the query image among the images to be traded will be described. In the following description, the images of the transaction target to be searched are collectively referred to as the search target images. In the following description, an example of a process of searching for a search target image similar to a query image by using a feature amount will be described, but the embodiment is not limited to this. As will be clarified in the explanation described later, if the information providing device 10 searches for a search target image similar to the query image by comparing the feature quantities with each other, the information providing device 10 is similar to the query image by any search method. You may search for the image to be searched.

For example, in a search using feature points, a plurality of characteristic points are selected as feature points from the search target image by the processing, and a range for calculating the feature amount is set for each feature point. Will be done. For example, the range in which such a feature amount is calculated is set so that the feature amount does not change so much even when various affine transformations are performed. As a result, each feature point becomes an irregular position, and the range (radius) in which the feature amount is calculated is also various. In FIG. 1, the range in which the feature amount is calculated is simulated by a dotted circle, but in reality, an arbitrary shape such as an ellipse or a rectangle is adopted as the range in which the feature amount is calculated. ..

In the following description, each feature amount calculated from the search target image is described as "feature amount of the search target image". Subsequently, the information providing device 10 similarly calculates the feature amount in the range based on the automatically selected characteristic point, range (radius), and direction for the query image. In the following description, each feature amount calculated from the query image is described as "feature amount of the query image".

The features of the above-mentioned search target image and query image include wide-area features such as GIST (http://ilab.usc.edu/siagian/Research/Gist/Gist.html), color histogram, and color distribution, and SIFT. (Scale-Invariant Feature Transform), SURF (Speed-Up Robust Features), feature amount, and any other technique for acquiring the feature amount of the image can be adopted. The features generated by the neural network can also be used.

Then, the information providing device 10 specifies a feature amount similar to the feature amount of the query image among the feature amounts of the search target image by comparing the feature amount of the query image and the feature amount of the search target image. .. For example, the information providing device 10 uses a technique such as NGT (Neighborhood Graph and Tree) for each feature amount of the query image to determine whether or not it is similar to the feature amount of each search target image. Then, the information providing device 10 associates the feature amount of the query image with the feature amount determined to be most similar to the feature amount of the query image among the feature amounts of each search target image.

Then, the information providing device 10 selects, among the search target images, a search target image in which the number of feature amounts associated with the feature amount of the query image exceeds a predetermined threshold value as a similar candidate for the query image. That is, the information providing device 10 selects a search target image similar to the query image as a candidate for the similar image in a voting format based on the similarity between the feature quantities. In the following description, the process of determining whether or not the above-mentioned feature amounts are similar to each other may be described as "feature amount collation process".

Here, a method of outputting a candidate of a similar image as a search result as it is can be considered, but the information providing device 10 executes the coordinate matching process described below in order to further improve the accuracy. For example, the information providing device 10 specifies a feature amount associated with each other among the feature amount of an image that is a candidate for a similar image (hereinafter, referred to as “candidate image”) and the feature amount of the query image. do.

Subsequently, the information providing device 10 performs an affine transformation (for example, rotation, translation, enlargement or reduction) of the coordinates of the feature points of each feature amount specified from the query image, and the information providing device 10 of each feature amount specified from the candidate image. Determine if it is similar to the coordinates. Then, when the information providing device 10 determines that the coordinates of the feature points of the query image and the coordinates of the feature points of the candidate image are similar, the information providing device 10 determines that the query image and the candidate image are similar.

That is, in the feature amount matching process described above, the information providing device 10 determines whether or not the feature amount of the search target image and the feature amount of the query image are similar, and the query image is obtained by voting based on the determination result. Some of the feature points possessed are associated with some of the feature points of the plurality of search target images. Then, the information providing device 10 collates the coordinates of the associated feature points by executing the coordinate matching process using some of the feature points associated with each other in the coordinate matching process described above. That is, the information providing device 10 determines whether or not the coordinates of the feature points of the query image match (or are similar to) the coordinates of the feature points of the search target image when the coordinates of each feature point of the query image are affine-transformed. It is determined whether or not the coordinate relationships between the corresponding feature quantities are similar based on whether or not the number exceeds a predetermined number.

In the coordinate matching process, the information providing device 10 re-compares all the feature amounts of the query image with all the feature amounts of the candidate images, and the feature amounts of the query images and the feature amounts of the candidate images that are similar to each other are re-compared. You may redo the correspondence with. Then, the information providing device 10 may determine whether or not the query image and the candidate image are similar based on the positional relationship between the newly associated feature quantities. As a result of such processing, the information providing device 10 can improve the search accuracy of similar images.

[1-3. About the outline of the search process]
However, in the feature quantity matching process and the coordinate matching process described above, it may not be possible to say that the similarity search is properly performed. For example, a process of extracting a feature amount from a plurality of images including the same image pickup target will be described. For example, when a feature point is extracted from a plurality of images including the same image pickup target, a characteristic location of the image pickup target can be selected as the feature point. For example, a logo attached to a transaction target or a position with a characteristic shape or color, that is, a position that better indicates the characteristics of the transaction target can be easily selected as a feature point. However, even if a position that better indicates the characteristics of the transaction target is extracted as a feature point, if the shooting position is different, the position and angle at which the transaction target is imaged, the presence or absence of a background, the difference in lighting, etc. Therefore, the feature amount may change.

To give a more specific example, an example of extracting a feature amount from a plurality of images of a predetermined position of a product to be traded taken from different directions will be described. For example, when a part having a characteristic color or shape of a product to be traded and which can stably indicate the transaction target from any angle is selected as a feature point. Even if the direction and lighting when the product is photographed are different, it is considered that similar features are extracted. On the other hand, when a part having a characteristic color or shape of the product to be traded and the part whose feature amount changes depending on the shooting direction is selected as a feature point, when the product is photographed. It is considered that dissimilar feature quantities are extracted when the directions and illuminations of the products are different.

That is, the feature amount extracted from the image includes a feature amount that stably shows the characteristics of the transaction target and a feature amount that is difficult to stably show the characteristics of the transaction target. As described above, the feature amount that stably shows the characteristics of the transaction target is considered to be a feature amount useful for a semantic search. For example, when comparing feature quantities that stably indicate the characteristics of a transaction target, it is considered possible to search for an image in which a transaction target having similar characteristics to the transaction target captured in the query image is captured.

On the other hand, a feature amount for which it is difficult to stably show the characteristics of a transaction target may be similar to the characteristics of a transaction target different from the transaction target. As a result, when a search using a feature amount for which it is difficult to stably show the characteristics of the transaction target is performed, it is considered that a semantic search cannot be performed accurately.

In addition, a plurality of feature points are selected from the query image and each search target image. Therefore, when all the feature quantities extracted from the image are compared with each other, the time required for the search and the calculation cost increase.

Therefore, the information providing device 10 executes the following search process. First, the information providing device 10 extracts a useful feature amount useful for searching for similar information from the feature amounts extracted from the query information or the information to be searched. Then, the information providing device 10 searches for information similar to the query information by using the extracted useful feature amount. For example, the information providing device 10 performs a similar image search using a useful feature amount useful for searching, such as a feature amount that stably indicates the feature of the transaction target.

For example, the information providing device 10 roughly extracts useful feature quantities from the feature quantities extracted from the information to be searched. For example, the information providing device 10 extracts in advance useful feature amounts useful for searching for similar images from the feature amounts extracted from the search target image. To give a more specific example, the information providing device 10 has a feature amount estimated to be useful for searching for a photographed transaction target, that is, an image having a similar image capture target (that is, a feature possessed by the image capture target). (Stable features) are extracted as useful features. Then, the information providing device 10 searches for an image similar to the query image by using the extracted useful feature amount.

[1-4. About useful features]
Here, the information providing device 10 may extract a feature amount satisfying an arbitrary condition as a useful feature amount as long as it can extract a feature amount useful for searching for similar images. For example, a useful feature amount may be extracted by comparing the feature amount extracted from the search target image with the feature amount detected from a similar image similar to the search target image.

For example, consider the case where a certain search target image # A exists as an image similar to a certain query image # A. In such a case, it is considered that the feature amount extracted from the query image #A includes a feature amount similar to the feature amount extracted from the search target image #A. Therefore, the information providing device 10 acquires a similar image of the search target image, and among the feature amounts extracted from the search target image, the feature amount whose similarity with the feature amount extracted from the similar image satisfies a predetermined condition. Is extracted as a useful feature amount. More specifically, the information providing device 10 is based on a comparison result between a feature amount extracted from a plurality of information to be searched and a feature amount extracted from similar information similar to any one of the information. Based on this, useful features are extracted from the features extracted from the information to be searched.

Hereinafter, an example of a process in which the information providing device 10 extracts useful feature quantities will be described. For example, the information providing device 10 selects one of the search target images as the processing target from the database in which the search target images are registered. Then, the information providing device 10 generates an image similar to the extracted search target image, that is, a similar image. To give a more specific example, the information providing device 10 generates a similar image that can be recognized as having a transaction target that is the same as or similar to the transaction target captured in the extracted search target image.

For example, the information providing device 10 performs image analysis of the selected search target image, identifies the range in which the background is photographed and the range in which the main photographing target is photographed, deletes the range in which the background is photographed, or deletes the range in which the background is photographed. You may combine different backgrounds. Further, the information providing device 10 may generate an image obtained by performing an affine transformation of the range in which the imaging target is captured, such as an image in which the imaging target is rotated, enlarged, reduced, or deformed. Further, the information providing device 10 may generate an image to which various affine transformations are applied to the entire search target image. Further, the information providing device 10 may generate an image in which each range or the entire image is blurred. Further, the information providing device 10 may generate a similar image by an arbitrary image conversion method such as changing the resolution.

As for how much the changed image is to be a similar image, any setting can be adopted according to the purpose of the similar image search executed by the information providing device 10. That is, if the information providing device 10 is similar to the extent that when an image generated as a similar image is input as a query image, the original search target image can be selected as an image similar to the query image. , An image that has undergone arbitrary image conversion may be adopted as a similar image. Further, the generation of such a similar image can be realized by various image analysis techniques and image generation techniques.

Subsequently, the information providing device 10 extracts the feature amount of the generated similar image, and compares the feature amount extracted from the similar image with the feature amount extracted from each search target image. Then, the information providing device 10 determines whether or not the feature amount extracted from the search target image and the feature amount extracted from the similar image are similar, and for each feature amount extracted from the search target image, The number of times it is determined to be similar to the feature amount extracted from the similar information is counted.

For example, the information providing device 10 sets in advance a comparative use count “0” and a comparative non-use count “0” for each feature amount extracted from each search target image. Subsequently, the information providing device 10 selects one feature amount to be processed from the feature amounts extracted from similar images, and each feature amount extracted from each search target image and the feature amount to be processed are determined. Determine if they are similar or not. Then, the information providing device 10 adds 1 to the value of the number of comparative uses for the feature amount determined to be most similar to the feature amount to be processed among the feature amounts extracted from each search target image. , Of the feature quantities extracted from each search target image, 1 is added to the value of the number of comparison non-uses for the feature quantities that are not determined to be the most similar to the feature quantities to be processed. Further, the information providing device 10 executes the above-mentioned processing for all the feature quantities extracted from similar images.

Further, the information providing device 10 executes the above-mentioned processing for all the search target images. Then, the information providing device 10 extracts a useful feature amount from the feature amount extracted from the search target image based on the counted number of times. For example, among a plurality of feature quantities extracted from a search target image, a feature quantity in which the number of comparative uses is larger than a predetermined threshold value is likely to match the feature quantities of a plurality of similar images similar to the search target image. It becomes a feature quantity. It can be said that such a feature amount is a feature that the search target image and a plurality of similar images have in common. In other words, among the plurality of features extracted from the search target image, the feature amount in which the number of comparative uses is larger than a predetermined threshold value stabilizes the features of the transaction target even if the shooting conditions and the like change. It can be said that it is a feature amount that can be shown, that is, a feature amount that does not easily fluctuate.

When a similar image search is performed using such a stable feature amount, it is considered that the accuracy of the similar image search can be improved. In addition, when feature quantities other than such stable feature quantities are excluded from the targets of similar image search, the number of feature quantities to be compared can be reduced, and as a result, the time and search cost required for similar image search can be reduced. can. Therefore, the information providing device 10 extracts a feature amount in which the value of the number of comparative uses exceeds a predetermined threshold value as a useful feature amount. Then, the information providing device 10 deletes the feature amount other than the extracted useful feature amount. As a result of such processing, the information providing device 10 can more appropriately perform a similar image search.

It is considered that a feature amount in which the value of the number of comparative non-uses exceeds a predetermined threshold value cannot be said to be a feature amount useful for similar image retrieval. Therefore, the information providing device 10 may delete a feature amount whose number of comparative non-uses exceeds a predetermined threshold value. Further, the information providing device 10 may use a feature amount in which the number of comparative uses is larger than a predetermined threshold value and the number of times of comparative non-use is smaller than a predetermined threshold value as a useful feature amount. Further, the information providing device 10 may calculate a score indicating the usefulness of the feature amount from the number of comparative uses and the number of times of comparative non-use, and use the feature amount whose calculated score exceeds a predetermined threshold value as the useful feature amount. Further, the information providing device 10 may use a predetermined number of feature quantities as useful feature quantities in order from the one with the highest value of the number of comparative uses.

[1-5. Counting using coordinate matching processing]
Here, in the above-mentioned example, the information providing device 10 uses a feature amount having a number of times determined to be most similar to the feature amount of a similar image to exceed a predetermined threshold value among the feature amounts of the search target image as a useful feature amount. .. That is, the information providing device 10 extracted the useful feature amount based on the number of times used in the feature amount matching process (the number of times the feature amount of the similar image was matched). However, the embodiments are not limited to this.

For example, the information providing device 10 sets a plurality of feature amounts determined to be similar to the feature amounts extracted from similar images among the plurality of feature amounts extracted from the search target image as the target feature amounts, and extracts them from the similar information. Of the plurality of feature quantities that have been determined, a plurality of feature quantities that are determined to be similar to the target feature quantity are defined as similar feature quantities. Then, the information providing device 10 determines whether or not the positional relationship of the target feature amount and the positional relationship of the similar feature amount are similar, counts the number of times it is determined that the positional relationship is similar, and counts the number of times. You may extract useful features based on. That is, the information providing device 10 may extract useful features based on the number of times used in the coordinate matching process.

For example, the information providing device 10 sets in advance the number of times the coordinates are used "0" and the number of times the coordinates are not used "0" for each feature amount extracted from each search target image. Subsequently, the information providing device 10 executes a feature amount matching process using a plurality of similar images as query images, and for each feature amount extracted from the search target image, the number of times it is determined that the feature amount is most similar to the feature amount of the query image. Is counted as the number of comparative uses. Further, the information providing device 10 associates the feature amount of the query image determined to be similar to each other with the feature amount of the search target image. Further, the information providing device 10 extracts as a candidate image a search target image in which the number of feature amounts associated with the feature amount of the query image exceeds a predetermined threshold value.

Subsequently, in the information providing device 10, the positional relationship of the feature amount associated with the feature amount of the query image in the candidate image is similar to the positional relationship of the feature amount associated with the feature amount of the candidate image in the query image. If it is determined whether or not the image is similar, 1 is added to the number of times the coordinate of the feature amount associated with the feature amount of the query image is used in the candidate image. On the other hand, in the information providing device 10, the positional relationship of the feature amount associated with the feature amount of the query image in the candidate image is not similar to the positional relationship of the feature amount associated with the feature amount of the candidate image in the query image. If is determined, 1 is added to the number of times the feature amount associated with the feature amount of the query image is not used in the candidate image.

Then, the information providing device 10 determines whether or not the feature amount is a useful feature amount according to the number of times the coordinates are used and the number of times the coordinates are not used. For example, the information providing device 10 may use a feature amount in which the number of comparative uses exceeds a predetermined threshold value and the number of times the coordinates are used exceeds a predetermined threshold value as a useful feature amount. Further, the information providing device 10 calculates a score indicating whether or not the feature amount is useful based on the number of times of comparative use, the number of times of comparative non-use, the number of times of coordinate use, and the number of times of coordinate non-use, and the calculated score is A feature amount satisfying a predetermined threshold value may be used as a useful feature amount.

[1-6. About feature matching process]
In the process described above, the information providing device 10 identifies the feature amount most similar to the feature amount extracted from the query image among the feature amounts extracted from the plurality of search target images, and determines the number of comparative uses of the specified feature amount. Counted. However, the embodiments are not limited to this.

For example, the information providing device 10 extracts all the feature amounts extracted from a plurality of search target images and whose similarity with the feature amount extracted from the query image exceeds a predetermined threshold value, and the extracted feature amounts. Add 1 to the number of comparative uses of. Then, the information providing device 10 may use a feature amount in which the number of comparative uses satisfies a predetermined threshold value as a useful feature amount. That is, the information providing device 10 has a feature in which, among the feature quantities extracted from a plurality of search target images, the number of times it is determined that the similarity with the feature quantities extracted from similar images exceeds a predetermined threshold value satisfies a predetermined threshold value. The amount may be a useful feature amount.

When searching for a search target image similar to the query image, the information providing device 10 extracts all the feature amounts whose similarity with the feature amount extracted from the query image exceeds a predetermined threshold value, and retrieves the search target image. Among them, a search target image in which the number of feature amounts determined to have a similarity with the feature amount extracted from the query image exceeds a predetermined threshold may be equal to or higher than the predetermined threshold, and the search target image may be extracted as a candidate image.

[1-7. An example of search processing]
Hereinafter, an example of the search process executed by the information providing device 10 will be described with reference to FIG. For example, the information providing device 10 extracts in advance the feature amount of the search target image to be searched (step S1). For example, the information providing device 10 selects a plurality of feature points PT1-1, PT1-2, ... From the search target image T1 by a predetermined algorithm, and features FT-1, FT for each selected feature point. -2 ... is extracted.

Subsequently, the information providing device 10 extracts a feature amount from a similar image similar to a certain search target image (step S2). For example, the information providing device 10 is an image in which the same transaction target as the transaction target captured in the search target image T1 is captured as a similar image similar to the search target image T1, and is different from the search target image T1. Similar images AT1, AT2, ... It should be noted that such similar images AT1 and AT2 may be images generated from the search target image T1 or may be images provided by a predetermined user as images similar to the search target image T1. good.

Subsequently, the information providing device 10 selects feature points PAT1-1, PAT1-2, ... From the similar image AT1 by using the same algorithm as the algorithm used when the feature points are extracted from the search target image T1. , The feature quantities FAT1-1, FAT1-2, ... Are calculated from each feature point. Similarly, the information providing device 10 selects feature points PAT2-1, PAT2-2, ... From the similar image AT2 by using the same algorithm as the algorithm used when the feature points are extracted from the search target image T1. , The feature quantities FAT2-1, FAT2-2, ... Are calculated from each feature point.

Then, the information providing device 10 performs a similar image search using the similar image as a query image, and counts the number of times the search is used for each feature amount (step S3). More specifically, the information providing device 10 performs a similar image search not only on the search target image T1 but also on all the search target images registered in the database.

For example, the information providing device 10 compares all the feature amounts extracted in step S2 with all the feature amounts of all the search target images registered in the database, and sets the feature amount of the similar image having the highest degree of similarity. Corresponds to the feature amount of the search target image. Then, the information providing device 10 searches for a similar image based on the coordinate relationship between the feature amount of the similar image and the feature amount of the search target image. That is, the information providing device 10 executes the feature amount matching process and the coordinate matching process.

Here, the information providing device 10 counts the number of times the feature amount of the similar image is associated with the feature amount of the similar image in the feature amount matching process for each feature amount of the search target image as the number of comparative uses, and the feature amount matching process of the similar image. The number of times that is not associated with the feature amount is counted as the number of times of comparison and non-use. Further, the information providing device 10 counts the number of times it is determined that the coordinate relationship matches the feature amount of the similar image associated in the coordinate matching process for each feature amount of the search target image as the number of times the coordinates are used, and coordinates. The number of times it is determined that the coordinate relationship does not match the feature amount of the associated similar image in the collation process is counted as the number of times the coordinates are not used. That is, the information providing device 10 performs a coefficient of the number of times used and the number of times not used in the feature amount matching process and the coordinate matching process.

Then, the information providing device 10 extracts useful feature quantities useful for searching based on the number of times counted (step S4). For example, the information providing device 10 associates the type "useful" with a feature amount whose value of the number of comparative uses exceeds a predetermined threshold value, and associates the type "non-useful" with other feature amounts. The information providing device 10 may delete useful feature amounts other than useful feature amounts from the database.

Here, when the query image is acquired from the terminal device 100 (step S5), the information providing device 10 searches for an image similar to the query image based on the comparison result of the feature amount of the query image with the useful feature amount. (Step S6). For example, the information providing device 10 extracts feature quantities FQ1, FQ2, ... From the query image. Then, the information providing device 10 includes feature quantities FQ1, FQ2, ..., Feature quantities FT1-1, 1-3 extracted as useful feature quantities among the feature quantities extracted from the search target image T1, and search targets. Among the feature quantities extracted from the image T2, the feature quantities FT2-4, 2-7, etc. extracted as useful feature quantities are compared. Further, the information providing device 10 executes the feature amount matching process and the coordinate matching process described above based on the comparison result, and searches for an image similar to the query image based on the execution result.

Then, the information providing device 10 transmits the search result to the terminal device 100 (step S7). For example, the information providing device 10 transmits data, thumbnails, and the like of the candidate images T1 and T3 to the terminal device 100. The information providing device 10 may distribute the information of the searched search target image in a ranking format according to the result of the feature amount matching process or the coordinate matching process, for example. For example, the information providing device 10 calculates the accuracy similar to each search target image and the query image based on the result of each collation process, and obtains the information of the searched search target image in a ranking format according to the calculated accuracy. It may be delivered.

[1-8. Processing using the results of search processing]
Here, in the above description, the information providing device 10 generates a similar image from the search target image, and in the similar image search using the generated similar image as the query image, the number of times it is determined to be similar to the feature amount of the query image. Based on the above, useful feature quantities were extracted. However, the embodiments are not limited to this. For example, the information providing device 10 may perform a similar image search similar to the query image received from the terminal device 100, that is, extraction of useful features based on the operation result of the similar image search.

For example, the information providing device 10 searches for information similar to the first query image among the search target images, and the feature amount extracted from the search target image based on the search result of the image similar to the first query image. Extract useful features from. Then, the information providing device 10 may search for an image similar to the second query image by using the extracted useful feature amount. For example, the information providing device 10 determines whether or not the feature amount extracted from the search target image and the feature amount extracted from the first query image are similar, and based on the determination result, the first query image. Search for images similar to. Here, the information providing device 10 extracts a useful feature amount from the feature amount of the search target image similar to the first query image based on the search result, and the second query image is based on the extracted useful feature amount. Search for information similar to.

For example, the information providing device 10 searches for a search target image similar to the query image provided by the user during the operation of the similar image search, and provides the search result to the user. Here, among the search target images indicated by the search results, the image selected by the user is considered to be the image most similar to the query image.

Therefore, the information providing device 10 has a feature amount associated with the feature amount extracted from the query image received from the user among the feature amounts extracted from the search target image selected by the user from the search candidates. (That is, the most similar feature amount) or the feature amount determined to have the similarity exceeding a predetermined threshold value is specified, and 1 is added to the number of comparative uses of the specified feature amount. Then, the information providing device 10 may use a feature amount in which the value of the number of comparative uses exceeds a predetermined threshold value as a useful feature amount. That is, the information providing device 10 has a number of times associated with the feature amount extracted from the query image received from the user among the feature amounts extracted from the search target image selected by the user from the search candidates. , The feature amount in which the number of times the similarity with the feature amount extracted from the query image received from the user is determined to exceed the predetermined threshold exceeds the predetermined threshold may be extracted as the useful feature amount.

[1-9. About the features of useful features]
Here, a predetermined feature may appear in the feature amount extracted as a useful feature amount. Therefore, the information providing device 10 may make the model learn the features of the useful feature amount. For example, the information providing device 10 may make an arbitrary model, such as a DNN (Deep Neural Network) or an SVM (Support Vector Machine), learn a feature having a useful feature amount.

Further, the information providing device 10 may make the model learn the features of the non-useful features. For example, the information providing device 10 extracts a feature amount in which the number of times of comparative non-use or the number of times of coordinate non-use exceeds a predetermined threshold value as a useful feature amount. The non-useful feature amount extracted in this way is a feature amount other than the useful feature amount, which is presumed to be less useful for the search. Then, the information providing device 10 may learn the features unique to the useful feature amount with high accuracy by learning the difference between the useful feature amount and the non-useful feature amount in the model.

It is presumed that the learning model in which such learning is performed can extract useful features from the features. Therefore, the information providing device 10 may extract a useful feature amount from the feature amount extracted from the search target image by using the learning model. For example, the information providing device 10 inputs the feature amount extracted from the search target image into the learning model, determines whether or not the feature amount is a useful feature amount, and only the feature amount determined to be the useful feature amount. May be registered in the database.

Further, the information providing device 10 extracts a useful feature amount from the feature amount extracted from the query image using the learning model, and compares the extracted useful feature amount with the feature amount extracted from the search target image. You may search for an image similar to the query image based on. For example, the information providing device 10 uses a learning model to extract useful features from the features extracted from the query image during operation. Then, the information providing device 10 may search for an image similar to the query image based on the comparison result between the extracted useful feature amount and all the feature amounts extracted from the search target image.

Further, the information providing device 10 is a feature amount extracted from the feature amount of the query image using the learning model, a useful feature amount extracted from the feature amount of the search target image by the various methods described above, or a search target image. An image similar to the query image may be searched based on the comparison result with the useful feature amount extracted from the feature amount of the above using the learning model.

The information providing device 10 may have different useful feature quantities used for image retrieval and useful feature quantities used for model learning. For example, in order to accurately learn the features of the useful features in the model, the probability that the useful features used for learning the model are actually useful is the probability that the useful features used for the image search are actually useful. It is considered better to be higher than. Therefore, the information providing device 10 may individually extract the useful feature amount used for image retrieval and the useful feature amount used for model learning by using different threshold values.

For example, the information providing device 10 determines whether or not the feature amount extracted from the search target image is similar to the feature amount extracted from the similar image. Further, the information providing device 10 counts the number of times it is determined that the feature amount is similar to the feature amount extracted from the similar image for each feature amount extracted from the search target image. Then, the information providing device 10 extracts a feature amount in which the number of times counted is equal to or greater than a predetermined first threshold value as the first useful feature amount, and at the same time, the number of times counted is equal to or greater than the second threshold value larger than the first threshold value. The feature amount is extracted as the second useful feature amount. After that, the information providing device 10 searches for information similar to the query image using the first useful feature amount. On the other hand, the information providing device 10 may make the model learn the features of the second useful feature amount. As a result, the information providing device 10 can generate a learning model that accurately determines whether or not the feature amount extracted from the image is a useful feature amount useful for image retrieval.

[1-10. About the relationship of each process]
Here, with reference to FIG. 2, the relationship between the processes executed by the information providing apparatus 10 will be described. FIG. 2 is a diagram showing the relationship of processing executed by the information providing device according to the embodiment. For example, when the information providing device 10 acquires the query image (step S10), the query feature amount, which is the feature amount of the query image, is extracted (step S11), and the search process is executed (step S12). For example, the information providing device 10 performs a feature amount matching process for comparing a query feature amount with a feature amount of a search target image, and coordinates for comparing the positional relationship of feature amounts similar to each other between the query image and the search target image. The collation process is executed and the similarity judgment is performed based on the execution result.

Here, the information providing device 10 determines the number of times that the feature amount of the search target image is similar to the feature amount of the query image and the coordinates are determined to be similar in the feature amount matching process and the coordinate matching process. The number of times of various uses such as the number of times of use is counted (step S13). Further, the information providing device 10 sets the useful feature amount and the non-useful feature amount of the search target image based on the counting result (step S14). Then, the information providing device 10 extracts a useful feature amount from the feature amounts of the search target image (step S15), and executes a search process using the extracted useful feature amount and the query feature amount (step S12). ..

On the other hand, the information providing device 10 generates a similar image from the search target image (step S16), uses the similar image as a search query (step S17), and executes the search process (step S12). As a result, the information providing device 10 determines that the number of times the feature amount of the image to be searched is similar to the feature amount of the similar image and the coordinates are similar in the feature amount matching process and the coordinate matching process. Various times of use such as the number of times of use can be counted (step S13), and useful feature amounts and non-useful feature amounts can be set based on the count results (step S14).

Further, the information providing device 10 learns the features of the useful feature amount (step S18), extracts the query feature amount from the query image based on the learning result (step S11), and uses the extracted query feature amount. The search process is executed (step S12). More specifically, the information providing device 10 extracts and extracts a feature amount having features similar to the useful feature amount extracted from the search target image as a useful feature amount among the feature amounts extracted from the query image. The search process is executed using the useful feature amount of the query image (step S12).

[1-11. About similar images]
In the above-mentioned example, the information providing device 10 uses a feature amount extracted from the search target image as a useful feature amount in which the number of times determined to be similar to the feature amount extracted from the query image satisfies a predetermined threshold value. And said. As a result of such processing, the information providing device 10 can efficiently search for a search target image similar to the query image.

Here, in the above-mentioned example, as an image similar to the query image, a process for searching a search target image in which the same or similar transaction target as the transaction target captured in the query image is captured is described. For example, the information providing device 10 uses an image in which a transaction target that is the same as or similar to the transaction target captured in the search target image is captured as a similar image, and is similar to the feature amount of the similar image among the feature amounts of the search target image. A feature amount in which the number of times determined to be determined exceeds a predetermined threshold value was defined as a useful feature amount. However, the embodiments are not limited to this.

For example, the information providing device 10 uses an image having a similar appearance to the search target image as a similar image, and among the feature amounts of the search target image, the number of times determined to be similar to the feature amount of the similar image exceeds a predetermined threshold value. By setting the amount as a useful feature amount, a feature amount useful when searching for an image having a similar appearance to the image to be searched may be used as a useful feature amount. That is, the "similarity" in the embodiment is a concept that includes not only a semantic "similarity" such that the imaging object is "similar" but also an appearance "similarity".

Further, the information providing device 10 may extract useful feature quantities depending on the viewpoint from which "similarity" is adopted at the time of operation. For example, the information providing device 10 sets a query image to be determined to be "similar" to a certain search target image during operation as a similar image, and among the feature quantities of the search target image, the number of times it is determined to be similar to the feature quantity of the similar image. A feature amount whose value exceeds a predetermined threshold may be regarded as a useful feature amount.

[1-12. About the target of processing]
Here, in the above-mentioned example, the information providing device 10 searches for a search target image similar to the query image. However, the embodiments are not limited to this. For example, the information providing device 10 can apply the above-mentioned search process to arbitrary contents such as sentences and music.

For example, when the search target is a sentence, the information providing device 10 selects some words as feature points from the query sentence and the sentence to be searched, and selects a plurality of words existing before and after the selected word. Extract. Further, the information providing device 10 calculates the feature amount of the extracted word as the feature amount, and the relative appearance position of the word (for example, a word existing between the word selected as the feature point and the next word). , Etc.) are regarded as coordinates. Then, the information providing device 10 performs a search process based on the comparison result of the feature amount, and also uses the feature amount of the sentence to be searched as a useful feature amount according to the number of times it is determined to be similar to the feature amount of the query sentence. May be extracted.

Further, for example, when the search target is voice, the information providing device 10 selects some playback positions as feature points from the query voice and the voice to be searched, and the voice existing before and after the selected playback position. The feature amount is calculated from the frequency characteristic of the above, and the information providing device 10 regards the reproduction position as a coordinate. Then, the information providing device 10 performs a search process based on the comparison result of the feature amount, and also uses the feature amount of the voice to be searched as a useful feature amount according to the number of times it is determined to be similar to the feature amount of the query voice. May be extracted.

The information providing device 10 may perform a similarity search between different species. For example, the information providing device 10 may search for a target voice similar to the query text, or may search for a target text similar to the query text. Further, a feature amount useful in such a similar search between different species may be extracted from a query or a voice or a sentence to be searched. That is, the information providing device 10 applies the above-mentioned search process to any content as long as the content can set a multidimensional amount indicating a local feature as the feature amount, and extracts useful feature amounts. You may go.

[2. Configuration of information providing device]
Hereinafter, an example of the functional configuration of the information providing device 10 that realizes the above-mentioned search process will be described. FIG. 3 is a diagram showing a configuration example of the information providing device according to the embodiment. As shown in FIG. 3, the information providing device 10 includes a communication unit 20, a storage unit 30, and a control unit 40.

The communication unit 20 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 20 is connected to the network N by wire or wirelessly, and transmits / receives information to / from the terminal device 100.

The storage unit 30 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. Further, the storage unit 30 stores the search target database 31 and the learning model 32.

Target information to be searched is registered in the search target database 31. For example, FIG. 4 is a diagram showing an example of information registered in the search target database according to the embodiment. In the example shown in FIG. 4, the search target database 31 has a "search target ID (Identifier)", a "search target data", a "feature amount", a "comparison usage count", a "comparison non-usage count", and a "coordinate usage". Information such as "number of times", "number of times of non-use of coordinates", and "type" is registered.

Here, the "search target ID" is an identifier that identifies the search target image. Further, the "search target data" is the data of the search target image. The "feature amount" is a feature amount. Further, the "number of comparison usage", "number of comparison non-use", "number of coordinate non-use", and "number of coordinate non-use" are the number of comparison use, the number of comparison non-use, and the coordinates of the associated feature amount, respectively. The number of uses and the number of times the coordinates are not used. Further, the "type" is information indicating whether the associated feature amount is a useful feature amount or a non-useful feature amount.

For example, in the example shown in FIG. 4, the search target database 31 has the search target ID “T1”, the search target data “TD # 1”, the number of comparison uses “5”, the number of comparison non-uses “2”, and the number of coordinates usage. "3", the number of times the coordinates are not used "2", and the type "useful" are registered in association with each other. Such information indicates that the data of the search target image indicated by the search target ID "T1" is "TD # 1" and the feature amount is "FT1-1". Further, for such information, the number of comparative uses of the feature amount "FT1-1" is "5", the number of times of comparative non-use is "2", the number of times of coordinate use is "3", and the number of times of coordinate non-use is "2". Indicates that. Further, such information indicates that the feature amount "FT1-1" is a useful feature amount.

It should be noted that the various values shown in FIG. 4 are conceptual, and in reality, numerical values indicating each feature amount and the like are registered in the search target database 31. Further, in the search target database 31, in addition to the information shown in FIG. 4, arbitrary information such as feature points corresponding to the feature amount and various information regarding the transaction target corresponding to the search target image may be registered. ..

Returning to FIG. 3, the explanation will be continued. The learning model 32 is a learning model in which the features of the useful features are learned. For example, the learning model is various parameters of the learning model that has learned the features of useful features such as DNN and SVM.

The control unit 40 is a controller, and for example, various programs stored in a storage device inside the information providing device 10 by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) are RAM or the like. Is realized by executing as a work area. Further, the control unit 40 is a controller, and may be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 4, the control unit 40 includes an extraction unit 41, a determination unit 42, a counting unit 43, a search unit 44, a generation unit 45, and a learning unit 46.

The extraction unit 41 extracts useful features useful for searching for similar information from the features extracted from the query image or the search target image. For example, the extraction unit 41 extracts a useful feature amount from the feature amount extracted from the search target image.

For example, the extraction unit 41 is based on the comparison result between the feature amount extracted from the search target image and the feature amount extracted from the similar information similar to the information, and from the feature amount extracted from the search target image. Extract useful features. Further, the extraction unit 41 extracts from the search target image based on the comparison result between the feature amount extracted from the plurality of search targets and the feature amount extracted from the similar information similar to any of the search target images. Useful features are extracted from the features.

Here, the extraction unit 41 extracts a useful feature amount useful for searching for a similar image from the feature amounts extracted from the query image or the image to be searched. For example, the extraction unit 41 extracts useful features useful for searching for images similar to the imaging target as useful features useful for searching for similar images.

More specifically, the extraction unit 41 extracts a useful feature amount from the feature amount extracted from the search target image based on the number of times counted by the counting unit 43. For example, the extraction unit 41 extracts as a useful feature amount a feature amount in which the number of times counted by the counting unit 43 is equal to or more than a predetermined threshold value among the feature amounts extracted from the search target image.

For example, when the extraction unit 41 receives the search target image from a predetermined information processing device (not shown), the extraction unit 41 extracts the feature amount of the search target image and searches the search target image and each feature amount in the search target database. Register at 31. Further, the extraction unit 41 refers to the search target database 31 at a predetermined time interval, and refers to and refers to the number of comparative use, the number of comparative non-use, the number of coordinate use, and the number of coordinate non-use of each feature amount. Based on the number of times, a score indicating whether or not each feature amount is useful for similar image search is calculated. For example, the extraction unit 41 calculates a score in which the value increases as the number of comparative uses and the number of coordinate uses increases, and the value decreases as the number of comparative non-uses and the number of coordinate non-uses increases.

Then, when the calculated score value exceeds a predetermined threshold value, the extraction unit 41 associates the feature amount with the type "useful", and when the calculated score value falls below the predetermined threshold value, the feature amount is classified into the type. Correspond to "unuseful". The extraction unit 41 may use the same threshold value or different threshold values for distinguishing between useful and non-useful.

The extraction unit 41 may delete the feature amount other than the useful feature amount from the storage device in which the feature amount extracted from the search target image is registered. For example, the extraction unit 41 may delete the feature amount associated with the type “unuseful” from the search target database 31.

Further, the extraction unit 41 may extract a useful feature amount from the feature amount extracted from the query image by using the model learned by the learning unit 46. For example, when the query image is received from the terminal device 100, the extraction unit 41 extracts the feature amount of the query image. Further, the extraction unit 41 reads the learning model 32 from the storage unit 30, and uses the learning model to extract useful feature quantities from the feature quantities of the query image.

The extraction unit 41 may use the learning model 32 to extract useful feature quantities of the search target image. For example, the extraction unit 41 may input each feature amount of the search target image into the learning model 32 and determine whether or not each feature amount is a useful feature amount.

Further, the extraction unit 41 is a feature amount extracted from the search target image based on a comparison result between the feature amount extracted from the search target image and the feature amount extracted from the similar image generated by the generation unit 45. Useful features may be extracted from. For example, when the extraction unit 41 acquires a similar image from the generation unit 45, the feature amount is extracted from the acquired similar image, and the extracted feature amount is provided to the counting unit 43. Then, the extraction unit 41 refers to the search target database 31 and extracts useful features from the search target image based on the various usage counts counted by the counting unit. The extraction unit 41 may use the learning model to extract useful features from the features extracted from similar images, and provide the extracted useful features to the counting unit 43.

Here, each number of uses registered in the search target database 31 is counted in the search by the determination unit 42, the counting unit 43, and the search unit 44. Therefore, the extraction unit 41 extracts the useful feature amount from the feature amount of the information similar to the past image (that is, the first query image) based on the search result by the search unit 44. That is, the extraction unit 41 extracts the useful feature amount from the feature amount extracted from the search target image based on the search result of the image similar to the past query image.

The extraction unit 41 extracts the feature amount whose number of times counted by the counting unit is equal to or higher than the predetermined first threshold value as the first useful feature amount, and the number of times counted by the counting unit is larger than the first threshold value. A feature amount having a second threshold value or more may be extracted as a second useful feature amount. For example, the extraction unit 41 registers the feature amount in which the number of times counted by the counting unit 43 is equal to or higher than a predetermined first threshold value in the search target database 31 in association with the type “useful”. On the other hand, the extraction unit 41 registers the feature amount counted by the counting unit 43 in the search target database 31 in association with the type "useful" for the feature amount having the second threshold value or more larger than the first threshold value. At the same time, the learning unit 46 may be notified as learning data.

The determination unit 42 determines whether or not the feature amount extracted from the search target image is similar to the feature amount extracted from the similar information. Further, the determination unit 42 is extracted from the positional relationship of the plurality of target feature quantities determined to be similar to the feature quantities extracted from the similar information among the plurality of feature quantities extracted from the search target image, and the similar information. Among the plurality of feature quantities, it is determined whether or not the positional relationship of the plurality of similar feature quantities determined to be similar to the plurality of target feature quantities is similar.

For example, the determination unit 42 receives the feature amount extracted from the query image by the extraction unit 41 during the operation of the similar image search. Further, for example, the determination unit 42 receives the feature amount extracted by the extraction unit 41 from the similar image generated by the generation unit 45. The feature amount of such a query image or a similar image may be a useful feature amount extracted by using the learning model 32.

In such a case, the determination unit 42 refers to the search target database 31, identifies the feature amount most similar to the feature amount of the received query image or similar image, and the feature amount of the query image or similar image and the search target. Corresponds to the feature amount specified from the database 31. Then, the determination unit 42 determines whether or not the positional relationship between the associated feature quantities is similar. That is, the determination unit 42 executes the feature amount matching process and the coordinate matching process. In the feature amount matching process, the determination unit 42 may specify from the search target database 31 all the feature amounts whose similarity with the feature amount of the received query image or similar image exceeds a predetermined threshold value.

The counting unit 43 counts the number of times it is determined that the feature amount is similar to the feature amount extracted from the similar image for each feature amount extracted from the search target image. Further, the counting unit 43 counts the number of times it is determined that the feature amount is most similar to the feature amount extracted from the similar image for each feature amount extracted from the search target image. Further, the counting unit 43 counts the number of times it is determined that the positional relationship between the plurality of target feature quantities and the positional relationship between similar feature quantities are similar.

For example, when the determination unit 42 determines that the feature amount registered in the search target database 31 is similar to the feature amount extracted from the similar image, the counting unit 43 sets 1 for the number of comparative uses of the feature amount. to add. Further, when the determination unit 42 determines that the feature amount registered in the search target database 31 is not similar to the feature amount extracted from the similar image, the counting unit 43 determines the number of comparative non-uses of the feature amount. Add 1 to.

Further, for example, the counting unit 43 uses the feature amount associated with the feature amount extracted from the similar image as the target feature amount among the feature amounts registered in the search target database 31, and the feature extracted from the similar image. Of the quantities, the feature quantity associated with the target feature quantity is defined as a similar feature quantity. Then, when the determination unit 42 determines that the positional relationship between the target feature amount of a certain search target image and the Tsukiji feature amount of a similar image is similar, the counting unit 43 determines the number of times the coordinates of the target feature amount are used. Add 1 to. On the other hand, when the determination unit 42 determines that the positional relationship between the target feature amount of a certain search target image and the Tsukiji feature amount of a similar image is not similar, the counting unit 43 does not use the coordinates of the target feature amount. Add 1 to the number of times.

On the other hand, the counting unit 43 executes the following processing during the operation of the similar image search. First, the determination result by the determination unit 42 is temporarily held. Further, when the counting unit 43 receives the notification of the search target image selected by the user from the search target images included in the search results from the terminal device 100, the counting unit 43 queries among the feature quantities of the notified search target images. 1 is added to the number of comparative uses of the feature amount determined to be similar to the feature amount extracted from the image. Further, the counting unit 43 determines by the determination unit 42 that the feature amount extracted from the query image among the feature amounts of the notified search target image has a similar positional relationship with the associated feature amount. Add 1 to the number of times the features are used.

The search unit 44 searches for information similar to the query image by using the useful feature amount. For example, the search unit 44 searches for a search target image similar to the query image based on the determination result by the determination unit 42. For example, the search unit 44 acquires the results of the feature amount matching process and the coordinate matching process by the determination unit 42 during the operation of the similar image search. Then, the search unit 44 selects an image to be searched, which is determined to have a similar positional relationship between similar feature quantities as a result of the coordinate matching process, as an image similar to the query image.

Further, the search unit 44 generates a search result in which various information such as thumbnails of images similar to the query image are arranged. Then, the search unit 44 provides the generated search result to the terminal device 100 that is the source of the query image.

The generation unit 45 generates a similar image from the search target image. For example, when the generation unit 45 extracts a useful feature amount using a similar image, the generation unit 45 selects one of the search target images from the search target database 31 and uses various image conversion techniques to select the selected search target image. Generate a similar image of. Then, the generation unit 45 provides the generated similar image to the extraction unit 41.

The learning unit 46 causes the model to learn the features of the useful features. For example, the learning unit 46 trains the model using a feature amount that is more useful than the useful feature amount used when searching for a search target image similar to the query image. For example, when the learning unit 46 receives the notification of the useful feature amount extracted by the extraction unit 41, the learning unit 46 causes the learning model 32 to learn the features of the notified useful feature amount. The learning unit 46 may, for example, train the learning model 12 to learn the difference between the feature of the useful feature amount and the feature of the non-useful feature amount.

[3. An example of the flow of processing executed by the information providing device]
Subsequently, an example of the flow of processing executed by the information providing device 10 will be described with reference to FIG. FIG. 5 is a flowchart showing an example of the flow of the search process executed by the information providing device according to the embodiment.

For example, the information providing device 10 extracts a feature amount from a search target image (step S101). Subsequently, the information providing device 10 counts the number of times used for the search for each feature amount when the similar image is used as the query image (step S102). For example, the information providing device 10 counts the number of times it is determined that the feature amount of the query image is similar to the feature amount of the query image, the number of times that the positional relationship is determined to be similar to the feature amount of the query image, and the like.

Then, the information providing device 10 extracts a useful feature amount based on the counted number of times (step S103). That is, the information providing device 10 extracts a feature amount that is frequently determined to be similar to the feature amount of the query image in the similar image search, that is, a feature amount useful for the similar image search as a useful feature amount.

Here, the information providing device 10 determines whether or not the query image has been acquired from the terminal device 100 (step S104), and if not (step S104: No), waits until the query image is acquired. Then, when the query image is acquired from the terminal device 100 (step S105: Yes), the information providing device 10 compares the feature amount of the query image with the useful feature amount (step S105), and based on the comparison result, the information providing device 10 compares the feature amount and the useful feature amount. An image similar to the query image is searched from the search target image (step S106), and the process is terminated.

[4. Modification example]
In the above, an example of the search process by the information providing device 10 has been described. However, the embodiments are not limited to this. Hereinafter, variations of the search process will be described.

[4-1. Comparison of features]
In the above-mentioned example, the information providing device 10 compares the feature amount of the query information with the query feature amount of the information to be searched (hereinafter, may be referred to as “target information”). Here, the information providing device 10 may use a feature amount of arbitrary information as a feature amount. For example, when the search target is an image, the information providing device 10 may use the average value of the saturation and brightness of the pixels around the feature point as the feature quantity, and when the search target is voice, the periphery of the feature point. A frequency characteristic around several seconds may be used as a feature quantity. In addition to the above-mentioned processing, the information providing device 10 may also use an arbitrary feature amount comparison technique.

[4-2. others〕
Further, among the processes described in the above-described embodiment, all or a part of the processes described as being automatically performed can be manually performed, and conversely, the processes described as being manually performed. It is also possible to automatically perform all or part of the above by a known method. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown in the figure.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

In addition, the above-described embodiments can be appropriately combined as long as the processing contents do not contradict each other.

[4-3. program〕
Further, the information providing device 10 and the terminal device 100 according to the above-described embodiment are realized by, for example, a computer 1000 having a configuration as shown in FIG. FIG. 6 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to the output device 1010 and the input device 1020, and the arithmetic unit 1030, the primary storage device 1040, the secondary storage device 1050, the output IF (Interface) 1060, the input IF 1070, and the network IF 1080 are connected by the bus 1090. Have.

The arithmetic unit 1030 operates based on a program stored in the primary storage device 1040 or the secondary storage device 1050, a program read from the input device 1020, or the like, and executes various processes. The primary storage device 1040 is a memory device that temporarily stores data used by the arithmetic unit 1030 for various operations such as RAM. Further, the secondary storage device 1050 is a storage device in which data used by the calculation device 1030 for various calculations and various databases are registered, such as a ROM (Read Only Memory), an HDD (Hard Disk Drive), and a flash memory. Is realized by.

The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various information such as a monitor and a printer. For example, USB (Universal Serial Bus), DVI (Digital Visual Interface), and the like. It is realized by a connector of a standard such as HDMI (registered trademark) (High Definition Multimedia Interface). Further, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, a keyboard, a scanner, and the like, and is realized by, for example, USB.

The input device 1020 is, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), or a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. Further, the input device 1020 may be an external storage medium such as a USB memory.

The network IF 1080 receives data from another device via the network N and sends it to the arithmetic unit 1030, and also transmits the data generated by the arithmetic unit 1030 to the other device via the network N.

The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 10, the arithmetic unit 1030 of the computer 1000 realizes the function of the control unit 40 by executing the program loaded on the primary storage device 1040. Further, for example, when the computer 1000 functions as the terminal device 100, the arithmetic unit 1030 of the computer 1000 realizes the function of the control unit 40 by executing the program loaded on the primary storage device 1040.

[5. effect〕
As described above, the information providing device 10 is useful for searching for similar information among the feature quantities extracted from the information to be searched, such as query information which is information to be a query such as a query image or an image to be searched. Extract useful feature quantities. Then, the information providing device 10 searches for information similar to the query information by using the extracted useful feature amount. As a result, the information providing device 10 performs a similarity search using features useful for the similarity search, so that the search accuracy is improved and the time and calculation cost required for the search are reduced, resulting in the efficiency of the similarity search. Can be improved.

Further, for example, the information providing device 10 extracts a useful feature amount from the feature amount extracted from the information to be searched. For example, the information providing device 10 extracts from the information to be searched based on the comparison result between the feature amount extracted from the information to be searched and the feature amount extracted from similar information similar to the information. Useful features are extracted from the features. Further, the information providing device 10 is based on a comparison result between a feature amount extracted from a plurality of information to be searched and a feature amount extracted from similar information similar to any one of the information. Useful features are extracted from the features extracted from the information to be searched.

Further, the information providing device 10 determines whether or not the feature amount extracted from the information to be searched is similar to the feature amount extracted from the similar information. Further, the information providing device 10 counts the number of times it is determined that the feature amount is similar to the feature amount extracted from the similar information for each feature amount extracted from the information to be searched. Then, the information providing device 10 extracts a useful feature amount from the feature amount extracted from the information to be searched based on the counted number of times. For example, the information providing device 10 counts the number of times it is determined that the feature amount is most similar to the feature amount extracted from the similar information for each feature amount extracted from the information to be searched. As a result of these processes, the information providing device 10 can extract a feature amount useful for a similarity search.

Further, the information providing device 10 has a positional relationship between a plurality of target feature quantities determined to be similar to the feature quantities extracted from similar information among the plurality of feature quantities extracted from the information to be searched, and similar information. Among the plurality of feature quantities extracted from, it is determined whether or not the positional relationship of the plurality of similar feature quantities determined to be similar to the plurality of target feature quantities is similar, and the positional relationship of the plurality of target feature quantities is determined. The number of times it is determined that the positional relationship of the similar feature amount is similar is counted, and the useful feature amount is extracted from the feature amount extracted from the information to be searched based on the number of times counted by the counting unit. Therefore, the information providing device 10 can extract a feature amount useful in the similarity search for executing the coordinate matching process as a useful feature amount.

Further, the information providing device 10 determines whether or not the feature amount extracted from the information to be searched is similar to the feature amount extracted from the first query information, and based on the determination result, the first Search for information similar to the query information. Here, the information providing device 10 extracts a useful feature amount from the feature amount of information similar to the first query information based on the search result, and is similar to the second query image based on the extracted useful feature amount. Search for information to do. Therefore, since the information providing device 10 can extract useful feature quantities based on the search results during the operation of the similar search, for example, it is possible to save the trouble of preparing information similar to the information to be searched.

Further, the information providing device 10 extracts as a useful feature amount a feature amount whose counted number of times is equal to or more than a predetermined threshold value among the feature amounts extracted from the information to be searched. Therefore, the information providing device 10 can extract an appropriate useful feature amount.

Further, the information providing device 10 deletes a feature amount other than the useful feature amount from the storage device in which the feature amount extracted from the information to be searched is registered. Therefore, the information providing device 10 can improve the efficiency of the similarity search.

Further, the information providing device 10 searches for information similar to the first query information among the information to be searched, and extracts from the information to be searched based on the search result of the information similar to the first query information. A useful feature amount is extracted from the obtained feature amount, and information similar to the second query information is searched for using the extracted useful feature amount. Therefore, since the information providing device 10 can extract useful feature quantities based on the search results during the operation of the similar search, for example, it is possible to save the trouble of preparing information similar to the information to be searched.

Further, the information providing device 10 generates similar information similar to the information to be searched from the information to be searched, and is extracted from the feature amount extracted from the information to be searched and the information generated by the generation unit. Based on the comparison result with the feature amount, the useful feature amount is extracted from the feature amount extracted from the information to be searched. Therefore, the information providing device 10 can extract a feature amount useful for similar search as a useful feature amount.

Further, the information providing device 10 causes the model to learn the features of the useful feature amount. Further, the information providing device 10 uses the learned model to extract useful features from the features extracted from the query information. Further, the information providing device 10 uses the learned model to extract useful feature quantities from the feature quantities extracted from the information to be searched. Therefore, the information providing device 10 can improve the efficiency of the similarity search.

Further, the information providing device 10 determines whether or not the feature amount extracted from the information to be searched is similar to the feature amount extracted from similar information similar to the information. Further, the information providing device 10 counts the number of times it is determined that the feature amount is similar to the feature amount extracted from the similar information for each feature amount extracted from the information to be searched. Further, the information providing device 10 extracts a feature amount in which the number of times counted is equal to or greater than a predetermined first threshold value as the first useful feature amount, and at the same time, the number of times counted is equal to or greater than the second threshold value larger than the first threshold value. The feature amount is extracted as the second useful feature amount. Then, the information providing device 10 searches for information similar to the query information using the first useful feature amount, and causes the model to learn the features of the second useful feature amount. As a result, the information providing device 10 can learn a model capable of appropriately extracting useful features.

Further, the information providing device 10 extracts a useful feature amount useful for searching a similar image from the feature amount extracted from the query image or the image to be searched, and queries using the useful feature amount. Search for images that are similar to the image. Further, the information providing device 10 extracts useful features useful for searching for images similar to the imaging target as useful features useful for searching for similar images. Therefore, the information providing device 10 can improve the efficiency of the similarity search.

Although some of the embodiments of the present application have been described in detail with reference to the drawings, these are examples, and various modifications are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure column of the invention. It is possible to carry out the present invention in other modified forms.

Further, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the extraction unit can be read as an extraction means or an extraction circuit.

10 Information providing device 20 Communication unit 30 Storage unit 31 Search target database 32 Learning model 40 Control unit 41 Extraction unit 42 Judgment unit 43 Counting unit 44 Search unit 45 Generation unit 46 Learning unit 100 Terminal device

Claims (18)

Of the feature amounts extracted from the predetermined search subject information to be search target, and the predetermined search target information be different from the search target information, extracted from the similarity information similar to the predetermined search target information an extraction unit a feature quantity is extracted as useful useful feature quantity similar to feature amount,
A search device comprising a search unit that searches for information similar to query information using the useful feature amount extracted by the extraction unit. The extraction section, based on the comparison result of the feature amount extracted from the search target information for multiple, and feature quantity extracted from similar information similar to any of the search target information of the search target information, Search device according to claim 1, characterized in that extracting the useful feature quantity from the feature quantity extracted from the search pair Zojo paper. A feature value extracted from the search pair Zojo report, a determination section for determining whether or not the feature amount extracted from the similarity information is similar,
For each feature amount extracted from the search pair Zojo paper, and a counting section for counting the number of times it is determined similar to the feature amount extracted from the similarity information,
The extraction section, based on the number of times counted by the counting unit, a feature amount extracted from the search pair Zojo paper according to claim 1 or 2, characterized in that extracting the useful feature quantity Search device. The counting unit, for each feature amount extracted from the search pair Zojo paper, according to claim 3, characterized in counting the number of times said is determined that the feature quantity extracted from the similarity information most similar to Search device. The determination unit, among the plurality of feature values extracted from the search pair Zojo paper, the positional relationship of the plurality of object feature amount that has been determined to be similar to the feature amount extracted from the similarity information, from the similarity information Among the extracted plurality of feature quantities, it is determined whether or not the positional relationship of the plurality of similar feature quantities determined to be similar to the plurality of target feature quantities is similar.
The counting unit counts the number of times it is determined that the positional relationship between the plurality of target feature quantities and the positional relationship between the similar feature quantities are similar.
The extraction section, based on the number of times counted by the counting unit, a feature amount extracted from the search pair Zojo paper according to claim 3 or 4, characterized in that extracting the useful feature quantity Search device. The determination section determines the feature amount extracted from the search pair Zojo report, whether features and extracted from the first query information is similar,
The search unit searches for information similar to the first query information based on the determination result by the determination unit.
The extraction unit extracts useful features from the features of information similar to the first query information based on the search results by the search unit.
The search according to any one of claims 3 to 5 , wherein the search unit searches for information similar to the second query image based on the useful feature amount extracted by the extraction unit. Device. The extraction section, among the feature quantities extracted from the search pair Zojo report, the number of times counted by the counting unit and extracting a feature amount equal to or greater than a predetermined threshold value as the useful feature quantity The search device according to any one of claims 3 to 6. Claim search pair Zojo feature quantity extracted from report from the storage device that is registered, characterized in that it has a deletion unit for deleting the feature amount other than the useful feature amount extracted by the extraction unit 1-7 The search device described in any one of them. The search unit of the search pair Zojo paper, retrieves information similar to the first query information,
The extraction section, based on the search result of the information similar to the first query information, extracts the useful feature quantity from the feature quantity extracted from the search pair Zojo paper,
The search according to any one of claims 1 to 8 , wherein the search unit searches for information similar to the second query information using the useful feature amount extracted by the extraction unit. Device. From the search pair Zojo paper has a generation unit for generating similarity information similar to the information,
The extraction unit includes a feature amount extracted from the search pair Zojo paper, based on a result of comparison between the feature amount extracted from the generated information by the generation section, it is extracted from the search pair Zojo paper The search device according to any one of claims 1 to 9 , wherein the useful feature amount is extracted from the feature amount. The search device according to any one of claims 1 to 10 , wherein the search device has a learning unit that causes a model to learn the features of the useful feature amount extracted by the extraction unit. The extraction unit, using said learning model by learning unit, the search device according to claim 1 1, characterized in that extracting the useful feature quantity from the feature quantity extracted from the query information. The extraction unit, using the model that is learned by the learning section, to claim 1 1 or 1 2 and extracting the useful feature quantity from the feature quantity extracted from the search pair Zojo paper The search device described. A feature value extracted from the search pair Zojo report, a determination unit and a feature amount extracted from the similarity information whether similar similar to the information,
For each feature amount extracted from the search pair Zojo paper, and a counting section for counting the number of times it is determined similar to the feature amount extracted from the similarity information,
The extraction unit extracts a feature amount whose number of times counted by the counting unit is equal to or higher than a predetermined first threshold value as a first useful feature amount, and the number of times counted by the counting unit is higher than the first threshold value. A feature amount having a large second threshold value or more is extracted as a second useful feature amount, and the feature amount is extracted.
The search unit searches for information similar to the query information using the first useful feature amount.
The learning section, search apparatus according to any one of claims 1 1 to 1 3, characterized in that to learn the features of the second useful feature quantity has the model. The extraction unit extracts useful features useful for searching similar images from the feature quantities extracted from the query image or the image to be searched.
The search unit is configured using a useful feature quantity, the search device according to any one of claims 1 to 1 4, characterized in that retrieving the image similar to the query image. The search device according to claim 15 , wherein the extraction unit extracts useful features useful for searching for images similar to the imaging target as useful features useful for searching for similar images. .. It is a search method executed by the search device.
Of the feature amounts extracted from the predetermined search subject information to be search target, and the predetermined search target information be different from the search target information, extracted from the similarity information similar to the predetermined search target information An extraction step of extracting a feature amount similar to the obtained feature amount as a useful useful feature amount, and
A search method comprising a search step of searching for information similar to query information using the useful feature amount extracted by the extraction step. Of the feature amounts extracted from the predetermined search subject information to be search target, and the predetermined search target information be different from the search target information, extracted from the similarity information similar to the predetermined search target information An extraction procedure for extracting a feature amount similar to the feature amount as a useful useful feature amount,
A search program for causing a computer to execute a search procedure for searching information similar to query information using useful features extracted by the extraction procedure.

Images