JP6997656B2 - Search device, search method and search program - Google Patents

Description

The present invention relates to a search device, a search method, and a search program.

Conventionally, there has been known a technique for searching information similar to a search query by comparing the feature amount of the information to be a search query with the feature amount of the information to be searched. As an example of such a technique, in addition to the similarity between a plurality of local features extracted from an image to be a search query and a plurality of features of the image to be searched, the range in which the features are extracted. A technique for searching an image similar to an image used as a search query based on the degree of similarity of coordinate relationships is known.

Japanese Unexamined Patent Publication No. 2014-157443

However, in the above-mentioned conventional technique, it may not be possible to say that information retrieval is performed efficiently.

For example, a plurality of images to be searched may be composed of a transaction target and a background. In such a case, if a similar image search is performed using the results of comparing the feature quantities extracted from the background portion, it may be determined that the images captured by different transaction targets are similar to each other. There is. In addition, as the number of feature quantities extracted from search queries and search targets increases, the search cost required for information retrieval increases.

The present application has been made in view of the above, and an object thereof is to improve the efficiency of information retrieval.

The search device according to the present application is extracted by an extraction unit that extracts non-useful feature amounts that are not useful for searching similar information among the feature amounts extracted from query information or information to be searched, and the extraction unit. It is characterized by having a search unit for searching information similar to query information by using a feature amount other than the non-useful feature amount.

According to one aspect of the embodiment, the efficiency of information retrieval can be improved.

FIG. 1 is a diagram showing an example of a search process executed by the information providing device according to the embodiment. FIG. 2 is a diagram showing an example of a process in which the information providing device according to the embodiment extracts frequently-appearing feature quantities using similar images. FIG. 3 is a diagram showing a configuration example of the information providing device according to the embodiment. FIG. 4 is a diagram showing an example of information registered in the search target database according to the embodiment. FIG. 5 is a flowchart showing an example of the flow of the search process executed by the information providing apparatus according to the embodiment. FIG. 6 is a diagram showing an example of a hardware configuration.

Hereinafter, a search device, a search method, and a mode for carrying out the search program (hereinafter, referred to as “embodiment”) according to the present application will be described in detail with reference to the drawings. Note that this embodiment does not limit the search device, search method, and search program according to the present application. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description will be omitted.

[Embodiment]
[1-1. About an example of information providing device]
First, with reference to FIG. 1, an example of a search process executed by an information providing device, which is an example of a search device, will be described. FIG. 1 is a diagram showing an example of a search process executed by the information providing device according to the embodiment.

In FIG. 1, the terminal device 100 and the information providing device 10 used by each user can communicate with each other via a predetermined network N (see, for example, FIG. 3) such as the Internet. Further, it is assumed that the information providing device 10 can communicate with various external servers other than the terminal device 100. Further, the information providing device 10 may be able to communicate with any number of terminal devices 100.

The terminal device 100 is a smart device such as a smartphone or tablet, and is a mobile terminal device capable of communicating with an arbitrary server device via a wireless communication network such as 3G (3rd Generation) or LTE (Long Term Evolution). be. The terminal device 100 may be not only a smart device but also an information processing device such as a desktop PC (Personal Computer) or a notebook PC.

The information providing device 10 is an information processing device that executes a search process, and is realized by, for example, a server device, a cloud system, or the like. For example, when the information providing device 10 receives an image to be a search query from the terminal device 100, the information providing device 10 searches for an image having contents similar to the received query image, and the searched image or information indicating the searched image (for example, a URL). (Uniform Resource Locator), thumbnails, etc.) are provided as search results to the terminal device 100, and a search process is executed.

To give a more specific example, the information providing device 10 holds an image showing a transaction target in an electronic shopping district. For example, the terminal device 100 holds an image of a product to be traded and an image related to a service to be traded (hereinafter, collectively referred to as “image of transaction target”). Such an image of a transaction target is, for example, an image registered by a seller who sells the transaction target in an electronic shopping district.

For example, when the information providing device 10 receives the query image from the terminal device 100 as a search query for searching the transaction target exhibited in the electronic shopping district, the information providing device 10 searches for an image of the transaction target similar to the received query image. .. Then, the information providing device 10 uses the terminal device 100 as a search result that summarizes the content explaining the transaction target corresponding to the searched image (that is, the transaction target captured in the image) and the link to the content for purchasing the transaction target. Send to.

[1-2. Search using features]
Hereinafter, as the search process executed by the information providing device 10, the flow of the process of searching for an image similar to the query image among the images to be traded will be described. In the following description, the images of the transaction target to be searched are collectively referred to as the search target images. In the following description, an example of a process of searching for a search target image similar to a query image using a feature amount will be described, but the embodiment is not limited to this. As will be clarified in the explanation described later, if the information providing device 10 searches for a search target image similar to the query image by comparing the feature quantities with each other, the information providing device 10 is similar to the query image by any search method. You may search for the image to be searched.

For example, in a search using feature points, a plurality of characteristic points are selected as feature points from the search target image by the processing, and a range for calculating the feature amount is set for each feature point. Will be done. For example, the range in which such a feature amount is calculated is set so that the feature amount does not change so much even when various affine transformations are performed. As a result, each feature point becomes an irregular position, and the range (radius) in which the feature amount is calculated is also various. In FIG. 1, the range in which the feature amount is calculated is simulated by a dotted circle, but in reality, an arbitrary shape such as an ellipse or a rectangle is adopted as the range in which the feature amount is calculated. ..

In the following description, each feature amount calculated from the search target image is described as "feature amount of the search target image". Subsequently, the information providing device 10 similarly calculates the feature amount in the range based on the automatically selected characteristic point, range (radius), and direction for the query image. In the following description, each feature amount calculated from the query image is described as "feature amount of the query image".

The features of the above-mentioned search target image and query image include wide-area features such as GIST (http://ilab.usc.edu/siagian/Research/Gist/Gist.html), color histogram, and color distribution, and SIFT. (Scale-Invariant Feature Transform), SURF (Speed-Up Robust Features), feature amount, and any other technique for acquiring the feature amount of the image can be adopted. The features generated by the neural network can also be used.

Then, the information providing device 10 specifies a feature amount similar to the feature amount of the query image among the feature amounts of the search target image by comparing the feature amount of the query image and the feature amount of the search target image. .. For example, the information providing device 10 uses a technique such as NGT (Neighborhood Graph and Tree) for each feature amount of the query image to determine whether or not it is similar to the feature amount of each search target image. Then, the information providing device 10 associates the feature amount of the query image with the feature amount determined to be most similar to the feature amount of the query image among the feature amounts of each search target image.

Then, the information providing device 10 selects, among the search target images, a search target image in which the number of feature amounts associated with the feature amount of the query image exceeds a predetermined threshold value as a similar candidate for the query image. That is, the information providing device 10 selects a search target image similar to the query image as a candidate for the similar image in a voting format based on the similarity between the feature quantities. In the following description, the process of determining whether or not the above-mentioned feature amounts are similar to each other may be described as "feature amount collation process".

Here, a method of outputting a candidate of a similar image as a search result as it is can be considered, but the information providing device 10 executes the coordinate matching process described below in order to further improve the accuracy. For example, the information providing device 10 specifies a feature amount associated with each other among the feature amount of an image that is a candidate for a similar image (hereinafter, referred to as “candidate image”) and the feature amount of the query image. do.

Subsequently, the information providing device 10 performs an affine transformation (for example, rotation, translation, enlargement or reduction) of the coordinates of the feature points of each feature amount specified from the query image, and the information providing device 10 of each feature amount specified from the candidate image. Determine if it is similar to the coordinates. Then, when the information providing device 10 determines that the coordinates of the feature points of the query image and the coordinates of the feature points of the candidate image are similar, the information providing device 10 determines that the query image and the candidate image are similar.

That is, in the feature amount matching process described above, the information providing device 10 determines whether or not the feature amount of the search target image and the feature amount of the query image are similar, and the query image is obtained by voting based on the determination result. Some of the feature points possessed are associated with some of the feature points of the plurality of search target images. Then, the information providing device 10 collates the coordinates of the associated feature points by executing the coordinate matching process using some of the feature points associated with each other in the coordinate matching process described above. That is, the information providing device 10 determines whether or not the coordinates of the feature points of the query image match (or are similar to) the coordinates of the feature points of the search target image when the coordinates of each feature point of the query image are affine-transformed. It is determined whether or not the coordinate relationships between the corresponding feature quantities are similar based on whether or not the number exceeds a predetermined number.

In the coordinate matching process, the information providing device 10 re-compares all the feature amounts of the query image with all the feature amounts of the candidate images, and the feature amounts of the query images and the feature amounts of the candidate images that are similar to each other are re-compared. You may redo the correspondence with. Then, the information providing device 10 may determine whether or not the query image and the candidate image are similar based on the positional relationship between the newly associated feature quantities. As a result of such processing, the information providing device 10 can improve the search accuracy of similar images.

[1-3. About the outline of the search process]
However, in the feature amount matching process and the coordinate matching process described above, it may not be possible to say that the information retrieval is properly performed. For example, a process of extracting a feature amount from a plurality of images including the same image pickup target will be described. For example, when a feature point is extracted from a plurality of images including the same image pickup target, a feature point having a background can be selected as a feature point in addition to the characteristic place having the image pickup target. As a result, for example, even if the query image and the search target image include different imaging targets, if the backgrounds are common or similar, the query image and the search target image are determined to be similar images. There is a fear.

Further, for example, when a feature point is extracted from an image including a certain image pickup target, a boundary between the image pickup target and the background, that is, a position indicating the shape of the image pickup target may be selected as the feature point. For example, a corner or a straight line portion of the image pickup target is selected as a feature point. From such feature points, a feature amount indicating an angle or a straight line portion of an imaging target is calculated, and many imaging targets have such an angle or a straight line portion. Therefore, it cannot be said that it is useful for similar image search even if features commonly possessed by many imaging targets such as corners and straight lines are used as comparison targets.

That is, the feature amount extracted from the image is a feature amount that is a feature of the transaction target and that shows the feature that many transaction targets have in common, unlike the feature amount that shows the feature that the transaction target has uniquely. (Hereinafter, referred to as "frequent feature amount") exists. When a similar image search is performed using such a frequent feature amount, there is a possibility that images with different transaction targets captured may be determined to be similar images.

In addition, a plurality of feature points are selected from the query image and each search target image. Therefore, when all the feature quantities extracted from the image are compared with each other, the time required for the search and the calculation cost increase.

Therefore, the information providing device 10 executes the following search process. First, the information providing device 10 extracts a non-useful feature amount that is not useful for searching for similar information from the feature amounts extracted from the query image or the search target image. Then, the information providing device 10 searches for information similar to the query image by using the feature amount other than the extracted non-useful feature amount.

For example, the information providing device 10 extracts a non-useful feature amount from the feature amount extracted from the search target image. More specifically, the information providing device 10 extracts a plurality of features that are similar to each other, that is, frequently appearing features, as non-useful features, among the features extracted from the search target image. Then, the information providing device 10 deletes the extracted frequent feature amount from the database in which the feature amount extracted from the search target image is registered.

As a result of such processing, the information providing device 10 compares the feature quantities other than the frequently-appearing feature quantities, that is, the feature quantities showing the features uniquely possessed by the search target image and the query image. As a result, the information providing device 10 can reduce erroneous determinations such as similar images having similar backgrounds, so that the search accuracy can be improved. Further, since the information providing device 10 reduces the number of feature quantities to be compared, it is possible to reduce the time required for searching and the calculation cost.

[1-4. Extraction of frequent features]
Here, if the information providing device 10 can extract a feature amount that is not useful for searching for similar images, such as a frequently-appearing feature amount, the feature amount satisfying an arbitrary condition is extracted as a non-useful feature amount. good.

For example, consider the case where there are a search target image # A and a search target image # B in which different transaction targets are captured. In such a case, among the feature quantities extracted from the search target image # A and the search target image # B, the feature quantities that are similar to each other are not so much for the semantic similar image search such as the background and the general shape. It is considered to be a feature quantity that is not useful. Therefore, the information providing device 10 determines whether or not the feature amounts extracted from the plurality of search target images are similar to each other, and determines the number of times that the feature amounts determined to be similar to the other feature amounts exceed a predetermined threshold value. Extract as frequent features (ie, non-useful features).

For example, the information providing device 10 determines whether or not the feature amount extracted from the search target image is similar to a plurality of predetermined feature amounts, and a predetermined feature amount is determined for each feature amount extracted from the search target image. The determined number of times similar to the feature amount is counted. Then, the information providing device 10 may extract a feature amount in which the number of times counted exceeds a predetermined threshold value as a frequent feature amount. Further, the information providing device 10 has a plurality of predetermined feature amounts, which are a predetermined number or more, a feature amount generated by a predetermined method, and a feature amount extracted from the search target image. May be determined whether or not they are similar.

For example, the information providing device 10 selects an arbitrary image from the search target image, and determines whether or not the feature amount of the selected search target and the feature amount of the other search target image are similar. Then, the information providing device 10 may extract a feature amount whose number of times determined to be similar to other feature amounts exceeds a predetermined threshold value as a frequent feature amount.

Further, the information providing device 10 acquires an arbitrary image such as various images existing on the network, and extracts a feature amount from the acquired image. Then, the information providing device 10 compares the feature amount of the search target image with the feature amount of the acquired image, and among the feature amounts of the search target image, the number of times determined to be similar to the feature amount of the acquired image is predetermined. A feature amount exceeding the threshold value of may be extracted as an extraction feature amount.

Further, for example, the information providing device 10 determines whether or not the feature amount extracted from the query image received within a predetermined period is similar to the feature amount extracted from the search target image. Then, the information providing device 10 may extract a feature amount having a number of times determined to be similar to the feature amount of the query image among the feature amounts of the search target image exceeds a predetermined threshold value as a frequent feature amount.

Further, for example, the information providing device 10 generates a plurality of pseudo-features (hereinafter, referred to as "pseudo-features") by a predetermined algorithm or randomly, and the generated pseudo-features are the feature quantities to be searched. Determine if and are similar. Then, the information providing device 10 may extract a feature amount having a number of times determined to be similar to the pseudo feature amount among the feature amounts of the search target image to exceed a predetermined threshold value as a frequent feature amount.

That is, among the feature quantities to be compared in the similar image search, the information providing device 10 frequently features feature quantities other than the feature quantities that indicate the characteristics unique to the imaging target (that is, the transaction target) indicated by the search target image. If it can be an amount, the frequently-appearing feature amount is extracted based on the comparison result between the feature amount extracted from an arbitrary image or the feature amount generated by an arbitrary means and the feature amount of the search target image. You can do it.

As described above, the information providing device 10 extracts the feature amount that frequently appears among the feature amounts to be compared in the similar image search as a feature amount that is not so important, that is, a feature amount that indicates a feature that is not so characteristic. .. Then, the information providing device 10 performs a similar image search using a feature amount other than the extracted feature amount. As a result, the information providing device 10 can improve the efficiency of similar image retrieval.

[1-5. About category consideration]
Here, the information providing device 10 may extract the frequently-appearing feature amount in consideration of the category. For example, the information providing device 10 may extract a non-useful feature amount based on the comparison result between the feature amounts extracted from the information belonging to different categories in the search target image. For example, the information providing device 10 compares a feature amount extracted from information related to the first transaction target with a feature amount extracted from information related to a second transaction target of a type different from the first transaction target. Based on the results, non-useful features may be extracted. That is, the information providing device 10 selects an arbitrary image from the search target images, and the feature amount of the selected search target and the feature amount of another search target image belonging to a category different from the selected search target image. May be determined whether or not they are similar.

For example, when the information providing device 10 selects an image of a transaction target belonging to category # A, the information providing device 10 extracts the image of the transaction target belonging to category # B as a comparison target. Then, the information providing device 10 determines whether or not the feature amount extracted from the image of the transaction target belonging to the category #A and the feature amount extracted from the image of the transaction target belonging to the category #B are similar to each other, and determines whether or not the feature amount is similar to the category. Of the feature quantities extracted from the image of the transaction target belonging to # B, the feature quantity in which the number of times determined to be similar to the feature quantity extracted from the image of the transaction target belonging to category #A exceeds a predetermined threshold is defined as the frequent feature quantity. May be good.

Here, any relationship can be adopted for category # A and category # B. For example, the information providing device 10 is a category different from the search target image in which the transaction target belonging to the category # A is captured in the electronic shopping district and the category # A, and the category # B in which the category one step higher is common. The feature amounts may be compared with the search target image in which the transaction target belonging to the category is captured. Further, the information providing device 10 is a category different from the search target image in which the transaction target belonging to the category # A is captured in the electronic shopping district and the category # A, and the category # in which the categories higher than two levels are common. The feature amounts may be compared with the search target image in which the transaction target belonging to B is captured.

Further, the information providing device 10 compares the feature amounts of the search target images in which the transaction targets belonging to the same category are captured, and counts the number of times determined to be similar to the other feature amounts for each feature amount. When such processing is performed, a feature amount whose number of times determined to be similar to other feature amounts exceeds a predetermined threshold can be said to be a feature amount commonly possessed by transaction targets belonging to the category. Therefore, the information providing device 10 may set a feature amount in which the number of times determined to be similar to other feature amounts is less than a predetermined threshold value as a frequent feature amount.

Further, the information providing device 10 compares the feature amounts of the search target images with each other, and determines that the transaction target belonging to the same category is similar to the feature amount of the captured search target image, and the number of transactions belonging to different categories. The number of times that the target is determined to be similar to the feature amount of the image to be searched is counted. Then, in the information providing device 10, the number of times that the transaction target belonging to the same category is determined to be similar to the feature amount of the captured search target image falls below a predetermined threshold value, and the transaction target belonging to a different category is captured. A feature amount in which the number of times determined to be similar to the feature amount of the image exceeds a predetermined threshold may be extracted as a frequent feature amount.

[1-6. Processing using similar images]
Further, the information providing device 10 generates a similar image from an arbitrary image such as a search target image or a query image, and the feature amount extracted from the generated similar information is similar to the feature amount extracted from the search target image. Determine whether or not to do so. Then, the information providing device 10 may extract a feature amount whose number of times determined to be similar to other feature amounts exceeds a predetermined threshold value as a frequent feature amount.

For example, FIG. 2 is a diagram showing an example of a process in which an information providing device according to an embodiment extracts a frequent feature amount using a similar image. For example, the information providing device 10 selects the search target image T1 as the processing target from the database in which the search target image is registered. Then, the information providing device 10 generates an image similar to the extracted search target image T1, that is, similar images TV1 to TV3.

For example, the information providing device 10 performs image analysis of the selected search target image, identifies the range in which the background is captured and the range in which the main imaging target is captured, and deletes the range in which the background is captured. Generate image TV1. The information providing device 10 may generate a similar image obtained by synthesizing different backgrounds. Further, the information providing device 10 may generate a similar image TV 2 in which the range in which the image pickup target is captured is affine-transformed, such as an image in which the image pickup target is rotated, enlarged, reduced, or deformed. Further, the information providing device 10 may generate a similar image TV 3 in which the color to be imaged is converted.

Further, the information providing device 10 may generate an image to which various affine transformations are applied to the entire search target image. Further, the information providing device 10 may generate an image in which each range or the entire image is blurred. Further, the information providing device 10 may generate a similar image by an arbitrary image conversion method such as changing the resolution.

As for how much the changed image is to be a similar image, any setting can be adopted according to the purpose of the similar image search executed by the information providing device 10. That is, if the information providing device 10 is similar to the extent that when an image generated as a similar image is input as a query image, the original search target image can be selected as an image similar to the query image. , An image that has undergone arbitrary image conversion may be adopted as a similar image. Further, the generation of such a similar image can be realized by various image analysis techniques and image generation techniques.

Subsequently, the information providing device 10 extracts the feature quantities of the generated similar images TV1 to TV3. For example, the information providing device 10 extracts feature quantity groups TV1 to TV3 from similar images TV1 to TV3. Further, the information providing device 10 acquires the feature amount group T1 of the original search target image T1. Subsequently, the information providing device 10 selects the search target image T2 in which the transaction target in a category different from the search target image T1 is captured, and extracts the feature amount group T2 of the selected search target image T2. Then, the information providing device 10 compares each feature amount included in the feature amount groups T1, T2, and TV1 to TV3 with each other, and among the feature amounts included in the feature amount group T2, the feature amount group T1, TV1 to TV3. A feature amount in which the number of times determined to be similar to the feature amount contained in the above exceeds a predetermined threshold is extracted as a frequent feature amount.

[1-7. Necessity of category consideration]
It is not always necessary to consider the category in the various processes described above. For example, in the actual processing, the information providing device 10 repeatedly executes a process of comparing the feature amounts of the search target images belonging to various categories and counting the number of times determined to be similar to the other feature amounts. If such processing is performed a sufficient number of times, or if the number of categories and transaction targets corresponding to the search target image is sufficiently large, the transaction targets belonging to a certain category have common features and are useful for searching. Other features are features that are not useful for searching (features that show features that many images have in common), rather than the number of times that features that show features are determined to be similar to other features. It is considered that the number of times that it is judged to be similar to the amount is larger.

Therefore, the information providing device 10 counts the number of times determined to be similar to other feature quantities by repeatedly executing the above-mentioned processes in various arbitrary combinations, and the feature quantity whose counted number exceeds a predetermined threshold value. It may be extracted as a frequent feature amount.

[1-8. About the threshold]
Further, the information providing device 10 may adopt an arbitrary threshold value as a threshold value when extracting the frequently-appearing feature amount. For example, the information providing device 10 selects a predetermined number of feature quantities in descending order of the number of times it is determined to be similar to the other feature quantities among all the feature quantities of all the search target images, and selects the selected feature quantities. It may be extracted as a frequent feature amount. Further, the information providing device 10 may use all the feature quantities in which the number of times determined to be similar to the other feature quantities is equal to or higher than a predetermined threshold value as the frequent feature quantities.

On the other hand, if a large number of features are deleted from the features extracted from the search target image, the search accuracy may decrease. Therefore, the information providing device 10 may set an upper limit on the number of frequently-used feature amounts to be deleted. For example, the information providing device 10 may extract a predetermined number or less of non-useful features from the features extracted from the query image or the search target image. More specifically, the information providing device 10 extracts the frequently appearing feature amount so that the number of the feature amount other than the frequently appearing feature amount does not fall below a predetermined threshold value among the feature amount per search target image. You may. Further, the information providing device 10 may set a lower limit value or an upper limit value of the ratio of the feature amount to be the frequently appearing feature amount among all the feature amounts of the search target image.

[1-9. An example of search processing]
Hereinafter, an example of the search process executed by the information providing device 10 will be described with reference to FIG. For example, the information providing device 10 extracts in advance the feature amount of each search target image to be searched (step S1). For example, the information providing device 10 selects a plurality of feature points PT1-1, PT1-2, ... From the search target image T1 by a predetermined algorithm, and features FT1-1, FT1 for each selected feature point. -2 ... is extracted. Further, the information providing device 10 selects a plurality of feature points PT2-1, PT2-2, ... From the search target image T2 by a predetermined algorithm, and the feature quantities FT2-1, FT2 for each selected feature point. -2 ... is extracted.

Subsequently, the information providing device 10 compares the extracted feature amounts with each other, that is, the feature amounts of the images registered as the search target images, and counts the number of times for each feature amount to be determined to be similar to the other feature amounts. do. Then, the information providing device 10 extracts a feature amount whose number of times determined to be similar to other feature amounts exceeds a predetermined threshold value as a frequent feature amount (step S2). For example, the information providing device 10 compares the feature amount FT1-1 with the other feature amounts FT1-2, FT2-1, FR2-2, ..., And the degree of similarity with the feature amount FT1-1 is a predetermined threshold value. The following number of features is counted as the number of similar features. Further, the information providing device 10 similarly counts the number of similar feature quantities for other feature quantities. Then, the information providing device 10 extracts a predetermined number of feature quantities as frequent feature quantities in order from the one with the highest value of the counted similar feature quantities.

Subsequently, the information providing device 10 deletes the extracted frequently-appearing feature amount (step S3). For example, the information providing device 10 deletes the feature amounts FT1-1, FT2-2, and the like in which the number of similar feature amounts exceeds a predetermined threshold value from the database in which the feature amounts of each search target image are registered.

Here, when the query image is acquired from the terminal device 100 (step S4), the information providing device 10 is similar to the query image based on the comparison result between the feature amount of the query image and the feature amount other than the frequent feature amount. Search for the image to be used (step S6). For example, the information providing device 10 searches for an image similar to the query image based on the comparison result between the feature amount registered in the database and the feature amount extracted from the query image.

Then, the information providing device 10 transmits the search result to the terminal device 100 (step S6). For example, the information providing device 10 transmits image data, thumbnails, and the like determined to be similar to the terminal device 100. The information providing device 10 may distribute the information of the searched search target image in a ranking format according to the result of the feature amount matching process or the coordinate matching process, for example. For example, the information providing device 10 calculates the accuracy similar to each search target image and the query image based on the result of each collation process, and obtains the information of the searched search target image in a ranking format according to the calculated accuracy. It may be delivered.

[1-10. Processing using the results of search processing]
The information providing device 10 will compare whether or not the query image received by the feature amount matching process described above and the feature amount of the search target image are similar to each other. Therefore, for example, the information providing device 10 may count the number of times that the feature amount of the search target image is determined to be similar to the feature amount of the query image by the feature amount matching process. For example, in the case of the information providing device 10, if a certain feature amount extracted from the search target image is determined to be similar to the feature amount of the query image by the feature amount matching process, the similar feature amount number of the feature amount is determined. 1 may be added to.

[1-11. About the features of frequent features]
Here, a predetermined feature may appear in the feature amount extracted as the frequent feature amount. Therefore, the information providing device 10 may make the model learn the features of the frequent feature quantities. For example, the information providing device 10 may make an arbitrary model, such as a DNN (Deep Neural Network) or an SVM (Support Vector Machine), learn the features of the frequent features.

Further, the information providing device 10 may make the model learn the features of the feature amount other than the frequently appearing feature amount. For example, the information providing device 10 extracts a feature amount in which the number of similar feature amounts is less than a predetermined threshold value as a non-frequent feature amount. Then, the information providing device 10 may learn the feature unique to the frequent feature amount with high accuracy by learning the difference between the feature possessed by the frequent feature amount and the feature possessed by the non-frequent feature amount in the model. ..

It is presumed that the learning model in which such learning is performed can extract the frequent features from the features. Therefore, the information providing device 10 may extract the frequently appearing feature amount from the feature amount extracted from the search target image by using the learning model. For example, the information providing device 10 inputs the feature amount extracted from the search target image into the learning model, determines whether or not the feature amount is a frequent feature amount, and only the feature amount determined not to be a frequent feature amount. May be registered in the database.

Further, the information providing device 10 uses a learning model to extract a frequent feature amount from the feature amount extracted from the query image, a feature amount other than the extracted frequent feature amount, and a feature amount extracted from the search target image. You may search for an image similar to the query image based on the comparison result with. For example, the information providing device 10 uses a learning model to extract frequently appearing features from the features extracted from the query image during operation. Then, the information providing device 10 may search for an image similar to the query image based on the comparison result between the extracted feature amount other than the frequently-appearing feature amount and all the feature amounts extracted from the search target image. ..

Further, the information providing device 10 has a feature amount other than the frequent feature amount extracted from the feature amount of the query image using the learning model, and a feature amount other than the frequent feature amount extracted from the feature amount of the search target image by the above-mentioned various methods. You may search for an image similar to the query image based on the feature amount of the above or the comparison result with the feature amount other than the frequent feature amount extracted from the feature amount of the search target image by using the learning model.

The information providing device 10 may have different frequently-used feature quantities used for image retrieval and useful feature quantities used for model learning. For example, in order to make the model learn the features of the frequent features with high accuracy, it is considered that the probability that the frequent features used for learning the model are not actually useful is higher. Therefore, the information providing device 10 may individually extract the frequently-appearing feature amount to be considered in the image search and the frequently-appearing feature amount to be used for model learning by using different threshold values.

For example, the information providing device 10 determines whether or not the feature amount extracted from the search target image is similar to the various feature amounts. Further, the information providing device 10 counts the number of times it is determined that the feature amount is similar to the various feature amounts for each feature amount extracted from the search target image. Then, the information providing device 10 extracts a feature amount whose counted number of times is equal to or more than a predetermined first threshold value as a first frequent feature amount, and sets the number of counted times to a second threshold value or more larger than the first threshold value. The feature amount is extracted as the second frequent feature amount. After that, the information providing device 10 searches for information similar to the query image by using a feature amount other than the first frequent feature amount. On the other hand, the information providing device 10 may make the model learn the features of the second frequent feature amount. As a result, the information providing device 10 can generate a learning model that accurately determines whether or not the feature amount extracted from the image is a frequent feature amount that is not useful for image retrieval.

[1-12. About the target of processing]
Here, in the above-mentioned example, the information providing device 10 searches for a search target image similar to the query image. However, the embodiments are not limited to this. For example, the information providing device 10 can apply the above-mentioned search process to arbitrary contents such as sentences and music.

For example, when the search target is a sentence, the information providing device 10 selects some words as feature points from the query sentence and the sentence to be searched, and selects a plurality of words existing before and after the selected word. Extract. Further, the information providing device 10 calculates the feature amount of the extracted word as the feature amount, and the relative appearance position of the word (for example, a word existing between the word selected as the feature point and the next word). , Etc.) are regarded as coordinates. Then, the information providing device 10 may extract the frequently appearing feature amount based on the comparison result between the feature amounts.

Further, for example, when the search target is voice, the information providing device 10 selects some playback positions as feature points from the query voice and the voice to be searched, and the voice existing before and after the selected playback position. The feature amount is calculated from the frequency characteristic of the above, and the information providing device 10 regards the reproduction position as a coordinate. Then, the information providing device 10 may extract the frequently appearing feature amount based on the comparison result between the feature amounts.

The information providing device 10 may perform information retrieval between different types. For example, the information providing device 10 may search for a target voice similar to the query text, or may search for a target text similar to the query text. Further, in such an information retrieval between different species, a frequently-appearing feature amount may be extracted, and similar information may be searched using a feature amount other than the extracted frequently-appearing feature amount. That is, the information providing device 10 applies the above-mentioned search process to any content as long as the content can set a multidimensional amount indicating a local feature as the feature amount, and extracts useful feature amounts. You may go.

[2. Configuration of information providing device]
Hereinafter, an example of the functional configuration of the information providing device 10 that realizes the above-mentioned search process will be described. FIG. 3 is a diagram showing a configuration example of the information providing device according to the embodiment. As shown in FIG. 3, the information providing device 10 includes a communication unit 20, a storage unit 30, and a control unit 40.

The communication unit 20 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 20 is connected to the network N by wire or wirelessly, and transmits / receives information to / from the terminal device 100.

The storage unit 30 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. Further, the storage unit 30 stores the search target database 31 and the learning model 32.

Target information to be searched is registered in the search target database 31. For example, FIG. 4 is a diagram showing an example of information registered in the search target database according to the embodiment. In the example shown in FIG. 4, the search target database 31 includes "search target ID (Identifier)", "search target data", "category", "feature amount", "similar feature amount number", and "type". Information is registered.

Here, the "search target ID" is an identifier that identifies the search target image. Further, the "search target data" is the data of the search target image. Further, the "category" is a category to which the transaction target related to the associated search target image belongs, and is, for example, information indicating a category to which the transaction target imaged in the electronic shopping district belongs. The "feature amount" is a feature amount. The "number of similar feature quantities" is the number of times determined to be similar to other feature quantities by the various processes described above. Further, the "type" is information indicating whether the associated feature amount is a frequent feature amount or another normal feature amount.

For example, in the example shown in FIG. 4, the search target database 31 contains the search target ID “T1”, the search target data “TD # 1”, the category “beverage”, the number of similar feature quantities “230”, and the type “frequent”. It is registered in association with each other. Such information is an image in which the data of the search target image indicated by the search target ID "T1" is "TD # 1", the transaction target related to the category "beverage" is captured, and the feature amount is "FT1-1". ". Further, in such information, the number of similar feature quantities of the feature quantity "FT1-1", that is, the number of other feature quantities similar to the feature quantity "FT1-1" is "230", and the type is "frequently appearing". That is, it indicates that it is a frequent feature amount.

It should be noted that the various values shown in FIG. 4 are conceptual, and in reality, numerical values indicating each feature amount and the like are registered in the search target database 31. Further, in the search target database 31, in addition to the information shown in FIG. 4, arbitrary information such as feature points corresponding to the feature amount and various information regarding the transaction target corresponding to the search target image may be registered. ..

Returning to FIG. 3, the explanation will be continued. The learning model 32 is a learning model that learns the features of the frequent features. For example, the learning model is various parameters of the learning model that has learned the features of the frequent feature quantities such as DNN and SVM.

The control unit 40 is a controller, and for example, various programs stored in a storage device inside the information providing device 10 by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) are RAM or the like. Is realized by executing as a work area. Further, the control unit 40 is a controller, and may be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 4, the control unit 40 includes an extraction unit 41, a determination unit 42, a counting unit 43, a search unit 44, a generation unit 45, and a learning unit 46.

The extraction unit 41 extracts a non-useful feature amount that is not useful for searching for similar information, that is, a frequently-appearing feature amount, from the feature amounts extracted from the query image or the search target image. For example, the extraction unit 41 extracts the frequently-appearing feature amount from the feature amount extracted from the search target image.

Further, the extraction unit 41 extracts a plurality of features that are similar to each other as frequently appearing features among the features extracted from the search target image. For example, the extraction unit 41 may extract frequently-appearing feature amounts based on the comparison results of the feature amounts extracted from the information belonging to different categories in the search target image. To give a more specific example, the extraction unit 41 is extracted from the feature amount extracted from the information related to the first transaction target and the information related to the second transaction target of a type different from the first transaction target. Frequent feature quantities may be extracted based on the comparison result with the feature quantities.

For example, when the extraction unit 41 receives the search target image from a predetermined information processing device (not shown), the extraction unit 41 extracts the feature amount of the search target image and searches the search target image and each feature amount in the search target database. Register at 31. Further, the extraction unit 41 refers to the search target database 31 at predetermined time intervals, refers to the number of similar features of each feature, and based on the number of referenced similar features, each feature frequently appears. It is determined whether or not it is. For example, the extraction unit 41 selects a predetermined number of features as a frequent feature amount in order from a feature amount in which the number of similar feature amounts exceeds a predetermined threshold value or a feature amount having a higher number of similar feature amounts. Then, the extraction unit 41 associates the type "frequent" with the feature amount selected as the frequent feature amount, and associates the type "normal" with the other feature amounts. That is, the extraction unit 41 extracts the feature amount in which the number of times counted by the counting unit 43 exceeds a predetermined threshold value as the frequent feature amount.

In addition, when extracting the frequent feature amount used for learning of the learning model 32, the extraction unit 41 may extract the frequent feature amount with a threshold value different from that when performing a similar image search. For example, the extraction unit 41 extracts a feature amount whose number of times counted by the counting unit 43 is equal to or higher than a predetermined first threshold value as a first frequent feature amount, and associates the extracted feature amount with the type “frequent”. Further, the extraction unit 41 uses the frequently-appearing feature amount registered in the search target database 31 as a frequently-appearing feature amount whose number of times counted by the counting unit 43 is greater than or equal to the second threshold value and is equal to or higher than the second threshold value. Extract as a quantity. Then, the extraction unit 41 notifies the learning unit 46 of the extracted second frequent feature amount as learning data.

The extraction unit 41 may extract the frequently-appearing feature amount from the feature amount extracted from the search target image by using the model learned by the learning unit. For example, the extraction unit 41 may input each feature amount of the search target image into the learning model 32 and determine whether or not each feature amount is a frequent feature amount. Further, the extraction unit 41 may use the learning model 32 to extract frequently occurring features from the features extracted from the query image. For example, when the extraction unit 41 receives the query image, the extraction unit 41 extracts the feature amount from the query image. Then, the extraction unit 41 may input the extracted feature amount into the learning model 32 and provide the determination unit 42 with a feature amount other than the feature amount determined to be a frequent feature amount.

Further, the extraction unit 41 may extract a predetermined number or less of frequent feature quantities from the feature quantities extracted from the query image or the search target image. For example, the extraction unit 41 may extract the frequently appearing feature amount so that the number of the feature amounts other than the frequently appearing feature amount among the feature amounts extracted from one image is equal to or more than a predetermined threshold value. For example, the extraction unit 41 may preferentially set a feature amount having a larger number of similar feature amounts or a feature amount estimated by the learning model 32 to have a higher probability of being a frequent feature amount as a frequent feature amount. Further, the extraction unit 41 may delete the extracted frequently-appearing feature amount from the search target database 31 in which the feature amount extracted from the search target image is registered.

The determination unit 42 determines whether or not the feature amount extracted from the search target image is similar to a plurality of predetermined feature amounts. For example, the determination unit 42 has a plurality of predetermined feature amounts, which are a predetermined number or more of the feature amounts and are generated by a predetermined method and the feature amounts extracted from the search target image. Determine if they are similar. Then, the counting unit 43 counts the determined number of times similar to the predetermined feature amount for each feature amount extracted from the search target image.

For example, the determination unit 42 generates a feature amount by a predetermined algorithm or randomly, and determines whether or not the generated feature amount is similar to each feature amount registered in the search target database 31. Then, the counting unit 43 adds 1 to the number of similar feature quantities of the feature quantities determined to be similar to the generated feature quantity among the feature quantities registered in the search target database 31.

Further, the determination unit 42 may determine whether or not the feature quantities registered in the search target database 31 are similar to each other. For example, the determination unit 42 selects one of the feature amounts registered in the search target database 31, and determines whether or not the selected feature amount is similar to the other feature amount. Then, the counting unit 43 adds 1 to the number of similar feature quantities of other feature quantities determined to be similar to the selected feature quantity.

Further, the determination unit 42 determines whether or not the feature amount extracted from the similar information generated by the generation unit 45 is similar to the feature amount extracted from the search target image. For example, the determination unit 42 includes a feature amount extracted from the similar information generated by the generation unit 45, and a feature amount extracted from information belonging to a category different from the image in which the similar information is generated among the search target images. Determine if they are similar. For example, when the determination unit 42 acquires the similar image generated by the generation unit 45, the determination unit 42 extracts the feature amount of the similar image. The extraction of such a feature amount may be realized by the extraction unit 41.

Subsequently, the determination unit 42 refers to the search target database 31 and identifies the category of the search target image that is the source of the similar image. Then, the determination unit 42 may determine whether or not the feature amount of the search target image associated with the category different from the specified category is similar to the feature amount extracted from the similar image. In such a case, the counting unit 43 adds 1 to the number of similar feature quantities of the feature quantity determined to be similar to the feature quantity extracted from the similar image.

Further, the determination unit 42 may determine, as a predetermined feature amount, whether or not the feature amount extracted from the query image and the feature amount extracted from the search target image are similar to each other. For example, the determination unit 42 determines whether or not the feature amount extracted from the query image is similar to the feature amount registered in the search target database 31. Then, the counting unit 43 adds 1 to the number of similar feature quantities of the feature quantity determined to be similar to the feature quantity extracted from the query image among the feature quantities registered in the search target database 31.

The determination unit 42 holds the query image received during the operation of the similar image search in the storage unit 30, and extracts from the feature amount extracted from the query image held in the storage unit 30 at a predetermined timing and the search target image. It may be determined whether or not the feature quantities are similar to each other. Further, the determination unit 42 may execute the above-mentioned process when searching for a search target image similar to the query image received during the operation of the similar image search.

For example, the determination unit 42 receives the feature amount of the query image extracted by the extraction unit 41 during the operation of the similar image search. Here, the feature amount received from the extraction unit 41 may be all the feature amounts extracted from the query image, or may be a feature amount other than the frequent feature amount extracted by the extraction unit 41. In such a case, the determination unit 42 associates each feature amount of the query image with the feature amount determined to be the most similar among the feature amounts other than the frequently appearing feature amounts registered in the search target database 31. At this time, the counting unit 43 adds 1 to the number of similar feature quantities of the feature quantities associated with each feature quantity of the query image.

Then, the determination unit 42 determines whether or not the coordinate relationships between the associated feature quantities are similar. That is, the determination unit 42 executes the feature amount matching process and the coordinate matching process described above.

The search unit 44 searches for information similar to the query image by using a feature amount other than the frequently-appearing feature amount. For example, the search unit 44 searches for a search target image similar to the query image based on the determination result by the determination unit 42. For example, the search unit 44 acquires the results of the feature amount matching process and the coordinate matching process by the determination unit 42 during the operation of the similar image search. Then, the search unit 44 selects an image to be searched, which is determined to have a similar positional relationship between similar feature quantities as a result of the coordinate matching process, as an image similar to the query image. Since the extraction unit 41 extracts the feature amount having the number of similar feature amounts equal to or higher than the first threshold value as the frequent feature amount, the search unit 44 uses the search unit 44 for features other than the feature amount having the similar feature amount number equal to or higher than the first threshold value. The amount is used to search for a search target image similar to the query image.

Further, the search unit 44 generates a search result in which various information such as thumbnails of images similar to the query image are arranged. Then, the search unit 44 provides the generated search result to the terminal device 100 that is the source of the query image.

The generation unit 45 generates a similar image from the search target image. For example, when the generation unit 45 extracts a frequent feature amount using a similar image, the generation unit 45 selects one of the search target images from the search target database 31 and uses various image conversion techniques to select the selected search target image. Generate a similar image of. Then, the generation unit 45 provides the generated similar image to the determination unit 42.

The learning unit 46 causes the model to learn the features of the frequent features. For example, the learning unit 46 causes the model to learn the features of the second frequent feature amount. For example, when the learning unit 46 receives the notification of the frequently-appearing feature amount extracted by the extraction unit 41, the learning unit 46 causes the learning model 32 to learn the features of the notified frequent-appearing feature amount. The learning unit 46 may, for example, cause the learning model 12 to learn the difference between the feature possessed by the frequently-appearing feature quantity and the feature possessed by the other feature quantity.

[3. An example of the flow of processing executed by the information providing device]
Subsequently, an example of the flow of processing executed by the information providing device 10 will be described with reference to FIG. FIG. 5 is a flowchart showing an example of the flow of the search process executed by the information providing apparatus according to the embodiment.

For example, the information providing device 10 compares the feature amount of the search target image with the predetermined feature amount (step S101), and counts a number similar to other feature amounts for each feature amount (step S102). Then, the information providing device 10 extracts a feature amount whose counted number exceeds a predetermined threshold value as a frequent feature amount (step S103).

Here, the information providing device 10 determines whether or not the query image has been acquired from the terminal device 100 (step S104), and if not (step S104: No), waits until the query image is acquired. Then, when the query image is acquired from the terminal device 100 (step S105: Yes), the information providing device 10 compares the feature amount of the query image with the feature amount other than the frequent feature amount (step S105), and the comparison result. Based on the above, an image similar to the query image is searched from the search target image (step S106), and the process is terminated.

[4. Modification example]
In the above, an example of the search process by the information providing device 10 has been described. However, the embodiments are not limited to this. Hereinafter, variations of the search process will be described.

[4-1. Comparison of features]
In the above-mentioned example, the information providing device 10 compares the feature amount of the query information with the query feature amount of the information to be searched (hereinafter, may be referred to as “target information”). Here, the information providing device 10 may use a feature amount of arbitrary information as a feature amount. For example, when the search target is an image, the information providing device 10 may use the average value of the saturation and brightness of the pixels around the feature point as the feature quantity, and when the search target is voice, the periphery of the feature point. A frequency characteristic around several seconds may be used as a feature quantity. In addition to the above-mentioned processing, the information providing device 10 may also use an arbitrary feature amount comparison technique.

[4-2. others〕
Further, among the processes described in the above-described embodiment, all or a part of the processes described as being automatically performed can be manually performed, and conversely, the processes described as being manually performed. It is also possible to automatically perform all or part of the above by a known method. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown in the figure.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in any unit according to various loads and usage conditions. Can be integrated and configured.

In addition, the above-described embodiments can be appropriately combined as long as the processing contents do not contradict each other.

[4-3. program〕
Further, the information providing device 10 and the terminal device 100 according to the above-described embodiment are realized by, for example, a computer 1000 having a configuration as shown in FIG. FIG. 6 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to the output device 1010 and the input device 1020, and the arithmetic unit 1030, the primary storage device 1040, the secondary storage device 1050, the output IF (Interface) 1060, the input IF 1070, and the network IF 1080 are connected by the bus 1090. Have.

The arithmetic unit 1030 operates based on a program stored in the primary storage device 1040 or the secondary storage device 1050, a program read from the input device 1020, or the like, and executes various processes. The primary storage device 1040 is a memory device that temporarily stores data used by the arithmetic unit 1030 for various operations such as RAM. Further, the secondary storage device 1050 is a storage device in which data used by the calculation device 1030 for various calculations and various databases are registered, such as a ROM (Read Only Memory), an HDD (Hard Disk Drive), and a flash memory. Is realized by.

The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various information such as a monitor and a printer. For example, USB (Universal Serial Bus), DVI (Digital Visual Interface), and the like. It is realized by a connector of a standard such as HDMI (registered trademark) (High Definition Multimedia Interface). Further, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, a keyboard, a scanner, and the like, and is realized by, for example, USB.

The input device 1020 is, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), or a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. Further, the input device 1020 may be an external storage medium such as a USB memory.

The network IF 1080 receives data from another device via the network N and sends it to the arithmetic unit 1030, and also transmits the data generated by the arithmetic unit 1030 to the other device via the network N.

The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 10, the arithmetic unit 1030 of the computer 1000 realizes the function of the control unit 40 by executing the program loaded on the primary storage device 1040. Further, for example, when the computer 1000 functions as the terminal device 100, the arithmetic unit 1030 of the computer 1000 realizes the function of the control unit 40 by executing the program loaded on the primary storage device 1040.

[5. effect〕
As described above, the information providing device 10 is not useful for searching for similar information such as a frequent feature amount among the feature amounts extracted from the query information such as a query image or the information to be searched such as a search target image. Extract no non-useful feature quantities. Then, the information providing device 10 searches for information similar to the query information by using the feature amount other than the extracted frequent feature amount. As a result, the information providing device 10 improves the search accuracy of similar information and reduces the search cost in the time required for searching for similar information, so that the efficiency in information retrieval can be improved.

Further, the information providing device 10 extracts the frequently appearing feature amount from the feature amount extracted from the information to be searched. Further, the information providing device 10 extracts a plurality of features that are similar to each other as frequently appearing features among the features extracted from the information to be searched. For example, the information providing device 10 determines whether or not the feature amount extracted from the information to be searched is similar to a plurality of predetermined feature amounts. Further, the information providing device 10 counts a determined number of times similar to a predetermined feature amount for each feature amount extracted from the information to be searched. Then, the information providing device 10 extracts a feature amount in which the number of times counted exceeds a predetermined threshold value as a frequent feature amount. Therefore, the information providing device 10 can appropriately extract frequently-appearing feature quantities that are not useful for information retrieval.

Further, the information providing device 10 has a plurality of predetermined feature quantities, which are a predetermined number or more of the feature quantities, and are the feature quantities generated by a predetermined method and the features extracted from the information to be searched. Determine if the quantity is similar. Further, the information providing device 10 generates similar information similar to the information from the information to be searched, and the feature amount extracted from the generated similar information and the feature amount extracted from the information to be searched. Determine if and are similar. Here, the information providing device 10 includes a feature amount extracted from the generated similar information and a feature amount extracted from information belonging to a category different from the information that generated the similar information among the information to be searched. May be determined whether or not they are similar. Further, the information providing device 10 may determine whether or not the feature quantities extracted from the information to be searched are similar to each other. Further, the information providing device 10 may determine whether or not the feature amount extracted from the query information and the feature amount extracted from the information to be searched are similar to each other as a predetermined feature amount. By executing such a process, the information providing device 10 can appropriately extract frequently-appearing feature quantities that are not useful for information retrieval, so that the efficiency of information retrieval can be improved.

Further, the information providing device 10 deletes the frequently-appearing feature amount extracted by the extraction unit from the storage device in which the feature amount extracted from the information to be searched is registered. As a result, the information providing device 10 can improve the efficiency in information retrieval.

Further, the information providing device 10 extracts frequently appearing feature quantities of a predetermined number or less from the feature quantities extracted from the query information or the information to be searched. Therefore, the information providing device 10 can improve the efficiency of information retrieval while preventing deterioration of accuracy in information retrieval.

Further, the information providing device 10 causes the model to learn the features of the extracted frequent feature quantities. Then, the information providing device 10 uses the trained model to extract the frequently-appearing feature amount from the feature amount extracted from the query information. Further, the information providing device 10 uses the learned model to extract the frequently-appearing feature amount from the feature amount extracted from the information to be searched. Therefore, the information providing device 10 can improve the efficiency of information retrieval.

Further, the information providing device 10 determines whether or not the feature amount extracted from the information to be searched is similar to a plurality of predetermined feature amounts, and for each feature amount extracted from the information to be searched. In addition, the number of times determined to be similar to the predetermined feature amount is counted, and the feature amount having the counted number of times equal to or higher than the predetermined first threshold value is extracted as the first frequent feature amount, and the counted number of times is the first. A feature amount having a second threshold value larger than the threshold value and equal to or higher than the second threshold value is extracted as a second frequent feature amount. Then, the information providing device 10 searches for information similar to the query information using a feature amount other than the first frequent feature amount, and causes the model to learn the feature of the second frequent feature amount. Therefore, the information providing device 10 can learn a model capable of extracting frequently occurring features with high accuracy.

Further, the information providing device 10 extracts the frequently-appearing feature amount based on the comparison result between the feature amounts extracted from the information belonging to different categories among the information to be searched. For example, the information providing device 10 has a feature amount extracted from information related to the first transaction target and a feature amount extracted from information related to a second transaction target of a type different from the first transaction target. Frequent features are extracted based on the comparison results. Therefore, the information providing device 10 can improve the efficiency of information retrieval because the feature quantity that is not useful can be used as the frequent feature quantity while retaining the feature quantity that is useful for information retrieval.

Further, the information providing device 10 extracts frequently-used feature amounts that are not useful for searching for similar images from the feature amounts extracted from the query image or the image to be searched. Therefore, the information providing device 10 can improve the efficiency of similar image retrieval.

Although some of the embodiments of the present application have been described in detail with reference to the drawings, these are examples, and various modifications are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure column of the invention. It is possible to carry out the present invention in other modified forms.

Further, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the extraction unit can be read as an extraction means or an extraction circuit.

10 Information providing device 20 Communication unit 30 Storage unit 31 Search target database 32 Learning model 40 Control unit 41 Extraction unit 42 Judgment unit 43 Counting unit 44 Search unit 45 Generation unit 46 Learning unit 100 Terminal device

Claims (23)

Of the feature quantities extracted from the query information or the information to be searched, the extraction unit that extracts the non-useful feature quantities that are not useful for searching similar information, and the extraction unit.
It has a search unit that searches for information similar to query information by using a feature amount other than the non-useful feature amount extracted by the extraction unit.
The extraction unit extracts the non-useful feature amount based on the comparison result between the feature amounts extracted from the information belonging to different categories from the information to be searched.
A search device characterized by that. Of the feature quantities extracted from the query information or the information to be searched, the extraction unit that extracts the non-useful feature quantities that are not useful for searching similar information, and the extraction unit.
A search unit that searches for information similar to query information using features other than the non-useful features extracted by the extraction unit.
Have,
The extraction unit is based on the comparison result between the feature amount extracted from the information related to the first transaction target and the feature amount extracted from the information related to the second transaction target of a type different from the first transaction target. Based on this, the non-useful features are extracted.
A search device characterized by that. The search device according to claim 1 or 2 , wherein the extraction unit extracts the non-useful feature amount from the feature amount extracted from the information to be searched. The search device according to claim 3 , wherein the extraction unit extracts a plurality of feature quantities similar to each other as the non-useful feature quantities among the feature quantities extracted from the information to be searched. .. A determination unit for determining whether or not the feature amount extracted from the information to be searched is similar to a plurality of predetermined feature amounts.
For each feature amount extracted from the information to be searched, it has a counting unit for counting a determined number of times similar to the predetermined feature amount.
The search device according to claim 3 or 4 , wherein the extraction unit extracts a feature amount whose number of times counted by the counting unit exceeds a predetermined threshold value as the non-useful feature amount. The determination unit is a plurality of predetermined feature quantities, which are a predetermined number or more of the feature quantities and are generated by a predetermined method and the feature quantities extracted from the information to be searched. The search device according to claim 5 , wherein it is determined whether or not the two are similar to each other. It has a generator that generates similar information similar to the information from the information to be searched.
The determination unit is characterized in that it determines whether or not the feature amount extracted from the similar information generated by the generation unit is similar to the feature amount extracted from the information to be searched. Item 5. The search device according to item 5 or 6 . A generator that generates similar information similar to the information from the information to be searched,
A determination unit for determining whether or not the feature amount extracted from the information to be searched is similar to the feature amount extracted from a plurality of information belonging to a category different from the information that generated the similar information.
For each feature amount extracted from the information to be searched, a counting unit for counting the number of times determined to be similar to other feature amounts, and a counting unit.
Of the features extracted from the information to be searched, the features whose number of times counted by the counting unit exceeds a predetermined threshold are extracted, and the non-useful features that are not useful for searching similar information are extracted. Department and
A search unit that searches for information similar to query information using features other than the non-useful features extracted by the extraction unit.
A search device characterized by having . The search device according to any one of claims 5 to 8 , wherein the determination unit determines whether or not the feature amounts extracted from the plurality of search target information are similar to each other. .. Any of claims 5 to 9 , wherein the determination unit determines whether or not the feature amount extracted from the query information is similar to the feature amount extracted from the information to be searched. The search device described in one. Any of claims 3 to 10 , wherein the storage device has a deletion unit for deleting the non-useful feature amount extracted by the extraction unit from the storage device in which the feature amount extracted from the information to be searched is registered. The search device described in one. One of claims 1 to 1 is characterized in that the extraction unit extracts the non-useful feature amount of a predetermined number or less from the feature amount extracted from the query information or the information to be searched. The search device described in one. The search device according to any one of claims 1 to 12, wherein the search device has a learning unit that allows a model to learn the characteristics of the non-useful feature amount extracted by the extraction unit. The search device according to claim 13 , wherein the extraction unit extracts the non-useful feature amount from the feature amount extracted from the query information by using the model learned by the learning unit. Claim 1 3 or 14 characterized in that the extraction unit extracts the non-useful feature amount from the feature amount extracted from the information to be searched by using the model learned by the learning unit. The search device described in. A determination unit for determining whether or not the feature amount extracted from the information to be searched and a plurality of predetermined feature amounts are similar, and
For each feature amount extracted from the information to be searched, it has a counting unit for counting the number of times determined to be similar to the predetermined feature amount.
The extraction unit extracts a feature amount whose number of times counted by the counting unit is equal to or greater than a predetermined first threshold value as a first non-useful feature amount, and the number of times counted by the counting unit is from the first threshold value. A feature amount having a large second threshold value or more is extracted as a second non-useful feature amount.
The search unit searches for information similar to the query information using a feature amount other than the first non-useful feature amount.
The search device according to any one of claims 1 to 15 , wherein the learning unit causes the model to learn the features of the second non-useful feature amount. Claims 1 to 1 are characterized in that the extraction unit extracts a non-useful feature amount that is not useful for searching a similar image from the feature amounts extracted from the query image or the image to be searched. The search device according to any one of 6 . It is a search method executed by the search device.
Of the feature quantities extracted from the query information or the information to be searched, the extraction process for extracting non-useful feature quantities that are not useful for searching similar information, and
Including a search step of searching for information similar to query information using a feature amount other than the non-useful feature amount extracted by the extraction step.
The extraction step extracts the non-useful feature amount based on the comparison result between the feature amounts extracted from the information belonging to different categories from the information to be searched.
A search method characterized by that. Extraction procedure for extracting non-useful features that are not useful for searching similar information from the features extracted from query information or information to be searched, and
It is a search program for causing a computer to execute a search procedure for searching information similar to query information by using a feature amount other than the non-useful feature amount extracted by the extraction procedure .
The extraction procedure extracts the non-useful feature amount based on the comparison result between the feature amounts extracted from the information belonging to different categories from the information to be searched.
A search program that features that . It is a search method executed by the search device.
Of the feature quantities extracted from the query information or the information to be searched, the extraction process for extracting non-useful feature quantities that are not useful for searching similar information, and
A search step for searching for information similar to query information using a feature amount other than the non-useful feature amount extracted by the extraction step.
Including
The extraction step is based on the comparison result between the feature amount extracted from the information related to the first transaction target and the feature amount extracted from the information related to the second transaction target of a type different from the first transaction target. Based on this, the non-useful features are extracted.
A search method characterized by that. Extraction procedure for extracting non-useful features that are not useful for searching similar information from the features extracted from query information or information to be searched, and
A search procedure for searching for information similar to query information using features other than the non-useful features extracted by the extraction procedure.
Is a search program that allows a computer to run
The extraction procedure is based on the comparison result between the feature amount extracted from the information related to the first transaction target and the feature amount extracted from the information related to the second transaction target of a type different from the first transaction target. Based on this, the non-useful features are extracted.
A search program that features that. It is a search method executed by the search device.
A generation process that generates similar information similar to the information from the information to be searched,
A determination step of determining whether or not the feature amount extracted from the information to be searched is similar to the feature amount extracted from a plurality of information belonging to a category different from the information that generated the similar information.
For each feature amount extracted from the information to be searched, a counting step for counting the number of times determined to be similar to other feature amounts, and
Of the features extracted from the information to be searched, the features whose number of times counted by the counting step exceeds a predetermined threshold are extracted, and the non-useful features that are not useful for searching similar information are extracted. Process and
A search step for searching for information similar to query information using a feature amount other than the non-useful feature amount extracted by the extraction step.
A search method characterized by including. A generation procedure for generating similar information similar to the information to be searched, and a generation procedure.
A determination procedure for determining whether or not the feature amount extracted from the information to be searched is similar to the feature amount extracted from a plurality of information belonging to a category different from the information that generated the similar information.
For each feature amount extracted from the information to be searched, a counting procedure for counting the number of times determined to be similar to other feature amounts, and a counting procedure.
Of the features extracted from the information to be searched, the features whose number of times counted by the counting procedure exceeds a predetermined threshold are extracted, and the non-useful features that are not useful for searching similar information are extracted. Procedure and
A search procedure for searching for information similar to query information using features other than the non-useful features extracted by the extraction procedure.
A search program that lets your computer run.

Images