JP5345918B2 - Document search method, document search apparatus, and document search program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce a calculation cost in retrieving retrieval words by previously performing the calculation of distances or similarities between the feature vectors of words constituting the retrieval words and the feature vectors of documents to be retrieved. <P>SOLUTION: In a step S3 for selecting the documents to be retrieved from the aggregation of the documents to be retrieved, a retrieval result determining means selects the documents to be retrieved corresponding to the distances from the aggregation of the documents to be retrieved in the ascending order of the distances between the feature vectors of the retrieval words and the feature vectors of the documents to be retrieved, which are calculated by obtaining, with the use of the weights of the words, the weightsum of the distances between the feature vectors of the words constituting the prescribed retrieval words and the feature vectors of the documents to be retrieved, which are previously calculated via steps S1, S2. The similarities may be adapted in place of the distances. In this case, the documents to be retrieved corresponding to the similarities are selected from the aggregation of the documents to be retrieved in the descending order of the similarities. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to a technique for expressing features of a document by a vector based on a weighted average of feature vectors of words that are constituent elements, and performing a high-speed search when performing a similar search based on a distance between vectors.

  With the spread of the Internet, there is an increasing need to search a huge amount of documents at high speed. In search technology, a method is widely used in which document features are represented by vectors and based on the distance between the vectors.

  Up to now, many techniques for speeding up multidimensional vector searches have been proposed. Prior to 1998, a multidimensional vector search technique mainly using a tree structure was proposed.

  However, a multidimensional vector search technique using a tree structure causes a so-called “curse of dimension” problem when the number of dimensions increases, and requires a cost equivalent to that of linear search (see, for example, Non-Patent Document 1). Therefore, a speed-up technique using a data structure called VA-file (see Non-Patent Document 2) and a locality-detectable hash (LSH (Locality Sensitive Hashing)) that overcomes this VA-file problem (Non-Patent Document) 3) has been proposed. While LSH guarantees the search accuracy stochastically, the search cost is (hashed number) × (number of dimensions). Theoretically, it functions even when the number of dimensions is large, and has a feature that a similar search can be performed at high speed, and is the most promising as a high-speed similar search technique.

Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, Uri Shaft, `` When Is `` Nearest Neighbor '' Meaningful? '', ICDT 2005, p.158-172 Roger Weber, Hans-J.Schek, Stephen Blott, “A Quantitative Analysis and Performance Study for Similarity-Search Methods in High-Dimensional Spaces”, Very Large Data Bases, 1998, p.194-205 Piotr Indyk, Rajeev Motwani, "Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality", Annual ACM Symposium on Theory of Computing, 1998, p.603-613 Kita Kenji, Tsuda Kazuhiko, Sasabori Masatomi, “Information Retrieval Algorithm”, Kyoritsu Shuppan, January 1, 2002, p. 34-35

  In the above prior art, there is a problem that the cost increases in proportion to the number of dimensions of the target vector in any method.

  In the present invention, when the feature vector of a search word (document) is calculated by the weighted average of the feature vectors of the words that are the constituent elements of the search word, an increase in cost proportional to the number of dimensions of the vector is avoided, and high speed is achieved. The purpose is to perform search processing.

  In the present invention, distance calculation is performed in advance between the feature vectors of all words that can be components of the search word (document) and the feature vectors of the searched document. The magnitude relationship between the distance between the feature vector of the search word and the feature vector of the searched document that is the search target is the magnitude relationship of the weighted sum of the distance between the feature vector of the word and the feature vector of the searched document. Based on the property that is in an equivalence relation, a weighted sum of distances calculated in advance is calculated at the time of retrieval. This avoids calculations proportional to the number of vector dimensions. The same applies when the similarity is replaced instead of the distance.

  That is, the document search method of the present invention is configured such that when a feature vector of a predetermined search word is represented by a weighted average using the weight of the word of the feature vector of the word constituting the search word, A document search method for selecting a document to be searched based on a distance or similarity with a feature vector of a search word, wherein the search result determination means and a feature vector of a word constituting a predetermined search word and a search target The ascending order of the distance or the descending order of the similarity between the feature vector of the search word and the feature vector of the searched document calculated by weighting and summing the distance or similarity with the feature vector of the document using the weight of the word And a step of selecting a searched document corresponding to the distance or similarity from the set of searched documents.

  Further, the document search device of the present invention is configured such that when a feature vector of a predetermined search word is represented by a weighted average using the weight of the word of the feature vector of the word constituting the search word, A document search apparatus that selects a document to be searched based on a distance or similarity with a feature vector of a search word, and includes a feature vector of a word constituting a predetermined search word calculated in advance and a feature vector of the document to be searched In the ascending order of the distance between the feature vector of the search word calculated by weighting and summing the distance or the similarity using the weight of the word and the feature vector of the searched document or in descending order of the similarity, Search result determining means for selecting a search target document corresponding to the distance or similarity from the set is provided.

The principle of the present invention will be described. First, the feature vector of the search word (document) is d, the feature vector of the search target document (K is assumed) is g _k (k = 1,..., K), and the word (n) included in the search word (document). X _i (i = 1,..., N), the word weight is represented by T _i , and the total weight is represented by T _all . Then, when the weight normalized by summing the weights to 1 is represented by t _i , the following equation is established.

  Here, the “weight” is obtained by multiplying the “local weight” calculated based on the appearance frequency of the word and the “global weight” of the word in consideration of the entire document set. That way is shown.

The precondition “when the feature vector of a search word (document) is calculated by a weighted average of the feature vectors of words that are constituent elements of the search word”, which is a precondition that the present invention can be used, The following expression (1) is shown by using the word x _i (i = 1,..., N) and its weight t _i (i = 1,..., N), which are constituent elements of the search term.

At this time, the characteristic magnitude of the vector d and the distance between the feature vector g _k of each of the retrieved documents (or similarities) (for different k) is some distance (similarity) defined search terms (document) When used, this is a weighted sum of the distance (or similarity) between the feature vector x _i of the word that is a component of the search term and the feature vector g _k of the searched document.

When such a relationship exists, calculation of the distance (or similarity) between the feature vector x _i and the feature vector g _k requires a calculation cost depending on the number of dimensions of the vector. If the calculation is performed in advance as in the document search method and document search apparatus according to the present invention, this calculation can be avoided in the actual search.

  That is, the document search method and apparatus shown in FIG. 1 and FIG. 2 are weighted using the feature vector of a predetermined search word (or document) using the weight of the word of the feature vector of the word constituting the search word. A document search method and apparatus for selecting a search target document based on a distance or similarity to a feature vector of the search term from a set of search target documents when represented by an average, wherein a search result determination unit calculates in advance The feature vector of the search word calculated by weighting and summing the distance or similarity between the feature vector of the word constituting the predetermined search word and the feature vector of the searched document using the weight of the word, and the Extracted in ascending order of the distance from the feature vector of the searched document or descending order of similarity For words constituting the predetermined search word, the distance between the vector of the word and the feature vector of the searched document is calculated in advance.

  As a specific mode of the above document search method, as shown in FIG. 1, the appearance frequency of each word constituting the inputted predetermined search word in the search word is calculated and stored in advance. A word weight calculating step (S1) in which the word weight calculating means calculates the weight of the word from the importance of the word, the feature vector of the word calculated for each word, and the feature of the searched document The calculation for calculating the distance between the feature vector of the search word and the feature vector of the searched document as the weighted sum using the calculated weight of the word is calculated for all the searched documents. A distance calculating step (S2) performed by the calculating means; and a search result corresponding to each distance selected by the search result determining means in the ascending order of the calculated distances from the set of searched documents. Having a search result determination step (S3) and to search results. Similarity may be used instead of distance, and the height of similarity may be used instead of distance. In this case, a search target document corresponding to each similarity selected in descending order among the calculated similarities is set as a search result.

  As an aspect of the document search apparatus corresponding to the document search method, an input unit 11 for inputting a predetermined search word, a word importance storage unit 12 for storing the importance of a word, and the input search word A word weight calculating means 13 for calculating the weight of the word from the appearance frequency of each word constituting the search word and the importance of the word extracted from the word importance storage means 12; The distance between the feature vector of the word stored in the word / search target distance storage means 14 and the feature vector of the searched document is weighted using the weight of the word calculated by the word weight calculation means 13. A distance calculation means 15 for performing a calculation for calculating all the searched documents as a distance between the feature vector of the search term and the feature vector of the search document; And a search result determination unit 16, the search results to be retrieved documents corresponding to each a selected distance in order from smallest of distances the calculated from a set of documents. It should be noted that means that uses similarity instead of distance and uses high similarity instead of distance is also included.

  In the document search method and apparatus according to the above invention, as the distance or similarity index, the magnitude relationship between the distance between the feature vector of the predetermined search word and the feature vector of the searched document or the value of similarity is the search. A distance or similarity index that is equivalent to the distance between the feature vector of the word constituting the word and the feature vector of the document to be searched or the magnitude relation of the weighted sum of the similarities is used.

  Examples of the distance index include any one of Euclidean square distance between the feature vector of the word and the feature vector of the search target document, Cullback-Liber divergence, and cross-entropy using this divergence. An example of the similarity index is an inner product of both feature vectors.

  That is, when using the Euclidean square distance of both vectors as a distance index, the following equation (2) is obtained.

  Also in the following example, “weight” when considering “weighted sum of distance between feature vector of word and feature vector of searched document” may be any before and after normalization.

When the inner product of both feature vectors x _i and g _k is used as an index of similarity, the following equation (3) is obtained.

This is the weighted sum of inner products of the feature vector x _i of the word and the feature vector g _k of the searched document as it is. The weighted sum of the inner products of both vectors can be regarded as the weighted sum of the similarity between the vectors.

In addition, when using the Cullback / Lailer divergence of both feature vectors x _i and g _k as an index of distance, if both feature vectors x _i and g _k are regarded as probability distributions, the vector g _{k changes} to the vector x _i . Can be regarded as a weighted sum of the similarity between the vector x _i and the vector g _k . That is, when the cullback, librarian divergence from the feature vector d to the feature vector g _k is calculated, the following equation (4) is obtained.

In addition, when the feature vectors x _i and g _k of both are regarded as probability distributions, as another index of the distance between the two, the vector x using the kullback, librarian divergence from the vector g _k to the vector x _i Cross entropy of _i and g _k can be mentioned. The weighted sum of the cross entropy can also be regarded as a weighted sum of the distance between the vector x _i and the vector g _k .

  It should be noted that the present invention may be a document search program for causing a computer to function as each means constituting the document search apparatus.

  According to the above invention, since the distance or similarity between the feature vector of the word constituting the search word and the feature vector of the searched document can be calculated in advance, the calculation cost when searching for the search word is reduced. The

The flowchart figure explaining the principle of the one aspect | mode of the document search method of this invention. 1 is a block diagram showing an aspect of a document search apparatus according to the present invention. 1 is a block diagram showing a document search apparatus according to an embodiment of the present invention. The flowchart figure explaining the document search method concerning embodiment of this invention.

  Hereinafter, a document search apparatus 100 according to an embodiment of the invention will be described with reference to FIG.

  As shown in FIG. 3, the document search apparatus 100 includes a CPU 51, a memory 52, a display 53, a keyboard 54, a processing program 55, a processing target storage means 56, an OS 57, a word / search target distance storage means 58, and a word importance storage. Means 59.

  The CPU 51 functions as each means for executing steps S101 and S102 shown in FIG. 4 and steps S103 to S106 according to the invention in cooperation with a processing program 55 operating on an OS (operating system) 57.

  That is, the CPU 51 uses the weight of the word to calculate the distance between the feature vector of the word constituting the predetermined search word calculated in advance and the feature vector of the searched document. Search result determination for executing a step (S106) of selecting a search target document corresponding to the distance from a set of search target documents in ascending order of distance between the feature vector and the feature vector of the search target document (in ascending order). Functions as a means. Similarity may be applied instead of the distance. In this case, the search target document corresponding to the distance is selected from the set of search target documents in descending order of similarity between the feature vector of the search word and the feature vector of the search target document (in descending order).

  Further, the CPU 51 calculates the weight of the words constituting the predetermined search word and calculates the distance between the feature vector of the search word and the feature vector of the searched document by a weighted sum using the weight. When not performed (S103), it functions as a word weight calculating means for executing the step (S104) of calculating the weight of the word from the appearance frequency of the word in the search word and the importance of the word. Further, the CPU 51 sets the distance between the feature vector of the word and the feature vector of the searched document as a weighted sum using the calculated weight of the word, and the feature vector of the search word and the feature of the searched document. It is calculated as a distance to the vector, and functions as a calculation means for executing the step (S105) in which this distance is calculated for all searched documents. Similarity may be applied instead of the distance.

  The memory 52 stores a search word (or document) that is a processing target input by the keyboard 54 that is one mode of the input unit 11 shown in FIG. 2 or the processing target extracted from the processing target storage unit 56. This is temporarily stored for use in processing related to 55. The memory 52 also temporarily stores information from the storage means 58 and 59 for use in processing related to the processing program 55.

  The display 53 is a display unit for displaying a search word input by the keyboard 54 or a search result obtained in steps S101 to S106 described later based on the search word.

  The processing program 55 is an application program for causing the CPU 51 to execute S101 to S106 as described above, and is configured to operate on the OS (operating system) 57. In the above procedure, S103 to S105 are procedures corresponding to the function means 13 to 15 shown in FIG. The processing program 55 is stored in a storage unit exemplified by a hard disk device in which the OS 57 is installed.

  The processing target storage means 56 stores the search term input by the keyboard 54 as a processing target. The processing target storage means 56 may be a known storage means exemplified in the form of a hard disk device or a file server.

  The word / search target distance storage means 58 stores the distance between the feature vector of the word calculated in advance and the feature vector of the searched document. Examples of the distance include any one of the square distance defined by the above-described equation (2), the divergence defined by the above-described equation (4), and the cross entropy to which this divergence is applied. The storage means 58 stores the similarity between the vectors instead of the distance. As the similarity, for example, the inner product defined by the above-described equation (3) can be given.

  The word importance storage unit 59 stores in advance the importance for calculating the weight of the word from the appearance frequency in the predetermined search word and the importance of the word constituting the search word. Examples of the importance include “global weight” disclosed in Non-Patent Document 4.

  The storage means 58 and 59 may be a known storage means in the same manner as the storage means 56.

  Steps S101 to S106 executed by the CPU 51 of the document search apparatus 100 will be described with reference to the flowchart shown in FIG. In S101 to S106, the distance between the feature vector of the word and the feature vector of the searched document is calculated in advance. As the distance, for example, the Euclidean square distance of the two vectors represented by the above-described equation (2) is applied.

  S 101: A search word (or document) to be processed is extracted from the processing target storage unit 56 and read into the memory 52.

S102: All elements of the array of distances between the feature vector of the search word on the memory 52 and the feature vector of the search target (searched document) are initialized to zero. The number of elements of the distance array D is equal to the number of documents to be searched K, and each element is represented by D _k (k = 1,..., K).

  S103: The words that are not subjected to the processing of S104 and S105 are extracted from the words constituting the search word (or document) read onto the memory 52. If there is no unprocessed word, the process proceeds to S106.

  S104: The importance level of the word to be processed is extracted from the word importance level storage means 59, and the product of the word frequency and the appearance frequency in the search word is used as the weight of the word. For example, in determining the weight, a method of determining the weight of “index word” (corresponding to “word” in the present invention) described in Non-Patent Document 4 may be applied.

  S105: For the word to be processed, the distance between the feature vector of the word and the feature vector of the search target document that is the search target (this is array data having the number K of search target numbers) is stored as the word / search target distance storage means 58. Then, the value obtained by multiplying each element of the extracted distance by the weight of the word calculated in S104 is added to each element of the array D indicating the distance between the feature vector of the search word and the feature vector of the search target.

This processing can be expressed as the following equation (5), where T is the word weight, and D ^w _k is the k-th element of the distance between the word feature vector and the feature vector to be searched.

  S103 to S105 are executed until there is no unprocessed word constituting the search word.

  S106: The element values of the feature vector of the search word, the feature vector of the search target, and the distance array D are sorted in ascending order, and the search target corresponding to the top Y items is output as the search result. This search result is output from the display 53 and displayed. The number of cases Y may be determined in advance or may be determined by the user inputting an arbitrary number by operating the keyboard 54.

  In S101 to S106 described above, the distance between the feature vector of the word and the feature vector of the searched document is calculated in advance, but the similarity may be calculated instead of the distance. In this case, in S106, the element values of the search term feature vector, the search target feature vector, and the similarity array D are sorted in descending order, and the search target corresponding to the top Y items is output as the search result. As the similarity index, for example, the inner product of both feature vectors represented by the above-described equation (3) may be applied. Further, as the distance index, instead of the Euclidean square distance, any of the Cullback-Liber divergence expressed by the above-described equation (4) and the cross entropy using this divergence may be applied.

  As described above, the document search apparatus 100 according to the embodiment of the present invention assumes that the feature vector of the search word (or search document) is given by the weighted average of the feature vector of the word, and the feature vector of the word and the searched document The distance or similarity between the feature vector of the search term and the feature vector of the searched document is regarded as a weighted average of the distance or the similarity with the feature vector. The weighted sum of the distance or similarity between the feature vector of the word constituting the search word and the feature vector of the searched document based on the magnitude relationship between the distance between the feature vector of the search word and the feature vector of the searched document or the value of the similarity This is because there is an equivalence relationship with the magnitude relationship of the values of. Therefore, if the distance or similarity between the word and the document to be searched is calculated in advance, the search can be performed only by comparing the weighted averages.

  Therefore, according to the document search apparatus 100, the calculation of the distance or the similarity between the feature vector of the word constituting the search word and the feature vector of the searched document can be performed in advance, and the calculation cost is high. The calculation cost when searching for a search term is reduced.

  In addition, since a search target document corresponding to the distance or similarity is output from the set of search target documents in ascending order of distance between the feature vector of the search word and the feature vector of the search target document or in descending order of similarity. And efficient search for users is realized.

  When the processing program 55 is stored in a computer-readable recording medium, the program itself read from the medium by the computer realizes functional means constituting the document search apparatus according to the present invention. Therefore, the processing program 55 or a recording medium storing the processing program 55, such as a CD-ROM, a DVD-ROM, a CD-R, an MO, or an HDD, also constitutes one aspect of the present invention.

12, 59 ... Word importance storage means 13 ... Word weight calculation means 14, 58 ... Word / search target distance storage means 15 ... Distance calculation means 16 ... Search result determination means 55 ... Processing program (document search program)
100: Document search device

Claims (7)

When a feature vector of a predetermined search word is represented by a weighted average using the weight of the word of the word constituting the search word, the distance or similarity between the search document set and the feature vector of the search word A document search method for selecting a search target document based on a degree,
Calculating the weight of the word from the appearance frequency of each word constituting the predetermined search word inputted by the word weight calculating means and the importance of the word calculated in advance;
The distance calculation means calculates the distance or similarity between the feature vector of the word calculated in advance for each word and the feature vector of the searched document as a weighted sum using the calculated weight of the word. Performing a calculation for all searched documents to calculate the distance or similarity between the feature vector of the search term and the feature vector of the searched document;
The search result determining means determines the search target document corresponding to each distance selected in ascending order or the similarity selected in descending order among the calculated distances or similarities from the set of search target documents as a search result. Step and
Document search method characterized by having a. As the distance or similarity index, the feature vector of the word constituting the search word and the search target document are the magnitude relationship between the distance or similarity value of the feature vector of the predetermined search word and the feature vector of the search target document. The document search method according to claim 1 , wherein a distance or similarity index that is equivalent to a magnitude relationship between a distance to a feature vector or a weighted sum of similarity values is used. The index of the distance is any one of a Euclidean square distance between the feature vector of the word and the feature vector of the searched document, Cullback-Liber divergence, and cross-entropy using this divergence,
The document search method according to claim 2 , wherein the similarity index is an inner product of the feature vectors of the two . When a feature vector of a predetermined search word is represented by a weighted average using the weight of the word of the word constituting the search word, the distance or similarity between the search document set and the feature vector of the search word A document search device for selecting a search target document based on a degree,
Word weight calculating means for calculating the weight of the word from the appearance frequency of the word constituting the input search word in the search word and the importance of the word calculated in advance;
The feature of the search word is a weighted sum of the distance or similarity between the feature vector of the word calculated in advance for each word and the feature vector of the searched document, using the calculated weight of the word. Distance calculation means for performing calculation for all searched documents as a distance or similarity between a vector and a feature vector of the searched document;
Search result determining means for determining, as a search result, a search target document corresponding to each distance selected in ascending order from the set of search target documents or the similarity selected in descending order among the calculated distances or similarities.
Document retrieval system according to claim <br/> further comprising a. As an indicator of the distance or similarity, the predetermined search word feature vector and the feature vector and the to-be-searched documents words magnitude of distance or similarity values of the feature vector of the to-be-searched document constitutes the search term 5. The document search apparatus according to claim 4 , wherein a distance or similarity index that is equivalent to a magnitude relationship between a distance to a feature vector or a weighted sum of similarity values is used. The index of the distance is any one of a Euclidean square distance between the feature vector of the word and the feature vector of the searched document, Cullback-Liber divergence, and cross-entropy using this divergence,
6. The document search apparatus according to claim 5 , wherein the similarity index is an inner product of the feature vectors of the two. A document search program for causing a computer to function as each means constituting the document search device according to any one of claims 4 to 6 .

Images