JP2009193219A - Indexing apparatus, method thereof, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable document retrieval of high ranking accuracy at processing speed applicable to a search engine even for a document set composed of documents including miscellaneous contents. <P>SOLUTION: By a keyword identification section 4, a position is identified at which a word or a word sequence in respective analyzed retrieval object documents matches a keyword. By a neighborhood word acquisition section 5, a neighborhood word for the keyword is extracted from the respective analyzed retrieval object documents and a co-occurrence frequency of the keyword and the neighborhood word is counted. By a relevance ratio calculation section 7, a co-occurrence frequency of the keyword and the neighborhood word in the entire set of the analyzed retrieval object documents and an appearance frequency of the keyword, and a relevance between the keyword and the neighborhood word from the appearance frequency of the neighborhood word and an entire word count are calculated. By a document score calculation section 9, a relevance between the keyword and the neighborhood word is used to calculate a score of the keyword in the respective analyzed retrieval object documents, and the score is output as an index. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for searching a document that matches a query (a processing request represented by a character string) from a set of documents and creating an index for ranking in the order of matching.

  Conventionally, various methods have been proposed as a method of searching documents that match a query from a set of documents and ranking (ranking) them in the order of matching.

For example, a vector space model is representative as a method for expressing the contents of a document, and similarity is obtained according to a cosine measure between a document vector and a query vector, and ranking is performed in descending order of similarity. Furthermore, as a method for improving the search accuracy, a word that is included in a document that the user has determined that matches or does not match after searching once is weighted in consideration of the number of documents that match and the number of documents that do not match, and is newly added to the query. There has been proposed a method called relevance feedback for performing a search again by adding (see Non-Patent Documents 1 and 2). In addition, by calculating the mutual information amount between the word and the query contained in the document from the top document of the first search result without giving feedback from the user, the query and the word related to the query are automatically detected. A method of adding to the above has also been proposed (see Non-Patent Document 3).
Takenobu Tokunaga, “Language and Computation 5 Information Retrieval and Language Processing”, The University of Tokyo Press, 1999, p. 154-159 Kita Kenji, Tsuda Kazuhiko, Sasabori Masatomi, “Information Retrieval Algorithm”, Kyoritsu Publishing Co., Ltd., 2002, p. 60-65 Satoshi Kanaya, Koji Umemura, “Analysis of Empirical Weights Using Correlation Coefficients and Search Query Expansion”, IPSJ SIG Technical Report, 2003-FI-13, 2003, P.A. 17-24

  However, there are some problems in applying the above-described conventional technique to a search engine.

  In other words, the relevance feedback method uses the top document of the first search result, but the number of keywords input by the user in the search engine is often very small, and the search request of the user is sufficiently reflected in the first search. There is a problem that the higher-order document of the retrieved result cannot be obtained.

  Further, the documents targeted by the search engine include documents that are freely described by the user. Such a document may contain miscellaneous contents in one document, and it is difficult to extract words related to the query in one document, and the conventional conformity feedback and automatic query expansion methods are applied as they are. However, there is a problem that it does not contribute to accuracy improvement.

  Furthermore, the search engine requires real-time processing, but with the conventional relevance feedback and automatic query expansion methods, calculations for detecting related words based on the initial search results are executed and updated each time. There was a problem in terms of processing speed because the query was searched again.

  It is an object of the present invention to provide a high-accuracy processing speed applicable to a search engine even for a set of documents that can be composed of documents including various contents when a query input by a user is given by a small number of keywords. It is to enable ranking.

In order to achieve the above object, in the present invention,
・ Calculate the degree of relevance between keywords and neighboring words in order to focus on neighboring words that are strongly related to keywords so that a collection of documents containing miscellaneous contents can be targeted.
-In addition, the document score is calculated by using the degree of relevance with the keyword for the neighboring word of the keyword on the assumption that the document having the word closely correlated with the keyword is well matched with the keyword. .

Specifically, the index creation device of the present invention is:
An apparatus for creating an index for searching documents that match a query from a set of search target documents and ranking them in the order of suitability, an analyzed document database that stores a set of search target documents that have been subjected to morphological analysis, and a query A keyword identifying unit for identifying a position where a word or a word string in each analyzed search target document matches a keyword given as: a position of a word or a word string that matches the keyword in each analyzed search target document The word in the whole set of analyzed documents to be searched is extracted as content words existing in the vicinity of the word and the co-occurrence frequency of the word or word string that matches the keyword and the co-occurrence frequency of the neighborhood word is counted. Count the appearance frequency of words or word strings corresponding to, the appearance frequency of neighboring words, and the total number of words. Corresponding to the keyword in the entire set of analyzed search target documents, together with the co-occurrence frequency of the word or word string that matches the keyword in the entire set of analyzed search target documents and the neighboring word. Relevance calculator that calculates the degree of association between the keyword and the neighborhood word from the appearance frequency of the word or word string, the appearance frequency of the neighborhood word, and the total number of words, and the degree of association between the keyword and the neighborhood word It has at least a document score calculation unit that calculates a score for the keyword in the analyzed search target document and outputs it as an index.

Furthermore, in the present invention,
In order to avoid multiple searches and reduce the amount of calculation during the search, the document score of the document included in the document set is calculated for each keyword in advance and stored in the index.

That is, the index creation device of the present invention
In addition to the above, a keyword database for storing a set of keywords is provided, and all the processes in the keyword identification unit, the neighborhood word acquisition unit, the word frequency measurement unit, the relevance calculation unit, and the document score calculation unit are included in the keyword database. It repeats about this keyword, The index which makes the score the element corresponding to the combination of each keyword and each analyzed search object document is generated.

  As a result, in the present invention, it is possible to reflect information on a word strongly related to the keyword in the search ranking. In particular, (1) the use of peripheral word information for a keyword is effective for a document set including miscellaneous information. (2) It is not necessary to use feedback from the user. (3) By using an index storing the document score calculated in advance, the retrieval document ranking can be output at high speed during retrieval.

  According to the present invention, even when a query is given only with a small number of keywords and a document set including various contents is targeted, a high score is obtained for a document including many words that are highly correlated with keywords existing around the keyword. Therefore, as a result, it is possible to display a document having many words related to the keyword in the vicinity of the search ranking. In addition, since keywords and documents that can be assumed as queries are indexed in advance, there is no need to incorporate feedback from users or perform calculations using search results. Compared to this, it is possible to output the results quickly when searching. As described above, the present invention has an effect of realizing a high-speed and highly accurate document search ranking.

  Next, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows an example of an embodiment of an index creation apparatus according to the present invention (however, here also includes a part related to document search). The index creation apparatus of the present embodiment includes an analyzed document database 1 and , Keyword database 2, keyword input unit 3, keyword identification unit 4, neighborhood word acquisition unit 5, word frequency measurement unit 6, relevance level calculation unit 7, relevance level table 8, and document score calculation unit 9. And a keyword document index 10 and a document search unit 11.

  The analyzed document database 1 assigns each word in the search target document with word information such as word notation, reading, and part of speech, and position information indicating the position of the word in the document that has appeared. A set of analyzed search target documents is stored. The processed search target document is subjected to a well-known morphological analysis process for each search target document (search target document described in natural language) in the search target document database in which a set of search target documents is stored in advance. Obtained by.

  FIG. 2 shows an example of the analyzed search target document when the search target document is “Yokosuka Department Store Takaaki Hasegawa may be a really good person” (however, location information is omitted).

  The keyword database 2 stores a set of keywords (given as a query) used for document search. The keyword here is one word, compound word or phrase. A compound word is a series of nouns, and a phrase includes not only content words but also function words. As a keyword, a keyword having a high appearance frequency in a set of search target documents may be selected, or a unique expression extracted by an existing specific expression extraction technique may be selected. Alternatively, as an information source other than the document set, a query log obtained by aggregating search keywords input by the user may be used.

  The keyword input unit 3 acquires a keyword from the keyword database 2 and inputs it to the keyword identification unit 4.

  The keyword identification unit 4 matches a word (morpheme) or a word string (morpheme string) that is a sequence of a plurality of words in each analyzed search target document stored in the analyzed document database 1 with the input keyword. If they match, information on the start position and end position is acquired and output to the neighborhood word acquisition unit 5.

  For example, if the keyword is “Takaaki Hasegawa” and the document is the example shown in FIG. 3, the word notation and the keyword are collated from the head position by front matching, and further consecutive words are connected. By shifting the position of the word until the word string notation completely matches the keyword, the position from “Hasegawa” at position 4 to “Ryoaki” at position 5 matches the keyword, and the start of the word string that matches the keyword Position 4 and end position 5 are obtained. If there are multiple matching positions in the document, all are detected. It should be noted that the keyword may be detected by using a method such as using collation with a character string, and it is not always necessary to use a word or a word string.

  The neighborhood word acquisition unit 5 matches the keyword from the analyzed search target documents stored in the analyzed document database 1 based on the information on the start position and end position of the word or word string that matches the keyword. A content word existing in the vicinity of the position of the word or word string is extracted as a neighborhood word, stored in the relevance table 8 as a pair with the word or word string matching the keyword, and the word or word matching the keyword The co-occurrence frequency of the column and the neighboring word is counted, that is, stored / updated in the association degree table 8. At this time, the co-occurrence frequency is counted only for an analyzed search target document that detects a word or a word string that matches the keyword and a neighboring word, and for a whole set of analyzed search target documents. Shall.

  Here, the content word is defined in advance by, for example, part-of-speech information given to the word. As part-of-speech information of a content word, for example, as shown in FIG. 3, “noun” or “noun: unique” is designated. Further, the content words may be limited to words having large values using a scale such as IDF (inverse document frequency) or residual IDF, or may be used in combination with part-of-speech information. For example, the residual IDF has a large value with respect to the number of documents in which a word appears, and the total appearance frequency of the entire document set has a large value and is regarded as an important keyword. If it is such a scale, it will not be limited to these. Furthermore, the content word may be limited to a specific expression such as a person name, a place name, or an organization name, or may be used in combination with a word. A specific expression can be extracted using an existing specific expression extraction technique.

  The neighborhood is, for example, a set of words in the range of the number of words before and after a keyword specified in advance. The example of FIG. 3 shows a case where the three words before and after the keyword are designated as neighbors. In this example, three words from the start position 4 of the keyword “Takaaki Hasegawa”, that is, the words from position 1 to position 3 , The three words from the end position 5 of the keyword “Takaaki Hasegawa”, that is, the words from the position 6 to the position 8 are in the vicinity, but by specifying the part-of-speech information of the content word described above, “Yokosuka”, “ "Department store" and "store manager" are obtained.

  The neighborhood is not limited to the number of words before and after the keyword, but may be a word having a dependency relationship with the keyword obtained as a result of dependency analysis, or within the same phrase as the keyword. It may be a word included.

  When a specific expression is targeted, the keyword and the specific expression are not necessarily composed of a single word, so the co-occurrence of the keyword and the specific expression is specified in advance by the difference between the start positions of the two. It may be within the window size range. As shown in the example of FIG. 4, the case where the specific expression is not completely included in the range of the predetermined window size or the case where the keyword is included in the specific expression is regarded as having a co-occurrence relationship. May be.

  FIG. 5 shows the flow of processing in the keyword identification unit and the neighborhood word acquisition unit.

  That is, first, a pointer is set at the head of the analyzed search target document (s1), and a word and its position information are acquired (s2). Next, it is compared whether or not the acquired word matches the input keyword (s3), and if it matches, the position of the acquired word is acquired as the position of the word that matches the keyword (s4). If not, the pointer is advanced by one (s5). The above processing is repeated until the end of the document (s6).

  Next, one word is extracted from the analyzed search target document together with its position information (s7), and the difference between the extracted word position and the word position where the stored keyword matches is within a predetermined value. (S8), if it is within the specified value, the extracted word is stored in the relevance table 8 as a word near the keyboard, and the co-occurrence frequency between the word matching the keyword and the extracted word is determined as the relevance Store / update (count) in the table 8 (s9), and if it is larger than the specified value, do nothing. The above processing is repeated for all the words in the analyzed search target document (s10).

  FIG. 5 shows the processing for one analyzed search target document, and the actual processing is similarly performed for all the analyzed search target documents stored in the analyzed document database 1.

  The word frequency measurement unit 6 performs a search with respect to all the analyzed search target documents stored in the analyzed document database 1 using the above-described keyword and the neighborhood word stored in the relevance degree table 8, and the analyzed search target Measures the frequency of occurrence of words or word strings corresponding to keywords in the entire set of documents and the frequency of appearance of neighboring words in the entire set of analyzed search target documents, as well as the number of words in each analyzed search target document and the analyzed search The number of all words in the entire set of target documents is measured and stored in the relevance table 8.

  The relevance calculation unit 7 stores the analyzed search target document together with the co-occurrence frequency of the word or word string that matches the keyword and the neighborhood word in the entire set of analyzed search target documents stored in the relevance level table 8. From the appearance frequency of the word or word string corresponding to the keyword, the appearance frequency of neighboring words, and the total number of words in the whole set, the degree of association between the keyword and the neighboring word is calculated based on a pre-specified calculation formula. The calculation formula only needs to obtain the strength of the correlation between the keyword and the neighboring word. For example, the calculation formula may be a formula for obtaining the mutual information amount. An example of the calculation formula is shown below.

However,
Q: Keyword W: Keyword near the keyword C (Q): Appearance frequency of the keyword Q in the document set C (W): Appearance frequency of the neighborhood word W in the document set C (Q, W): Keyword Q and the neighborhood in the document set Co-occurrence frequency of word W N: The total number of words in the document set.

  The calculation of the degree of relevance is limited to words or specific expressions that satisfy a predetermined number of words or specific expressions that co-occur with keywords, according to the number of co-occurrence frequencies, or a measure such as residual IDF. It may be limited to a value larger than a certain threshold value, or may be limited in combination.

  The relevance table 8 shows the appearance frequency of the word or word string corresponding to the keyword and the entire set of analyzed search target documents, the word corresponding to the keyword or a word near the word string and the entire set of analyzed search target documents. The frequency of occurrence of the word, the co-occurrence frequency of the word or word string that matches the keyword and the neighboring word (however, only the whole set is shown here), the degree of association between the calculated keyword and the neighboring word Store.

  FIG. 6 shows an example of the degree-of-association table 8, and here shows an example in which the degree of association between a keyword and a neighboring word is calculated by the equation (1). 6 is a numerical value when the total number of words N in the entire set is 1,000,000.

  The document score calculation unit 9 calculates the document score for the keyword in each analyzed search target document using the degree of association between the keyword and the neighboring word stored in the degree-of-association table 8, and uses this as an index to obtain the keyword document index 10 is output. The calculation formula of the document score may be an expression that gives a high score to a document having many neighboring words strongly related to the keyword. The rules for the neighborhood and the content word may be the same as or different from the calculation of the relevance, but the same case will be described below.

  For example, in the case of using the mutual information amount, a value of the sum of mutual information amounts with neighboring words is obtained for each occurrence position of the word or word string corresponding to the keyword in the analyzed search target document, and further corresponding to the keyword. The sum of the values obtained for each occurrence position of the word or word string is used as the document score of the search target document for the keyword. Further, the document score may be a value normalized with respect to this value using the total number of words in the document, the number of different content words, and the like. The content word used for the calculation of the document score may be restricted so that it is selected from the top in descending order of the degree of association with the keyword.

  An example of the calculation formula of the document score of the document D with respect to the keyword Q is shown below.

However,
D: document Q: keyword W _i: each word Co to co-occur with keyword _{Q (D, Q, W i} ): co-occurrence frequency of the word Wi to co-occur with keyword Q in the document _D N D: the number of words in the document D It is.

For example, the document score for the document “Takaaki Hasegawa at Yokosuka Department Store may be a really good person” is obtained for the keyword “Takaaki Hasegawa”. If a word whose part of speech existing in the range of the three morphemes before and after the keyword “Takaaki Hasegawa” is “noun” or “noun: proper” is a neighboring word, three neighboring words are obtained as shown in FIG. Each co-occurrence frequency in the document is 1. The degree of association PMI between the keyword “Takaaki Hasegawa” and the neighboring words “Yokosuka”, “Department Store”, and “Store Manager” is 3.51, 3.22, 6.21, respectively, as shown in FIG. Since Equation 13 is satisfied, the document score of the document for the keyword is expressed by the following equation (2): Score = (1 * 3, 51 + 1 * 3.22 + 1 * 6.21) / 13
= 0.995
Is calculated.

In addition, the score of the document includes a score (Score1) based on the appearance frequency of the word or word string corresponding to the keyword normalized by the number of words in the document, and the word or word string corresponding to the keyword described above. From the score based on the mutual information (Score2) and the score based on the mutual information of the specific expression co-occurring with the word or word string corresponding to the keyword (Score3), each score is set to an appropriate constant α, β, γ After normalizing with, linear sum as shown in the following equation: Score = α * Score1 + β * Score2 + γ * Score3
And a score may be given to the document containing the keyword according to this value.

  All keywords in the keyword database 2 are sequentially acquired by the keyword input unit 3, and in the keyword identification unit 4, the neighborhood word acquisition unit 5, the word frequency measurement unit 6, the relevance calculation unit 7, and the document score calculation unit 9 described above. A series of processing is performed, and a document score is calculated for a combination of all keywords in the keyword database 2 and all analyzed search target documents in the analyzed document database, and each of the calculated keywords and each analyzed search is calculated. An index having a document score corresponding to a combination with the target document as an element is stored in the keyword document index 10.

  FIG. 7 shows an example of the keyword document index. If the number of analyzed search target documents stored in the analyzed document database 1 is N and the number of keywords stored in the keyword database 2 is M, keywords 1 to M are compared to documents 1 to N. As a matrix of document scores.

  The document search unit 11 ranks the search target documents according to the document score stored in the keyword document index 10 and outputs the ranking. For example, for a keyword with a query at the time of search, the search target documents may be ranked so that the document scores stored for the keyword are in descending order. When the query is a plurality of keywords, it may be ranked in descending order of the sum of the document scores for each keyword, or instead of using the reciprocal of the word document frequency and the word document frequency in the vector space model. The similarity between the query and the document may be calculated using the reciprocal of the keyword document score and the keyword document frequency, and the documents may be ranked in the descending order of similarity. When a keyword included in the query does not exist in the keyword document index, an existing vector space model may be applied to the keyword.

  The present invention can also be realized by installing a program for realizing the functions shown in the configuration diagram of FIG. 1 via a recording medium or a communication line in a known computer.

The block diagram which shows an example of embodiment of the index production apparatus of this invention Explanatory diagram showing an example of an analyzed search target document Explanatory drawing which shows the state of keyword identification and neighborhood word acquisition processing for the analyzed search target document Explanatory diagram showing an example of co-occurrence relationship between keywords and proper expressions Flow chart of processing in keyword identification unit and neighborhood word acquisition unit Explanatory drawing which shows an example of an association degree table Explanatory drawing showing an example of keyword document index

Explanation of symbols

  1: analyzed document database, 2: keyword database, 3: keyword input unit, 4: keyword identification unit, 5: neighborhood word acquisition unit, 6: word frequency measurement unit, 7: relevance calculation unit, 8: relevance table 9: Document score calculation unit, 10: Keyword document index, 11: Document search unit.

Claims (6)

A device that searches a document that matches a query from a set of search target documents and creates an index for ranking in order of matching,
An analyzed document database for storing a set of search target documents that have been subjected to morphological analysis;
A keyword identification unit for identifying a position where a word or a word string in each analyzed search target document matches a keyword given as a query;
A content word existing in the vicinity of the position of the word or word string matching the keyword is extracted as a neighborhood word from each analyzed search target document, and the co-occurrence frequency of the word or word string matching the keyword and the neighborhood word is extracted. A neighborhood word acquisition unit that counts
A word frequency measuring unit that counts the appearance frequency of words or word strings corresponding to keywords in the entire set of analyzed search target documents, the appearance frequency of neighboring words, and the total number of words;
The frequency of occurrence of the word or word string corresponding to the keyword in the entire set of analyzed search target documents, together with the co-occurrence frequency of the word or word string that matches the keyword in the entire set of analyzed search target documents and neighboring words, A degree-of-association calculating unit that calculates the degree of association between the keyword and the neighboring word from the appearance frequency of the neighboring word and the total number of words;
An index creation apparatus comprising: at least a document score calculation unit that calculates a score for the keyword in each analyzed search target document using a degree of association between the keyword and a neighboring word, and outputs the score as an index. In addition to the above
It has a keyword database that stores a set of keywords,
The keyword identification unit, neighborhood word acquisition unit, word frequency measurement unit, relevance calculation unit, and document score calculation unit are repeated for all keywords included in the keyword database, and each keyword and each analyzed search target document The index creation device according to claim 1, wherein an index having a score corresponding to a combination of and as an element is generated. A method for searching documents that match a query from a set of search target documents and creating an index for ranking in the order of matching,
Using an analyzed document database that stores a set of search target documents that have undergone morphological analysis,
A step in which a keyword identification unit identifies a position where a word or a word string in each analyzed search target document matches a keyword given as a query;
The neighborhood word acquisition unit extracts, as a neighborhood word, a content word that exists in the vicinity of the position of the word or word string that matches the keyword from each analyzed search target document, and the word or word string that matches the keyword and the neighborhood Counting the frequency of co-occurrence with a word;
A word frequency measurement unit that counts the appearance frequency of words or word strings corresponding to keywords in the entire set of analyzed search target documents, the appearance frequency of neighboring words, and the total number of words;
The relevance calculator calculates a word corresponding to the keyword in the entire set of analyzed search target documents, together with the co-occurrence frequency of the word or word string that matches the keyword in the entire set of analyzed search target documents and neighboring words, or Calculating the degree of association between the keyword and the neighborhood word from the appearance frequency of the word string, the appearance frequency of the neighborhood word, and the total number of words;
The document score calculation unit includes at least a step of calculating a score for the keyword in each analyzed search target document using a degree of association between the keyword and a neighboring word, and outputting the score as an index. How to make. In addition to the above
Using a keyword database that stores a set of keywords,
The keyword position identification step, the neighborhood word acquisition step, the word frequency measurement step, the relevance calculation step, and the document score calculation step are repeated for all keywords included in the keyword database, and each keyword and each analyzed search target document are The index creation method according to claim 3, wherein an index having a score corresponding to the combination as an element is generated.   The program for functioning a computer as each means of the index production apparatus of Claim 1 or 2.   A computer-readable recording medium on which the program according to claim 5 is recorded.

Images