JP5184438B2 - Document signature generation apparatus, document signature generation method, and document signature generation program for detecting similar documents - Google Patents

Description

  The present invention relates to a document signature generation technique for detecting similar documents included in a large-scale document set that constitutes an index of a Web full-text search engine.

  As a technique for efficiently retrieving documents from documents included in a large-scale document set, a method using a document signature has been known for a long time. For example, Non-Patent Document 1 introduces a plurality of document signature creation methods. The document signature has a smaller capacity than the original document set, and the document signature can be used to search for a document faster than the original document set.

  In order to apply the document signature method to similar document retrieval, Non-Patent Document 2 introduces a document signature generation technique that takes into account the appearance frequency of words by extending the “Superimposed Coding” method introduced in Non-Patent Document 1. ing.

  Here, a method of generating an F-bit document signature using the technique introduced in Non-Patent Document 2 will be described. First, an F-bit hash value is calculated for each word included in the document. Next, each hash value is treated as a bit string, and each element is {-1, 1 at the position where the bit is 1 and -1 at the position where the bit is 0. Convert each bit string to a vector of F elements that is 1}. Thus, each word is expressed by a vector of F elements. Further, the sum of the elements at the same position of each vector is calculated, and a vector of F elements having 1 as an element if the sum is non-negative and 0 as an element is obtained. A document signature is obtained by converting the elements of this vector into a bit string.

Faloutos, C.I. and Christodoulakis, S .; “Description and performance analysis of signature file methods for office filling.” ACM Transactions on Office Information Systems. Vol. 5, no. 3, July 1987, pp. 237-257. Search on May 1, 2009 <Internet URL; http: // doi. acm. org / 10.1145 / 2764128057> Henzinger, M.M. "Finding near-duplicate web pages:. A large-scale evaluation of algorithms" In Proceedings of the 29th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (. Seattle, Washington, USA, August 06-11,2006) SIGIR '06. ACM, New York, NY, pp. Search on May 1, 284-291.209 <Internet URL; http: // doi. acm. org / 10.1145 / 1148170.1148222>

  The present invention solves two problems of the background art as a document signature generation method for retrieving similar documents. Hereinafter, the two problems will be described as a problem 1 and a problem 2, respectively.

  Problem 1 is that the background technology treats all words in the document equally, so that large and small differences are treated equally. As an example, consider the following two sets of sentences.

  (Example 1-a) I went to Osaka.

  (Example 1-b) I went to Osaka.

  (Example 2-a) I went to Tokyo.

  (Example 2-b) I went to Osaka.

  Example 1-a and Example 1-b are the only differences between “ni” and “to”. Similarly, Example 2-a and Example 2-b are different from each other by only one word “Tokyo” and “Osaka”. In the background art, since each word is handled in the same way, the difference between Example 1-a and Example 1-b or the difference between Example 2-a and Example 2-b is not different. However, Example 1-a and Example 1-b are similar documents, whereas Example 2-a and Example 2-b are not similar documents.

  Problem 2 is to generate the same document signature even for sentences in which the order of sentences is changed because the background art does not consider the order of words in the document. The following two sentences are given as an example.

  (Example 3-a) I went to Osaka. After that, I went to Tokyo.

  (Example 3-b) I went to Tokyo. After that, I went to Osaka.

  Since the background technology does not consider the word order, the same document signature is generated for Example 3-a and Example 3-b. In the background art, it is not possible to meet the request for distinguishing these documents by document signatures instead of similar documents.

For Task 1, consider the importance of each word when calculating the document signature. Specifically, in Non-Patent Technology 2, each word is converted into a bit string, and each element is set to {−1, 1} when converted into a vector, depending on the importance of the word {−w _t , Let w _t }. Here, w _t is a numerical value of the importance of each word, and is larger for important words and smaller for unimportant words.

For assignment 2, the degree of contribution of each word to the document signature is determined according to the appearance position. Specifically, in Non-Patent Technology 2, each word is converted into a bit string, and each element is set to {−1, 1} when converted into a vector, and {−w _p , _Let w _p }. However, w _p is a numerical value of the degree of contribution due to the appearance position (importance corresponding to the appearance position). In general, since important information tends to concentrate at the beginning of a document, w _p is made larger in the preceding word and smaller in the following word.

To solve Problem 1 and Problem 2 at the same time, each word is converted into a bit string in Non-Patent Technology 2, and each element is set to {-1, 1} when converted to a vector. And {−w _t w _p , w _t w _p } depending on the appearance position.

The document signature generation apparatus for detecting a similar document according to claim 1 of the present invention includes at least the importance w _{t of} each word included in the document or the importance w _p corresponding to the appearance position of each word. Importance calculating means for calculating importance w by calculating either one of them and {-w, w} for each word as an element according to importance w of each word obtained by the importance calculating means And a document signature calculating unit that generates a vector, obtains a document signature vector of an F element from the generated vector, and uses the document signature vector as document signature information.

(1) According to the present invention, Problem 1 can be solved, so that it is not affected by a trivial word difference when generating a document signature. In the background art, even when different document signatures are generated due to differences in words in a document, the present invention increases the number of cases where the same document signature can be generated for similar documents. By using the document signature of the present invention, it is possible to detect more similar documents than the background art.
(2) Further, according to the present invention, the problem 2 can be solved, and therefore, the difference in word order in the document can be taken into account when generating the document signature. In the background art, the same document signature is generated for a plurality of documents having different word orders. However, in the present invention, different document signatures are generated if the word orders are different. In the background art, there are cases where a document having only a different word order is erroneously detected as a similar document. However, such erroneous detection can be avoided by using the document signature of the present invention.

1 is a block diagram showing an example of an embodiment of a document signature generating apparatus according to the present invention. The flowchart which shows the document signature calculation method in one Embodiment of the document signature production | generation method of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments. FIG. 1 is a block diagram of an embodiment of a document signature generating apparatus according to the present invention. In the figure, a broken line arrow indicates a data flow, and a solid line arrow indicates a processing flow.

  In FIG. 1, reference numeral 100 denotes a document set database, which is a storage device that stores similar documents to be detected.

Reference numeral 200 denotes word statistical information calculation means for calculating the importance w _t of each word included in each document stored in the document set database 100.

Reference numeral 300 denotes a word importance database, which is a storage device that stores the importance w _t for each word calculated by the word statistical information calculation means 200.

400 obtains the importance w _p according to the appearance position of each word included in each document stored in the document set database 100, and the importance w _p stored in the word importance database 300. Document signature calculation means for calculating a document signature for each document included in the document set based on _t .

  Reference numeral 500 denotes a document signature database, which is a storage device that stores a document signature for each document.

  Each function to be described later of the word statistical information calculation unit 200 and the document signature calculation unit 400 is achieved by a computer, for example.

The importance calculation means of the present invention is achieved by the word statistical information calculation means 200 obtaining the importance w _t of each word and the document signature calculation means 400 obtaining the importance w _p according to the appearance position of each word. Is done.

  Next, the operation of the apparatus configured as described above will be described. First, the word statistical information calculation means 200 calculates the importance of a word. The word statistical information calculation means 200 reads a document from the document set database 100 and extracts words included in each document by a morphological analyzer.

Next, the IDF (reverse sentence frequency) of each word is calculated, and the IDF is recorded in the word importance database 300 as the importance w _t (the following formula (1)) of each word t.

Here, the IDF of idf word t of formula (1), N denotes the total number of documents in the set document, df _t is the number of documents that contain term t.

  Next, the document signature calculation means 400 calculates a document signature for each document included in the document set according to the flowchart shown in FIG. The document signature calculation unit 400 reads a document from the document set database 100 and extracts words included in each document by a morphological analyzer (step S1). When a morphological analyzer cannot be used, for example, n-gram may be extracted as a word.

  Next, an F-bit hash value is calculated for each extracted word using a hash function (step S2). As the hash function, a function that satisfies the conditions that a hash value close to similar words is not generated and that hash value collisions do not easily occur is used. Standardized hash functions such as MD5 (Message Digest Algorithm 5), SHA (Secure Hash Algorithm) -1, and SHA-256 satisfy these conditions. Subsequently, the word importance of each word is acquired from the word importance database 300.

Further, the importance w _p corresponding to the word appearance position _p is calculated. In the embodiment, assuming that the closer to the beginning of the document, the more important it is, and assuming that the word appears pth in a document whose document length is N words,
w _p = (N−p) / N (2)
_Let w _{p be} the importance of the appearance position.

For each word, the hash value of each word is treated as a bit string, the element at the position where the bit is 0 is −w _t w _p , the element at the position where the bit is 1 is w _t w _p , and so on. An F element vector (first vector) is calculated (step S3). Here, w _p = 1 when not using the importance w _p corresponding to the appearance position of the word, and w _t = 1 when not using the word importance w _t .

  Further, the sum of the elements at the same position of each F element vector is calculated to be a new vector (second vector) (step S4). If the sum is non-negative, 1 is set, and if the sum is negative, 0 is set. A document signature vector of the F element is obtained (step S5). Finally, the document signature vector converted into an F-bit bit string is stored in the document signature database 500 as a document signature (step S6).

  It should be noted that the hash value calculated by the document signature calculation unit 400 may be stored and used in a memory (not shown), for example.

  In addition, the present invention can be realized by configuring some or all of the functions of each means in the document signature generating apparatus of the present embodiment by a computer program and executing the program using the computer. It is needless to say that the procedure in the document signature generation method of the above can be configured by a computer program and the program can be executed by the computer, and a program for realizing the function by the computer can be read by the computer, For example, FD (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact Disk) -ROM, DVD (Digital Versati) e Disk) -ROM, CD-R, CD-RW, HDD, and recorded in a removable disk, or stored, it is possible or distribute. It is also possible to provide the above program through a network such as the Internet or electronic mail.

DESCRIPTION OF SYMBOLS 100 ... Document set database 200 ... Word statistical information calculation means 300 ... Word importance database 400 ... Document signature calculation means 500 ... Document signature database

Claims (5)

An apparatus for generating document signature information for detecting a similar document,
Importance calculating means for calculating importance w by calculating at least one of importance w _{t of} each word included in the document or importance w _p corresponding to the appearance position of each word;
A vector having {−w, w} as an element is generated for each word according to the importance w of each word obtained by the importance calculating means, and a document signature vector of F element is generated from the generated vector. A document signature generation apparatus for detecting a similar document characterized by comprising: a document signature calculation means that obtains the document signature vector as document signature information. The document signature calculation means
Calculate an F-bit hash value for each word contained in the document;
Based on the importance w _t of each word obtained by the importance calculation means and the importance w _p according to the appearance position of each word, the calculated hash value is treated as a bit string, and the bit becomes 0 Find a first vector with the element at the position -w _t w _p and the element at the position where the bit is 1 w _t w _p ,
Calculating a sum of elements at the same position of the first vector to obtain a second vector;
2. The document signature generation apparatus for detecting a similar document according to claim 1, wherein a document signature vector of an F element in which a non-negative element of the second vector is 1 and a negative element is 0 is obtained. A method of generating document signature information for detecting similar documents,
Importance calculation step in which the importance calculation means calculates importance w by calculating at least one of importance w _{t of} each word included in the document or importance w _p corresponding to the appearance position of each word When,
The document signature calculating means generates a vector having {−w, w} as an element for each word according to the importance w of each word obtained by the importance calculating means, and F is generated from the generated vector. A document signature generation method for detecting a similar document, comprising: obtaining a document signature vector of an element; and a document signature calculation step using the document signature vector as document signature information. The document signature calculation step includes:
Calculating an F-bit hash value for each word contained in the document;
Based on the importance w _t of each word obtained by the importance calculation means and the importance w _p according to the appearance position of each word, the calculated hash value is treated as a bit string, and the bit becomes 0 Obtaining a first vector having an element at a certain position as −w _t w _p and an element at a position where the bit is 1 as w _t w _p ;
Calculating a sum of elements at the same position of the first vector to obtain a second vector;
A method for detecting a similar document according to claim 3, further comprising: obtaining a document signature vector of an F element in which the non-negative element of the second vector is 1 and the negative element is 0. Document signature generation method. A document signature generation program for detecting a similar document that causes a computer to function as each unit according to claim 1.

Images