KR100834292B1 - Document processing method and system - Google Patents

Abstract

The present invention relates to a method and a system for processing a document, the method comprising: distinguishing an original document from at least one copy document from a plurality of documents including at least one common chunk, and based on the common chunk; Calculating the scores of the copied document in different ways. According to the present invention, the degree of copying of a document can be grasped by calculating and assigning a document score to the document.

Document processing, chunks, hash values, original documents, copy documents, index volumes, document scores, duplicate documents

Description

Document processing method and system {DOCUMENT PROCESSING METHOD AND SYSTEM}

1 is a block diagram illustrating a document processing system according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating an example of the hash generation server shown in FIG. 1.

3 is a schematic diagram illustrating an example of a document collection file, a hash collection file, and a score collection file used in a document processing system according to an embodiment of the present invention.

4 is a schematic diagram illustrating an example of extracting chunks according to an embodiment of the present invention.

FIG. 5 is a block diagram illustrating an example of the score generation server shown in FIG. 1.

6-9 are schematic diagrams showing examples of calculating document scores according to an embodiment of the present invention.

<Description of Drawing>

100: document processing system, 110: document database,

120: hash generation server, 122: document parsing module,

124: chunk extraction module, 126: hash calculation module,

128: hash collection generation module, 130: hash indexing server,

140: hash index volume, 150: score generation server,

152: hash parsing module, 154: search module,

156: sorting module, 158: score calculation module,

159: score collection generation module, 160: document indexing server,

170: document index volume, 180: score collection database,

190: database query server, 200: communication network,

300: user terminal, 400: search system

The present invention relates to a document processing method and system, and more particularly, to a document processing method and system that can derive a higher quality search results in information retrieval using the Internet.

With the rapid spread of high speed internet in recent years, it has become an indispensable necessity in modern life. Most users who use the Internet access the Internet through a browser and search for information through an Internet portal site. When the user enters a query in the search box of the Internet portal, the Internet portal extracts various information corresponding to the query from the database and provides the search result to the user. The extracted information is divided into categories such as dictionaries, knowledge information (eg, Naver's knowledge iN), blogs, cafes, specialty materials, sites, books, news, web pages, videos, and the like, and is displayed and delivered to the user.

Increasingly, users do not create original documents on specific subjects, but rather copy documents written by other users to answer knowledge information or post their blogs or cafes. This is because a document used on the Internet has a feature that can be easily copied. The user creates a document mainly by copying the original document itself or selectively copying only the necessary parts from newspaper articles or specialized materials or blogs or cafes of others. In some cases, additional text may be added to the copied document to supplement the contents, or explanation may be modified. Thus, the documents thus created are often the same or substantially the same as the original document.

Among the documents displayed to the user as a search result, the one most relevant to the search query is displayed first, and the display order is determined according to the relevance. However, if the copied documents are not filtered but included in the search results and displayed to the user, the quality of the search service is degraded. A user may spend a lot of time opening a duplicate document, may not open a document that is pushed in the display rank even though it is highly relevant, and may have to re-search the search result.

Therefore, the technical problem to be achieved by the present invention is to provide a document processing method and system that can derive a higher quality search results from information search.

The document processing method according to an embodiment of the present invention for achieving the technical problem, the step of distinguishing the original document and at least one copy document from a plurality of documents including at least one common chunk, and based on the common chunk Calculating the scores of the original document and the copied document in different ways.

The score of the original document may be the sum of the number of chunks of the original document and the number of the common chunks.

The score (OS) of the original document is calculated by the following equation,

OD is the number of chunks of the original document, CD _i is the number of common chunks of the i-th copy document, and n may be the number of the copy documents.

The score of the copy document may be a ratio of the number of chunks of the copy document and the number of chunks of the copy document.

Generating a score collection file recording document identification marks, document scores, and copy lists for each document, wherein document identification codes of the copied documents are described in the copy list of the original document, and the copy list of the copied documents. Document identification code of the original document can be described.

The document discriminating step may include extracting chunks from a plurality of documents, extracting a duplicate document having the chunk in common by querying a storage medium, and comparing a document generation time of the duplicate document with the original document and the copy document. It may include the step of distinguishing.

The duplicate document may have more than a certain number of the common chunks.

A computer readable medium according to another aspect of the present invention records a program for causing a computer to execute any of the above methods.

According to another aspect of the present invention, a document processing system includes a document discrimination module that distinguishes an original document from at least one copy document from a plurality of documents including at least one common chunk, and the original document and the based on the common chunk; And a score calculation module for calculating the score of the copy document in different ways.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention.

First, a document processing system and method according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 5.

1 is a block diagram illustrating a document processing system according to an embodiment of the present invention.

Referring to FIG. 1, a document processing system 100 according to an embodiment of the present invention may include a document database 110, a hash generation server 120, a hash indexing server 130, a hash index volume 140, and a score generation. A server 150, a document indexing server 160, a document index volume 170, a score collection database 180, and a database lookup server 190 are included and connected to the search system 400. The search system 400 is connected to the plurality of user terminals 300 through the communication network 200.

The document database 110 stores a plurality of various documents. Documents include documents posted in cafes, blogs, etc., news, technical data, and user-written knowledge information documents. The document database 110 may divide a category such as a cafe, a blog, news, specialized material, knowledge information, and the like and store the document according to the corresponding category. Each document has a global document ID that uniquely identifies it, includes document creation time, document body and formatting, and a hypertext markup language. , HTML) or a standard generalized markup language (SGML).

FIG. 2 is a block diagram illustrating an example of the hash generation server shown in FIG. 1.

Referring to FIG. 2, the hash generation server 120 includes a document parsing module 122, a chunk extraction module 124, a hash calculation module 126, and a hash collection generation module 128, and a document database 110. Create a hash collection file by referencing the document collection file from.

The document collection file is a file of at least one document stored in the document database 110. Each document included in the document collection file includes a document identification code (gdid), a document generation time (time), and a body, and has a certain format as shown in FIG. 3 is a schematic diagram illustrating an example of a document collection file, a hash collection file, and a score collection file used in a document processing system according to an embodiment of the present invention.

Here, &quot; ID1 &quot; appearing between &gt;# gdid # and a line break control character (#n) means a document identification code. The size of the document identification code is preferably the same for each document for unification of the interface. For example, the size of the document identification code may be set to 20 bytes, and the document identification code may include a separator at the head of the document identification code so that categories of each document can be distinguished. The delimiter may be composed of at least one byte. For example, news may be set to 0110, knowledge information to 0120, cafe to 0130, blog to 0140, and the like.

The document generation time includes time information in the unit of date (yyyymmdd) and hour and minute seconds (hhmmss), and is located between &gt;# time # and the line break control character (#n) on the line following the document identifier. The body is located between &gt;# body # and the line break control character (#n). For example, as shown in FIG. 3 (a), the body includes a plurality of sentences CA, CB, CC, CD, CE, CF. It includes. As such, the text may include at least one sentence, but may not necessarily have a sentence form.

The document format shown in (a) of FIG. 3 is one example, and the document collection file may have a different format.

The document processing system 100 according to the embodiment of the present invention extracts a document from the document database 110, generates a document collection file, and transmits the document collection generating module (not shown) to the hash generation server 120. It may include.

The document parsing module 122 receives the document collection file from the document database 110 or the document collection generating module and parses each document included therein, so that the hash generation server 120 based on the body of each document. Allow chunk extraction.

The chunk extraction module 124 receives the parsed text from the document parsing module 122 and extracts chunks and valid chunks. The chunk refers to each chunk when the text is divided into at least one chunk, and is extracted based on a predetermined pivot character that is a character that distinguishes the chunk from the chunk. Pivot characters include at least one of a period, a newline control character (＼n), a space character, a semicolon (;), and a colon (:) in sentences that include hot points (.), Question marks (?), And exclamation points (!). can do. Pivot characters may be included in any of the delimited chunks, but may not be included. Pivot characters can be arbitrarily set as needed, and in addition to those listed above, various characters can be used as pivot characters as necessary.

If the pivot character is a period, one chunk is created based on one sentence; if it is a line break control character, one chunk is created based on a new line; if it is a space character, one chunk is based on one word. Chunks of are generated.

When using periods or line breaks as pivot characters, it is unlikely that a non-duplicate document will be considered as a duplicate document, it is generally faster to judge whether to copy, and simply copies and pastes without modification. It is effective for judging whether or not a document generated by the method is copied. In the case of using a space character as a pivot character, it is effective to determine whether to copy a document that has been modified at a large amount at random after copying content from another document.

In order to be chunked, the document must be at least as large as the body. If it is smaller than this, the document is excluded from chunking. The minimum body size can be set, for example, 128 bytes.

An effective chunk is a chunk that satisfies a predetermined condition. For example, if the size of the chunk is larger than the minimum chunk size, the chunk may be included in the valid chunk. The minimum chunk size may be set to 40 bytes, for example, but is not limited thereto, and may be set to another size as needed. Chunks less than the minimum chunk size can be ignored. Alternatively, a chunk that is smaller than the minimum chunk size may be merged with the chunk that appears after the chunk, and may be included in the valid chunk when the size of the merged chunk is larger than or equal to the minimum chunk size.

The number of valid chunks that can be extracted from a document can be limited to less than the maximum chunk count, for example 50. Chunks exceeding the maximum number of chunks may be ignored.

For example, certain characters and certain strings, such as '==', '^^', numbers, and conventional phrases, can be excluded from the chunk. You can also exclude strings enclosed in parentheses ([], {}, ()) if necessary.

For documents without pivot characters, you can extract the chunk in the following way: A hash value is calculated by applying a first hash function to each word or phrase included in the document. A hash value may be calculated by dividing the document into strings of a constant size and applying a first hash function to each string. Hash functions, also called summary functions or message digest functions, generate a fixed-length pseudorandom number from a given string.

The calculated hash value M is divided by the predetermined natural number K and the remainder N is obtained (M% K = N, 0≤N≤K-1). At this time, the chunk may be extracted based on a word / phrase / string having a specific value (for example, 0) among the rest (N). In other words, these words / phrases / strings can be used as criteria for separating chunks as if they were pivot characters. Except for extracting the chunks in this manner, the same limitations can be applied to a document without a pivot character as well as a document with a pivot character. You can also extract chunks in this way with or without pivot characters.

An example of extracting chunks will be described with reference to FIG. 4.

4 is a schematic diagram illustrating an example of extracting chunks according to an embodiment of the present invention.

Shown in FIG. 4A is a chunk extraction target document, and shown in FIG. 4B is an extracted chunk. A period was used as the pivot character, and the minimum body size was set to 128 bytes and the predetermined size was set to 40 bytes. In (b), the chunks from the first to the third row are valid chunks. The chunks from the fourth row to the last row are not valid chunks because they are less than 40 bytes (shown in dashed vertical lines). On the other hand, the parts of the first line of the document that are enclosed in parentheses ([]) and parentheses ([]) are excluded from the chunk, but can be included in the chunk as needed.

The hash calculation module 126 calculates a hash value by applying a second hash function to each valid chunk from the chunk extraction module 124. The second hash function may be the same as or different from the first hash function described above. The hash calculation module 126 is a second hash function, for example, a well-known message digest algorithm (MD) series, a secure hash algorithm (SHA) series, a race integrity primitives evaluation message digest (RIPEMD) series, or the like. Hash functions may be used, but the present invention is not limited thereto, and other types of hash functions may be used.

The hash collection generation module 128 receives the hash value generated from the hash calculation module 126 and generates a hash collection file corresponding to the document collection file. Each hash document included in the hash collection file includes a document identification code, a document generation time, a hash count, and a hash value, each of which is divided into <gdid>, <time>, <count>, and <hash>. It has a certain form as shown in (b).

As an example, FIG. 3B is a hash document corresponding to the document represented in FIG. 3A. This hash document has a document identification code 'ID1' and has the same document generation time as that document. In (a) of FIG. 3, if each sentence of the body is a chunk and CC and CF are invalid chunks, the hash values ha, hb, hd, and he in FIG. 3 (b) are valid chunks CA, CB, Hash values for CD and CE. In this case, the hash count '4' is the number of hash values and is equal to the number of valid chunks.

The hash indexing server 130 receives the hash collection file from the hash generation server 120 and performs an indexing operation based on the hash value included in the hash collection file to update the hash index stored in the hash index volume 140. do.

The hash index volume 140 stores a hash index for all hash values as a storage medium. Hash values are arranged in alphabetical order in order to facilitate retrieval. For each hash value, the document identification code and document generation time of the hash document including the corresponding hash value are listed.

FIG. 5 is a block diagram illustrating an example of the score generation server shown in FIG. 1.

Referring to FIG. 5, the score generation server 150 includes a hash parsing module 152, a search module 154, a sorting module 156, a score calculation module 158, and a score collection generation module 159, A score collection file is generated using the hash collection file from the hash generation server 120 and the hash index of the hash index volume 140.

The hash parsing module 152 receives a hash collection file from the hash generation server 120 and parses each hash document included therein, so that the score generation server 150 generates a document based on the hash value of each hash document. Allow the score to be calculated.

The search module 154 receives the hash document parsed from the hash parsing module 152 and inquires the hash index volume 140 for each hash value included in the hash document. As a result of the search, a document having a corresponding hash value is extracted from the hash index volume 140. If the number of hash values that the parsed hash document and the extracted document have in common is more than a predetermined set value, it is determined that the documents overlap each other. The predetermined set value can be set by varying the number of valid chunks that each document has.

The search module 154 performs an inquiry on the entire hash document included in the hash collection file, and transmits information about the document without the duplicate document and the document determined to be the duplicate document to the sorting module 156.

The sorting module 156 compares documents determined to be duplicates from the search module 154 based on document generation time or document identification code, and arranges them in order, and then distinguishes the original document from the copy document. The document with the earliest document generation time is the original document, and the rest is the copy document. In the case of duplicate documents having the same document generation time, the document in the news category may be determined to be generated first, and the document generation order may be determined in the order of knowledge information, cafe, and blog category. Therefore, in this case, a document with a small document identification code can be determined as the original document and the remaining documents as copy documents. This is because the news can be identified as the first document generated according to the delimiter as described above.

On the other hand, a document without duplicate documents becomes an original document.

The score calculation module 158 calculates document scores of the original document and the copy document in different ways and assigns document scores to the corresponding documents, respectively.

The document score (OS) of the original document is a value obtained by adding 1 to the sum of the proportion of the portion copied to the copy document in the original document, and is calculated as in Equation 1 below.

Where 'OD' is the number of valid chunks of the original document, and 'CD _i ' is the number of valid chunks that the i th copy document and the original document have in common. n is the number of copy documents determined to be a duplicate of the original document.

The document score (OS) of the original document means how many valid chunks of the original document have been copied to the copy document, which is a real number that is one or more. Therefore, the larger the document score (OS) of the original document, the more effective chunks of the original document can be determined to have been copied to the copy document.

Of course, the document score of the original document may have a value different from the calculated value of [Equation 1], for example, may have a value not adding 1 in [Equation 1].

The document score CS of the copied document is a ratio occupied by the portion of the copied document except the portion copied from the original document, and is calculated as in Equation 2 below.

Where 'CT' is the number of valid chunks of the copy document, and 'CD' is the number of valid chunks that the copy document and the original document have in common.

The document score (CS) of the copy document indicates how much different content is added from the original document and is a real number less than one. Therefore, as the document score CS of the copied document is smaller, it can be determined that there is no content added to the copied document in addition to the copied portion.

The score collection generation module 159 receives the document score calculated from the score calculation module 158 and generates a score collection file corresponding to the hash collection file. Each score document included in the score collection file includes a document identifier, a document score, a copy count, and a copy list, each divided into <gdid>, <cscore>, <ccount>, and <clist>. As shown in c), it has a certain form.

The copy count indicates the number of copy documents judged to be duplicates of the original document in the case of the original document, and is 1 in the case of the copy document. In the copy list, the document identification code of the copy document determined to be a duplicate of the original document in the case of the original document, and the document identification code of the original document in the copy document is displayed. However, in the case of a copy document, the document identification code of another copy document of the original document may be displayed in the copy list.

The score document shown in Fig. 3C has a document identification code 'ID1', and the document score is 3.5. Therefore, this document is identified as the original document because the document score is 1 or more. The valid chunks of this original document were copied into three copy documents, and the document identifiers of these copied documents were 'ID2', 'ID3' and 'ID4'.

The score generation server 150 sends the score collection file thus generated to the document indexing server 160. In addition, the score generation server 150 updates the document score information stored in the score collection database 180 using the generated score collection file.

The score collection database 180 may store score information including a document identification code, a document score, a copy list, and the like, and may separately receive and store the score collection file itself from the score generation server 150.

The document indexing server 160 performs a document indexing operation using the document collection file from the document database 110 or the document collection generation module and the score collection file from the score generation server 150. The document index stored in the document index volume 170 is updated. In addition, the document indexing server 160 may directly request the score collection database 180 and receive score collection information therefrom to perform document indexing.

The document index volume 170 is a storage medium and stores a document index for all documents. In the document index, the keyword for inquiry is arranged, and for each keyword, the document identification code and the document score of the document including the keyword are listed.

The document processing system 100 according to the embodiment of the present invention may perform document processing in a batch processing method. As time passes, new documents created by the user are accumulated in the document database 110. When new documents are accumulated in the document database 110 by a predetermined amount, a document collection file is generated for each newly accumulated document, Document processing can be performed in the manner described so far, such as by assigning a document score. Alternatively, document processing may be performed at regular intervals for documents newly accumulated in the document database 110.

Unlike the batch processing method, the database inquiry server 190 may query or calculate a document score of a specific document according to a request of an operator or an administrator of the search system 400 or the document processing system 100. If the document score of a particular document is already stored in the score collection database 180, the document score may be simply retrieved from the score collection database 180 and returned. However, if not stored, the database query server 190 may calculate a document score for the particular document. That is, in this case, the database query server 190 extracts a valid chunk for a specific document, generates a hash value, queries the hash index volume 140, extracts duplicate documents, and retrieves the original document and the copy document according to the document generation time. After the determination, the document score may be returned after document processing such as calculating the document score.

The document processing system 100 according to the embodiment of the present invention may optionally include a score collection database 180 or a database query server 190.

The search system 400 may appropriately respond to a user search query based on document scores assigned to each document. That is, the search system 400 receives the query from the user terminal 300 and searches the document index volume 170 based on the contents of the query. In addition, after finding documents related to the query in the document index volume 170, display ranks of the documents to be displayed on the user terminal 300 may be determined based on document scores of the documents. Documents with a high document score may be given priority to be displayed first on the search result screen, and documents having a document score of less than 1 may not be displayed. Alternatively, documents determined to be duplicate documents may be displayed by grouping / clustering. However, the task of determining or grouping the display order of documents based on document scores of documents related to the query may be performed in the document processing system 100 according to an embodiment of the present invention.

6 to 9, a method of calculating document scores according to an embodiment of the present invention will be described in detail, for example.

6-9 are schematic diagrams showing examples of calculating document scores according to an embodiment of the present invention.

The hash document of the hash collection file is displayed above the inverted triangle of each figure, and the score document of the corresponding score collection file is displayed below. In order to avoid duplication, some hash documents are omitted and only score documents are shown. For convenience of description, the hash collection file may display not only generated in the current batch process but also generated in the previous batch process.

Referring first to FIG. 6, hash documents having document identification IDs ID11, ID12, ID13, and ID14 are listed side by side. Hereinafter, the document having the document identification code ID11 will be simply referred to as 'ID11 document' or 'document ID11', and other documents will be described in the same manner.

Document ID11 has hash values A1, B1, C1, D1, E1, and F1, document ID12 has the same hash value as document ID11 (ie, document ID12 is substantially the same as document ID11), and document ID13 has a hash value. With A1, B1 and C1, document ID14 has the same hash value and hash values G1 and H1 as document ID11. When the number of duplicate hash values that can be determined as duplicate documents is set to 2, these documents may be determined to be duplicate documents by the search module 154.

Documents ID11, ID12, ID13, and ID14 have document generation times T11, T12, T13, and T14, respectively. Assuming that time T11 is the time earlier than the other document generation time, document ID11 becomes an original document, and document ID12, ID13, and ID14 become copy documents.

According to [Equation 1], the document score of the original document ID11 is 3.5. That is, the number of hash values of document ID11 is 6, the number of hash values common to documents ID11 and ID12 is 6, so the ratio by document ID12 is 1, and the number of hash values that document ID11 and ID13 have in common. Since it is 3, the ratio by document ID13 is 0.5, and since the number of hash values which document ID11 and ID14 have in common is 6, the ratio by document ID14 is 1. Therefore, 3.5 which adds 1 to these sums becomes the document score of document ID11.

According to [Equation 2], the copy documents ID12 and ID13 do not have hash values other than the hash values of the original document ID11, so the document scores of the copy documents ID12 and ID13 are zero. Since the copy document ID14 has hash values G1 and H1 in addition to the hash value possessed by the original document ID11, the document score of the copy document ID14 is 2/8 = 0.25.

In the copy list of the original document ID11, copy document ID12, ID13, and ID14 are listed, and the copy count is three. The original document ID11 is displayed in the copy list of the copy documents ID12, ID13, and ID14, and the copy count is one.

Referring to FIG. 7, document ID21 was generated at time T21 and has hash values A2, B2, C2, D2, E2, and F2. Document ID22 was generated at time T22 and has hash values A2, B2, C2, D2, E2, F2, G2 and H2. Document ID23 was generated at time T23 and has hash values D2, E2, F2, G2, and H2. Document ID24 was generated at time T24 and has hash values G2 and H2.

Assuming that time T21 is the time earlier than other document generation time, document ID21 becomes an original document, and document ID22 and ID23 become a copy document.

The document score of the original document ID21 is 2.5, the copy documents ID22 and ID23 are displayed in the copy list, and the copy count is two. The document scores of the copy documents ID22 and ID23 are 0.25 and 0.4, respectively, the original document ID21 is displayed in the copy list, and the copy count is one.

On the other hand, hash values G2 and H2 are not included in the original document ID21 but are included in the copy documents ID22, ID23, and ID24. Assuming that time T22 is a time earlier than times T23 and T24, document ID22 serves as the original document for hash values G2 and H2, and document ID24 becomes a copy document of document ID22, as indicated by the arrow in FIG. Therefore, the document score of the document ID24 is 0 according to [Equation 2], the document ID22 is displayed on the copy list, and the copy count is 1.

Referring to FIG. 8, the documents ID32 and ID33 are documents included in the document collection file generated during the previous document processing operation and are documents for which the document score is calculated. If time T32 is earlier than time T33, document ID32 is the original document, and document ID33 is a copy document of document ID32 because document ID32 and hash values A3, B3, and C3 overlap. Therefore, document ID32 is displayed in the copy list of document ID33. Document scores of documents ID32 and ID33 are 1.5 and 0.4, respectively.

Assume that the document processing system 100 performs a batch processing document processing operation at time T3. Documents ID31 and ID34 are included in the new document collection file, and because document ID31 has hash values G3 and H3 and document ID34 has hash values A3, B3, and C3, these documents ID31 and ID34 are duplicates of documents ID32 and ID33. It is considered a document. Here, if time T31 is ahead of time T32 and time T32 is ahead of time T34, the original document of document ID33 is changed from document ID32 to document ID31 because document ID31 is generated before document ID32. In other words, the original document of the copied document, the contents of which are copied from several documents, becomes the first generated document among several documents. Of course, the original document of document ID34 becomes document ID32.

As a result, the document score of document ID33 is changed to 0.6 and the copy list is changed to document ID31. The document score of document ID32 is changed to 2 and document ID34 is added to the copy list. The document score of document ID31 is 2, and document ID33 is described in the copy list. Document ID34 has a document score of 0 and document ID32 is described in the copy list.

A score collection file containing such document score information is generated at the time point T3 'when the document processing operation is completed, as shown in the bottom row of FIG.

As such, even if a document whose document score has been previously calculated is newly inputted, a document score and a copy list may be changed by this document.

9, document ID41 has hash values A4 and B4, document ID42 has hash values A4, B4 and C4, and document ID43 has hash values A4, B4, A4, B4, A4, B4 and C4. have. Document ID43 is a document created by repeatedly copying the contents corresponding to A4 and B4. If time T41 is earlier than time T42 and T43, document ID41 becomes the original document, and document ID42 and ID43 become the copy document.

The document score of document ID41 is 3, and document ID42 and ID43 are described in a copy list. The document score of document ID42 is 0.33, and document ID41 is described in a copy list. However, Document ID43 can vary the value of the document score, depending on how the number of hash values is handled.

First, in case (a), document ID43 has seven hash values, there are two hash values overlapping document ID41 (A4 and B4), and there are five non-duplicate hash values. 5/7 = 0.71. In the case of (b), document ID43 has seven hash values, there are six hash values overlapping document ID41 (all duplicated A4 and B4), and one non-overlapping hash value (C4). In this case, the document score is 1/7 = 0.14. Finally, in the case of (c), it treats duplicate hash values as one, so that document ID43 has three hash values, there are two hash values overlapping document ID41, and one non-duplicate hash value. In this case, the document score is 1/3 of 0.33.

If necessary, the document processing system 100 may calculate a document score by appropriately selecting any one of the above three cases with respect to a document having duplicate contents by itself.

The server included in the document processing system 100 according to the exemplary embodiment of the present invention may be a process for performing a corresponding function, or may be implemented as hardware for performing such a function. In addition, the modules included in each server are not necessarily included in the server, but may be included in other servers or separated independently.

The document processing system 100 according to the embodiment of the present invention may further include a management module for managing the document database 110, the hash index volume 140, the document index volume 170, and the score collection database 180. Can be.

Embodiments of the invention include a computer readable medium containing program instructions for performing various computer-implemented operations. This medium records a program or process for executing a method of processing a document as described so far. The media may include, alone or in combination with the program instructions, data files, data structures, and the like. Examples of such media include, but are not limited to, magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CDs and DVDs, floppy and magnetic disks, programs such as ROM, RAM, flash memory, and the like. Hardware devices configured to store and execute instructions. Alternatively, the medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

Although the preferred embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

As described above, according to the present invention, it is possible to determine whether an original document or a copy document is obtained by extracting chunks from each document and comparing hash values based on the extracted chunks.

In addition, by calculating and assigning a document score to the number of duplicate hash values, you can determine how much of the content of the original document was copied to another document in the case of the original document. You can see how much of this has been added.

By creating a copy list for duplicate documents, the original document can be identified in the case of the original document, and in the case of the copy document, the original document can be identified.

In addition, the user may provide a higher quality search result to the user by ranking or displaying the search results using the document scores assigned to the documents.

Claims (15)

Extracting chunks from the plurality of documents, Querying a storage medium to extract duplicate documents having the chunk in common; Comparing the document generation time of the duplicate document to distinguish the original document from the at least one copy document, and Calculating scores of the original document and the copied document based on a common chunk that the duplicate documents have in common Document processing method comprising a. In claim 1, And the score of the original document is the sum of the number of chunks of the original document and the number of the common chunks. In claim 1, The score (OS) of the original document is calculated by the following equation,

OD is the number of chunks of the original document, CD _i is the number of common chunks of the i-th copy document, and n is the number of the copy documents. In claim 1, And a score of the copy document is a ratio of the number of chunks of the copy document minus the number of the common chunks and the number of chunks of the copy document. In claim 1, Generating a score collection file recording document identification marks, document scores, and copy lists for each document, The document identification code of the copy document is described in the copy list of the original document, and the document identification code of the original document is described in the copy list of the copy document. delete In claim 1, And the duplicated document has the common chunk of a predetermined number or more. A computer-readable medium having recorded thereon a program for causing a computer to execute the method of any one of claims 1 to 5 and 7. A chunk extraction module for extracting chunks from a plurality of documents, A search module for querying a storage medium to extract duplicate documents having the chunks in common; A sorting module for comparing the document generation time of the duplicate document to distinguish the original document from the at least one copy document, and A score calculation module for calculating a score of the original document and the copied document based on a common chunk that the duplicate document has in common Document processing system comprising a. In claim 9, And a score of the original document is a sum of a ratio of the number of chunks of the original document and the number of the common chunks. In claim 9, The score (OS) of the original document is calculated by the following equation,

OD is the number of chunks of the original document, CD _i is the number of common chunks of the i th copy document, and n is the number of the copy documents. In claim 9, And a score of the copy document is a ratio of the number of chunks of the copy document minus the number of the common chunks and the number of chunks of the copy document. In claim 9, And further comprising a score collection generation module for generating a score collection file for each document documenting document identification marks, document scores, and copy lists. The document identification code of the copy document is described in the copy list of the original document, and the document identification code of the original document is described in the copy list of the copy document. delete In claim 9, And the duplicated document has a predetermined number or more of the common chunks.

Images