KR101399272B1 - Document similarity estimation method - Google Patents

Abstract

The present invention relates to a method for estimating document similarity, wherein the method comprises: a step of extracting multiple first representative words from a first group including at least one document, and extracting weighted values indicating the degree of representing the first group for each of the multiple first representative words together; a step of extracting multiple second representative words from a second group including at least one document, and extracting weighted values indicating the degree of representing the second group for each of the multiple second representative words; and a step of estimating the similarity between the first group and the second group by measuring the similarity between the multiple first representative words and the multiple second representative words.

Description

{Document Similarity Estimation Method}

The present invention relates to a method for inferring similarity of a document, and more particularly, to a method for inferring a similarity between a document and a document, between a document and a document group, and between a document group and a document group.

With the rapid development of the Internet, the sharing of information has benefited mankind. Through the development of computers and the Internet, various information can be provided or provided. Even in the case of academic journals of specialized academic organizations, it is possible to offer and subscribe to papers online.

There is a limit to human information acquisition ability compared to the massive amount of information, and various information search functions are being studied. Especially, most of the academic papers use text (text). Research on text mining, which is useful for the processing of data composed of such texts, is actively being conducted.

It is necessary to provide services such as appropriate submission of articles written in such an environment or search for articles.

Korean Patent Publication No. 10-2010-0038378 (Apr. 14, 2010)

SUMMARY OF THE INVENTION The present invention is directed to provide a method of inferring similarity of a document, which has been devised to meet the needs of the prior art. At this time, the similarity of the document may include not only between the document and the document, but also between the document and the document group, between the document group and the other document group.

The technical objects to be achieved by the present invention are not limited to the above-mentioned technical problems, and other technical subjects which are not mentioned can be clearly understood by those skilled in the art from the description of the present invention .

A document similarity reasoning method according to an embodiment of the present invention includes extracting a plurality of first representative words from a first group including one or more documents, the degree of representing the first group for each of the plurality of first representative words Extracting a weight with a weight indicating a weight; Extracting a plurality of second representative words from a second group including one or more documents together with a weight indicating a degree of representing the second group for each of the plurality of second representative words; And inferring the similarity between the first group and the second group by measuring a degree of similarity between the plurality of first representative words and the plurality of second representative words.

In an embodiment of the present invention, the extraction of the plurality of first representative words and the extraction of the plurality of second representative words may be performed using a TFIDF (Term Frequency Inverse Document Frequency) technique.

According to an embodiment of the present invention, a method of deducing the similarity of a document can be provided.

In embodiments of the present invention, the similarity of a document may include not only between the document and the document, but also between the document and the group of documents, between the group of documents and the group of other documents. Thus, according to embodiments of the present invention, the similarity between a particular document set and another document set can be deduced. In addition, according to the embodiment of the present invention, it is possible to deduce the similarity between the created document and the research field between the specific document group. In addition, according to the embodiment of the present invention, the degree of similarity between the created document and the specific document can be deduced.

Also, according to the embodiment of the present invention, it is possible to find a suitable publication society and field suitable for the thesis, research field, and subject field, and enable the thesis unit search service.

1 shows a flowchart of a document similarity reasoning method according to an embodiment of the present invention.
Figure 2a relatively illustrates the size and concept of word weights in documents and groups.
Figure 2b shows the magnitude and concept of word weights relatively in two different groups.
Figure 2c shows the magnitude and concept of word weights relatively in two different documents.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a detailed description of preferred embodiments of the present invention will be given with reference to the accompanying drawings. However, the embodiments of the present invention may be modified into various other forms, and the scope of the present invention is not limited to the embodiments described below. The shape and the size of the elements in the drawings may be exaggerated for clarity of explanation and the same reference numerals are used for the same elements and the same elements are denoted by the same quote symbols as possible even if they are displayed on different drawings Should be. In the following description, well-known functions or constructions are not described in detail to avoid unnecessarily obscuring the subject matter of the present invention.

1 shows a flowchart of a document similarity reasoning method according to an embodiment of the present invention. As shown in FIG. 1, a document similarity inference method according to an exemplary embodiment of the present invention extracts a plurality of first representative words from a first group, extracts a first representative word from a plurality of first representative words, (S110) extracting a plurality of second representative words from the second group, and extracting the second representative words together with a weight indicating a degree of representing the second group for each of the plurality of second representative words (S120), measuring a degree of similarity between a plurality of first representative words and a plurality of second representative words (S200), and estimating a degree of similarity between the first and second groups based on the result of step S200 (S300).

The first group may include one or more documents. A document may represent a certain doctrine, idea, or thought in text, such as a letter or symbol. In the embodiment of the present invention, the independent document may be a paper prepared for a specific research topic. The first group may represent a collection of one or more articles. For example, the first group may be a set of papers belonging to a specific society or a set of papers belonging to the same research field. The description of the first group can be equally applied to the second group.

The plurality of first representative words representing the first group may be extracted from one or more documents of relatively high importance among the one or more documents included in the first group. For example, representative words representative of a first group, which is a collection of one or more articles for a particular research topic, may be extracted. At this time, the representative words may be words representing the research topic of the first group. At this time, in order to increase the reliability of extracted representative words, representative words can be extracted from documents having high importance among documents included in the first group.

A document of high importance in the first group may be, for example, a document of high recognition among the first group. For example, if the first group is a collection of one or more articles, the article with a high degree of importance may be a article having a large number of citations in a search engine such as google. In addition, high-priority papers can be selected through the recognition method used by Microsoft. This is because superior papers in the same academic or research field will be highly recognized and cited accordingly. At this time, the determination of importance can be variously performed according to the embodiment. Thus, reliability of representative words extracted from papers with high importance can be increased. At this time, depending on the reliability required for the representative word representing the first group, how many papers in the first group can be selected as the papers with high importance can be determined.

If only one document is included in the first group, words representative of one document can be extracted. At this time, the document itself included in the first group may represent the first group. The description of the first group can be applied to the second group as well.

The step S100 of extracting a plurality of representative words from a specific group may be performed using a TFIDF (Term Frequency Inverse Document Frequency) technique. TFIDF is a method of evaluating the importance of each word in the document by weighting the frequency of use of the word and the number of times used in various documents in one document. The more frequently used words in a document are, the higher the term frequency (TF) is, which increases the weight. However, the word often referred to in many documents other than the document has a lower weight (Inverse Document Frequency, IDF). This is because, if a word is frequently referred to in a document other than the document, the word is likely to be a general word, which is inappropriate as a representative word of the document.

At this time, representative words representing a specific group can be sorted in descending order of weight among words appearing in the group. Here, a word having a high weight may be referred to as a word having a high priority. Representative words representing a particular group may include a certain number or a certain percentage of words in the order of the highest weight among the words appearing in the group. The specific number or specific ratio may be applied differently in each embodiment.

In this case, if only one document is included in the group, that is, if the document itself represents a group, the representative words extracted from the group may be important terms indicating the subject or research topic of the document. If the group contains multiple documents, the representative words extracted from the group may be important terms that indicate the subject of the group, the research topic, or the field of research.

Figure 2a relatively illustrates the size and concept of word weights in one document and group. For example, the difference in weights of representative words extracted according to the TFIDF technique for documents and groups is conceptually shown in FIG. 2A. In Figs. 2A-2C, the document may be, for example, a separate article having a research topic, and the group may be a collection of articles including a plurality of articles in a specific research field.

Generally, for a particular word, the weight in the group is greater than the weight in one document. In FIG. 2A, it is exemplified that the weight in the group of words 1 to 3 is larger than the weight in the document. If a document and a group have a similar research field, the word representing that research field is mentioned more often in a group than in a single document.

Figure 2b shows the magnitude and concept of word weights relatively in two different groups. For example, the weight difference of the representative words extracted according to the TFIDF scheme for two different groups is conceptually shown in FIG. 2B.

Even with the same word, the weights may be different in the two groups of research fields or subjects with each other. 2B illustrates that the weights in the first group of words 1 and 2 are greater than the weights in the second group but the weights in the first group of words 3 are less than the weights in the second group.

Figure 2c shows the magnitude and concept of word weights relatively in two different documents. For example, the weight difference of the representative words extracted according to the TFIDF technique for two different documents is conceptually shown in FIG. 2C.

Even with the same word, the weights may be different in the two documents that differ from each other in the research field or subject. 2C, the weights in the first document of words 1 to 3 are larger than the weights in the second document.

Referring again to FIG. 1, in the document similarity inference method according to the embodiment of the present invention, similarity between a plurality of first representative words extracted in S100 and a plurality of second representative words may be measured (S200). The similarity between these representative words can be measured by various methods and the following three methods are exemplified.

First, the measurement of the degree of similarity between the plurality of first representative words and the plurality of second representative words is performed based on the weight in the first group for the common word between the plurality of first representative words and the plurality of second representative words, Can be measured using the difference in weights in the group. The similarity between the first representative word and the second representative word according to the first method can be calculated according to the following equation (1).

수식(1)

Equation (1)

Here, w1 to wn represent weights in the first group of words common to both the first representative word and the second representative word. c1 to cn represent weights in the second group of words corresponding to the weight values w1 to wn. As described above, the similarity Sall between the first representative words and the second representative words is calculated by summing the weight difference values of common words among the first representative word and the second representative word, Can be calculated.

The first similarity measure method may be suitable for a case where the second group includes only one document, the document itself is a second group, and the first group includes a plurality of documents. Equation (1) utilizes the fact that the weight of words in the first group is much larger than the weight in the second group, which is relatively one document, as illustrated in FIG. 2A. Since the weight of the word in the first group is much larger than the weight of the same word in the second group (the document itself), according to Equation (1), the presence or absence of a common word between the first representative word and the second representative word May have a large influence on the similarity value (Sall). However, if the weights in the second group of words common to the first representative word and the second representative word are large, the similarity value Sall may be measured to be low.

Second, the measurement of the degree of similarity between the plurality of first representative words and the plurality of second representative words can be performed by examining the presence or absence of a common word between a plurality of first representative words and a plurality of second representative words. The similarity between the first representative word and the second representative word according to the second method can be calculated according to the following equation (2).

수식(2)

Equation (2)

Here, n represents the number of a plurality of second representative words extracted from the second group. At this time, when the i-th word among the second representative words is common to the first representative word and the second representative word, the Si value has a value of one. The Si value has a value of 0 when the i-th word of the second representative word is present in the second representative word but not in the first representative word. The similarity value (Sall) value between the first representative word and the second representative word according to the second method may have a value of the number of common words between the first representative word and the second representative word. However, according to the embodiment, it is possible to calculate the similarity Sall which is numerically differentiated by additionally adding the weights of the common words in the first group or the weights of the common words in the second group.

The second similarity measure method may also be suitable when the second group includes only one document, the document itself is a second group, and the first group includes a plurality of documents.

Third, the similarity measure between the plurality of first representative words and the plurality of second representative words may be performed by measuring the degree of similarity between the plurality of first representative words and the plurality of second representative words in the first group By using an exponential function such that the smaller the difference between the weights and the weights in the second group is, the greater the degree of similarity is. The similarity between the first representative word and the second representative word according to the third method can be calculated according to the following equation (3).

수식(3)

Equation (3)

Here, n represents the number of common words between the first representative word and the second representative word. σ is a variable for adjusting the magnification of the degree of similarity. If the value of σ is set small, the resultant similarity value becomes large. On the contrary, if the value of σ is set large, the similarity value may become small. The magnitude of this sigma value can be set according to the scale of the similarity value value required according to the embodiment. Equation (3) can be used to measure the differentiated similarity (Sall) when the common words between the first representative word and the second representative word have similar weights in the first and second groups. According to Equation (3), as the weight wi in the first group and the weight ci in the second group have similar values by applying the exponential function, higher similarity Sall can be calculated . This third similarity measurement method is a method in which both the first group and the second group include a plurality of documents or the document itself is a first group and a second group including only one document for each of the first group and the second group Lt; / RTI >

In addition, the first through third similarity measurement methods described above can be used in combination of one or more of them.

The document similarity inference method of the present invention can infer the similarity between the first group and the second group based on the similarity (Sall) between the first representative word and the second representative word measured as described above . For example, the similarity (Sall) itself between the first representative word and the second representative word may be estimated as the similarity between the first group and the second group.

The document similarity reasoning method according to the embodiment of the present invention may be implemented in a programming language and executed in a computer.

Although the document itself has been used for extracting representative words in the embodiment of the present invention, a paragraph including the largest number of representative words among a plurality of paragraphs included in one document is additionally used as a representative document .

According to an embodiment of the present invention, a method of deducing the similarity of a document can be provided.

In embodiments of the present invention, the similarity of a document may include not only between the document and the document, but also between the document and the group of documents, between the group of documents and the group of other documents.

Thus, according to embodiments of the present invention, the similarity between a particular document set and another document set can be deduced. In addition, according to the embodiment of the present invention, it is possible to deduce the similarity between the created document and the research field between the specific document group. In addition, according to the embodiment of the present invention, it is possible to infer the similarity between a research field and a subject between a created document and a specific document.

In addition, according to the embodiment of the present invention, it is possible to search for appropriate articles and societies in terms of articles, research fields and subjects, and to enable a thesis unit search service.

According to an embodiment of the present invention, a data search system and a service can be provided based on a document (or a thesis) itself, rather than a key word based search. Therefore, according to the embodiment of the present invention, it is possible to present the results of the articles retrieved in descending order of similarity by using the articles themselves for retrieval.

In addition, according to the embodiment of the present invention, it is possible to infer the degree of association or similarity between the article prepared and the research field of the society to be submitted.

As described above, by using the topic model according to the embodiment of the present invention, data having a story flow such as a novel or a history document from the viewpoint of an object or an object group can be divided into a flow of a paragraph or a flow of a predetermined section You can then analyze the topic. In addition, readers of the document can automatically obtain results about how the story is going, centering on characters, places, or events without reading the document. In addition, according to the embodiment of the present invention, the topic flow can be analyzed using only the document itself as data without additional data or metadata.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. will be. Therefore, it should be understood that the above-described embodiments are to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than the foregoing description, It is intended that all changes and modifications derived from the equivalent concept be included within the scope of the present invention.

Claims (7)

Extracting a plurality of first representative words from a first group including at least two documents and extracting the plurality of first representative words together with a weight indicating a degree of representing the first group for each of the plurality of first representative words;
Extracting a plurality of second representative words from a second group including at least two documents and extracting the second representative words together with a weight indicating a degree of representing the second group for each of the plurality of second representative words; And
And inferring the similarity between the first group and the second group by measuring the similarity between the plurality of first representative words and the plurality of second representative words,
The similarity (Sall) between the plurality of first representative words and the plurality of second representative words is

, ≪ / RTI >
Where n is the number of words common to the plurality of first representative words and the plurality of second representative words, wi represents the weight of the common word in the first group, and ci represents the weight of the common word in the second group ≪ / RTI >
Document similarity inference method. The method according to claim 1,
Wherein the extraction of the plurality of first representative words and the extraction of the plurality of second representative words are performed using a TFIDF (Term Frequency Inverse Document Frequency) technique.
Document similarity inference method. The method according to claim 1,
Wherein the plurality of first representative words are extracted from at least one of the at least two documents included in the first group.
Document similarity inference method. delete delete delete A computer-readable medium storing a program for causing a computer to execute a document similarity inference method according to any one of claims 1 to 3.

Images