KR20160098084A - System for filtering documents of interest and method thereof - Google Patents

Abstract

The present invention relates to a system and a method for filtering documents of interest, which can filter noise or key documents from a plurality of documents by extracting words included in the documents such as patents or papers, selecting a reference seed document through information quantity analysis using the extracted words, and grouping the documents similar to the reference seed through semantic similarity relation analysis. The method for filtering documents of interest comprises the steps of: selecting, as the reference seed, at least one specific document which is an object of interest among the documents; analyzing the topic of each document based on word statistical information on the documents to calculate topic-specific probability distribution and calculating a distance to the reference seed by using the topic-specific probability distribution of each document; and selecting and filtering the documents of interest according to the calculated distance.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document filtering system,

The present invention relates to a system and method for filtering interest documents for automatically extracting noise or key data of interest from collected population data.

In general, technical documents (populations) such as patents or papers collected through keyword search contain noise documents that must be removed in addition to the effective documents in the target technical field to be analyzed. In addition, the effective documentation also includes core documents that are more closely related to the technical field of interest.

In order to efficiently analyze the patent, effective patent documents must be collected. In order to do this, a filtering process is required to remove noise documents or extract key documents.

For example, in order to remove a noise document, a human being directly examines a patent document directly in a population and removes a noise document through a simple repetitive operation of removing noise, so that a lot of time and manpower are put into. Of course, it takes a lot of time to manually extract the core document and extract it while reviewing the document contents.

The following patent documents relate to a document retrieval and classification method and system thereof, but fail to provide a solution to the above-mentioned problems.

Korean Patent Publication No. 10-2006-0047306

In order to solve the problems of the prior art described above, the present invention extracts words included in a document such as a patent or a paper, selects a reference seed document through analysis of information using the same, The present invention provides a system and method for filtering a document of interest that can filter noise or key documents from multiple documents by clustering documents similar to seeds.

A method of filtering a document of interest according to an embodiment of the present invention includes the steps of: (a) selecting at least one specific document of interest as a reference seed among a plurality of documents; (b) analyzing a topic of each document based on the word statistical information about the plurality of documents to calculate a subject probability distribution, and calculating a probability distribution of the subject based on the subject's probability distribution, ; And (c) selecting and filtering an interest document according to the calculated degree of similarity.

The reference seed of step (a) is selected on the basis of information entropy calculated through a probability of occurrence of a predetermined interest in a plurality of documents, and the reference seed is selected from a range of information entropy of the calculated information entropy and a condition Can be selected.

The step (b) includes: calculating a probability distribution of each document by applying a topic modeling algorithm to the word statistical information; Calculating a degree of similarity between a reference seed and a document to be compared using a subject probability distribution of each document and a subject probability distribution of the document selected as the reference seed; And clustering and providing documents similar to the reference seed based on the calculated similarity.

A system for filtering a document of interest according to an embodiment of the present invention includes: a reference seed selection unit for selecting at least one specific document of interest as a reference seed among a plurality of documents; A seed clustering unit for calculating a probability distribution of the subject through subject analysis of each document based on the word statistical information about the plurality of documents and calculating a degree of similarity with the selected reference seed using the subject classified probability distribution of each document; And an interest document filtering unit for selecting and filtering an interest document according to the calculated similarity for each document.

The filtering system may further include a word extracting unit for extracting a word for each document from the collected plurality of documents; A statistical information acquiring unit for measuring the occurrence frequencies of the extracted words in the document and acquiring the word statistical information using the appearance frequencies of the words and the inverse document frequencies; And an information entropy calculation unit for calculating information entropy of each document through a probability of appearance frequency in a document of a preset interest word with respect to the number of occurrences of all words in the document measured in the above.

According to the embodiment of the present invention, in the case of a filtering system of interest documents such as noise or a core document, it is possible to discriminate the extent of information about the technical group to be analyzed only by qualitative analysis by the Shannon entropy method It is possible to express the possibility of the document of interest in quantified numerical value to help the expert judgment.

This is because it is possible to help to maintain and maintain the consistency of problems that are inconsistent with the judgment of the interest documents, which is one of the problems of the long time simple repetitive working methods of the existing experts, so that the processes that may cause errors or errors are systematically supplemented Thereby improving the quality of the work.

Topic analysis modeling, which is a topic analysis, plays a role of collecting and grouping documents that are semantically similar to the reference seed, extracting the documents of interest that the experts are confident of, So that it can be extracted collectively. It is possible to drastically reduce the chances of filtering interest documents that an expert should perform.

In the present invention, since the process of filtering the target document is performed a plurality of times from the process of selecting the reference seed, the range of the document set requiring the sorting operation can be sufficiently reduced. This is because it can reduce the time investment of the professional manpower in comparison with the time cost which has been consumed by the conventional method, thereby increasing the speed and efficiency of research and development.

1 is a conceptual diagram for explaining a system for filtering a document of interest according to the present invention.
2 is a block diagram illustrating a system for filtering a target document according to an exemplary embodiment of the present invention.
3 is a diagram illustrating the seed clustering unit shown in FIG.
4 is a flowchart illustrating a method for filtering a target document according to an embodiment of the present invention.
5 is a UI screen illustrating a document filtering system according to an embodiment of the present invention.
6 is a flowchart for explaining the word extraction and appearance frequency calculation process of FIG.
7 is a view showing the process of FIG.
FIG. 8 is a flowchart for explaining the seed clustering process of FIG.
9A to 9C are diagrams for explaining the LDA applied to the calculation of the subject probability distribution of FIG.
10 is a screen for extracting and displaying a document similar to the reference seed according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

However, the embodiments of the present invention can be modified into various other forms, and the scope of the present invention is not limited to the embodiments described below. Further, the embodiments of the present invention are provided to more fully explain the present invention to those skilled in the art.

In the drawings referred to in the present invention, elements having substantially the same configuration and function will be denoted by the same reference numerals, and the shapes and sizes of the elements and the like in the drawings may be exaggerated for clarity.

As used in this embodiment, the term " portion " refers to a hardware component such as software or an FPGA (field-programmable gate array) or ASIC, and 'part' performs certain roles.

However, 'part' is not meant to be limited to software or hardware. &Quot; to " may be configured to reside on an addressable storage medium and may be configured to play one or more processors.

Thus, by way of example, 'parts' may refer to components such as software components, object-oriented software components, class components and task components, and processes, functions, , Subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

The functions provided in the components and components may be further combined with a smaller number of components and components or further components and components.

In addition, the components and components may be implemented to play back one or more CPUs in a device or a secure multimedia card.

1 is a conceptual diagram for explaining a system for filtering a document of interest according to the present invention.

Generally, in order to obtain the largest possible population in a patent or thesis search, a search formula is roughly created, and then various documents with various application technologies or heterogeneous features are collected. As a result, noise is included in the population and the process of removing the noise or extracting the core document necessarily occurs.

Accordingly, the present invention proposes an efficient noise or key document extraction method. The method applied in the present invention is based on a crystal growth method which is a method used to obtain pure crystals and is subjected to a repetitive purification process. In the course of crystal growth, a method of growing relatively small crystals of pure crystals is used to obtain pure crystals of sufficient size. At this time, the crystal of a small size is a crystal seed, and the crystal seed is caused to induce crystallization of the same kind of solute in a stable saturated solution or the like to grow into a crystal of sufficient size.

Likewise, in the method proposed in the present invention, some of the documents of interest recommended by the system are selected as the reference seeds using the Shannon information entropy and semantic similarity analysis, and based on this, documents having high semantic similarity are clustered And then filtering and repeating the extraction and repetition.

2 is a block diagram illustrating a system for filtering a target document according to an exemplary embodiment of the present invention.

2, the system for filtering target documents according to an exemplary embodiment of the present invention includes a document storage unit 10, an interest storage unit 50, and an information processing unit 100. The information processing unit 100 Includes a word extracting unit 110, a statistical information obtaining unit 120, an information entropy calculating unit 130, a reference seeding unit 140, a seed clustering unit 150 and an interest document filtering unit 160 .

The document storage unit 10 can receive and store a plurality of documents collected through the keyword search formula by the document search system. In an embodiment, the document may be a patent document or a paper, and the document retrieval system may be a Google patent, a Delphi-in, a Kipris, a WIPSON, a WINTELIPS, , NDSL, and the like.

The interest word storage unit 50 stores a word of interest input from the user, that is, a keyword of interest. The word of interest is composed of at least one or more keywords, and the word of interest is used in information entropy calculation of a document to be described later.

The word extracting unit 110 extracts words from each of the plurality of collected documents. The word extracting unit 110 extracts sentences included in the document through natural language processing, extracts words corresponding to adjectives and nouns through parsing analysis of the extracted sentences, and extracts words included in a preset imperative word list The word is removed and the necessary word is extracted.

The statistical information obtaining unit 120 is configured to measure the frequency of appearance of the extracted words in the document and to obtain the word statistical information by using the appearance frequency of the word and the reverse document frequency.

Here, the statistical information obtaining unit 120 may include a frequency measuring unit 121 and a statistical information generating unit 125. The frequency measurement unit 121 is configured to measure the occurrence frequencies of words extracted through the word extraction unit 110, respectively. The statistical information generation unit 125 calculates an inverse document frequency by dividing the number of documents including an arbitrary word by the total number of documents for the extracted word and outputs the inverse document frequency multiplied by the occurrence frequency of the arbitrary word and the inverse document frequency Thereby acquiring word statistical information.

Meanwhile, the information entropy calculation unit 130 is configured to calculate the information entropy of each document by summing probabilities of occurrence frequencies in the document of a preset interest word with respect to the number of occurrences of all words in the measured document. A keyword of interest may be a key keyword or a noise keyword, and the weight may be set differently for each keyword to be reflected in the information entropy calculation.

Information entropy is an important concept of information theory, in which uncertainty is measured. That is, it has a high information entropy value in a high uncertainty state and a low information entropy value in a low uncertainty state. For example, a coin throwing event has lower uncertainty and information entropy values than six cases where there are two cases of uncertainty, that is, events that can occur, than the case of throwing a die. Also, the amount of information of one system changes according to the probability of occurrence of each event in the same situation. In other words, assuming that the probability of occurrence of each event is different in the same situation with the number of events, if the occurrence probability of each event is the same, prediction of the event becomes more difficult, and in this case, the highest information entropy value is obtained.

Such information entropy can be measured using the Shannon entropy algorithm of the following equation (1) for the discrete probability distribution. Using the mathematical expression, the amount of information about one system can be expressed as a numerical value, which means the degree of information diversity of the system. That is, if the document contains the information that the user desires, that is, the interest word, the information entropy of the document is measured by measuring the amount of information on the word of interest because the probability of the document being interested is high.

[Equation 1]

Where i is the word of interest and pi is the probability value of the word i of interest.

The reference seeding unit 140 selects at least one specific document of interest from among a plurality of documents based on information entropy of each document calculated by the information entropy calculating unit 130 as a reference seed. At this time, at least one reference seed may be selected, and the information entropy and the set reference value may be compared with each other to recommend a document that is out of the reference value as the reference seed.

The seed clustering unit 150 calculates the subject probability distribution based on the subject analysis of each document based on the word statistical information about the plurality of documents generated by the statistical information generation unit 125 and uses the subject probability distribution And calculates the similarity with the selected reference seed.

Here, the seed clustering unit 150 may include a subject probability distribution calculating unit 151, a similarity calculating unit 153, and a similar document clustering unit 155 as shown in FIG. The subject probability distribution calculating unit 151 is configured to calculate a probability distribution in which each document belongs to each topic through the subject modeling algorithm with respect to the word statistical information generated by the statistical information generating unit 125. The similarity calculating unit 153 compares the subject probability distribution of the document selected as the reference seed with the subject probability distribution of the other comparison target document at the interested seed selection unit 140 and calculates the similarity between the document selected as the reference seed and the other comparison target document And the similar document clustering unit 155 is configured to cluster and provide documents similar to the reference seed based on the similarity calculated by the similarity calculating unit 153. [

As the subject modeling algorithm, any one of LDA (Latent Dirichlet Allocation), LSA (Latent Semantic Analysis), and PLSA (Probabilistic Latent Semantic Analysis) may be used, and the similarity measurement algorithm may be Hellinger distance, cosine similarity, and Jaccard similarity coefficient may be used. The above-described algorithms are well known in the art and will not be described in detail.

The interest document filtering unit 160 selects interest documents according to the similarity information for each document calculated by the seed clustering unit 150, and filters the interest documents. Here, the interested document filtering unit 160 may compare the calculated similarity information for each document with a predetermined threshold value to recommend an interest document similar to the reference seed, and may transmit an interest document including the reference seed to the document storage unit 10 Can be displayed separately on the stored population document data. Thereafter, if the document of interest is noise, it will be removed from the population, and if the document of interest is a core patent, it will be extracted from the population.

Thus, in the present invention, information entropy for each document is obtained using a word of interest, and a reference seed for interest is selected based on the information entropy for each document. Thereafter, documents similar to the reference seed are searched and clustered.

Here, the word of interest is used for document entropy calculation, the reference seed is selected based on the information entropy of each document, the TF-IDF value for each document is used to calculate the subject probability distribution for measuring the similarity, It is calculated by measuring the vector between the subject probability distribution and the subject probability distribution of other documents.

In addition, the Shannon entropy algorithm can be used to compute the information entropy of each document, and Latent Dirichlet Allocation, Latent Semantic Analysis or Probabilistic Latent Semantic Analysis algorithms can be used to obtain the subject probability distribution of each document, Hellinger distance, Cosine similarity, or Jaccard similarity coefficient algorithm can be used to measure liver similarity.

The overall operation of the interest document filtering system configured as above will be described in more detail with reference to FIG. 4 to FIG. Hereinafter, the case where the document of interest is a patent document will be described as an example.

4 is a flowchart illustrating a method of filtering interest documents according to an embodiment of the present invention.

First, the related patent document is retrieved from the patent database by receiving the keyword retrieval expression or patent number (application number, public number or registration number) input from the user. The retrieved patent document is loaded into the document filtering system 1, And may be stored in the storage unit 10. Then, the document filtering system 1 may receive a keyword for a word of interest and store it in the interest word storage unit 50 (S100). Here, the patent document is an excel-type patent data including a noise and a key patent when it is searched by a keyword, and the document includes bibliographic information (application number, filing date, public number, publication date, applicant, And summaries, claims, and citation information.

5, the user can select the patent loading (1) of the filtering system 1, load the retrieved patent population into the system 1, And stores the keyword (2) for the word of interest in the storage unit (10), and stores the keyword (2) in the interest word storage unit (50). At this time, the keyword of interest may be a key keyword or a noise keyword.

The word extracting unit 110 of the filtering system 1 extracts words contained in a plurality of documents stored in the document storage unit 10 and the statistical information obtaining unit 120 obtains a word included in each of the plurality of documents The word statistics information for a plurality of documents can be generated (S200).

In the embodiment, the word extracting unit 110 extracts a sentence included in the document through natural language processing on the document as shown in FIG. 6 (S210), and then, based on the analysis of parts of speech in the extracted sentence, (S220). Here, the word extracting unit 110 may remove a word included in a predetermined stopword list among the extracted words (S230).

Then, the statistical information obtaining unit 120 may calculate the occurrence frequencies of words extracted for each document through the extracted words (S240). The statistical information obtaining unit 120 may calculate an inverse document frequency (idf) indicating the number of documents including the corresponding word with respect to the total number of documents (S250).

Here, the appearance frequency (tf) of a word is a value indicating how often a particular word appears in the document, and can be calculated by adjusting the frequency of the word according to the length of the document. For example, the frequency measurement unit 121 may use the frequency of appearance as the frequency of occurrence of the word tf, but may also be calculated by the following equation (2). At this time, the word having the highest occurrence frequency in the document will have a value of '1', and other words will have a value smaller than 1.

&Quot; (2) "

Where t is any word, d is any document, w is any word in document d, and f (t, d) is the frequency of word t in document d.

In addition, the statistical information generation unit 125 calculates a reverse document frequency idf indicating how common an arbitrary word is included in all of a plurality of documents, And dividing the number of documents by the number of documents including the log scale.

For example, the inverse document frequency idf can be calculated by the following equation (3).

&Quot; (3) "

Here, t; Any word, d; Any document, D; Total number of documents, | d? D: t? D |; The number of documents containing the word t.

Next, the statistical information generating unit 125 may calculate a TF-IDF (TF-IDF) value by multiplying the word occurrence frequency tf and the inverse document frequency idf, May be acquired as word statistical information (S260).

For example, the TF-IDF value can be calculated by the following equation (4). Here, the reason for adding 'to the reverse document frequency is to prevent the inverse document frequency from being negative depending on the base of the log scale. When the base of the log scale is larger than 1,' is not added It is possible.

&Quot; (4) "

Here, TF (t, d) is the number of occurrences of word t in document d, | D | is the total number of documents, and | d∈D: t∈d |

The TF-IDF value becomes larger as the frequency of occurrence of a word in a specific document increases and as the number of documents including the word in the entire document increases.

The statistical information generation unit 125 can form a plurality of documents stored in the document storage unit 10 using the TF-IDF values of the extracted words for each document.

For example, when the number of documents stored in the document storage unit 10 is eight and the total number of words extracted from the document is seven, the statistical information generation unit 125 generates TF-IDF Value can be calculated and can be shaped as shown in Table 1 below.

[Table 1]

The operations of the word extracting unit 110 and the statistical information obtaining unit 120 are illustrated in FIG.

Meanwhile, the information entropy calculation unit 130 may calculate an information entropy (Shannon Entorpy) based on a word of interest stored in the interest word storage unit 50 (S300). The keyword of interest is related to the noise or core patent, and may be set by the user, and the weight may be set differently for each word of interest. Here, the weight means that weights can be applied to the frequency of occurrence of the concerned word.

Specifically, the information entropy calculating unit 130 may calculate information entropy for a word of interest using the frequency of words included in the word of interest and words not included in the words included in the document. The frequency information of each word can be used as the frequency information measured by the frequency measuring unit 121. In the user interface of FIG. 5, the 'seed patent calculation (3)' button may be an information entropy calculation command.

Here, the information entropy can be calculated by the following equation (5).

&Quot; (5) "

Here, n is the value of the required information amount of each document, k is the number of classes of interest, h _i is the probability of occurrence of each word of interest, The probability of occurrence is.

For example, if the keyword of interest is 3 in Stereo, Lithography and 3D, the frequency of words included in the word of interest and words not included in the word of interest in a specific document may be as shown in Table 2 below.

[Table 2]

Here, hi of Stereo in a specific document can be a value obtained by dividing 4, which is the appearance frequency of Stereo, by 50, which is the appearance frequency of the whole word. On the basis of this, the information entropy of the corresponding document may be '1.08' when the words of interest (k1 to k3) of Table 2 are applied to Equation (5).

The matrix of Table 2 above occurs for each document. In the present invention, words other than the word list of interest are treated as one word to form word groups (non-preferred words), thereby maximizing the composition of information amounts for the core words (words of interest).

Then, the interested seed selection unit 140 compares the information entropy of each document calculated and the preset information entropy to select a reference seed document (S400). If the information entropy of each document is smaller than a preset reference value, the document will be selected as the reference seed. If the information is extracted from the core patent, if the information entropy of each document is larger than a preset reference value, Will be selected as the reference seed.

The seed clustering unit 150 may extract and cluster at least one document similar to the reference seed selected by the seed seeding unit 140 (S500). The seed clustering unit 150 extracts at least one similar document similar to the reference seed. In order to extract a similar document, the seed clustering unit 150 may analyze the vector similarity between the documents using a subject modeling algorithm, have. If a plurality of reference seeds are selected by the interested seeding unit 140, the seed clustering unit 150 may extract at least one similar document for each reference seed document.

For example, as shown in FIG. 8, the seed clustering unit 150 applies a subject modeling algorithm to each word weight value (TF-IDF matrix) of each document generated through the statistical information generating unit 125, A probability distribution in which the document belongs to each subject can be calculated (S510). For example, the subject modeling algorithm may be a Latent Dirichlet Allocation (LDA) algorithm.

The LDA algorithm is a tool generally used in classification according to a topic of a document according to a known technology, and will be briefly described with reference to a MATLAB code as shown in FIG. Basically, the LDA algorithm begins on the premise that the document is a bunch of words, the document has a specific topic, and the topic is shared for each document. For example, assuming that there are eight documents as shown in FIG. 9A, and each document is made up of a total of 16 words, it is possible to display the document in color according to the appearance frequency of the word. The more green the word is, the higher the occurrence frequency of words. The more blue the word, the lower the occurrence frequency. In the case of the document No. 7 in FIG. 9A, only the words of the matrix (3, 4) are found to have a significantly high appearance frequency.

FIG. 9B shows a distribution on a topic, which has eight topics (Topic 1 to Topic 8) and shows which words have a topic. That is, the subject means a distribution of words. For example, in the case of topic 1, the frequency of appearance of words from the first to the fourth ((1,1) to (1,4)) is high. Therefore, applying the LDA to the weight of each word for each document shows a similar pattern as shown in FIG. 9B, thereby finding each topic.

Figure 9c shows the distribution of the subject for each document, the red being used to create the data, and the blue being found through the LDA. In other words, if you ignore the order of the subject in the x-axis, you can see that the LDA can eventually find the subject of each document similarly.

In addition to the Latitude Dirichlet Allocation (LDA), Latent Semantic Analysis (LSA) or Probabilistic Latent Semantic Analysis (PLSA) may be used as a subject modeling algorithm.

The number of topics (corresponding to the 'technical field') can be preset in the filtering system 1, and it has been found most appropriate to classify the number of topics into 8 to 10 over several tests. Therefore, as shown in Table 3 below, the subjects were first classified into 9 categories, and then LDA was applied to a plurality of documents and classified into respective subjects.

As shown in Table 3 below, the number of patent documents belonging to each topic derived as a result of the LDA and the main keyword information constituting each topic can be displayed. Using the main keyword information corresponding to each topic, Can be determined. For example, in Topic 1, it can be inferred that this is a technical clustering of adhesive particulate bonding.

[Table 3]

In this way, the subject probability distribution calculating unit 151 extracts the subject keywords and calculates the probability distribution belonging to each subject in each patent document as shown in Table 4 below.

[Table 4]

The similarity calculator 153 may calculate similarities between documents by performing similarity analysis between documents using a probability distribution belonging to each topic. The similarity may be calculated using Hellinger distance, cosine similarity, And a Jacquard similarity coefficient (S520).

For example, the similarity calculating unit 153 calculates the similarity between the reference seed selected by the interested seed selecting unit 140 and the other comparison target document by the Hellinger distance (H (P, Q)) of Equation (6) The degree of similarity can be calculated.

&Quot; (6) "

Here, i is the subject, k is the number of subjects, pi is the subject probability distribution of the reference seed document, and qi is the subject probability distribution of the comparison document.

The result value (H (P, Q)) calculated by the hellinger distance has a value between 0 and 1. The smaller the result value, the greater the degree of similarity between the two documents. The larger the result, It is small. Therefore, the final similarity value S (P, Q) is obtained by subtracting the result value H (P, Q) of the hellenistance to 1 as shown in the following Equation 7 so as to intuitively understand it easily It can also be used as the similarity value.

&Quot; (7) "

Table 5 below shows an example of the similarity matrix for each document calculated by Equations (6) and (7).

[Table 5]

Next, the similar document clustering unit 155 clusters similar documents similar to the reference seed based on the similarity information calculated by the similarity calculating unit 153 (S530). At this time, if there are a plurality of reference seeds, it is also possible to group similar documents for each reference seed.

Thereafter, the interest document filtering unit 160 may filter or remove the reference seed and its similar documents clustered through the seed clustering unit 150 based on a preset threshold value (S600). That is, the interest document filtering unit 160 may compare the similarity information between the reference seed and the comparison target document with a preset threshold value, and select interest documents similar to the reference seed to recommend. In addition, the interest document filtering unit 160 may separately display the interest document including the reference seed on the population document data stored in the document storage unit 10. [ Thereafter, if the document of interest is noise, it will be removed from the population, and if the document of interest is a core patent, it will be extracted from the population.

In other words, as shown in FIG. 10, in the filtering system 1, documents similar to the reference seed are clustered and provided. At least one reference seed (4) and similar documents (5) The information entropy value (⑦) of the document, and the similarity (distance: ⑧) with the reference seed. In FIG. 10, Patent No. 2002-018463, which is one of the reference seeds (4), and similar patents (5) similar to the reference patent are shown. Here, the similar patent (5) is only a patent having a similarity degree (8) lower than the preset threshold value (6). Of course, if another reference seed is selected, it will be re-extracted and displayed as patents similar to the reference seed.

The reference seed selection step (S400) to the similar document filtering step (S600) may be repeatedly performed as necessary to filter the core patent or noise step by step.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not intended to limit the invention to the particular forms disclosed. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

1: document filtering system 10: document storage unit
50: interest word storage unit 100: information processing unit
110: word extracting unit 120: statistical information obtaining unit
121: frequency measuring unit 125: statistical information generating unit
130: Information entropy computing unit 140: Interest seeding unit
150: seed clustering unit 151: subject probability distribution calculating unit
153: similarity calculating unit 155: similar document clustering unit
160: Interest document filtering unit

Claims (19)

(a) selecting at least one specific document of interest as a reference seed among a plurality of documents;
(b) analyzing a topic of each document based on the word statistical information about the plurality of documents to calculate a subject probability distribution, and calculating a probability distribution of the subject based on the subject's probability distribution, ; And
(c) selecting and filtering an interest document according to the calculated degree of similarity.
2. The method of claim 1, wherein the reference seed of step (a)
Wherein the predetermined interest is selected based on information entropy calculated through a plurality of document occurrence frequency probabilities.
The method of claim 2,
Wherein the reference seed is selected by comparing a range or a condition of information entropy of the calculated information entropy with a predetermined seed.
The method of claim 2, wherein the information entropy (n)
A method of filtering a document of interest generated by the following equation: Shannon entropy.
[Equation]

Here, n is the value of the required information amount of each document, k is the number of classified interest words, h _i is the probability of occurrence of each word of interest, The frequency of occurrence of word is probability.
The method of claim 1, wherein the word statistical information of step (b)
A method of filtering a document of interest, the method comprising: extracting words from a plurality of documents, respectively, and measuring the frequency of occurrence of the extracted words in the document.
The method according to claim 1,
Wherein the reference seed is a document of interest to be extracted, the query being one of noise and key data.
The method of claim 1, wherein the word statistical information of step (b)
Extracting words included in a plurality of documents;
Measuring a frequency of appearance of the extracted words in the document for each of the plurality of documents;
Calculating an inverse document frequency by dividing the number of documents in which the word is included in the total number of documents with respect to the extracted word; And
And multiplying the appearance frequency of the word by the inverse document frequency.
The method of claim 7, wherein extracting the word comprises:
Extracting a sentence included in the document through natural language processing on the document;
Extracting words corresponding to adjectives and nouns through analysis of parts of speech of the extracted sentences; And
And removing a word included in a predetermined stopword list among the extracted words.
The method of claim 1, wherein the step (b)
Applying a topic modeling algorithm to the word statistical information to calculate a probability distribution of each document belonging to each topic;
Calculating a degree of similarity between a reference seed and a document to be compared using a subject probability distribution of each document and a subject probability distribution of the document selected as the reference seed; And
And clustering and providing documents similar to the reference seed based on the calculated similarity.
The method of claim 9,
The subject modeling algorithm is any one of Latent Dirichlet Allocation (LDA), Latent Semantic Analysis (LSA) and Probabilistic Latent Semantic Analysis (PLSA), and the similarity is determined by Hellinger distance, Cosine Similarity, And a Jaccard similarity coefficient. The method of claim 1,
2. The method of claim 1, wherein the degree of similarity in step (b)
Wherein the probability distribution of each document belonging to the subject is calculated by applying the Hellinger distance (H (P, Q)) of the following formula.
[Equation]

Here, k is the distribution being subject, t is the number of topics, p _k are probability distributions of the subject based on the seed, q _k is the probability the subject of the comparison target document.
The method of claim 11,
Since the value of 'H' (P, Q) is equal to the probability distribution of documents, the final value (S (P, Q), similarity) is determined by the following equation How to filter documents of interest.
[Equation]

The method according to claim 1,
Wherein the interest comprises at least one of a core keyword and a noise keyword.
A reference seed selection unit for selecting at least one specific document as a reference seed among a plurality of documents as a reference seed;
A seed clustering unit for calculating a probability distribution of the subject through subject analysis of each document based on the word statistical information about the plurality of documents and calculating a degree of similarity with the selected reference seed using the subject classified probability distribution of each document; And
And an interest document filtering unit that selects and filters an interest document according to the similarity of each document calculated in the above.
[Claim 15] The method according to claim 14,
When selecting the reference seed, selects a preset interest based on information entropy calculated through a plurality of document occurrence frequency probabilities.
15. The method of claim 14,
A word extracting unit for extracting a word for each document from a plurality of collected documents;
A statistical information acquiring unit for measuring the occurrence frequencies of the extracted words in the document and acquiring the word statistical information using the appearance frequencies of the words and the inverse document frequencies; And
And an information entropy calculation unit for calculating information entropy of each document through a probability of appearance frequency in a document of a preset interest word with respect to the number of occurrences of all words in the document measured in the above step.
17. The system according to claim 16,
A frequency measurement unit for measuring a frequency of appearance of a word extracted through the word extraction unit; And
Calculating an inverse document frequency by dividing the number of documents including an arbitrary word in the total number of documents by the extracted word and multiplying the appearance frequency of the arbitrary word by the inverse document frequency to obtain word statistical information And a statistical information generating unit that generates a statistical information based on the statistical information.
15. The seed clustering unit according to claim 14,
A subject probability distribution calculating unit for calculating a probability distribution of each document included in each topic through a subject modeling algorithm with respect to the generated word statistical information;
A similarity calculating unit for comparing the subject probability distribution of the document selected as the reference seed from the interest seed selection unit with the subject probability distribution of the other comparison target document to calculate the similarity between the document selected as the reference seed and another comparison target document; And
And a similar document clustering unit for clustering and providing documents similar to the reference seed based on the similarity calculated by the similarity calculating unit.
15. The method of claim 14,
Wherein the interest includes at least one of a core keyword and a noise keyword.

Images