JP2006293616A - Document aggregating method, and device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To aggregate documents which deal with the same subject of the input documents and to multiply classify the documents respectively into groups of contained topics even when the documents describing the plurality of topics. <P>SOLUTION: Keywords for subsequent processing are limited by acquiring general words from an input document, choosing keyword candidates from the words according to conditions, such as proper nouns, and obtaining occurrence frequencies of the keyword candidates. Peripheral general words for each of the limited keywords are collected and summed up and the number of occurrence of the general words for each keyword is found, and if the number of occurrence is a predetermined number or more, an aggregation of documents is clustered as having topics with common keywords. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a document aggregation method, apparatus, and program, and more particularly to a technique for acquiring a document that deals with the same subject from a large number of documents and simultaneously acquiring a central keyword representing the acquired document group. The present invention relates to a document aggregation method, apparatus, and program for documents of broken style such as diaries and impressions.

  Clustering is a typical method for collecting documents with similar contents. For example, this is one of the clustering methods for expressing elements to be classified as vectors and aggregating elements having a small distance between vectors into the same cluster, and is widely used. When applied to document classification, a document vector obtained by weighting each word with TF-IDF (Term Frequency & Inverse Document Frequency) or the like is used (for example, see Non-Patent Document 1).

In addition, a technique has been proposed in which word weighting is improved and clustering accuracy is increased by using a document vector in which words having particularly high importance are emphasized.
Translated by Morio Onoe "Pattern Identification" New Technology Communications pp. 528−529

  Existing document classification techniques can be expected to have a certain level of accuracy for relatively well-written sentences such as news articles and papers. However, documents such as personal diaries and impressions not only have stylistic fluctuations and differences in document length, but may also include spoken language and dialects. Also, in the case of a document that includes individual subjectivity, such as an impression of a movie or a document that pleases a large record of a certain athlete, even if the document deals with the same topic, the viewpoint differs depending on the person who describes it. Often, the composition of the sentence and the characteristics of the words used vary.

  In addition, according to the conventional technology, a single document vector is given to a document in which multiple subjects are described in one document, and classification processing is performed. Therefore, the document vector may become ambiguous depending on the number of topics included. Classification accuracy may be reduced.

  For the above reasons, the existing document classification technique does not work effectively for miscellaneous documents such as personal diaries and impressions, and it is impossible to correctly form a set of documents that are similar in content. Also, it cannot be determined whether or not the obtained document set is actually composed of the same topic.

  The present invention has been made in view of the above points, and has dealt with the same subject matter for a large group of documents, particularly a broken document such as a diary or impression written by an unspecified number of people in an arbitrary style. To provide a document aggregation method, apparatus, and program capable of aggregating documents and capable of multiple classification into all the topic groups included even when documents describing a plurality of topics exist With the goal.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) aggregates documents describing the same topic among a large number of documents, and at the same time, a central keyword representing the aggregated document group, that is, a keyword serving as a topic axis. A document aggregation method in a document aggregation device to obtain,
A general word extraction step (step 1) for analyzing the input document to acquire words (hereinafter referred to as general words) in the document and appearance position information of the general words and storing them in the general word storage means;
A keyword extraction step (step 2) of acquiring a keyword as a keyword candidate that forms a topic axis in the input document and an appearance position of the keyword and storing the keyword in the keyword storage unit;
A keyword selection step (step 3) for selecting keywords based on the results of reading out and summing up the keywords from the keyword storage means;
Whether or not the document group including the keyword is a document group dealing with the same topic based on the result of totaling the general words appearing around the keyword selected in the keyword selection step with reference to the general word storage means And an aggregation determination step (step 4) for determining whether or not.

Further, the present invention (Claim 2) is a keyword selection step (Step 3).
A step of selecting frequently occurring keywords appearing in a plurality of documents is performed.

Further, according to the present invention (Claim 3), in the aggregation determination step (Step 4),
Aggregate general words that appear around the keywords selected in the input document,
A step of determining whether or not a document group including a keyword is a document group dealing with the same topic is performed based on the number of general words appearing at a high frequency of a certain ratio or more.

Further, the present invention (Claim 4), in the aggregation determination step (Step 4),
If the number of documents in which a certain keyword appears is X, and a general word acquired from the periphery of the keyword in the X documents appears in Y documents,
When the number of general words having a high ratio of Y to X is greater than a certain level, a step of determining that X document groups are the same topic document group having a keyword as a topic axis is performed.

Further, according to the present invention (Claim 5), in the aggregation determination step (Step 5),
When it is determined that a certain keyword is a topic axis, among the X documents including the keyword, only documents including a certain number of general words having a high frequency Y are included in the same topic document group having the keyword as an axis. The step which determines that it is is performed.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 6) aggregates documents describing the same topic among a large number of documents, and at the same time, a central keyword representing the aggregated document group, that is, a keyword serving as a topic axis. A document aggregating device to acquire,
General word storage means 203 for storing the extracted general word and the appearance position information of the general word;
Keyword storage means 205 for storing the extracted keyword and the appearance position information of the keyword;
A document analysis unit 202 that analyzes an input document to acquire words in the document (hereinafter referred to as a general word) and appearance position information of the general word, and stores the information in a general word storage unit;
A keyword extraction unit 204 that acquires a keyword as a keyword candidate that forms a topic axis in the input document and an appearance position of the keyword, and stores the keyword in the keyword storage unit;
A keyword totaling unit 206 that reads out and aggregates keywords from the keyword storage unit 205 and selects a keyword that is particularly likely to be a topic axis;
Based on the result of aggregating general words appearing around the keywords selected by the keyword selection unit 206 with reference to the general word storage unit 203, the document group including the keyword is a document group that handles the same topic. And an aggregation determination unit 207 for determining whether or not.

Further, the present invention (Claim 7) is the keyword aggregation means 206,
Means for selecting high-frequency keywords appearing in a plurality of documents are included.

Further, according to the present invention (claim 8), the aggregation determination means 207
Means for counting general words appearing around keywords selected in the input document;
Means for determining whether or not a document group including a keyword is a document group dealing with the same topic based on the number of general words appearing at a high frequency of a certain ratio or more.

Further, according to the present invention (claim 9), the aggregation determination means 207
Means for counting the number of documents X in which each keyword appears;
Means for acquiring a general word appearing at a high frequency Y around the keyword in the X documents when the number of keywords in the X documents is more than a certain value;
Means for determining that X document groups are the same topic document group having a keyword as a topic axis when the number of general words having a high ratio of Y to X is greater than a certain value.

Further, according to the present invention (claim 10), the aggregation determination means 207
When it is determined that a certain keyword is a topic axis, among the X documents including the keyword, only documents including a certain number of general words having a high frequency Y are included in the same topic document group having the keyword as an axis. Means for determining that the

The present invention (Claim 11) is a centralized system that aggregates documents describing the same topic among a large number of documents and represents an aggregated document group in a computer having at least a storage device and a CPU. Is a document aggregation program that simultaneously acquires various keywords, that is, keywords that become the topic axis,
A computer having, in a storage device, general word storage means for storing the extracted general word and appearance position information of the general word, and keyword storage means for storing the extracted keyword and appearance position information of the keyword, A program that functions as means according to claim 6.

  The document aggregation method, apparatus and program of the present invention first finds a keyword that can be a topic axis, and determines whether or not the keyword is an actual topic axis by aggregating words appearing around the keyword. It is a method to do. Because the judgment is based on the statistical information of the axis candidate keywords and the surrounding words (general words), even documents with broken style such as sentences without phrases, documents written in spoken language, documents with dialects, etc. It can be classified with high accuracy. In particular, because words that have meanings such as place names and names, and words that occur frequently, are used as the topic axis, the topic is topical, such as a diary or an extremely short document that describes individual actions. It is possible to exclude documents that are not present and aggregate only useful documents that are written with the same interest.

  In addition, even when there are descriptions of a plurality of topics in one document, the topics are aggregated using words appearing around the noted keyword. For example, a document in which the first half is written about topic A and the second half is written about topic B Are aggregated according to the topic they contain, such as grouping documents of topic A from the distribution of words appearing in the first half and grouping documents of group of topics B from the distribution appearing in the second half. Can be classified.

  Furthermore, not only the documents including the same classification are aggregated but also the keywords that are the basis of the aggregation can be acquired, so various responses such as document search and summary using the implementation result of the present invention are possible.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows a configuration of the document aggregation device according to the embodiment of the present invention. The document aggregation device shown in FIG. 1 is based on a document storage unit 201 that stores an input document, an input document analysis, a document analysis unit 202 that divides the document into elements constituting the document, and an analysis result of the document analysis unit 202. A general word storage unit 203 that records the acquired general word and a position in the document, a keyword extraction unit 204 that extracts a keyword that characterizes each document based on the analysis result of the document analysis unit 202, a keyword and a position in the document The keyword storage unit 205 to record, the keywords recorded in the keyword storage unit 205 are aggregated, and the keyword aggregation unit 206 that selects a particularly important keyword that can be the topic axis, appears in a document including the selected keyword. The aggregation determination unit 207 that aggregates general words to be used and finally determines whether or not the keyword is the axis of the topic, Keyword is determined to be, and a aggregated document storage unit 208 for storing the documents that make up the subject.

  FIG. 4 is an example of document data in the document storage unit according to the embodiment of the present invention.

  Each document shown in the figure is given a unique ID for distinction.

  The document analysis unit 202 acquires the document recorded in the document storage unit 201 and divides the document into morphemes that are grammatical minimum units constituting the document. After that, all or a part of morphemes or a compound word obtained by concatenating consecutive morphemes in the document is stored in the general word storage unit 203 as a “general word”. At this time, each general word is recorded together with the appearance position in the document.

  FIG. 5 is an example of a general word in the general word storage unit according to the embodiment of the present invention.

  In the example shown in the figure, the morpheme determined to be a noun as a result of the morpheme analysis processing is regarded as a general word and recorded together with the appearance position in the document. The appearance position is represented by the number of bytes obtained by counting the positions of the beginning and end of the general word from the beginning of the document. Since the amount of information for a Japanese double-byte character is 2 bytes, if all double-byte characters are used, assuming that the beginning of the document is 0 bytes, the first character begins at position “0” and the second character begins at position “ 2 ”and the third character starts at position“ 4 ”. In the example of the document with the document ID “0001”, the general word “Yesterday” appears at a position of 0 to 3 bytes, and the general word “movie” appears at a position of 16 to 19 bytes.

  The following description will be made using an example in which only nouns are registered as general words as shown in FIG.

  The keyword extraction unit 204 uses the processing result of the document analysis unit 202 to extract keywords that characterize each sentence. The purpose of this process is to acquire words that serve as an axis for collecting documents that deal with the same topic in subsequent processes. For example, in the case of a document written focusing on a certain athlete, the word that is the axis of topic aggregation is the name of the athlete, in the case of a document written about a merger between companies, the company name, etc. Mainly proper nouns.

  For keyword extraction, the existing technology “Hideki Isozaki“ Japanese Named Expression Extraction Using Meta-rules and Decision Tree Learning ”, IPSJ Transactions, Vol.43, No.5 pp.1481-1491 (2002) Can be used.

  This is based on the results of the morpheme analysis process, and based on the morpheme and its part-of-speech information, and information such as the type of characters such as hiragana, katakana, and kanji, the “organization name” such as the company name and sports team name from the input document , “Person name”, “place name”, “artifact name” such as product name and building name, “property”, etc.

  In the present embodiment, it is assumed that only the “organization name” and “person name” acquired by the above-described existing technology among these unique expressions are recorded in the keyword storage unit 205 as keywords.

  FIG. 6 shows an example of keywords in the keyword storage unit according to the embodiment of the present invention. A keyword is recorded in the keyword storage unit 205 together with the document ID and the appearance position from the top of the document. The appearance position is represented by the number of bytes obtained by counting the position of the beginning and end of the keyword from the beginning of the document.

  For example, it represents that the keyword “× Tanioo” extracted as a person name was located at the position of 44 to 51 bytes in the document with the document ID “0001”. Similarly, keywords extracted from documents with document IDs “0002” and “0004” are recorded. Assuming that no unique expression of “organization name” and “person name” appears in the diary-like document with the document ID “0003”, nothing is output for the section relating to the document ID “0003” as shown in FIG. Not.

  The keyword totaling unit 206 totals the appearance frequencies of the words recorded in the keyword storage unit 205, and collects a list of keywords that are particularly likely to be established as a topic axis and the IDs of documents that include the keywords. To send. Generally, when the same subject is handled in many documents, the subject can be called a topic. Therefore, a keyword that can be a topic axis is a keyword with a large number of appearing documents. Therefore, for example, keywords whose appearance frequency is a certain level or more as a result of aggregation are preferably sent to the aggregation determination unit 207.

  The aggregation determination unit 207 determines whether the keyword acquired from the keyword aggregation unit 206, that is, the candidate word for the topic axis is an axis for collecting documents. Processing performed by the aggregation determination unit 207 will be described.

  FIG. 7 is a flowchart of the process of the aggregation determination unit in one embodiment of the present invention.

  Step 601) A keyword is acquired from the keyword totaling unit 206, and a process for determining whether or not this keyword is a topic axis is started.

  Step 602) In a document including a keyword, general words appearing around the keyword are acquired and aggregated.

  For example, when the aggregation determination unit 207 acquires the keyword “○ mountain” acquired as the keyword of the person name, the ID (“0002” or “0004”) of the document including “○ mountain”, and the respective Appearance positions of keywords in the document database are acquired. As shown in FIG. 6, “○ mountain” appears at the 20th to 23rd bytes of the document with the document ID “0002”, and appears at the 28th to 31st bytes of the document with the document ID “0004”. Therefore, such information is acquired.

  In step 602, for the document with the first document ID “0002” including the keyword “○ mountain”, the appearance position of the keyword is checked against the general word list recorded in the general word storage unit 203, and the document ID “0002”. The general word appearing around the keyword “Yama” in the document is acquired.

  When a general word is acquired from a document including a keyword of interest, various acquisition ranges can be set. FIG. 8 is a diagram (part 1) for explaining an example of a range in which general words are collected. a is a document with the document ID “0002” including the keyword “○ mountain”. The keyword storage unit 205 records the appearance position of the keyword “Oyama”, and the general word storage unit 203 records the appearance position of general words such as “Senior House” and “Dissolution”.

  For example, in Example 1 shown in FIG. 8B, the collection range is a general word of up to two words before and after the keyword “Oyama”. “Dissolution” and “Prime Minister” appearing before the keyword “Mt. Yama”, and “decisions” and “general elections” appearing later are general words to be collected.

  Example 2 shown in FIG. 8C is an example in which the entire document including the keyword is used as the collection range. In this case, “general house”, “dissolution”, “prime”, “decision”, “general election”, “depressed” and all general words in the document with the document ID “0002” are collected.

  In the example of the document with the document ID “0002” shown in FIG. 8A, there are only three general words before “○ mountain”. Can only get 3 words before the keyword. In this case, all three words before the keyword and five words after the keyword may be acquired.

  In the description so far, only “document ID”, “general word (or keyword)”, and “appearance position” are recorded as in the examples of FIGS. 5 and 6, but morphological analysis performed by the document analysis unit 202 is performed. It is also possible to acquire the punctuation mark “.” In the document from the processing result, determine the sentence and sentence breaks, and simultaneously record which words and which words are extracted from the same sentence. If this is used, as shown in Example 3 shown in FIG. 8D, several sentences around the sentence including the keyword “Oyama” can be used as the general word collection range. Example 3 in FIG. 8D is an example in which only sentences including keywords are used as the collection range, and therefore only four words “House of Representatives”, “Dissolution”, “Prime Minister”, and “Decision” are acquired. The range may be expanded to the preceding and following sentences, such as “two sentences before and after the sentence.”

  In addition to the punctuation mark “.”, A period “.”, An exclamation mark “!”, A question mark “?”, Etc. may be considered as a sentence delimiter.

  On the other hand, an example of a range designation method when acquiring a peripheral general word by paying attention to the keyword “XX newspaper” acquired as an organization name is shown. FIG. 9 is an example (part 2) of the general word collection range in the embodiment of the present invention.

  According to the document aggregation method of the present invention, when a specific general word appears around a keyword of interest with a high probability, the document group is a document describing the same topic, and the keyword is a topic axis. It is determined that Assuming that the character string “(XX newspaper issue)” in the sentence shown in FIG. 9 is a fixed character string used for the headings of all news articles issued by XX newspaper publishers, The general word “issue” will appear with a very high probability. For this reason, all news articles issued by XX newspaper companies are mistakenly aggregated as being the same topic. The range designation example shown in FIG. 9 is one method for avoiding this. By not acquiring a general word that appears adjacent to the keyword of interest, it is possible to prevent the same issuer from mistakenly determining that the phrase used frequently and fixedly is the axis.

  Instead of the adjacent general word, the general word closest to the keyword of interest may be excluded from the acquisition target. In the case of FIG. 9, “inundation” and “issue” are excluded from the target.

  Not only the range setting method shown in FIGS. 8 and 9, but also various general word acquisition methods can be used. For example, when no general word is included in a range specified by a fixed length, such as one sentence before and after, the search range is expanded, and general words of M words before and after are acquired in order from the keyword of interest, etc. You may acquire in combination with a plurality of methods.

  Use either method to get a general word from the surroundings for all documents that contain the keyword of interest. For example, for the keyword “Yama,” “the House of Representatives” is 5 times, “Dissolution” is 7 times, etc. The number of documents from which general words are acquired is counted.

  Step 603) It is determined whether the search has been completed for all documents including the keyword of interest. If it is determined, the process proceeds to Step 604, and if not, Step 602 is repeated.

  Step 604) Using the general word frequency information aggregated for each keyword, it is determined whether or not each keyword is a phrase that forms the topic axis.

  The determination method will be described below.

  In performing document aggregation, whether or not a certain keyword is a topic axis is determined based on whether or not a specific word appears frequently around the keyword.

  FIG. 10 shows an example of a general word acquired from the vicinity of a keyword in an embodiment of the present invention. The figure shows an example of the top five words with the largest number of appearing documents as a result of totaling the general words in the vicinity by paying attention to each of the keywords “XX” and “XX newspaper”. “Oyama” appears in 28 documents in all documents, and general words obtained from the surroundings are listed in descending order of “common house” (appears in 25 documents), “dissolve” (appears in 22 documents), “ It is assumed that “Prime” (appears in 21 documents), “Tomorrow” (appears in 15 documents), and “General Election” (appears in 13 documents).

  At this time, the aggregation determination unit 207 counts the number of appearances of general words up to the top five listed here that exceeds 14 which is half of the number of appearance documents 28 of the keyword of interest “○ mountain”. In this example, the general words having the number of appearing documents exceeding 14 are the four words “the House of Representatives”, “Dissolution”, “Prime Minister”, and “Tomorrow”. In other words, four words out of the top five general words acquired from the periphery appear at a frequency higher than a certain level.

  Similarly, in the example of “XX newspaper” similarly shown in FIG. 8, there is no single general word that appears with a high frequency exceeding half of the number of appearing documents 16.

  The aggregation determination unit 207 counts the number of general words having a high frequency when satisfying the condition among the upper words (for example, the upper five words) of the general words arranged in the order of high frequency (four words in the example of “○ mountain”, “ In the example of “Newspaper”, 0 word) is examined, and when the ratio is 60% or more, for example, it is determined that the keyword of interest is the topic axis.

  As a result, “Yama” is determined to be a topical axis because 4 words satisfying the condition among the top 5 words correspond to 80% and exceed 60%. On the other hand, since there is no word that satisfies the condition for “XX newspaper” (0%), it is determined that it is not the topic axis.

  When documents containing a keyword are written on the same subject, the appearance pattern of words around the keyword is similar between documents. Make it easier to do. In other words, words appearing around the keyword are biased. On the other hand, a generality of a certain keyword is high, and in the case of a word used in documents in various fields, words appearing in the vicinity vary, and therefore it is difficult to increase the frequency even if the number of occurrences of each keyword is counted. In other words, the words appearing around the keyword are not biased. The above determination is based on this feature, and it is determined whether or not it is the axis of the topic by examining the ratio of words with a particularly high frequency among general words that frequently appear around the keyword of interest. It is automatically determined.

  The determination method focusing on this feature is not limited to the above example, and various applications are possible.

  For example, it is possible to construct a database in which the possibility that all two words are used at the same time is constructed in advance, and determine whether or not high-frequency general words acquired from the periphery are related to each other. In this case, in the example of FIG. 9, general words such as “the House of Representatives”, “dissolution” and “Prime Minister” acquired from the vicinity of the keyword “Yama” are words closely related to each other by referring to the database. It can be used as a judgment material that “○ mountain” is the topic axis. Conversely, “Arrest”, “Seismic Intensity”, and “Home Run” obtained from the vicinity of “XX Newspaper” have low correlation, and “XX Newspaper” is a highly general word. It can be determined that it should not.

  Alternatively, a database that quantifies the possibility of any two words or phrases appearing simultaneously in one sentence or one document may not be constructed in advance, but may be constructed dynamically from an input document. In this case, calculate the probability that high-frequency general words such as “House of Parliament” and “Dissolution” obtained from the keyword “○ Mountain” will appear simultaneously within a certain range, such as in the same sentence or in the same document. If the probability is high, it can be determined that it is the topic axis.

  As a result of the determination process, if it is determined that the keyword of interest is the topic axis, the process proceeds to step 605. If not, the process ends.

  Step 605) The keyword and the ID of the document including the keyword are recorded in the aggregated document storage unit 208.

  When outputting the ID of the topic composition document, instead of outputting the IDs of all the documents including the keyword determined to be the topic axis, out of the top five general words acquired from the periphery of the keyword, For example, the ID of only a document that includes three or more words may be output. In the example of the keyword “Yama” shown in FIG. 10, only a document including three or more words among “the House of Representatives”, “Dissolution”, “Prime Minister”, “Tomorrow”, and “General Election” is a topic centered on “Yama”. By outputting as a group of documents, it is possible to carefully select only documents that describe a specific topic such as “the general election accompanying the dissolution of the House of Representatives” without accidentally mixing documents that contain the keyword “○ mountain”. it can.

  As a result of the above processing, a keyword that is positioned at the center of the topic, such as the example of “Yama”, the surname of the prime minister, is not a topic, but a general keyword such as “XX newspaper” in the above embodiment. A group of documents constituting the topic is output at the same time.

  In the above-described embodiment, a word called a specific expression is a keyword candidate and a noun is a general word. However, the present invention is not limited to this, and various changes can be made.

  In addition, the operations of the document analysis unit 202, the keyword extraction unit 204, the keyword aggregation unit 206, and the aggregation determination unit 207 of the document aggregation device in the above embodiment are constructed as programs, and the computer having storage means such as a hard disk and a memory is built. It can be installed and executed, or distributed via a network.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to a technique for acquiring a common topic of a given document set.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. 1 is a system configuration example of a document aggregation device according to an embodiment of the present invention. It is an example of the document data of the document storage part in one embodiment of this invention. It is an example of the general word of the general word storage part in one embodiment of this invention. It is an example of the keyword of the keyword storage part in one embodiment of this invention. It is a flowchart of the process of the aggregation determination part in one embodiment of this invention. It is an example (the 1) of the general word collection range in one embodiment of this invention. It is an example (the 2) of the general word collection range in one embodiment of this invention. It is an example of the general word acquired from the keyword periphery in one embodiment of this invention.

Explanation of symbols

201 Document storage unit 202 Document analysis unit, document analysis unit 203 General word storage unit, general word storage unit 204 Keyword extraction unit, keyword extraction unit 205 Keyword storage unit, keyword storage unit 206 Keyword aggregation unit, keyword aggregation unit 207 Aggregation determination unit , Aggregation determination unit 208 Aggregated document storage unit

Claims (11)

Document aggregation method in a document aggregation apparatus that aggregates documents that describe the same topic among a large number of documents, and simultaneously acquires a central keyword that represents the aggregated document group, that is, a keyword that is a topic axis. Because
A general word extraction step of analyzing an input document to acquire words in the document (hereinafter referred to as general words) and appearance position information of the general words, and storing them in a general word storage means;
In the input document, a keyword extraction step of acquiring a keyword that is a keyword candidate that forms the topic axis and an appearance position of the keyword and storing the keyword in a keyword storage unit;
A keyword selection step of selecting keywords based on the results of reading out and summing up keywords from the keyword storage means;
Based on the result of totaling the general words appearing around the keyword selected in the keyword selection step with reference to the general word storage means, the document group including the keyword is a document group that handles the same topic. An aggregation determination step for determining whether or not,
A document aggregation method characterized by: In the keyword selection step,
Perform a step of selecting frequently occurring keywords that appear in multiple documents.
The document aggregation method according to claim 1. In the aggregation determination step,
Aggregating general words that appear around the keywords selected in the input document,
Performing a step of determining whether or not the document group including the keyword is a document group dealing with the same topic, based on the number of general words appearing at a high frequency of a certain ratio or more;
The document aggregation method according to claim 1 or 2. In the aggregation determination step,
If the number of documents in which a certain keyword appears is X, and a general word acquired from the periphery of the keyword in the X documents appears in Y documents,
A step of determining that the X document group is the same topic document group having the keyword as a topic axis when the number of general words in which the ratio of Y to X is high is more than a certain value;
The document aggregation method according to claim 3. In the aggregation determination step,
When it is determined that a certain keyword is a topic axis, among the X documents including the keyword, only documents including a certain number of general words having a high frequency Y are included in the same topic document group having the keyword as an axis. Performing the step of determining that
The document aggregation method according to claim 4. A document aggregation device that aggregates documents that describe the same topic among a large number of documents, and that also acquires a central keyword that represents the group of documents, that is, a keyword that is the axis of the topic,
General word storage means for storing the extracted general word and the appearance position information of the general word;
Keyword storage means for storing the extracted keyword and the appearance position information of the keyword;
A document analysis unit that analyzes an input document to acquire words in the document (hereinafter, referred to as a general word) and appearance position information of the general word, and stores the information in the general word storage unit;
In the input document, a keyword extraction unit that acquires a keyword that is a candidate for a keyword that forms the topic axis and an appearance position of the keyword, and stores the keyword in the keyword storage unit;
Keyword totalization means for reading out and totaling keywords from the keyword storage means and selecting keywords that are particularly likely to be topical axes;
Based on the result of totaling the general words appearing around the keyword selected by the keyword selection unit with reference to the general word storage unit, the document group including the keyword is a document group handling the same topic. Aggregation determination means for determining whether or not,
A document aggregating apparatus comprising: The keyword counting means is:
Including means for selecting frequently occurring keywords that appear in multiple documents,
The document aggregation device according to claim 6. The aggregation determination means includes
Means for counting general words appearing around the keywords selected in the input document;
Means for determining whether or not the document group including the keyword is a document group that handles the same topic, based on the number of general words that appear at a high frequency of a certain ratio or more,
The document aggregation device according to claim 6 or 7. The aggregation determination means includes
Means for counting the number of documents X in which each keyword appears;
Means for acquiring a general word appearing at a high frequency Y around the keyword in the X documents when the number of keywords in the X documents is more than a certain value;
Means for determining that the X document group is the same topic document group having the keyword as a topic axis when the number of general words in which the ratio of Y to X is high is greater than a certain value. ,
The document aggregation device according to claim 8. The aggregation determination means includes
When it is determined that a certain keyword is a topic axis, among the X documents including the keyword, only documents including a certain number of general words having a high frequency Y are included in the same topic document group having the keyword as an axis. Including means for determining that
The document aggregation device according to claim 9. At least in a computer having a storage device and a CPU, documents describing the same topic among many documents are aggregated, and a central keyword representing the aggregated document group, that is, a topic axis. A document aggregation program that also acquires keywords at the same time,
A computer having, in the storage device, general word storage means for storing the extracted general word and appearance position information of the general word; and keyword storage means for storing the extracted keyword and appearance position information of the keyword ,
11. A document aggregation program that functions as the means according to claim 6.

Images