JP4206961B2 - Topic extraction method, apparatus and program - Google Patents

Description

  The present invention relates to a topic extraction method, apparatus, and program, and in particular, in a situation where documents containing new information such as news articles classified by field can be obtained one after another, it has become a recent topic among documents in each field. The present invention relates to a topic extraction method, apparatus, and program for automatically extracting existing phrases.

  Information that can be obtained from media such as newspapers and television is increasing day by day, but the impact of the spread of the Internet is particularly significant, and flooded information is immediately buried. Under such circumstances, the information updated / added recently is likely to contain timely information such as current trends, interests, and new information. Therefore, by collecting and analyzing many documents with new creation times, it is possible to grasp recent trends and timely events.

As a technique for extracting a word representing a topic from a plurality of document information, there is a method using contextual rules and linguistic knowledge (see, for example, Patent Document 1). This is a method of detecting a topic candidate in consideration of the relationship between a topic after conversion and a topic before conversion, using a dictionary in which expressions used when the topic changes are learned.
JP-A-6-139276

  However, the above conventional method is a method that does not require domain knowledge, but before topic detection, it is a dictionary that collects expressions used at the time of topic change, such as “by the way”, “next”, “said”. Need to make. Also, if the target language changes, the dictionary needs to be rebuilt.

  SUMMARY OF THE INVENTION The present invention has been made in view of the above points. A topic extraction method that analyzes document data acquired one after another and extracts words that express topics and important words with high speed without requiring any knowledge. And an apparatus and a program.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) analyzes a large number of documents having creation time information, and investigates the frequency of appearance of the phrases that change with time for the phrases in the documents, thereby increasing the topical strength of the phrases. In the topic extraction method of quantifying and outputting
The phrase acquisition means sets a phrase having a creation time within the analysis time range from a recorded all-phrase accumulation database by combining the creation time of the document, an ID that uniquely identifies the document, and the phrase that appears in the document. A phrase acquisition step (steps 1 and 2) to be acquired;
Phrase topic degree calculation means, starting from the current time, the interval from Tplus to the past time (positive interval), and from the current time to the past time by Tplus, the interval from Tminus to the past time (negative interval) and interval analysis time range consisting of, negative and positive values with respect to the positive zone, the negative values for negative interval, in absolute value than the negative section of the integral value of the positive values in the positive section It is set so that the absolute value of the integral value of the value is larger , and the positive value is added to the appearance frequency of the word appearing in the positive interval and stored in the buffer, and the negative interval The negative value is added and stored in the buffer for the appearance frequency of the word that appears inside, and finally, the added positive value and negative value stored in the buffer are added together to add the negative value . A word topic level calculation step (step 3) for calculating the topic level;
An output step (step 4) in which the output means outputs the calculated topic level to the output word storage means;
Consists of
The word topic level calculation step (step 3) expands the analysis time range when the number of documents having the creation time within the analysis time range is less than a predetermined number.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 2) analyzes a large number of documents having creation time information, and investigates the frequency of appearance of the phrases that change with time for the phrases in the documents, thereby increasing the topical strength of the phrases. Is a topic extraction device that quantifies and outputs
An all-phrase storage database 13 that records a creation time of a document, an ID that uniquely identifies the document, and a phrase that appears in the document as a set;
A phrase acquisition unit 151 for acquiring a phrase having a creation time within an analysis time range from the all-phrases storage database 13;
Analyzed time range from the current time to Tplus past time (positive interval) and Tplus from the current time Tplus past time to the past time (negative interval) and then, a positive value with respect to the positive zone, the negative values for negative period, of the absolute value of the integrated value of the negative value in absolute value than the negative section of the integral value of the positive values in the positive section Is set to be larger, and the positive value is added to the appearance frequency of the phrase appearing in the positive section and stored in the buffer, and the appearance frequency of the phrase appearing in the negative section The phrase topic level is calculated by adding the negative value and storing the result in the buffer, and finally adding the positive value and the negative value stored in the buffer to calculate the topic level of the phrase. Calculating means 152;
Output means 153 for outputting the calculated topic level to the output phrase storage means;
Have
The phrase topic degree calculation means 152
When the number of documents having the creation time within the analysis time range is less than the predetermined number, means for expanding the analysis time range is included.

The present invention (Claim 3) is a topic extraction program for causing a computer to function as each means constituting the topic extraction apparatus according to claim 2 .

  As described above, according to the present invention, a collection of a large number of documents that are high in speed as a news article and that increases from time to time is automatically extracted, and words with high topicality are automatically extracted from the words in the documents. Therefore, it is possible to grasp recent trends and topics only by reviewing only high-topic words and phrases without looking through each article.

  Also, apply a filter that has a positive weight in the near past and a negative weight in the far past with respect to the appearance frequency of words that increase or decrease over time (the number of documents in which the word appears). Can detect the excitement of the current topic. By making the negative effect larger than the positive effect in the filter used here, it is possible to reduce the topic extraction error and capture a larger topic change.

  Further, even when an information source whose property is such that the number of documents increases or decreases depending on the time zone of the filter or the day of the week is used, it is possible to effectively detect a topic that has continued to the present time.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows the configuration of the hot topic extraction apparatus according to an embodiment of the present invention.

  Connected to the topic extraction device 10 shown in FIG. 1 is a document data buffer 11 for storing document data to be input to the device, and an output phrase recording device 16 for recording the words output by the device 10 and the topic level. ing.

  The topic extraction device 10 includes a document analysis unit 12, an all-phrase accumulation database 13, a trigger issue unit 14, and a phrase topic level calculation unit 15.

  The document data buffer 11 is assigned a unique document ID to a newly created document, and is input and recorded one after another along with creation time information, for example, like an article published on a news site on the Internet. In addition, it is desirable to target information sources in which documents containing new information are sequentially updated. In this case, it is preferable to monitor the update status of document data in the site and collect the update time as the document creation time. The document data buffer 11 is a queue that temporarily stores input documents, and waits for the document data stored therein to be sent to the document analysis unit 12.

  The document analysis unit 12 acquires document data stored in the document data buffer 11 for each document, and performs text analysis. Performs morphological analysis on the input text and breaks it down into parts of speech. At this time, if necessary, consecutive nouns may be combined to form a compound noun, and the compound noun may be treated as a single noun. As a word representing a topic, a noun (or compound noun) is more suitable than verbs such as “walking” and “instruct” and adjectives such as “blue” and “high”. Only compound nouns) are extracted from the document. In the following description, nouns (or compound nouns) are collectively referred to as “phrases”.

  In the all-word / phrase accumulation database 13, the document creation time and document ID acquired from the document analysis unit 12 and the words / phrases cut out from the document are recorded as a set.

  The trigger issuing unit 14 is a timer that generates a trigger for performing topic extraction processing at regular intervals. At each preset time interval, a topic extraction process start command is sent to the phrase topic level calculation unit 15. The phrase topic level calculation unit 15 receives the trigger from the trigger issuing unit 14 and starts topic extraction processing.

  The phrase topic level calculation unit 15 acquires the phrase information having a past creation time from the current time from the all phrase storage database 13. At this time, only the phrase having the creation time within the topic extraction target period determined by the phrase topic level calculation unit 15 is acquired, but at the same time, the number of documents is considered in consideration of the number of documents input to the document data buffer 11. When the number is small, the time range to be acquired (topic extraction target period) is expanded until a certain amount of document is obtained.

  The topic level is calculated for each of the acquired words and the result is output to the output word recording device 16.

  The topic level of the phrase w is a numerical value that indicates how suitable the phrase w is as a phrase that represents the current topic. The greater the topic level, the higher the topicality.

  The phrase topic level calculation unit 15 excludes a phrase that is regularly used as a phrase that expresses a topic and excludes it as a phrase that expresses a topic. It is preferable to configure the following functions as those having a function of giving a high degree of topic by highly evaluating the words that appear.

  The graph of FIG. 4 is a filter used when the phrase topic level calculation unit 15 calculates the topic level of the phrase. Starting from the current time, a positive value W + is taken from the present to Tplus until the past (referred to as a positive interval), and further, a negative value W− is taken from Tminus to the past (referred to as a negative interval). The feature is that the area S− of the negative section is larger than the area S + of the positive section.

  The phrase handled when the phrase topic level calculation unit 15 calculates the topic level is a phrase in which the creation time recorded in the all phrase storage database 13 is included in either the positive interval or the negative interval in FIG. , Words with a past creation time are not used for analysis. That is, the topic extraction target period described above is from the present to the past by Tplus + Tminus. However, when the number of input documents is small, the topic extraction target period is expanded. The method will be described in detail later.

  The word topic level calculator 15 adds the weight of the negative value W− to the appearance frequency during the period of Tminus and the weight of the positive value W + to the frequency of appearance during the period of Tplus for each word. The total value of all periods is the topic level. Therefore, the topicality of each word is determined by comparing the amount of appearance in the distant past (negative interval) with the amount of appearance in the near past (positive interval). Here, as the appearance frequency of the word or phrase, the number of appearances in the document may be used, but in the following, the number of documents including the word or phrase is treated as the appearance frequency. That is, if there is a document that includes the word even once, the number of documents in which the word appears is counted as one. This is because, for example, in news articles, proper nouns such as subject names and organization names appear only once in many cases, and the number of appearances in a document and the size of topicality are not directly linked. This is because, if the number of documents including the phrase is large, it is considered that the topicality is high. The value of a word that is always used at a constant frequency from the negative interval to the positive interval has a small value, and the topic level of a word whose amount of appearance has increased recently (the range of the positive interval) has a large positive value. In order to prevent a phrase having a small number of appearing documents from being extracted as a topic and to detect a larger topic change, the negative section area S− is made larger than the positive section area S +.

  When the calculation of the word topic level calculation unit 15 is finished, the result is recorded in the output word recording device 16. Since the phrase and its topic level are recorded in the output phrase recording device 16, the current topic can be known if a plurality of phrases are acquired as needed from phrases with a high topic level.

  Next, the actual operation in the topic extraction device 10 of the present invention will be described.

  FIG. 5 is a flowchart of processing of the document analysis unit according to the embodiment of the present invention.

  Step 101) The document analysis unit 12 checks whether or not the document data is stored in the document data buffer 11. If there is a document waiting to be processed, the process goes to Step 102, and there must be a document waiting to be processed. For example, the process proceeds to step 106, and the process is repeated until an end command is given, and the process waits for a document to be input.

  Step 102) The document analysis unit 12 acquires data for one document from the document data buffer 11.

  Step 103) Next, the document analysis unit 12 analyzes the acquired document. Document analysis is a process of dividing an input sentence into parts of speech.

  Step 104) As a result of the analysis, it is determined whether or not a word (noun or compound noun) to be subject to topic extraction is included in the document. For example, if the sentence is extremely next time and no nouns or compound nouns are included at all, or if nouns or compound nouns are not acquired at all due to an analysis error, the processing related to the acquired document is finished, and step 101 is performed again. Return to and process the next document.

  Step 105) When a phrase (noun or compound noun) is included, the phrase is recorded in the all-phrase accumulation database 13 together with the document creation time, and the process proceeds to Step 106.

  Step 106) Return to Step 101 and repeat the process until an end command is given.

  FIG. 6 is an example of data recorded in the all-words accumulation database in one embodiment of the present invention. In the example shown in the figure, the words “Iraq”, “Self-Defense Force dispatch”, “Today” and “Experience” extracted from the document with the document ID “doc00275” created at 16:45 on January 8, 2004 are “2004 / 1/8 16:45 ”and a document ID“ doc00275 ”are recorded. In the figure, the phrase “Tokyo” 401 and the phrase “Tokyo” 402 both have time information “2004/1/8/16: 48”, but the phrase “Tokyo” has a plurality of times with the same creation time. It is extracted from the documents (doc00277 and doc00278).

  FIG. 7 is a flowchart of topic extraction processing according to an embodiment of the present invention, and shows a flow of processing when topic extraction is performed using information recorded in the all-phrase accumulation database 13.

  Step 201) When the process is started, the trigger issuing unit 14 instructs the word topic level calculating unit 15 to start the process.

  Step 202) The word topic level calculation unit 15 provisionally determines the past (Tplus + Tminus) period as the analysis time range starting from the present. For example, when using a filter as shown in FIG. 8 in which the positive interval is 8 hours and the negative interval is 24 hours, the provisional analysis time range is from 32 hours before the present interval including the positive interval and the negative interval to the present.

  Step 203) Subsequently, the word topic level calculation unit 15 accesses the all word storage database 13 and checks the information of the creation time and the document ID, thereby counting the number of documents having the creation time included in the temporary analysis time range. To do.

  Step 204) If the number of documents having the creation time between 32 hours ago and the present is small and less than a certain number (for example, 100 documents), the process proceeds to Step 205. If so, the process proceeds to step 206. Here, when the number of documents in the past 32 hours exceeds a certain number (for example, 100), the temporary analysis time range of 32 hours becomes the actual analysis time range as it is.

  Step 205) If the number of documents having the creation time of the past 32 hours is less than a certain number (100), the period is further extended to the past until the number of documents in the period reaches a certain number (100). .

  The period from the creation time of the point where the number of documents reaches a certain number to the present is the analysis time range. When the number of documents exceeds 100 after going back to the document having the creation time 48 hours before the current time, the filter is 48 hours in total, with the positive interval being 12 hours and the negative interval being 36 hours as shown in the graph of FIG. The filter has a time width.

  The reason for expanding the analysis time range according to the amount of input documents is as follows.

  Document data used in the topic extraction device of the present invention is text written by a human. For this reason, fewer documents are created at night. Also, many exchange-related documents tend to be created during the daytime on weekdays. There are few nighttime creations of sports results in Japan. If the analysis time range is made constant regardless of the day of the week or time, the topic may not be extracted at all, so the analysis time range is dynamically changed.

  In the following description, processing when the analysis time range is expanded and the filter shown in FIG. 9 is used will be described.

  Step 206) It is determined whether there is an unacquired phrase within the analysis time range determined so far, and if there is, the process proceeds to Step 207.

  Step 207) The phrase topic level calculation unit 15 accesses the all phrase storage database 13 and acquires any phrase having a creation time within the analysis time range. When the current time is 18:00 on January 8, 2004, the analysis time range is after 9:00 on January 7, 2004, 32 hours ago. The phrase “Minister of Finance” with the creation time of “/ 8/16: 48” is eligible for acquisition.

  Step 208) Since the creation time “2004/1/8 16:48” is 12 minutes before the current time and is within the range of the positive section in FIG. 9, the weight is “1”. This value “1” is temporarily stored in the buffer in the phrase topic level calculation unit 15 as a provisional value of the topic level of the phrase “Minister of Finance”.

  The process proceeds to step 206, and the processing in steps 206 to 208 is performed for all words having the creation time of the analysis time range, that is, the creation time after 32 hours. When the phrase “Minister of Finance” appears at other times, the weight is determined from the creation time in the same manner as described above, and is stored in addition to the buffer in the phrase topic degree calculation unit 15.

  Step 209) In step 206, when the provisional values of the topic level of all the phrases are completed, the provisional value held for each phrase in the buffer in the phrase topic level calculation unit 15 is the topic level of each phrase. Therefore, this value is recorded in the output phrase recording unit 16.

  By the processing so far, the topic levels of all the words / phrases appearing in the past 48 hours which are the analysis time range are calculated and recorded in the output word / phrase recording device 16 as shown in FIG. Among them, “Iraq” with a big topic level “12.75” and “SDF dispatch” with “10.00” are words that strongly express the topic at 17:00 on January 8, 2004. is there.

  Step 210) If there is an end command, the process ends here. If there is no end command, the process proceeds to step 211.

  Step 211) If there is no end command, the trigger issuing unit 14 waits for a predetermined period of time set in advance, moves to Step 201, and issues a topic level calculation process start trigger again.

  By repeating the above processing, the latest topic is always output at regular intervals.

  In the present invention, the operation shown in FIGS. 5 and 7 is constructed as a program, installed in a computer used as a topic extraction device, executed by a control means such as a CPU, or acquired through a network. It is also possible to execute it.

  The present invention is not limited to the above-described embodiments and examples, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to a technique for automatically extracting words and phrases that have become a recent topic from a document.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a block diagram of the hot topic extraction apparatus in one embodiment of this invention. It is an example of the filter of the phrase topic degree calculation part in one embodiment of this invention. It is a flowchart of the process of the document analysis part in one embodiment of this invention. It is an example of the all phrase storage database in one embodiment of this invention. It is a flowchart of the topic extraction process in one embodiment of the present invention. It is an example of the filter which made the positive area 8 hours and the negative area 24 hours in one embodiment of this invention. It is an example of the filter which expanded the analysis time range in one embodiment of this invention. It is an example of the phrase and topic level stored in the output phrase recording device in one embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Topic extraction apparatus 11 Document data buffer 12 Document analysis part 13 All phrase storage database 14 Trigger issue part 15 Phrase topic degree calculation part 16 Output phrase memory | storage means, Output phrase recording device 151 Phrase acquisition means 152 Phrase topic degree calculation means

Claims (3)

A topic extraction method for analyzing a number of documents having creation time information and quantifying and outputting the topical intensity of the words by examining the frequency of appearance of the words that change with time for the words in the documents In
The phrase acquisition unit sets a phrase having a creation time within an analysis time range from a recorded all-phrase storage database by combining a document creation time, an ID that uniquely identifies the document, and a phrase that appears in the document. A phrase acquisition step to acquire;
Phrase topic degree calculation means, starting from the current time, the interval from Tplus to the past time (positive interval), and from the current time to the past time by Tplus, the interval from Tminus to the past time (negative interval) and said analysis time range a section consisting of a positive value with respect to the positive zone, the negative values for negative interval, said at negative section than the absolute value of the integrated value of the positive values in the positive section The absolute value of the negative integrated value is set to be larger, and the positive value is added and stored in the buffer for the frequency of occurrence of the word that appears in the positive interval. The negative value is added and stored in the buffer for the appearance frequency of the word that appears in the section, and finally, the added positive value and negative value stored in the buffer are added to the word. A word topic level calculation step for calculating the topic level of
An output means for outputting the calculated topic level to an output word storage means;
Consists of
The phrase topic degree calculation step includes:
A topic extraction method characterized by expanding the analysis time range when the number of documents having the creation time within the analysis time range is less than a predetermined number. A topic extraction device that analyzes a large number of documents having creation time information and quantifies and outputs the intensity of topicality of the phrases by investigating appearance frequency transitions of the phrases that change with time for the phrases in the documents Because
An all-phrase storage database that records a creation time of a document, an ID that uniquely identifies the document, and a phrase that appears in the document;
A phrase acquisition means for acquiring a phrase having a creation time within an analysis time range from the all phrase storage database;
The analysis time is a section consisting of a section from the current time to Tplus past time (positive section) and a section from the current time to Tplus past time to Tminus from the current time to the past time (negative section). in the range, positive values with respect to the positive zone, the negative values for negative interval, the absolute value of the integrated value of the negative value in the negative zone than the absolute value of the integrated value of the positive values in the positive section Is set to be larger, and the positive frequency is added to the appearance frequency of the phrase appearing in the positive section and stored in the buffer, and the appearance of the phrase appearing in the negative section The phrase topic that adds the negative value for the frequency and stores it in the buffer, and finally calculates the topic level of the phrase by adding the added positive and negative values stored in the buffer Degree calculation means;
Output means for outputting the calculated topic level to output phrase storage means;
Have
The phrase topic degree calculating means includes:
A topic extraction device comprising means for expanding the analysis time range when the number of documents having the creation time within the analysis time range is less than a predetermined number.   A topic extraction program for causing a computer to function as means for constituting the topic extraction device according to claim 2.

Images