JP4466334B2 - Information classification method and apparatus, program, and storage medium storing program - Google Patents

Description

  The present invention relates to an information classification method and apparatus, a program, and a storage medium storing the program, and in particular, from a large amount of Web information newly transmitted every day, from information obtained by a search keyword that is of interest to the user, The present invention relates to an information classification method and apparatus for extracting and classifying topic words, a program, and a storage medium storing the program.

  In recent years, information has been updated every day and provided to users. In particular, the update speed and amount of information on the Internet is remarkable. Under such circumstances, it is difficult to obtain the information that the user wants in accordance with the topic in line with the interests of the world.

  However, it is highly likely that the information that is updated every day includes many topics that the user is interested in, such as public interests, new events, events, and trends. Therefore, by analyzing these pieces of information, it is possible to extract topics that indicate the user's interest.

  In addition, when a desired content is obtained from a lot of information, there is a case where it is difficult to obtain desired information only by searching. Therefore, many techniques for automatically classifying the obtained information have been proposed.

  In topic extraction, a plurality of techniques for extracting from a plurality of document information have been proposed. For example, there is a technique for extracting a word representing the subject of a conversation from data in which contents of transmission sources of a plurality of speakers are documented. In this method, nouns are extracted from the documented data of conversation content by morphological analysis, and the word weight (possibility of expressing the topic) is determined based on their appearance frequency and appearance interval in the conversation flow. Technology. In this technique, words that are frequently used in one statement or words that have been used after being used for a while are evaluated as having high importance (hereinafter referred to as the first conventional technique) (for example, , See Patent Document 1).

  In addition, it is a technique for calculating the momentum of the information from message information such as conversations sent one after another, and extracting the strong phrase as a topic word (hereinafter referred to as the second conventional technique) ( For example, refer nonpatent literature 1).

  In the classification of information, it learns in advance what kind of information applies to a predetermined classification category, and when unknown information is input, it determines which classification the information applies to. A plurality of techniques for analysis have been proposed (hereinafter referred to as a third conventional technique) (for example, see Non-Patent Document 2).

  In addition, there is a technique in which a phrase vector is calculated based on the appearance frequency of words in a document, and the documents are clustered according to the similarity between documents. In addition, there is a technique for naming clusters according to the similarity of the word vectors (hereinafter referred to as the fourth conventional technique) (see, for example, Patent Document 2).

In addition, there is a technique for automatically classifying a result of inputting a keyword in a search engine. In some cases, an ontology is used for classification, or a classification category of information is originally attached.
Japanese Patent No. 2931553 Japanese Patent No. 3385297 Megumi Ishii et al., “Proposal of Topic Extraction Method Using Noun Phrase and Word Momentum” Information Processing Society of Japan Research Report-vol.2004-no.23, 2004-NL-160, pp.79-84 Nobuyoshi Ueda et al., “Probabilistic model / parametric mixed model of multi-topic text”, IEICE Transactions (D-II), Vol.J87-DII, No.3, March 2004, pp.872-883

  In the first prior art, the frequency of a certain word in one utterance and the topic being spoken in many cases are often not linked. Moreover, it can be said that a topic is also expressed when a relatively frequently used word is used intensively and frequently, but it is not suitable for extracting such a word.

  In the second prior art, since the momentum of information is calculated from message information such as a query, it is suitable for extracting topic words for a large number of documents that are spoken from completely different viewpoints. Absent.

  Further, the third prior art needs to determine categories to be classified in advance, and is not suitable for information in which topics change one after another.

  In the fourth prior art, a method of calculating a phrase vector in a document is used instead of clustering from the viewpoint of topic words that are taken up by many people. It is desirable to classify the topic words from the viewpoint of words / phrases handled in a large number of documents, not how much the words / phrases appear in the same document. Therefore, this method is not suitable for extracting topic words. In particular, in web pages such as blogs, news, diaries, etc., various topics are often described in one page, and it is difficult to extract topics in clustering using the appearance frequency of words in a document. Become.

  In addition, in the conventional technology that automatically classifies the results searched by the search engine, it is necessary to have an ontology or dictionary in advance or classify the categories. In such a case, the information is constantly updated, which is not suitable for extracting new trends or topics.

  The topic word in the present invention is a phrase that is taken up in many documents, and changes over time (a phrase that appears concentrated in many documents in the short term, many in the long term) For example). Among them, as an attractive topic word, it is desirable that the word has a strong impact and the contents can be immediately imaged.

  The present invention has been made in view of the above points, and analyzes document data or a large amount of document data obtained one after another based on a certain keyword in real time without prior learning. From the viewpoint of relevance and temporal freshness, it is possible to classify the topic data by classifying document data based on the topic word and then classifying the document data according to the topic word. An object of the present invention is to provide an information classification method and apparatus, a program, and a storage medium storing the program.

  FIG. 1 is an explanatory diagram of the principle of the present invention.

This onset Ming, from the document data obtained on the basis of a keyword, to extract the topic words, by classifying the said document data by the topic words, information classification for performing classification with distinctive involved more the keyword In the method
Using the specified keyword as a search keyword, the update date, the search result output order, the text (sentence), or a sentence including a part of the text is obtained from the search result, and the text or the text If a part of the text cannot be acquired, a data collection step of supplementarily collecting the text based on the publication location of the document data obtained from the search result and storing it in the document database; (Step 1);
Reading document data from the document database, extracting topic word candidates from the document data by referring to topic word rules using combinations of parts of speech stored in the topic word rule storage means, and storing them in the topic word database Topic word candidate extraction step (step 2)
Based on the topic word aggregation rules stored in the topic word aggregation rule storage means, a topic word aggregation step of aggregating the topic word candidate that is read from the topic word database (step 3),
In each topic word candidate topic word database, a topic word score calculating step (Step 5) calculating a topic word score by the relevant height and update time of the document data in the document data and the retrieval keyword including the topic word ,
A document classification step (step 6) for selecting a topic word from the relationship between the topic word score of each topic word candidate in the topic word database and the document data including the topic word candidate, and classifying the document data for each topic word. Done
In the topic word score calculation step (step 5),
The topic word scores of the topic word candidates are summed with the document topic word score obtained from the document data including the character string of the corresponding topic word candidate,
The document topic word score is determined from the search result output rank of one document data including the character string of the corresponding topic word candidate and the update date of the document data obtained from the search result.

In the above step 1, document data (update date required for the topic word extraction from the source of information that wants the user based on the keyword, the search result output order, before and after the sentence of the body or part of the body (summary, keywords, etc.) Etc.) from the search result, if the text or part of it cannot be obtained, the document publication location is also obtained from the search result as part of the document data, and the data is supplementally collected based on the document publication location. . When the entire text can be obtained or when a long sentence is obtained, the text before and after the search keyword is acquired in the document and used as the text. When multiple search keywords appear in a document, all the sentences in the vicinity are collectively treated as a text.

  In the topic word candidate extraction step of step 2 above, topic word candidates are extracted based on the topic word rule (after adjusting the fluctuation of the notation) based on the document data acquired in step 1 (perform morphological analysis). The topic word rules are stored in a topic word rule storage means or the like, and certain conditions (part of speech combinations, etc.) are stored so that phrases suitable as topic words can be extracted. Note that the adjustment of the shaking of the notation will be described in detail in the section of the embodiment.

  In the topic word aggregation step of step 3, words and phrases that can have the same meaning are aggregated based on topic word aggregation rules stored in topic word aggregation rule storage means or the like, which are certain conditions. I will do it. The topic word aggregation rule will be described in detail in the section of the embodiment.

In the time elapsed inspection step in step 4, to the topic word candidate that consolidates in step 3, to examine the temporal freshness of the topic words. A word or phrase that has been extracted as a topic word for a certain period T (a positive integer) or a word or phrase that is registered in the NG word list is excluded from the topic word candidates (also NG words) If there are words that are not in the list and are newly excluded from the topic word candidates, they are added to the NG word list). The step 4 is a step used particularly when the search is repeatedly executed periodically under the same conditions, and may be skipped when executed once.

  The topic word score calculation step of step 5 calculates a topic word score and outputs the document data acquired in step 1 for each topic word. The update date of the document data attached to the topic word candidate uses the search output order to calculate the topic word score. At this time, the topic word extracted from the document data with the newest update date and the highest search output rank has a high document topic word score for the document data. One topic word score is determined by summing up the document topic word scores.

  The document classification step of Step 6 can classify the topic words together with the document data in descending order of the topic word score and output them.

  FIG. 2 is a principle configuration diagram of the present invention.

According to the present invention (claim 1 ), a topic word is extracted from document data acquired based on a certain keyword, and is classified into the document data based on the topic word, thereby further classifying the characteristic relating to the keyword. An information classification device for
A topic word rule storage means 620 storing a topic word rule 22 using a combination of parts of speech as a condition for extracting a topic word;
A topic word aggregation rule storage unit 630 that stores topic word aggregation rules 23 for aggregating topic words that have the same meaning and are composed of combinations of parts of speech ;
As specified keyword search keyword, update date on which the document data, the search result output ranking, text (sentence) or obtained from the search results sentence containing a portion of the body, the body or the said body The data collection means 300 for supplementarily collecting the text based on the public location of the document data obtained from the search result and storing it in the document database 20;
Topic stored in the topic word database 21 by reading out document data from the document database 20, extracting topic word candidates from the document data with reference to the topic word rule 22 stored in the topic word rule storage unit 620. Word candidate extraction means 310;
Based on the topic word aggregation rule 23 topic word stored in the aggregation rule storage unit 630, a topic word aggregation unit 320 for aggregating topic word candidate that is read from the topic word database 21,
In each topic word candidate topic word database 21, a topic word score calculating means 340 for calculating a topic word score by the update time of the relevant height and the document data of the search keyword and document data including the topic word,
The topic word score of each of the topic word candidates in the topic word database 21 is narrowed down to a topic word candidate having a predetermined number or more, and the number of documents including the character string of the topic word candidate is narrowed to a topic word candidate having a predetermined number or more. the selected topic word, possess a document classification means 350 for the topic word score to classify the document data for each topic word in descending order, and
The topic word score calculation means 340
Means for summing the topic word scores of the topic word candidates with the document topic word score obtained from the document data including the character string of the corresponding topic word candidate;
Means for determining a document topic word score from a search result output rank of one document data including a character string of a corresponding topic word candidate and an update date of document data obtained from the search result;
Including

The present invention (Claim 2 ) is the information classification apparatus according to Claim 1 ,
The topic word aggregation means 320
As a topic word,
A phrase using a combination of parts of speech that satisfies the rules predefined by the topic word rule,
Terms not included in search terms,
A phrase that does not exist in the NG word list 24 in which phrases that are not suitable as topic words are stored;
Means for extracting phrases that are aggregated and selected as one topic word from phrases that can have the same meaning based on the topic word aggregation rule.

Further, the present invention (Claim 3), in the information classification apparatus according to claim 1 above, when repeatedly executing the information classification by topic words, the topic word is once extracted, a certain period of time T (positive integer ), The time passage inspection means 330 for excluding the phrase from the topic word is further included.

In the above SL data acquisition unit 300, an external input, or based on pre-set keyword, collects target document data from the search engine. Then, it is stored in the document database 20. When the entire text cannot be obtained, or when a long sentence is obtained, the text before and after the search keyword is acquired and used as the text. When multiple search keywords appear in a sentence, all the sentences in the vicinity are treated as a body.

  The topic word candidate extraction unit 310 acquires the document information acquired by the data collection unit 300 from the document database 20 and extracts the topic word candidate based on the topic word rule 22. The topic word rule 22 is stored in the topic word rule storage unit 620, and certain conditions (part of speech combinations, etc.) are stored so that words suitable as topic words can be extracted. Note that the adjustment of the shaking of the notation will be described in detail in the section of the embodiment. The extracted topic word candidates are stored in the topic word database 21. At this time, the topic word database 21 also stores related document information.

  The topic word aggregation means 320 can take the same meaning of the topic word candidates in the topic word database 21 based on the topic word aggregation rule 23 stored in the topic word aggregation rule storage unit 630 which is a certain condition. Concentrate words together. A phrase that continues to be extracted as a topic word for a certain period T (a positive integer) is excluded from the topic word candidates. The topic word aggregation rule 23 is indicated by a combination of parts of speech. Details of the rule will be described in detail in the section of the embodiment.

  The time passage inspection unit 330 acquires the topic word candidates extracted by the topic word aggregation unit 320 from the topic word DB 21 and inspects the temporal freshness of the words. In addition, words such as those registered in the NG word list 24 stored in the NG word list storage unit 640 are excluded from the topic word candidates. In addition, if there is a phrase that does not exist in the NG word list 24 and is newly excluded from the topic word candidates, it is added to the NG word list. The remaining topic word candidates are stored again in the topic word database 21. At this time, the topic word database 21 stores the aggregated topic word candidate information and associated document information.

  This means is used especially when the search is repeatedly executed periodically under the same conditions, and may be skipped when executed once.

  The topic word score calculation means 340 calculates the topic word score from the topic word database 21 using the update date and search output order of the document data attached to the topic word candidate. At this time, the topic word candidate having a new update date and extracted from the document data with the higher search output order has a higher document topic word score for the document data. One topic word score is determined by summing up the document topic word scores.

  The document classification unit 350 outputs topic words together with document data in descending order of topic word score. Further, the topic word score can be stored in the topic word database 21.

The present invention (Claim 4 ) is an information classification program for causing a computer to function as each means constituting the information classification apparatus according to any one of Claims 1 to 3.

The present invention (Claim 5 ) is a computer-readable recording medium storing the information classification program according to Claim 4 .

  This makes it possible to analyze document data or a large amount of document data that is acquired one after another based on keywords in real time, and from there, it is a topic in terms of relevance to keywords and temporal freshness. It is possible to provide a method of classifying search results by topic words, which can extract a phrase and classify document data according to the topic words, thereby enabling more characteristic classification related to the keywords.

  As described above, according to the present invention, a topic word is extracted mainly from a summary of a text obtained from a search result, a document around a search keyword, or the like, and therefore, the topic word may be deeply related to the keyword. Becomes higher. Therefore, it is possible to extract a topic word related to the keyword more deeply.

  In addition, some retrieved documents contain all matters in an exhaustive manner, so a normal classification device may determine that a document cannot be classified or has no special features. . In the present invention, it is possible to classify the vicinity of the keyword, that is, the topic of the desired information by obtaining a document in the vicinity of the search keyword or a summary of the document related to the keyword from the search result.

  In addition, it is possible to analyze the document data or a large amount of document data acquired and updated one after another based on keywords in real time without prior learning, and extract the topical phrases from there. It becomes.

  Also, by performing topic word scoring that takes into account the freshness of topic words based on the date of update of the document, etc., it is different from simple classification of search results, and based on topics that are seen from the results, that is, trends and user concerns, etc. Since the classification is performed, the classification is easy for the user to understand and reflects the current time.

  As described above, execute “extract topic words from surrounding documents related to keywords” and “topic words have higher topic word scores as the extracted documents are higher in terms and the output ranking of search results” Therefore, it becomes easier for the user to talk about the topic than the conventional technology, and the classification can be freely changed according to the fashion at the time.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, the outline of the present invention will be described.

  FIG. 3 is a diagram for explaining the outline of the present invention.

  The present invention analyzes document data acquired one after another or a large amount of document data based on a certain keyword in real time, and from the viewpoint of relevance to the keyword and temporal newness, The extracted words are classified, and the document data is classified according to the topic words, thereby classifying the keywords more specifically with respect to the keywords.

  In step 10, based on the keyword, the document data necessary for extracting the topic word from the information source desired by the user (update date, search result output order, text, or part of the text (summary, text before and after the keyword, etc.) Etc.) from the search result, and if the text or a part thereof cannot be acquired, it is acquired from the search result as a part of the document data at the publication location of the document such as URL (URI), and the document such as URL (URI) Supplementary data is collected based on the public location. When the entire text can be obtained, or when a long text is obtained, the documents before and after the search keyword are acquired and used as the text. When a plurality of search keywords appear in a sentence, all of the sentences in the vicinity are processed as a body text.

  In step 20, morphological analysis is performed based on the document data acquired in step 10, and after adjusting the notation fluctuation based on the topic word rule, topic word candidates are extracted using the topic word rule. The topic word rules are stored in a storage means such as a topic word rule storage unit, and certain conditions (such as combinations of parts of speech) are stored so that phrases suitable as topic words can be extracted. Here, the adjustment of the notation fluctuation means that, since there are various notation methods depending on the writer of the document, when candidates that can be taken as the same word are acquired, they are combined into one. For example, “dog” and “dog”, “precipitation” and “sedimentation”, “interface” and “interface”, “Daikoku Empire” and “Daikoku Empire”, “meeting”, “meeting”, “meeting”, “ Meeting ". Rules relating to the shaking of the notation are also stored in the topic word rules. If the topic word candidate is the same as or included in the keyword, the topic word candidate is excluded from the topic word candidate because it is not suitable as a topic word. For example, when the keyword is “Koshien Stadium” and “Koshien Stadium”, “Koshien”, and “Baseball Stadium” are acquired as topic word candidates, these three phrases are all included in the keyword as character strings. Can be removed from the candidate.

  In step 30, words that can be taken to have the same meaning are aggregated based on a certain condition: topic word aggregation rules stored in storage means such as a topic word aggregation rule storage unit. . The topic word aggregation rule is indicated by a combination of parts of speech or the like, and can be applied to the following cases, for example. If "programming language", "programming language", or the keyword is "dog", "dog trimming" and "trimming" can be judged as words that have the same meaning, so they are aggregated in either one be able to.

  In step 40, the temporal freshness of the topic word is examined for the topic word candidates collected in step 30. A phrase that continues to be extracted as a topic word for a certain period T (a positive integer) is excluded from the topic word candidates. Phrases that are registered in the storage means as an NG word list are removed from the topic word candidates. Further, if there is a phrase that does not exist in the NG word list and is newly excluded from the topic word candidates, the step 40 of adding it to the NG word list is not essential.

  Step 40 is a process that is used particularly when the search is repeatedly executed periodically under the same conditions, and will be described in the second embodiment of the present invention. In the case of execution only once, the process of the step may be skipped (first embodiment).

  In step 50, a topic word score is calculated, and the document data acquired in step 10 is output for each topic word. The topic word score is calculated using the update date of the document data attached to the topic word candidate and the search output order. At this time, a topic word extracted from document data with a new update date and a higher search output order has a higher document topic word score for the document data. One topic word score is determined by summing up the document topic word scores. The topic words are output together with the document data in descending order of the topic word score.

[First Embodiment]
An object of the first embodiment of the present invention is to extract topic words from a document from a Web page or the like acquired based on a certain keyword, and to cluster the documents for each topic word. This document should have some features for the purpose of extracting topic words. For example, based on Weblog, news, and new arrival pages of general Web pages, it is possible to extract topics that are exciting from time to time. In addition, if a page describes a specific matter, for example, a document explaining a history of France over a plurality of pages, a main topic of the history can be extracted. Details of the processing will be described below.

  FIG. 4 shows the configuration of the information classification device based on topic words in the first embodiment of the present invention.

  The information classification device based on topic words shown in FIG. 1 includes a computer 10, a document database (document DB) 20 connected to the computer 10 via a network 40, a topic word database (topic word DB) 21, and topic word rules. The topic word rule storage unit 620 stores the topic word aggregation rule storage unit 630 that stores the topic word aggregation rule 23.

  The computer 10 includes a memory such as a RAM, a ROM, and a magnetic disk, a CPU, a display unit 11 using a display, and an instruction input unit 12 such as a mouse and a keyboard, and is realized by a software program executed by the CPU. A data collection processing unit 500, a topic word candidate extraction processing unit 510, a topic word aggregation processing unit 520, a topic word score calculation processing unit 540, and a document classification processing unit 550 are provided.

  The document DB 20 stores meta information such as a URL (URI) of a content to be subject to topic word extraction, a title, a text sentence representing the content of the text (such as an outline), an update date and time, and an output order of search results.

  In the topic word DB 21, the number of topic words extracted from the content, the number of contents related to the topic word candidate, information about the content, the topic word selected by the topic word selection process, and the contents related to the topic word are stored. The number and content information, and the topic word candidates, but the topic word candidates aggregated during the topic word selection process are stored.

  The topic word rule 22 is stored in the topic word rule storage unit 620 and describes conditions (part of speech combination, etc.) for extracting the topic word. This rule can be freely added and changed.

  The topic word aggregation rule 23 is stored in the topic word aggregation rule storage unit 630 and describes conditions (such as combinations of parts of speech) for aggregating topic words that can have the same meaning. This rule 23 can be freely added and changed.

  The data collection processing unit 500, the topic word candidate extraction processing unit 510, the topic word aggregation processing unit 520, and the topic word score calculation processing unit 540 are the basis of the topic word extraction system configured as described above, and will be described below. It operates so that this invention may be realized by performing.

  The process will be described below.

(1) Data collection processing unit 500:
FIG. 5 is a flowchart of the data collection process in the first embodiment of the present invention.

Step 501) First, the data collection processing unit 500 acquires a topic word extraction target and parameters such as related keywords for which a topic word is to be extracted from an external input or a preset value. Various parameters can be used as necessary. For example,
(A) Extraction information target: Existing DB, general web pages on the Internet, web logs, news articles, etc .;
(B) Related keywords: Keywords related to information that the user wants to obtain, topics of particular interest, etc .;
It is. Since (a) performs a search based on the keyword of (b), it must be searchable data (data that can use a DB or a search engine). When the database is not used in (a), the search engine may use a publicly available search site, or can be realized by building and setting a search server in advance.

Step 502) A related keyword is sent as a search query to the acquired extracted information target. At this time, when the search destination is a search engine, there are sites where a search result can be obtained simply by adding a search keyword in a URL (URI) address and sending it to the search engine. For example, for gooblog search,
"Http://blog.goo.ne.jp/search/serch.php?status=select&tg=all&ts=goo&st=time&dc=10&dp=all&ts=all&MT=search keyword & da = all"
A search result (search result output order, page title, update date, text before and after the search keyword, etc.) is returned by simply sending an address such as However, this search keyword needs to be URL encoded (escaped). For example, “soccer” becomes “% A5% B5% A5% C3% A5% AB% A1% BC”.

  When acquiring data from an existing DB, it may be obtained from the entire text. Depending on the search engine, an outline of the content may be returned as a search result. In either case, the content is regarded as a body describing the content and will be handled below.

  When the text can be obtained or when a long sentence is obtained, the text before and after the search keyword is acquired in the document and used as the text. For example, it is possible to acquire 256 characters before and after the keyword in the middle, or to acquire one sentence before and after the document including the keyword. When multiple search keywords appear in a sentence, all the sentences in the vicinity are treated as a body.

  Step 503) The content information acquired in the above step is stored in the document DB 20.

(2) Topic word candidate extraction processing unit 20:
FIG. 6 is a flowchart of topic word candidate extraction processing according to the first embodiment of this invention.

  Step 601) First, the topic word candidate extraction processing unit 20 extracts target document information from the document DB 20.

  Step 602) Next, the document is subjected to morphological analysis. After that, the fluctuation of the notation of each word is adjusted. The target for the morphological analysis is the text and title of the document acquired by the data collection processing unit 500. In addition, the adjustment of the notation here means that the notation method varies depending on the writer of the document, and therefore when candidates that can be taken as the same word are acquired, they are combined into one. For example, “dog” and “dog”, “precipitation” and “precipitation”, “interface” “interface”, “Dainippon Empire” and “Daikoku Empire”, “meeting”, “meeting”, “meeting”, “ Meeting ".

  Step 603) Extract topic word candidates from the morphological analysis results. A topic word may have a different suitable phrase depending on a sentence or a field to be handled or a preference of a user. In the present embodiment, three types of topic words are used for the following reasons, and noun phrases are extracted, and the rules are described in the topic word rule 22.

  First of all, many people take up the content, and those that change over time (concentrated in the short term or spoken in the long term) are suitable as topic words. Among them, those that have a strong impact that attracts the user's interest and those that can give an image of the content at first glance have higher utility value.

  Proper nouns and new words are examples of words that have a strong impact. New words are often not registered in the morphological analysis dictionary and are therefore treated as a series of unknown words. Therefore, this time, we will adopt a series of unknown words in Katakana as proper nouns and a sequence of unknown words in alphabet as nouns.

  At first glance, words that can be used to imagine the content of the content include specific proper nouns that are specific to each other, a series of nouns that become more specific when they are consecutive, and words that sandwich nouns with the case particle “no”. It is done. Also, words that are too long to be imaged at first glance are not suitable because the user must read them aloud. Therefore, it can be said that it is desirable that it is below a certain length.

  From the above, even though it is treated as a topic word, a noun phrase in which a proper noun, a series of nouns, a noun + each particle "no" + noun is described with a certain length or less is adopted in this embodiment. . The regular expression is shown below. “?” Means “0” or the first occurrence of the previous word, “|” means to take or of the word before and after the symbol, and “{A, B}” It means the repetition of the previous word A times or more and B times or less. However, A and B are positive integers. Also, a: case particle “no”, n: noun, N: proper noun, p: noun prefix, s: noun suffix.

1. (P? (N | N) s?) {2,4}
2. (P? (N | N) s?) {1,3} a (p? (N | N) {1,3}
3. N
The result of the morphological analysis is connected, and the longest matching of the word strings corresponding to any of the above is acquired as the topic candidate. If this is followed, for example, “crime statement of terrorist organization”, “chiffon cake”, “US presidential election”, etc. can be acquired. In addition to this, it is also possible to extract a vocabulary that is easy to imagine a scene by using a noun phrase that employs a combination of adjectives and nouns or a combination of adjective verbs and nouns.

  The contents as described above are described in the topic word rule 22.

  Step 604) Topic word candidates that are all included in the keywords, that is, stories that are “keywords” ⊇ “topic word candidates” are excluded from the topic word candidates. For example, when the keyword is “Koshien Stadium” and “Koshien Stadium”, “Koshien”, and “Baseball Stadium” are acquired as topic word candidates, these three phrases are all included in the keyword as character strings. Can be removed from the candidate.

  Step 605) The topic word candidates extracted above are stored in the topic word DB 21. At this time, the document information of the base word extracted from the topic word is also stored in association with the topic word candidate. Further, when a topic word candidate extracted from each document becomes the same as another document, it is handled as one topic word candidate. Therefore, the topic word DB 21 does not have duplicate topic words in the entire document acquired at one time.

  The above processing is repeated for the number of topic word extraction target documents, and topic word candidates for each document are extracted.

(3) Topic word aggregation processing unit 520:
FIG. 7 is a flowchart of topic word selection processing in the first embodiment of the present invention.

  Step 701) The topic word aggregation processing unit 520 acquires a topic word candidate from the topic word DB 21.

  Step 702) If the following two types of topic words exist among the topic word candidates, they are collected into one. “DaE” and “DE”. Here, D and E are (p? (N | N) s?) {1, 3}, and a is the case particle "no". The regular expressions D and E are as described in the topic word candidate extraction processing unit (2) above. At this time, which topic word candidate is aggregated is not stipulated, but in this embodiment, it is aggregated into DaE. The relationship between the topic word candidate and the document is handed over to the aggregated topic word candidate. For example, when “programming language” and “programming language” appear as topic word candidates, they can be collected into “programming language”. Then, the document data including the “programming language” is succeeded to the “programming language” (that is, it is treated as the document data including the “programming language”).

Step 703) Further, if the acquired topic word candidate is a phrase including a keyword in part, the following processing is performed. Among the topic word candidates,
1. [1] KD
[2] KaD
[3] D
When the candidates [1] and [2] coexist, they are collected into [2].

  If any of the candidates [1] [3], [2] [3], [1] [2] [3] coexist, they are collected into [3].

2. [1] DK
[2] DaK
[3] D
When the candidates [1] and [2] coexist, they are collected into [2].

  If any of the candidates [1] [3], [2] [3], [1] [2] [3] coexist, they are collected into [3].

However,
a: Case particle "no"
K: Keyword D: (p? (N | N) s?) {1, 3}
Suppose that

  For example, in the case where the keyword is “dog”, “dog trimming” and “treeing” can be collected as “dog trimming” when “dog trimming” and “trimming” are raised as topic word candidates. However, the relationship between the topic word candidate and the document is inherited to the aggregated topic word candidate.

  Step 704) The aggregated topic word candidates are associated as accessory information of the aggregated topic word candidates and stored in the topic word DB 21. At this time, related document information is also stored together.

  The above is processed for all topic word candidates. With this process, the aggregation of topic word candidates is completed.

(4) Topic word score calculation processing unit 540:
FIG. 8 is a flowchart of the topic word score calculation process and the document classification process in the first embodiment of the present invention. Steps 801 to 804 are processes performed by the topic word score calculation processing unit 540. Steps 805 to 808 are processes performed by the document classification processing unit 550 described later.

  Step 801) The topic word score calculation processing unit 540 acquires, from the document DB 20, document information including a character string of a certain topic word candidate or a topic word candidate of the attached information in the topic word DB 21.

  Step 802) If the document is obtained from the search result and there is update date information of the document, a document topic word score for the document is calculated based on the update date information of the document and the search result output order information. For example, there is a method of calculating a document topic word score by adding a date score and a search result score as follows. For example, the date score may be “1” if it is within 24 hours from the time when the document was acquired, and if it is after that, a value obtained by dividing “1” by the number of days passed may be used as the date score. A scoring method such as that used in search result output rank information or a search engine may be used.

  For example, the search result score may be a value obtained by dividing “1” by the output rank.

  Step 803) If the document is obtained from the search result and there is no date information of the document, the processing of step 802 is skipped, and the document topic word score for the document is calculated based on the search result output rank information. For example, a method in which “1” is divided by the output order may be used.

  Step 804) The document topic word scores of all the documents including the topic word candidates or the character strings of any of the aggregated topic word candidates that are the associated information are summed, and the value is set as the topic word score. FIG. 9 shows a document example and an output example in the first embodiment of the present invention. According to FIG. 9, “Document A” includes “Asia Cup” and “Japan” as topic word candidates, “Document B” includes “World Cup Qualifier”, “Japan”, and “Korea”, and “Document C”. If there is “Asia Cup” as a topic word candidate, and each document topic word score is as shown in FIG. 9A, the topic word score is shown in FIG. 9B. As shown. Then, the topic word score is stored in the topic word DB 21 as accessory information of the topic word candidate.

(5) Document classification processing unit 550:
The operation of the document classification processing unit 550 will be described using the flowchart of FIG.

  Step 805) When the document is acquired from the search result, the document classification processing unit 550 performs the processing of the above-described steps 801 to 804 on all the topic word candidates. Then, the topic word candidates are rearranged in descending order of the topic word score.

  Step 806) A topic word candidate whose topic word score is lower than the constant Y and whose topic word candidate or the number of documents including the character string of the topic word candidate (attached information) aggregated thereto is less than the constant M. , Remove from candidates and set the rest as topic words. At this time, Y and M are positive integers.

  Step 807) If there is a sentence that does not contain any topic words or character strings of topic word candidates (attached information) aggregated thereto, it is stored in the topic word DB 21 as “not applicable”.

  Step 808) From the document DB 20 and the topic word DB 21, the topic words and the documents related to the topic words are output in a list format in descending order of the topic word score or in descending order of the number of documents. An example of the output is shown in FIG.

  This information can be displayed in a map-like format instead of a list format. Various display methods such as two-dimensional and three-dimensional can be applied.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

[Second Embodiment]
In the second embodiment of the present invention, the latest RSS is acquired from the news provided in RSS format or the WebBlog portal, the topic word is extracted from the information obtained from the RSS, and the content for each topic word is extracted. Is intended to be clustered. Since the information provided in the RSS format is updated at any time, this information can be updated periodically, new arrival information can be acquired at any time, and new topic words and their contents can be provided to the user.

  Some sites return search results in RSS format when a search keyword is transmitted to a certain URI. Therefore, it is also possible to obtain information related to the keyword to be examined by periodically accessing this site. Some sites also provide information related to a keyword in the RSS format whenever a keyword is registered. For this reason, it is possible to follow information with keywords that are of interest even among new arrivals. For example, it is possible to grasp what is being talked about in “soccer”. Moreover, even if it is information that cannot be acquired based on a keyword, by using the present invention and a search engine as a set, information that is updated at any time is acquired in RSS format, content information is searched by keyword, It is also possible to use only the information of interest as the input source.

  Details of the processing will be described below.

  FIG. 10 is a configuration diagram of an information classification device based on topic words according to the second embodiment of the present invention.

  The information classification device based on topic words shown in FIG. 1 includes a computer 10, a document database (document DB) 20, a topic word database (topic word DB) 21, and topic word rules 22 connected to the computer 10 via a network 40. The topic word rule storage unit 620 stores the topic word aggregation rule storage unit 630 that stores the topic word aggregation rule 23, and the NG word list storage unit 640 that stores the NG word list 24.

  The computer 10 includes a memory such as a RAM, a ROM, and a magnetic disk, a CPU, a display unit 11 using a display, and an instruction input unit 12 such as a mouse and a keyboard, and is realized by a software program executed by the CPU. A data collection processing unit 500, a topic word candidate extraction processing unit 510, a topic word aggregation processing unit 520, a time course inspection result processing unit 530, a topic word score calculation processing unit 540, and a document classification processing unit 550 are provided.

  The document DB 20 stores meta information such as a URL (URI) of a content to be extracted from a topic word, a title, a text sentence representing a body content (such as an outline), and an update date and time.

  In the topic word DB 21, the number of topic words extracted from the content, the number of contents related to the topic word candidate, information about the content, the topic word selected by the topic word selection process, and the contents related to the topic word are stored. The number, the time information at which the content information is acquired, the information on the content, and the topic word candidates that have become topic word candidates but are aggregated during the topic word selection process are stored.

  The topic word rule 22 is stored in the topic word rule storage unit 620 and describes conditions for extracting a topic word. This rule can be freely added and changed.

  The topic word aggregation rule 23 is stored in the topic word aggregation rule storage unit 630 and describes conditions for aggregating topic words that have the same meaning. This rule can be freely added and changed.

  The NG word list 24 is a list in which phrases that are not suitable as topic words are collected. The NG word list 24 can be simplified and processed without preparation.

  The data collection processing unit 500, topic word candidate extraction processing unit 510, time course inspection processing unit 530, topic word aggregation processing unit 520, and topic word score calculation processing unit 540 are the topic word extraction system configured as described above. By performing the processing described below, the operation is performed to realize the present invention.

  The process will be described below.

(1) Data collection processing unit 500:
FIG. 11 shows a flowchart of data collection processing in the second embodiment of the present invention.

Step 1001) The data collection processing unit 500 first acquires a content group described in the RSS format from the URI of the RSS providing site registered in advance based on the designated keyword. For example, for gooblog search,
http://blog.goo.ne.jp/search/search.php?status&tg=all&st=time&dc=50&dp=all&bu=&ts=all&MT="Search Key "& da = all & rss = 1 & fr = 1
By simply sending an address like this, 50 search results using the “search key” as a search keyword are returned in RSS format. However, this search key needs to be URL encoded (escape). “Soccer” becomes “% A5% B5% A5% C3% A5% AB% A1% BC”. It is also possible to register a plurality of sites (URIs) in advance. When a plurality of sites (URIs) are registered, it can be realized by repeating the flow of the data collection processing unit 500 for the number of registrations.

  FIG. 12 shows an RSS description example in the second embodiment of the present invention.

  Step 1002) Regarding the acquired information, information sandwiched between <item> and </ item> tags is regarded as one content. Then, <title>: title, <link>: link information, and <description>: text (content outline, etc.) are acquired.

  Step 1003) “description” may or may not be described. In addition, there are cases where there are too few character strings to handle “description” for analysis. For example, if the character string is less than one sentence, it can be said that it is not suitable for extracting a topic word. As described above, when the text cannot be acquired, the text of the content is directly acquired based on the link information. At this time, there are various ways of acquiring the text. For example, in the case of an html file, it is possible to obtain the text in which the longest sentence surrounded by <p> tags is described. If a long sentence can be obtained, the sentences before and after the search keyword are acquired and used as the text. For example, 256 characters may be acquired before and after the keyword in the middle, or one sentence before and after the document including the keyword may be acquired. When multiple search keywords appear in a sentence, all the sentences in the vicinity are treated as a body. In either case, the content is regarded as a body describing the content and will be handled below.

  Step 1004) The content information obtained in the above steps, that is, the title, link information, text, and RSS acquisition time are set as a set of content and stored in the document DB 20. In RSS, a plurality of pieces of content information are often described. For this reason, information for the number of contents is acquired and stored in the document DB 20.

(2) Topic word candidate extraction processing unit 510:
FIG. 13 shows a flowchart of topic word candidate extraction processing in the second exemplary embodiment of the present invention.

  Step 1101) The topic word candidate extraction processing unit 510 first extracts the body and title of the content acquired from the latest RSS from the document DB 20. This is a content document. Only the text may be used as a document.

  Step 1102) Next, the document is subjected to morphological analysis.

  Step 1103) Extract topic word candidates from the morphological analysis results. The topic word may have a different suitable phrase depending on the sentence and field to be handled and the preference of the user. In the present embodiment, three types of topic words are used for the following reasons, and noun phrases are extracted, and the rules are described in the topic word rule 22.

  First of all, many people take up the content, and those that change over time (concentrated in the short term or spoken in the long term) are suitable as topic words. Among them, those that have a strong impact that attracts the user's interest and those that can give an image of the content at first glance have higher utility value.

  Examples of words that have a strong impact include proper nouns and new words. New words are often not registered in the morphological analysis dictionary and are therefore treated as a series of unknown words. Therefore, this time, we will adopt a series of unknown words in Katakana as proper nouns and a sequence of unknown words in alphabet as nouns.

  At first glance, words that can be used to imagine the content of the content include specific nouns that are specific to themselves, a series of nouns that become more specific by being continuous, or words that are sandwiched between “no” of case particles between nouns, etc. Can be given. Also, too long a word that can be imaged at a glance is not suitable because the user has to read it out. Therefore, it is desirable that the length is not more than a certain length.

  From the above, even though it is treated as a topic word, a proper noun, a series of nouns, a noun + case particle `` no '' + noun, a noun phrase described below a certain length is adopted in this embodiment. To do. The regular expression is shown below. “?” Means 0 or 1 occurrence of the immediately preceding word, “|” means to take or of the word before and after the symbol, and “{A, B}” Is repeated from A to B times. However, A and B are positive integers. Also, a: case particle “no”, n: noun, N: proper noun, p: noun prefix, s: noun suffix.

1. (P? (N | N) s?) {2,4}
2. (P? (N | N) s?) {1,3} a (p? (N | N) s?) {1,3}
3. N
The result of the morphological analysis is connected, and the longest matching of the word string corresponding to any of the above is acquired as the topic word candidate. In addition to this, it is also possible to extract a vocabulary that is easy to imagine a scene by using a noun phrase that employs a combination of adjectives and nouns or a combination of adjective verbs and nouns.

  The contents as described above are described in the topic word rule 22.

  Step 1104) Topic word candidates that are all included in the keyword, that is, words that are “keyword” ⊇ “topic word candidate” are excluded from the topic word candidates.

  Step 1105) The topic word candidates extracted above are stored in the topic word DB 21. At this time, the topic word candidate is also stored in association with the document information of the basis from which the topic word is extracted. Further, when a topic word candidate extracted from each document becomes the same as another document, it is handled as one topic word candidate. Therefore, the topic word DB 21 does not have duplicate topic words in the entire document acquired at one time.

  The above processing is repeated for the number of topic word extraction target documents, and topic word candidates for each document are extracted.

(3) Topic word aggregation processing unit 520:
FIG. 14 is a flowchart of topic word aggregation processing according to the second embodiment of this invention.

  In the flowchart shown in the figure, steps 1201 to 1203 and 1205 are the same as the operations of steps 701 to 704 in FIG. 7 in the first embodiment described above.

  In the present embodiment, if there is an NG word list 24 in the topic word aggregation processing unit 520, it is checked against the list and the corresponding topic word candidate is deleted from the candidates (step 1204).

(4) Time-lapse inspection processing unit 530:
FIG. 15 shows a flowchart of the time lapse inspection process in the second embodiment of the present invention.

  Step 1301) The time-lapse inspection processing unit 530 stores all the topic words in the topic word DB 21 from the time TN at which the latest content information is acquired to the time TP that is back by a constant T (a positive value representing time). The time when the content information was acquired is acquired.

  Step 1302) The topic word from time TP to TN (including the topic word candidate that is auxiliary information of the topic word) and the topic word candidate acquired at TN are collated, and they are continuously extracted even when the time changes. If there is a topic word or a topic word candidate, the phrase is deleted from the topic word candidates of TN. If the NG word list 24 is present, the topic word candidate that is the topic word or attached information is added to the list.

  By this processing, it is possible to prevent a risk that a word unsuitable as a topic word is extracted, for example, when information unrelated to the text is misrecognized and the same document is always extracted.

(5) Topic word score calculation processing unit 540 and document classification processing unit 550:
Since this processing is the same as the topic score calculation processing unit 540 and the document classification processing unit 550 in the first embodiment, description of these processing units is omitted.

  The information finally obtained can be displayed not in a list format but in a map-like format. About a display method, it can apply variously, such as two dimensions and three dimensions. In addition, when the document is repeatedly acquired at regular intervals, the topic words are stored in time series, so that the transition of the topic words can be seen. This transition is made possible by acquiring and outputting a topic word from the topic word DB 21 at each time when RSS is acquired.

  In addition, the operation | movement of each component of the information classification apparatus of said 1st Embodiment and 2nd Embodiment is built as a program, and it installs in the computer utilized as an information classification apparatus, or via a network Can be distributed.

  In addition, the constructed program can be stored in a hard disk device connected to a computer used as an information classification device, a portable storage medium such as a flexible disk or a CD-ROM.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to a technique for automatically extracting words that are the latest topics from information that is updated daily, such as news and blogs, and classifying the information.

It is a principle explanatory view of the present invention. It is a principle block diagram of this invention. It is a figure for demonstrating the outline | summary of this invention. It is a block diagram of the information classification device by the topic word in the 1st Embodiment of this invention. It is a flowchart of the data collection process in the 1st Embodiment of this invention. It is a flowchart of the topic word candidate extraction process in the 1st Embodiment of this invention. It is a flowchart of the topic word aggregation process in the 1st Embodiment of this invention. It is a flowchart of the topic word score calculation process and document classification process in the 1st Embodiment of this invention. It is an example of a document and an example of output in a 1st embodiment of the present invention. It is a block diagram of the information classification device by the topic word in the 2nd Embodiment of this invention. It is a flowchart of the data collection process in the 2nd Embodiment of this invention. It is an RSS description example in the 2nd Embodiment of this invention. It is a flowchart of the topic word candidate extraction process in the 2nd Embodiment of this invention. It is a flowchart of the topic word aggregation process in the 2nd Embodiment of this invention. It is a flowchart of the time passage inspection process in the second embodiment of the present invention.

Explanation of symbols

10 Computer 11 Display unit 12 Instruction input unit 20 Document DB
21 Topic DB
22 topic word rule 23 topic word aggregation rule 24 NG word list 40 network 300 data collection means 310 topic word candidate extraction means 320 topic word aggregation means 330 time course inspection means 340 topic word score calculation means 350 document classification means 500 data collection processing unit 510 topic word candidate extraction processing unit 520 topic word aggregation processing unit 530 time course inspection processing unit 540 topic word score calculation processing unit 550 document classification processing unit 620 topic word rule storage unit, topic word rule storage unit 630 topic word aggregation rule storage unit , Topic word aggregation rule 640 NG word list storage means, NG word list storage unit

Claims (5)

An information classification apparatus for performing a classification with a feature related to a keyword by extracting a topic word from document data acquired based on a keyword and classifying the document data according to the topic word.
Topic word rule storage means for storing topic word rules using combinations of parts of speech as conditions for extracting topic words;
A topic word aggregation rule storage means that stores topic word aggregation rules for aggregating topic words that can be taken in the same meaning, consisting of combinations of parts of speech ;
As specified keyword search keyword, update date on which the document data, the search result output ranking, text (sentence) or obtained from the search results sentence containing a portion of the body, the body or the said body A data collection means for supplementarily collecting the text based on the publication location of the document data obtained from the search result and storing it in the document database;
Topic words that read out document data from the document database, extract topic word candidates from the document data with reference to the topic word rules stored in the topic word rule storage means, and store them in the topic word database Candidate extraction means;
Based on the said topic word aggregation rules stored in the topic word aggregation rule storage means, and the topic word aggregation means for aggregating the topic word candidates read out from the topic word database,
In each of the topic word candidates of the topic word database, and the topic word score calculating means for calculating a topic word score by the update time of the relevant height and the document data of the document data and the retrieval keyword including the topic word,
The topic word score of each of the topic word candidates in the topic word database is narrowed down to a predetermined number of topic word candidates, and further, the number of documents including the character string of the topic word candidate is narrowed down to a predetermined number or more of topic word candidates. the selected topic words having a document classification means for classifying the topic word score descending order of document data for each topic word, a,
The topic word score calculating means includes:
Means for summing the topic word scores of the topic word candidates with a document topic word score obtained from document data including a character string of the corresponding topic word candidate;
Means for determining the document topic word score from a search result output rank of one document data including a character string of a corresponding topic word candidate and an update date of document data obtained from the search result;
An information classification device characterized by comprising: The topic word aggregation means includes:
As the topic word,
A phrase using a combination of parts of speech that satisfies a rule predefined by the topic word rule,
A phrase not included in the search keyword,
The phrase is not suitable as a topic word phrase does not exist in the N G word list that is stored,
Phrases that are aggregated and selected as a single topic word based on the topic word aggregation rule, such that they can have the same meaning,
Information classification apparatus of claim 1 further comprising a means for extracting. When the information classification based on the topic word is repeatedly executed, if the extracted topic word continues to be extracted for a certain period T (positive integer), the word / phrase is excluded from the topic word. information classification apparatus according to claim 1, further comprising a time course testing means. The information classification program for functioning a computer as each means which comprises the information classification device of any one of Claims 1 thru | or 3. A computer-readable recording medium storing the information classification program according to claim 4 .

Images