JP4134975B2 - Topic document presentation method, apparatus, and program - Google Patents

Description

  The present invention relates to a topic document presentation method, apparatus, and program, and in particular, to efficiently allow a user to browse a large number of document data whose creation times can be specified, a set of documents that meet the same theme. Topic document presentation method that calculates the score representing the degree of topicality of each document in consideration of the document creation time, and arranges and presents the documents from the viewpoint of both similarity and topicality And an apparatus and a program.

  Due to the spread of the Internet, a huge amount of documents are overflowing on the network. In particular, in the case of a document that is updated and added many times a day, such as representing news, it takes a lot of labor to browse the information one by one and grasp the information.

  In addition, while the source of information is distributed, it is a feature of the Internet, but since documents written from various viewpoints along the same theme are released from multiple sources, they are temporarily selected. Even if you want to read only the necessary information, browsing operations become extremely difficult as the total amount of documents distributed increases. Furthermore, an increase in the amount of documents causes a document of interest to be lost, and it is difficult to obtain necessary information.

  In such a situation, there is a technique for organizing a document set and presenting it to a user in order to browse a large number of documents efficiently (see, for example, Patent Document 1).

The above technique is a method for searching for a document while narrowing down candidates to be presented by a user operation. A group of documents obtained as a result of performing a basic search based on a user's information request is divided into a set of similar documents, a word list representing each set is presented, and the set selected by the user is again displayed. This is a method of interactively searching while repeating the process of classifying similar documents and presenting a word list.
Japanese Patent Laid-Open No. 11-213000

  However, since the above technology is rather a technology that is conscious of users who actively search for documents, and is not a presentation technology that takes time into consideration, it is monitored in a situation where new documents continue to increase. Therefore, it is difficult to use for the purpose of grasping information.

  The present invention has been made in view of the above points. In a situation where an enormous amount of documents are distributed, an enormous amount of information is provided to assist passive users in grasping the topicality of information that changes every day. Provides an efficient information acquisition environment for users who want to acquire the latest topic information from a passive standpoint by organizing and presenting documents based on two viewpoints: similarity between documents and time swell of topics An object of the present invention is to provide a topic document presentation method, apparatus, and program that can be used.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) is a topic document presentation method in a topic document presentation apparatus that automatically selects and presents a highly topical document from a huge document group with creation time information.
Similarity calculation means determines attribute values for words included in the input document, calculates similarity between documents using a document vector obtained by converting each input document into a vector of attribute value strings, and similarity A similarity calculation step (step 1) to be recorded in the degree recording means;
The clustering means divides the input document group into at least one subset from the similar documents using the similarity between documents recorded in the similarity recording means, and stores the documents belonging to each subset in the cluster recording means A clustering step (step 2),
The provisional topic degree calculation means calculates the provisional topic degree for each document by adding all similarities between the subsets divided in the clustering step and other documents in the subset to which each document belongs. A provisional topic level calculation step;
A freshness determination step in which the freshness determination means determines the freshness of each document by a function having a value that increases as the creation time information of each document is closer to the current time, and decreases as the distance from the current time
Document topic calculation means, for each document, the provisional topic of a by multiplying the freshness calculates the document topic of the document topic calculation step of storing the document topic of the recording means (Step 3),
The presentation data creating means uses the document belonging to each subset stored in the cluster recording means and the document topic level stored in the document topic level recording means, and sets each subset to the subset. A presentation data creation step (step 4) is performed for creating and outputting data in which the included document group is arranged in descending order of the document topic level .

Further, the present invention (Claim 2), in the new freshness determining step,
The freshness determination means uses a function represented by an exponential function as a function for determining the freshness.

The present invention (Claim 3 ) further performs a category classification step in which the category classification means classifies the input document into one or more of a plurality of categories according to the contents,
Each of the similarity calculation step, clustering step, provisional topic level calculation step, freshness determination step, document topic level calculation step, and presentation data creation step is performed for each classified category.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 4 ) is a topic document presentation device that automatically selects and presents a highly topical document from a huge document group with creation time information,
An attribute value is determined for a word included in the input document, a similarity between each document is calculated using a document vector obtained by converting each input document into an attribute value sequence vector, and recorded in the similarity recording unit 207 A similarity calculation means 208,
Clustering that divides the input document group into at least one subset from the similar documents using the similarity between documents recorded in the similarity recording unit 207, and stores the documents belonging to each subset in the cluster recording unit 209 Means 208;
For each document, the provisional topic degree calculating unit 210 that calculates a provisional topic of all similarities addition to between the other documents in the subset the document belongs among the subsets divided in the clustering means 208 ,
A freshness determination means for determining the freshness of each document by a function in which the value is larger as the creation time information of each document is closer to the current time, and the value is smaller as it is farther from the current time;
For each document, the provisional topic of the integrated and freshness calculates the document topic degree, document topic degree calculating unit 210 to store the document topic of the recording unit 211,
Using the documents belonging to each subset stored in the cluster recording unit 209 and the document topic level stored in the document topic level recording unit 211, the document group included in the subset for each of the subsets and create a data arranged in order of the document topic degree, having a presentation data creating means 212 to be output.

Further, the present invention (Claim 5), in the new freshness determining means,
A function represented by an exponential function is used as a function for determining freshness.

The present invention (Claim 6 ) further includes category classification means for classifying the input document into any one or more of a plurality of categories according to the contents,
The processes of the similarity calculation means, clustering means, provisional topic degree calculation means, document topic degree calculation means, freshness determination means, and presentation data creation means are performed for each classified category.

The present invention (Claim 7) is a topic document presentation program for causing a computer to function as each means constituting the topic document presentation apparatus according to any one of claims 4 to 6 .

  As described above, according to the present invention, the similarity between the documents is calculated, the similar documents are aggregated, and the topic level of each document is assigned to the document as the topic level. Is freed from the hassle of reading multiple documents that deal with the same content, and it is possible to preferentially browse only highly topical documents for each of similar document groups (clusters), so that effective information acquisition is possible. An environment is provided.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 3 shows the configuration of the topic document presentation device in the first embodiment of the present invention.

  The topic document presentation device 20 shown in the figure is connected to a topic output device 21 for outputting topic document information.

  The topic document presentation device 20 includes a document collection unit 201, a document data recording unit 202, a document analysis unit 203, a word totaling unit 204, a document vector recording unit 205, a similarity calculation unit 206, a similarity recording unit 207, a clustering unit 208, A cluster recording unit 209, a document topic level calculation unit 210, a document topic level recording unit 211, and a presentation data creation unit 212 are configured.

  The document topic level calculation unit 210 includes a time information acquisition unit 2101, a similarity acquisition unit 2102, a similarity addition unit 2103, a freshness coefficient calculation unit 2104, and an integration unit 2105.

  Hereinafter, the function of each processing unit constituting the topic document presentation device 20 will be described.

  The document collection unit 201 collects document data from an external information source. For example, news article data published on a newspaper company site on the Internet is acquired, a unique identifier (document ID) is assigned to each document, and stored in the document data recording unit 202. At that time, the creation time information of each document is also acquired and recorded together.

  As shown in FIG. 4, the document data recording unit 202 records the creation time and text of the document, and the document ID uniquely assigned to each document by the document collection unit 202.

  The document analysis unit 203 acquires the document data stored in the document data recording unit 202, divides the text into words by morphological analysis processing, and temporarily outputs the obtained word list to the document vector recording unit 205. The document analysis unit 203 simultaneously sends a word list to the word totaling unit 204, and the word totaling unit 204 totals the number of documents in which each word appears.

  The morpheme analysis process in the document analysis unit 203 is a process of dividing a sentence into components (morphemes) such as parts of speech such as verbs and particles and symbols such as “, (punctuation)” and “. (Reading)”. In the processing performed by the topic document presentation method of the present invention, the document is expressed as a vector for the purpose of calculating similarity between documents, and it is not necessary to handle morphemes that are not suitable for characterizing the documents. Accordingly, punctuation marks and further morphemes such as particles and auxiliary verbs are excluded from the results of the morphological analysis processing, and nouns, verbs, and adjectives are targeted and sent to the document vector recording unit 205 and the word counting unit 204. .

  The word totaling unit 204 totals the number of documents in which each word (in this embodiment, excluding punctuation marks and particles) sent from the document analysis unit 203 appears, and uses the result to calculate the weight for each word. decide. For example, it is assumed that n types of words W1 to Wn are obtained as a result of the document analysis unit 203 dividing the document with the document ID “DOC-012345” for each word. A method for determining the weight of the word will be described later. As a result of the word totaling unit 204 considering the number of appearances of these words in all documents, the weights V1 to Vn corresponding to the respective words are obtained. The document with the document ID “DOC-012345” has a vector format in which weights of words appearing in the document are enumerated such that the attribute value corresponding to the word W1 is V1, the attribute value corresponding to the word W2 is V2,. Can be expressed.

  The processing flow in which the word totaling unit 204 determines the weight of each word will be described in detail with reference to FIG.

  First, the document analysis unit 204 acquires a word list obtained as a result of dividing a document into words, and totals the number of documents in which each word appears even once (hereinafter referred to as “appearing document number”). At the same time, the number of processed documents is counted (step 201).

  When the step 201 is repeated for all the documents processed by the document analysis unit 203 (step 202), the number of occurrences of the word W1 is “5”, the number of occurrences of the word W2 is “8”, and so on. The number of appearance documents of each word is totaled.

  Subsequently, the weight of each word is determined (step 203).

  Usually, the TF-IDF method is often used to determine the weight representing the importance of a word appearing in a document. The TF-IDF method uses a TF (Term Frequency: number of occurrences in a document) of a word and DF (Document Frequency: the number of occurrences of a document), and calculates the weight TF-IDF (d, w) of the word w in the document d. It is given by equation (1). TF (d, w) is the number of occurrences of the word w in the document d, N is the total number of documents, and DF (w) is the number of occurrences of the word w in all N documents. The TF-IDF method is an evaluation method that considers that words that appear more frequently in a document or words that appear in a small number of documents in all documents are more important.

他にも単語の重み付けに使用可能な方法はあるが、本実施の形態では、特にニュース記事を収集して話題文書提示方法を適用する場合を想定し、上記の式（１）のＴＦ−ＩＤＦ（ｄ，ｗ）ではなく、式（２）のWeight(w)で単語ｗの重み付けを行う。ニュース記事では主題となる人名や団体名などの名詞の１文書内における出現回数が必ずしも高いわけではなく、出現回数と単語の重要度が直結しないことに基づくものであり、ＴＦ−ＩＤＦ法から文書内出現回数（ＴＦ）の効果を省いたものである。

Although there are other methods that can be used for weighting words, in this embodiment, it is assumed that the news document is collected and the topic document presentation method is applied, and the TF-IDF of the above formula (1) is applied. Instead of (d, w), the weighting of the word w is performed by Weight (w) of the formula (2). In news articles, the number of occurrences of nouns such as subject names and organization names in a document is not necessarily high, and is based on the fact that the number of occurrences and the importance of words are not directly linked. The effect of the number of appearances (TF) is omitted.

単語集計部２０４は全単語の重みWeight(w)が決定すると（ステップ２０３）、文書ベクトル記録部２０５に保存されている各文書の単語リストを参照し、各文書に現れた単語に対して属性値を重みWeight(w)で与え、文書ベクトルを決定し、文書ベクトル記録部２０５に保存する（ステップ２０４）。

When the total weight Weight (w) is determined (step 203), the word totaling unit 204 refers to the word list of each document stored in the document vector recording unit 205, and sets the attribute for the word appearing in each document. A value is given by weight Weight (w), a document vector is determined, and stored in the document vector recording unit 205 (step 204).

  FIG. 6 is an example of a document vector stored in the document vector recording unit according to the first embodiment of the present invention. The document ID “DOC-012345” is represented by a vector having an attribute value “0.0053” for “this morning”, an attribute value “2.38” for “Ox”, an attribute value “1.02” for the player, and so on. As can be seen from the above equation (2), the weight of each word determined by the word totaling unit 204 is a value determined only by the total number of documents N and the number of appearing documents DF (w) of each word, For example, as the attribute value of the word “this morning” included in “DOC-012345” and “DOC-012346”, the weight “0.0053” of the word “this morning” determined in step 203 is used.

  The similarity calculation unit 206 refers to each document vector stored in the document vector recording unit 205 and calculates the similarity between the documents. FIG. 7 is a flowchart of the process of the similarity calculation unit in the first embodiment of the present invention.

  First, referring to the document vector recording unit 205, document vectors of two different documents are acquired (step 301).

Subsequently, the similarity between the two acquired documents is calculated (step 302). The simplest method of calculating similarity using two document vectors is a method using cosine similarity. This is a method in which the cosine of the angle formed by two vectors is used as a similarity between two documents. The more words that appear in two documents, the smaller the angle formed by the two document vectors, and thus the larger the cosine. Become. In the present embodiment, the similarity between the two documents d _i and d _j is defined by the cosine similarity S (d _i , d _j ) in the equation (3). d _i →, d _j → is a vector of each document, | d _i → |, | d _j → | is the size of each document vector, and θ _ij is an angle formed by the vectors of each document.

このようにして算出した類似度を類似度記録部２０７に保存し（ステップ３０３）、上記のステップ３０１〜３０３の処理を文書ベクトル記録部２０５に保存された文書ベクトルの全組み合わせについて行う（ステップ３０４）。

The similarity calculated in this way is stored in the similarity recording unit 207 (step 303), and the above steps 301 to 303 are performed for all combinations of document vectors stored in the document vector recording unit 205 (step 304). ).

FIG. 8 is an example of inter-document similarity data stored in the similarity recording unit in the first embodiment of the present invention. This indicates that the degree of similarity between the documents “DOC-012345” and “DOC-012346” was “0.021”. When there is no word that appears in common in two documents, the value of the inner product (d _i →, d _j →) of the vector in equation (3) becomes “0”. The degree of similarity is “0” like the combination of “DOC-012345” and “DOC-012347”.

  The clustering unit 208 uses the similarity between documents stored in the similarity recording unit 207 to perform a process of grouping similar documents together. This process is a process in which it is considered that documents similar to each other are likely to be documents written along the same subject, and are combined into one. Hereinafter, a group of documents grouped by the process is referred to as “cluster”, and the process of dividing the entire document group into a plurality of clusters is referred to as “clustering”.

  Various existing methods can be used for clustering, and the clustering method used in the present invention is not limited to any method, but in this embodiment, the number of clusters to be divided cannot be estimated in advance. It is assumed that the maximum distance method, which is one of the available methods, is used. As a means for realizing the maximum distance method, for example, “Iwanami Shoten Multimedia Informatics 2“ Organization of Information ”pp.192 to 193” is used.

First, in performing clustering, a value obtained by subtracting the inter-document similarity from 1 is defined as a distance between documents. The similarity between documents has a value from 0 to 1 according to the above equation (3), and takes a larger value closer to 1 as the documents are similar. Therefore, Genzure document Similarity 1 takes small value close to about 0 2 document is similar, it is possible to define a "dissimilarity" which approaches 1 as the similarity decreases. This dissimilarity is treated as a distance between documents.

Hereinafter, clustering processing based on the maximum distance method will be described. The maximum distance method, to form a new cluster in the center of the document that is far enough away from an existing cluster, is a method to proceed while increasing one after another the number of clusters.

  FIG. 9 is a flowchart of clustering processing according to the first embodiment of this invention.

  Step 401) First, a first cluster centered on any one document is formed. There is only one document belonging to the first cluster at this time.

  Step 402) Search for the cluster closest to each document. Here, the distance from the document d to the cluster c refers to the distance between the document at the center of the cluster c and the document d. Note that when step 402 is performed for the first time, there is only one cluster, so the nearest cluster is determined without searching.

Step 403) In step 402, the distance (shortest distance) from each document to the nearest cluster is obtained, but the document in which these shortest distances are maximized is examined. That is, the document d _k farthest from the center of the existing cluster is _obtained .

  Step 404) When the maximum value of the shortest distance obtained in Step 403 is m and the maximum value of the distance between the centers of the existing clusters is MAX, whether r satisfies a conditional expression “m / MAX> r” with r as a constant. to decide. Only the constant r needs to be determined in advance. If it is satisfied, the process proceeds to step 405. If it is not satisfied, the process is terminated.

Step 405) If the above conditional expression is satisfied, a new cluster centered on the document d _k is formed, and the processing from Step 402 is repeated until the conditional expression is not satisfied.

  When the above processing is completed, the clustering processing unit 208 transfers the processing result to the cluster recording unit 209 and stores it. FIG. 10 is an example of clustering result data in the cluster recording unit according to the embodiment of the present invention. C1, C2, C3, C4,... Are cluster identifiers (cluster IDs), and IDs of documents belonging to the respective clusters are recorded. As a result of clustering by the longest distance method, there is a cluster formed by only one document such as the cluster “C2”, but all documents always belong to one of the clusters.

  The document topic level calculation unit 210 uses the similarity data between the documents stored in the similarity recording unit 207, digitizes the topic level of each document, and outputs it to the document topic level recording unit 211. Hereinafter, the flow of processing performed by the topic level calculation unit 210 will be described with reference to FIG.

  FIG. 11 is a flowchart of processing performed by the topic degree calculation unit according to the first embodiment of the present invention.

  First, the time information acquisition unit 2101 sets one set of document ID and creation time from the document data recording unit 202, for example, the document “DOC-012345” from the data shown in FIG. 4 recorded in the document data recording unit 202. The creation time “2004/9/11 23:00” is acquired (step 501).

  When the document ID “DOC-012345” acquired by the time information acquisition unit 2101 is transmitted to the similarity acquisition unit 2012, the similarity acquisition unit 2012 indicates the similarity between the document ID and another document. Obtained from 207 and stored in the memory (step 502). In the example illustrated in FIG. 8, the similarities “0.021”, “0”, “0.300”,... Between “DOC-012345” and other documents are sequentially acquired.

  Subsequently, the similarity adding unit 2103 adds the similarities acquired by the similarity acquiring unit 2102 one after another and stores it in the memory (step 503). As for the document “DOC-012345”, it is assumed that “0.021”, “0”, “0.300” are added one after another, and the total value finally becomes “1.28”. As described above, a value “1.28” obtained by adding all similarities between a document “DOC-0123345” and other documents is referred to as a provisional topic level of the document “DOC-012345”. The provisional topic level is a topicality evaluation method in which when a certain document is focused, the higher the number of similar documents, the higher the level of attention. In the case of newspaper articles and the like, it can be said that this is a particularly effective evaluation method because a plurality of newspaper companies publish related articles one after another and the number of similar documents increases for highly topical matters.

  The freshness coefficient calculation unit 2104 compares the difference between the time information “2004/9/11 23:00” received from the time information acquisition unit 2101 and the current time, and determines the freshness coefficient of the acquired document “DOC-012345”. And stored in the memory (step 504).

The freshness coefficient is a value determined by a function of time such that the value is larger as the document creation time is closer to the current time, and the value is smaller as the document creation time is closer to 0. The function having this property is stored in the memory in advance, and for example, it is preferable to use the forgetting curve shown in FIG. 12 that shows how the storage of information obtained in the past by humans fades with time. This shows that the information acquired recently (time close to the present) keeps a large amount of memory even now, and the information acquired in the past (time far from the present) shows that the amount of memory is now small. The vertical axis indicates the storage amount F (t) in the past (time-t where the current time is 0) that is back by t from the current time, which is the storage amount, by the equation (4). In the equation (4), the value becomes F ₀ at t = 0 (current), and the value approaches 0 as it goes back in the past.

F (t) = F ₀ × exp (−t / T ₀ ) (4)
T _{0 in} equation (4) is a parameter that determines the rate at which the freshness coefficient decays as the time goes back (forgetting rate). If the value is increased, the attenuation will be moderate, and even if the document has an old creation time, a certain degree of freshness coefficient is given to make it easy to determine that the document is a topic document. On the other hand, the attenuation by reducing the T ₀ becomes steep curve, document with very new creation time is considered the topic document, made in the old document it is less likely to be considered a hot topic trend. T ₀ is a constant that can be variously set according to the nature of the document to be collected and the usage scene.

  In the following, the difference between the time information “2004/9/11 23:00” of “DOC-012345” and the current time is obtained in step 504 and applied to equation (4), resulting in a freshness coefficient of “0.7”. The explanation will continue as if

  The accumulating unit 2105 accumulates the provisional topic levels “1.28” and “0.7” of the document “DOC-012345” to obtain the topic level “0.896” of “DOC-012345” (step 405). When the integration is completed, the temporary topic level “1.28” held in the similarity adding unit 2103 is cleared to “0” for calculation of the document topic level of the next document.

  The calculated document topic level of the document “DOC-012345” is stored in the document topic level recording unit 211 together with the document ID (step 506).

  The processing from step 501 to step 506 is repeated for all documents acquired by the time information acquisition unit 2101 (step 507), and the processing ends.

  FIG. 13 is an example of the document topic level data recorded in the document topic level recording unit in the first embodiment of the present invention. In the document topic level recording unit 211 shown in the figure, a set of the document ID “DOC-012345” and the document topic level “1.28” is recorded. The document topic level is similarly recorded for other documents.

  The presentation data creation unit 212 refers to each data recorded in the cluster recording unit 209 and the document topic level recording unit 211, and creates data to be output from the topic submission device 20 of the present invention. A list of document IDs belonging to each cluster is acquired from the cluster recording unit 209, and a document topic level corresponding to each document ID is acquired from the document topic level recording unit 211. Further, the creation time and the text corresponding to each document ID are acquired from the document data recording unit 202, and document list data for each cluster is created. The documents in each cluster are rearranged in descending order according to the document topic level. The data thus created is output to the topic output device 21 and presented to the user.

  A display example of the topic document output on the screen of the topic output device 21 is shown in FIG. In the horizontal direction on the screen, clusters 1, 2,... Are arranged in order from the left, and documents belonging to each cluster are displayed side by side in the vertical direction. In the figure, a is an area in which documents belonging to the cluster with the cluster ID “C1” are displayed, and the documents belonging to the cluster are arranged from the top in descending order of the document topic level. B in the figure is the creation time information of the document having the third highest document topic level in the cluster, c in the figure is the document topic level, and d in the figure is the text.

  In the example of FIG. 14, four documents related to the retirement of a certain athlete are grouped as “Cluster 1”, “Cluster 2” is a cluster consisting of only one document related to museum events, and “Cluster 3” is A cluster of many documents related to the merger of baseball teams. The museum events handled by the documents in “Cluster 2” are not handled by other documents, and are therefore a cluster composed of only one document. Conversely, “Cluster 3” is related to one topic of baseball team merger, and related documents such as documents dealing with fan signature activities, documents dealing with strikes, and documents dealing with the effects of mergers are linked. It is a large cluster gathered in

  The document topic level increases as the number of similar documents increases, and increases as the document creation time is new. For this reason, a document having a low topicality such as a document belonging to “Cluster 2” has a low document topic level. Further, when attention is focused on documents belonging to “cluster 3”, the document topic level of a document with a new creation time is relatively large and tends to be displayed at the top.

  Users can grasp topic information in a short period of time by reading the documents above each cluster, and by reading each document in detail for the cluster they are interested in, it is possible to view the document efficiently. .

[Second Embodiment]
FIG. 15 shows the configuration of a topic document presentation device according to the second embodiment of the present invention.

  The topic document presentation device 130 is connected to a topic output device 1314 for outputting topic document information and an input device 1315 for receiving an operation from a user.

  The topic document presentation device 130 includes a document collection unit 131, a document data recording unit 132, a document analysis unit 133, a document classification unit 134, a word totaling unit 135, a document vector recording unit 136, a similarity calculation unit 137, and a similarity recording unit 138. A clustering unit 139, a cluster recording unit 1310, a document topic level calculation unit 1311, a document topic level recording unit 1312, and a presentation data creation unit 1313. The document topic level calculation unit 1311 is further similar to a time information acquisition unit 13111. A degree acquisition unit 13112, a similarity addition unit 13113, a freshness coefficient calculation unit 13114, and an integration unit 13115 are included.

  Hereinafter, the function of each processing unit constituting the topic document presentation device 130 will be described.

The document collection unit 131 collects document data from an external information source, and has the same function as the document collection unit 201 in the first embodiment. A unique identifier (document ID) is assigned to the collected document and stored in the document data recording unit 132. At that time, the creation time information of each document is also acquired and recorded together. The document collection unit 131 further sends the collected document to the document analysis unit 133.

  The document analysis unit 133 divides the document received from the document collection unit 131 into words by morphological analysis processing, and temporarily outputs the obtained word list to the document vector recording unit 136. In addition, the document analysis unit 133 simultaneously sends a word list to the word totaling unit 135, and the word totaling unit 135 totals the number of documents in which each word appears.

  Similar to the word totaling unit 204 in the first embodiment described above, the word totaling unit 135 determines the weight from the formula (2) using the total number of documents for each word, and constructs a document vector for each document. And stored in the document vector recording unit 136. As in the first embodiment, the document vector recording unit 136 stores a document ID and a document vector as shown in FIG.

  The document analysis unit 133 further sends the word list of the document to the document classification unit 134. Using this word list, the document classification unit 134 classifies the document into one or more categories among a plurality of predetermined categories. When classifying documents into categories, existing document classification techniques include, for example, “Nobuo Ueda, Kazuaki Saito,“ Probability model of multi-topic texts—parametric mixed model ”, and D-II Vol. J87-D-II No.3 pp.872-883, March 2004 ”is used.

  An outline of the above existing technology will be briefly described.

  First, a machine learns a set of documents whose categories are known, and predicts which category an unknown document belongs to using the learned machine. This prediction process corresponds to document classification.

The document analysis unit 133 sends the word list of each document to the document classification unit 134. At this time, each word is sent with the number of appearances in the document added. When the number of occurrences in the document _{d m} word Wn and _{X mn,} word frequency vector of document _{d m} is, _{X m} → _{is, X m → = (X m1} , X m2, ..., X mv) is represented by The

The total number of categories is L, the category vector indicating the category to which the document d _m belongs,
Y _m → = (Y _m1 , Y _m2 ,..., Y _mL )
It shall be expressed as Here, Y _{m, k} is "1" when the document d _m belongs to the k category, it takes a value of "0" when not belong. Although it is allowed to belong to a plurality of categories, it belongs to at least one category.

First, the machine is trained so that the category is a document set D = {(X _m , Y _m )} (m = 1 to N) to X _{m with} a known category. Next, using this machine, Y _* is estimated by inputting the word frequency vector X _* of the document d _* whose category is unknown. Y _{* k} of the category vector Ymk = (Y _{* 1} , Y _{* 2} ,..., Y _{* L} ) calculated by prediction indicates the degree of coincidence (probability of belonging) whether or not the document d _* belongs to the kth category. .

  In the above existing document classification technology, the output for the word frequency vector of an input document whose category is unknown is a vector listing the matching degrees to each category, so the input documents are classified into several categories in descending order of matching degrees. . Although a method of use such as classifying into a category where the degree of coincidence exceeds a certain threshold value can be considered, in this embodiment, description will be continued assuming that only one category with the maximum degree of coincidence is classified.

  The category information in which the document classification unit 134 classifies the document using the existing document classification technology is stored in the document data recording unit 132.

  FIG. 16 shows an example of data stored in the document data recording unit in the second embodiment of the present invention. In addition to the example of data stored in the document data recording unit 202 of the first embodiment (FIG. 4), the belonging category name is recorded.

  The similarity calculation unit 137 calculates the similarity between the documents with reference to each document vector stored in the document vector recording unit 136. At this time, the category information stored in the document data recording unit 132 is used as the category information. Refer to and calculate the similarity between documents belonging to the same category.

  The method for calculating the similarity is the same as that in the first embodiment. FIG. 17 is an example of inter-document similarity data stored in the similarity recording unit in the second embodiment of the present invention. In FIG. 16, the similarity “0.021” between the documents “DOC-012345” and “DOC-012347” in which the category name “sports” is recorded is recorded in the “sports” column of FIG. Similarly for other categories and other documents, the similarity between documents is recorded for each category.

  The clustering unit 139 performs document clustering processing using the inter-document similarity recorded in the similarity recording unit 138. The same processing as the clustering unit 208 in the first embodiment is performed except that the clustering processing is performed for each category.

  FIG. 18 is an example of clustering result data stored in the cluster recording unit according to the second embodiment of the present invention. The clustering results are recorded in categories such as “sports”, “society”, “movies”, etc. For example, in the first cluster “C11” of the category “sports”, document IDs “DOC-012345” “DOC” are stored. Three documents “-012349” and “DOC-012355” are included. The “sports” category includes a plurality of clusters “C12”, “C13”... In the “society” category, “C21” “C22” “C23”. Clustering results such as “C32”, “C33”... Are stored.

  The document topic level calculation unit 1311 uses similarity data between documents stored in the similarity level recording unit 138, quantifies the topic level of each document, and outputs it to the document topic level recording unit 1312.

  The processing performed by the topic level calculation unit 1311 is the same as the processing performed by the document topic level calculation unit 210 in the first embodiment described above, except that the processing is performed for each category. The flow of processing will be described with reference to FIG.

  FIG. 19 is a flowchart of processing performed by the topic level calculation unit according to the second embodiment of the present invention.

  First, the time information acquisition unit 13111 receives one set of document ID, creation time, and category name from the document data recording unit 132, for example, the document “DOC-012345 from the data in FIG. 16 recorded in the document data recording unit 132. ”And its creation time“ 2004/9/11 23:00 ”and the category name“ sports ”are acquired (step 601).

  When the document ID “DOC-012345” acquired by the time information acquisition unit 13111 and the category name “sports” are sent to the similarity acquisition unit 13112, the similarity acquisition unit 13112 and the document ID classified into the “sports” category The similarity with “DOC-012345” is acquired from the similarity recording unit 138 (step 602). In the example of the similarity recording unit 138 shown in FIG. 17, the similarity “0.021”, “0”,... Between “DOC-012345” recorded in the “sports” category and other documents is acquired one after another. .

  A step of calculating the provisional topic level of the document “DOC-012345” by the similarity adding unit 13113 adding these similarities (step 603), and a freshness coefficient calculating unit 13114 calculating the difference between the creation time of each document and the current time The step of determining the freshness coefficient from the step (step 604) and the step of calculating the document topic level by multiplying the provisional topic level by the freshness factor (step 605) by the accumulating unit 13115 are the respective steps in the first embodiment. Are the same.

  The document topic level of each document calculated by the integrating unit 13115 is stored in the document topic level recording unit 1312 together with the category information and document ID to which each document belongs (step 606).

  The processing in steps 601 to 606 is repeated for all documents acquired by the time information acquisition unit 13111 (step 607), and the processing ends.

  FIG. 20 is an example of document topic level data stored in the document topic level recording unit according to the second embodiment of the present invention. In the document topic level recording unit 1312 shown in the figure, a set of the document ID “DOC-012345” and the document topic level “1.28” is recorded in the “sports” category, and the documents classified into other categories are recorded. Similarly, the document topic level is recorded for each.

  The presentation data creation unit 1313 refers to each data recorded in the cluster recording unit 1310 and the document topic level recording unit 1312 and creates data to be output from the topic document presentation device 130 of the present invention. A list of document IDs of documents belonging to each cluster of each category is acquired from the cluster recording unit 1310, and a document topic level corresponding to each document ID is acquired from the document topic level recording unit 1312. Further, the creation time and the text corresponding to each document ID are acquired from the document data recording unit 132, and the document list data for each cluster is created for each category. The documents in each cluster are rearranged in descending order according to the document topic level. The data thus created is output to the topic output device 1314 and presented to the user.

  A user uses an input device 1315 such as a mouse, a keyboard, and a touch panel operation connected to the topic document presentation device 130 of the present invention to interactively switch a display screen while selecting a button displayed on the screen, and Can be viewed.

  A display example of the topic document output on the screen of the topic output device 1314 is shown in FIG. Although a screen is displayed for each category, FIG. 21 is an example of a state in which the “sports” category is displayed.

  FIG. 21A is a button for switching the display to a screen displaying topic documents of the category “Society”, and the document classification unit 134 classifies “movie”, “economic”, “entertainment”, and the like. Topic documents for each category can be displayed.

  In FIG. 21, at the bottom of the screen, only the documents having the highest document topic level for each cluster are displayed among the documents included in the displayed category. The topic level (b), creation time (c), and text (d) are arranged in order from the left. As can be seen from FIG. 16, the document displayed at the top is the document with the document ID “DOC-012347”, and the topic level “3.51” (see FIG. 20) calculated by the document topic level calculation unit 1311 is displayed. It is displayed.

  The document displayed here is the document having the highest document topic level in each of the categories “C11”, “C12”, “C13”,... Among the documents recorded in the item “Sports” in FIG. Are selected from “C11” to one document, “C12” to one document, and so on. The document “DOC-012347” is a document selected from the cluster “C13”.

  Accordingly, in FIG. 21, a document equal to the number of clusters included in the “sports” category is displayed. The selected documents are further sorted in descending order according to the document topic level, and are displayed as representative documents of each cluster as shown in FIG. FIG. 21E is a button for obtaining detailed information of the cluster including the representative document displayed on the screen. When the user selects this button, a transition is made to a screen for displaying all the documents included in the corresponding cluster (cluster “C13” to which the document “DOC-012347” belongs in the example of FIG. 21).

  FIG. 22 is a display example of the screen after transition when the user selects the “display details” button e of the first document displayed on the screen shown in FIG. 22b is the document displayed in FIG. 21 as well, and is the document having the highest document topic level in the selected cluster “C13”. Hereinafter, the documents included in “C13” are displayed in descending order of the document topic level.

  22a is a button for returning to the screen of FIG. 21 again.

  With the screen display described above, the user browses representative documents of any category, and displays all the documents in the corresponding cluster by selecting the “Show details” button for the document that is particularly interested. More detailed information.

  In both the first embodiment and the second embodiment described above, the document topic level calculation process is independent of the clustering process, and the document topic level calculation units 210 and 1311 add the inter-document similarity. Processing is performed on all documents (all documents in a category in the second embodiment). However, instead of this, the degree of provisional topic may be calculated by adding the similarity only between documents in each cluster. In this case, since the number of similarities to be added differs depending on the cluster, it is impossible to determine the magnitude of importance by comparing the document topic levels of two documents belonging to different clusters. However, this is a method that can be used when determining the importance only between documents belonging to one cluster. In addition, a reduction in processing time required for topic document presentation can be expected.

  In the first and second embodiments described above, the topic output devices 21 and 1314 are assumed to be monitor devices, and are displayed on the screen so that the user can browse them. As a storage device for storing the information presented on the screen in the above embodiment, a user or another device may be readable.

  Further, based on the processing described in the above embodiment, it can be constructed as a program that operates on a computer, and can be installed on a computer that operates as a topic document presentation device, or can be distributed via a network. .

  It is also possible to store the constructed program in a portable storage medium such as a hard disk device, a flexible disk, or a CD-ROM connected to a computer used as a topic document presentation device.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention is applicable to a technique for allowing a user to view a large amount of document data based on topicality.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a block diagram of the topic document presentation apparatus in the 1st Embodiment of this invention. It is an example of the data preserve | saved at the document data recording part in the 1st Embodiment of this invention. It is a flowchart of the process which the word totalization part in the 1st Embodiment of this invention performs. It is an example of the data preserve | saved at the document vector recording part in the 1st Embodiment of this invention. It is a flowchart of the process which the similarity calculation part in the 1st Embodiment of this invention performs. It is an example of the similarity data between documents preserve | saved in the similarity recording part in the 1st Embodiment of this invention. It is a flowchart of the clustering process in the clustering part in the 1st Embodiment of this invention. It is an example of the data of the clustering result preserve | saved at the cluster recording part in the 1st Embodiment of this invention. It is a flowchart of the process which the topic degree calculation part in the 1st Embodiment of this invention performs. It is an example of the function which determines the freshness coefficient in the 1st Embodiment of this invention. It is an example of the document topic level data preserve | saved in the document topic level recording part in the 1st Embodiment of this invention. It is a display example of topic document data output to the topic output device of the first exemplary embodiment of the present invention. It is a block diagram of the topic document presentation apparatus in the 2nd Embodiment of this invention. It is an example of the data preserve | saved in the document data recording part in the 2nd Embodiment of this invention. It is an example of the similarity data between documents preserve | saved in the similarity recording part in the 2nd Embodiment of this invention. It is an example of the clustering result data preserve | saved at the cluster recording part in the 2nd Embodiment of this invention. It is a flowchart of the process which the topic degree calculation part in the 2nd Embodiment of this invention performs. It is an example of the document topic degree data preserve | saved in the document topic recording part in the 2nd Embodiment of this invention. It is a display example of topic document data output to the topic output device in the second embodiment of the present invention. It is the example of a display after the screen transition of the topic document data output to the topic output device in the 2nd Embodiment of this invention.

Explanation of symbols

20, 130 Topic document presentation device 21, 1314 Topic output device 201, 131 Document collection unit 202, 132 Document data recording unit 203, 133 Document analysis unit 134 Document classification unit 204, 135 Word aggregation unit 205, 136 Document vector recording unit 206 , 137 Similarity calculation unit 207, 138 Similarity recording unit 208, 139 Clustering unit 209, 1310 Cluster recording unit 210, 1311 Document topic level calculation unit 211, 1312 Document topic level recording unit 212, 1313 Presentation data creation unit 2101, 13111 Time information acquisition unit 2102, 13112 Similarity acquisition unit 2103, 13113 Similarity addition unit 2104, 13114 Freshness coefficient calculation unit 2105, 13115 Accumulation unit 1315 Input device

Claims (7)

In a topic document presentation method in a topic document presentation device that automatically selects and presents a highly topical document from a huge document group with creation time information,
Similarity calculation means determines attribute values for words included in the input document, calculates similarity between documents using a document vector obtained by converting each input document into a vector of attribute value strings, and similarity A similarity calculation step for recording in the degree recording means;
Clustering means divides the input document group into at least one subset from the similar documents using the similarity between the documents recorded in the similarity recording means, and the clusters belonging to each subset are cluster recording means A clustering step to store in
The provisional topic level calculation means of the document topic level calculation means adds all similarities between each document and the other documents in the subset to which each document belongs among the subsets divided in the clustering step. A provisional topic degree calculation step for calculating a provisional topic degree;
The freshness determination means of the document topic level calculation means determines the freshness of each document by a function that has a value that increases as the creation time information of each document is closer to the current time and decreases as the distance from the current time Steps,
The document topic calculation means, for each document, by integrating with the freshness and the provisional topic of calculating a document topic of a document topic calculation step of storing the document topic of the recording means,
The presentation data creation means uses the documents belonging to each subset stored in the cluster recording means and the document topic level stored in the document topic level recording means for each of the subsets. , A presentation data creation step for creating and outputting data in which document groups included in the subset are arranged in descending order of document topics; and
The topic document presentation method characterized by performing . In the freshness determination step,
The freshness determining means is
Wherein as a function of determining the freshness, the topic document presentation method of claim 1 wherein using a function represented by an exponential function. Category classification means further performs a category classification step of classifying the input document into any one or more of a plurality of categories according to the contents,
The similarity calculation step, the clustering step, the provisional topic degree calculating step, the freshness determining step, the document topic calculation step, each of the presentation data generating step, according to claim 1, wherein performing the classification category Topic document presentation method. A topic document presentation device that automatically selects and presents a highly topical document from a huge document group with creation time information,
An attribute value is determined for a word included in the input document, a similarity between each document is calculated using a document vector obtained by converting each input document into a vector of attribute value strings, and recorded in the similarity recording unit. Similarity calculation means;
Clustering that divides an input document group into at least one subset from similar documents using the similarity between the documents recorded in the similarity recording means, and stores the documents belonging to each subset in the cluster recording means Means,
For each document, provisional topic degree calculation means for calculating a provisional topic degree by adding all the similarities with other documents in the subset to which the document belongs among each subset divided by the clustering means ;
A freshness determination means for determining the freshness of each document by a function in which the value is larger as the creation time information of each document is closer to the current time, and the value is smaller as it is farther from the current time;
For each document, by integrating with the freshness and the provisional topic of calculating a document topic of a document topic calculation means for saving the document topic of the recording means,
Each of the subsets is included in the subset by using the document belonging to each subset stored in the cluster recording unit and the document topic level stored in the document topic level recording unit. Creating presentation data creating means for creating and outputting data in which document groups are arranged in descending order of document topics ; and
Topic document presentation device which is characterized in that it has a. In the previous Symbol freshness determination means,
The topic document presentation apparatus according to claim 4, wherein a function represented by an exponential function is used as the function for determining the freshness. Further comprising category classification means for classifying the input document into any one or more of a plurality of categories according to the contents;
The similarity calculation means, said clustering means, the provisional topic degree calculating section, the document topic calculation means, the freshness determining means, the respective processing of the presentation data generating unit, classified performed by category Claim 4 or 5. The topic document presentation device according to 5 . A topic document presentation program for causing a computer to function as each means constituting the topic document presentation device according to any one of claims 4 to 6 .

Images