JP2012194961A - Device and method for calculating document significance - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device for calculating document significance.SOLUTION: The device for calculating the document significance according to the present invention includes: a semantic relation forming section configured to form a semantic relation between a target document in a target document set and an external document in an external document set; and a document significance calculating section configured to calculate a significance score of the target document based on the semantic relation.

Description

  The present invention relates to the field of information retrieval, and more particularly to an apparatus and method for calculating document importance.

  With the ever-increasing electronic information, a large amount of diverse information exists on different distributed systems. This causes great difficulty for the user to search for useful information.

  Information Retrieval (IR) technology can be used to retrieve specific information from a set of documents. The IR operation is further subdivided as follows. Search for information contained in a document, search for the document itself, search for metadata describing the document, databases (relational personal databases such as Ethernet or content / document management systems, hypertext network databases) Search text, audio, images or data from.

In the main bibliographic search process, given a query, the document is queried as a query-relevant
approach and query-query-irrelevant
ranked by using a combination of approaches). The query-related method measures the similarity between a query and a document. On the other hand, the query-unrelated method ranks documents in consideration of features that are not related to the degree of matching with a specific query. In practice, the use of query-irrelevant methods to calculate document importance plays an important role in both general document search engines and specific query / response or data mining systems.

  Conventional IR technology mainly uses internal information of a document to measure the similarity (query related score) between a query and a document. The web hyperlink structure plays an important role in web page ranking. As a specific example, PageRank uses the position of the page in the web graph structure to determine the importance of the page (query-irrelevant score).

Non-Patent Document 1 (“The PageRank citation ranking: Bringing order
to the web ”, L. Page, S. Brin, R. Motwani, and T. Winograd, Technical Report,
Stanford University, 1999) discloses how to assign importance levels to web pages. This method mainly includes the following steps. (1) Extract hyperlinks for a given set of web pages. (2) Build a link diagram where each page is considered a node and each hyperlink is considered a directional edge. (3) Ranking the importance of web pages. Here, the link diagram can be interpreted as a Markov chain in which web pages are regarded as states and links between pages are regarded as state transitions. By giving an initial probability distribution, it is possible to calculate a Markov chain stationary probability distribution.

  US Pat. No. 6,285,999 B1 describes a method for ranking nodes in a link database. The level of a document in the link database can be calculated based on the level of other documents that cite the document.

US Pat. No. 6,285,999 B1

“The PageRank citation ranking: Bringing order to theweb”, L. Page, S. Brin, R. Motwani, and T. Winograd, Technical Report, Stanford University, 1999

  However, the link diagram can be added because web content creators can optionally add or delete hyperlinks in the web so that website publishers can fake importance scores. It's not a reliable data source. In addition, for a general document set, there is no hyperlink relationship between documents. For this reason, the related art method cannot be applied to a general document set.

  In order to solve the above technical problem, the present invention utilizes a series of semantic associations between documents in the target document set and documents from external sources to calculate document importance. Specifically, according to the present invention, an external document set is used as an implicit knowledge source. First, a semantic association is formed between the target document and the external document. The importance score of the target document is then calculated based on the semantic association formed.

  The apparatus for calculating document importance of the present invention comprises a semantic association forming unit configured to form a semantic association between a target document in a target document set and an external document in an external document set, and a semantic And a document importance calculator configured to calculate an importance score for the target document based on the relevance.

  According to a preferred aspect, the semantic relation forming unit is configured to form a semantic relation between the target document and the external document by measuring text similarity between the target document and the external document.

  According to a preferred aspect, the semantic relation forming unit defines the target document and the external document as a class document and an instance document, respectively, and calculates a probability that the instance document is classified into the class document, thereby obtaining the target document and the external document. Configured to form a semantic relationship between.

  According to a preferred aspect, the document importance calculator is configured to calculate an importance score for the target document based on the number of external documents associated with the target document.

  According to a preferred aspect, the document importance calculator is a graph structure including the target document as a node, and each edge connecting two nodes in the graph structure is associated with two nodes connected by an edge. A graph structure having a weight determined by the number of external documents is generated and configured to calculate an importance score for the target document based on the sum of the edge weights of the target document.

  According to a preferred aspect, the document importance calculator is a graph structure including the target document as a node, and each edge connecting two nodes in the graph structure is associated with two nodes connected by an edge. Generate a graph structure with weights determined by the number of external documents, and calculate the importance score for the target document based on the sum of the edge weights of the target document and the number of external documents associated with the target document Composed.

  According to a preferred embodiment, the document importance calculation unit calculates an intermediate importance score of the target document based on the number of external documents associated with the target document, and calculates the intermediate importance score and the weight of the edge of the target document. It is configured to calculate an importance score for the target document based on the sum.

  According to a preferred aspect, it further includes a semantic association store configured to store the semantic association formed by the semantic association creator.

  The method for calculating document importance of the present invention includes the steps of forming a semantic relationship between a target document in a target document set and an external document in an external document set, and the importance of the target document based on the semantic relationship. Calculating a sex score.

  According to a preferred aspect, the forming step includes forming a semantic association between the target document and the external document by measuring text similarity between the target document and the external document.

  According to a preferred aspect, the forming step defines the target document and the external document as a class document and an instance document, respectively, and calculates the probability that the instance document is classified as a class document, thereby determining the target document and the external document. Forming a semantic association.

  According to a preferred aspect, the calculating step includes calculating an importance score for the target document based on the number of external documents associated with the target document.

  According to a preferred aspect, the calculating step is a graph structure including the target document as a node, wherein each edge connecting two nodes in the graph structure is an external document associated with the two nodes connected by the edge. Generating a graph structure having a weight determined by a number, and calculating an importance score for the target document based on a sum of the edge weights of the target document.

  According to a preferred aspect, the calculating step is a graph structure including the target document as a node, wherein each edge connecting two nodes in the graph structure is an external document associated with the two nodes connected by the edge. Generating a graph structure having a weight determined by a number, and calculating an importance score for the target document based on the sum of the edge weights of the target document and the number of external documents associated with the target document.

  According to a preferred aspect, the calculating step calculates an intermediate importance score for the target document based on the number of external documents associated with the target document, and is based on the sum of the intermediate importance score and the edge weight of the target document. Calculating an importance score for the target document.

  According to a preferred aspect, the method further includes storing a semantic association between the target document and the external document.

  According to the present invention, when there is no hyperlink between documents, the importance score of the document can be calculated, thereby improving the accuracy of the document search.

The above and other features of the present invention will become more apparent from the following detailed description with reference to the drawings.
It is a block diagram which shows the structure of the apparatus which calculates the document importance by one embodiment of this invention. It is a block diagram which shows the structure of the apparatus which calculates the document importance by other embodiment of this invention. 6 is a flowchart illustrating a method for calculating document importance according to another embodiment of the present invention.

  The principles and practice of the present invention will become more apparent from the following description of specific embodiments of the invention with reference to the drawings. However, the present invention is not limited to the specific embodiments described herein. Furthermore, in order to make it simple, description of the general element relevant to this invention is abbreviate | omitted.

First, some terms used in this specification will be described below with reference to Table 1.

  FIG. 1 shows an apparatus 10 for calculating document importance according to one embodiment of the present invention. As shown in FIG. 1, the apparatus 10 for calculating document importance includes a semantic relation formation unit 120 and a document importance calculation unit 130.

  The semantic relation forming unit 120 is configured to form a specific relationship between the target document of the target document set and the external document of the external document set as a semantic relation. The documents of the target document set and the external document set include information such as document number, title, author, time, phrase 1 and phrase 2, respectively. The target document set and external document set are stored in one or more external memories implemented by various types of memory, including volatile and non-volatile memory (eg, read only memory (ROM), flash memory). Is possible. However, it will also be appreciated that the target document set and the external document set are stored in one or more memories within the apparatus 10 for calculating document importance.

  In one embodiment, the semantic association can be formed based on the degree of association. Specifically, the semantic relation forming unit 120 uses the text similarity between the document of the target document set and the document of the external document set in order to measure the degree of relation between these two documents.

  In other embodiments, semantic associations can be formed based on instance relationships. Specifically, the semantic association forming unit 120 uses a classification method to form a class-instance relationship between two documents. That is, the documents in the target document set are regarded as class documents (templates), and the documents in the external document set are regarded as instance documents. Thereafter, the probability that the instance document is classified as a class document is calculated.

  The output of the semantic relation forming unit 120 is, for example, [relation degree (document number 1, document number 2)] or [instance relation (document number 1, document number 2)].

  The document importance calculator 130 is configured to calculate a document importance score based on the semantic association formed by the semantic association creator 120. According to the calculated document importance score, the document importance calculation unit 130 can rearrange the target documents in the target document set.

  In one embodiment, the document importance calculator 130 uses the number of related criteria (ie, the number of high related documents / instances in the external source) to calculate the importance score of the target document. This is referred to as document importance calculation based on the relevant basis number. Specifically, the document importance calculation unit 130 first calculates, for each document in the target document set, the number of documents / instances in the external source having high relevance to the document. Next, the document importance calculation unit 130 finds a document having the maximum number of highly related documents, divides the number of highly related documents of each target document by the maximum number, and acquires an importance score for the target document.

For example, assume that there are three target documents d1, d2, and d3. Accordingly, these three target documents have 10, 8 and 6 highly related documents / instances in the external source, respectively. Therefore, d1 is a document having the maximum number (10) of highly related documents. As a result, importance scores for d1, d2, and d3 are calculated as 10/10 = 1, 8/10 = 0.8, and 6/10 = 0.6, respectively.
As a result, the document importance ranking is d1>d2> d3.

In other embodiments, the document importance calculator 130 models the relationship between target documents and then calculates the document importance by graph analysis based on the semantic relationship between the target document and the external document. Generate a graph structure. This is called document importance calculation based on the graph structure.
Specifically, the document importance calculation unit 130 calculates the number of highly related documents / number of instances common to each pair of target documents. Thereafter, the document importance calculation unit 130 uses each target document as a node, and uses the number of common high-relevance documents / number of instances for each pair of target documents as a weight of an edge between the pair of target documents (nodes). As a result, a graph structure is formed. Next, the document importance calculation unit 130 calculates the sum of the edge weights of each node. Finally, the document importance calculation unit 130 finds the target document (node) having the maximum sum of the edge weights, and obtains the importance score of the target document by using the maximum sum of the edge weights. Divide the sum of the weights.

  For example, assume that there are three target documents d1, d2, and d3. Accordingly, the target documents d1 and d2 have a common highly relevant document number / instance number 5 in the external source. The target documents d2 and d3 have a common highly related document number / instance number 6 in the external source. Also, the target documents d1 and d3 have a common highly related document number / instance number 4 in the external source. In this case, the target document d1 has a sum of the weights of 5 + 4 = 9 sides as nodes. The target document d2 has a sum of weights of 5 + 6 = 11 sides as nodes. The target document d3 has 6 + 4 = 10 side weight sums as nodes. Therefore, the target document d2 is the target document having the maximum sum of edge weights (11). Accordingly, importance scores for d1, d2, and d3 are calculated as 9 / 11≈0.81, 11/11 = 1, 10 / 11≈0.9, respectively. Thereby, the document importance ranking is d2> d3> d1.

  In another embodiment, the document importance calculator 130 uses a combination of related reference numbers and graph structures to calculate a document importance score. The basic concept is that the use of either the associated reference number or the graph structure for calculating document importance (one-step calculation) may be unilateral to some extent. The related criterion number method only considers the number of highly related documents / instance documents for the target document. On the other hand, the graph structure method only considers a common number of subjects for two documents. In this case, if the obtained document importance scores are very close to each other, it may be necessary to further differentiate the importance of the document. For example, target documents whose importance scores are calculated by using one method (eg, number of related criteria) are clustered into multiple groups based on their importance scores (ie, absolute or relative Documents with importance scores that are relatively close in distance are grouped together), and then the next calculation (two-step calculation) is performed on the grouped documents by using another method (ie, graph structure). It is possible to apply. In this method, documents with very similar importance scores obtained by the first method are further classified for importance in order to comprehensively determine the importance of the target document. Here, the absolute distance between any two target documents is defined as the difference between the importance scores of the two target documents. The smaller the difference, the shorter the absolute distance. Furthermore, the relative distance between any two target documents is defined as the ratio of the difference between the importance scores of the two target documents to the maximum of the two importance scores. The smaller the percentage, the shorter the relative distance.

  In particular, target documents having importance scores calculated based on the relevant reference number are divided into two groups. All target documents in each group have the same importance score S1. S1 is the average importance score of all target documents in the group. At the same time, every target document has its own importance score S2 calculated based on the graph structure. Thereby, the importance score of a specific target document can be calculated as S = S1 × W + S2. Here, W is a predetermined coefficient value (for example, 10).

  As an example, assume there are three target documents d1, d2, and d3. Accordingly, these three target documents each have 10, 8 and 6 highly related documents / instances in the external source. Furthermore, the target documents d1 and d2 have 5 common highly related documents / instances in the external source, and the target documents d2 and d3 have 6 common highly related documents / instances in the external source. And the target documents d1 and d3 have four common highly related documents / instances in the external source.

  First, the document importance calculation unit 130 sets the importance scores of d1, d2, and d3 to 10/10 = 1, 8/10 = 0.8, and 6/10 = 0 based on the related reference numbers, respectively. .6. Further, the document importance calculation unit 130 calculates importance scores of d1, d2, and d3 as 9 / 11≈0.81, 11/11 = 1, and 10 / 11≈0.9 based on the graph structure. To do.

  Thereafter, the documents d1, d2, and d3 are divided into two groups according to the calculation result based on the related reference number. The first group includes d1 and d2, and the second group includes d3. A specific grouping process is as follows. First, d1 having the highest importance score is distributed to the first group, and d3 having the lowest importance score is distributed to the second group. After that, since the relative distance between d2 and d1 ((1-0.8) /1=0.2) is shorter than the relative distance between d2 and d3 ((0.8-0.6) /0.8=0.25), d2 Are distributed to the first group.

As a result, all target documents in the first group have an importance score S1 = (1 + 0.8) /2=0.9, and all target documents in the second group have an importance score S2 = 0. 6. Therefore, the importance scores for d1, d2, and d3 are calculated as follows in the combined method:
Important score for d1: 0.9 × 10 + 0.81 = 9.81
Important score of d2: 0.9 × 10 + 1 = 10
Important score for d3: 0.6 × 10 + 0.9 = 6.9
Therefore, the final ranking of document importance is d2>d1> d3.

  If the above combined calculation does not satisfy a given requirement (eg, there are still two or more target documents with the same or similar importance score), the above principle of the combined calculation satisfies the requirement. Until then, it will be readily appreciated by those skilled in the art that the target document can be subdivided into three, four or more groups for calculation.

  FIG. 2 shows an apparatus 20 for calculating document importance according to another embodiment of the present invention. As shown in FIG. 2, the apparatus 20 for calculating document importance is the same as the semantic relation formation unit 120 and the document importance calculation unit 130 shown in FIG. Each unit 230 is provided. Furthermore, the apparatus 20 for calculating document importance further comprises a semantic relation storage 240.

  The semantic association storage unit 240 is configured to store the semantic association formed by the semantic association formation unit 220. For example, the semantic relation storage unit 240 indicates a specific relation (semantic relation) between a document from the target document set and a document from the external document set by [relation degree (document number 1, document number 2)] or [ Instance relationship (document number 1, document number 2)] is stored. In addition, the semantic relation storage unit 240 can be realized by various types of memories including volatile and nonvolatile memories (for example, read-only memory (ROM), flash memory, and the like).

  The use of the semantic relation storage unit 240 is effective in the performance of storing the formed semantic relation. This eliminates the need to recreate semantic associations when recalculating document importance, thus eliminating many actions.

  FIG. 3 is a flowchart illustrating a method 30 for calculating document importance according to another embodiment of the present invention. The method 30 starts at step S310.

  In step S330, a specific relationship between the target document of the target document set and the external document of the external document set is formed as a semantic relationship. Here, the external document set is used as an implicit knowledge source for calculating the importance of the target document in the target document set. Specifically, text similarity between two documents can be used to measure the degree of association between the two documents, and a semantic association can be formed based on the degree of association. In addition, a classification method is used to form a class-instance relationship between two documents, and a semantic association can be formed based on the instance relationship. That is, the documents in the target document set are regarded as class documents (templates), and the documents in the external document set are regarded as instance documents. Thereafter, the probability that the instance document is classified as a class document is calculated.

  In optional step S340, the semantic association formed in step S330 can be stored in memory. This eliminates the need to re-execute step S330 when recalculating document importance. Thereafter, the method 30 proceeds to step S350. If S340 is omitted, the method 30 proceeds immediately to step S350 after step S330.

  In step S350, a document importance score is calculated based on the semantic association. In this step, a one-step calculation (e.g., document importance calculation based on the relevant reference number or document importance calculation based on the graph structure described above) or two-step calculation (referred to as a combination calculation, i.e. related reference number and graph). Document importance calculation based on a combination of structures can be used. The target documents in the target document set are then sorted based on the calculated document importance score.

  Finally, method 30 ends at step S360.

  According to the present invention, when there is no hyperlink between documents, the importance score of the document can be calculated and the documents can be ranked based on the importance score, thereby improving the accuracy of the literature search. .

  Although the present invention has been described with reference to preferred embodiments thereof, various other modifications, changes and additions can be made by those skilled in the art without departing from the spirit and scope of the present invention. It will be clear that this is possible. Accordingly, the scope of the invention is not limited to the specific embodiments described above, but only by the appended claims.

  Further, a part or all of the above-described embodiment can be described as in the following supplementary notes, but is not limited thereto.

(Appendix 1)
A semantic association formation unit configured to form a semantic association between a target document in the target document set and an external document in the external document set, and the importance of the target document based on the semantic association A document importance calculator configured to calculate a score and
An apparatus for calculating document importance, comprising:

(Appendix 2)
The semantic relation forming unit is configured to form a semantic relation between the target document and the external document by measuring a text similarity between the target document and the external document. An apparatus for calculating document importance according to appendix 1.

(Appendix 3)
The semantic relation forming unit defines the target document and the external document as a class document and an instance document, respectively, and calculates a probability that the instance document is classified into the class document, thereby calculating the target document and the external document. The apparatus of claim 1, wherein the apparatus is configured to form a semantic association between documents.

(Appendix 4)
The document importance of claim 1, wherein the document importance calculator is configured to calculate an importance score of the target document based on the number of the external documents associated with the target document. Device to calculate.

(Appendix 5)
The document importance calculation unit is a graph structure including the target document as a node, and each edge connecting two nodes in the graph structure is an external document associated with the two nodes connected by the edge Appendix 1 is configured to generate a graph structure having weights determined by the number of and to calculate an importance score for the target document based on a sum of edge weights of the target document A device that calculates document importance as described in.

(Appendix 6)
The document importance calculation unit is a graph structure including the target document as a node, and each edge connecting two nodes in the graph structure is an external document associated with the two nodes connected by the edge Generating a graph structure having a weight determined by a number of and calculating an importance score for the target document based on a sum of edge weights of the target document and the number of external documents associated with the target document The apparatus for calculating document importance as set forth in appendix 1, wherein the apparatus is configured.

(Appendix 7)
The document importance calculation unit calculates an intermediate importance score of the target document based on the number of external documents associated with the target document, and calculates a sum of the intermediate importance score and the edge weight of the target document. The apparatus for calculating document importance according to claim 6, wherein the apparatus is configured to calculate an importance score of the target document based on the document.

(Appendix 8)
The apparatus of claim 1, further comprising a semantic association store configured to store a semantic association formed by the semantic association formation unit.

(Appendix 9)
Forming a semantic association between a target document in the target document set and an external document in the external document set; calculating an importance score for the target document based on the semantic association;
A method of calculating document importance, characterized by comprising:

(Appendix 10)
The supplementary note 9, wherein the forming step includes the step of forming a semantic relationship between the target document and the external document by measuring a text similarity between the target document and the external document. How to calculate the document importance of.

(Appendix 11)
The forming step defines the target document and the external document as a class document and an instance document, respectively, and calculates a probability that the instance document is classified as the class document, thereby forming the target document and the external document. The method of calculating document importance according to claim 9 including the step of forming a semantic association of:

(Appendix 12)
The method of claim 9, wherein the calculating step includes calculating an importance score for the target document based on the number of the external documents associated with the target document.

(Appendix 13)
The calculating step includes a graph structure including the target document as a node, and each edge connecting two nodes in the graph structure is determined by the number of external documents associated with the two nodes connected by the edge. The document importance of claim 9 including the step of generating a graph structure having determined weights and calculating an importance score for the target document based on a sum of edge weights of the target document. How to calculate gender.

(Appendix 14)
The calculating step includes a graph structure including the target document as a node, and each edge connecting two nodes in the graph structure is determined by the number of external documents associated with the two nodes connected by the edge. Generating a graph structure having determined weights, and calculating an importance score for the target document based on a sum of edge weights of the target document and the number of external documents associated with the target document. A method of calculating document importance according to appendix 9, which is characterized.

(Appendix 15)
The calculating step calculates an intermediate importance score of the target document based on the number of external documents associated with the target document, and sets the target importance based on the sum of the intermediate importance score and the weight of the edge of the target document. 15. The method for calculating document importance according to appendix 14, comprising the step of calculating a document importance score.

(Appendix 16)
The method of claim 9, further comprising the step of storing a semantic association between the target document and the external document.

10: Device for calculating document importance 120: Semantic association forming unit 130: Document importance calculating unit 20: Device for calculating document importance 220: Semantic association forming unit 230: Document importance calculating unit 240: Semantic Related storage

Claims (10)

A semantic association generator configured to form a semantic association between a target document in the target document set and an external document in the external document set;
An apparatus for calculating document importance, comprising: a document importance calculator configured to calculate an importance score of the target document based on the semantic association.   The semantic relation forming unit is configured to form a semantic relation between the target document and the external document by measuring a text similarity between the target document and the external document. An apparatus for calculating document importance according to claim 1.   The semantic relation forming unit defines the target document and the external document as a class document and an instance document, respectively, and calculates a probability that the instance document is classified into the class document, thereby calculating the target document and the external document. The apparatus for calculating document importance of claim 1, wherein the apparatus is configured to form a semantic association between documents.   The document importance of claim 1, wherein the document importance calculator is configured to calculate an importance score of the target document based on a number of the external documents associated with the target document. A device that calculates gender.   The document importance calculation unit is a graph structure including the target document as a node, and each edge connecting two nodes in the graph structure is an external document associated with the two nodes connected by the edge Generating a graph structure having a weight determined by a number of and calculating an importance score for the target document based on a sum of edge weights of the target document. An apparatus for calculating document importance according to 1.   The document importance calculation unit is a graph structure including the target document as a node, and each edge connecting two nodes in the graph structure is an external document associated with the two nodes connected by the edge Generating a graph structure having a weight determined by a number of and calculating an importance score for the target document based on a sum of edge weights of the target document and the number of external documents associated with the target document The apparatus of claim 1, wherein the apparatus calculates document importance.   The document importance calculation unit calculates an intermediate importance score of the target document based on the number of external documents associated with the target document, and calculates a sum of the intermediate importance score and the edge weight of the target document. 7. The apparatus for calculating document importance of claim 6, wherein the apparatus is configured to calculate an importance score for a target document based on the document.   The apparatus of claim 1, further comprising a semantic association store configured to store a semantic association formed by the semantic association creator. Forming a semantic association between a target document in the target document set and an external document in the external document set;
Calculating the importance score of the target document based on the semantic association.   The method of claim 9, wherein the forming step includes forming a semantic association between the target document and the external document by measuring text similarity between the target document and the external document. How to calculate the document importance described.

Images