JP3604069B2 - Apparatus for calculating relevance between documents, method therefor, and recording medium therefor - Google Patents

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an inter-document relevance calculation apparatus and method for calculating the relevancy between documents, and more particularly, to an inter-document relation calculating the relevancy between documents referenced by hyperlinks according to hyperlinks. The present invention relates to a degree calculation device and method.
[0002]
[Prior art]
By calculating the degree of relevance between documents, documents related to the document specified by the user can be efficiently searched, and a large number of documents can be classified and used. In order to realize this, a method of calculating the degree of relevance between documents has been conventionally proposed.
[0003]
For example, a document (G. Salton, “Automatic Text Processing”, Addison Wesley, Reading, Mass, 1989) discloses that the relevance between documents is calculated based on the frequency of words included in each document. I have. That is, if two frequently appearing words are similar to each other, the degree of relevance is high. Conversely, if the frequently appearing words are low in similarity, the degree of relevance is low. Think.
[0004]
However, even when describing the same concept, there are cases where synonyms are used and languages such as Japanese and English are different, so the accuracy of inter-document relevance obtained by statistical processing of words is Not necessarily high.
[0005]
Furthermore, the relevance between documents is not determined only by the similarity of the words used, but can be defined from various viewpoints.
[0006]
Therefore, when providing a service such as a related document search, it is necessary to find a document set that many people recognize as being related to each other.
[0007]
[Problems to be solved by the invention]
However, in the conventional example, there is a problem in that it is not possible to find the inter-document relevance recognized by many people as described above.
[0008]
For example, over 300 million pages of WWW documents are published on the Internet (S. Lawrence and CL Giles, "Searching the World Wide Web", Science, Vol. 280, No. 5360, p. 98, 1998). Many users create and publish links that link documents that match their interests and are accessed many times from their own perspective. In other words, it can be considered that the documents included in the link collection have a high quality to some extent, and that the relevance from a predetermined viewpoint is defined. Therefore, by collecting these relevances, a high-quality related document set can be created. As a result, it is possible to provide an effective service such as a related page search.
[0009]
However, in the above-described conventional example, there is a problem that it is not possible to calculate the degree of relevance between documents in a form closer to a human viewpoint.
[0010]
The present invention provides an inter-document relevance calculation apparatus and method for calculating the relevance between documents in a form closer to the human viewpoint by totalizing related document sets described in a link collection or the like created by a human. The purpose is to provide.
[0011]
[Means for Solving the Problems]
The present invention extracts the hyperlink described in each document when calculating the relevance between a plurality of documents referred to by a hyperlink such as a URL in a reference source document, and extracts the extracted hyperlink. The relevance between each of the documents linked by the hyperlinks is calculated, and the calculated relevance is totaled.
[0012]
Embodiments and Examples of the Invention
FIG. 1 is a block diagram showing an inter-document relevance calculating apparatus 100 according to an embodiment of the present invention.
[0013]
The inter-document relevance calculation device 100 is a device that calculates the relevance between a plurality of documents referenced by a hyperlink such as a URL in a reference source document. The system includes a memory 11, a URL extracting unit 20, a memory 21 for increase / decrease rules, an inter-document relevance calculating unit 30, an inter-document totaling unit 40, and an inter-document relevance memory 41.
[0014]
Here, in the above embodiment, it is assumed that the document whose relevance is to be calculated is a document described in HTML (Hyper Text Markup Language), and the linked document position (document storage location) is URL. (Uniform Resource Locator).
[0015]
The document selecting means 10 is a means for selecting a desired document from the HTML documents stored in the HTML document collection memory 11.
[0016]
The HTML document set memory 11 is a memory in which a large number of HTML document sets are stored.
[0017]
The URL extracting unit 20 is an example of a hyperlink extracting unit that extracts a hyperlink described in each document. In the above embodiment, the URL extracting unit 20 is a unit that extracts a URL from each document.
[0018]
The increase / decrease rule memory 21 is a memory that stores data of an increase / decrease rule for increasing / decreasing a distance according to the type of tag.
[0019]
The inter-document relevance calculating unit 30 is a unit that calculates the inter-document relevance between each of a plurality of documents linked by the URL extracted by the URL extracting unit 20.
[0020]
The inter-document counting means 40 is a means for counting the inter-document relevance calculated by the inter-document relevance calculation means 30.
[0021]
The inter-document relevance memory 41 is a memory in which inter-document relevance is stored.
[0022]
FIG. 2 is a flowchart showing the operation of the inter-document relevance calculation device 100.
[0023]
First, an HTML document to be processed is selected (S1). The URL described in the selected HTML document and the storage location of the document referred to by the URL are extracted (S2), and the degree of association between the extracted URLs is determined. That is, the degree of relevance between the document referenced by one URL and the document referenced by another URL is determined (S3). Then, the degree of association between the URLs is totaled. That is, the calculated inter-document relevance is counted (S4). Then, the above processing (S1 to S4) is repeated until the above calculation is completed for all HTML documents (S5).
[0024]
The above embodiment can be understood as a recording medium invention. That is, in the above embodiment, the document selection procedure for selecting the HTML document to be processed, the hyperlink displayed in the selected HTML document, and the display for displaying the hyperlink in the selected HTML document An extraction procedure for extracting the position of the hyperlink in the application description, and a process for extracting a document referenced by the extracted one hyperlink and a document referenced by the extracted other hyperlink. It is an example of a computer-readable recording medium in which a program for causing a computer to execute the inter-document relevance calculation procedure for calculating the inter-document relevance and the counting procedure for summing the calculated inter-document relevance is described.
[0025]
In this case, the recording medium may be an FD, a CD, a DVD, a semiconductor memory, or the like.
[0026]
FIG. 3 is a diagram showing a description example of an HTML document used in the above embodiment.
[0027]
FIG. 4 is a diagram showing an example in which the HTML document shown in FIG. 3 is displayed on a browser.
[0028]
As shown in FIG. 3, an HTML document is a mixture of tags such as “<HEAD>” and “</ HEAD>”, which start with “<” and end with “>”, and normal text data. is there.
[0029]
A hyperlink to another document is represented, for example, as “<A HREF=“URL1”> document 1 </A>” described in the twelfth line in FIG. That is,
(1) A tag “<A>” indicating a hyperlink, ““ URL1 ”” indicating a storage location of a linked document,
(2) A display text “document 1” of the hyperlinked document,
(3) A tag “</A>” indicating the end of the description of the hyperlink indicates a hyperlink to another document.
[0030]
In step S2 shown in FIG. 2, a storage location (that is, a URL) of another document described as a hyperlink and a display text corresponding to the URL are extracted from the HTML document.
[0031]
In the HTML document shown in FIG. 3, hyperlinks are described on the twelfth, thirteenth, seventeenth, and eighteenth lines. From this HTML document, URL1, URL2, URL11, and URL12 are extracted as storage locations of other documents.
[0032]
On the other hand, the texts referred to by the hyperlinks are document 1, document 2, document 11, and document 12, respectively. The position where these (display) texts are described is determined according to a predetermined position calculation rule. The position calculation rule in the above embodiment is the number of bytes from the first part of the HTML document to the position where the text is described. In the description of FIG. 3, the first line starts from the first byte.
[0033]
When counted in this way, the description positions of Document 1, Document 2, Document 11, and Document 12 are 129, 158, 215, and 246, respectively, as shown in FIG.
[0034]
In the above embodiment, the rule for calculating the description position of the display text of each link is the number of bytes from the first part of the HTML document to the position where the text is described. May be used.
[0035]
For example, a rule may be adopted in which the number of bytes excluding the tag from the first part of the HTML document to the position where the text is described is the description position of the display text of each link. You may. According to this rule, the description positions of Document 1, Document 2, Document 11, and Document 12 are 44, 50, 65, and 72, respectively, as shown in FIG.
[0036]
Further, an increase / decrease rule for increasing / decreasing the number of bytes from the first part of the HTML document to the position where the text is described is determined according to the type of tag, and the text is described in consideration of the increase / decrease rule. The number of bytes up to the position may be obtained.
[0037]
FIG. 5 is a diagram showing an example of an increase / decrease rule according to the type of tag in the above embodiment.
[0038]
In FIG. 5, the <HR> tag is +100, the <UL> tag and the <H1> tag are +50, the <H2> tag is +30, and the other tags do not increase or decrease. As a result, for example, the related document collection described in the eighth line in FIG. 3 is described in the number of bytes excluding the tag, the description position is 24. However, when the increase / decrease rule is applied, <H1> Since "50" is added later, the description position is 74.
[0039]
As described above, by applying the increase / decrease rule, the description positions of document 1, document 2, document 11, and document 12 are 274, 280, 375, and 382, respectively, as shown in FIG.
[0040]
In addition, the description position can be represented by the number of the displayed line when displayed on a browser. In this case, the description positions of Document 1, Document 2, Document 11, and Document 12 are 4, 5, 8, and 9, respectively, as shown in FIG.
[0041]
In step S3 in FIG. 2, based on each URL extracted in step S2 and the position of the URL in the display description for displaying the URL, the degree of association between the URLs (that is, the URL between the documents referenced by the URL) Relevance). In the above embodiment, the degree of association is the reciprocal of the difference between the description positions of the display text.
[0042]
FIG. 6 is a diagram illustrating the degree of association between URLs calculated in the above embodiment.
[0043]
As described above, the inter-document relevance is obtained from the reciprocal of the difference between the description positions of the display text, and the relevancy between the URL1, URL2, URL11, and URL12 thus obtained is as shown in FIG. Is calculated.
[0044]
As a method of calculating the degree of association between URLs, a method based on the description position of the display text may be adopted other than the method of using the reciprocal of the square of the difference between the description positions of the display text.
[0045]
In step S4 in FIG. 3, the relevance between URLs calculated for each HTML document is totaled.
[0046]
By executing steps S1 to S4 for all target HTML documents, the degree of relevance between URLs, that is, the degree of relevance between HTML documents indicated by the URLs can be obtained.
[0047]
The degree of relevancy obtained in this way has a high degree of relevance if it is described close to each other in many HTML documents, even if the original HTML documents are not directly linked by hyperlinks. become. Therefore, in the case of a WWW document on the Internet in which many users create related link collections from various viewpoints, it is possible to provide a service such as a related page search by counting the relatedness. Convenience can be significantly improved.
[0048]
FIG. 7 is an operation explanatory diagram of steps S2 and S3 in the above embodiment.
[0049]
FIG. 7A is a diagram illustrating each extracted link and the description position of the display text. FIG. 7B is a diagram illustrating the reciprocal of the difference between the description positions of the texts referred to by the two links. Is a diagram showing the degree of relevance between documents. FIG. 7 is the same in content as FIG.
[0050]
FIG. 8 is a flowchart illustrating a specific example of calculating the inter-document relevance, which is the reciprocal of the difference between the description positions of the texts referred to by two links in the above embodiment.
[0051]
The number of input links is set to N (N = 4 in the example shown in FIG. 7), each link is set to LINK [i] (i = 1, 2, 3, 4 in the example shown in FIG. 7), and each link is set. Is described as POS [i], and the degree of association between LINK [i] and LINK [j] is REL [i, j].
[0052]
In FIG. 8, the link order i is set to 1 (S11), and if this i reaches N-1 (S12), the link is output. (S13), the degree of association REL [i, j] between LINK [i] and LINK [j] is calculated as the reciprocal of the difference (S14), and the order j of the link of the partner is incremented by 1 (S15). If the order j of the link of the partner is N or less (S16), the above processing (S14, S15) is repeated (S16). If the link order j of the partner is larger than N (S16), the link order i is incremented by 1 (S17), and the process returns to step S12.
[0053]
In other words, one HTML document is selected from a plurality of HTML documents, the hyperlink displayed in the selected HTML document is extracted, and the hyperlink displayed in the selected HTML document is displayed in the display description for displaying the hyperlink. After extracting the position of the hyperlink, the inter-document association between the document referenced by the extracted first hyperlink and the document referenced by the extracted second hyperlink The degree is calculated by the inter-document relevance calculating procedure. In the above embodiment, as an example of the inter-document relevance calculating means, the number of input hyperlinks is N, and each hyperlink is LINK [i]. The description position of the display text of each hyperlink is POS [i], and the degree of association between LINK [i] and LINK [j] And REL [i, j], LINK [i] and LINK [j] and the relevance REL [i, j] to be calculated as the reciprocal of the difference.
[0054]
That is, the position of the hyperlink in the display description for displaying the first hyperlink in the selected HTML document, and the display description for displaying the second hyperlink in the selected HTML document. The reciprocal of the difference from the position of the hyperlink in is calculated as the inter-document relevance.
[0055]
FIG. 9 is a diagram for explaining another operation of steps S2 and S3 in the above embodiment.
[0056]
FIG. 9A is a diagram illustrating an example of a case where the same link is present among a plurality of input links. That is, there are two URL1s. In this case, the maximum value of the positions of the two texts referred to by the two links is adopted. The average value of the positions of the two texts referred to by the two links may be adopted.
[0057]
FIG. 9B is a diagram showing, as the inter-document relevance, the reciprocal of the difference between the description positions of the texts referred to by the two links in the case shown in FIG. 9A.
[0058]
FIG. 10 shows a case where the same link is present among a plurality of input links in the above embodiment, and the maximum value of the positions of the two texts referred to by the two links is adopted. It is a flowchart which is calculated | required.
[0059]
The flowchart shown in FIG. 10 is basically the same as the flowchart shown in FIG. 8, except that steps S21 to S24 are provided instead of step S14 in the flowchart shown in FIG.
[0060]
Note that LINK [1] = LINK [3], and a unique ID determined from the link name of each link is ID [link name]. Also, the degree of association between ID [LINK [i]] and ID [LINK [j]] is REL [ID [LINK [i]], ID [LINK [j]]], and max (a, b) Is the smaller value of a and b.
[0061]
That is, after setting the order j of the link of the partner for which the document relevance is determined to be n + 1 (S13), the reciprocal R of the difference between LINK [i] and LINK [j] is determined (S21), and REL [ID [LINK [i] ]], ID [LINK [j]]] is completed (S22), and max (REL [ID [LINK [i]], ID [LINK [j]], R) is converted to REL [ ID [LINK [i]], ID [LINK [j]]] (S23), while if the calculation of REL [ID [LINK [i]], ID [LINK [j]]] has not been completed (S23). S22), and let R be REL [ID [LINK [i]], ID [LINK [j]]] (S24).
.
[0062]
That is, when there are two first hyperlinks, the maximum value or the average value of the positions of the hyperlinks in the display description for displaying the first hyperlink is determined by Position.
[0063]
In the above-described embodiment, a description method using HTML is adopted as a method for describing a document. However, another description language may be used as long as it is a language capable of describing the relationship between documents. This description language includes, for example, XML (extensible Markup Language).
[0064]
Note that the above embodiment is an example in which documents to be totaled are collected in advance and stored in a database or the like. May be calculated.
[0065]
According to the above embodiment, first, a hyperlink described in each document is extracted, and the hyperlink displays a document location (URL) for uniquely specifying a linked document and the link on the screen. When extracting a hyperlink, the document location (URL) and the position of the display description in the document are extracted. Next, the degree of association between the extracted document locations (URLs) is calculated based on the position of the display description in the document. At this time, the relevance between the documents intended by the creator of the document is determined by calculating such that the closer the description position is, the higher the relevance is, and the further the description position is, the lower the relevance is. Finally, the calculated inter-document relevance can be summed up to finally obtain the relevancy between the documents.
[0066]
【The invention's effect】
According to the present invention, the relevance between documents is calculated by totalizing a link collection or the like described by a human, so that the relevance can be obtained in a form closer to the human viewpoint. The effect is that the operability of the information providing system such as presentation and search is extremely improved.
[Brief description of the drawings]
FIG. 1 is a block diagram showing an inter-document relevance calculating apparatus 100 according to an embodiment of the present invention.
FIG. 2 is a flowchart showing an operation of the inter-document relevance calculating apparatus 100;
FIG. 3 is a diagram showing a description example of an HTML document used in the embodiment.
FIG. 4 is a diagram showing an example in which the HTML document shown in FIG. 3 is displayed on a browser.
FIG. 5 is a diagram showing an example of an increase / decrease rule according to a tag type in the embodiment.
FIG. 6 is a diagram showing a degree of association between URLs calculated in the embodiment.
FIG. 7 is an operation explanatory diagram of steps S2 and S3 in the embodiment.
FIG. 8 is a flowchart illustrating a specific example of calculating the inter-document relevance, which is the reciprocal of the difference between the description positions of texts referred to by two links in the embodiment.
FIG. 9 is another operation explanatory view of steps S2 and S3 in the embodiment.
FIG. 10 shows a case where the same link is present among a plurality of input links in the above embodiment and the maximum value of the positions of the two texts referred to by the two links is adopted. It is a flowchart which is calculated | required.
[Explanation of symbols]
10. Document selection means
11. HTML document collection memory
20 URL extraction means,
21: memory for increase / decrease rules,
30 ... Document relevance calculating means
40 ... means for counting between documents
41: Memory for the degree of association between documents.

Claims (15)

An inter-document relevance calculator that calculates a relevance between a plurality of documents referenced by a hyperlink in at least one document of a reference source ,
Each hyperlink described in the above reference source in a document, and hyperlinks extracting means for extracting a description location of each hyperlink in the referencing document;
The closer the description position of the predetermined hyperlink described in the reference document and the description position of another hyperlink described in the reference document are closer to each other, the higher the degree of association is, and as both described position is far from one another, by the degree-of-association calculation rules to be less relevance, the document relevancy calculating means which calculates a degree of association between the document;
An inter-document relevance counting means for summing up all the relevances between the documents obtained by the inter-document relevance calculation means for each of the plurality of documents referenced by the hyperlinks in the reference source document ;
An inter-document relevance calculation apparatus characterized by having: In claim 1,
The relevance calculation rule is a reciprocal of a difference between a description position of a predetermined hyperlink described in the reference source document and a description position of another hyperlink described in the reference source document. The inter-document relevance calculating apparatus is characterized in that the relevancy is a rule . In claim 1 or claim 2 ,
The description position of the hyperlink described in the reference source document is the number of bytes from the head of the document in the reference source document to the position where the hyperlink is described. Inter-association degree calculation device. In claim 1 or claim 2 ,
The description position of the hyperlink described in the reference source document is the number of bytes excluding the tag from the head of the document in the reference source document to the position where the hyperlink is described. Characteristic inter-document relevance calculator. In claim 1 or claim 2 ,
According to the type of tag appearing in the document of the reference source, an increase / decrease rule for increasing / decreasing the number of bytes of the tag is defined,
The description position of the hyperlink described in the above referenced document is
For tags that exist in the number of bytes from the beginning of the source document to the position where the hyperlink is described, and between the head of the source document and the position where the hyperlink is described An inter-document relevance calculation device, characterized in that it is a position obtained by adding the number of bytes to which the increase / decrease rule is applied . In claim 1 or claim 2 ,
The description position of the hyperlink described in the above referenced document is
An inter-document relevance calculation apparatus , wherein the number of lines on which the hyperlink is displayed when the reference source document is displayed according to a predetermined display rule . What is claimed is: 1. A control method for an inter-document relevance calculating apparatus for calculating a relevance between a plurality of documents referenced by a hyperlink in at least one document as a reference source, comprising: a hyperlink extracting unit; In a control method of an inter-document relevance calculating device having a calculating unit and an inter-document relevance totaling unit,
Extracting the hyperlinks described in the reference source document and the description positions of the hyperlinks in the reference source document;
The inter-document relevance calculating means determines that a description position of a predetermined hyperlink described in the reference source document is close to a description position of another hyperlink described in the reference source document. Determining the relevance between the documents by a relevance calculation rule that determines that the higher the relevance is, the lower the two description positions are, and the lower the relevance is.
The inter-document relevance summing means sums up all the inter-document relevance calculated by the inter-document relevance calculation means for each of a plurality of documents referenced by hyperlinks in the reference source document. Storing in a memory;
A method for controlling an inter-document relevance calculating apparatus, comprising: In claim 7,
The relevance calculation rule is a reciprocal of a difference between a description position of a predetermined hyperlink described in the reference source document and a description position of another hyperlink described in the reference source document. The control method of the inter-document relevance calculating device, wherein the rule is a rule that sets the relevance . In claim 7 or claim 8 ,
The description position of the hyperlink described in the reference source document is the number of bytes from the head of the document in the reference source document to the position where the hyperlink is described. A control method of the inter-association degree calculation device . In claim 7 or claim 8 ,
The description position of the hyperlink described in the reference source document is the number of bytes excluding the tag from the head of the document in the reference source document to the position where the hyperlink is described. A control method of an inter-document relevance calculation apparatus . In claim 7 or claim 8 ,
According to the type of tag appearing in the document of the reference source, an increase / decrease rule for increasing / decreasing the number of bytes of the tag is defined,
The description position of the hyperlink described in the above referenced document is
For tags that exist in the number of bytes from the beginning of the source document to the position where the hyperlink is described, and between the head of the source document and the position where the hyperlink is described A control method of an inter-document relevance calculating apparatus, characterized in that the position is a position obtained by adding the number of bytes to which the increase / decrease rule is applied . In claim 7 or claim 8 ,
The description position of the hyperlink described in the above referenced document is
A control method for an inter-document relevance calculating apparatus, wherein when the reference source document is displayed according to a predetermined display rule, the number of lines in which the hyperlink is displayed is provided . A program for causing a computer to function as a hyperlink extracting unit, an inter-document relevance calculating unit, and an inter-document relevance counting unit, and which is referred to by a hyperlink in at least one referencing document. A computer-readable storage medium storing a program for calculating the degree of relevance between a plurality of documents,
A procedure in which the hyperlink extracting means extracts each hyperlink described in the reference source document and a description position of each hyperlink in the reference source document;
The inter-document relevance calculating means determines that a description position of a predetermined hyperlink described in the reference source document is close to a description position of another hyperlink described in the reference source document. Determining the degree of relevance between the documents according to a relevance calculation rule that determines that the degree of relevance is higher and the distance between the two description positions is farther from each other;
The inter-document relevance summing means sums up all the inter-document relevance calculated by the inter-document relevance calculation means for each of a plurality of documents referenced by hyperlinks in the reference source document. Storing the information in a memory;
Computer-readable recording medium storing a program for causing a computer to execute the program. In claim 13,
The procedure for determining the relevance between the above documents is as follows:
The position of the hyperlink in the display description that displays the first hyperlink in the reference source document, and the hyperlink in the display description that displays the second hyperlink in the reference source document A computer-readable recording medium, which is a procedure for obtaining a reciprocal of a difference from a position of the document as the inter-document relevance. In claim 14,
When two of the first hyperlinks are present, the maximum value or the average value of the positions of the hyperlinks in the display description for displaying the first hyperlink is calculated as the position of the hyperlinks. A computer-readable recording medium characterized by the following.

Images