JP2002259407A - Document collection device for particular application, its method and program for execution with computer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To permit the quick collection of a document for a particular application through a network. SOLUTION: In the document collection device for collecting, through a network, a document for a community, prior to the collection start, an initial document group 20 that is to be the start point of the collection is provided. A reference relation extracting means 3, from the initial document group 20, extracts a reference relation. A next candidate determining means 5, on the basis of the reference relation for the collected document group 20, determines the uncollected document that satisfies a fixed condition as a next collection candidate 21 that is a candidate of the document to be collected subsequently. A document collection means 2 collects the document that has been determined as the next collection candidate and adds it to the collected document 20. Unless the number of documents in the collected document group 20 is equal to or more than a prescribed number, the reference relation extracting means 3, from the newly collected documents, further extracts a reference relation to repeat the above processing. In this way, the document collection is repeated until the number of the documents in the collected document group 20 is equal to or more than the prescribed value.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to document collection, and more particularly, to a document collection apparatus and a document collection method for efficiently collecting documents according to specific applications.

[0002]

2. Description of the Related Art A search engine for documents on a network such as an intranet or WWW is realized by a document collection device (robot) for collecting documents from the network and a search engine for creating a keyword index for the collected documents. .

A document collection device is provided with a given material URL (Unif
orm Resorce Locator) (starting collection when starting collection) starts document collection, and collects uncollected documents referenced by anchor (reference relationship) from collected documents as next collection candidates The operation is performed by repeating such a process a certain number of times. In this way, the document collection robot periodically collects documents from tens of millions to hundreds of millions of URLs. Here, the URL refers to a description method for specifying the location of the information on the network and the method of obtaining the information.

[0004] By the way, today, the number of documents on the network is increasing rapidly, and in January 2000, Inktomi et al. Published a survey result that the number of unique documents on the Internet reached one billion. In July 2000, a report from Cyveillance of the United States announced that the size of the Internet was about 2.1 billion documents, and is expected to double in 2001.

When it comes to collecting documents from one billion URLs, one million URLs per day (about 10 URLs per second = 40
Even if it has been collected, it will take three years to complete the collection, and by the time collection is completed, information on documents collected in the early days will become obsolete. Therefore, there has been a demand for an intelligent document collection device that efficiently collects only information having high importance in accordance with the application.

There are the following document collecting apparatuses for collecting documents of a specific use with priority. -For example, as in the invention disclosed in JP-A-9-311802,
Collect new information first.・ Collect documents whose contents are considered to be similar. At that time, the following concept is introduced.

A) The collection range is limited by the number of layers. For example, as in the invention disclosed in Japanese Patent Laid-Open No. 9-218876, a document having a reference relationship is considered to be similar in content,
The concept of collecting documents by limiting the collection range by the number of layers is because there is no meaningful connection if the layers are too far apart.

[0008] b) Only documents having similar semantic contents are collected. For example, as in the invention disclosed in JP-A-10-105572,
The idea is to calculate the semantic closeness from the matching of the contents of the documents and collect only the semantically close documents from the documents having a reference relationship.

C) Only documents whose character strings indicating the reference destination are appropriate are collected. For example, as in the inventions disclosed in JP-A-10-260979 and JP-A-2000-9011, a reference expression that is an expression representing a reference destination, for example, in the case of HTML, the reference expression based on the content of an anchor tag is used. The idea is to determine whether to collect the next referenced document referenced by. -Generally, documents with higher popularity are collected first.

A document having a large number of references (the number of other documents referencing the document) is considered to have high popularity. The idea that documents with high popularity can be preferentially collected by collecting documents in descending order of the number of documents referenced from documents in the collected document group.

[0011]

However, the above-described prior art framework alone is insufficient for use in collecting documents required for a portal site of a community such as a company. For example, the following points are required as requirements for a portal site in a company, that is, a corporate portal.・ Automatically collect a huge amount of documents generated in real time inside and outside the company. -Automatically perform semantic analysis and categorization. -Collect the documents and feed the classified results to the appropriate place on the screen (personally).

[0012] Of these, in document collection, it is necessary to select and collect a large number of documents from inside and outside of the company from the viewpoint of being related to the business, rather than collecting them unnecessarily. A business-related perspective is slightly different from having a specific semantic content or importance. For example, in the intranet community of a company of a certain size, the contents of documents are also semantically diverse. In addition, for documents outside the company (for example, the Internet), information on hobbies is also very popular, and such information is not always useful for a corporate portal.

[0013] However, the framework used in the conventional document collection, such as the priority acquisition of the latest information, the priority acquisition of the information in a specific field, and the priority acquisition of the information with a high degree of popularity alone, is not the information relating to such hobbies. like,
There is a problem in that documents that are generally important but not necessarily useful to the community are collected.

Further, for example, when documents are collected by the above-mentioned conventional technique of "collecting only documents having similar semantic contents", there are the following problems in each concept.・ The idea of simply limiting the number of layers is simple, but the process is simple, but do you prefer to collect documents that have really similar meanings, or do you miss important documents?
There is no guarantee. According to the method of comparing the contents of documents to determine whether the semantic contents are close or not, generally, using natural language processing, analyze the text described in the document to extract keywords, and resemble the extracted keywords. Analyze by degree. Therefore, processing takes time. At the earliest, it can process only about 100 documents per second. Therefore, it is difficult to process billions of documents one by one in a realistic time. Also, even if it takes such a long time,
Its accuracy is about 70 to 80%. Further, since this process greatly depends on the type of language, it is necessary to provide a determination tool for each language. -Even if it is determined whether or not to collect based on the reference expression, the character strings used in the reference expression include fixed words (such as "homepage", "return to top", and "click here"). Word), and does not always represent the semantic content of the reference destination.

[0015] In view of the above problems, it is an object of the present invention to make it possible to quickly and accurately collect documents suitable for use without depending on languages.

[0016]

The present invention is based on an apparatus or method for collecting documents from a network.
In the apparatus according to each aspect of the present invention, in the document collection apparatus that collects documents from the network, a next collection candidate that is a candidate for a document to be collected next is determined based on a reference relation of a group of collected documents. And a document collection unit that collects the next collection candidate from the network and adds it to the collected document group. The next candidate determination unit performs until the number of documents in the collected document group reaches a certain number or more. The determination of the next collection candidate and the collection of documents by the document collection means are repeated.

The above apparatus may be configured as a document collection apparatus for a community that collects documents having a high degree of usefulness for a community on a network. To this end, in the above configuration, after the document collection unit has evenly collected the documents from within the community on the network, the next candidate determination unit determines the next collection candidate from the documents inside and outside the community based on the reference relation of the collected document group. You may do it.

By collecting documents evenly from within the community before collecting documents from inside and outside the community, information on documents in various fields required in the community can be obtained. By collecting documents from inside and outside the community using the reference relation of the document group regarding the various fields obtained in this way, it becomes possible to accurately collect documents having a high degree of usefulness for the community. Further, since the content of the document body is not analyzed, it is possible to quickly collect documents having high utility for the community without depending on the language.

[0019] In the above configuration, the apparatus further comprises ranking means for calculating the importance based on the reference relation of the collected document group and the information indicating the location of the document on the network, for example, a URL. Alternatively, the next collection candidate may be determined based on the importance.

In the above-mentioned document collection apparatus for a community, the ranking means divides the document into and out of the community based on the degree of importance, and the next candidate determination means comprises:
Within and outside the community,
A document with a high ranking may be set as the next collection candidate. As a result, it is possible to prevent the next collection candidates from being concentrated in or outside the community and collecting documents only from inside or outside the community.

Further, the community document collection device may further include a presentation unit for presenting a search result of the collected document group inside and outside the community. This allows a client belonging to the community to obtain a document search result separately from inside and outside the community.

Further, the document collection device for the community further determines whether or not the document is a document in the community by indicating information indicating a place of the document on the network, for example, UR.
L may be provided. By determining based on information indicating the location of the document on the network, it is possible to quickly determine whether the document is a document in a community.

Further, the document collection device for collecting documents from the network may be configured as a document collection device for a specific field that collects documents related to a specific field. Therefore, according to still another aspect of the present invention, in a device for collecting documents from a network, prior to the collection of documents, a set of positive documents that are documents related to a specific field, and a document that is less relevant to the specific field. A document group relating to the field and a negative example document group are given as a collected document group, and the document collection unit adds the collected next collection candidate to the positive example document group. The determination of the next collection candidate by the next candidate determination unit and the collection by the document collection unit are repeated until the number of documents reaches a certain number or more. This makes it possible to quickly collect documents related to a specific field based on the reference relationship without analyzing the contents of the document body.

Further, in the above-mentioned document collecting apparatus for specific fields, further, a reference degree calculating means for calculating a reference degree which is a degree of reference from only the documents of the positive example document group based on the reference relation of the collected documents. And the next candidate determination means includes:
A document with a high degree of reference may be determined as a next collection candidate. Further, in the above-described document collecting apparatus for a specific field, further, based on the reference relationship of the collected documents, the collected documents which are referred to from the collected documents which refer to the documents of the positive example documents are collected. A co-reference degree calculating unit that calculates a co-reference degree indicating a degree of reference from the group may be provided, and the next candidate determination unit may determine a document having a high co-reference degree as a next collection candidate. By using the reference degree and the co-reference degree, it is possible to quickly collect documents related to the field to be collected without considering the contents of the document body.

Further, the above-mentioned document collecting apparatus for a specific field may be a document collecting apparatus which targets a plurality of fields and collects documents related to each field at the same time. Therefore, in the above-described document collecting apparatus for a specific field, when collecting collected documents given before collection as a union of documents related to a plurality of fields and collecting documents as a group of documents related to a certain field as a positive document group Then, the union of the documents in the remaining fields is set as the negative example document.

Further, each document collection device may further include means for collecting and collecting a group of collected documents based on a reference expression used in the collected documents. Among the reference expressions, there is a reference expression indicating that the contents of the reference destination document and the reference source document are the same, but are distributed and stored on the network. For example, “next”, “Next”, “previous”, “Prev”, etc. correspond to such reference expressions. The grouping unit groups together two or more documents having a reference relationship by such a reference expression.

Each of the document collection devices further includes a keyword assigning unit that assigns a keyword to the collected document based on a reference expression used in the collected document which is a document in the collected document group. It is good. As a result, it is possible to use the acronyms of various keywords as keywords without analyzing the semantic content of the document body.

The keyword assigning means may not use a keyword when the reference expression is a reference expression used irrespective of the reference destination document. Here, examples of the reference expression used regardless of the reference destination document include “return to top” and “return to home”.

The keyword assigning means may count the number of different documents referred to by the reference expression, and if the number of different documents is equal to or more than a certain number, the keyword may not be used as the reference expression. This is because such a reference expression is likely to be a reference expression used regardless of the reference destination document.

When the number of different documents referred to by the reference expression is less than a certain number, the keyword assigning means further counts the number of times of reference, which is the number of times each collected document is referred to by the reference expression. Alternatively, it may be determined whether or not the reference expression is a keyword based on the number of different documents and the number of references.

The keyword assigning means may combine a keyword based on the reference expression with a keyword extracted from the body of the collected document and a keyword extracted from information indicating a location on the network of the collected document. This makes it possible to combine keywords extracted by various methods.

The above-mentioned problem can also be solved by a method comprising the steps of the processing performed by each configuration of the present invention. In addition, a program that causes a computer to perform the same control as the function performed by each configuration of the present invention described above can also solve the above-described problem by being executed by the computer. In addition, a computer-readable recording medium that stores the above-described program can also solve the above-described problem by reading the program from the recording medium to a computer and executing the program.

[0033]

Embodiments of the present invention will be described below with reference to the drawings. The present invention relates to a document collection device that collects documents suitable for use from a network. In the following description, a case where a document is described in HTML will be described, but this is not intended to limit the present invention. Change the language to HTML (HyperText Markup Languag
The purpose is not limited to e). XML (eXtensibleMarkup L) is a markup language that describes the structure of a document.
anguage) and XSL (eXtensible Stylesheet Languag)
e) and other languages. As information indicating the location of the document on the network, a URL (Uniform Resource) is used.
Locators), but is not intended to limit the present invention. The information need not be a URL as long as the information indicates the location of the document on the network. The URL is UR
It is a part of the function of I (Uniform Resource Identifiers) and is currently widely used on networks.

FIG. 1 shows the principle of the present invention. As shown in FIG. 1, the document collection device 1 is connected to a network such as the Internet or an intranet. The document collection device 1 includes a document collection unit 2, a reference relationship extraction unit 3, a community determination unit 4, a next candidate determination unit 5, a ranking unit 6, a URL determination unit 7, a reference / co-reference calculation unit 8, and a grouping unit 9. , Keyword extracting means 10. In FIG. 1, the components indicated by dotted lines, that is,
The community discriminating means 4 and the reference / co-reference degree calculating means 8 may or may not be used depending on the embodiment. Similarly, the arrow indicated by the dotted line, that is, the ranking result of the document by the ranking unit 6 may or may not be used for the next collection candidate determination by the next candidate determination unit 15 depending on the embodiment.

A document collection device according to one embodiment of the present invention collects documents for a community on a network. For this purpose, the community document collection device according to one embodiment includes a document collection unit 2, a reference relation extraction unit 3, a community determination unit 4, a next candidate determination unit 5, a ranking unit 6, a grouping unit 9, and a keyword assignment unit 10. Prepare. In a document collection device for a community, first, documents are uniformly collected from within the community, and then documents having a high degree of usefulness to the community are collected from inside and outside the community.

The reference relation extracting means 3 collects the collected document group 2
The reference relation is extracted from 0, and the inter-document reference relation 22 is extracted. At the start of collection, an initial document group is given as the collected document group 20 in advance. The community discriminating means 4 discriminates whether or not an uncollected document which is a reference destination document of the collected document group 20 is a document in the community.

The next candidate judging means 5 includes the collected document group 20
And the uncollected document in the community is determined as the next collection candidate 21. The document collection unit 2 collects the documents determined as the next collection candidates 21 and adds a newly collected document group (a newly collected document group) to the collected document group 20 to make a new collected document group 20. Document collection means 2
Determines whether the number of documents in the collected document group 20 is equal to or greater than a prescribed value. If the number of documents in the collected document group 20 is smaller than the prescribed value, the process of collecting documents from within the community is repeated as described above. As described above, information on various fields to which the documents in the community belong can be acquired by uniformly collecting the documents in the community in a specified number or more. This information is useful for collecting documents that are highly useful for the community from inside and outside the community.

If the number of documents in the collected document group 20 is equal to or greater than a specified value, documents having the next highest usefulness for the community are collected from inside and outside the community. The reference relation extracting means 3 extracts the reference relation from the newly collected document group,
The community discriminating means 4 discriminates whether or not the reference document which has not been collected is a document in the community. The ranking unit 6 ranks uncollected documents, which are the reference destinations of the collected documents, in and out of the community based on the reference relationship and the information indicating the location of the documents on the network, for example, the URL. The ranking unit 6 includes a URL determination unit 7, and the URL determination unit 7 determines the similarity of the reference destination document and the reference source document on the URL character string. The ranking unit 6 ranks the documents based on the character string similarity of the URL determined by the URL determination unit 7.

The next candidate judging means 5 judges the uncollected documents ranked high in the community and outside as the next collection candidate 21 which is the document to be collected from the network next time. The document determined as 21 is collected. As described above, the community document collection device according to the embodiment of the present invention collects documents having high utility for the community in multiple stages. When documents exceeding a certain specified level are collected from inside and outside the community, the grouping means 9 groups together the collected documents 20 based on the reference expression. The keyword assignment unit 10 assigns a keyword to the collected document 20 based on the reference expression and the appearance frequency of the reference expression. The ranking means 6 ranks the collected documents 20 this time as described above. The collected and collected documents 20 that are finally compiled, assigned keywords, and ranked
Is stored as the collected document file 23. As described above, since the content of the document body is not analyzed in the document collection device for the community, it is possible to quickly collect documents suitable for the purpose without depending on the language.

A document collection device according to another embodiment of the present invention collects documents related to a specific field. For this purpose, the document collection device relating to the specific field includes a document collection unit 2, a reference relation extraction unit 3, a next candidate determination unit 5,
The system includes a ranking unit 6, a reference / co-reference calculation unit 8, a grouping unit 9, and a keyword assignment unit 10. In a document collection device for a specific field, there is no need to distinguish between documents inside and outside the community, so there is no processing related to community discrimination.

In a document collection device for a specific field,
Prior to collection, a document group relating to a specific field is given as a positive example document group, and a document group having little relation to the specific field is given as a negative example document group. The collected document group 20 is a union of the positive example document group and the negative example document group. The reference / co-reference degree calculation means 8 determines the degree of relevance of the document to a specific field based on the reference relation between a certain document and a group of positive examples, and the reference relation between the document and a group of negative examples. Calculate as degrees.
The next candidate determination unit 5 determines an uncollected document having a high reference degree or a high co-reference degree calculated by the reference degree / co-reference degree calculation unit 8 as a next collection candidate instead of the ranking by the ranking unit 6. Further, among the collected documents 20 included in the negative example document group, a document having a high reference degree or co-reference degree is excluded from the negative example document group and added to the positive example document group. Document collection means 2
The document determined as the next collection candidate 21 is collected and added to the positive document group. The determination of the next collection candidate and the collection of documents are repeated until the number of documents in the positive example document group becomes equal to or more than the specified number. Other operations are as described above.

The following is a description of a community document collection device for collecting documents having high utility for the community according to the first embodiment. As the community on the network described in the first embodiment of the present invention, for example, an in-house site, an industry site, and a user group on the network of a specific topic can be considered. Here, the in-house site is often represented by an intranet. The industry site is represented by an extranet consisting of systems from multiple companies. Note that a document collection device that collects documents required for an in-house site is applicable to an intranet portal in a company, also called a corporate portal (EIP: EnterpriseInformation Portal).

In the portal of the community, there is a requirement that documents having a high degree of usefulness to the community be automatically collected with priority. For example, in the case of a corporate portal, it is necessary to automatically collect documents related to business. According to the first embodiment of the present invention, such automatic document collection is realized. For this purpose, the following concept is adopted in the document collection device according to the first embodiment. -A document that is highly useful for a specific community is considered to be a document that is frequently referred to by many documents in the community or a document that is referred to by important documents in the community.

FIG. 2 shows the configuration of the document collection device according to the first embodiment. As shown in FIG. 1, the document collection device 10
Reference numeral 0 includes a document collection unit 101, a reference relationship extraction unit 102, a community determination unit 103, a next candidate determination unit 104, a ranking unit 105, a grouping unit 106, and a keyword assignment unit 107.

As described above, the document collection apparatus 100 first collects documents in a community a plurality of times, and then collects documents in and outside a community a plurality of times. One of the features of the document collection apparatus 100 is that document collection is performed a plurality of times in a multi-step manner.

Prior to the start of collection, first, an initial document group is given as a collected document group S. This initial document group is a starting point for collection. As the initial document group, for example, a top page of the site, a reference collection (link collection) of the top page, and the like can be considered. Specifically, the collected document group S or the initial document group is stored as a URL table 120 in the document collection device 1.
00 is provided.

Subsequently, the reference relationship extracting unit 102 extracts a reference relationship from the collected document group S, and URLs of documents to be referred to by the collected document group S (hereinafter referred to as reference destination documents).
Is stored in the URL table 120, and the extracted reference relation is stored in the reference relation table 121. The community determining unit 103 determines whether the reference destination document of the collected document group S extracted by the reference relationship extracting unit 102 is a document in the community or a document outside the community based on the URL. The result is stored in the reference relation table 121.

The document collection apparatus 100 first collects documents in a community at least once. At this time, collection is performed evenly. The next candidate determination unit 104 is a candidate of the next document to be collected next among the documents in the community that have not been collected among the reference documents of the collected document group S extracted by the reference relationship extraction unit 102 (hereinafter, the next candidate). (Referred to as a collection candidate N). The document collection unit 101 collects the document group determined as the next collection candidate N, adds the collected document to the collected document group, and sets a new collected document group S. The collection of documents in the community is performed until a specified number of documents are collected. It is not necessary to collect all documents in the community, and it may be about １ ／ to の of all documents in the community. Obtain information about the areas of documentation that are useful within the community by uniformly collecting documents within the community.

After the document collection unit 101 collects a specified number of documents in the community, the document collection device 100
Then, the document inside and outside the community is collected at least once. In this case, as described above, the document collection unit 101 collects documents, and the reference relationship extraction unit 102 and the community determination unit 103 store information in the URL table 120 and the reference relationship table 121, and then further
The ranking unit 105 assigns importance to the referenced document based on the reference relationship and the URL of the document, and ranks the referenced document based on the importance.

The candidate determining unit 104 includes a ranking unit 105
Based on the result of the determination, the reference documents that have not been collected yet, and the top n among the documents in the community
The document group in the first place and the document group in the top n2 place among the documents outside the community are determined as the documents to be the next collection candidates N. By determining the next collection candidate N separately inside and outside the community, it is possible to prevent the documents from being unevenly collected either inside the community or outside the community.

Subsequently, similarly to the collection of documents in the community, the document collection unit 101 collects next collection candidates N from inside and outside the community, and adds the collected documents to a collected document group to newly collect collected documents. A document group S is assumed. The document collection device 100 repeats document collection from inside and outside the community until the specified number of documents is collected.

After the document collection unit 101 collects a specified number of documents from inside and outside the community, the collected documents are sorted out. The selection of documents is performed by the grouping unit 106, the keyword assigning unit 107, and the ranking unit 105.
First, the grouping unit 106 groups documents that have the same content but are divided into a plurality of documents among the collected documents based on a character string (also referred to as a reference expression) used when referring to another document in the document. .

The keyword assigning unit 107 determines a keyword based on the reference expression in the document, and assigns the keyword to the document. More specifically, the keyword assigning unit 10
Reference numeral 7 excludes reference expressions, such as “return to top” and “to home”, which are often used regardless of the contents of the referenced document. Subsequently, the keyword assigning unit 107 counts the number of different documents referenced by each reference expression, and stores the counted number in the reference expression table 122 (not shown in FIG. 2). Also, the frequency of referring to each collected document by a certain reference expression is counted and stored in the reference count table 123 (not shown in FIG. 2). Keyword assignment unit 107
Calculates the weight of the reference expression for each collected document based on these counting results, and assigns a certain number of reference expressions in descending order of the weight to each collected document as a keyword.

The ranking unit 105 assigns importance to each document based on the reference relation and the URL of the document, and ranks the documents based on the importance. As described above, the community document collection device 100 according to the present embodiment is described.
Collects documents based on reference relationships and URLs, summarizes, assigns keywords, and ranks documents without analyzing the contents of the document body.

As described above, the document collection device 100
Provides a group of documents that have been put together, given a keyword, and ranked as superior content 130. The excellent content 130 is provided as an index 141 via a search engine 140,
0 to the server 160, or a directory edited by the classification engine 150 and provided to the server 160. The client of the server 160 can browse the excellent content 130 provided to the server 160 via the browser 170.

The data structure of each table will be described below with reference to FIGS. FIG. 3 shows URL table 1
20 shows an example of the data structure of No. 20. As shown in FIG.
The RL table contains a document I that identifies the document for each document.
D (Identification information), the URL of the document, a collected flag indicating whether the document has been collected, a community flag indicating whether the document is in the community, and the importance of the document are stored. The document ID and the URL are stored when the reference relation extracting unit 102 extracts the reference destination document of the collected document. The collected flag is set in the document collection unit 10
1 is turned on (1) when the document is collected.
The community flag is set to “ON (1)” when the community determination unit 103 determines that the document is a document in the community. The importance is determined by the ranking section 10
5 is calculated and stored based on the document reference relationship and the URL character string features.

FIG. 4 shows an example of the data structure of the reference relation table 121. As shown in FIG. 4, the reference relation table 121 stores information on the reference relation of the document. More specifically, the reference relation table 121 includes a reference source document ID which is a document ID of a reference source document, a reference destination document ID ₁ which is a document ID of a document in a community referred to by the reference source document, and Reference destination document ID ₂ which is the document ID of a document outside the community referenced by the original document
Is stored. These pieces of information are stored by the reference relation extracting unit 102.

FIG. 5 shows an example of the data structure of the reference expression table 122. As shown in FIG. 5, the reference expression table 122 stores information on the frequency at which each reference expression is used in a collected document. More specifically, the reference expression table 122 includes, for each reference expression, an expression ID for identifying the reference expression, a reference expression (character string), and a document frequency DF (w) indicating the number of different documents referred to by the reference expression. ,as well as,
A necessity flag indicating whether the keyword should be used is stored. All of this information is stored in the
Stored by

FIG. 6 shows an example of the data structure of the reference count table 123. As shown in FIG. 6, the reference count table 123 stores a reference expression frequency TF (d, w) which is the number of times each collected document is referred to by each reference expression.
All of this information is stored by the keyword assignment unit 107. For example, when a reference destination document doc2 is obtained by referring to a link embedded in a certain reference expression rw1 in a certain document, the TF of the reference destination document doc2 is obtained.
(Doc2, rw1) is incremented by one. In FIG. 6, the document whose document ID is doci is represented by the expression ID RW
This indicates that the reference expression j is referred to TF (doci, rwj) times. For example, in FIG. 6, it can be seen that the document whose document ID is doc1 has been referenced 19 times by the reference expression whose expression ID is rw1.

Hereinafter, a description will be given of a method of collecting documents having a high degree of usefulness for a specific community realized by the document collection apparatus according to the first embodiment. The following notation is used in the description. LT (S) indicates a document group as a reference destination of the document group S. XY indicates a difference set between the set X and the set Y.

First, a general flow of a process of collecting documents for a specific community will be described with reference to FIG. First, at the start of collection, a document in the community is given as an initial document group (a document group serving as a collection start point) of the collected document group S.

Based on the result of the reference relationship extraction by the reference relationship extraction unit 102 and the result of the community determination unit 103 determining whether or not the referenced document is a document in the community, the candidate determination unit 104 N is extracted (step S1). The process of extracting the next collection candidate N will be described later in detail.

Subsequently, the document collection unit 101 collects the next collection candidate N based on the URL stored in the URL table 120 (step S2), and turns on the collection completed flag for the collected document. Accordingly, the document collection unit 101 adds the newly collected next collection candidate N to the collected document group S. That is, the group of documents represented by the expression S∪N is set as a newly collected document group S.

The document collection unit 101 determines whether or not the number of documents included in the collected document group S is equal to or greater than the specified number of documents (step S3). This determination is made by counting the number of documents for which the collected flag stored in the URL table 120 is “ON (1)”. If the number of documents included in the collected document group S is not equal to or greater than the specified number of documents (step S3: No), the next candidate determination unit 104
Determines the next collection candidate again (step S4), and returns to step S2. In the determination of the next collection candidate after the second time, a reference relation extraction result by the reference relation extraction unit 102 for a document newly collected by this collection (hereinafter, referred to as a newly collected document) and a new determination by the community determination unit 103 Based on the result of determining whether or not the reference document of the collected document is a document in the community, the candidate determination unit 104
Extracts the document in the community from the uncollected reference destination documents as the next collection candidate N. The processing in step S4 is
Since this is the same as step S1, step S1 will be described together when it is described later.

When the number of documents included in the collected document group S is equal to or more than the specified number of documents (step S3: Ye)
s) This time, the candidate determination unit 104 determines a next collection candidate from documents inside and outside the community. First of all,
The reference relationship extraction unit 102 extracts the reference relationship of the newly collected document, and the community determination unit 103 determines whether the reference destination document of the newly collected document is a document in the community. After that, the ranking unit 105 assigns importance to the collected document and its reference destination document, that is, S∪LT (S), and based on the importance, the uncollected reference destination document, that is, LT (S). -S ranking (step S
5). Details of the processing in step S5 will be described later.

Subsequently, the next candidate determination unit 104
Among (S) -S, a document group included in the top n1 documents in the ranking of documents in the community and a document group included in the top n2 documents in the ranking of documents outside the community are set as the next collection candidate N. (Step S6). By extracting the next collection candidate N while distinguishing the inside and outside of the community as described above, it is possible to prevent the collected documents from being biased inside or outside the community.

The document collection unit 101 includes the URL table 12
The next collection candidate N is collected based on the URL stored in 0 (step S7), and the collected flag of the collected document is set to “ON (1)”. The document collection unit 101 has a URL
By counting the number of documents for which the collected flag stored in the table 120 is “ON (1)”,
It is determined whether or not the number of documents included in the collected document group S is equal to or greater than the specified number of documents (step S8).

If the number of documents included in the collected document group S is not equal to or greater than the specified number of documents (step S8: No),
It returns to step S5. When the number of documents included in the collected document group S is equal to or more than the specified number of documents (step S8:
Yes), the documents of the collected document group S are selected by the ranking unit 105, the grouping unit 106, and the keyword unit 107 (step S9). Details of the processing in step S9 will be described later.

The process of determining the next collection candidate when collecting documents in a community will be described below in detail. This processing is performed in steps S1 and S4 in FIG.
Is equivalent to

First, the reference relation extracting unit 102 extracts a reference destination document referred to from a newly collected document (step S11). The reference relation extracting unit 102 determines, for each extracted reference destination document, the same URL as the reference destination document.
If it is not stored in the table 120, the URL of the referenced document is stored in the URL table 120 (step S
12). This is because it is not necessary to store the same URL redundantly. When storing the information, the reference relationship extracting unit 102
The collected flag is set to “off (0)”.

Subsequently, the community determining unit 103
Based on the character string of the URL of the reference document stored in the RL table 120, it is determined whether or not the extracted reference document is a document in the community. The determination unit 103
The community flag of the URL table 120 is set to “ON (1)”. In other cases, the community determination unit 103 sets the community flag to “off (0)” (step S13). Further, the reference relation extracting unit 102
Is based on the determination result of the community determination unit 103,
The reference relation is stored in each column of the reference relation table 121.

Here, according to the present embodiment, the community is given as a set of documents on the network, that is, a document group. Therefore, whether or not the document belongs to the same community can be determined based on the URL indicating the document group. More specifically, the determination as to whether or not the document is in the community is performed as follows based on the characteristics of the URL in the character string. When the community is an in-house site, a document whose domain name is the same as the domain name of the in-house site (fujitsu.co.jp or the like) is usually determined to be a document in the community. If the community is an industry site, a document having the same domain name as any of the domain names of the sites of a plurality of companies belonging to the industry site is determined to be a document in the community. If the community is a user group, the URL (for example, ht) of each user's site (also referred to as a home document)
tp: //www.fujitsu.co.jp/foo/) is determined as a document in the community if the URL includes the same character string as any of the characters.

The next candidate judging section 104 is a document LT which is a reference destination document of the collected document and which is an uncollected document.
Among (S) -S, the document in the community is determined as the next collection candidate N. Specifically, the next candidate determination unit 104
Refers to the URL table 120, and determines, as the next collection candidate N, a document whose collection flag is “off (0)” and whose community flag is “on (1)” (step S14). Such a next collection candidate N can be represented by the following equation (1).

N = {d | d} LT (S) −S, d is in the community} (1) In this way, the next collection candidate N is determined, and the documents in the community are collected evenly. Accordingly, information on semantically diverse documents required in the community can be obtained without bias.

Next, a process for ranking collected documents and their reference destination documents will be described with reference to FIG.
This processing corresponds to step S5 in FIG. The reference relationship extraction unit 102 and the community determination unit 103 extract the reference relationship of the newly collected document, and store the reference relationship together with the community determination result in the URL table 120 and the reference relationship table 121 (steps S21 to S2).
3). The processing of steps S21 to S23 is the same as the processing of steps S11 to S13 described in FIG.
Detailed description is omitted.

Subsequently, the ranking unit 105 determines the reference relation stored in the reference relation table 121 and the URL stored in the URL table 120 for the collected document and its reference destination document, that is, S∪LT (S). Is calculated based on the characteristics of the character string of
It is stored in the L table 120 (step S24). The ranking unit 105 ranks uncollected reference documents, that is, LT (S) -S, inside and outside the community, based on the community flag and importance stored in the URL table 120 (step S25).

Hereinafter, the process of calculating the importance in step S24 will be described in detail. As described above, the ranking unit 105 uses the reference relation of the document and the URL,
The importance of a document is calculated without analyzing the semantic content of the collected document. Hereinafter, the importance given to the document based on the reference relation is referred to as link importance. The basic concept when assigning link importance is as follows. Documents that are frequently referenced by URLs with low similarity are important.

For example, generally, a plurality of documents provided in the same site are referred to by other documents in the site, but the URLs of the documents are similar to each other. Therefore, it can be estimated that the importance of a document referenced from a URL having a high similarity is low. A document that is referred to by many documents is a more important document, and a document that is referred to by an important document and has a low URL similarity is important.

For example, famous directory services and government offices are referred to by many documents, and documents referred to by such important documents are considered to have high importance. In addition, many documents and documents provided in a service (site) having a mirror site are often referred to in the site, but since the URLs of documents in the same site are usually similar, " It is possible to avoid that many documents at the same site are searched for many times, by introducing the idea that "documents with low URL similarity are important." The similarity of the URL is defined based on the face information of the URL such that a server address, a path, and a file name that are all different are the smallest, and a mirror site or a document in the same server has a high similarity.

By introducing the above three concepts, all reference relations are not treated equally, and a weight corresponding to the link importance is given to the reference relation. More specifically, the weight is given as the reciprocal of the similarity between the URLs of the reference source document and the reference destination document. Hereinafter, the calculation of the link importance will be described in more detail.

DOC = {p ₁ , p ₂ ,... P _N }, the link importance of the document p is W _p , the document set of the reference destination of the document p is Ref (p), Referred (p) is the set of documents referencing document p, and UR of documents p and q
L similarity is sim (p, q), difference is diff (p, q) = 1 / sim (p, q)
Then, when it is assumed that a reference is made from the document p to q, the weight lw (p, q) of the reference is defined by the following equation (2).

[0082]

(Equation 1)

As can be seen from equation (2), lw (p, q)
Becomes larger as the similarity sim (p, q) of the URLs of p and q is lower and the number of references from p is smaller. The link importance of each document is obtained by defining Cq as a constant (a lower limit of the importance, which may be different for each document) for each p∈DOC.

[0084]

(Equation 2)

Is defined as a solution of the simultaneous linear equation. The ranking unit 105 assigns link importance to each document by solving this simultaneous linear equation. In addition,
Description of the solution of the simultaneous linear equations is omitted because there are many existing algorithms. From the expressions (2) and (3), it can be seen that the above concept is realized.

Next, the URL similarity sim (p, q) between the documents p and q in the equations (2) and (3) will be described. URL
The similarity is calculated by a URL determination unit (not shown) of the ranking unit 105. Generally, the URL of a document is composed of three types of information: a server address, a path, and a file name. For example, the URL of a WWW document, http: //www.fla
b.fujitsu.co.jp/hypertext/news/1999/product1.html
Is the server address (www.flab.fujitsu.co.jp), path (hypertext / news / 1999), file name (product1.htm
l) consists of three types of information.

In the present embodiment, given two documents p
And q are defined by the above three combinations. As similarity sim (p, q), for example, domain similarity sim_domain (p, q) and fusion similarity sim described below
_Merge (p, q) is considered.

The domain similarity sim_domain (p, q) is calculated based on the domain similarity. The domain is the latter half of the server address and represents a company or organization. For servers in the United States ending with .com, .edu, .org, etc., from the end of the server address to the second, and for servers in other countries ending with .jp, .fr, etc., from the end of the server address Up to the third corresponds to a domain.

The domain similarity between the document p and the document q is defined by the following equation. sim_domain (p, q) = 1 / α (when p and q are in the same domain) = 1 (when p and q are in different domains) Here, α is a constant and a real value larger than 0 and smaller than 1 Take it.

As sim (p, q), a similarity sim_merge (p, q) obtained by fusing the above three types of information is defined as follows. sim_merge (p, q) = (similarity of server address) + (similarity of path) + (similarity of file name) Hereinafter, a method of calculating each term on the right side will be described.

As for the similarity of the server address, the hierarchy of the address is viewed from the rear, and when the levels match up to n levels, the similarity is set to 1 + n. For example, www.fujitsu.co.jp and ww
w.flab.fujitsu.co.jp is 4 because it matches up to 3 levels. www.fujitsu.co.jp and www.fujitsu.com are 1
Since the levels do not match (match 0 level), the similarity is 1.

The path similarity is compared for each element separated by "/" of the path from the head, and the similarity is determined up to the matching level. For example, /doc/patent/index.html and / doc / paten
Since t / 1999/2 / file.html matches up to two levels, the similarity is 2.

The file name similarity is set to 1 when the file names match. This sim_merge (p, q) can also prevent many documents with similar URLs from being searched.

In this way, ranking section 105
Documents are assigned importance, and documents with high importance are ranked higher. As described above, according to the present embodiment, the ranking unit 105 does not analyze the semantic content of the document body based on the acquired reference relationship of the document and the characteristics of the character string of the URL. With high accuracy,
Documents can be assigned importance, and the documents can be ranked based on the importance.

Hereinafter, the process of selecting collected documents will be described in detail with reference to FIG. This processing corresponds to step S9 in FIG. First, the grouping unit 106
The collected documents S are put together based on the reference expression used in the collected documents S (step S3).
1). The reference expression is, for example, HTML (HyperT
In ext Mark-up Language), the part surrounded by the anchor tag corresponds to it.

More specifically, reference expressions (character strings used at the time of reference) such as "next" and "previous" are stored in advance in a collective reference expression table (not shown). Documents using these reference expressions such as “next” and “previous” have the same content in the reference source document and the reference destination document, but have the UR.
It is estimated that L is a distributed document. Consolidation part 1
In step 06, the reference expressions stored in the grouping reference expression table are extracted from the document, and the documents are grouped as follows. -"Next", "Continue", "N" from document doc1
When the document doc2 is referred to by an expression such as “ext”, the grouping unit 106 outputs the document doc2
Is reduced to the document doc1. Repeat this operation as much as possible. -From the document doc1, "Previous", "Return to previous", "P
When the document doc2 is referred to by an expression such as “rev”, the grouping unit 106 converts the document doc1 to d.
Degenerate to oc2. Repeat this operation as much as possible.

Subsequently, the keyword assigning unit 107 assigns a keyword to the collected document S based on the reference expression (step S32). The keyword assignment processing will be described later in detail. Finally, the ranking unit 105 determines whether the
In the same manner as in step S24, importance is assigned to the collected documents, and the importance is stored in the URL table 120.
The ranking unit 105 ranks the collected documents based on the importance (Step S33).

Next, the keyword assignment processing in step S32 will be described in detail with reference to FIG. First, of the reference expressions used in the collected documents in advance,
A frequently used reference expression such as "go home" or "return to top" is stored in an unnecessary word dictionary (not shown) regardless of the reference document (not shown). The keyword assignment unit 107
A reference expression is extracted from the collected document group S, and each reference expression w is extracted.
, The number DF (w) of different documents referenced using the reference expression w is aggregated, and the expression I that identifies the reference expression w
D, together with the reference expression (character string), the total result of DF (w) is stored in the reference expression table 122 (step S)
41). At this stage, the necessity flag is set to “OFF (0)”.

The keyword assigning unit 107 excludes the reference expressions w whose DF (w) is equal to or more than a predetermined number from the keyword candidates (step S42). In other words, assuming that the total number of documents including the reference destination document is N, the reference expression w corresponding to the following expression is omitted.

DF (w)> αN Here, α is a constant, and may be, for example, 0.1. The keyword assigning unit 107 excludes the specific reference expressions stored in the unnecessary word dictionary from the keyword candidates among the reference expressions w (step S43). This is because these reference expressions are used irrespective of the reference destination document and are not suitable for use as keywords.

The keyword assigning unit 107 extracts the document d from the collected documents S, and sets the difference set between the collected documents S and d, that is, Sd, as a new collected documents S (step S44).

The keyword assigning unit 107 counts the number of times TF (d, w) referred to by each reference expression w in the document d, and calculates the number of times TF (d, w) in the document d using the following equation (4). , The weight W (d, w) of each reference expression w is calculated (step S45).

W (d, w) = TF (d, w) log (N / DF (w)) (4) The keyword assigning unit 107 accesses the reference expression table 122 and weights the reference expression. The necessity flags of at most n reference expressions are set to “on (1)” in the descending order of W. That is, at most n reference expressions are set as keywords of the document d in descending order of the weight W.

One of the features of the keyword based on the reference expression thus obtained is to acquire various aliases as keywords, unlike the keyword based on the word included in the text of the document d. For example, various names (official names, abbreviations, common names, English names, etc.) of the company can be obtained from the reference expression to the home page of the company. Further, for example, regarding the term “Linux”, various aliases such as “Linux” and “Linux” can be acquired as keywords. On the other hand, in general, only one of such aliases is used in the text of one document in a unified manner, so that when a keyword is obtained from the text, the alias cannot be obtained as a keyword.

The keywords obtained from the reference expression include keywords that are frequently used among words appearing in the body of the document d and keywords obtained from the URL indicating the document d, for example, http://www.fujitsu. If it is com /, fujitsu may be added as a keyword.
This makes it possible to assign various keywords to the document d.

FIG. 12 shows an example of a screen for providing a user with documents collected using the document collection device according to the first embodiment. In FIG. 12, the collected excellent contents 1
30 is divided into directories using the classification engine 150 and provided to the client of the server 160 as an example. By inputting a keyword or selecting a category on the screen 180, the client can display a link or a collection of links to a document to be viewed on the screen.

When the client inputs a keyword, a link to a document searched based on the keyword is displayed together with the importance as shown in a screen 181. According to the present embodiment, it is possible to perform a search with an alias of the input keyword. When a category is selected, a link collection to documents related to the selected category is displayed as shown in a screen 182.

Here, as shown in screens 181 and 182, when presenting the retrieved document, based on the community flag stored in the URL table 120,
The document may be presented separately inside and outside the community.

Hereinafter, a document collection device according to the second embodiment will be described. The document collection device according to the second embodiment collects documents related to a specific field. The following concept is adopted in the document collection device according to the present embodiment.・
In a network, documents in a parent / child / brother relationship of reference tend to be similar in content. Documents that are often described as having parent-child / sibling relationships with a certain set of documents,
It is highly likely that the content is in the same field as the original document group.
The operation of collecting documents having a high reference degree (parent-child relation) or a high degree of co-reference (sibling relations) among documents in parent-child / sibling relations from the original document group, and nesting the original document group in multiple steps. With this, documents related to the relevant field can be collected.

FIG. 13 shows the configuration of a document collection device according to the second embodiment. As shown in FIG. 13, the document collection device 200 according to the second embodiment includes a document collection unit 101, a reference relation extraction unit 102, a candidate determination unit 104, a reference / co-reference degree calculation unit 201, a ranking unit 105, and a grouping unit. 10
6 and a keyword assigning unit 107. The reference degree / co-reference degree calculation unit 201 calculates the degree to which a certain document is related to a specific field based on the reference relation of the documents. The functions of the other units are as described in the first embodiment.

In the document collection apparatus according to the second embodiment, prior to the start of collection, first, representative documents in a certain field are collected using an existing search engine or link collection, and given as a positive example document group PS. Similarly, documents in any field that does not overlap with the field are also collected and given as a negative example document group NS, and PS @ NS is set as a collected document group S. This collected document group S is the starting point of collection.

The reference relation extracting unit 102 extracts a reference relation from the collected document group S, stores the URL of the reference destination document of the collected document group S in the URL table 120, and stores the extracted reference relation in the reference relation table. 121. Here, in the document collection device according to the second embodiment, the UR
The L table 120 includes a column of a positive example flag indicating whether or not the document is included in the positive example document group PS instead of the community flag. The positive example flag is turned on (1) when the document is included in the positive example document group PS. When storing the reference relation in the reference relation table 121, it is not necessary to divide the reference relation inside and outside the community.

The reference degree / co-reference degree calculation section 201 compares the positive example document group PS and the negative example document group NS with the reference document of the collected document S based on the reference relation extracted by the reference relation extraction section 102. The reference degree and the co-reference degree indicating the relationship are calculated. The next candidate determination unit 104 is a reference destination document of the collected document group S and is included in the positive example document group PS based on the reference degree and the co-reference degree calculated by the reference degree / co-reference degree calculation unit 201. A document that satisfies a predetermined condition is determined as a next collection candidate N from the documents that do not exist. The next candidate determination unit 104 excludes the documents included in the negative example document group NS among the next collection candidates N from the negative example document group NS and adds the documents to the positive example document group PS.

The document collection unit 101 sets the URL table 12
0, and collects uncollected documents among the next collection candidates N,
The collected documents are added to the positive document group PS. The document collection device 200 according to the second embodiment extracts the reference relation of the collected documents S as described above until the number of documents in the positive example document group PS becomes equal to or more than the specified number, and based on the reference relation. Then, the next collection candidate N is determined, and the process of collecting the next collection candidate N is repeated.

When the number of collected documents S exceeds the specified number, the grouping unit 106 compiles the collected document group S based on the reference expression, and the keyword assigning unit 107 collects the collected documents S based on the frequency of use of the reference expression. A keyword is assigned to the document group S. The ranking unit 105 determines each of the collected documents S based on the reference relationship and the characteristics of the URL character string.
Is calculated, and the collected documents S are ranked based on the importance. As a result, excellent content 2 by field
Create 10. As described above, according to the document collection device according to the second embodiment, it is possible to collect, collect, and assign keywords to a specific field without analyzing the contents of the document body.

[0116] The excellent content 210 for each field is provided to the server 160 via the search engine 140. The client of the server can receive the search service using the browser 160.

Hereinafter, a document collection method for a specific field realized by the document collection device according to the second embodiment will be described. First, the notation used will be described. LT (B) indicates a reference destination document set of the document group B. LT (p) indicates a document set referred to by the document p. LS (d, X) = {c {X | c refers d}} indicates a set of documents that refer to document d in document set X. LS (A, X) = {c} X | {d {A, crefers d}
Indicates a set of documents that refer to at least one document in the set A of the document set X. CC (d, A, X) = LS (d, X) ∩LS (A,
X) indicates a set of documents that refer to both the document d and the documents of the set A (at least one document) in the document set X.

In FIG. 14, LT (S), LT (p), LS
For (d, X) and LS (A, X), the reference relation of the document which each set means is shown. In FIG. 14, a black circle indicates a document, an arrow indicates a reference relationship, a source of the arrow indicates a reference source, and a tip of the arrow indicates a reference destination. As shown in FIG.
(B) and LS (A, X) and LT (p) and LS (d,
In X), it can be seen that the arrows are reversed, that is, the reference destination document and the reference source document are interchanged. FIG. 15 shows a reference relationship of a document represented by CC (d, A, X).

The processing for collecting documents relating to a specific field will be described below with reference to FIG. According to the document collection device according to the second embodiment, “XML” or “Linux”
In the case of preferentially collecting semantically similar documents related to a specific field (genre), such as "x", it is possible to collect documents based on a reference relationship without performing a process of analyzing the contents of the document body. .

First, a typical document belonging to this field is
Search and collect from existing search engines and link collections,
This is a positive example document group PS. Similarly, documents belonging to a field that does not overlap with the relevant field are searched for and collected, and set as a negative example document group NS. The positive example document group PS and the negative example document group NS become the initial document group. And the UR of the PS and NS documents
L, a collected flag (all “ON (1)”), and a positive example flag (“ON (1)” in the case of a positive example document) are stored in the URL table 120. Positive document group PS and negative document group N
The union set PS∪NS of S is set as a collected document group S (step S51). Here, for example, if the field is “computer”, examples of fields that do not overlap with the field include “handicraft”, “cooking”, and “beauty”.

The reference relation extracting unit 102 extracts a reference relation from the initially collected document group S (initial document group) at the start of collection and from a newly collected document thereafter (step S5).
2) The URL of the reference destination document is stored in the URL table 120, and the reference relation is stored in the reference relation table 121.
This processing is the same as in the first embodiment.

The reference degree / co-reference degree calculation unit 201 generates a document set T (S) by removing the documents included in the positive example document group PS from the reference destination documents of the collected document group S based on the extracted reference relation. ) = LT (S) -document d∈T included in PS
Regarding (S), the reference degree R is calculated using the following equation (5).
Calculate _score (d, PS, S). Next candidate determination unit 10
Reference _numeral 5 denotes a document group in which the reference _score R _score (d, PS, S) is in the top n1 documents, as N1. (Step S53).
Whether or not the collected document is included in the positive example document group PS can be determined by referring to the positive example flag of the URL table 120.

[0123]

(Equation 3)

The first term in equation (5) refers to document d.
2 shows the logarithm of the number of documents in the positive example document group PS. Also, (5)
The second term in the formula is the number of collected documents that refer to document d.
On the other hand, the number of documents in the positive document group PS referencing the document d
Indicates the ratio of Therefore, the positive example sentence in the collected document group S
The more the document d is referred to only from the book group PS, the more R
_scoreYou can see that (d, PS, S) takes a large value
You.

That is, the next candidate determination unit 105 determines that the reference degree R
_{Based on the score} (d, PS, S), among the reference documents of the newly collected documents, a large number of positive example documents PS related to a specific field refer to a large number of negative example documents NS not related to a specific field. A document that has not been subjected to the determination is determined as N1.
In FIG. 17, when calculating the reference degree for the document d,
(5) The reference relation that each set included in the expression means is shown.

Subsequently, the reference degree / co-reference degree calculation section 201
Calculates the co-reference degree C _score (d, PS, S) for the document d @ T (S) -N1 using the following equation (6). The next candidate determination unit 105 sets a document group in d の う ち T (S) -N1 in which the co-reference degree C _score (d, PS, S) is in the top n2 items, as N2 (step S54).

[0127]

(Equation 4)

The content of the logarithm of the first term of the equation (6) is the document d
And the sum of the number of documents included in the positive example document group PS, which is the reference destination document of the document p, for all the collected documents p referencing both the documents of the positive example document group PS. Accordingly, the co-reference degree C _score (d, PS, S) is such that the greater the number of collected documents p that refer to both the document d and the documents of the positive example document group PS, and such a document d It can be seen that the value of the document d, which is the reference document of the document p and includes the number of documents included in the positive example document group PS, is larger. In other words, with respect to the document d referenced from the collected documents referencing the documents of the positive example document group PS, the document d in which the number of collected documents referencing the document d is large.
The larger the co-reference degree C _score (d, PS, S), the larger the value.

The second term of the expression (6) indicates the ratio of the number of documents p referenced together with the document d to the number of collected documents that are the reference sources of the document d. Co-reference score C _score
(D, PS, S) takes a larger value as this ratio increases. FIG. 18 shows a reference relationship that each set included in Expression (6) implies when calculating the co-reference degree for the document d.

The next candidate determination unit 105 determines that the next collection candidate N = N1
∪N2 is set (step S55). Next candidate determination unit 105
Searches the URL table 120 using the URL of the next collection candidate N as a key, and turns on (1) the positive example flag of the next collection candidate N. As a result of this processing, the documents included in the negative example document group NS but determined as the next collection candidates are removed from the negative example document group NS and added to the positive example document group PS (step S56). .

The document collection unit 101 stores the URL table 12
Based on the URL stored in 0, uncollected documents among the next collection candidates N are collected from the network, and the collected flag corresponding to the collected documents is set to “ON (1)” (step S57). By this process, a newly collected document is added to the positive document group PS. The document collection unit 101 refers to the URL table 120 and determines whether or not the number of documents in the positive example document group PS is equal to or greater than a specified number (step S58). If the number of documents in the positive example document group PS is not equal to or greater than the specified number (step S58: No), the process returns to step S52 and repeats the processing.

If the number of documents in the positive example document group PS is equal to or greater than the specified number (step S58: Yes), the documents in the positive example document group PS are selected (step S59), and the process is terminated. The document selection process is the same as in the first embodiment, and a description thereof will not be repeated.

As described above, according to the present embodiment, it is possible to accurately and quickly collect documents related to a specific field without analyzing the contents of the document body.
Hereinafter, a modified example of the second embodiment will be described. Since it is difficult to collect the negative example document group NS, it is desirable to effectively use the document group NS so as not to discard it after the collection process.
Therefore, according to the document collection device according to the modification of the second embodiment, the negative example document group NS collected in the above processing is effectively used. This makes it possible to collect documents in a plurality of fields as independent as possible, for example, “Java (registered trademark) language”, “knitted”, and “French cuisine”. Therefore, when documents in a certain field are collected, a document group in the field is treated as a positive example document group PS, and a document group in a field other than the field is treated as a negative example document group NS.

The configuration of the document collection device is as described with reference to FIG. 13, and a description thereof will not be repeated. Hereinafter, FIG.
A process performed by the document collection device according to the modification of the second embodiment will be described with reference to FIG.

First, n independent document groups D _i (i
= 1, the · · · n), collected searched from a search engine or links like, URL of the document of the document set D _i, collecting flag, and the field identification information is information for identifying the areas URL It is stored in the table 120. In the document collection device according to the modification of the second embodiment, the positive example flag is unnecessary. The document group _Di is an initial document group of the field i. The collected document group _{D = (D 1, D 2} , ···, D n) to (step S61).

First, the reference relation extracting unit 102 gives i (step S62). At the start of collection, the reference relationship extraction unit 102 sets i to 1. Subsequently, the reference relationship extraction unit 102 determines whether i exceeds n (step S63). If i exceeds n (step S6)
3: Yes), proceed to step S71. Otherwise (step S63: No), the reference relation extracting unit 102
From the new collection documents of the document group D _i corresponding to the field i (collection start time from the initial document group), to extract the reference relationship, the URL of the referenced document in the URL table 120, the reference relationship table 121 reference relationships Each is stored (Step S
64). This processing is the same as in the first embodiment.

[0137] Referring degree / co-reference calculation unit 201, a referenced document of the document set D _i, is not included in the collected document group D document group _{T (D i) = LT (} D i) -D Is the next collection range, and the document d∈T included in the next collection range T (D _i )
For (D _i ), the reference degree R is calculated by using the above equation (5).
Calculate _score (d, D _i , D). Next candidate determination unit 105
Let N1 _i be a document group in which the reference _score R _score (d, D _i , D) is in the top n1 cases. (Step S65). The field containing the collected documents is the URL table 1
The determination can be made by referring to the twenty field identification information.

The reference degree / co-reference degree calculation unit 201 performs the next collection
Range T (D_i) To N1_iDocument d included in the set excluding
∈T (D_i) -N1_iThen, using the above equation (6),
Reference C_score(D, D_i, D) are calculated. Next candidate judgment
The unit 105 calculates the co-reference degree C_{scor e}(D, D_i, D) is the top n
N2 sets of documents included in two documents_iAnd (Step S
66).

[0139] The following candidate determination unit 105, and collected next candidate N _i of areas i of N1 _i ∪N2 _i (step S6
7). Next candidate determining unit 105 accesses the URL table 120, given the classification identification information corresponding to the value of the current i to be collected next candidate N _i. Document collection unit 101 collects the collected next candidate N _i from the network (step S68).
The document collection unit 101 accesses the URL table 120 and sets the collected flag of the collected next collection candidate N _i (newly collected document group) to “ON (1)”. Thus, the document collection unit 101 adds the new collection documents and new documents D _i in document group D _i (step S69).

Subsequently, the reference relation extracting unit 102 sets i to 1
Increment (step S70), step S63
Return to The document collection device 200 repeats the above processing until i exceeds n.

If i exceeds n (step S63: Ye
s), the reference relation extracting unit 102
Refers to the, based on the collected flag and sector identification information, counts the number of documents in the document group D _i, determines whether the number of documents for each document group D _i is defined number or more (step S71). If there is a document group D _k (k is an arbitrary number from 1 to n) in which the number of documents is not more than the specified number, step S62
Then, the reference relationship extracting unit 102 repeats the processing of step S63 and subsequent steps with i = k.

The document group D in which the number of documents is not more than the prescribed number
_When there are a plurality of _k , for example, when there are D _k1, D _k2 and D _k3, and when i = k1, k2 and k3, the processing after step S63 is repeated. If D document number for all the collected document group D _i to ₁ from D _n is greater than or equal to the prescribed number (step S71: Yes), the process ends.

Thus, when documents in a certain field are collected, it is possible to use a document group in that field as a positive document group PS and use the union of documents in other remaining fields as a negative document group NS. Therefore, the processing related to the negative example document group NS is not wasted.

[0144] Also, according to a modification of the second embodiment, the document group D ₁ of the certain areas as positive sample document group PS, when focusing for gathering documents about the field, used as a negative sample document group NS The documents in other fields are larger than the positive documents PS. Furthermore, since the negative example document group NS itself is a document group related to another field, it is semantically constant. In the second embodiment which is not a modified example, if the collection proceeds to a certain extent or more, the positive example document group PS becomes larger while the documents are moved from the negative example document group NS to the positive example document group PS. R shown in
_A situation can occur in which the second term of _score (d, PS, S) increases. As a result, the accuracy of the collection may have been reduced, but in a modified example, the possibility is reduced.

Hereinafter, the accuracy of collecting a document related to a specific field in the document collection device according to the second embodiment will be described with reference to FIGS. FIG. 20 shows a document set of about 6.70 million URLs collected from the network as a whole set D, and 15,000 URLs including "Linux" in the URL
Is the correct answer L, and about 5,000 URLs arbitrarily selected are the positive document set PS and the other URLs (D-PS) are the negative document set NS as the initial document. Show.

In FIG. 20, the horizontal axis indicates the number of times of repeated collection i, and the vertical axis indicates the precision or recall. The recall is shown by a polygonal line, and the precision is shown by a square plot. Here, the relevance and the recall for the positive example set S _i obtained in the i-th iteration are shown by the following equations (7) and (8).

Precision = | S _i ∩L | / | S _i | (7) Reproducibility = | S _i ∩L | / | L | (8) That is, the precision is This is the ratio of correct examples L included in the correct example document group S in the correct example set S _i , and indicates the number of documents (so-called garbage) not included in the target field. The recall rate is a ratio of the correct answer examples L included in the correct example document group S _i in the correct answer examples L, and indicates a small possibility that documents included in the target field are not collected (so-called omission). FIG.
As shown in the figure, when the number of repetitions reaches about 73, the recall rate sharply decreases, but it can be seen that the relevance rate and the recall rate are good after several dozen repetitions. The reason why the recall decreases when the number of repetitions becomes about 73 is as follows.
It is considered that so-called garbage is for calling garbage.

In FIG. 21, the URL includes "What's New".
The result of a similar experiment performed when 14,000 URL was used as Correct Example L is shown. As shown in FIG. 21, when the number of repetitions becomes about several times, the matching rate sharply decreases. This is probably because contents such as What's New do not have much semantic association (connection) with each other.

From the experimental results shown in FIG. 20, it can be seen that the document collection apparatus according to the present embodiment can efficiently collect a semantically related document group. Each server and each terminal described above can be configured using an information processing device (computer) as shown in FIG. The information processing device 300 of FIG.
1, memory 302, input device 303, output device 304,
An external storage device 305, a medium drive device 306, and a network connection device 307 are provided, and these are connected to each other by a bus 308.

The memory 302 is, for example, a ROM (Read O
nly Memory), RAM (Random Access Memory), etc., and stores programs and data used for processing.
The CPU 301 performs necessary processing by executing a program using the memory 302.

Each device and each unit constituting each server and each terminal described above is stored as a program in a specific program code segment of the memory 302. The input device 303 is, for example, a keyboard, a pointing device, a touch panel, or the like, and is used for inputting an instruction or information from a user. The output device 304 is, for example,
A display, a printer, or the like;
Is used for inquiring the user of the program and outputting the processing results and the like.

The external storage device 305 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, or the like. The above-described programs and data are stored in the external storage device 305, and are stored in the memory 30 as necessary.
2 can also be used.

The medium driving device 306 is a portable recording medium 30.
9 to access the recorded contents. As the portable recording medium 309, a memory card, a memory stick,
Floppy (registered trademark) disk, CD-ROM (Comp
A recording medium that can be read by any information processing device, such as an act disc read only memory), an optical disk, a magneto-optical disk, and a DVD (Digital Versatile Disk) is used.
The above-described program and data can be stored in the portable recording medium 309, and can be used by loading them into the memory 302 as needed.

The network connection device 307 is a LAN,
An external device is communicated via an arbitrary network (line) such as a WAN, and data conversion accompanying the communication is performed. Also,
If necessary, the program and data described above can be received from an external device, and loaded into the memory 302 for use.

FIG. 23 shows a recording medium and a transmission signal which can be read by an information processing apparatus capable of supplying a program and data to the information processing apparatus 300 of FIG. Note that, when the present invention is used by an information processing apparatus, it can be read by the information processing apparatus for causing the information processing apparatus to perform the same function as the function realized by each configuration of the above-described embodiment of the present invention. It can also be configured as a simple recording medium 309.

In the embodiment, a program for causing an information processing apparatus to execute the same processing as that performed by each apparatus is stored in a recording medium 309 readable by the information processing apparatus in advance, and as shown in FIG. The program is read from the recording medium 309 by the information processing device 300 and temporarily stored in the memory 302 or the external storage device 305 of the information processing device 300. The stored program is stored in the CPU 301 of the information processing device 300. Read and execute.

Also, the program (data) provider 310
The transmission signal itself transmitted via the line 311 (transmission medium) when the program is downloaded from the computer to the information processing apparatus 300 also has the function corresponding to each apparatus described in the above-described embodiment of the present invention as general-purpose information. It can be performed by a processing device.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various other modifications are possible. For example, by configuring the document collection device 100 according to the first embodiment and the document collection device 200 according to the second embodiment to be assembled, documents may be collected according to fields for a community.

Further, each unit and each DB constituting the document collection device 100 or 200 operate in cooperation with each other to realize a series of business processes. These units and DBs may be provided in the same server, or may be provided in different servers and operate in cooperation via a network.

(Supplementary Note 1) A document collection method for collecting documents from a network, the method comprising: collecting at least a predetermined number of documents from a community on the network based on a reference relation of the documents; Collecting a predetermined number of documents or more, and then collecting documents from inside and outside the community based on the reference relation of the collected documents.

(Supplementary Note 2) The importance indicating the degree of importance of the collected documents is calculated based on the reference relation of the collected documents and the information indicating the location on the network.
The document collection method according to claim 1, wherein a document to be collected is determined based on the reference relation and the importance.

(Supplementary Note 3) The document collection method according to Supplementary Note 2, wherein the documents to be collected are determined for each of the inside and outside of the community.

(Supplementary Note 4) The document collection method according to Supplementary Note 3, wherein a result of searching the collected document group is presented separately inside and outside the community.

(Supplementary note 5) The document collection method according to Supplementary note 2, wherein whether or not the document is in the community is determined based on information indicating a location on the network.

(Supplementary Note 6) A document collection method for collecting documents from a network, wherein a positive example document group that is a document group related to a certain field and a negative example document group that is a document group related to a field that is less relevant to the field. A document to be collected relating to the field is determined based on a reference relationship between the positive example document group and the negative example document group, and the document to be collected is collected from the network. Collection method.

(Supplementary Note 7) On the basis of the reference relation, a reference degree indicating a degree of reference from only the documents in the positive example document group is calculated, and a document having a high reference degree is determined as the document to be collected. 6. The document collection method according to claim 6, further comprising:

(Supplementary Note 8) A co-reference degree indicating a degree of reference from the collected documents with respect to a document referred to from the collected documents referring to the documents in the positive document group based on the reference relation. The document collection method according to claim 6, wherein a document having a high degree of co-reference is determined as a document to be collected.

(Supplementary Note 9) The negative example document group is a union of documents relating to a plurality of fields.
Document collection method according to attachment 6.

(Supplementary Note 10) The document collection method according to Supplementary Note 1, wherein the collected document group is put together based on a reference expression used in the collected document.

(Supplementary Note 11) The document collection method according to Supplementary Note 1, wherein a keyword is assigned to the collected document based on a reference expression used in the collected document.

(Supplementary Note 12) The document collection method according to Supplementary Note 11, wherein if the reference expression is a reference expression used irrespective of a reference destination document, the reference expression is not used as a keyword.

(Supplementary Note 13) The number of different documents referenced by the reference expression is counted, and if the number of different documents is equal to or greater than a certain number, the reference expression is not used as a keyword. 11. The document collection method according to item 11.

(Supplementary Note 14) If the number of the different documents is less than a certain number, the number of references, which is the number of times each collected document is referred to by the reference expression, is counted, and the number of the different documents and 12. The document collection method according to appendix 11, wherein it is determined whether or not the reference expression is a keyword based on the reference count.

(Supplementary Note 15) The keyword based on the reference expression is combined with a keyword extracted from the body of the collected document and a keyword extracted from information indicating a location on the network of the collected document. The document collection method described in Supplementary Note 11.

(Supplementary Note 16) A search method for searching for a document belonging to a community on a network, in which information for searching for a document is transmitted to a server, and the information is divided into and out of the community based on the information for searching. Receiving the retrieved documents together with information indicating the degree of importance to the community.

(Supplementary Note 17) A document collection device that collects documents from a network, comprising: a next candidate determination unit that determines a next collection candidate that is a candidate for a document to be collected next based on the reference relation of the documents. Community determining means for determining whether or not the document is a document in a community on the network based on information indicating a location of the document on a network; and document collecting means for collecting the next collection candidate from the network Wherein the document collection means collects a predetermined number or more of documents from inside the community and then collects documents from inside and outside the community.

(Supplementary Note 18) A document collection device for collecting documents from a network, wherein a reference is made to a group of positive documents which is a group of documents related to a certain field and a group of negative examples which is a group of documents related to a field which is less relevant to the field. A document comprising: a next candidate determining unit that determines a next collection candidate that is a candidate for a document to be collected next based on the relationship; and a document collection unit that collects the next collection candidate from the network. Collection device.

(Supplementary Note 19) A computer-readable recording medium in which a program for causing a computer to execute a control for collecting a document from a network by being executed by the computer is recorded. Collecting at least a predetermined number of documents from the community on the network, collecting at least the first predetermined number of documents from the community, and then collecting documents from inside and outside the community based on the reference relation of the collected documents. A recording medium on which a program for causing a computer to perform control including collecting is recorded.

(Supplementary Note 20) A computer-readable recording medium in which a program for causing a computer to collect documents from a network by executing the computer is recorded, and is a group of documents related to a certain field. Based on the reference relationship between the positive document group and the negative document group that is a document group related to the field with a small relation to the field, a document to be collected is determined for the field, and the document to be collected is collected from the network. A recording medium on which a program for causing a computer to perform control including the above is recorded.

(Supplementary Note 21) A computer data signal embodied in a carrier wave, which represents a program for causing a computer to perform control for collecting documents from a network, wherein the program causes the computer to execute the following: Based on the document reference relationship, a predetermined number or more of documents are collected from within the community on the network, and after collecting the first predetermined number or more of documents from the community, based on the reference relationship of the collected documents,
(Supplementary Note 22) A computer program which, when executed by a computer, causes the computer to perform control of collecting a document from a network, based on a reference relationship of the document. After collecting a predetermined number or more of documents from the community on the network, collecting the first predetermined number of documents or more from the community, and collecting documents from inside and outside the community based on a reference relation of the collected documents. A computer program for causing the computer to perform control including:

(Supplementary Note 23) A computer program which, when executed by a computer, causes the computer to control the collection of documents from a network, wherein a set of positive documents, which is a group of documents related to a certain field, And a negative example document group, which is a document group relating to a field having a small relation, determine a document to be collected regarding the field based on the reference relationship between the positive example document group and the negative example document group, from the network. A computer program for causing the computer to perform control including collecting the document to be collected.

[0182]

As described in detail above, the present invention provides
When collecting documents for a certain use, the documents to be collected are determined based on the reference relation between the documents, and by collecting the determined documents, the use can be quickly performed without depending on the language. Documents can be selected and collected.

Also, by collecting collected documents based on the reference expression and assigning a keyword to each collected document, access to the collected documents can be facilitated. Further, since the content of the document body is not analyzed, it is possible to quickly assign a keyword without depending on the language.

[Brief description of the drawings]

FIG. 1 is a principle diagram of the present invention.

FIG. 2 is a configuration diagram of a document collection device according to the first embodiment.

FIG. 3 is a diagram illustrating an example of a data structure of a URL table.

FIG. 4 is a diagram illustrating an example of a data structure of a reference relation table.

FIG. 5 is a diagram illustrating an example of a data structure of a reference expression table.

FIG. 6 is a diagram illustrating an example of a data structure of a reference count table.

FIG. 7 is a flowchart illustrating a rough flow of a process performed by the document collection device according to the first embodiment.

FIG. 8 is a flowchart illustrating a process of determining a next collection candidate when collecting documents in a community.

FIG. 9 is a flowchart illustrating a process of ranking collected documents and reference destination documents.

FIG. 10 is a flowchart illustrating a process of selecting collected documents.

FIG. 11 is a flowchart illustrating a keyword assignment process.

FIG. 12 is a diagram illustrating an example of a screen for providing collected documents.

FIG. 13 is a configuration diagram of a document collection device according to a second embodiment.

FIG. 14 shows LT (S), LT (p), LS (d, X),
FIG. 14 is a diagram illustrating a reference relationship of documents that is indicated by LS (A, X).

FIG. 15 is a diagram illustrating a reference relationship of a document represented by CC (d, A, X).

FIG. 16 is a flowchart illustrating a process performed by the document collection device according to the second embodiment.

FIG. 17 is a diagram illustrating a reference relationship that each set included in an expression for calculating a reference degree means.

FIG. 18 is a diagram illustrating a reference relationship implied by each set included in an equation for calculating a co-reference degree.

FIG. 19 is a flowchart illustrating processing performed by a document collection device according to a modification of the second embodiment.

FIG. 20 is a diagram (part 1) illustrating an experimental result of the collection accuracy of the document collection device.

FIG. 21 is a diagram (part 2) illustrating an experimental result of the collection accuracy of the document collection device.

FIG. 22 is a configuration diagram of an information processing apparatus.

FIG. 23 is a diagram illustrating a recording medium, a transmission signal, and a transmission medium that supply a program and data to the information processing apparatus.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1, 100, 200 Document collection device 2 Document collection means 3 Reference relation extraction means 4 Community discrimination means 5 Secondary candidate judgment means 6 Ranking means 7 URL judgment means 8 Reference / co-reference degree calculation means 9 Grouping means 10 Keyword assignment means 20 Collected document group 21 Primary collection candidate 22 Reference relationship between documents 23 Collection document file 101 Document collection unit 102 Reference relationship extraction unit 103 Community determination unit 104 Candidate determination unit 105 Ranking unit 106 Grouping unit 107 Keyword assignment unit 120 URL table 121 Reference relationship Table 122 Reference expression table 123 Reference count table 130 Excellent content 140 Search engine 141 Index 150 Classification engine 160 Server 170 Browser 180, 181, 182 Screen 201 Reference / co-reference Degree table 210 sectoral superior content 300 the information processing apparatus 301 CPU 302 memory 303 input device 304 output device 305 external storage device 306 medium drive 307 network connection device 308 bus 309 portable recording medium 310 a program (data) provider 311 lines

Claims (12)

[Claims] 1. A document collection method for collecting documents from a network, comprising: collecting at least a predetermined number of documents from a community on the network based on a reference relationship of the documents; Collecting a plurality of documents and then collecting documents from inside and outside the community based on a reference relationship of the collected documents. 2. An importance degree indicating a degree of importance of the collected document is calculated based on a reference relation of the collected document and information indicating a location on a network, and based on the reference relation and the importance. The document collection method according to claim 1, wherein a document to be collected is determined. 3. The document collection method according to claim 2, wherein the document to be collected is determined for each of the inside and outside of the community. 4. The document collection method according to claim 3, wherein a result of searching for the collected documents is presented separately inside and outside the community. 5. A document collection method for collecting documents from a network, comprising: providing a group of positive documents as a group of documents related to a certain field and a group of negative documents as a group of documents related to a field that is less relevant to the field. Determining a document to be collected in the field based on a reference relationship between the positive example document group and the negative example document group, and collecting the document to be collected from the network. . 6. A reference degree indicating a degree of being referred to only from the documents of the positive example document group is calculated based on the reference relation, and a document having a high reference degree is determined as a document to be collected. The document collection method according to claim 5, wherein 7. A co-reference degree indicating a degree of reference from a collected document with respect to a document referred to from a collected document referring to a document in the positive document group based on the reference relation. The document collection method according to claim 5, wherein the document having a high degree of co-reference is determined as a document to be collected. 8. The document collection method according to claim 5, wherein the negative example document group is a union of documents relating to a plurality of fields. 9. The document collection method according to claim 1, wherein the collected documents are compiled based on a reference expression used in the collected documents. 10. The document collection according to claim 1, wherein a keyword is assigned to the collected document based on a reference expression used in the collected document. Method. 11. A computer program which, when executed by a computer, causes the computer to control the collection of documents from a network, wherein the computer program executes the control based on a reference relationship of the documents from within a community on the network. Collecting at least a predetermined number of documents from the community, collecting at least the first predetermined number of documents from the community, and then collecting documents from inside and outside the community based on a reference relationship of the collected documents. A computer program characterized by being performed by a computer. 12. A computer program which, when executed by a computer, causes the computer to control the collection of documents from a network, comprising: a set of regular documents which is a group of documents related to a certain field; And a negative example document group, which is a group of documents related to a small number of fields, determine a document to be collected based on the reference relationship between the positive example document group and the negative example document group, and collect the document from the network. A computer program for causing the computer to perform control including collecting documents to be performed.