JP2010140373A - Method and device for detecting document group - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a document group detecting method and document group detection device for detecting a targeted document group provided on a network. <P>SOLUTION: A document collecting means 12 detects a specific document and documents belonging to a document group composed of a plurality of documents connected to the specific document on the basis of a set keyword, and specifies the specific document. Then the document collecting means 12 uses the specific document to collect collected documents. A feature accumulating means 13 analyzes the collected documents and accumulates feature items representing features of the document group. A document group determining means 14 analyzes an accumulation result of the feature items on the basis of a feature rule, and determines whether or not the document group is a document group to be detected. When it is a document group to be detected, the document group is registered as a candidate for a document group and presented to a user. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a document group detection method and a document group detection apparatus, and more particularly to a document group detection method for detecting a predetermined document group which is a set of documents provided on a network and managed by one or more computers. And a document group detection apparatus.

  In recent years, technical terms that have been increasing day by day due to rapid progress in technology have become difficult to deal with in encyclopedias and dictionaries printed on paper. On the other hand, on the network, there is a group of documents that explain such technical terms and are managed by one or more computers. As the most widespread currently available, the World Wide Web (hereinafter referred to as the WWW) provided on the Internet is a group of documents that explain such technical terms. There are many. Such a document on the WWW is called a web page, a document group, or a place on the Internet where a document group is placed. Hereinafter, a website in which documents that explain technical terms are gathered in this way is called a dictionary site or a term explanation site, and a web page is called an explanation page. The explanation page of the dictionary site is updated daily, and you can see the explanation of the latest term by using these pages.

  Further, there is a system that automatically puts a link to an explanation page of a term for a term in an arbitrary Web page. Such a system is called an auto link system. In the autolink system, an autolink dictionary that associates a word to be autolinked with a URL (Uniform Resource Locator) of an explanation page of the word is created in advance. When a word registered in the autolink dictionary is detected in the sentence of the target HTML (Hyper Text Markup Language) document, a link to the URL associated with the word is pasted. An HTML (with link) document created in this way is output and provided to the user.

In creating these auto-link dictionaries, information on the words registered in the dictionary site and the explanation page URL can be used as each word and its link destination URL.
However, it is not easy to detect a dictionary site from a large number of Web sites existing on the network. In a conventional search engine that is generally used for Web site detection, search is performed in units of Web pages. Therefore, a Web site that is a set of pages has to be detected manually. Detection of a dictionary site to be registered in the autolink dictionary is also performed manually, and management such as registration of the dictionary site and periodic maintenance of registration information has been costly.

Therefore, in order to search for information in units of websites, the meta information of each page is used to classify into link types, the parent page is searched based on the classification to estimate the internal structure of the website, and A method for outputting a search result has been proposed (see, for example, Patent Document 1). Also, a method has been proposed in which the URL of the hit page and the score reflecting the weight of the word are calculated for each scoring site representing the degree of suitability of the search results, and the search results are output in the order of the scores (for example, patents) Reference 2).
JP 2003-186883 A JP 2003-186901 A

However, a conventional search in units of documents (for example, Web sites) on a network has a problem that it is not possible to identify whether the searched documents are desired.
In a conventional search in units of Web sites, pages hit in the search are analyzed, the internal structure of the Web site that is an aggregate of the pages is estimated, and a Web site suitable for the search purpose is detected based on the internal structure. However, conforming to the search purpose means that the degree of matching with a search request by a keyword or the like is high, and does not determine whether the Web site itself is suitable for the purpose.

  For example, when linking an explanation of a term to a term in a sentence with an auto link system, it is desirable that the linked information is from a dictionary site. This is due to the fact that the neutrality and generality of the information content of web page information that is not a dictionary site is often not guaranteed. Therefore, the neutrality and generality of the content of the explanation cannot be guaranteed simply by linking the term explanation Web page to the term in the sentence. For this reason, it is necessary to eliminate this type of page as much as possible as a link destination and perform a search oriented to a dictionary site.

  For these reasons, in the conventional auto link system, the dictionary site is detected manually. However, detecting an appropriate dictionary site from a huge number of Web sites is not an easy task. In addition, since it is a manual operation, the management cost of the dictionary is high, and there is also a problem that a service provider that provides an auto link service cannot frequently add a dictionary.

  In addition to auto-link usage, it is important to automate the maintenance of a dictionary that associates terms with the URLs of pages associated with them. It is essential to be.

  SUMMARY An advantage of some aspects of the invention is that it provides a document group detection method and a document group detection apparatus for detecting a target document group provided on a network.

  In order to solve the above problems, there is provided a document group detection method for detecting a predetermined document group which is a set of documents provided on a network and managed by one or more computers. In this document group detection method, a collection procedure, a feature counting procedure, a document group determination procedure, and a procedure for outputting a document group are executed by a computer. In the collection procedure, one of the subordinate documents is searched using a specific keyword for a document group having a hierarchical structure in which a plurality of subordinate documents exist under the specific document. Then, the specific document is detected based on the searched subordinate document, and a plurality of subordinate documents under the specific document are collected. In the feature counting procedure, for each of the collected specific document of the document group and a plurality of subordinate documents, connection information indicating a relationship with a specific other document added to an arbitrary character string in the subordinate document is extracted. Then, the number of states having a specific relationship between the subordinate document and the associated contact document is totaled. In the document group determination procedure, the feature rule is read out from the feature rule storage means in which the feature rule that is a condition using a specific relationship is stored. Then, it is determined whether the number of states of the specific relationship of the document group satisfies the characteristic rule condition, and the document group satisfying the condition is registered in the target document group candidate. In the procedure for outputting the document group, the document group registered in the target document group candidate is output.

  According to such a document group detection method, a subordinate document is searched using a specific keyword for a document group having a hierarchical structure in which a plurality of subordinate documents exist under the specific document. A specific document is detected based on the retrieved document, and a plurality of subordinate documents under the specific document are collected. Then, the link information added to the character string of the collected subordinate document is extracted. Based on this link information, the number of states in which the relationship between the subordinate document and the link destination document associated with the subordinate document becomes a specific relationship is totaled. Then, a feature rule that is a condition using a specific relationship is read from the feature rule storage unit, and it is determined whether or not the number of states of the specific relationship satisfies the condition of the feature rule. Then, the document group satisfying the condition is registered as a target document group candidate and output.

  In order to solve the above problem, a document group detection apparatus is provided in which a computer executes the document group detection method.

  According to the disclosed document group detection method and document group detection apparatus, a document group in which a plurality of subordinate documents exist under a specific document is searched using a keyword. It is determined whether the retrieved document group satisfies the feature rule of the document group to be detected, and the target document group candidate is determined. Thus, if a keyword is set as an example, a target document group including a document that matches the keyword and having a feature based on the feature rule is automatically detected. As a result, the user's task of detecting a document group can be greatly reduced.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, an outline of the invention will be described, and then specific contents will be described.
Here, the document group to be detected is a collection of documents provided on the network, and is managed by one or more computers. Further, this document group has a hierarchical structure in which a specific document and a plurality of subordinate documents exist under the specific document. The specific document is a document including a character string to which link information relating other documents belonging to the same document group is added. For example, the specific document is a table of contents for browsing subordinate documents, an index, or the like. In the specific document, concatenation information for associating a corresponding subordinate document with a title such as a table of contents or a character string such as a word characterizing the subordinate document is added. The subordinate document is a document associated with the specific document by the connection information of the specific document. In addition, for a subordinate document, when a character string appearing in a document is associated with another document, connection information that associates the character string with another document is added to the character string. For example, if the document group is a dictionary document group, it has a hierarchical structure of a specific document in which an index of terms to be explained is described and a term explanation document that explains terms. In the specific document, the link information of the corresponding glossary document is added to the character string representing the term explained in the glossary document. In addition, when a term explained in another glossary document appears in the glossary document, the linked information of the glossary document corresponding to the character string representing the term is also added.

In the present invention, the connection relationship between documents is analyzed based on the connection information. Then, a document group satisfying the characteristics to be detected is detected and presented to the user.
FIG. 1 is a diagram showing an outline of the invention.

  The document group detection device 10 includes a document storage unit 11a, a feature rule storage unit 11b, a total information storage unit 11c, a document group candidate storage unit 11d, and a document group storage unit 11e, a document collection unit 12, and a feature total unit. 13, each of the processing means of the document group determination means 14 and the document group presentation means 15.

  The document data collected by the document collection unit 12 is stored in the document storage unit 11a. The feature rule storage unit 11b stores a feature item representing the feature of the document group to be detected, a feature rule for determining whether the searched document group is the target document group, and the like. The totaling information storage unit 11c stores the totaling result of the feature items that the characteristic totaling unit 13 analyzed and totalized the collected documents. The document group candidate storage unit 11d stores information regarding the target document group candidate that the document group determination unit 14 determines to be a detection target document group candidate based on the feature rule. Of the target document group candidates presented by the document group presentation unit 15, the document group storage unit 11e is described in the identification information of the document group designated as the target document group by the user and the specific document belonging to this document group. The document group information including the character string to which the link information is added is stored.

  The document collection unit 12 searches the network based on the set keyword and detects a predetermined document group including documents that match the keyword. The keyword is specified by the user as an arbitrary phrase that represents the characteristics of the contents of the document group to be detected, or as an example. A word or phrase for obtaining the target information is set. For example, when it is desired to obtain network-related information, “network”, “LAN (Local Area Network)”, and the like are set. In addition, an address on a network for acquiring a specific document may be designated. When an arbitrary phrase is set as a keyword, the address of the document related to the keyword is acquired by the search engine. At this time, it is also possible to expand the search target by further searching by adding a preset phrase to the keyword. The address of the specific document is detected based on the address having the identification information on the network of the retrieved document. In general, an address indicating the position of a document on a network has a hierarchical structure similar to the structure of a document group. Therefore, it is possible to predict the address of the specific document in the upper hierarchy from the address of the subordinate document searched. In addition, a method for acquiring the address of a higher-level document is well known, and here, any one method is used. A specific document is acquired based on the address of the specific document acquired in this way. When the address of the specific document is directly designated, the process starts from acquisition of the specific document. In the specific document, the link information for associating the corresponding subordinate document is added to the character string provided with the information related to the subordinate document. Therefore, the link information added to the character string of the specific document is extracted, and the document data of the subordinate document is collected based on the link information. The document data of the collected specific document and subordinate document is stored in the document storage unit 11a for each document group.

  The feature counting unit 13 extracts and analyzes the link information added to the character string of the document including the specific document stored in the document storage unit 11a for each document group, and analyzes the original document in which the character string is described. The number of states that have a specific relationship with the linked document is counted. This is called feature item aggregation. In other words, it is analyzed whether the relationship between the original document and the link destination document associated by the link information satisfies the feature item (specific relationship) that characterizes the document group to be detected, and the number of states that are satisfied For each feature item. These aggregation processes are performed for each document group. The total result is stored in the total information storage unit 11c for each document group.

  The document group determination unit 14 determines whether or not this document group satisfies the condition of the document group to be detected based on the result of the feature item for each document group by the feature totaling unit 13. The characteristic items used for the determination and the determination conditions such as the threshold value are stored in advance in the characteristic rule storage unit 11b. When the document group satisfies the condition, this document group is selected as the target document group candidate, and the document group identification information is registered in the target document group candidate table. Further, the evaluation result based on the feature rule may be quantified as a feature score. In this case, the feature score calculation method is also defined in the feature rule. The target document group candidate table is stored in the document group candidate storage unit 11d. At this time, together with the document group identification information, the calculated feature score, the connection information of the specific document, and the like may be stored in the document group candidate storage unit 11d. Further, the determination is performed by combining the total results of arbitrary feature items. By determining by combining a plurality of feature items, the accuracy (probability) that the target document group candidate is the detection target document group increases.

  The document group presentation unit 15 presents identification information of the document group registered in the target document group candidate by the document group determination unit 14 to the user. Then, the identification information of the document group selected by the user among the target document group candidates is registered as the target document group. The identification information of the selected target document group is stored in the document group storage unit 11e. At this time, the connection information of the specific document may be stored in the document group storage unit 11e together with the document group identification information.

An operation of the document group detection apparatus 10 having such a configuration and a document group detection method to be executed will be described.
When a keyword is input, the document collection unit 12 searches for a document including the keyword provided on the network based on the keyword. Then, based on the address of the retrieved document, the address of the specific document is detected, and the specific document is acquired. When the address of the specific document is specified as a keyword, the specific document is directly acquired without performing the search process. Link information with the subordinate document as the link destination is added to the character string described in the specific document. Therefore, by acquiring the specific document, connection information to the subordinate document is also acquired. The document collection unit 12 collects the subordinate document based on the link information to the subordinate document thus acquired, and stores the collected subordinate document data in the document storage unit 11a. A series of processing is performed for each document searched using a keyword. Thereby, a plurality of document data including specific documents belonging to the document group is stored in the document storage unit 11a for each document group corresponding to the retrieved document.

  Next, for each document group, the feature counting unit 13 analyzes the connection information added to the document data stored in the document storage unit 11a, and the relationship between the original document and the connection destination document is a feature item. Count the number that meets. In the character string described in the subordinate document, if there is another document related to the character string, the link information is added. The feature counting means 13 extracts the link information added to each document in this way, and the feature item that characterizes the document group to be detected by the relationship between this document and the link destination document specified by the link information. Analyze whether it meets. For each feature item, the number of pieces of linked information that satisfy the feature item is totaled. The total result is stored in the total information storage unit 11c as total information. Subsequently, the document group determination unit 14 reads out the feature rule from the feature rule storage unit 11b. Further, the total information stored in the total information storage unit 11c is read out, and it is determined whether or not the document group is a target document group candidate. In the feature rule, a criterion is defined as to whether or not it can be determined that the document has the feature of the target document group based on the total result for each feature item. The totaling result for each feature item by the feature tabulating unit 13 is checked against the feature rule to make a determination. At this time, the probability that the document group is the target document group may be quantified as a feature score based on the feature rule. When the condition defined by the feature rule is satisfied, this document group is set as a candidate document group candidate, and document group identification information is stored in the document group candidate storage unit 11d. At this time, if necessary, the calculated feature score, the connection information of the specific document, and the like are also stored in the document group candidate storage unit 11d. When the condition defined by the feature rule is not satisfied, this document group is not set as a target document group candidate.

  When there is a document group candidate presentation request from the user, the document group presentation unit 15 reads out identification information of the document group candidate stored in the document group candidate storage unit 11d and presents it to the user. For example, the identification information of the target document group candidate is displayed on the display device. At this time, a feature score, connection information of a specific document, and the like may be provided. When the user determines that the presented target document group candidate is the target document group, the user designates the target document group candidate as the target document group. Upon receiving the designation, the document group presentation unit 15 sets the designated target document group candidate as the target document group, and registers the identification information of the document group in the document group storage unit 11e. At this time, the connection information of the specific document may be stored in the document group storage unit 11e together with the identification information of the document group.

  As a result of the above processing, when a keyword is set as an example of information desired by the user, a document group having a document including the keyword and having a feature specified in advance by a feature rule is automatically generated. And a list of detected document groups is presented. As described above, since the target document group is automatically detected, the work of detecting the document group can be greatly reduced. In addition, management work such as periodic maintenance is facilitated. Furthermore, the specific document includes a predetermined term (character string) and concatenation information that indicates the location of the document related to the character string. When creating a dictionary, the specific document is concatenated with the character string. Information can be used as it is. Thus, there is also an advantage that a dictionary can be easily created.

  Hereinafter, the invention will be described in detail with reference to the drawings, taking as an example a case where the invention is applied to a document site provided on the Internet, for example, a dictionary site detection system that detects a dictionary site. The detected dictionary site is used as a dictionary candidate applied to an auto link system or the like. In the embodiment, a document acquired by a search by a viewer is a Web page (hereinafter referred to as a page), and a document group is a Web site (hereinafter referred to as a site) that is a set of pages. The site is composed of a top page corresponding to a table of contents and an index and other pages linked from the top page. Further, the site is managed by one or more computers, and an identifier on the Internet of such a computer group becomes a domain. Thus, a site can be identified by a domain common to the page URL. Many of the pages are described in HTML. In HTML, a character string in a page and another page can be linked as anchor text.

FIG. 2 is a diagram illustrating a configuration example of the dictionary site detection system.
In the dictionary site detection system, a dictionary site detection server 100 that detects a dictionary site and a client device 200 of a user that issues a search site detection instruction are connected via a network 300.

  The dictionary site detection server 100 is a document group detection device, and detects a candidate dictionary site that provides a document explaining a predetermined term on the network in response to a request from the client device 200. The client device 200 is a creator's device that creates an autolink dictionary, and includes a browser 210 and an input unit 220. The browser 210 displays an HTML format detection result obtained from the dictionary site detection server 100 on a display device (not shown). The input means 220 inputs the creator's instruction and notifies the dictionary site detection server 100. The network 300 is, for example, the Internet.

  The configuration of the dictionary site detection server 100 will be described. The dictionary site detection server 100 includes an extended search rule (storage device) 111, an acquisition site (storage device) 112, a link feature database (hereinafter referred to as DB) 113, a link feature rule (storage device) 114, a dictionary site determination rule ( Storage device) 115, dictionary addition rule (storage device) 116, dictionary candidate DB 117 and dictionary DB 118, site acquisition unit 120, link information extraction unit 130, link feature totaling unit 140, dictionary site determination unit 150, dictionary Each of the processing means of the entry candidate creation unit 160 and the user presentation unit 170.

  The extended search rule (storage device) 111 stores an extended search rule that defines a rule for extending a keyword input for a search. For example, character strings that frequently appear on the explanation page of terms such as “to”, “term”, and “explanation” are defined together with use conditions as necessary. By adding such an extended character string to the input keyword, it is possible to narrow down the search to a result that seems to be an explanation page.

The acquisition site (storage device) 112 is the document storage unit 11a, and stores the URL of the site acquired by the site acquisition unit 120, the page data of the collected pages, and the like.
The link feature DB 113 is a tabulation information storage unit 11c, and stores tabulation results for each feature item representing features of the tabulated site and features related to link information regarding the acquired site.

The link feature rule (storage device) 114 stores a link feature rule that defines a feature item extraction rule that represents the feature of the dictionary site.
The dictionary site determination rule (storage device) 115 stores a dictionary site determination rule that presents a rule for determining whether or not a site is a dictionary site based on characteristics of the dictionary site. In the dictionary site determination rule, a condition for determining whether or not the site is a dictionary site is defined by using the total result of the feature items. The link feature rule (storage device) 114 and the dictionary site determination rule (storage device) 115 correspond to the feature rule storage unit 11b.

The dictionary addition rule (storage device) 116 stores a dictionary addition rule that defines a rule for adding the detected dictionary site candidate to the dictionary.
The dictionary candidate DB 117 is a document group candidate storage unit 11d, and stores a dictionary candidate table in which information about sites determined as dictionary candidates by the dictionary entry candidate creation unit 160 is set.

The dictionary DB 118 is a document group storage unit 11e, and stores a dictionary site table in which information related to sites registered in the dictionary site by the user is set.
The site acquisition unit 120 is the document collection unit 12 and collects documents on a site including documents searched by keywords. When a keyword is input, the extended search rule stored in the extended search rule (storage device) 111 is read, and an extended character string is added to the keyword input according to the extended search rule. In this way, the keyword is expanded, and the page is searched using the expanded keyword. Then, the domain name is extracted from the retrieved page and the top page (specific document) is detected. The top page is a table of contents, an index, and the like, and link information (link information) to related pages is added to a character string representing the item. Therefore, the subordinate pages are collected based on the link information of the top page. The collected page data is stored in the acquisition site (storage device) 112 for each site. The page data is document data described in HTML. When the URL of the top page is directly specified as a keyword, the process up to the top page detection is omitted and the process is performed thereafter.

  The link information extraction unit 130 and the link feature totaling unit 140 are the feature totaling unit 13. The link information extraction unit 130 analyzes the page data acquired by the site acquisition unit 120 and extracts the anchor text, the link destination URL added to the anchor text, and the title of the link destination page as link information. The link information is extracted for all pages acquired by the site acquisition unit 120, and the top page and other pages are also targeted. Each piece of extracted information is stored in the link feature DB 113. The link feature totaling unit 140 analyzes the link information extracted by the link information extracting unit 130 based on the link feature rule stored in the link feature rule (storage device) 114. In the link feature rule, feature items corresponding to the features of the dictionary site are defined, link information is analyzed, and the number of link information satisfying the feature items is totaled. For example, if the link feature rule is “there is a feature that the link destination is in the same site”, in order to analyze the feature, the number of links to the same site and the number of all links are totaled. The obtained tabulation results are stored in the link feature DB 113.

  The dictionary site determination unit 150 and the dictionary entry candidate creation unit 160 are the document group determination unit 14. The dictionary site determination unit 150 reads the dictionary site determination rule stored in the dictionary site determination rule (storage device) 115. Then, based on the dictionary site determination rule and the tabulation result of the feature items stored in the link feature DB 113, it is determined for each site whether the site is a dictionary site. The site determined to be a dictionary site registers information such as the URL in the dictionary candidate table as a dictionary candidate. The dictionary candidate table is stored in the dictionary candidate DB 117. Subsequently, the dictionary entry candidate creation unit 160 creates dictionary entry candidates for the sites registered as dictionary candidates. In the case of a dictionary site, the relationship between the character string of the anchor text and the URL of the corresponding linked page is the same as the URL registered in the autolink dictionary and the explanation page of the word. Therefore, for the site determined to be a dictionary site, the character string of the anchor text is extracted as the dictionary entry candidate from the URL of the linked page, and dictionary entry candidate information is generated. At this time, dictionary entry candidate information is registered with reference to dictionary addition rule information stored in the dictionary addition rule (storage device) 116. For example, if an exclusion keyword is set in the dictionary addition rule information, processing such as not registering an entry corresponding to this keyword is performed. The generated dictionary entry candidate information is stored in the dictionary candidate DB 117 in association with the corresponding dictionary site.

  The user presenting unit 170 is the document group presenting means 15 and presents information related to the detected dictionary candidate site to the user. Then, the site designated by the user is registered in the dictionary site, information on this dictionary site is set in the dictionary site table, and stored in the dictionary DB 118.

Here, the hardware configuration of the dictionary site detection server will be described. FIG. 3 is a block diagram illustrating a hardware configuration example of the dictionary site detection server.
The entire dictionary site detection server 100 is controlled by a CPU (Central Processing Unit) 101. A random access memory (RAM) 102, a hard disk drive (HDD) 103, and a communication interface 104 are connected to the CPU 101 via a bus 105.

  The RAM 102 temporarily stores at least part of an OS (Operating System) program and application programs to be executed by the CPU 101. The RAM 102 stores various data necessary for processing by the CPU 101. The HDD 103 stores the OS and application programs. The communication interface 104 is connected to the network 300 and transmits / receives data to / from the client device 200 via the network 300.

  With such a hardware configuration, the processing function of the dictionary site detection server 100 can be realized. The instruction to the dictionary site detection server 100 is sent via the network 300 from the input unit 220 of the client device 200. Further, the detection result and the like are transmitted to the client device 200 by the display information generated by the dictionary site detection server 100 and displayed on the display device by the client device 200.

Next, features of the dictionary site that is the object of detection will be described. FIG. 4 is a diagram for explaining the characteristics of the dictionary site.
A typical dictionary site has a hierarchical structure including a top page 500 corresponding to an index or a table of contents, and explanation pages 510, 520, and 530 explaining each term entered in the top page 500. The top page 500 is a page for providing a user with a list 501 of terms that provide the explanation pages 510, 520, and 530 on the dictionary site. Each term set in the top page 500 is linked to a respective explanation page 510, 520, 530. For example, “VPN (Virtual Private Network)” at the top of the term list 501 is linked to an explanation page (file name is vpn.html) 510 explaining the term “VPN”. Similarly, “LAN” is an explanation page explaining the term “LAN” (file name is LAN.html) 520, and “RSS (Rich Site Summary)” is an explanation page explaining the term “RSS” (file name is RSS.html). html) 530.

  The explanation page 510 of “VPN” has a title “What is VPN? IT Glossary: DICDIC” 511 and an explanation sentence explaining “VPN”. In addition, when a term explained in another explanation page appears in the explanation sentence, a link to the explanation page is set for the term. For example, “LAN” appearing in the description sentence is linked to the explanation page 520 of “LAN”.

  The same is true for other commentary pages. The explanation page 520 of “LAN” has a title “What is LAN? IT Glossary: DICDIC” 521 and an explanation sentence. Further, the explanation network “network” 522 is linked to an explanation page of “network” (not shown). The explanation page 530 of “RSS” has a title “What is RSS? IT Glossary: DIDICIC” 531 and an explanation sentence. The commentary “Web site” 532 is linked to a comment page of “Web site” (not shown).

  As described above, the dictionary site has a high percentage of closed links in the site, and (Characteristic 1) the character string (anchor text) specified as the link source and the character string in the title tag of the linked page ( (Characteristic 2) Anchor text and link destination file name are highly consistent, (Characteristic 3) The titles of other pages in the site and character strings other than terms match It can be said that the ratio is high. Therefore, in the case of a dictionary site, when viewed in units of sites (in the example of FIG. 4, the top page 500 and the explanation pages 510, 520, and 530 linked under the top page 500), the above characteristics 1 to 3 There is a high percentage of pages that fall into any of these features. Therefore, the link structure of the pages in the site is analyzed, and the number of links to the site exceeds a certain percentage of the total number of links, and any one or more of the above feature 1, feature 2, and feature 3 is It is possible to determine whether or not this site is a dictionary site by checking whether or not the ratio of the number of links to be satisfied is equal to or greater than a certain value.

A feature detection method will be described with a specific example.
A feature 1 is that the anchor text matches the title in the title tag of the linked page. In HTML, an <a> tag is used for setting a link, and anchor text is enclosed by <a href=“... ”> and <a/>. “...” represents a link destination. For example, assuming that VPN is an anchor text, <a href=“http://.../vpn.html”> VPN <a/> has a character string “VPN” and “http: // ... “//Vpn.html” indicates that it is linked. Here, the title part (character string indicated by <title>) of “http: //... /Vpn.html” and the anchor text “VPN” are collated. If “VPN” is included in the title part, this link is determined to satisfy the feature 1.

  Characteristic 2 is that the anchor text and the linked file name match. As described above, the link destination file name can be extracted from <a href=“... ”>. In the case of the feature 1, the file name “vpn.html” of “http: //... /Vpn.html” is extracted and collated with the anchor text “VPN”. If they match, it is determined that this link satisfies feature 2.

  Characteristic 3 is that the title of the other page in the site matches the character string other than the term. The titles of other pages can be extracted in the same manner as the above feature 2. For example, from <a href=“...”> VPN <a/> the title “What is VPN? IT Glossary: DICDIC”, <a href=“...”> RSS <a/> from title “RSS” It is assumed that “IT Glossary: DICDIC” is extracted. Thereafter, it is compared whether or not the character strings excluding the extracted terms in the title portion match. In this example, “What is IT Glossary: DICDIC” except “VPN” is compared with “What is IT Glossary: DICDIC” except “RSS”. If they match, it is determined that this link satisfies feature 3.

  Note that the dictionary site in the example of FIG. 4 is composed of two layers, a top page 500 and comment pages 510, 520, and 530, but the present invention is not limited to this. For example, even in the case of a hierarchical structure in which a field-specific index page is provided between the top page 500 and the explanation pages 510, 520, and 530, the characteristics of the dictionary site are the same.

Hereinafter, the operation of the dictionary site detection system that detects a dictionary site using the characteristics of such a dictionary site and the procedure of the dictionary site detection process will be described using a specific example.
First, a keyword set by the user is input. The site acquisition unit 120 searches for a page including the keyword and detects the top page. Then, based on the link information of the top page and the link information in the page linked from the top page, all the pages in the range connected by the link from the top page are acquired. In this way, if a word or phrase related to a field that the user wants to create is set, a dictionary site in that field can be detected. For example, if “VPN” is set, a dictionary site for IT terms is detected. If “sub prime loan” is set, a dictionary site for financial terms is detected. A specific process will be described.

FIG. 5 is a diagram showing a flow of processing from when a keyword is input until the page information of the site is acquired.
A keyword 600 is input by the user. In the example of FIG. 5, the keyword 600 is “VPN”. When the keyword 600 is input, the site acquisition unit 120 expands the query based on the extended search rule stored in the extended search rule (storage device) 111 and generates an extended keyword 610. For example, when “to” and “term” are set in the extended word, two types of extended keywords 610 of “what is VPN” 612 and “VPN term” 613 are based on “VPN” of the keyword 600. Is generated. As a result, it is possible to narrow down the search to a result that seems to be an explanation page. Subsequently, a search is performed using the extended keyword 610. A page including a keyword in each of “What is VPN” 612 and “VPN term” 613 is searched. FIG. 5 shows that the search page group 620 of pages 621, 622, and 623 including “What is VPN” is detected for “What is VPN” 612. For each domain, page 621 is http: // dicdic. com, page 622 is http: // abc. com, page 623 is http: // a. co. jp. A search page group is similarly obtained for the other extended keywords 613, but the description thereof is omitted.

  Next, the top page of each page of the search page group 620 is detected and its document data is acquired. As a top page search method, a page having a URL of the domain of the search page group 620 is designated as the top page. Alternatively, an anchor text including “top page” in each page of the search page group 620 is searched, and a link destination page of the anchor text is determined as a top page. FIG. 5 shows the top page 630 of the page 621. Here, the top page 500 shown in FIG. 4 is detected. When the keyword 600 directly designates the URL of the top page, the processing is started from the point where the top page document data is acquired. The top pages are obtained in the same manner for the other pages 622 and 623, but description thereof is omitted here.

  Next, the subordinate explanation page group 640 is collected. As described with reference to FIG. 4, the top page 500 includes anchor text that links to the subordinate explanation page. Therefore, all anchor texts included in the top page 500 and link destination information thereof are extracted, and a crawling process for acquiring an explanation page is performed. As a result, a comment page group 640 corresponding to the anchor text written on the top page 500 is acquired. In the example of FIG. 5, vpn. html, lan. html and the rss. A comment page group 640 including html is acquired.

  Further, when the explanation pages 510, 520, and 530 also include link information to other explanation pages, the explanation pages of the link destinations are also acquired. Thus, the explanation page group 640 includes all pages that can be linked from the top page.

  The above processing is executed for the search page group 620 detected for each extended keyword 610, and the top page 630 and the explanation page group 640 under it are collected for each.

Using the top page 630 of the site thus searched and the explanation page group 640 under it, it is determined whether or not this site has the characteristics of a dictionary site.
First, the link information extraction unit 130 reads out HTML document files of the top page 630 and the explanation page group 640 for each site (common domain), and extracts link information in the page. That is, the HTML document of each read page is analyzed, and the domain of the site, the URL of the processing target page that has been analyzed, the anchor text, the link destination URL, and the title of the link destination page are extracted. Then, the URL-title table and the anchor text-link destination URL table are registered. Since the domain is common to the top page and the explanation page under it, it is not necessary to extract it every time.

FIG. 6 is a diagram showing an example of the URL-title table. The URL-title table is stored in the link feature DB 113.
In the URL-title table 1131, information items of the domain 1131 a, the URL 1131 b, and the title 1131 c are registered.

  The domain 1131a is an identifier for identifying a site, and a domain included in common in the URL of the top page and the explanation page group under the top page is registered. The link information extraction unit 130 extracts a domain from the top page or an arbitrary explanation page and registers it in the domain 1131a.

As the URL 1131b, the URL of the link destination page of the anchor text extracted by analyzing the HTML is registered.
In the title 1131c, the title of this page extracted from the linked page stored in the URL 1131b is registered.

  As described with reference to FIGS. 4 and 5, for example, “VPN” on the top page 500 is anchor text linked to the VPN explanation page 510, and in HTML, <a href = “http: // dicdic. com / vpn.html "> VPN <a/>". From here, “http://dicdic.com/vpn.html” is extracted as a link destination and registered in the URL 1131b. If the URL extracted at this time is described in a relative path, it is converted into an absolute path and then registered in the URL 1131b. Further, the vpn. The html is analyzed, and the title (character string indicated by <title>) is extracted. In the example of FIG. 6, “VPN IT Glossary: DICDIC” is extracted and registered in the title 1131c. Similarly, the URL of the link destination page described in the top page 500 and the title of the link destination page are extracted and registered in the URL-title table 1131. After the process for the top page 500 is completed, the same process is performed for each page of the explanation page group 640. At this time, if the same item has already been registered in the URL-title table 1131, the registration is not performed.

  In this way, the link information extraction unit 130 extracts the link destination URL that appears on the page and the title of the link destination page for each site, and registers them in the URL-title table 1131.

FIG. 7 shows an example of the anchor text-link destination URL table. The anchor text-link destination URL table is stored in the link feature DB 113.
In the anchor text-link destination URL table 1132, information items of the domain 1132a, the processing target URL 1132b, the anchor text 1132c, and the link destination URL 1132d are registered.

The domain 1132a is the same as the domain 1131a in FIG.
In the processing target URL 1132b, the URL of the processing target page subjected to the HTML analysis is registered.

In the anchor text 1132c, the anchor text extracted from the page registered in the processing target URL 1132b is registered.
In the link destination URL 1132d, the URL of the link destination page corresponding to the anchor text 1132c is registered.

  Similar to FIG. 6, for example, when analyzing the top page 500, the URL of the top page 500 (here, http://dicdic.com/index.html) is registered in the processing target URL 1132b. . Then, the anchor text “VPN” to which the explanation page 510 of “VPN” is linked is registered in the anchor text 1132c. Similarly to FIG. 6, the link destination “http://dicdic.com/vpn.html” is extracted and registered in the link destination URL 1132d.

  After the process for the top page 500 is completed, the same process is performed for each page of the explanation page group 640. At this time, if the same item has already been registered in the anchor text-link destination URL table 1132, the registration is not performed.

  In this way, the link information extraction unit 130 extracts the URL of the processing target page from which the anchor text has been extracted, the anchor text, and the URL of the link destination for each site, and registers them in the anchor text-link destination URL table 1132.

  Next, the link feature totaling unit 140 totals the feature items based on the URL-title table 1131 and the anchor text-link destination URL table 1132 set by the link information extracting unit 130. As described above, if the site is a dictionary site, there is a high percentage of links that are closed in the site, and the ratio of the number of links satisfying one or more of the features 1, features 2, and features 3 is as follows. There is more than a certain value. The link feature totaling unit 140 counts the following feature items in order to detect these features. First, in order to detect the ratio of links closed in the site, the total number of links and the number of links closed in the site (referred to as internal links) are totaled. Further, as the feature item of feature 1, the number of links where the link source anchor text matches the character string (title) in the title tag of the link destination page is totaled. As the feature item of feature 2, the number of links whose anchor text matches the file name of the link destination is totaled. Then, as a feature item of feature 3, a title in which the title of the other page in the site matches a character string other than the term is detected, and the number of appearances of the title is totaled.

  FIG. 8 is a diagram illustrating an example of the total information based on the feature 1. (A) is a feature 1 counter table, and (B) is a feature 1 entry table. Both are stored in the link feature DB 113. The total information by the feature 1 is obtained by analyzing the URL-title table 1131 and the anchor text-link destination URL table 1132 and totaling the number of links that match the feature 1.

  (A) The feature 1 counter table 1133 includes information items of a processing target URL (domain) 1133a, an all link counter 1133b, an internal link counter 1133c, and a feature 1 counter 1133d.

  In the processing target URL (domain) 1133a, the URL of the processing target site is registered. The URL read from the domain 1132a of the anchor text-link destination URL table 1132 is registered. The corresponding counter is a value aggregated for each site.

  The total value of link information detected for each site is registered in the all link counter 1133b. The aggregated link information includes information that links a page inside the site to a link destination and information that links a page outside the site. Specifically, the total number of link information registered in the link destination URL 1132d in which the domain 1132a of the anchor text-link destination URL table 1132 matches the domain of the target site is counted.

  In the internal link counter 1133c, of the link information detected for each site, a total value of link information that links to a page inside the site is registered. The number of link information in which the domain of the target site matches the domain 1132a and the domain part of the link destination URL 1132d matches the domain name of the domain 1132a is counted.

  In the feature 1 counter 1133d, a total value of link information in which the anchor text and the title of the page of the link destination URL match is registered for each site. The link destination URL 1132d corresponding to the anchor text read from the anchor text 1132c of the anchor text-link destination URL table 1132 is extracted. Then, a URL that matches the extracted link destination URL is detected from the URL-title table 1131. The title of this URL is extracted from the title 1131c corresponding to the matched URL, and collated with the anchor text first read from the anchor text 1132c. If they match, the feature 1 counter 1133d is incremented. At this time, if a URL that matches the link destination URL 1132d is not detected by the URL 1131b, adjustment is performed by changing the presence or absence of “index.html” included in the URL, the presence or absence of “#”, “?”, And the like. Is detected.

(B) The feature 1 entry table 1134 includes information items of a processing target URL (domain) 1134a, a word 1134b, and a URL 1134c.
Similar to the feature 1 counter table 1133, the URL of the processing target site is registered in the processing target URL (domain) 1134a. An anchor text that satisfies the condition 1 is registered in the word 1134b. In the URL 1134c, a URL that satisfies the condition of feature 1 is registered in association with the anchor text. When the link feature totaling unit 140 collates the anchor text with the title of the link destination URL and determines that they match, the anchor text is stored in the word 1134b and the link destination URL is stored in the URL 1134c. That is, when the feature 1 counter 1133d of the feature 1 counter table 1133 is incremented by 1, the anchor text and the link destination URL determined to satisfy the feature 1 are registered. The domain of the processing target site is registered in the processing target URL (domain) 1134a.

  FIG. 9 is a diagram illustrating an example of the total information based on the feature 2. (C) is a feature 2 counter table, and (D) is a feature 2 entry table. Both are stored in the link feature DB 113.

  (C) The feature 2 counter table 1135 includes information items of a processing target URL (domain) 1135a, an all link counter 1135b, an internal link counter 1135c, and a feature 2 counter 1135d.

  The URL of the processing target site is registered in the processing target URL (domain) 1135a. The total value of link information detected for each site is registered in the all link counter 1135b. In the internal link counter 1135c, of the link information detected for each site, a total value of link information that links to a page inside the site is registered. The processing target URL (domain) 1135a, the all link counter 1135b, and the internal link counter 1135c are the same as the information items having the same name in the feature 1 counter table 1133, and thus detailed description thereof is omitted.

  In the feature 2 counter 1135d, for each site, the total value of the link information in which the anchor text and the file name of the page of the link destination URL match is registered. The anchor text 1132c of the anchor text-link destination URL table 1132 and the corresponding link destination URL 1132d are read. Then, the read anchor text is URL-encoded, and the obtained character string is collated with the link destination file name included in the read link destination URL. If they match, the feature 2 counter 1135d is incremented.

(D) The feature 2 entry table 1136 includes information items of a processing target URL (domain) 1136a, a word 1136b, and a URL 1136c.
Similar to the feature 2 counter table 1135, the URL of the processing target site is registered in the processing target URL (domain) 1136a. An anchor text satisfying the condition 2 is registered in the word 1136b. In the URL 1136c, a URL satisfying the condition of feature 1 is registered in association with the anchor text. The processing is the same as in the case of the feature 1 except that the condition to be satisfied is not the feature 1 but the feature 2. That is, when the feature 2 counter 1135d of the feature 2 counter table 1135 is incremented by 1, the anchor text determined to satisfy the feature 2 and the link destination URL are registered.

  FIG. 10 is a diagram illustrating an example of the total information according to the feature 3. (E) is a feature 3 counter table, (F) is a common title table, and (G) is a feature 3 entry table. All are stored in the link feature DB 113.

  (E) The feature 3 counter table 1137 includes information items of a processing target URL (domain) 1137a, an all link counter 1137b, an internal link counter 1137c, a feature 3 counter 1137d, and a common title 1137e.

  The URL of the processing target site is registered in the processing target URL (domain) 1137a. The total value of link information detected for each site is registered in the all link counter 1137b. In the internal link counter 1137c, of the link information detected for each site, a total value of link information that links to a page inside the site is registered. Since the processing target URL (domain) 1137a, the all link counter 1137b, and the internal link counter 1137c are the same as the information items having the same name in the feature 1 counter table 1133, detailed description thereof is omitted.

  In the feature 3 counter 1137d, for each site, a total value of link information in which the title of the link destination URL page excluding the anchor text is common is registered. In addition, the title is registered in the common title 1137. The link feature totaling unit 140 analyzes the URL-title table 1131 and the anchor text-link destination URL table 1132 to create a common title table. Therefore, among the common titles registered in the common title table 1138, the one with the largest number of link information is registered. The common title is registered in the common title 1137e.

(F) The common title table 1138 includes information items of a processing target URL (domain) 1138a, a common title 1138b, and a counter 1138c.
The URL of the processing target site is registered in the processing target URL (domain) 1138a. The common title of the extracted link destination URL is registered in the common title 1138b. In the counter 1138c, the total value of the link information in which the common title appears is registered.

  The URL that matches the link destination URL 1132d corresponding to the anchor text 1132c of the anchor text-link destination URL table 1132 is detected from the URL-title table 1131. The title 1131c corresponding to the URL 1131b of the URL-title table 1131 is extracted with the URL corresponding to the matched URL 1131b, and the character string excluding the anchor text 1132c read first is extracted. If this character string is registered in the common title 1138b of the common title table, the corresponding counter 1138c is incremented. If not registered in the common title 1138b, a new record is added to the common title table 1138 and registered.

(G) The feature 3 entry table 1139 includes information items of a processing target URL (domain) 1139a, a word 1139b, a URL 1139c, and a common title 1139d.
Similar to the feature 3 counter table 1137, the URL of the processing target site is registered in the processing target URL (domain) 1139a. An anchor text that satisfies the condition 3 is registered in the word 1139b. In the URL 1139c, a URL satisfying the condition of feature 3 is registered in association with the anchor text. The corresponding title is registered in the common title 1139d.

  In this way, the link feature totaling unit 140 analyzes the URL-title table 1131 and the anchor text-link destination URL table 1132 in which the extracted link information is registered. Then, together with the total link counter and the internal link counter, the feature 1 counter, the feature 2 counter, and the feature 3 counter, which are the number of links that satisfy the features of the dictionary site, are totaled, and an entry candidate table that satisfies the respective requirements is created. For feature 1, a feature 1 counter table 1133 and a feature 1 entry table 1134 are generated. For feature 2, a feature 2 counter table 1135 and a feature 2 entry table 1136 are generated. For feature 3, a feature 3 counter table 1137 and a feature 3 entry table 1139 are generated. Note that aggregation is performed only for features designated in advance.

  The dictionary site determination unit 150 uses the aggregated feature 1 counter table 1133, feature 2 counter table 1135, and feature 3 counter table 1137, and based on the dictionary site determination rule, the processing target URL (domain) is a dictionary site. Determine whether or not. The dictionary site determination rule is read out from the dictionary site determination rule (storage device) 115 and applied.

  For example, a description will be given of a case where the dictionary site determination rule is “the ratio of the number of links satisfying feature 1 is 90% or more among the total links, and 90% or more in-site links”. In this case, the ratio of intra-site links and the ratio of links satisfying the characteristic (1) are calculated with reference to the feature 1 counter table 1133.

Since the ratio of intra-site links is the ratio of internal links to all links, it can be calculated by the number of internal links / total number of links. For example, when the processing target URL is “http: // dicdic / com /”, the total link counter 1133b is “110” and the internal link counter 1133c is “101”.
R = 101/110 (= 0.9181)
And can be calculated. Further, the ratio R1 of the number of links satisfying the feature 1 is
R1 = 100/101 (= 0.99)
And calculated. When the determination rule is the above, the value of B is a dictionary-like score.

  Furthermore, a combination of features can be used as a determination rule. For example, “Of the total number of links, 90% or more of the links in the site and 90% or more of the links satisfying feature 1 or 90% or more of the links satisfying feature 2 and feature 3 A case where the ratio of the number of links to be satisfied is 90% or more will be described.

  In this case, in the same manner as described above, the values of all link counters and internal link counters of the corresponding URL are extracted from the feature 1 counter table 1133, the feature 2 counter table 1135, or the feature 3 counter table 1137, and the ratio of the internal links (R) is calculated. Further, the ratio (R1) of the number of links satisfying the feature 1 is calculated from the feature 1 counter 1133d and the internal link counter 1133c of the feature 1 counter table 1133. The ratio (R2) of the number of links satisfying the feature 2 is calculated from the feature 2 counter 1135d and the internal link counter 1135c of the feature 2 counter table 1135. Then, the ratio (R3) of the number of links satisfying the feature 3 is calculated from the feature 3 counter 1137d and the internal link counter 1137c of the feature 3 counter table 1137. Then, it is determined whether or not the rule is established based on the calculated R, R1, R2, and R3. If the rule is established, this site can be determined as a dictionary site. Sites determined to be dictionary sites are registered in the dictionary candidate table.

Alternatively, the score may be calculated by weighting the ratio of the number of links for each feature. As the weighting coefficient α of feature 1, the weighting coefficient β of feature 2, and the weighting coefficient γ of feature 3, the score S is
S = αR1 + βR2 + γR3 (1)
Can be calculated.

  FIG. 11 is a diagram showing an example of a dictionary candidate table and its entry candidate table. (H) is an example of a dictionary candidate table, and (I) is an example of an entry candidate table. Both are stored in the dictionary candidate DB 117.

(H) The dictionary candidate table 1171 includes information items of a site URL (domain) 1171a and a score 1171b.
In the site URL (domain) 1171a, the URL of the target site determined to be a dictionary candidate is registered.

The score 1171b stores a dictionary-like score calculated by the equation (1).
(I) The entry candidate table 1172 includes information items of a processing target URL (domain) 1172a, a word 1172b, and a URL 1172c.

  In the processing target URL (domain) 1172a, the URL of the target site determined to be a dictionary candidate is registered. In the word 1172b, a word registered in any of the feature 1 entry table 1134, the feature 2 entry table 1136, and the feature 3 entry table 1139 created for this site is set. Similarly, any URL is set as the URL.

  When registering dictionary candidates and entry candidates, processing is performed based on the dictionary addition rules stored in the dictionary addition rule (storage device) 116. For example, if the URL determined as a dictionary candidate is the same as the URL prohibited from being registered by the dictionary addition rule, the registration to the dictionary candidate is not performed. In addition, in the case of entry candidate registration, if an exclusion keyword or the like is set, anchor text that matches the exclusion keyword is not registered in the entry candidate table 1172.

After the dictionary candidate is determined in this way, if there is a display request from the user, the user presenting unit 170 displays it on the display unit of the client device 200.
FIG. 12 is a diagram showing an example of a dictionary site candidate list screen.

  The dictionary site candidate list screen 2100 is displayed on a display device connected to the client device 200. The dictionary site candidate list screen 2100 displays a dictionary site candidate list table 2101 showing candidate dictionary sites, a detailed entry selection button 2102, an NG site registration button 2103, a dictionary site registration button 2104, and the like.

  The dictionary site candidate list screen 2100 displays a check column 2101a, an identification number 2101b, a URL 2101c, an example entry 2101d, and a score 2101e based on the dictionary candidate table 1171 and the entry candidate table 1172.

  The check column 2101a is a column for selecting a site to be registered as a dictionary site or a site to be registered as an NG site. Checked sites are processed. The identification number 2101b is a number assigned to the dictionary site candidates in order. The URL 2101c is a URL of the site, and is displayed based on the site URL 1171a of the dictionary candidate table 1171. The entry example 2101d is an example of an entry that can be referred to by this dictionary site candidate. An arbitrary word / phrase is selected from the URL 1172c of the corresponding site in the entry candidate table 1172, and the URL is displayed. Here, the anchor text may be selected from the corresponding word 1172b and displayed. In the score 2101e, the dictionary-like score of this site is extracted from the score 1171b of the dictionary candidate table 1171 and displayed.

  A detailed entry selection button 2102 is prepared for each site. For example, no. 1 is operated, the detailed entry selection screen 2110 is opened. An entry candidate list 2111 corresponding to one dictionary site candidate is displayed. In the entry candidate list 2111, registrations of the entry candidate table 1172 of the selected dictionary site candidate are extracted and displayed. The check column 2111a is a column for selecting whether or not to register this entry. The identification number 2111b is a number assigned to each entry candidate in order. In the URL 2101c, the anchor text registered in the word 1172b of the entry candidate table 1172 of the selected dictionary site candidate is displayed with the link information of the URL 1172c added.

  The user displays a dictionary site candidate list screen 2100 and selects a dictionary site. At this time, if necessary, a detailed entry selection screen 2110 can be displayed to confirm the contents of the entry candidate. At this time, whether or not to be an entry candidate can also be selected. If there is a candidate to be registered in the dictionary site, the check column 2101a is checked, and the dictionary site registration button 2104 described as “Register in dictionary site” is operated. As a result, the selected dictionary site candidate and entry candidate are registered in the dictionary site table as a dictionary and stored in the dictionary DB 118. If it is not desired to register as a dictionary site, the NG site registration button 2103 described as “Register in NG site” is operated. As a result, this dictionary site is deleted from the dictionary site candidates. At this time, the site may be registered as an NG site in the dictionary addition rule and stored in the dictionary addition rule (storage device) 116 so that this site will not be a dictionary site candidate in the future.

  In this way, when a user sets an example word as a keyword, a dictionary site of that genre is automatically detected, and at the same time, a dictionary entry (a pair of a term and an explanation page URL) is extracted. Since the user only has to confirm this on the dictionary site candidate list screen 2100 and decide whether or not to use it as a dictionary, the dictionary can be easily added. Also, if it is executed regularly, dictionary maintenance becomes easy.

Hereinafter, a processing procedure of the dictionary site detection method in the dictionary site detection system will be described with reference to a flowchart.
FIG. 13 is a flowchart showing the entire processing procedure of the dictionary site detection method. This flowchart shows a processing procedure from input of a keyword to presentation of dictionary site candidates. The keyword set by the user is input and the process is started.

  [Step S01] Based on the input keyword, the site acquisition unit 120 searches for a site having a page including the keyword. A site acquisition process is performed for acquiring the top page and other pages linked to the top page for the searched site. The acquired page group is stored in the acquisition site (storage device) 112 for each site. Details of the site acquisition process will be described later.

  [Step S02] The link information extraction unit 130 selects one candidate site from the sites stored in the acquisition site (storage device) 112 by the site acquisition process in step S01, and analyzes the page group. Then, link information is extracted from the page and registered in the URL-title table 1131 and the anchor text-link destination URL table 1132. Details of the link information extraction processing will be described later.

  [Step S03] The link feature totaling unit 140 analyzes the URL-title table 1131 and the anchor text-link destination URL table 1132. Then, the number of feature items indicating features unique to the dictionary site is added to the link information. Then, total information and an entry table are generated for each feature. Details of the link feature aggregation processing will be described later.

  [Step S04] The dictionary site determination unit 150 calculates a dictionary-likeness score from the aggregate information based on the dictionary site determination rule stored in the dictionary site determination rule 115 (storage device). A site having a dictionary-likeness score of a predetermined value or more is determined as a dictionary site candidate. Details of the dictionary site determination process will be described later.

  [Step S05] It is checked whether or not this site is determined to be a dictionary site by the dictionary site determination processing in step S04. If it is determined to be a dictionary site, the process proceeds to step S06. If it is not determined to be a dictionary site, the process proceeds to step S07.

  [Step S06] When the dictionary site is determined, dictionary entry candidates are created based on the entry table extracted halfway. Details of the dictionary entry candidate creation processing will be described later.

  [Step S07] It is determined whether or not the processing of all candidate sites detected by the keyword is completed. When the processes for all candidate sites are completed, the process proceeds to step S08. If all candidate sites have not been processed, the process returns to step S02 to process the next candidate site.

  [Step S08] When processing of all candidate sites is completed, a process of presenting sites registered as dictionary site candidates and their entries to the user is performed. Details of the user presentation process will be described later.

  By executing the above processing procedure, a site that may be a dictionary site of a desired genre is detected based on the input keyword, and the site is found in the site based on link information between pages. It is determined whether or not it has the characteristics as follows. And when it determines with a dictionary site candidate, the site is shown to a user. At the same time, dictionary entry candidates necessary for creating an autolink dictionary are also generated from the link information. As a result, the user can easily detect a dictionary site of a desired genre simply by setting a keyword related to the desired genre. In addition, dictionary entries (information in which words are associated with URLs of their explanation pages) can be obtained at the same time, making it easy to create an autolink dictionary.

Details of each process will be described below.
FIG. 14 is a flowchart showing a procedure of site acquisition processing.
A keyword is input and processing starts.

[Step S11] The extended search rule stored in the extended search rule (storage device) 111 is read, the keyword is changed based on the extended search rule, and the query is extended.
[Step S12] A search is performed using the query created in Step S11. Thereby, a page including the keyword is detected. In the search, a plurality of keywords may be accepted and AND of the respective search results may be taken.

  [Step S13] One of the pages searched in step S12 is selected, and the top page is detected. The detected top page is stored in the acquisition site (storage device) 112.

[Step S14] Link information described in the top page detected in step S13 is extracted.
[Step S15] Based on the link information detected in step S14, a subordinate comment page to be linked is acquired. The acquired explanation page is stored in the acquisition site (storage device) 112.

[Step S16] Link information described in the comment page acquired in step S15 is extracted.
[Step S17] Based on the link information extracted in Step 14 and Step S16, it is determined whether or not all the explanation pages traced by links from the top page have been collected. If collected, the process proceeds to step S17. If not all have been collected, the process returns to step S15 to collect the next commentary page.

  [Step S18] It is determined whether or not the processing has been completed for all search pages searched in step S12. If not completed, the process returns to step S13 to perform processing for acquiring the next site of the search result. When finished, the site acquisition process is finished.

  By executing the above processing procedure, the top page of the site to which the page searched based on the keyword belongs and the subordinate pages linked to the top page are collected and stored in the acquisition site (storage device) 112. The

FIG. 15 is a flowchart showing the procedure of link information extraction processing.
One site stored in the acquisition site (storage device) 112 by the site acquisition process (step S01) is set as a candidate site, the page group (top page and comment page group) is read, and the process is started.

  [Step S21] The domain name of the candidate site is extracted. For example, the domain name is extracted from the URL of the top page. The extracted domain name is stored in the domain 1131a of the URL-title table 1131 and the domain 1132a of the anchor text-link destination URL table 1132 when other information items are registered.

[Step S22] One unprocessed page is extracted from the page group of the candidate site. The pages to be processed include both the top page and the explanation page.
[Step S23] The HTML of the extracted page is analyzed. Further, the URL of the extracted page is stored in the processing target URL 1132 b of the anchor text-link destination URL table 1132.

[Step S24] The link information A in the page is extracted and registered in the URL-title table 1131. Details of the processing will be described later.
[Step S25] The link information B in the page is extracted and registered in the anchor text-link destination URL table 1132. Details of the processing will be described later.

  [Step S26] It is determined whether or not the processing of all pages belonging to the candidate site has been completed. If not completed, the process returns to step S22 to process the next page. When finished, the link information extraction process is finished.

FIG. 16 is a flowchart showing the procedure of the link information A extraction process.
The link information A (the URL of the linked page and the title of the page) is extracted and registered in the URL-title table 1131.

[Step S241] One unprocessed link information is extracted from the page to be processed.
[Step S242] The designation information of the link destination URL is extracted, and if the URL is a relative path, the URL of the link destination page is converted into an absolute path from the file position of the processing target page. Since the link destination URL may be described as a relative path from the page to be processed, such a relative path is converted into an absolute path.

[Step S243] The page of the link destination URL extracted in Step S242 is acquired, the HTML is analyzed, and the title is extracted.
[Step S244] The URL of the link destination page converted into the absolute path in step S242 is registered in the URL 1131b of the URL-title table 1131. In addition, the title of the linked page extracted in step S243 is registered in the title 1131c in association with the previously registered URL. The domain name extracted in the previous stage is also registered in the domain 1131a. The URL-title table 1131 is stored in the link feature DB 113.

  [Step S245] It is determined whether it is the last link information of the page to be processed. If it is not the last link information, the process returns to step S241 to process the next link information. If it is the last link information, the link information A extraction process is terminated.

  By executing the above processing procedure, the link destination URL described in the link information of the processing target page and the title of the link destination page are registered in the URL-title table 1131.

FIG. 17 is a flowchart showing the procedure of the link information B extraction process.
The link information B (the URL of the processing target page, the anchor text, and the URL of the link destination page) is extracted and registered in the anchor text-link destination URL table 1132.

[Step S251] One piece of unprocessed link information is extracted from the page to be processed.
[Step S252] An anchor text and a link destination URL are extracted from the link information.

  [Step S253] It is determined whether the link destination URL extracted in step S252 is a relative path. If it is a relative path, the process proceeds to step S254. If it is not a relative path, the process proceeds to step S255.

[Step S254] When the link destination URL is a relative path, the URL of the link destination page is converted into an absolute path from the file position of the page to be processed.
[Step S255] The anchor text extracted in step S252 is registered in the anchor text 1132c of the anchor text-link destination URL 1132. The link destination URL (absolute path) is also registered in the link destination URL 1132d in association with the anchor text. The domain name extracted in the previous stage is registered in the domain 1132a, and the URL of the processing target page is registered in the processing target URL 1132b. The anchor text-link destination URL 1132 is stored in the link feature DB 113.

  [Step S256] It is determined whether it is the last link information of the page to be processed. If it is not the last link information, the process returns to step S251 to process the next link information. If it is the last link information, the link information (feature 2) extraction process is terminated.

  By executing the above processing procedure, the anchor text described in the link information of the processing target page and the link destination URL are registered in the anchor text-link destination table 1132.

FIG. 18 is a flowchart showing the procedure of the link feature totaling process.
Using the URL-title table 1131 and the anchor text-link destination URL table 1132, data for each feature is aggregated.

  [Step S31] A link feature rule is read from the link feature rule (storage device) 114. It is determined whether or not to analyze feature 1 based on the link feature rule. When analyzing, the process proceeds to step S32. When the analysis is not performed, the process proceeds to step S33.

[Step S32] When it is determined that the feature 1 is to be analyzed, the analysis according to the feature 1 is performed, and the number of links satisfying the feature is tabulated. Details of the feature 1 analysis process will be described later.
[Step S33] Based on the link feature rule, it is determined whether or not to analyze feature 2. When analyzing, the process proceeds to step S34. When the analysis is not performed, the process proceeds to step S35.

[Step S34] When it is determined that the analysis of the feature 2 is performed, the analysis according to the feature 2 is performed, and the number of links satisfying the feature is totaled. Details of the feature 2 analysis processing will be described later.
[Step S35] Based on the link feature rule, it is determined whether to analyze feature 3. When analyzing, the process proceeds to step S36. When the analysis is not performed, the link feature totaling process is terminated.

  [Step S36] When it is determined that the analysis of the feature 3 is performed, the analysis according to the feature 3 is performed and the number of links satisfying the feature is totaled. Details of the feature 3 analysis process will be described later. After the feature 3 analysis process is completed, the link feature aggregation process is terminated.

By executing the above processing procedure, the dictionary site can be determined using any one of the features 1, 2, and 3.
FIG. 19 is a flowchart showing the procedure of the feature 1 analysis process.

  In the feature 1 analysis processing, based on feature 1, the number of links where the link source anchor text matches the character string (title) in the title tag of the linked page is counted as a feature 1 counter.

  [Step S321] One line is read from the anchor text-link destination URL table 1132. The domain, processing target URL, anchor text, and link destination URL are read out.

  [Step S322] All link counters 1133b of the feature 1 counter table 1133 corresponding to the domain read in step S321 are incremented by 1 and stored. When the domain corresponding to the domain 1133a of the feature 1 counter table 1133 is not set, a new record is created and 1 is set to the corresponding all link counter 1133b.

  [Step S323] It is determined whether the link destination URL read in step S321 is a link in the own site. If so, the process advances to step S324. If the link is not within the local site, the process proceeds to step S329.

[Step S324] The internal link counter 1133c of the feature 1 counter table 1133 corresponding to the domain read in step S321 is incremented by 1 and stored.
[Step S325] The URL 1131b of the URL-title table 1131 is searched, and a line in which the same URL as the link destination URL read in step S321 is registered is detected. Then, a title is extracted from the title 1131c corresponding to the detected URL.

  [Step S326] It is determined whether the anchor text read in step S321 is included in the title extracted in step S325. If the anchor text is included in the title, the process proceeds to step S327. If not included, the process proceeds to step S329.

  [Step S327] When the anchor text is included in the title, the feature 1 counter 1133d of the corresponding domain row in the feature 1 counter table 1133 is incremented by 1 and stored.

  [Step S328] The link information in which the anchor text is included in the title is entered in the feature 1 entry table 1134. The URL of the processing target page is registered in the processing target URL (domain) 1134a, the anchor text is registered in the word 1134b, and the link destination URL is registered in the URL 1134c.

  [Step S329] It is determined whether or not an unprocessed URL remains in the processing target URL 1132b of the anchor text-link destination URL table 1132. If it remains, the process returns to step S321, and the next process target URL is processed. If not, the feature 1 analysis process is terminated.

  By executing the above processing procedure, a total link counter, an internal link counter, and a feature 1 counter in which the number of links satisfying the feature 1 are totaled are obtained. The aggregation result is registered in the feature 1 counter table 1133 in association with the domain of the processing target site. At the same time, the feature 1 entry table 1134 in which the anchor text satisfying the requirement of the feature 1 is associated with the link destination URL is also generated.

FIG. 20 is a flowchart showing the procedure of the feature 2 analysis process.
In the feature 2 analysis process, based on the feature 2, the number of links whose link source anchor text matches the file name of the link destination page is counted as a feature 2 counter.

  [Step S341] One line is read from the anchor text-link destination URL table 1132. The domain, processing target URL, anchor text, and link destination URL are read out.

  [Step S342] The link counter 1135b of the feature 2 counter table 1135 corresponding to the domain read in step S341 is incremented by 1 and stored. When a domain corresponding to the domain 1135a of the feature 2 counter table 1135 is not set, a new record is created and 1 is set to the corresponding all link counter 1135b.

  [Step S343] It is determined whether the link destination URL read in step S341 is a link in the own site. If so, the process advances to step S344. If it is not a link in its own site, the process proceeds to step S346.

[Step S344] The internal link counter 1135c of the feature 2 counter table 1135 corresponding to the domain read in step S341 is incremented by 1 and stored.
[Step S345] The anchor text read in step S341 is URL-encoded. In general, symbols, double-byte characters, and the like that cannot be used as URLs are URL-encoded when they are incorporated into URLs, and converted to combinations of single-byte characters such as “% E3”. For this reason, when the file name included in the link destination URL is compared with the anchor text, the anchor text needs to be URL-encoded.

  [Step S346] The anchor text subjected to the URL encoding process in step S345 is collated with the file name included in the link destination URL read in step S341. If the anchor text matches the file name, the process proceeds to step S347. If not, the process proceeds to step S349.

  [Step S347] When the anchor text matches the file name, the feature 2 counter 1135d of the corresponding domain row in the feature 2 counter table 1135 is incremented by 1 and stored.

  [Step S348] The link information in which the anchor text is included in the title is entered in the feature 2 entry table 1135. The URL of the processing target page is registered in the processing target URL (domain) 1135a, the anchor text is registered in the word 1135b, and the link destination URL is registered in the URL 1135c.

  [Step S349] It is determined whether or not an unprocessed URL remains in the processing target URL 1132b of the anchor text-link destination URL table 1132. If it remains, the process returns to step S341, and the next URL to be processed is processed. If not, the feature 2 analysis process is terminated.

  By executing the above processing procedure, a feature 2 counter in which the total link counter, the internal link counter, and the number of links satisfying the feature 2 are totaled is obtained. The aggregation result is registered in the feature 2 counter table 1135 in association with the domain of the processing target site. Note that when the analysis process of feature 1 is performed at the same time, the process of counting all link counters and internal link counters may be performed on either side. At the same time, a feature 2 entry table 1136 in which the anchor text satisfying the requirement of feature 2 and the link destination URL are associated is also generated.

FIG. 21 is a flowchart showing the procedure of the feature 3 analysis process.
In the feature 3 analysis process, a character string obtained by removing the anchor text from the title of the link destination page is extracted as a common title based on the feature 3, and the number of appearances of the common title is totaled.

  [Step S361] One line is read from the anchor text-link destination URL table 1132. The domain, processing target URL, anchor text, and link destination URL are read out.

  [Step S362] The link counter 1137b of the feature 3 counter table 1137 corresponding to the domain read in step S361 is incremented by 1 and stored. When the domain corresponding to the domain 1137a of the feature 3 counter table 1137 is not set, a new record is created and 1 is set to the corresponding all link counter 1137b.

  [Step S363] It is determined whether the link destination URL read in step S361 is a link in the own site. If so, the process advances to step S364. If it is not a link in its own site, the process proceeds to step S369.

[Step S364] The internal link counter 1137c of the feature 3 counter table 1137 corresponding to the domain read in step S361 is incremented by 1 and stored.
[Step S365] The URL 1131b of the URL-title table 1131 is searched, and a line in which the same URL as the link destination URL read in step S361 is registered is detected. Then, the title corresponding to the detected URL is extracted from the title 1131c.

  [Step S366] It is determined whether or not the anchor text read out in step S361 is included in the title extracted in step S365. If it is included, the process proceeds to step S367. If not included, the process proceeds to step S369.

  [Step S367] When the anchor text is included in the title, a character string excluding the anchor text from the title is extracted and set as a common title. The extracted common title and the common title 1138b of the common title table 1138 are collated, and if there is a match, the corresponding counter 1138c is incremented by 1 and stored. If there is no match, a new record is registered in the common title 1138b, and 1 is set in the corresponding counter 1138c.

  [Step S368] The link information in which the common title is registered in Step S367 is entered in the feature 3 entry table 1139. The URL of the processing target page is registered in the processing target URL (domain) 1139a, the anchor text is registered in the word 1139b, the link destination URL is registered in the URL 1139c, and the common title is registered in the common title 1139d.

  [Step S369] It is determined whether or not an unprocessed URL remains in the processing target URL 1132b of the anchor text-link destination URL table 1132. If it remains, the process returns to step S361 to perform the process for the next processing target URL. If not, the process proceeds to step S370.

  [Step S370] The count values of the counter 1138c registered in the common title table 1138 by the processing up to step S369 are compared, and the maximum count value and the common title are extracted. Then, the maximum count value extracted in the feature 3 counter 1137d of the feature 3 counter table 1137 and the common title corresponding to the maximum count value are registered in the common title 1137e.

  [Step S371] The common title registered in step S370 is sequentially checked against the common title 1139 in the feature 3 entry table 1139. The record is deleted except for those that have a common title. As a result, only the entries having the same common title on the maximum number of pages remain in the feature 3 entry table 1139.

  By executing the above processing procedure, a feature 3 counter in which the total link counter, the internal link counter, and the number of links having a common title satisfying the feature 3 are totaled is obtained. The aggregation result is registered in the feature 3 counter table 1137 in association with the domain of the processing target site. In addition, when performing the analysis process of the feature 1 or the feature 2 at the same time, the process of counting the all link counter and the internal link counter may be performed on either one. At the same time, a feature 3 entry table 1139 in which the anchor text satisfying the requirement of the feature 3 is associated with the link destination URL is also generated.

When the link feature totaling process is thus completed, the dictionary site determination unit 150 performs a determination process as to whether or not the detected site is a dictionary site.
FIG. 22 is a flowchart showing the procedure of the dictionary site determination process.

When dictionary site processing is started, it is assumed that the domain name being processed is inherited from the previous processing.
[Step S41] The dictionary site determination unit 150 reads the dictionary site determination rule from the dictionary site determination rule (storage device) 115.

  [Step S42] The dictionary site determination unit 150 calculates a dictionary-likeness score based on the dictionary site determination rule read in step S41. At this time, the aggregation result of the corresponding site registered in the corresponding feature 1 counter table 1133, the feature 2 counter table 1135, or the feature 3 counter table 1137 is used.

  [Step S43] The dictionary-likeness score calculated in step S42 is compared with a threshold defined in the dictionary site determination rule. If the dictionary site determination rule is satisfied, it can be determined as a dictionary site candidate.

  [Step S44] If the dictionary site candidate is determined based on the comparison result in step S43, the process proceeds to step S45. If it is not determined as a dictionary site candidate, the dictionary site determination process ends.

  [Step S45] If it is determined as a dictionary site candidate, this site is set as a dictionary candidate and registered in the dictionary candidate table 1171. The processing target URL (domain) is registered in the site URL (domain) 1171a of the dictionary candidate table 1171. Also, the calculated dictionary-likeness score is registered in the score 1171b. After registration, the dictionary site determination process ends.

When the dictionary site candidate is determined, the dictionary entry candidate creation unit 160 performs a process of creating a dictionary entry candidate for the dictionary site candidate.
FIG. 23 is a flowchart showing a procedure of dictionary entry candidate creation processing.

  [Step S61] An entry is read from the entry tables (feature 1 entry table 1134, feature 2 entry table 1136, and feature 3 entry table 1139) stored in the link feature DB 113. One entry of a pair of the word 1134b, 1136b, 1139b and the URL 1134c, 1136c, 1139c of the corresponding processing target URL (domain) is extracted.

  [Step S62] It is determined whether the entry extracted in step S61 is registered in the corresponding processing target URL of the entry candidate table 1172 of the dictionary candidate DB 117. If not registered, the process proceeds to step S63. If not registered, the process proceeds to step S64.

[Step S63] If the entry is not registered in the entry candidate table 1172, the entry is registered in the entry candidate table 1172 as a new record.
[Step S64] It is determined whether or not there are still entries. If there is a next entry, the process returns to step S61 to perform the process from the entry extraction. If there is no next entry, the dictionary entry candidate creation process is terminated.

  With the above processing procedure, an entry candidate table of dictionary site candidates is created. The entry candidate registration rule may be determined in advance and stored in the dictionary addition rule (storage device) 116. For example, rules such as exclusion keywords, URL designation, and registration of only words detected at a plurality of sites are set. In the dictionary entry candidate creation process, only entries that satisfy the conditions defined in the rule are registered in the entry candidate table 1172.

When the dictionary site candidate is determined in this way and the dictionary entry candidate is created, the user presenting unit 170 performs a process of presenting the dictionary site candidate and the dictionary entry candidate to the user.
FIG. 24 is a flowchart showing the procedure of the user presentation process.

  [Step S <b> 81] Information on dictionary site candidates set in the dictionary candidate table 1171 and the entry candidate table 1172 stored in the dictionary candidate DB 117 is output to the client device 200. The client device 200 displays the dictionary site candidate list screen 2100 shown in FIG. 12 based on the acquired information.

[Step S82] Wait until an instruction from the user is received via the client device 200. If accepted, the process proceeds to step S83.
[Step S83] It is determined whether the instruction received in step S82 is a registration request. If it is a registration request, the process proceeds to step S84. If it is not a registration request, the process proceeds to step S86.

[Step S84] If it is a registration request, this dictionary site candidate is registered in the dictionary and stored in the dictionary DB 118.
[Step S85] An acceptance confirmation screen is output to the client device 200, and the user presentation process is terminated.

[Step S86] If it is not a registration request, the requested process is executed, and the user presentation process is terminated.
By executing the above processing procedure, a dictionary site of the genre is automatically detected based on a word input by the user as an example, and a dictionary entry (a pair of a term and an explanation page URL) is extracted at the same time. The As a result, the user's dictionary creation work can be greatly reduced. Also, if it is executed regularly, dictionary maintenance becomes easy.

  The above processing functions can be realized by a computer. In that case, a program describing the processing contents of the functions that the document group detection apparatus should have is provided. By executing the program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium.

  When distributing the program, for example, portable recording media such as a DVD (Digital Versatile Disc) and a CD-ROM (Compact Disc Read Only Memory) on which the program is recorded are sold. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. Further, each time the program is transferred from the server computer, the computer can sequentially execute processing according to the received program.

Regarding the above embodiment, the following additional notes are disclosed.
(Supplementary Note 1) In a document group detection method for detecting a predetermined document group which is a set of documents provided on a network and managed by one or more computers,
The computer is
For a group of documents having a hierarchical structure in which a plurality of subordinate documents exist under the specific document, the subordinate document is searched using a specific keyword, and the specific is determined based on the searched subordinate document. A collection procedure for detecting a document and collecting a plurality of subordinate documents under the specific document;
For each of the collected specific document and the plurality of subordinate documents in the collected document group, connection information indicating a relation with a specific other document added to an arbitrary character string in the subordinate document is extracted, and the subordinate document and A feature counting procedure that counts the number of states that have a specific relationship with the associated linked document,
Read out the feature rule from the feature rule storage means in which the feature rule that is a condition using the specific relationship is stored, determine whether the number of states of the specific relationship of the document group satisfies the condition of the feature rule, A document group determination procedure for registering a document group satisfying the condition as a target document group candidate;
A procedure for outputting the document group registered in the target document group candidate;
A document group detection method characterized by comprising:

  (Supplementary Note 2) When a keyword characterizing a document belonging to the detection target document group or the document group is acquired, the collection procedure generates an extended keyword by adding a preset extended word to the keyword, The document group detection method according to appendix 1, wherein a search is performed using the extended keyword in addition to the keyword.

(Supplementary Note 3) The document group to be detected is a document group related to commentary information on a plurality of terms in an arbitrary field, the specific document describes a list of terms to be explained, and the subordinate documents are A dictionary document group that is a glossary document in which explanations of terms are described,
Further, the connection information is added to a character string representing the term appearing in the specific document and the glossary document, and associates the character string with the glossary document corresponding to the character string,
The specific relationship in the feature counting procedure is a specific relationship between the character string associated by the connection information and the connection destination document.
The document group detection method according to supplementary note 1, wherein:

(Supplementary Note 4) In the feature aggregation procedure, the character string to which the link information is added and the link destination document corresponding to the character string are analyzed, and the link information belongs to the same document group. A specification in which the number of internally linked information associated with a glossary document matches the character string included in the title of the linked document and the character string to which the linked information is added as the specific relationship. The number of internally consolidated information and
In the document group determination procedure, a ratio of the specific internal connection information to the entire internal connection information is calculated as a dictionary-like score, and the dictionary-like score is compared with a threshold value based on the feature rule. Determine whether the conditions of the dictionary document group are satisfied,
The document group detection method according to attachment 3, wherein the document group detection method is a procedure.

(Supplementary Note 5) In the feature aggregation procedure, the character string to which the link information is added and the link destination document corresponding to the character string are analyzed, and the link information belongs to the same document group. The number of internal link information associated with the glossary document matches the character string added to the link information and the character string included in the file name of the link destination document as the specific relationship. The number of specific internal consolidated information and
In the document group determination procedure, a ratio of the specific internal connection information to the entire internal connection information is calculated as a dictionary-like score, and the dictionary-like score is compared with a threshold value based on the feature rule. Determine whether the conditions of the dictionary document group are satisfied,
The document group detection method according to attachment 3, wherein the document group detection method is a procedure.

(Additional remark 6) In the said feature totalization procedure, the said character string to which the said connection information was added and the said connection destination document corresponding to the said character string are analyzed, and the said belonging to the same said document group by the said connection information As the state of the specific relationship with the number of internal connection information associated with the glossary document, the connection information relating the connection destination document from the title of the connection destination document is added to the connection destination document. The common title part excluding the same part as the character string is extracted and compared, and the number of specific linked information that matches the common title part is totaled,
In the document group determination procedure, based on the feature rule, a ratio of the specific link information to the entire internal link information is calculated as a dictionary-like score, and the dictionary-like score is compared with a threshold value based on the feature rule. Determining whether the document group satisfies the conditions of the dictionary document group,
The document group detection method according to attachment 3, wherein the document group detection method is a procedure.

  (Supplementary Note 7) When a plurality of the specific relations are selected, the document group determination procedure is selected based on the feature rule in which a predetermined coefficient corresponding to the importance of each specific relation is defined. The dictionary-likeness score calculated for the specific relationship is multiplied by the predetermined coefficient and weighted to calculate a dictionary-likeness score corresponding to the selected specific relationship. The document group detection method according to appendices 4, 5, and 6, which is a feature.

(Additional remark 8) The said feature totaling procedure produces | generates the entry information which linked | related the identification information of the said character string and the said connection destination document further about the said character string and the said connection destination document which satisfy | filled the said specific relationship,
The outputting step outputs the entry information generated for the document group together with the identification information of the document group registered in the target document group candidate.
The document group detection method according to appendix 4, 5, or 6, wherein the document group detection method is a procedure.

  (Supplementary Note 9) In the output procedure, when the user designates the target document group candidate as the target document group, the target document group is registered with the identification information of the designated target document group candidate. It is a procedure for registering in the document group information and storing it in the document group storage unit, and storing the entry information corresponding to the target document group candidate in the document group storage unit in association with the document group information. The document group detection method according to appendix 8.

(Supplementary Note 10) In a document group detection apparatus for detecting a predetermined document group which is a set of documents provided on a network and managed by one or more computers,
For a group of documents having a hierarchical structure in which a plurality of subordinate documents exist under the specific document, the subordinate document is searched using a specific keyword, and the specific is determined based on the searched subordinate document. A collecting means for detecting a document and collecting a plurality of subordinate documents under the specific document;
For each of the collected subordinate documents of the specific document and the group of documents, connection information indicating a relationship with a specific other document added to an arbitrary character string in the subordinate document is extracted, and the subordinate document and , A feature counting unit that counts the number of states that have a specific relationship with the associated linked document,
Read out the feature rule from the feature rule storage means in which the feature rule that is a condition using the specific relationship is stored, determine whether the number of states of the specific relationship of the document group satisfies the condition of the feature rule, A document group determination means for registering a document group satisfying the condition as a target document group candidate;
Output means for outputting the document group registered in the target document group candidate;
A document group detection apparatus comprising:

(Supplementary Note 11) In a document group detection program for detecting a predetermined document group which is a set of documents provided on a network and managed by one or more computers,
Computer
For a group of documents having a hierarchical structure in which a plurality of subordinate documents exist under the specific document, the subordinate document is searched using a specific keyword, and the specific is determined based on the searched subordinate document. A collecting means for detecting a document and collecting a plurality of subordinate documents under the specific document;
For each of the collected subordinate documents of the specific document and the group of documents, connection information indicating a relationship with a specific other document added to an arbitrary character string in the subordinate document is extracted, and the subordinate document and , Feature counting means for counting the number of states that have a specific relationship with the linked document
Read out the feature rule from the feature rule storage means in which the feature rule that is a condition using the specific relationship is stored, determine whether the number of states of the specific relationship of the document group satisfies the condition of the feature rule, Document group determination means for registering a document group satisfying the condition as a target document group candidate,
Output means for outputting the document group registered in the target document group candidate;
A document group detection program characterized by functioning as

It is a figure which shows the outline | summary of invention. It is the figure which showed the structural example of the dictionary site detection system. It is a block diagram which shows the hardware structural example of a dictionary site detection server. It is a figure for demonstrating the characteristic of a dictionary site. It is the figure which showed the flow of the process after inputting a keyword until acquiring page information of a site. It is the figure which showed an example of URL-title table. It is the figure which showed an example of the anchor text-link destination URL table. FIG. 5 is a diagram illustrating an example of total information according to feature 1. FIG. 6 is a diagram illustrating an example of total information according to feature 2. FIG. 6 is a diagram illustrating an example of total information according to feature 3. It is the figure which showed an example of a dictionary candidate table and its entry candidate table. It is the figure which showed an example of the dictionary site candidate list screen. It is the flowchart which showed the whole process sequence of the dictionary site detection method. It is the flowchart which showed the procedure of site acquisition processing. It is the flowchart which showed the procedure of the link information extraction process. It is the flowchart which showed the procedure of the link information A extraction process. It is the flowchart which showed the procedure of the link information B extraction process. It is the flowchart which showed the procedure of the link characteristic total process. It is the flowchart which showed the procedure of the characteristic 1 analysis process. It is the flowchart which showed the procedure of the feature 2 analysis processing. It is the flowchart which showed the procedure of the characteristic 3 analysis process. It is the flowchart which showed the procedure of dictionary site determination processing. It is the flowchart which showed the procedure of dictionary entry candidate creation processing. It is the flowchart which showed the procedure of the user presentation process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Document group detection apparatus 11a Document storage means 11b Feature rule storage means 11c Total information storage means 11d Document group candidate storage means 11e Document group storage means 12 Document collection means 13 Feature totalization means 14 Document group determination means 15 Document group presentation means 15

Claims (6)

In a document group detection method for detecting a predetermined document group that is a set of documents provided on a network and managed by one or more computers,
The computer is
For a group of documents having a hierarchical structure in which a plurality of subordinate documents exist under the specific document, the subordinate document is searched using a specific keyword, and the specific is determined based on the searched subordinate document. A collection procedure for detecting a document and collecting a plurality of subordinate documents under the specific document;
For each of the collected specific document and the plurality of subordinate documents in the collected document group, connection information indicating a relation with a specific other document added to an arbitrary character string in the subordinate document is extracted, and the subordinate document and A feature counting procedure that counts the number of states that have a specific relationship with the associated linked document,
Read out the feature rule from the feature rule storage means in which the feature rule that is a condition using the specific relationship is stored, determine whether the number of states of the specific relationship of the document group satisfies the condition of the feature rule, A document group determination procedure for registering a document group satisfying the condition as a target document group candidate;
A procedure for outputting the document group registered in the target document group candidate;
A document group detection method characterized by comprising: The document group to be detected is a document group regarding explanation information of a plurality of terms in an arbitrary field, the specific document describes a list of terms to be explained, and the subordinate documents have explanations of terms. A dictionary document group that is a glossary document to be described,
Further, the connection information is added to a character string representing the term appearing in the specific document and the glossary document, and associates the character string with the glossary document corresponding to the character string,
The specific relationship in the feature counting procedure is a specific relationship between the character string associated by the connection information and the connection destination document.
The document group detection method according to claim 1. In the feature counting procedure, the character string to which the link information is added and the link destination document corresponding to the character string are analyzed, and the glossary document belonging to the same document group is determined by the link information. The number of associated internal connection information and the specific internal connection information in which the character string to which the connection information is added and the character string included in the title of the connection destination document match as the state of the specific relationship. The number and
In the document group determination procedure, a ratio of the specific internal connection information to the entire internal connection information is calculated as a dictionary-like score, and the dictionary-like score is compared with a threshold value based on the feature rule. Determine whether the conditions of the dictionary document group are satisfied,
3. The document group detection method according to claim 2, wherein the document group detection method is a procedure. In the feature counting procedure, the character string to which the link information is added and the link destination document corresponding to the character string are analyzed, and the glossary document belonging to the same document group is determined by the link information. Specific internal link information in which the number of associated internal link information and the character string to which the link information is added as a state of the specific relationship match the character string included in the file name of the link destination document And the number of
In the document group determination procedure, a ratio of the specific internal connection information to the entire internal connection information is calculated as a dictionary-like score, and the dictionary-like score is compared with a threshold value based on the feature rule. Determine whether the conditions of the dictionary document group are satisfied,
3. The document group detection method according to claim 2, wherein the document group detection method is a procedure. In the feature counting procedure, the character string to which the link information is added and the link destination document corresponding to the character string are analyzed, and the glossary document belonging to the same document group is determined by the link information. The number of associated internal linkage information and the character string to which the linkage information relating the linkage destination document from the title of the linkage destination document is added to the linkage destination document as the state of the specific relationship. Extract and compare the common title part excluding the same part, and count the number of specific linked information that matches the common title part,
In the document group determination procedure, based on the feature rule, a ratio of the specific link information to the entire internal link information is calculated as a dictionary-like score, and the dictionary-like score is compared with a threshold value based on the feature rule. Determining whether the document group satisfies the conditions of the dictionary document group,
3. The document group detection method according to claim 2, wherein the document group detection method is a procedure. In a document group detection apparatus for detecting a predetermined document group which is a set of documents provided on a network and managed by one or more computers,
For a group of documents having a hierarchical structure in which a plurality of subordinate documents exist under the specific document, the subordinate document is searched using a specific keyword, and the specific is determined based on the searched subordinate document. A collecting means for detecting a document and collecting a plurality of subordinate documents under the specific document;
For each of the collected subordinate documents of the specific document and the group of documents, connection information indicating a relationship with a specific other document added to an arbitrary character string in the subordinate document is extracted, and the subordinate document and , A feature counting unit that counts the number of states that have a specific relationship with the associated linked document,
Read out the feature rule from the feature rule storage means in which the feature rule that is a condition using the specific relationship is stored, determine whether the number of states of the specific relationship of the document group satisfies the condition of the feature rule, A document group determination means for registering a document group satisfying the condition as a target document group candidate;
Output means for outputting the document group registered in the target document group candidate;
A document group detection apparatus comprising:

Images