JP2001060198A - Information collecting method and recording medium recording information collection program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information collecting method for appropriately acquiring an information page deeply related with a keyword out of a little links of a network. SOLUTION: This information page collecting method has a step for acquiring the link candidate list of related information pages from the sets of retrieval group sets pairing two words of retrieval word and object word for each of retrieval words more than one determined beforehand concerning the object word, a set for acquiring the information page pointed by a link concerning all the links of the link candidate list and adding the link of information pages to Hub link list when the number of link published on the information page is more than a predetermined number and a step for tracing links for predetermined steps among the links on the Hub link list, acquiring all the information pages pointed by the traced links and defining them as an information page set.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for collecting necessary information from the Internet, and more particularly to an automatic construction of a knowledge base for handling hypertext (hereinafter referred to as "Web page") in which a plurality of texts are linked by links. The present invention relates to an information collection method and a recording medium on which an information collection program is recorded.

[0002]

2. Description of the Related Art In recent years, with the spread of the Internet, many texts have been provided electronically.
In view of these circumstances, research has been conducted to automatically construct a knowledge base used for intelligent systems and machine translation from texts on the Internet. In the automatic construction of such a knowledge base, an important technical issue is how to efficiently collect texts useful for the construction of a knowledge base from a huge text group on the Internet.

As an example, consider an intelligent system for determining the performance of a digital camera. In order to construct a knowledge base of such an intelligent system, it is necessary for each model to have knowledge such as "What is the number of pixels of a CCD?" And "What are its features?" Such knowledge is often described on a Web page provided by a digital camera maker (for example, a product announcement article or the like often provides a main feature or specification table of a product). Therefore, if such Web pages can be collected, a knowledge base can be automatically configured using a technology for acquiring knowledge from text.

In collecting texts from the Internet, a search based on a keyword appearance check has been conventionally used. In such a method, first, a target (a digital camera in the above example) handled by the intelligent system is used as a keyword, and a link in text in which the keyword appears is searched. Then, necessary text is extracted by examining the text pointed to by the link thus obtained one by one.

[0005]

However, on the Internet, there are a huge number of texts including words of digital cameras and the like, and an enormous number of links are often obtained as a result of a search using a certain keyword. In such a case, finding a text necessary for the knowledge base requires a large number of links to be searched one by one, so that much labor / time is required. In order to solve such a problem, a method using a search focusing on the frequency of occurrence of a keyword has been proposed.

In this method, attention is paid to the frequency of appearance of a keyword, and a text link having a high frequency of appearance by the keyword is presented with priority. However, the text required for the construction of the knowledge base is not necessarily the text in which many keywords appear. For example, in a product announcement article of a digital camera presented by a manufacturer, the keyword of a digital camera is a redundant description, and thus does not appear so often. Therefore, even if such a method based on the frequency of occurrence is used, it is not possible to narrow down the links pointing to the texts necessary for the construction of the knowledge base, and in the end, avoid the difficulty of finding the necessary text from many links. Can not.

As described above, since the prior art is a method of presenting a link of a text in which a keyword appears, many links are searched, and enormous effort is required to search for an appropriate text therefrom. There is a problem that it takes time.

[0008] An object of the present invention is to collect Web pages closely related to a keyword by using a Web page in which selected links closely related to a keyword are collected, and to select appropriate Web pages from a small number of carefully selected links. It is an object of the present invention to provide an information collection method for acquiring a page and a recording medium on which an information collection program is recorded.

[0009]

According to the information page collecting method of the present invention, when a target word which is a certain keyword is given, "a set of hypertexts in which a plurality of texts are linked by a link" In an information collection method for obtaining a set of “hypertexts closely related to a target word” (hereinafter referred to as an information page) from a network, a Hub that collects links of information pages describing a target indicated by the target word When acquiring a link candidate list pointing to a page, for each of the search words that are one or more predetermined words, configure a set of search groups in which two words of the search word and the target word are set, For information pages on the network, link the information pages that are relevant to at least one of the search sets. In a link candidate obtaining step of obtaining a link candidate list, and in extracting a Hub link list including only links of Hub pages from the link candidate list, an information page indicated by the link is obtained for all links in the link candidate list. A hub page determining step of adding the link of the information page to a hub link list when the number of links on the information page is larger than a predetermined number;
When obtaining a set of information pages from the web link list, the links in the hub link list are followed by a predetermined number of steps, and all the information pages indicated by the searched links are obtained as an information page set. The method further comprises a page acquisition step of setting the information page set to a set of information pages closely related to the target word.

[0010] Further, a recording medium on which the information collection program of the present invention is recorded is a recording medium on which an information collection program for acquiring a set of information pages from a network when a target word as a certain keyword is given. In order to obtain a link candidate list indicating a Hub page in which links of information pages describing the target indicated by the target word are obtained, the search is performed for each of one or more predetermined search words. A link that constitutes a set of search pairs in which two words, a word and a target word, are combined, and obtains a link candidate list in which information pages on the network are linked to information pages that are related to at least one of the search pairs. A Hub consisting of only a link of the Hub page from the candidate acquisition program and the link candidate list When taking out a Nkurisuto,
For all links in the link candidate list, an information page pointed to by the link is acquired, and when the number of links on the information page is larger than a predetermined number, the link of the information page is changed to a Hub link list. To obtain a set of information pages from a Hub link list,
b From the links in the link list, follow the link by a predetermined number of steps, acquire all the information pages pointed to by the followed link to form an information page set, and use the information page set to make an information page closely related to the target word. Page acquisition program.

In summary, the information page collection method of the present invention includes three steps: a link candidate acquisition step, a Hub page determination step, and a page acquisition step. In the link candidate obtaining step, for each of one or more predetermined search words, a set of search groups in which two words of the search word and the target word are set is configured. Then, for Web pages on the Internet, a list is obtained in which links of Web pages related to at least one of the search groups are arranged.

[0012] In the Hub page determination step, for all links in the link candidate list, the Web pointed to by the links is referred to.
When a page is acquired and the number of links posted on the Web page is larger than a predetermined number,
The link of the Web page is added to the Hub link list.

In the page obtaining step, the link is followed by a predetermined number of steps from the links in the Hub link list, all the Web pages pointed to by the followed link are obtained as an information page set, and the information page set is obtained. It is a set of Web pages closely related to the target word.

Therefore, in the link candidate obtaining step of the present invention, a search is performed by combining a search word obtained by collecting words related to the Hub page with the target word. Thereby, Hub
A candidate for a link pointing to the page can be obtained.

Next, in the Hub page determination step, it is possible to determine whether or not the collected Web pages are Hub pages by paying attention to the feature of the Hub page that “the number of the posted links is large”. . As a result, a list including only the links of the Hub pages can be extracted from the link candidate list.

The link posted on the Hub page is
This is a collection of Web page links closely related to the target word. The page acquisition step collects Web pages indicated by the links posted on these Hub pages. As a result, Web pages closely related to the target word can be collected.

As described above, according to the present invention, Web pages closely related to a target word can be collected using the three steps of the link candidate acquisition step, the Hub page determination step, and the page acquisition step. .

[0018]

Next, an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram for explaining an embodiment of an information collecting method according to the present invention, and is a block diagram of a Web page collecting method according to the present embodiment on the Internet.

Referring to FIG. 1, the information page collecting method of the present invention comprises a link candidate (URL candidate) acquiring step 11;
It comprises three steps: a Hub page determination step 12 and a page acquisition step 13.

The link candidate obtaining step 11 includes a Web page describing a target indicated by the target word which is a keyword.
Web that is a candidate for a Hub page that collects page links
Get page b. These steps are configured as, for example, steps of performing two processes of a search group configuration process and a Web page search process.

Here, the search set construction process forms, for each of the search words, a set of search sets in which two words of the search word and the target word are combined. The search group composition process
In this way, a set of search sets is formed. In the Web page search process, a search for a Web page closely related to each search group is performed for the search group received from the search group configuration process. Such a Web page search process can be realized, for example, as the following process.

First, for each search set, a text containing both words included in the search set is extracted from a database (hereinafter, a Web database) that collects Web pages. Then, W extracted for each search set
The web pages are combined into a Hub candidate page. Web
In the page search process, Web pages that are candidates for Hub pages can be collected as described above.

In a Web page search process, a Web page is searched from a search group.
The process of retrieving page b is realized by, for example, the following two processes of a primary search process and a secondary search process. Here, in the primary search processing, the first word of the search set is set as keyword 1, and only the text including the keyword 1 is extracted from the Web database. In this retrieval, the primary search process uses a Web page as an input character string. Then, the character string is extracted from the first character of the input character string and collated with the first character of keyword 1. If the collation is successful, the input character string is collated with the next character of the keyword 1 and the second character of keyword 1. Such collation is performed for all characters of keyword 1, and if all characters of keyword 1 are successfully collated, it is determined that the Web page includes keyword 1. As described above, in the primary search processing, only the text including the keyword 1 is extracted from the Web database. In the secondary search process, the second word of the search set is set as keyword 2. Then, a text including the keyword 2 is extracted from the Web page extracted by the primary search process. Such retrieval can be realized in the same manner as the primary search processing. The link candidate acquisition step is configured as, for example, a step of performing the two processes of the search group configuration process and the Web page search process as described above.

Link candidate acquisition step 11 as described above
Can be realized using, for example, a net search engine provided on the Internet. In such a net search engine, W
You can search for links on web pages. Such a search is basically realized by focusing on the appearance of a word, as described above. Using such a net search engine, a word in which a word included in a search group appears together
This can be realized by acquiring the link of page b and acquiring the Web page indicated by the link thus obtained. The link candidate acquisition step 11 can be configured as, for example, a step using a net search engine as described above. As described above, the link candidate acquisition step 11 can be configured to collect Hub page candidates.

The Hub page determination step 12 extracts a Hub page from the Hub page candidates collected in the link candidate acquisition step 11. These steps are configured as, for example, steps of performing three processes of a link extraction process, an external link determination process, and an external link number evaluation process.

Here, the link extraction process extracts, for each of the candidates for the Hub page, a list of links listed on the Hub page. Such a link extraction process can be realized by focusing on the HTML href tag. HTM that describes Web pages
In the L language, an href tag is used as a tag indicating a link destination. Therefore, the link name can be extracted by extracting the portion surrounded by the href tag. The link extraction processing is performed as described above.
A list of links can be extracted from each of the candidates for the ub page.

The external link determination processing determines whether the link extracted by the link extraction processing is an external link or an internal link. Here, the external link is a link in which the link points to a site other than the Web page to which the link is provided. Also, an internal link is
The link is a link pointing to the same site as the Web page to which the link is provided. These determinations are based on whether the link format is a link starting with http: //
Alternatively, it can be determined by simply starting with a directory name. The external link determination process can determine whether each link is an external link or an internal link as described above.

The external link number evaluation process determines, for each Hub page candidate, whether the candidate page is a Hub site. First, the external link number evaluation process
For each Hub page candidate, the number of external links listed on the candidate page is counted. Such a count process can be easily realized using a list of links extracted by the link extraction process and a determination result for each link determined by the external link determination process. Next, in the external link number evaluation processing, candidate pages whose number of external links is larger than a predetermined value are set to H.
It is determined that the page is a web page. Conversely, candidate pages whose number of external links does not exceed a predetermined value are set to Hu.
It is determined that the page is not page b. As described above, the external link number evaluation process can determine, for each Hub page candidate, whether the candidate page is a Hub site.

As described above, the Hub page determination step 12 can be configured as a step of performing three processes of the link extraction process, the external link determination process, and the external link number evaluation process.

The page obtaining step 13 obtains a Web page that is deeply related to the target word by using a link listed on the Hub page. Such a step is configured as, for example, a step of repeating two processes of a link extraction process and a link destination acquisition process a predetermined number of times. Here, the link extraction processing can be realized in the same manner as in the Hub page determination step 12. The link extraction process in the page acquisition step 13 is first performed by Hu
Extract all links on the page determined to be page b.
Then, the link destination acquisition processing acquires a Web page indicated by the link extracted by the link extraction processing, and sets this as a secondary Hub page. When the secondary Hub page is obtained, the link extraction process extracts all listed links for each of the secondary Hub pages. In the link destination acquisition process, a Web page indicated by the link extracted by the link extraction process is acquired again.
The next Hub page is set. The page acquisition step 13 repeats the above two processes a predetermined number of times. Then, by the above processing, all the Web pages obtained from the link destination obtaining processing are set as the Hub pages closely related to the target word. As described above, the page acquisition step 13 can be configured as a step of repeating the two processes of the link extraction process and the link destination acquisition process a predetermined number of times.

By the above three steps, a Web page closely related to the target word can be obtained.

Next, an embodiment of a recording medium on which the information collection program of the present invention is recorded will be described with reference to the drawings.

FIG. 2 is a diagram showing a configuration of a computer which acquires a Web page closely related to a target word from the Internet using the information collection program of the present invention. The input device 21
An input device that accesses the Internet and configures one search group together with the target word for each of the search words to form a search group from the target word and the search word. The storage device 22 reads various programs and tables of the information collection program from the recording medium 24 and passes them to the data processing device 25 for arithmetic processing.
5 is a storage device for storing data from the storage device 5. Output device 2
Reference numeral 3 denotes a display device for accessing the Internet, inputting a search set of a target word and a search word, displaying a process of each processing program, a printer device for printing a record, and the like when executing the information collection program. Output device. The storage medium 24 is a floppy (registered trademark) disk on which the information collection program of the present invention is recorded, C
It is a storage medium such as a D-ROM, a magneto-optical disk, and a semiconductor memory. The data processing device 25 reads the information collection program from the recording medium 24 and the search group from the target word and the search word from the input device 21 and executes a program for acquiring a Web page closely related to the target word from the Internet. is there. The communication device 26 is a communication device such as a modem or a DSU that accesses the Internet and controls communication with the Internet.

The information collection program of the present invention recorded on the storage medium 24 includes a link candidate acquisition program 241 and
It comprises a Hub page determination program 242 and a page acquisition program 243.

The link candidate acquisition program 241 configures a set of a search group in which two words, a search word and a target word, are set for each of one or more predetermined search words, and the information page on the network. Is a program for obtaining a link candidate list in which links of information pages related to at least one of the search sets are arranged.

The Hub page determination program 242 obtains, for all links in the link candidate list, information pages indicated by the links, and when the number of links on the information page is larger than a predetermined number. , A program for adding the link of the information page to the Hub link list.

The page acquisition program 243 follows a predetermined number of links from the links in the Hub link list, acquires all the information pages indicated by the visited links, and sets them as an information page set. Is a set of information pages closely related to the target word.

[0038]

Hereinafter, the collecting operation of the above steps will be described in detail based on an example of a target word and a search word shown in FIG. 3 and an example of a Web page shown in FIG.

In the example shown in FIG. 3, the target word is {digital camera}, and the search word is {ranking, list, link collection}. First, the link candidate acquisition step forms a search group from the target word and the search word. In constructing a search set, each search word is combined with the target word into one search set. As a result, in the example shown in FIG. 3, three search sets of {digital camera, ranking}, {digital camera, list}, {digital camera, link collection} are obtained. The link candidate acquisition step includes:
Based on the search sets obtained in this way, a search for a Web page closely related to each search set is performed. In describing the search operation, two Web pages shown in FIG. 4 will be used as an example. First, in the link candidate acquisition step, the first search group {digital camera, ranking} is extracted. further,
The “digital camera” in the first term of the search set is extracted as the keyword 1. Then, the We including the keyword 1
Take out page b. As a result, the two We shown in FIG.
Both b pages will be fetched. Next, the link candidate obtaining step extracts “ranking” in the second term of the search set as keyword 2. Then, a Web page including the keyword 2 is extracted from the Web page including the keyword 1. Web page 1 shown in FIG. 4 includes the word “ranking”, while Web page 1
b page 2 does not include the word "ranking". Therefore, only Web page 1 is extracted. On the Hub page, "ranking", "comparison",
Words related to the enumeration of the object, such as "list" and "link collection", are often used. Therefore, by preparing such a word as a search word, it is possible to extract the Hub page.

The Hub page determination step extracts a Hub page from the Hub page candidates collected in the link candidate acquisition step. The Hub page is a page that has many links, so in that page,
Many external links are listed. Therefore, by examining the number of external links, it is possible to determine whether or not the page is a Hub page. In the Hub page determination step, first, a list of links in the page is extracted. This process will be described using Web page 1 shown in FIG. 4 as an example. Hu
In the b-page determination step, first, the page is divided by a symbol “<ahref =”. As a result, Web page 1 is divided into six parts shown in the center of FIG. Then, the first <HTML >><HEAD> ‥‥ <Tab
le><tr><td> are excluded. Then, for the second and subsequent parts, the first ">" of each part
Extract up to the part. As described above, the link destination can be extracted from the Web page. In the case of this example, the number of links is five. Here, for example, if it is determined that a Web page having five or more links is determined as a Hub page, Web page 1 is determined as a
Can be retrieved as a page. The threshold value of the number of links is set as a small value when it is desired to find many Hub pages, and as a large value when it is not necessary to find so many Hub pages. The Hub page determination step can determine whether each Web page is a Hub page as described above.

The page acquisition step uses the link extracted in the Hub page determination step to generate a Web page.
You can get the page. This page acquisition step performs the same processing as the link extraction described in the Hub page determination step, and follows the link a predetermined number of times. Regarding the number of times to follow a link, the number is set to a large value when a large amount of text is required, and the number is set to a small value when a small number of texts are sufficient. Further, if the number of times is set to a large value, it takes a lot of time to collect, so if it is desired to perform collection in a short time, the number of times is set to a small value. In the page acquisition step, as described above, the page of the followed link is acquired at the same time as following the link. The page acquisition step can acquire a Web page as described above.

As described above, according to the present invention, Web pages can be collected.

[0043]

As described above, according to the present invention, when collecting Web pages closely related to a target word,
Use the links listed on the Hub page. Such a Hub page is a page in which important links carefully selected by human hands are collected. Therefore, it is possible to acquire an appropriate Web page from a small number of carefully selected links as compared with a conventional method in which important links are automatically collected by a net search engine. As described above, according to the present invention, there is an effect that a Web page closely related to a target word can be collected from fewer links.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining an embodiment of an information collecting method according to the present invention.

FIG. 2 is a diagram illustrating a configuration of a computer that acquires a Web page from the Internet using the information collection program of the present invention.

FIG. 3 is a diagram showing an example of a target word and a search word.

FIG. 4 is a diagram illustrating an example of a Web page.

FIG. 5 is a diagram illustrating an example of an information link list acquisition process.

[Explanation of symbols]

 11 Link Candidate (URL Candidate) Acquisition Step 12 Hub Page Determination Step 13 Page Acquisition Step 21 Input Device 22 Storage Device 23 Output Device 24 Recording Medium 25 Data Processing Device 26 Communication Device

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B075 KK07 ND20 NK02 NK44 PP02 PP03 PP12 PP13 PP22 PQ02 PQ42 PQ46 QS20 UU40 5B082 EA00 EA01 GC04

Claims (2)

[Claims] 1. When a target word which is a certain keyword is given, an information page which is a hypertext closely related to the target word from a network which is a set of hypertexts in which a plurality of texts are linked by a link. In the information collection method for obtaining a set, a search for one or more predetermined words is performed in obtaining a link candidate list indicating a Hub page in which links of information pages describing the target indicated by the target word are collected. For each of the words, form a set of search sets each of which is a pair of the search word and the target word. For information pages on the network, link links of information pages related to at least one of the search sets. A link candidate obtaining step of obtaining a linked list of links, When retrieving a Hub link list consisting only of Hub page links from the supplementary list, an information page indicated by the link is obtained for all the links in the link candidate list, and the number of links described in the information page is determined in advance. A hub page determination step of adding a link of the information page to a hub link list when the number is larger than a predetermined number; and acquiring a set of information pages from the hub link list, from the links in the hub link list, A page acquisition step of following links by a predetermined number of steps, acquiring all information pages indicated by the followed links to form an information page set, and using the information page set as a set of information pages closely related to the target word. Information page collecting method characterized by having 2. When a target word which is a certain keyword is given, an information page which is a hypertext closely related to the target word from a network which is a set of hypertexts in which a plurality of texts are linked by a link. A storage medium storing an information collection program for obtaining a set, wherein a link candidate list indicating a Hub page that collects links of information pages describing a target indicated by the target word is obtained by a predetermined one. For each of the two or more search words, a set of search sets is formed by combining the search word and the target word, and the information page on the network is associated with at least one of the search sets. Get link candidate list that links the links of a certain information page When extracting a program and a Hub link list including only Hub page links from the link candidate list, an information page indicated by the link is obtained for all the links in the link candidate list, and the information page is posted on the information page. A hub page determination program for adding a link of the information page to a hub link list when the number of links is greater than a predetermined number; and a hub link list for acquiring a set of information pages from the hub link list. A set of information pages deeply related to the target word is obtained by tracing the links by a predetermined number of steps from the inside links, obtaining all the information pages pointed to by the tracing links, and forming an information page set. Information having a page acquisition program to be used Recording medium recording a current program.