JP2000339316A - Method and device for collecting retrieval link type information and recording medium with its method stored therein - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide such collecting method and device as make the priority with which data is collected coincide with the request of a retrieving person and to provide a recording medium in which a computer program is stored. SOLUTION: These method store a retrieval word for retrieving a database, extract a retrieval word whose number of appearance times is a high frequency among stored retrieval words (S32), select one link with the highest priority from a list of links according to the extracted retrieval word (S34), acquire and output information indicated by the selected link, extract all the links included in the acquired information, calculate the priority of each extracted link according to the extracted retrieval word with the high frequency and add the priority and link to the above link (S40) and repeat operations that follow the link selection of the whole links in the list.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] The present invention relates to a document which is arranged such that a document including a link to another document is distributed to a plurality of servers,
And a search-linked information collection method and apparatus for comprehensively acquiring documents of a distributed hypertext system in which documents held by each server are independently added, deleted, and updated, and a recording medium recording the program About.

[0002]

2. Description of the Related Art World Wide Web (hereinafter referred to as W) is a distributed hypertext system on the Internet.
In WW), many servers add their content without permission.
Deleted / updated. Therefore, in order to search the entire WWW, a method of first collecting copies of documents placed on each server in one place and searching the data is adopted.

In such a search system, there is a process (hereinafter referred to as a crawling process) for collecting a copy of a document of each server. The crawling process collects distributed data by acquiring data and continuing to follow the link referred to by the data. At the same time, the same document is re-collected at regular intervals to keep the latest document. However, since such a collection method usually takes a long time, it is necessary to determine a priority of collecting data in order to efficiently collect data.

Conventionally, Lyco US
There is a method of "company giving priority to a document having many referenced links" which is a method of company s. In this method, “information frequently referred to from other documents”, that is, common sense and information that can be found immediately is given priority. However, searchers are only looking for information that they do not know or information that is not easily found, and there is a problem that this priority determination method does not increase the priority of such information. Was.

[0005]

SUMMARY OF THE INVENTION An object of the present invention is to record a search-linked information collection method, apparatus and program for providing a data collection method in which the priority of collecting data matches the needs of a searcher. It is to provide a recording medium.

[0006]

A search-linked information collecting method according to the present invention accumulates search terms for searching a database, extracts a search term having a high frequency of appearance from the accumulated search terms, Select one link with the highest priority from the list of links according to the extracted search term, acquire and output the information indicated by the selected link, and extract and extract all the links included in the acquired information The calculated priority of each link is calculated in accordance with the extracted high-frequency search term, and the priority and the link are added to the list. It is characterized by repeating.

[0007] A search-linked information collecting apparatus according to the present invention comprises: means for accumulating search terms for searching a database; means for extracting high-frequency search terms from the accumulated search terms;
Means are provided for collecting information linked to data containing high-frequency search terms and adding the information to a database. ADVANTAGE OF THE INVENTION According to this invention, richer and newer information can be collected about the frequently used search.

[0008] A recording medium storing the search-linked information collection program of the present invention stores search terms for searching a database, extracts high-frequency search terms from the stored search terms, and performs high-frequency search. Gather information linked to data containing words and add it to the database.

[0009] The main feature of the present program is to collect more information about frequent search requests. ADVANTAGE OF THE INVENTION According to this invention, more rich and new information can be collected about the search word used frequently.

[0010]

FIG. 1 shows an overall configuration of an information collecting apparatus according to an embodiment of the present invention. The input device 101 is for inputting a search word. The storage device 104 stores a search program, an information collection program, a database for storing collected data, and search history data for recording used search terms.

A search word is input from the input device 101. The control device 103 adds the search term to the search history data in the storage device 104. In addition, a search program is started, a database is searched using the search word, and the result is output to the output device 102.

On the other hand, the collection program is started independently and periodically regardless of the presence or absence of a search request. The collection program collects documents linked one after another using the link information in the document from a given starting point. Generally, one document includes a plurality of links, and uncollected links need to be prioritized and processed in order.

[0013] For this reason, a search term frequently used is checked by referring to the search history data. If a frequently used search word is highly related to a link in the document, the priority of the link is increased.

[0014] The collected documents are added to a database in the storage device 104 and provided for searching.

Although one storage device 104 is used for simplicity, a database or a program may be generally distributed to a plurality of storage devices 104 in order to reduce an input / output load. Further, the input device 101 and the output device 102 may be different computers connected via a network.

FIG. 2 is a diagram showing a hardware configuration when the apparatus of FIG. 1 is realized using a CPU 201. FIG.
In the CPU 201, a memory 202, a display 204 as a display device, and a keyboard 2 as an input device are provided.
03, a hard disk 205 as a storage device is connected. The hard disk 205 has a search program 2
06, a collection program 207, a database 208, and search history data 209 are stored.

FIG. 3 is a flowchart of the information collection program in FIG. A list of origin links from which information is desired to be acquired is put into a queue, parameters for extracting high-frequency search terms are given, the process is started (31), and high-frequency search terms are extracted (32). The extraction of a high-frequency search term will be described with reference to FIGS. If the link is empty (33), the process ends (41). If the link is not empty, the link with the highest priority is selected from the queue (34), and the selected link is deleted from the list (35). Then, the information indicated by the link is obtained (3
6) Output the obtained information (37). Next, all the links included in the acquired information are extracted (38), and the priority of each extracted link is calculated (39). The method of obtaining the priority will be described later (FIG. 7). Then, the priority and the link are added to the list (40). Next, the same processing is performed for another link, and the processing is repeated until the queue becomes empty (3.
3).

In actual operation, in order to prevent an infinite loop or taking too much time,
After a certain period of time, it is necessary to terminate even if the queue is not empty.

The priority of the link is calculated using a set of a keyword and a frequency of appearance of the keyword in search history data given in advance. For each keyword, (Relationship between link and keyword x Frequency of keyword appearance)
, And the larger the sum of the keywords, the higher the priority.

Here, as a method of calculating the degree of association between the link and the keyword, it is conceivable to divert the degree of association between the entire document including the link and the keyword. TF * IDF is known as the degree of association between a document and a keyword.
In this case, the links included in the same document are given the same priority. Here, TF (TermFr
EQF) indicates the frequency of occurrence of a word, and IDF
(Inversed Document Frequency
cy) is the reciprocal of the numerical value of what percentage of a plurality of documents a word appears in, and * represents multiplication.

In the case of HTML, since link information is often given to words or phrases,
It is also conceivable to use the appearance frequency of this word in the search history data as the relevance.

FIG. 4 is a diagram showing an example of calculating the priority of a link. Here, the frequency of appearance of the word to which the link is assigned in the search history data is used as the degree of association.

First, the words "Linux", "chat" and "MIDI" are used as words having a high frequency of appearance from the search history data.
It is assumed that “download” and “Windows” are obtained and the number of appearances is as shown in FIG. on the other hand,
Documents for which you want to calculate link priorities include "Linux
Page "and" MIDI page "are included. Then, since the link “Page of Linux” includes the word “Linux”, the relevance of the link to the search history is 45000. Similarly, the relevance of “MIDI page” is 3700
0. Assuming that the higher the numerical value, the higher the priority, the degree of relevance may be used as the priority as it is.

FIG. 5 shows an example of steps for extracting a high-frequency search word from the search words stored in the present invention. (A) is an example of accumulated search terms, (C) is an example of extracted high-frequency search terms, and (B) is the content of the extraction process.

In the search system, as shown in FIG. 5A, when and what search is performed are recorded in a search history file.

FIG. 5B is an operation flowchart of the extraction processing of a high-frequency search word, which is started by inputting the number of days D and the number of output words N (51). The search log from the day before D (for example, three days before) to the previous day is read (52), and the number of appearances of each search word in the log is checked (53). Next, the search words are rearranged in descending order of the number of appearances (54), and the search words and the number of appearances are output from the most frequently appearing N (for example, the top 20%) (55), and the process ends (56) . The output is
It is a table in which N ranks, search words, and the number of appearances of search words from D days ago to the previous day are arranged.

Here, how many days of history in the past are used, and what percentage of the search words are output may be changed depending on the frequency of use of the search system and variations in the search words. If the number of search words is very large, instead of rearranging all search words according to the number of appearances in (B), search words below a certain frequency are rearranged only for the remaining search words that have been truncated. May be performed.

FIG. 6 shows an example in which an expansion rate of the number of times of use of the search word is used instead of the number of times of use of the search word as a search word extraction criterion. First, the past calculation results of the ranking and the number of appearances of the search word obtained in FIG. 5 are recorded.

Next, the difference between the previous day and the current order for each search word is determined from the order of the search words newly calculated according to FIG. 5 and the order of each search word on the previous day. For example, FIG.
In the "interest rate" has risen from third to second place
1. Conversely, “shares” have dropped from second to third, so -1
Becomes

This variation is subtracted from the latest ranking. Then "interest rate" ranks first and "shares" rank fourth.

By using the change of the rank as described above,
The future ranking of each search term can be estimated and used in place of the actual ranking. In this example, the difference from the ranking one day ago is most simply used as the gradient of the change in the ranking. However, how much the ranking in the past is used and how many estimations are performed may be changed depending on the nature of the search system.

FIG. 7 is an operation flowchart showing a process of determining the priority of a link based on the order of each search word obtained in FIG. 5 or FIG. As an input, a list of links whose priority is to be calculated and an output result of a frequently searched word are given to start (71). If the priorities of all the links have been calculated (72), the process ends (78). If the link has not been calculated, one link whose priority has not been calculated is selected (7
3) Divide the link title into words (74). For the division, a method in which a sequence of kanji or katakana is regarded as a word or a morphological analysis using a dictionary can be used. Next, the number of appearances of each word of the title is obtained from the output result of the high-frequency search word calculated in advance (75),
The total number of appearances is calculated (76), and the calculation result is output as the priority of the selected link (77). The above operation is repeated for all links (72).

Often, there are only demonstrative pronouns such as "here" and "this" as link titles. For example, instead of creating a link called “XX newspaper page”,
This is a case where a link is set at the “here” portion of the sentence “XX newspaper is here”. By using the words of the entire sentence including the link instead of acquiring the title of the link in FIG. 7, such a case can be dealt with.

[0034]

As described above, according to the first aspect of the present invention,
By using the search-linked information collection method, richer and newer information about frequently used search words can be collected.

With the use of the search-linked information collection device according to the second aspect of the present invention, abundant and new information can be collected for frequently used search words.

By using the recording medium storing the search-linked information collection program according to the third aspect of the present invention, richer and newer information can be collected for frequently used search words.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating an overall configuration of hardware necessary for an embodiment of the present invention.

FIG. 2 is a diagram illustrating a hardware configuration when the device in FIG. 1 is implemented using a CPU.

FIG. 3 is a flowchart of an information collection program in FIG. 2;

FIG. 4 is a diagram illustrating a calculation example of link priorities;

FIG. 5 is a diagram illustrating an example of extracting a high-frequency search word from search words stored in the present invention.

FIG. 6 is a diagram illustrating an example of extracting a search word in a case where an expansion rate of the number of times of use of the search word is used instead of the number of times of use of the search word as a search word extraction criterion.

FIG. 7 is an operation flow of a process of determining a priority order of a link.

[Explanation of symbols]

 Reference Signs List 101 input device 102 output device 103 control device 104 storage device 201 CPU 202 memory 203 keyboard 204 display 205 hard disk 206 search program 207 collection program 208 database 209 search history data

Continued on the front page (72) Inventor Kazuo Tanaka 3-19-2 Nishishinjuku, Shinjuku-ku, Tokyo F-term (reference) in Nippon Telegraph and Telephone Corporation 5B075 ND03 ND36 NK31 PR04

Claims (3)

[Claims] In a method for collecting information, a search term for searching a database is accumulated, a search term having a high frequency of appearance is extracted from the accumulated search terms, and a link of a link is extracted according to the extracted search term. Select one link with the highest priority from the list, obtain and output the information indicated by the selected link, extract all the links included in the obtained information, and set the priority of each extracted link. Calculating according to the extracted high-frequency search term, adding a priority and a link to the list, and repeating the operation following the selection of the link for all the links in the list. Linked information collection method. 2. A search-linked information collection device for collecting information, comprising: means for accumulating search terms for searching a database; means for extracting a search term having a high frequency of appearance from the accumulated search terms. Means for selecting one of the links having the highest priority from the list of links according to the extracted search term; means for acquiring and outputting information indicated by the selected link; and linking included in the acquired information. Means for calculating the priority of each extracted and extracted link according to the extracted high-frequency search term, and adding the priority and the link to the list; Means for repeating the operation following the selection of a link. 3. A storage medium storing a computer program for collecting information, accumulating search terms for searching a database, extracting a search term having a high frequency of appearance from the accumulated search terms, and extracting the extracted search terms. One of the links with the highest priority is selected from the list of links according to the search term obtained, the information indicated by the selected link is obtained and output, and all the links included in the obtained information are extracted and extracted. A computer that calculates the priority of each link according to the extracted high-frequency search term, adds the priority and the link to the list, and repeats the above-mentioned operation of selecting the link for all the links in the list. A storage medium storing a search-linked information collection program, characterized by storing the program.