JP2002082923A - Information collating method, information collating device, and recording medium with information collection program recorded thereon - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make easily increasable simultaneously executable information collection processing units, and to make improvable the collecting performance (the number of information collectable per time) at need. SOLUTION: Computers CA, CB, and CC are connected through a network 7 to plural WWW servers. When the information of an URL received from a URL list reading part 1A is not collected, information processing parts 2A1 and 2A2 in the computer CA acquires a WWW page from a WWW indicated by the URL, and holds it on a file system. An URL extracting part 3A extracts the URL from the file in which the WWW page is prepared, and searches the number of the collection processing part in charge of collection according to calculation from the WWW server name indicated by the URL, and outputs the extracted URL to directories corresponding to the respective computers CA, CB, and CC to which the collection processing parts in charge of the collection belong on a disk 6A, and informs the respective computers CA, CB, and CC of the URL.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for providing information in parallel from a plurality of information providing servers which are provided on a network and provide information provided with an information address comprising an information providing server name and an identifier on the server. Information collection method for collecting information.

[0002]

2. Description of the Related Art Conventionally, when collecting information in parallel from a plurality of information providing servers, Sergey and Brin's paper "The
Anatomy of a Large-Scale Hypertextual Web Search E
ngine "(Proceedings of Seventh International Worl
d Wide Web Conference (1998)), an information address (called URL in the above paper) management module (URL ser ver in the above paper)
One is executed at the same time, and this module distributes information addresses so that a plurality of processing units do not execute collection from the same server, and causes the collection to be performed. (FIG. 2).

[0003]

However, in this method, all the information addresses found during collection must be processed by the information address management module, and the information addresses are found when the number of processing units increases. There is a problem that the processing of the management module cannot keep up with the speed, which causes a decrease in performance.

[0004] Therefore, it is assumed that a method has been provided for determining a processing unit responsible for collecting information addresses found for each processing unit. At this time, considering the number of connections between the processing units that notify the information addresses, if the number of processing units is n, the number of connections is 2n if the information addresses found by the processing units are notified via the information address management module. However, if notification is performed directly between the processing units, the number of connections is n ² , and the number of connections increases immediately as the number of processing units increases. When the number of processing units can be increased on one device, the notification between the processing units is relatively easily performed, but when more processing units are required, a plurality of devices are used, and the devices are straddled. There is a problem that notification is generally difficult. Further, when the processing unit to be notified is stopped, means for holding the information address to be notified at the notification source is required, and it is necessary to periodically check whether the processing unit on the other side has returned, which complicates the processing. There is also the problem of becoming.

An object of the present invention is to eliminate the need for an information address management module that hinders such performance improvement, simplify the method of notifying between processing units, and easily increase the number of simultaneously executable processing units. It is an object of the present invention to provide an information collection method and apparatus capable of improving performance (the number of information that can be collected per time) as needed, and a recording medium on which an information collection program is recorded.

[0006]

According to the information collection method of the present invention, a plurality of collection processing units collect information in parallel, and at that time, one collection processing unit can be connected to one information providing server at the same time. Each collection processing unit performs a calculation based on the information providing server name included in the information address found from the collected information, determines a collection processing unit in which information of the address should be collected, and assigns the information to the collection processing unit. Advertise the address.

The collection processing unit takes an information address as an input. The collection processing unit has a collected information address database, and can determine whether the information address has been collected. First, the input information address is confirmed as an uncollected information address using the collected information address database. If the information address has been collected, the information address is discarded.

[0008] The information address includes an information providing server name and an identifier on the information providing server. The collection processing unit connects to the information providing server based on the information address, and obtains information indicated by the information address by specifying an identifier on the server. After obtaining the information, the information address is registered as collected in the collected information address database. Since the information obtained from the server may include an information address, the information is analyzed and the information address is output. Information collection is performed by executing a plurality of such collection processing units simultaneously.

Each collection processing unit receives a name of an information providing server and shares a calculation method for determining a collection processing unit responsible for collecting information from the information providing server. Therefore, no matter which collection processing unit performs the calculation, if the same information server name is input, it can be determined that the same collection processing unit is in charge of collection.

Each of the collection processing units applies this calculation method to the information providing server name included in the information address found from the collected information, and determines a collection processing unit in which information of this information address should be collected. Notify the collection processing unit.

At the time of this notification, a specific directory on the file system is used. A directory is prepared in one-to-one correspondence with a set of collection processing units of an output source and an output destination. For this directory, the output source creates a file whose name is a number and outputs an information address to be notified. The file is closed for a certain time or at a certain information address, a file having a number one larger than the closed file is newly created, and the information address is output. Thus, the output source outputs the information addresses to sequentially numbered files. The output source does not make any changes to the file once it has been closed.

Regarding the file with the largest number in this directory, the output source may still write the information address, so that the collection processing unit to be notified is not read. The other files may be read because the output source has already output them and it is guaranteed that no changes will be made.

This notification method may be applied to notification between groups by collecting a plurality of collection processing units into one group.

As described above, when the notification between the collection processing units is transmitted via the file system, even if the collection processing unit receiving the notification is stopped due to some problem, the notification source is not affected and the information address is not changed. The collection processing unit to be notified can start reading the information address already written in the file system from the time when the activity is resumed. As a result, there is an advantage that the suspension of the individual collection processing unit does not affect the others.

Further, in this method, NFS (Network Fi
le System), which can be used across devices, can be used. Therefore, even if the number of devices used to increase the number of collection processing units is increased, there is no need to change the notification method between the collection processing units.

Further, when an information address found from information collected by each processing unit or a plurality of processing units as one group is notified, the processing unit or each of the plurality of processing units is regarded as one group. The information address is registered in the already notified information address list held by. When notifying the information address, the notification is performed after confirming whether the information address is registered in the already notified information address list.

To avoid duplicating a certain information address and notifying the collection processing unit responsible for collection, a database shared by all collection processing units may be prepared and notified while inquiring about it. However, it is clear that an inquiry to this database will hinder performance improvement, and is not a practical solution. By using this already notified information address list, it is guaranteed that a certain information address is output only once from a certain collection processing unit. Number of collection processing units,
Alternatively, only the number of groups obtained by grouping a plurality of collection processing units needs to be received, and the number of duplicate notifications can be suppressed.

[0018]

Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a system configuration diagram showing an embodiment of the present invention.

A plurality of WWW servers (not shown), which are information providing servers, are arranged on the network 7. WW
The WWW page, which is information that can be obtained from the W server,
Each is provided with an information address called a URL.
It consists of the name of the WWW server and an identifier on that server.

To collect information from the WWW server, one or a plurality of computers, in this case, three computers CA, CB, and CC are used, but the number of computers may be any number. Computer CA used for collection,
CB and CC are connected to each other by a network 7. Computers CA, CB, C used for collection
C and the WWW server are also connected by the network 7.

In the computers CA, CB and CC, UR
L list reading units 1A, 1B, 1C, collection processing unit 2A
₁ , 2A ₂ , 2B ₁ , 2B ₂ , 2C ₁ , 2C ₂ , and URL extraction units 3A, 3B, 3C are operating. Here, one U
RL list reading unit, two collection processing units, one UR
An example will be described in which the L extraction units operate in pairs,
The number of elements to be set is 1 for each module (part).
Any number may be used as long as the number is at least one. Although any number of such sets may be running on one computer, an example in which one set is running will be described here.

The collection processors 2A ₁ , 2A ₂ ,..., 2C ₂ are given serial numbers (0, ₁ , 2, ₃ , 4, 5).

URL list reading units 1A, 1B, 1
C, the URL extraction units 3A, 3B, and 3C have a calculation formula (assignment calculation formula) that uniquely determines the number of the collection processing unit that is in charge of collection from the WWW server by inputting the WWW server name. . This calculation formula can be realized by, for example, a hash function. Therefore, which computer UR
The same WW is used for both the L list reading unit and the URL extraction unit.
Given the W server name, the same collection processing unit number can always be determined.

URL list reading units 1A, 1B, 1C
First, a file (initial collection URL file) in which one or a plurality of URLs of WWW pages provided by a WWW server which is in charge of collection by a collection processing unit connected to the URL list reading unit is provided. This file is read and the URL
The WWW server name is taken out from L, this is given to the charge determination formula, and the URL is output to the collection processing unit according to the obtained number of the collection processing unit.

The collection processing units 2A ₁ , 2A ₂ , 2B ₁ , 2B ₂ ,
2C ₁ and 2C ₂ are the collected URL databases 4 respectively.
Have _{A 1, 4A 2, 4B 1} , 4B 2, 4C 1, 4C 2. Collected URL database 4A ₁ ~4C ₂ is, URL
Is a database that replies whether the URL is already registered as collected.

When the collection processing units 2A _{1 to} 2C ₂ receive the URL from the URL list reading units 1A to 1C, the collection processing units 2A _{1 to} 2C ₂ provide the collected URL databases 4A _{1 to} 4C ₂ with the URLs.
If the information indicated by the URL has not been collected, the WWW page indicated by the URL is collected. Otherwise, the URL is discarded, and the URL list reading unit 1A
For then obtained URL from ~1C, do a search of the collection already URL database 4A ₁ ~4C _2. URL to collect
For the URL, connect to the WWW server indicated by the URL
The WWW page indicated by L is acquired. Collection processing unit 2A
_{1 to} 2C ₂ store the obtained WWW pages on a file system (not shown) and collect the URL database 4A.
To register to _{1 ~4C 2} a URL as a collection already. After that, the file names of the files storing the collected WWW pages are notified to the URL extraction units 3A to 3C. URL extractor 3A
3C analyzes the file based on the notified file name, and extracts the URL if the file contains the URL. The name of the WWW server indicated by the extracted URL is given to the charge determination formula, and the number of the collection processing unit responsible for collection is obtained. Depending on this number, the UR of any computer
It is determined whether to notify the L list reading unit.

There are directories a, b and c under specific directories of the computers CA, CB and CC. On the disk 6A of the computer CA, / CA /
The directories a, / CA / b, and / CA / c are stored on the disk 6B of the computer CB as / CB / a, / C.
The directories B / b and / CB / c are stored on the disk 6C of the computer CC as / CC / a, / CC / b,
Assume that there is a directory / CC / c.

The URL extracted by the URL extraction unit 3A of the computer CA is output on the disk 6A. This means that even if the computers CB and CC are stopped,
As long as the computer CA is operating, the extracted URL
Is to be output. URL extractor 3A
Stores the URL to be collected by the collection processing units 2A ₁ and 2A ₂ of the computer CA in the directory a of the own disk 6A,
The URLs to be collected by the collection processing units 2B ₁ and 2B _{2 of the} computer CB are stored in the directory b of the own disk 6A, and the URLs to be collected by the collection processing units 2C ₁ and 2C ₂ of the computer CC.
The RL is output to the directory c of the own disk 6A. The file name at the time of output starts from a specific number, or when a certain number of URLs are output to one file, or when a certain time has elapsed after the output has started,
The file is closed, a file having a file name with the next higher number is created, and the output of the URL is continued. Here, description will be made assuming that the number 0 is used as the first file name.

The URL extraction units 3A, 3B, 3C
Is output to the output URL databases 5A, 5B, and 5C of the URL extraction units 3A, 3B, and 3C.
Inquires whether the URL has been output. If the URL has not been output, the URL is output to the output URL database 5
After registering in A, 5B, and 5C, the URL is output as described above.

Each of the computers CA, CB, and CC exports these directories to other computers using a network file system (NFS). Therefore, each computer CA, CB, CC
Now you can access directories and files on another computer's disk as if they were on your own disk.

URL of each computer CA, CB, CC
The list reading units 1A, 1B, and 1C use the initial collection URL
After reading the file, the URL extraction units 3A, 3B, 3
Attempts to read the URL file output by C. For example, the URL list reading unit 1A of the computer CA
Are / CA / a of computer CA and computer CB
/ CB / a of the computer and / CC / a of the computer CC. At this time, for each directory: It is assumed that n = 1.

2. Is a file with the number n created in that directory? If not, wait until you can. If there is, the file (n-1) is read.

3. Increment n by 1.

4. Repeat steps 1-3. Is performed. When this file is read, the same processing as that performed when the initial collection URL file is read is performed. As a result, the files being written by the URL extraction units 3A, 3B, 3C are read from the URL list reading units 1A, 1A.
There is a rule that B and 1C do not read.

[0036] Each computer CA, CB, URL in the CC list reading unit 1A, 1B, 1C, collection processing unit _{2A 1, 2A 2, 2B 1} , 2B 2, 2C 1, 2C 2, URL
The processing of the extraction units 3A, 3B, 3C is actually performed by a floppy disk, CD-ROM,
It is executed by being stored in a recording medium such as a magneto-optical disk.

[0037]

As described above, according to the present invention,
The effect is that the number of information collection processing units that can be executed simultaneously can be easily increased, and the collection performance (the number of pieces of information that can be collected per time) can be improved as needed.

[Brief description of the drawings]

FIG. 1 is a system configuration diagram showing an embodiment of the present invention.

FIG. 2 is a diagram for explaining a conventional technique.

[Explanation of symbols]

CA, CB, CC computer 1A, 1B, 1C URL list reading unit _{2A 1, 2A 2, 2B 1} , 2B 2, 2C 1, 2C 2 collection processing unit 3A, 3B, 3C URL extractor 4A _1, 4A _2, 4B _{1, 4B 2, 4C 1,} 4C 2 collected URL database 5A, 5B, 5C output already URL database 6A, 6B, 6C disk 7 network

Claims (9)

[Claims] 1. An information collection method for collecting information from a plurality of information providing servers which are provided on a network and provide information provided with an information address including an information providing server name and an identifier on the server. A plurality of collection processing units can collect information in parallel, in which case one collection processing unit can be connected to one information providing server at the same time, and each collection processing unit has an information address found from the collected information. An information collection method, comprising: performing a calculation based on an information providing server name included in the information processing server, determining a collection processing unit in which information of the address should be collected, and notifying the information processing unit of the information address. 2. A collection processing unit that, when a collection processing unit that has found an information address notifies another collection processing unit of the information address, sequentially outputs the information address to a numbered file on the file system and receives the notification. The method according to claim 1, wherein the communication is performed between the collection processing units or between the collection processing unit groups by reading the files other than the file with the largest number at that time in numerical order. 3. When outputting the file, a list of information addresses output in each collection processing unit is held, and the same information address is output only once.
The described method. 4. An information collecting apparatus for collecting information from a plurality of information providing servers provided on a network and providing information provided with an information address including an information providing server name and an identifier on the server. There is a collected information address database in which the information addresses from which the information has already been collected are registered. When the information addresses are received, the collected information address database is referred to, and the information of the information addresses is not collected. If so, connect to the information providing server indicated by the information address to collect the information, save the collected information in a file, register the information address as collected in the collected information address database, and then collect the collected information One or more collection processing units for notifying the file names of the files storing the information, and information for which the collection processing units are in charge of collection The information providing server name is extracted from each information address of the initial collection information address file in which the information address of the providing server is stored, and the collection processing unit responsible for collecting information from the information providing server is determined by calculation from the information providing server name. An information address list reading unit that outputs the information address to the collection processing unit; and a file having the file name notified from the collection processing unit is analyzed. If the information address is included, the information address is extracted and extracted. From the information providing server name indicated by the obtained information address, the collection processing unit of the own or other information collection device in charge of information collection is obtained, and the extracted information address is obtained on the file system of the own information collection device. An information address extraction unit that outputs the information to a file corresponding to the information collection device to which the collection processing unit belongs; The information address list reading unit, after reading the initial collection information address file, in the file system in which the information address extraction unit of its own and other information collection devices output the information address, the information collection device An information collection device that reads the corresponding file. 5. The information address extraction section sequentially outputs information addresses to numbered files on a file system, and the information address list reading section outputs information addresses other than the file with the largest number at that time. The apparatus according to claim 4, wherein the files are read in numerical order. 6. The information address extracting unit checks whether the extracted information address is registered in an output information address database when outputting the information address to the file system, and if not, outputs the information address. Outputting the information address to the file system after registering in the output information address database;
The device according to claim 4. 7. An information collection program which collects information from a plurality of information providing servers which are provided on a network and provide information to which an information address including a name of the information providing server and an identifier on the server is provided. When the information address is received, the collected information address database in which the information address for which the information has already been collected is registered is referred to. If the information of the information address has not been collected, the information provided by the information address is provided. Connect to the server to collect information, save the collected information in a file, register the information address as collected in the collected information address database, and then notify the file name of the file in which the collected information was stored One or more collection processes, and the first time the information address of the information providing server responsible for collection by the collection process is stored The information providing server name is extracted from each information address of the collected information address file, a collection process in charge of collecting information from the information providing server is determined by calculation from the information providing server name, and the information address is output to the collection process. The information address list reading process to be performed, the file of the file name notified from the collection process is analyzed, if the information address is included, the information address is taken out, and the information providing server name indicated by the taken out information address is calculated by the information providing server name, Requests the collection process of the own information collection program or other information collection program, and outputs the retrieved information address to a file on the file system of the own information collection program that corresponds to the information collection program to which the requested collection process belongs. Information address extraction processing, the information address list The reading process includes, after reading the initial collection information address file, information reading a file corresponding to the information collection program in the file system in which the information address extraction process of the own and other information collection programs output the information address. A recording medium that records a collection program. 8. The information address extracting process sequentially outputs information addresses to a numbered file on a file system, and the information address reading process includes a process for reading a file other than the file with the largest number at that time. 8. The recording medium according to claim 7, wherein the recording medium is sequentially read. 9. The information address extracting process, when outputting an information address to the file system, checks whether or not the extracted information address is registered in an output information address database. 9. The recording medium according to claim 7, wherein said information address is output to said file system after being registered in said output information address database.