JP2005135198A - Document collection apparatus and document collection method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a document collection apparatus and a document collection method in which the useless and repeated collection of documents of the same contents is prevented and an updating date of the contents of a document is highly accurately estimated in the case of automatically acquiring a document existing in another computer through a computer network. <P>SOLUTION: In the document collection apparatus and the document collection method, a date and time information extraction part calculates a date on which a prescribed document is to be collected next on the basis of the information of a plurality of dates extracted from the prescribed document and the prescribed document is collected on the calculated next collection date. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

The present invention relates to an apparatus and method for automatically collecting documents recorded in other computers via a computer network, and more particularly to an apparatus and method for collecting at a predetermined date and time.

  In recent years, it has been possible to exchange a large amount of electronic documents via a computer network such as the Internet and transmit information to an unspecified number of people.

  In addition, for documents expressed on the network, expressions that make use of their characteristics are used. Among them, a document called HTML on the WWW (World Wide Web) not only describes some information but also a “hyperlink” that can refer to a document written by another person existing on another computer. There is a function.

  This is used when, for example, trusting another document, supplementing information described by the user, and pointing to a document having the same content.

  HTML documents exist in a distributed manner on a plurality of computers, and are defined so that they can be accessed by a notation called “URL (Uniform Resource Locator)” so that each document can be easily referred to (for example, non-patent) Reference 1). Generally, it is represented by the notation "http :: // computer address / directory name /.../ file name".

  There are search services that collect and search for documents distributed on various computers (eg http://www.google.com/, http://www.goo.ne.jp/). In order to realize this search service, a device that periodically collects documents existing on other computers is required.

  From the perspective of effectively using computer resources, it is useless to collect the exact same documents over and over, while maintaining the freshness of information unless documents with changed contents are collected. I can't. In other words, it is necessary to determine the next collection date and time as accurately as possible.

Therefore, a method of estimating the date and time when the content of the document was updated using the result of collecting the document several times and collecting the document at the estimated date and time is known (for example, see Patent Document 1). .
JP 2003-099353 A "Uniform Resource Locator (URL) RFC1738" by T. Berners-Lee, L. Masinter, M. McCahill, December 1994

  In the above conventional example, in order to increase the estimation accuracy of the date and time when the content of the document is updated, the interval of the date and time of collection may be shortened. There is a problem of wasteful collection.

In the present invention, when a document existing on another computer is automatically acquired via a computer network, there is no waste of collecting a document having the same content over and over, and the content of the document is not reduced. It is an object of the present invention to provide a document collection apparatus and a document collection method with high accuracy in estimating the updated date and time.

In the present invention, the date and time information extraction unit calculates the date and time when the predetermined document is collected next time based on the information of a plurality of dates and times extracted from one predetermined document. A document collection apparatus and a document collection method for collecting a predetermined document.

The present invention calculates a date and time when the predetermined document is collected next time based on a plurality of date and time information extracted from one predetermined document, and collects the predetermined document at the calculated next collection date and time. Therefore, wasteful collection can be reduced, and the estimation accuracy of the date and time when the content of the document is updated is high.

  The best mode for carrying out the invention is the following examples.

  FIG. 1 is a block diagram illustrating a schematic configuration of a document collection apparatus 100 that is Embodiment 1 of the present invention.

  The document collection device 100 is a document collection device that automatically acquires a document existing on another computer via a computer network, and includes a collection target management unit 10, a document collection unit 20, and a date / time information extraction unit. 30, a next collection date calculation unit 40, and a management database 50.

  The collection target management unit 10 records the collection target document and the collection date and time.

  The document collection unit 20 collects documents according to the collection target document managed by the collection target management unit 10 and the collection date and time.

  The date information extraction unit 30 analyzes the document collected by the document collection unit 20 and extracts a plurality of date information described in the document.

  The next collection date and time calculation unit 40 calculates the date and time when the predetermined document is collected next time based on the plurality of date and time information extracted by the date and time information extraction unit 30 for one document, Information is sent to the collection target management unit 10.

  By the way, it is possible to describe a plurality of different information in one document (HTML file), and there is a document that simultaneously records the date and time indicating when each of the plurality of described information is described. Exists.

  For example, in an HTML file provided by a newspaper company, a headline of an article and a time indicating when the article was posted are also posted. In addition, in the case of an information sharing service called “Bulletin Board” on the Internet, unspecified / specific large numbers can enter and read information in one HTML file. In this case, the date and time of writing are also posted. doing. In addition, in a service called “diary” (a service in which an individual records and discloses the contents), information is disclosed in units of “what month and day”.

  These date / time information is a record indicating when the date / time is written, and at the same time, the timing for updating is considered. From such a document file, a plurality of dates and times listed in the file are extracted, rearranged in the oldest order, and when the next update is predicted is predicted by calculation.

To extract date / time information, a rule expressing the date / time is set in advance, and a character string that matches the set rule is extracted as the date / time. The following rules can be considered as examples of extraction rules. That means
YYYY / MM / DDhh: mm: ss (1)
YYYY-MM-DDhh: mm: ss (2)
YYYYMMDDhh: mm (3)
YYYY year MM month DD day hh hour mm minute ss second (4)
Yyyy MM month DD day hh hour mm minute ss second (5)
Etc. can be considered as rules.

  YYYY is the year, MM is the month, DD is the day, hh is the hour, mm is the minute, and ss is the second. Chinese numerals may be used. In addition, there are cases where the characters are enclosed in symbols such as parentheses (for example, [hh: mm]), and the month is expressed in alphabets (for example, April is “Apr” or “Apri”). Taking these into account, the extraction rule is made.

  Next, an example of a calculation method for calculating when the next update is performed in the above embodiment will be described.

  FIG. 2 is a diagram illustrating a display example of an HTML file.

  FIG. 3 is a flowchart showing the operation of the above embodiment.

  First, give the simplest example. First, the collection target management unit 10 describes the collection target and the collection timing. A URL is used to specify the collection target. Then, according to the schedule, the document collection unit 20 acquires an HTML file from another computer (S1). And the date information extraction part 30 extracts the date information which corresponds from the acquired HTML according to the rule prepared beforehand (S2).

  The extracted date and time information of the same format is sorted in ascending order (S3). That is, the date and time information extracted from the file is rearranged in the oldest order.

  Then, the next collection date and time economy section 40 determines the date and time for the next document collection using the date and time information that matches the notation rule in the date and time information with the highest number of appearances (S4). That is, it is checked which of the rules (1) to (5) above (or other rules) matches the date / time information appearing in the HTML file, and the most matched rule is obtained. Next, using only the date and time information that matches the most matched rule, it is determined what time to collect the next information.

For the results sorted in the old order, the date and time at which the n-th of the information has been published, and t _n, and the date and time t _n-1, which was last published, on the basis of the now published in the date and time t _n, to collect next time Predict the date and time (the date and time that will be posted next time) t _{n + 1} . In this case, the date and time t _n-1, which was last published, using the difference between the currently published in the date and time t _n, to determine the date and time t _{n + 1} to collect next time. In other words, the date and time t _{n + 1} to be collected next time is
t _{n + 1} = t _n + (t _n −t _n−1 ) + (6)
It is.

Information on the next collection date t _{n + 1} calculated in this way is transmitted to the collection target management unit 10.

Further, the date and time t _{n + 1} to be collected next time may be calculated using the least square method. In other words, in the HTML file, when the i-th information is posted as T _i and the i-th information is updated at a predetermined constant date and time as x _i ,

を満たすｘ_ｉを求めればよい。ここでｘ_ｉが
ｘ_ｉ＝ａ＊ｉ＋ｂ … （８）
という直線で近似できる。

What is necessary is just to obtain x _i satisfying. Here, x _i is x _i = a * i + b (8)
It can be approximated by a straight line.

  The least square method is described in “Tataka Tanaka and Kazumasa Wakimoto,“ Multivariate Statistical Analysis ”, Modern Mathematics, 1983, Chapter 1”.

  In addition, after the actual collection, the collected date / time information may be adjusted depending on whether or not the contents have been changed.

  Next, a specific example of the above embodiment will be described.

  First, the document to be collected and the date and time to be collected are described in advance in the collection target management unit 10. A URL is used to designate a collection target. When specifying the date and time, the following format is used.

YYYY / MM / DDhh: mm: ss (9)
Here, YYYY is the year, MM is the month, DD is the day, hh is the hour, mm is the minute, and ss is the second. The date and time of collection and the document to be collected are described as one pair. For example,
2003/10/22 12:00:14 http://www.aaa.co.jp/ (10)
2003/10/22 12:10:20 http://www.bbb.co.jp/pub/ (11)
2003/10/22 12:13:20 http://www.aaa.co.jp/ (12)
And so on.

  The document collection unit 20 collects the collection targets managed by the collection target management unit 10 according to the collection date and time. Using the above example, when the current time is “2003/10/22 12:00:14”, an attempt is made to acquire an HTML file that matches “http://www.aaa.co.jp”. .

The date / time information extraction unit 30 analyzes the document collected by the document collection unit 20 and extracts date / time information described in the document. For example, if a predetermined document is described as shown in FIG. 2, the date and time in accordance with the common format extracted from here is
1 2003/11/02 12:35:00 (13)
2 2003/11/02 13:00: 00 (14)
3 2003/11/02 13:15:00 (15)
4 2003/11/02 13:25:00 (16)
5 2003/11/02 13:38:00 (17)
It becomes.

  Note that the character string “April” is ignored from the viewpoint of a common format. In other words, “April” in “But this April…” written by “No. 3” speaker “Uu” in FIG. 2 does not indicate the time written by the speaker. In the above embodiment, it is not used when estimating the time when the file is updated next time.

Using the above five dates and times, try to find an approximate line using the method of least squares. Difference of each time,
1 0 (18)
2 25 (19)
3 15 (20)
4 10 (21)
5 13 (22)
(Unit: minutes) In addition, “0” in the above “1 0” means a starting point, and Equation 13−Equation 13 = 0 (that is, 2003/11/02 12: 35: 00-2003 / 11/02 12: 35: 00 = 0).

From this, when an approximate straight line is obtained using the above equations (19), (20), (21), and (22),
t_i = −4.1 × i + 26 (23)
It becomes. As a result, the sixth update date and time is −4.1 × 5 + 26 = 5.5 (24).
It is expected to be 2003/11/02 13:43:30 after a minute.

  Moreover, the said Example can be grasped | ascertained as invention of a program. In other words, in the above embodiment, in the document collection program for automatically acquiring documents existing on other computers via the computer network, the collection target document and the collection date and time are recorded in the collection target management unit. Depending on the target management procedure, the collection target document recorded in the collection target management unit and the collection date and time, the document collection unit collects the document, and the predetermined document collected in the document collection procedure Analyzing the date and time information described in the analyzed document into the date and time information extraction procedure extracted by the date and time information extraction and the date and time information extracted by the date and time information extraction procedure. Based on this, the computer calculates the next collection date calculation procedure for calculating the next collection date and time and sending the calculated next collection date and time information to the collection target management unit. It is an example of a program executed by the.

The program may be recorded on a recording medium such as an FD, CD, DVD, HD, or semiconductor memory.

1 is a block diagram illustrating a schematic configuration of a document collection apparatus 100 that is Embodiment 1 of the present invention. FIG. It is a figure which shows the example of a display of an HTML file. It is a flowchart which shows operation | movement of the said Example.

Explanation of symbols

100: Document collection device,
10 ... Collection target management department,
20 ... Document collection department,
30 ... date and time information extraction unit,
40 ... Next collection date calculation part,
50 ... management database,
60: Document database.

Claims (3)

In a document collection device that automatically acquires a document existing on another computer via a computer network,
A collection target management unit for recording a collection target document and a collection date and time;
A document collection unit that collects documents according to the collection target document recorded by the collection target management unit and the collection date and time;
A date and time information extraction unit that analyzes a predetermined document collected by the document collection unit and extracts information of a plurality of dates and times described in the analyzed predetermined document;
Based on the information of a plurality of dates and times extracted by the date and time information extraction unit, calculates the date and time when the predetermined document is collected next time, and sends the information of the calculated next collection date and time to the collection target management unit A date and time calculator;
A document collecting apparatus comprising: In a document collection method for automatically acquiring a document existing on another computer via a computer network,
A collection target management stage in which the collection target document and the date and time of collection are recorded in the collection target management unit;
A document collection stage in which the document collection unit collects documents according to the collection target document and the collection date and time recorded in the collection target management unit;
A date and time information extraction stage in which the predetermined document collected in the document collection stage is analyzed, and date and time information extraction extracts a plurality of date and time information described in the analyzed predetermined document;
Next time based on the plurality of date and time information extracted in the date and time information extraction stage, calculates the date and time when the predetermined document is collected next time, and sends the calculated next collection date and time information to the collection target management unit Collection date calculation stage;
A document collection method characterized by comprising: In a document collection program for automatically acquiring a document existing on another computer via a computer network,
A collection target management procedure for recording a collection target document and a collection date and time in a collection target management unit;
A document collection procedure in which the document collection unit collects documents according to the collection target document and the collection date and time recorded in the collection target management unit;
A date and time information extraction procedure for analyzing a predetermined document collected by the document collection procedure, and extracting a plurality of date and time information described in the analyzed predetermined document by date and time information extraction;
Next time based on the plurality of date and time information extracted in the date and time information extraction procedure, calculates the date and time when the predetermined document is collected next time, and sends the calculated next collection date and time information to the collection target management unit Collection date calculation procedure;
A program that causes a computer to execute.

Images