JP2009258923A - Information space search apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To specify a collection condition and collect webpages matching the condition from an information space in a short time. <P>SOLUTION: An information space search apparatus has a collection page discrimination rule creation means for creating a collection page discrimination rule determinative of whether or not to collect reached pages, and a termination condition setting means for setting a termination condition of collection. The process of collecting and storing in a webpage storage means webpages according to a source URL list, the termination condition and the collection page discrimination rule is repeated until satisfying the termination condition. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an information space search apparatus and program, and more particularly to an information space search apparatus and program for efficiently collecting Web pages according to the purpose in an information space having a network structure.

Regarding the network structure type information space search method, in addition to the method of collecting all designated points, by repeating the process of sequentially accessing the information resource group linked from the designated starting point, There is a method for exhaustively searching a network structure type information space (see, for example, Patent Document 1).
Japanese Patent No. 322889

  However, information spaces represented by the WWW contain enormous resources. When the information in the WWW is collected using the method of the above-mentioned Patent Document 1, all linked pages are collected, so that many items other than the page on which necessary information is posted are also collected.

  As a result, it takes a lot of time to collect, and there is a problem that information update of a page on which necessary information is posted is delayed.

  The present invention has been made in view of the above points, and an object of the present invention is to provide an information space search apparatus and program capable of collecting Web pages in an information space in a short time.

  FIG. 1 is a principle configuration diagram of the present invention.

The present invention (Claim 1) is an information space search apparatus for collecting Web pages from an information space having a network structure,
Starting point list storage means 101 for storing a starting point (starting point URL list) representing a collection start point;
A rule storage means 102 for storing a collection page discrimination rule that is a criterion for determining whether or not to collect the reached page;
Web page storage means 103 for storing the collected Web pages;
A collection page discrimination rule creating unit 104 that creates a collection page discrimination rule and stores it in the rule storage unit 102;
An end condition setting means 105 for setting an end condition at the time of collection;
A web page that collects web pages based on the origin URL list in the origin list storage unit 101, the end condition, and the collection page discrimination rule in the rule storage unit 102 and repeats the process of storing in the web page storage unit 103 until the end condition is reached. Collecting means 106.

Further, the present invention (Claim 2) is the Web page collecting means 106,
When the origin URL list is acquired from the origin list storage unit 101, the Web page of the URL of the origin URL list is acquired, and the Web page corresponds to the collection page determination rule of the rule storage unit 104, the Web page and Means for acquiring the URL of the link destination of the Web page;
Means for acquiring the first web page of the link destination URL and collecting the web page if the web page corresponds to the collection page discrimination rule.

Further, according to the present invention (claim 3), the collected page discrimination rule creating means 104 uses the collected page discrimination rule as
-For URL, similar notation URL, same domain,
・ For words used in Web documents, the presence or absence of specific keywords, bias of used words,
・ Set any or all of the contents of alt and the number of img tags for tags.

Further, according to the present invention (Claim 4), in the end condition setting means 105, as the end condition,
・ Based on the starting point;
・ Based on time;
・ Based on data volume;
The present invention (Claim 5) is an information space search program for causing a computer to function as each means constituting the information space search apparatus according to any one of Claims 1 to 4. .

  As described above, the present invention collects only a specific page based on a created rule, so that the collection can be completed in a shorter time than the conventional method. It becomes easy to hold the latest information about the page.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 2 shows a configuration of an information space search apparatus according to an embodiment of the present invention.

  The information space search apparatus shown in the figure includes a starting point list storage unit 101 that stores a starting point (starting point URL list) that represents a collection start point, and a rule that stores a rule that is a criterion for determining whether or not to collect a reached page. Storage unit 102, Web page storage unit 103 that stores collected Web pages, collection page discrimination rule creation unit 104 that creates a collection page discrimination rule, end condition setting unit 105 that sets an end condition for collection, and origin list The web page collection unit 106 collects Web pages based on the end condition and the discrimination rule, and the index assignment unit 107 assigns an index serving as a clue to the collected pages.

  The operation in the above configuration will be described below.

  FIG. 3 is a flowchart of an outline operation in one embodiment of the present invention.

  Step 100) First, a starting URL list that is a starting point of collection is set in the starting list storage unit 101. The operation is specified from the outside. The starting point may be a single point or a plurality of lists as shown in FIG. Specific examples of the method of creating the starting point include “URL list of search results output by inputting a query to a search engine” and “URLs published in books”.

Step 200) The end condition setting unit 105 sets an end condition for collecting information. The end condition is
・ Based on the starting point;
・ Based on time;
・ Based on data volume;
Etc. Specifically, when based on the starting point, “collect the starting URL list and n-hop linked pages from each page”, those based on time “collect 30 minutes from the start”, and those based on the data amount “ Setting conditions such as “collect until collected data reaches 1 GB” can be mentioned.

Step 300) The collection page discrimination rule creation unit 104 creates a rule for discriminating whether or not the reached page corresponds to the collection target. As a rule application target,
・ URL;
-Words used in web documents;
·tag;
Etc.

  Specific examples of rules include a similar notation URL and the same domain for “URL”, the presence / absence of a specific keyword and the bias of used words for “words used in a Web document”, “ For “tag”, the alt description content of the A tag and the number of img tags can be considered.

  Step 400) The web page collection unit 106 collects web pages based on the above starting point (starting point list), end condition, and collected page discrimination rule. The collected pages are transferred to the index creation unit 107.

  Step 500) The index assigning unit 107 assigns an index serving as a clue to search the data transmitted from the Web page collecting unit 106 and stores the index in the Web page storage unit 103.

  Next, a specific operation of the web page collection unit 106 in step 400 will be described.

  FIG. 5 is a sequence chart of detailed operations of the Web page collection unit according to the embodiment of the present invention. FIG. 6 shows an example of a start point list, end condition, and collection determination rule in one embodiment of the present invention.

  First, the web page collection unit 106 reads the origin URL list (FIG. 6A) stored in the origin list storage unit 101 (step 401), and acquires the first page (123.com/) (steps). 402), by referring to the rule storage unit 102, it is checked whether or not the page corresponds to a collected page discrimination rule (a page on which the word “XXX” is described) (step 403). If applicable (step 403, Yes), the page is collected (step 404), and all link destination URLs in the page are acquired (step 405). If not applicable, the page may or may not be collected depending on the purpose.

  For the acquired link destination page (step 406), the collected page discrimination rule storage unit 102 is similarly referenced to check whether or not the collected page discrimination rule is met (step 407). When it corresponds (step 407, Yes), a page is collected and it transfers to step 411 (step 408).

  On the other hand, in the above-described step 403, if the collection determination rule is not met, the collection page determination rule in FIG. 6C (link destinations of pages not including the word “xxx” are not collected) Accordingly, it is determined whether or not there is a next URL of the starting point (starting point list) (step 409). If there is (step 409, Yes), the process moves to the next URL list of the starting point URL list and proceeds to step 403 ( Step 410). If there is no next URL (step 409, No), the collection process is terminated.

  In addition, if it does not correspond to the collection determination rule in the upper step 407, or after collecting the page in step 408, it is determined whether there is another linked page (step 411), and if there is, move to the next URL of the link destination (Step 413). If not, the process proceeds to step 409 to determine whether there is a next URL in the URL list.

  The same processing is performed for all the linked pages, and when the check is completed for all the linked pages, the process returns to the starting point (starting list) and the collection determination rule is applied to the next URL.

  The operations of the components of the information space search device described above can be constructed as a program, installed in a computer used as the information space search device, executed, or distributed via a network.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention is applicable to a technique for collecting Web pages.

It is a principle block diagram of this invention. It is a block diagram of the information space search apparatus in one embodiment of this invention. It is a flowchart of the outline | summary operation | movement in one embodiment of this invention. It is an example of the starting point list in one embodiment of the present invention. It is a flowchart of detailed operation | movement of the web page collection part in one embodiment of this invention. It is an example of a starting point list, an end condition, and a collection determination rule in an embodiment of the present invention.

Explanation of symbols

101 starting point list storage unit, starting point list storage unit 102 rule storage unit, rule storage unit 103 Web page storage unit, Web page storage unit 104 collection page discrimination rule creation unit, collection page discrimination rule creation unit 105 end condition setting unit, end condition Setting unit 106 Web page collection means, Web page collection unit 107 Indexing unit

Claims (5)

An information space search device that collects web pages from an information space having a network structure,
Starting point list storage means for storing a starting point (starting point URL list) representing a collection start point;
Rule storage means for storing a collection page discrimination rule that is a criterion for determining whether or not to collect the reached page;
Web page storage means for storing the collected Web pages;
A collection page discrimination rule creating means for creating the collected page discrimination rule and storing it in the rule storage means;
An end condition setting means for setting an end condition when collecting,
A process of collecting Web pages based on the starting URL list of the starting list storage unit, the end condition, and the collected page determination rule of the rule storage unit and storing the Web page in the Web page storage unit until the end condition is reached. Repeating web page collection means;
An information space search device characterized by comprising: The web page collection means includes:
When the origin URL list is acquired from the origin list storage unit, the Web page of the URL of the origin URL list is acquired, and the Web page corresponds to the collected page determination rule of the rule storage unit, the Web Means for obtaining the URL of the page and the link destination of the web page;
Means for acquiring the first web page of the link destination URL and collecting the web page when the collected page discrimination rule is satisfied;
The information space search device according to claim 1, comprising: The discrimination rule creating means is a collection page discrimination rule,
-For URL, similar notation URL, same domain,
・ For words used in Web documents, the presence or absence of specific keywords, bias of used words,
The information space search device according to claim 1 or 2, wherein any one or all of the contents described in alt and the number of img tags are set for the tag. The end condition setting means includes the end condition as
・ Based on the starting point;
・ Based on time;
・ Based on data volume;
The information space search device according to claim 1, wherein any one of the above is set.   The information space search program for functioning a computer as each means which comprises the information space search apparatus of any one of Claims 1 thru | or 4.

Images