CN105550255A - Resource searching and scheduling method and device - Google Patents

Abstract

The present invention provides a method for obtaining the current main body link of the index page to be scheduled; comparing the obtained current main body link with the historical main body link of the index page to be scheduled; Perform page-turning scheduling until it is determined that there is no connection omission, then execute the subsequent scheduling operation; if it is determined that there is no link omission according to the comparison result, perform the subsequent scheduling operation. It can avoid the phenomenon of missing links in the process of resource search and scheduling, and improve the coverage of resource collection.

Description

Method and device for resource search and scheduling

technical field

The invention relates to the technical field of data search, in particular to a resource search scheduling method and device.

Background technique

In the network data search technology, the Spider system is located at the most upstream of the search engine data flow, and is responsible for collecting resources on the Internet to the local area and providing them for subsequent retrieval. It is one of the most important data sources of search engines. The goal of the spider system is to discover and crawl all valuable webpages on the Internet. To achieve this goal, the first thing is to find links to valuable webpages. The current spider system has a certain scheduling mechanism to discover resource links as quickly and completely as possible.

For example: when scheduling resource links, the following mechanisms can be set:

Mechanism 1: Schedule the excavated seeds according to a certain period (for example, 20 times a day), so as to cover all time-sensitive web pages.

Mechanism 2: Considering limited traffic and a large number of index pages, schedule general index pages (not within the scope of the seed collection) according to a certain period (for example, recrawl once a week).

The above scheduling mechanism has at least the following disadvantages:

For mechanism 1, when the seed scheduling cycle interval is short, there is generally no problem of missing links, but there may be a waste of traffic, that is, when the sampling point is not on time, it is a waste of traffic; when the seed scheduling cycle interval is long, there may be a waste of traffic. There is a leaky chain.

For mechanism 2, due to the long interval between scheduling cycles, there may be leaky chains.

The occurrence of missing links in the scheduling process will reduce the collection coverage of the Spider system.

Contents of the invention

In view of the above problems, the present invention is proposed to provide a resource search scheduling method and a corresponding resource search scheduling device that overcome the above problems or at least partially solve the above problems, which can avoid the phenomenon of missing links in the resource search scheduling process and improve resource collection. coverage.

The present invention provides a method for resource search and scheduling, including:

Obtain the current body link of the index page to be scheduled;

comparing the acquired current subject link with the historical subject link of the index page to be scheduled;

If it is determined that there is a link omission according to the comparison result, perform page-turning scheduling on the index page to be scheduled until it is determined that there is no connection omission, and then perform subsequent scheduling operations;

If it is determined according to the comparison result that there is no link omission, a subsequent scheduling operation is performed.

In some optional embodiments, if it is determined according to the comparison result that there is a link omission, the index page to be scheduled is page-turned until it is determined that there is no link omission, specifically including:

When there is no intersection between the current subject link and the historical subject link, obtain the current subject link of the next page of the to-be-scheduled index page, and return the execution history of the to-be-acquired current subject link and the to-be-scheduled index page The step of comparing subject links until the intersection of the current subject link and the historical subject link exists.

In some optional embodiments, the obtaining the current subject link of the index page to be scheduled specifically includes:

Determine the largest similar block in the index page to be scheduled to obtain the current body link of the index page to be scheduled.

In some optional embodiments, determining the largest similar block in the to-be-scheduled index page specifically includes:

Get the nodes with the same xpath in the Extensible Markup Language XML path to get similar blocks;

Determine the largest similar block in the to-be-scheduled index page according to the position and area of the similar block.

In some optional embodiments, the position of the similar block is determined according to the width, height, top margin, and left margin of the similar block in the scheduling index page;

The area of the similar block is determined according to the width and height of the similar block in the scheduling index page.

In some optional embodiments, the largest similar block in the to-be-scheduled index page is determined according to the position and area of the similar block, specifically including:

Determine the similar block with the largest area and containing the center point of the page among the similar blocks as the largest similar block.

In some optional embodiments, before performing paging scheduling on the index page to be scheduled, it also includes:

Obtain the page turning block in the index page to be scheduled, and determine the URL of the next page link by matching the anchor auchor of the link, so as to perform page turning and crawling according to the URL.

In some optional embodiments, obtaining the page turning block in the index page to be scheduled specifically includes:

By regularly matching the information contained in the linked auchor, the page-turning block in the to-be-scheduled index page is determined, and the determined page-turning block is obtained.

In some optional embodiments, if the auchor of the regular matching link includes the feature information of the page turning block, the page turning block in the index page to be scheduled is determined based on the page turning block information.

In some optional embodiments, the determining the Uniform Resource Locator URL of the next page link by matching the anchor auchor of the link specifically includes:

When the auchor of the regular matching link contains the next page link information, the link corresponding to the auchor is determined as the URL of the next page link.

In some optional embodiments, before comparing the acquired current subject link with the historical subject link of the to-be-scheduled index page, it further includes:

Sequentially extract time series according to the release time information included in each link in the current subject link;

According to the release time information of the adjacent links, it is judged whether the extracted time series has sequence or reverse sequence characteristics;

When not available, sort the links in the current subject links in chronological order or in reverse chronological order.

An embodiment of the present invention also provides a resource search scheduling device, including:

An acquisition module, configured to acquire the current body link of the index page to be scheduled;

A comparison module, configured to compare the obtained current subject link with the historical subject link of the index page to be scheduled;

The execution module is used to perform page-turning scheduling on the to-be-scheduled index page if it is determined that there is a link omission according to the comparison result, until it is determined that there is no connection omission, and then perform subsequent scheduling operations; if it is determined that there is no link omission according to the comparison result, perform subsequent scheduling operations .

In some optional embodiments, the execution module is specifically configured to:

When there is no intersection between the current subject link and the historical subject link, notify the acquisition module to obtain the current subject link of the next page of the index page to be scheduled, and the comparison module returns to execute the current subject link to be acquired and The step of comparing the historical main body links of the to-be-scheduled index page until the current main body link overlaps with the historical main body links is followed by a subsequent scheduling operation.

In some optional embodiments, the acquisition module is specifically used to:

Determine the largest similar block in the index page to be scheduled to obtain the current body link of the index page to be scheduled.

In some optional embodiments, the acquisition module is specifically used to:

Get the nodes with the same xpath in the Extensible Markup Language XML path to get similar blocks;

Determine the largest similar block in the to-be-scheduled index page according to the position and area of the similar block.

In some optional embodiments, the acquisition module is specifically used to:

Determine the position of the similar block according to the width, height, top margin, and left margin of the similar block in the scheduling index page, and determine the area of the similar block according to the width and height of the similar block in the scheduling index page.

In some optional embodiments, the acquisition module is specifically used to:

Determine the similar block with the largest area and containing the center point of the page among the similar blocks as the largest similar block.

In some optional embodiments, the acquisition module is also used for:

Obtain the page turning block in the index page to be scheduled, and determine the URL of the next page link by matching the anchor auchor of the link, so as to perform page turning and crawling according to the URL.

In some optional embodiments, the acquisition module is specifically used to:

By regularly matching the information contained in the linked auchor, the page-turning block in the to-be-scheduled index page is determined, and the determined page-turning block is obtained.

In some optional embodiments, the acquisition module is specifically used to:

If the auchor of the regular matching link includes the feature information of the page turning block, then determine the page turning block in the to-be-scheduled index page based on the page turning block information.

In some optional embodiments, the acquisition module is specifically used to:

When the auchor of the regular matching link contains the next page link information, the link corresponding to the auchor is determined as the URL of the next page link.

In some optional embodiments, the comparison module is also used for:

Sequentially extract time series according to the release time information included in each link in the current subject link;

According to the release time information of the adjacent links, it is judged whether the extracted time series has sequence or reverse sequence characteristics;

When not available, sort the links in the current subject links in chronological order or in reverse chronological order.

The resource search scheduling method and device of the present invention obtain the current subject link of the index page to be scheduled before performing the scheduling operation; compare the obtained current subject link with the historical subject link of the index page to be scheduled; to determine whether there is a link omission , when it is determined that there is no missing connection, the subsequent scheduling operation is performed, thereby avoiding the possibility of missing links in the resource scheduling process and improving the collection coverage of scheduling resources. This method does not need to be realized by shortening the scheduling cycle, and does not It will cause an increase in traffic overhead, and can avoid the phenomenon of missing links in the resource scheduling process while saving network traffic resources.

Furthermore, the above method of the present invention obtains the current main link of the scheduling index page page by page before scheduling, and compares them page by page, thereby avoiding the occurrence of missing links as much as possible.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Those skilled in the art will be more aware of the above and other objects, advantages and features of the present invention according to the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same components. In the attached picture:

FIG. 1 is a flowchart of a method for resource search and scheduling in Embodiment 1 of the present invention;

Fig. 2 is a screenshot of an index page in Embodiment 1 of the present invention;

FIG. 3 is a flowchart of a method for resource search and scheduling in Embodiment 2 of the present invention;

FIG. 4 is an example diagram of determining the largest similar block in an index page to be scheduled in Embodiment 2 of the present invention;

FIG. 5 is a flow chart of a method for resource search and scheduling in Embodiment 3 of the present invention;

Fig. 6 is a screenshot of another index page in Embodiment 3 of the present invention;

FIG. 7 is a flow chart of a method for resource search and scheduling in Embodiment 4 of the present invention;

Fig. 8 is a schematic structural diagram of a resource search scheduling device in an embodiment of the present invention.

detailed description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

In order to solve the problem of missing links in resource scheduling in the prior art, the embodiment of the present invention provides a resource search and scheduling method, which can avoid the phenomenon of missing links as much as possible, without increasing traffic overhead, and can improve the collection coverage of scheduling resources . The following will be described in detail through specific examples.

Embodiment one:

The resource search and scheduling method provided by Embodiment 1 of the present invention has a process as shown in FIG. 1 , including the following steps:

Step S101: Obtain the current subject link of the index page to be scheduled.

For each scheduling of the index page, parse the index page web page extraction and record the main link it finds. Specifically, for the index page to be scheduled, the current body link of the index page to be scheduled is obtained by determining the largest similar block in the index page to be scheduled.

Wherein, the index page refers to a web page whose main body is a link rather than content text. The subject link refers to a set of links corresponding to the subject on the index page webpage. For example, Figure 2 shows a screenshot of an index page. The main link in the index page http://roll.news.sina.com.cn/news/gnxw/gdxw1/index.shtml is the big box in Figure 2 , which includes links to the topics on this index page.

Step S102: Compare the acquired current subject link with the historical subject link of the index page to be scheduled.

After obtaining the current subject link of the index page to be scheduled, obtain the historical subject link of the scheduled index page, and compare the two to determine whether there is a missing link in the current subject link.

Optionally at this time, the index page series historical body link collection can also be acquired at one time. An index page and a series of corresponding turning pages are called an index page series, and the index page series historical subject link collection may include the sum of subject links extracted during index page series scheduling.

Step S103: Determine whether there is a missing link according to the comparison result. If yes, execute step S104; if no, execute step S105.

According to the comparison result of the current subject link of the to-be-scheduled index page and the historical subject link, it can be determined whether there is a missing link on the to-be-scheduled index page. For example, when the current main body link of the index page to be scheduled is completely different from the historical main body link, it proves that there is a leaky link between the two schedulings.

Step S104: perform page turning scheduling on the index page to be scheduled until it is determined that there is no missing connection, then execute step S105.

If it is determined that there is a link omission according to the comparison result, perform page-turning scheduling on the index page to be scheduled until it is determined that there is no link omission, and then perform subsequent scheduling operations.

When it is determined that there is a missing link on the index page to be scheduled, page-turning scheduling is performed on the index page to be scheduled, and the next page is scheduled to find the missing link. If there is still a missing link, continue to perform paging until it is determined that there is no connection missing.

Step S105: Execute subsequent scheduling operations.

When it is determined according to the comparison result that there is no link omission in the index page to be scheduled, a subsequent scheduling operation is performed. Thus, the phenomenon of missing links is avoided, and compared with the practice of shortening the scheduling cycle to reduce missing links in the prior art, this method can better avoid missing links without increasing the usage of network traffic. It can save system traffic resources.

Embodiment two:

The resource search and scheduling method provided by Embodiment 2 of the present invention has a process as shown in FIG. 3 and includes the following steps:

Step S201: Obtain the current subject link of the index page to be scheduled.

For the index page to be scheduled, the current subject link of the index page to be scheduled is obtained by determining the largest similar block in the index page to be scheduled, wherein the determination of the largest similar block in the index page to be scheduled specifically includes:

Get the extensible markup language (eXtensibleMarkupLanguage, XML) path, referred to as XPath, the same node, get the similar block; according to the position and area of the similar block, determine the largest similar block in the index page to be scheduled.

Optionally, the position of the similar block is determined according to the width, height, top margin, and left margin of the similar block in the scheduling index page; the area of the similar block is determined according to the width and height of the similar block in the scheduling index page.

Optionally, determining the largest similar block in the index page to be scheduled according to the location and area of the similar block, specifically includes: determining the largest similar block among the similar blocks that includes the center point of the page as the largest similar block.

For example: a collection of nodes with the same Xpath (the number of nodes > 4) is called a similar block; calculate the position information of similar blocks, that is, width, height, top margin, and left margin in the page, and these four items of information can determine the similarity The position of the block in the web page, and colleagues calculate the area of similar blocks based on information such as width and height. The similar block with the largest area and containing the center point is called the largest similar block.

As shown in FIG. 4 , it is an example of determining the largest similar block in the index page to be scheduled. For the similar blocks on the index page http://roll.news.sina.com.cn/news/gnxw/gdxw1/index.shtml, the three thick-lined boxes in Figure 4 circled three similar blocks, which is very Obviously the box circled at the bottom is the largest similar block.

The link in the most similar block is the main link, that is, the link corresponding to each news in the bottom box is the main link, such as:

After the Yangtze River sunken ship is righted, the next step is to improve the drainage as a whole:

http://news.sina.com.cn/c/2015-06-05/133131917716.shtml;

Many hospitals in Guangzhou are piloting one-day return home: surgery during the day and home at night:

http://news.sina.com.cn/c/2015-06-05/125531917706.shtml;

...

Step S202: Compare the obtained current subject link with the historical subject link of the index page to be scheduled.

According to the comparison result of the current subject link of the to-be-scheduled index page and the historical subject link, it is determined whether there is a missing link on the to-be-scheduled index page.

Step S203: Determine whether there is an intersection between the obtained current subject link and the historical subject link. If yes, execute step S205; if not, execute step S204.

According to whether there is an intersection between the current subject link and the historical subject link of the index page to be scheduled, it can be determined whether the two are the same.

Step S204: Obtain the current subject link of the next page of the to-be-scheduled index page. And return to continue to execute step S202.

When there is no intersection between the obtained current subject link and the historical subject link, obtain the current subject link of the next page of the index page to be scheduled, and return to the step of comparing the acquired current subject link with the historical subject link of the to-be-scheduled index page , until there is an intersection between the current subject link and the historical subject link.

As described in the above step S202-step S204, each time the index page to be scheduled is scheduled, compare the currently found subject link with the subject link found in the previous scheduling, if there is no intersection, that is, the subject link found in the last scheduling and the currently found The main links of the two schedules are completely different, which means that there is a missing link between the two schedules. It is necessary to schedule the page to turn the next page (ie page 2) to find the missing link, and add the currently found main link to the index page series In the collection of historical subject links. When scheduling the next page, similarly, compare the currently discovered subject link with the historical subject link set of the index page series. If there is no intersection, it means that there is still a missing link, and continue to initiate page turning to the next page (that is, the third page) page) scheduling. By analogy, until there is an intersection between the currently discovered main body link and the historical main body link set of the index page series, it means that the missing links have been found, and there is no need to initiate the scheduling of page turning.

Step S205: Execute subsequent scheduling operations.

When it is determined according to the comparison result that there is no link omission in the index page to be scheduled, a subsequent scheduling operation is performed.

The resource search and scheduling method provided by Embodiment 3 of the present invention has a process as shown in FIG. 5, including the following steps:

Step S301: Obtain the current subject link of the index page to be scheduled.

Step S302: Compare the obtained current subject link with the historical subject link of the index page to be scheduled.

Step S303: Determine whether there is an intersection between the obtained current subject link and the historical subject link. If yes, execute step S307; if not, execute step S304.

Step S304: Obtain the page turning block in the index page to be scheduled.

In this embodiment, on the basis of Embodiment 1 and Embodiment 2, before obtaining the current body link of the next page of the index page to be scheduled, the page turning block in the index page to be scheduled is obtained, so as to facilitate page turning scheduling.

Acquiring the page-turning block in the index page to be scheduled specifically includes: determining the page-turning block in the index page to be scheduled by matching the information contained in the anchor (auchor) of the regularization, and obtaining the determined page-turning block.

If the auchor of the regular matching link includes the feature information of the page turning block, the page turning block in the to-be-scheduled index page is determined based on the page turning block information.

For example, what is shown in the upper rectangular box in FIG. 2 is the page turning block. What is shown in the rectangular box in the middle of Fig. 4 is also a page turning block. The page turning blocks shown in Fig. 4 can also be extracted from similar blocks on the http://roll.news.sina.com.cn/news/gnxw/gdxw1/index.shtml web page.

Optionally, it can be determined whether it is a page-turning block by regularly matching the anchor of the link. For example, it can be determined as a page-turning block when it contains at least one of the following feature information of the page-turning block: "number", "<", " >", "<<", ">>", "Previous Page", "Next Page", "First Page", "Last Page" and other keywords.

Step S305: Determine the uniform resource locator (UniformResoureLocatorURL) of the link on the next page by matching the anchor auchor of the link in the regular pattern, so as to perform page flipping and crawling according to the determined URL.

When the auchor of the regular matching link contains the link information of the next page, the link corresponding to the auchor is determined as the URL of the next page link, so as to perform page-turning crawling according to the determined URL.

When determining the link to the next page, for the node in the page turning block, judge whether it matches the link information of the next page by matching the anchor of the link regularly, where the link information of the next page includes at least one of the following information: "specified number ", "next page", "next page", ">" and other keywords. If the link information of the next page is matched, record the link corresponding to the anchor as the URL of the next page link; otherwise, calculate the number of the next page based on the current page number, and then splicing out the URL of the next page link .

As shown in Figure 4, the anchor of the node in the page-turning block matches the keyword "next page", and the link to the next page is the link http://roll.news.sina whose anchor is "next page". com.cn/news/gnxw/gdxw1/index_2.shtml.

FIG. 6 is a partial screenshot of a webpage, which includes an example of a page turning block, such as the lower rectangular box in FIG. 6 is a page turning block. When the node anchor in the page turning block matches the link information of the next page, "next page", "next page", ">" and so on cannot be matched, so the number of next pages is calculated (the current page is the index page , that is, turn the first page, then the number of pages on the next page is 2). Find the link of the node anchor matching the number 2 from the page turning block, here is http://gold.jrj.com.cn/list/hjzx-2.shtml.

Step S306: Acquiring the current subject link of the next page of the to-be-scheduled index page. And return to continue to execute step S302.

When there is no intersection between the obtained current subject link and the historical subject link, obtain the current subject link of the next page of the index page to be scheduled, and return to the step of comparing the acquired current subject link with the historical subject link of the to-be-scheduled index page , until there is an intersection between the current subject link and the historical subject link.

Step S307: Execute subsequent scheduling operations.

For steps not described in detail above, refer to the description of the corresponding steps in Embodiment 1 and Embodiment 2.

Embodiment four:

The resource search and scheduling method provided in Embodiment 4 of the present invention has a process as shown in FIG. 7 , including the following steps:

Step S401: Obtain the current subject link of the index page to be scheduled.

Step S402: According to the release time information contained in each link in the obtained current main body link, sequentially extract the time series.

In this embodiment, before comparing the obtained current main body link with the historical main body link of the index page to be scheduled, determine whether each link in the obtained current main body link has a chronological order or a reverse order arrangement rule, and if not, the They are arranged to facilitate subsequent comparisons of subject links.

As shown in Figure 4, in the big box below, the underlined is the release time corresponding to the subject link, and each subject link has a release time information corresponding to it, based on which the time series can be sequentially extracted.

Step S403: According to the release time information of the adjacent links, judge whether the extracted time series has sequence or reverse sequence features; if yes, perform step S405; if not, perform step S404.

According to the publishing time information of the adjacent links, by judging whether the extracted time series has the characteristics of sequence or reverse order, it is realized whether the subject links are sorted in time order or reverse order.

Step S404: Sort the links in the current subject link in chronological order or in reverse chronological order.

When the extracted time series does not have the characteristics of chronological order or reverse order, it means that the subject links are not sorted in chronological order or in reverse order. In order to facilitate the comparison of subject links, the subject links can be sorted in chronological order or in reverse order.

Step S405: Compare the acquired current subject link with the historical subject link of the index page to be scheduled.

Step S406: Determine whether there is an intersection between the obtained current subject link and the historical subject link. If yes, execute step S408; if no, execute step S407.

Step S407: Acquiring the current subject link of the next page of the to-be-scheduled index page. And return to continue to execute step S402.

When there is no intersection between the obtained current subject link and the historical subject link, obtain the current subject link of the next page of the index page to be scheduled, and return to the step of comparing the acquired current subject link with the historical subject link of the to-be-scheduled index page , until there is an intersection between the current subject link and the historical subject link.

For index pages whose subject links are sorted in chronological order or in reverse order, each time an index page is scheduled, compare the currently discovered subject link and the historical subject link set, if there is no intersection, that is, the subject link found in the last scheduling and the currently discovered subject If the links are completely different, it means that there is a missing link between the two schedules, and the scheduling needs to turn page 2 to find the missing link, and add the currently discovered subject link to the historical subject link collection of the index page series. When scheduling to turn the second page, similarly, compare the currently discovered subject link with the historical subject link set of the index page series. If there is no intersection, it means that there is still a missing link, and continue to initiate the third page scheduling. By analogy, until there is an intersection between the currently discovered main body link and the historical main body link set of the index page series, it means that the missing links have been found, and there is no need to initiate page turning scheduling.

For the index pages whose main body links are sorted in chronological order, since the latest main body links are generally more concerned, it can be considered to schedule from the last page forward, and the mechanism is the same.

Step S408: Execute subsequent scheduling operations.

For the steps not described in detail above, refer to the description of the corresponding steps in Embodiment 1, Embodiment 2 and Embodiment 3.

Compared with the optional steps added in the first and second embodiments in the third and fourth embodiments above, a part of the steps described in each embodiment can be added separately as described in the third and fourth embodiments, The steps added in Embodiment 3 and Embodiment 4 can also be added on the basis of Embodiment 1 and Embodiment 2, and this implementation will not be described in detail with a new embodiment.

Based on the same inventive concept, an embodiment of the present invention also provides a device for resource search and scheduling. The structure of the device is shown in FIG. 8 , including: an acquisition module 801 , a comparison module 802 and an execution module 803 .

Obtaining module 801, configured to obtain the current body link of the index page to be scheduled.

The comparing module 802 is configured to compare the acquired current main body link with the historical main body link of the index page to be scheduled.

The execution module 803 is used to perform page-turning scheduling on the index page to be scheduled if it is determined that there is a link omission according to the comparison result, until it is determined that there is no connection omission, and then perform subsequent scheduling operations; if it is determined that there is no link omission according to the comparison result, perform subsequent scheduling operate.

Optionally, the above execution module 803 is specifically used to notify the acquisition module 801 to obtain the current subject link of the next page of the index page to be scheduled when there is no intersection between the acquired current subject link and the historical subject link, and the comparison module 802 returns to execute the A step of comparing the obtained current subject link with the historical subject link of the index page to be scheduled, until the acquired current subject link and the historical subject link overlap, and perform subsequent scheduling operations.

Optionally, the acquisition module 801 is specifically configured to determine the largest similar block in the index page to be scheduled, and obtain the current body link of the index page to be scheduled.

Optionally, the acquisition module 801 is specifically configured to acquire nodes with the same xpath in the Extensible Markup Language XML path to obtain similar blocks; and determine the largest similar block in the index page to be scheduled according to the position and area of the similar blocks.

Optionally, the acquisition module 801 is specifically configured to determine the position of the similar block according to the width, height, top margin, and left margin of the similar block in the scheduling index page, and determine the position of the similar block according to the width, height, and left margin of the similar block in the scheduling index page. The height determines the area of similar blocks.

Optionally, the acquisition module 801 is specifically configured to determine that among the similar blocks, the similar block with the largest area and including the center point of the page is the largest similar block.

Optionally, the above acquisition module 801 is also used to acquire the page turning block in the index page to be scheduled, and determine the Uniform Resource Locator URL of the link on the next page by matching the anchor auchor of the link regularly, so as to perform page turning according to the URL page fetch.

Optionally, the acquisition module 801 is specifically configured to determine the page-turning block in the index page to be scheduled by matching the information contained in the linked auchor in a regular pattern, and acquire the determined page-turning block.

Optionally, the acquisition module 801 is specifically configured to determine the page-turning block in the to-be-scheduled index page based on the page-turning block information if the auchor of the regular matching link includes the feature information of the page-turning block.

Optionally, the acquisition module 801 is specifically configured to determine the link corresponding to the auchor as the URL of the next page link when the auchor of the regular matching link contains the next page link information.

Optionally, the above-mentioned comparison module 802 is also used to sequentially extract the time series according to the release time information included in each link in the current subject link; and to judge whether the extracted time series have sequence or reverse order according to the release time information of adjacent links feature; when not available, sort the links in the acquired current subject links in chronological order or in reverse chronological order.

The resource search scheduling method and device provided by the embodiments of the present invention is a scheduling mechanism for a specific index page, realizes scheduling based on leaky link judgment, and improves resource search collection coverage at the most economical traffic cost.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, in order to streamline this disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure, or its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. Modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method or method so disclosed may be used in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components in the resource search and scheduling device according to the embodiment of the present invention. The present invention can also be implemented as an apparatus or an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

So far, those skilled in the art should appreciate that, although a number of exemplary embodiments of the present invention have been shown and described in detail herein, without departing from the spirit and scope of the present invention, the disclosed embodiments of the present invention can still be used. Many other variations or modifications consistent with the principles of the invention are directly identified or derived from the content. Accordingly, the scope of the present invention should be understood and deemed to cover all such other variations or modifications.

Based on one aspect of the present invention, A1. A method for resource search and scheduling is also disclosed, including:

Obtain the current body link of the index page to be scheduled;

comparing the acquired current subject link with the historical subject link of the index page to be scheduled;

If it is determined that there is a link omission according to the comparison result, perform page-turning scheduling on the index page to be scheduled until it is determined that there is no connection omission, and then perform subsequent scheduling operations;

If it is determined according to the comparison result that there is no link omission, a subsequent scheduling operation is performed.

A2. The method according to A1, wherein, if it is determined according to the comparison result that there is a link omission, the index page to be scheduled is page-turned until it is determined that there is no link omission, specifically including:

When there is no intersection between the current subject link and the historical subject link, obtain the current subject link of the next page of the to-be-scheduled index page, and return the execution history of the to-be-acquired current subject link and the to-be-scheduled index page The step of comparing subject links until the intersection of the current subject link and the historical subject link exists.

A3. The method according to A1, wherein said obtaining the current subject link of the index page to be scheduled specifically includes:

Determine the largest similar block in the index page to be scheduled to obtain the current body link of the index page to be scheduled.

A4. The method according to A3, wherein determining the largest similar block in the index page to be scheduled includes:

Get the nodes with the same xpath in the Extensible Markup Language XML path to get similar blocks;

Determine the largest similar block in the to-be-scheduled index page according to the position and area of the similar block.

A5. The method according to A4, wherein the position of the similar block is determined according to the width, height, upper margin, and left margin of the similar block in the scheduling index page;

The area of the similar block is determined according to the width and height of the similar block in the scheduling index page.

A6. The method according to A4, wherein, according to the position and area of the similar block, determine the largest similar block in the index page to be scheduled, specifically including:

Determine the similar block with the largest area and containing the center point of the page among the similar blocks as the largest similar block.

A7. The method according to A1, wherein, before performing paging scheduling on the index page to be scheduled, it also includes:

Obtain the page turning block in the index page to be scheduled, and determine the URL of the next page link by matching the anchor auchor of the link, so as to perform page turning and crawling according to the URL.

A8. The method according to A7, wherein obtaining the page turning block in the index page to be scheduled specifically includes:

By regularly matching the information contained in the linked auchor, the page-turning block in the to-be-scheduled index page is determined, and the determined page-turning block is obtained.

A9. The method according to A8, wherein if the auchor of the regular matching link includes the feature information of the page turning block, then determine the page turning block in the index page to be scheduled based on the page turning block information.

A10. The method according to A7, wherein the URL of the next page link is determined by matching the anchor auchor of the regular link, specifically including:

When the auchor of the regular matching link contains the next page link information, the link corresponding to the auchor is determined as the URL of the next page link.

A11. The method according to any one of A1-A10, wherein, before comparing the obtained current subject link with the historical subject link of the index page to be scheduled, it also includes:

Sequentially extract time series according to the release time information included in each link in the current subject link;

According to the release time information of the adjacent links, it is judged whether the extracted time series has sequence or reverse sequence characteristics;

When not available, sort the links in the current subject links in chronological order or in reverse chronological order.

Based on another aspect of the present invention, B12. A resource search scheduling device is also disclosed, comprising:

An acquisition module, configured to acquire the current body link of the index page to be scheduled;

A comparison module, configured to compare the obtained current subject link with the historical subject link of the index page to be scheduled;

The execution module is used to perform page-turning scheduling on the to-be-scheduled index page if it is determined that there is a link omission according to the comparison result, until it is determined that there is no connection omission, and then perform subsequent scheduling operations; if it is determined that there is no link omission according to the comparison result, perform subsequent scheduling operations .

B13. The device according to B12, wherein the execution module is specifically used for:

When there is no intersection between the current subject link and the historical subject link, notify the acquisition module to obtain the current subject link of the next page of the index page to be scheduled, and the comparison module returns to execute the current subject link to be acquired and The step of comparing the historical main body links of the to-be-scheduled index page until the current main body link overlaps with the historical main body links is followed by a subsequent scheduling operation.

B14. The device according to B12, wherein the acquisition module is specifically used for:

Determine the largest similar block in the index page to be scheduled to obtain the current body link of the index page to be scheduled.

B15. The device according to B14, wherein the acquisition module is specifically used for:

Get the nodes with the same xpath in the Extensible Markup Language XML path to get similar blocks;

Determine the largest similar block in the to-be-scheduled index page according to the position and area of the similar block.

B16. The device according to B15, wherein the acquisition module is specifically used for:

Determine the position of the similar block according to the width, height, top margin, and left margin of the similar block in the scheduling index page, and determine the area of the similar block according to the width and height of the similar block in the scheduling index page.

B17. The device according to B15, wherein the acquisition module is specifically used for:

Determine the similar block with the largest area and containing the center point of the page among the similar blocks as the largest similar block.

B18. The device according to B12, wherein the acquisition module is also used for:

Obtain the page turning block in the index page to be scheduled, and determine the URL of the next page link by matching the anchor auchor of the link, so as to perform page turning and crawling according to the URL.

B19. The device according to B18, wherein the acquisition module is specifically used for:

By regularly matching the information contained in the linked auchor, the page-turning block in the to-be-scheduled index page is determined, and the determined page-turning block is obtained.

B20. The device according to B19, wherein the acquisition module is specifically used for:

If the auchor of the regular matching link includes the feature information of the page turning block, then determine the page turning block in the to-be-scheduled index page based on the page turning block information.

B21. The device according to B18, wherein the acquisition module is specifically used for:

When the auchor of the regular matching link contains the next page link information, the link corresponding to the auchor is determined as the URL of the next page link.

B22. The device according to any one of B12-B21, wherein the comparison module is also used for:

Sequentially extract time series according to the release time information included in each link in the current subject link;

According to the release time information of the adjacent links, it is judged whether the extracted time series has sequence or reverse sequence characteristics;

When not available, sort the links in the current subject links in chronological order or in reverse chronological order.

Claims (10)

1. a resource searching dispatching method, comprising:

Obtain the current topic link of waiting to dispatch index page;

The described current topic link obtained is linked with described history body of waiting to dispatch index page and compares;

If determine that there is link omits according to comparative result, treat scheduling index page and carry out page turning scheduling, until when determining that there is not connection omits, perform subsequent scheduling operations;

If determine that there is not link omits, and performs subsequent scheduling operations according to comparative result.

2. method according to claim 1, wherein, if determine that there is link omits according to comparative result, treat scheduling index page and carry out page turning scheduling, until when determining that there is not connection omits, specifically comprise:

When the link of described current topic and history body link there is not common factor time, the current topic link of dispatching one page under index page is waited described in acquisition, and return to perform and described with described, the link of the current topic of acquisition is waited that the history body of dispatching index page links the step compared, until described current topic links and history body links and exists when occuring simultaneously.

3. method according to claim 1 and 2, wherein, the described current topic waiting to dispatch index page that obtains links, and specifically comprises:

Wait the maximal phase seemingly block dispatched in index page described in determining, obtain the current topic link of waiting to dispatch index page.

4. according to the arbitrary described method of claim 1-3, wherein, wait the maximal phase seemingly block dispatched in index page described in determining, specifically comprise:

Obtain the node that expandable mark language XML path xpath is identical, obtain similar piece;

According to position and the area of similar piece, described in determining, wait the maximal phase seemingly block dispatched in index page.

5. according to the arbitrary described method of claim 1-4, wherein, the position of described similar piece is according to described similar piece of width in scheduling index pages, highly, top margin, the left side is apart from determining;

The area of described similar piece is being dispatched the width in index pages according to described similar piece, is highly being determined.

6., according to the arbitrary described method of claim 1-5, wherein, according to position and the area of similar piece, wait the maximal phase seemingly block dispatched in index page described in determining, specifically comprise:

Determine that area in similar piece is maximum and comprise similar piece of page central point for maximal phase is like block.

7. according to the arbitrary described method of claim 1-6, wherein, described in treat before scheduling index page carries out page turning scheduling, also comprise:

Wait described in acquisition to dispatch the page turning block in index page, by the anchor auchor of canonical coupling link, determine the uniform resource position mark URL that lower one page links, to carry out page turning crawl according to described URL.

8. according to the arbitrary described method of claim 1-7, wherein, wait described in acquisition to dispatch the page turning block in index page, specifically comprise:

By the information that the auchor of canonical coupling link comprises, wait described in determining to dispatch the page turning block in index page, obtain the page turning block determined.

9. according to the arbitrary described method of claim 1-8, wherein, if the auchor of described canonical coupling link comprises page turning block feature information, then wait described in determining based on described page turning block message to dispatch the page turning block in index page.

10. a resource searching dispatching device, comprising:

Acquisition module, for obtaining the current topic link of waiting to dispatch index page;

Comparison module, compares for the described current topic link obtained being linked with described history body of waiting to dispatch index page;

Execution module, if for determining that according to comparative result there is link omits, treating scheduling index page and carrying out page turning scheduling, until when determining that there is not connection omits, perform subsequent scheduling operations; If determine that there is not link omits, and performs subsequent scheduling operations according to comparative result.