WO2015103997A1 - 一种基于关键词检索的网络爬虫调度方法及系统 - Google Patents

一种基于关键词检索的网络爬虫调度方法及系统 Download PDF

Info

Publication number
WO2015103997A1
WO2015103997A1 PCT/CN2015/070475 CN2015070475W WO2015103997A1 WO 2015103997 A1 WO2015103997 A1 WO 2015103997A1 CN 2015070475 W CN2015070475 W CN 2015070475W WO 2015103997 A1 WO2015103997 A1 WO 2015103997A1
Authority
WO
WIPO (PCT)
Prior art keywords
bucket
link address
page
task
keyword
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2015/070475
Other languages
English (en)
French (fr)
Inventor
廖耀华
黎小为
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Jingdong Shangke Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to US15/110,564 priority Critical patent/US10185771B2/en
Priority to RU2016132573A priority patent/RU2645266C1/ru
Publication of WO2015103997A1 publication Critical patent/WO2015103997A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/951—Indexing; Web crawling techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/162—Delete operations
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues

Definitions

  • the invention relates to the technical field of web crawlers, in particular to a web crawler scheduling method and system based on keyword retrieval.
  • Web crawlers based on keyword retrieval are the basis for obtaining keyword information, but the update of information may lead to repeated crawling or catching of web crawlers.
  • the target website information changes very frequently, such as Sina Weibo and other microblogging websites, Baidu and other search websites.
  • For popular keywords because the information is updated very quickly, it is difficult for the web crawler to capture the data.
  • For the unpopular keywords due to the slower information update, the information will be repeatedly crawled.
  • the current practice is to set different enthusiasm for keywords, and then crawl keywords according to keyword popularity, and hot keywords are more frequently crawled.
  • a web crawler scheduling method based on keyword retrieval comprising:
  • Step 12 The scheduling end receives a task request command from the crawling node.
  • Step 13 The scheduling end acquires the secondary download link address generation task from the priority bucket that saves the secondary download link address, and joins the task list. If the number of priority buckets is allowed to be added to the task list, step 16 is performed; otherwise, step 14 is performed.
  • the secondary download link address is obtained by the crawling node according to the task in the task list, and the crawling page is analyzed, and the link address that needs to be downloaded twice is analyzed from the crawling page;
  • Step 14 The dispatching end acquires a keyword link address from a dynamic bucket that stores a keyword multiple page code link address, and derives a derivative link address of the page number according to the number of page codes corresponding to the keyword link address, according to the number of the page number. Deriving the link address to generate the task of the number of page numbers and adding the task list. If the number of dynamic buckets allowed to join the task list has been reached, step 16 is performed; otherwise, step 15 is performed, and the keyword link address is based on the target website.
  • the link result address of the search result page generated by the keyword, and the search result page corresponding to the keyword link address in the dynamic bucket includes a page number greater than or equal to a preset page number threshold, and the page number threshold is greater than or equal to 2;
  • Step 15 The dispatching end acquires a keyword link address from the basic bucket that stores the keyword link address to generate a task and joins the task list. If the number of basic buckets is allowed to be added to the task list, step 16 is performed, and the keyword link address is a search result page corresponding to the keyword link address generated in the target bucket, and a search result page corresponding to the keyword link address in the basic bucket includes a page number smaller than a preset page number threshold, and the page number threshold is greater than Or equal to 2;
  • Step 16 The dispatcher returns a task list to the crawling node, and the crawling node executes the task in the task list according to the received task list.
  • a web crawler scheduling system based on keyword retrieval comprising: a dispatching end and at least one crawling node communicating with the dispatching end;
  • the dispatching end includes:
  • a task request command receiving module configured to receive a task request command sent by the crawling node
  • the priority bucket task generation module is configured to acquire a secondary download link address generation task from the priority bucket that saves the secondary download link address and join the task list, if the task list has been reached If the priority bucket number is allowed to be added, the task list is returned to the module, otherwise the dynamic bucket task generation module is executed, and the secondary download link address is obtained by the crawling node according to the task in the task list, and the crawling page is obtained from the crawling node.
  • the crawling page analyzes the link address that needs to be downloaded twice;
  • a dynamic bucket task generating module configured to store a dynamic bucket of a keyword multi-page code link address to obtain a keyword link address, and derive a derivative link address of the page number according to a page number corresponding to the keyword link address, according to the page number
  • the number of derived link addresses generates the task of the number of page numbers and joins the task list. If the number of dynamic buckets allowed to join the task list is reached, the task list return module is executed, otherwise the dynamic bucket task generation module is executed, and the keyword link address is executed.
  • a search result page corresponding to the keyword search link generated in the target website, and a search result page corresponding to the keyword link address in the dynamic bucket includes a page number greater than or equal to a preset page number threshold, the page number The quantity threshold is greater than or equal to 2;
  • a basic bucket task generating module configured to acquire a keyword link address generation task from a basic bucket storing a keyword link address and join the task list, and if the task list is allowed to join the basic bucket number, execute a task list return module,
  • the keyword link address is a link address of a search result page generated according to the keyword in the target website, and the search result page corresponding to the keyword link address in the basic bucket includes a page number smaller than a preset page number threshold.
  • the page number threshold is greater than or equal to 2;
  • a task list returning module for returning a task list to the crawling node
  • the crawling node includes:
  • a task request command sending module configured to send a task request command to the dispatching end
  • a task execution module configured to execute a task in the task list according to the received task list.
  • the priority bucket, the dynamic bucket, and the basic bucket are all virtual buckets.
  • virtual buckets By setting virtual buckets with different priorities, different types of link addresses are respectively stored in the virtual bucket, and the higher-priority link addresses are placed in higher priority.
  • Level in the virtual bucket Flexibly adjust the number of calls for different types of link addresses by adjusting the number of allowed tasks for each virtual bucket.
  • the heat of the link address is not set manually, but based on the link address. The actual situation of the page number that is captured is set, thus making it more frequent to grab popular keywords, avoiding data omissions, and reducing repeated crawling of unpopular keywords.
  • FIG. 1 is a flowchart of a network crawler scheduling method based on keyword retrieval according to the present invention
  • FIG. 2 is a schematic block diagram of a web crawler scheduling system based on keyword retrieval according to the present invention
  • FIG. 3 is a schematic structural diagram of a system according to an example of the present invention.
  • FIG. 4 is a working flow chart of an example bucket scheduling management module according to the present invention.
  • FIG. 5 is a flow chart showing the operation of an example bucket updater of the present invention.
  • FIG. 1 is a flowchart of a network crawler scheduling method based on keyword retrieval according to the present invention, including:
  • Step 11 The crawling node sends a task request command to the dispatching end.
  • Step 12 The scheduling end receives the task request command sent by the crawling node.
  • Step 13 The scheduling end acquires the secondary download link address generation task from the priority bucket that saves the secondary download link address, and joins the task list. If the number of priority buckets is allowed to be added to the task list, step 16 is performed; otherwise, step 14 is performed.
  • the secondary download link address is obtained by the crawling node according to the task in the task list, and the crawling page is analyzed, and the link address that needs to be downloaded twice is analyzed from the crawling page;
  • Step 14 The dispatching end acquires a keyword link address from a dynamic bucket that stores a keyword multiple page code link address, and derives a derivative link address of the page number according to the number of page codes corresponding to the keyword link address, according to the number of the page number. Deriving the link address to generate the task of the number of page numbers and adding the task list. If the number of dynamic buckets allowed to join the task list has been reached, step 16 is performed; otherwise, step 15 is performed, and the keyword link address is based on the target website.
  • the link address of the search result page generated by the keyword, and The search result page corresponding to the keyword link address in the dynamic bucket includes a page number greater than or equal to a preset page number threshold, and the page number threshold is greater than or equal to 2;
  • Step 15 The dispatching end acquires a keyword link address from the basic bucket that stores the keyword link address to generate a task and joins the task list. If the number of basic buckets is allowed to be added to the task list, step 16 is performed, and the keyword link address is a search result page corresponding to the keyword link address generated in the target bucket, and a search result page corresponding to the keyword link address in the basic bucket includes a page number smaller than a preset page number threshold, and the page number threshold is greater than Or equal to 2;
  • Step 16 The dispatcher returns a task list to the crawling node, and the crawling node executes the task in the task list according to the received task list.
  • step 13 the processing is specifically performed for the secondary download link address, and the second download of the link address means that the crawling node performs the capture to the crawling page according to the task in the task list, and the analysis needs to be performed from the crawling page.
  • the link address of the second download for example, searching for "Jingdong" in Baidu, a search result page including multiple search results will appear, each search result corresponding to a link address, and the link address corresponding to the search result is two Download link address. From the crawling page, the link address that needs to be downloaded twice is obtained, and there are already mature technologies for analysis, which will not be described here.
  • the keyword link address in steps 14 and 15 refers to the link address of the search result page generated according to the keyword in the target website. For example, if "Jingdong" is searched in Baidu, a key according to the key will appear.
  • the search result page generated by the word "Jingdong” if the keyword is a popular keyword, a search result page does not include all the search results, there will be a page number on the search page.
  • the number of page numbers is used to indicate that the search result page having the number of page numbers includes the keyword. Therefore, the more the number of page numbers, the more popular the keyword is. Therefore, the keyword link address whose page number exceeds the preset page number threshold is placed in the dynamic bucket, and is preferentially processed in step 14. Since it is judged based on the actual number of page numbers, it is possible to more realistically reflect the popularity of keywords.
  • the number of priority buckets allowed to be used is used to indicate the number of secondary download link addresses in the priority bucket allowed to join the task list
  • the number of dynamic buckets allowed to be added is used to indicate the number of keyword link addresses in the dynamic bucket allowed to join the task list.
  • the number of allowed basic buckets is used to indicate the number of keyword link addresses in the basic bucket that are allowed to join the task list.
  • the step 13 specifically includes: acquiring a secondary download link address generation task from the priority bucket holding the secondary download link address and joining the task list, and deleting the secondary download of the generated task from the priority bucket.
  • Link address if the number of priority buckets allowed to join the task list is reached, go to step 16. Otherwise, if there is a secondary download link address in the priority bucket, go to step 13. If all secondary download link addresses in the priority bucket are in the priority bucket If they have been deleted, go to step 14.
  • the secondary download link address is preferentially executed, and the secondary download link address is deleted only once after execution, thereby avoiding repeated crawling.
  • the step 14 specifically includes: acquiring an unscheduled keyword link address from a dynamic bucket storing a keyword link address, and deriving the page number according to the number of page codes corresponding to the keyword link address. Deriving a link address, generating a task of the page number according to the derived link address of the page number, adding a task list, and setting a state of the keyword link address of the generated task to be scheduled, if the task list is allowed to join the dynamic If the number of buckets is the same, go to step 16. Set the status of all the keyword link addresses in the dynamic bucket to unscheduled. Otherwise, if there are unscheduled keyword link addresses in the dynamic bucket, go to step 14. If the bucket is in the dynamic bucket. If the unscheduled keyword link address is not saved, go to step 15.
  • the keyword link address in the dynamic bucket is the link address corresponding to the popular keyword, it is called multiple times. Thereby avoiding the omission of popular keyword crawling.
  • the basic bucket includes an active bucket and a suspended bucket
  • the step 15 specifically includes: obtaining the scheduling time from the active bucket storing the keyword link address.
  • the keyword link address generates a task and joins the task list, and increases the scheduling time of the keyword link address of the generated task by a preset scheduling time increase amount, and then moves to the suspended bucket. If the task list has been allowed to join the basic bucket number Then, step 16 is performed. Otherwise, if the keyword link address is still stored in the activity bucket, step 15 is performed. If the keyword link address is not saved in the activity bucket, step 16 is performed.
  • the basic bucket since the keyword link address in the basic bucket is the link address corresponding to the unpopular keyword, the basic bucket is further set as the active bucket and the suspended bucket, and the link address in the active bucket is moved to Suspend the bucket and execute it after an interval increase of the scheduling time, so as to avoid repeated crawling of the unpopular keyword.
  • the method further includes:
  • the crawling node obtains the crawling page according to the task in the task list, and analyzes the crawling page to obtain analysis data including the secondary download link address, the information details or the page number, and sends the analysis data to the dispatching end;
  • the dispatcher receives the analysis data:
  • the analysis data is a secondary download link address
  • the secondary download link address is placed in the priority bucket
  • analysis data is information details
  • the information details are put into the data bucket
  • the keyword link address corresponding to the number of page numbers is adjusted in the dynamic bucket and the basic bucket.
  • Different analysis data is placed in different virtual buckets to update the link address of the virtual bucket.
  • the scheduling end adjusting the keyword link address corresponding to the number of the page numbers in the dynamic bucket and the basic bucket includes:
  • the number of page numbers corresponding to the corresponding keyword link address is modified to be a new page number
  • the corresponding keyword link address is moved to the dynamic bucket.
  • the keyword link address corresponding to the number of page numbers is adjusted in the dynamic bucket and the basic bucket according to the comparison between the number of old page numbers and the number of new page numbers.
  • the crawling node obtains the crawling page according to the task in the task list, and analyzes the crawling page to obtain analysis data including the secondary download link address, the information details or the page number, and sends the analysis data to the dispatching end;
  • the dispatcher receives the analysis data:
  • the analysis data is a secondary download link address
  • the secondary download link address is placed in the priority bucket
  • analysis data is information details
  • the information details are put into the data bucket
  • the analysis data is the number of page numbers, set the number of page numbers currently received to be the number of new page numbers, and set the number of page numbers received by the same keyword link address as the number of old page numbers, and if the number of old page numbers does not match the number of new page numbers, :
  • the number of page numbers corresponding to the corresponding keyword link address is modified to be a new page number
  • the page number threshold is used to retrieve the suspended bucket and move the keyword link address in the suspended bucket to the current time to the active bucket.
  • the keyword link address corresponding to the page number is adjusted in the dynamic bucket and the active bucket according to the comparison between the number of the old page number and the new page number, and the suspended bucket is retrieved at an appropriate time to avoid hanging.
  • the link address in the bucket is put on hold for a long time.
  • the number of allowed dynamic buckets is greater than the number of allowed basic buckets. By setting the number of allowed dynamic buckets to be larger than the number of allowed basic buckets, more resources are used to capture popular keywords.
  • FIG. 2 is a block diagram of a network crawler scheduling system based on keyword retrieval, which is characterized in that it includes: a dispatching end 21 and at least one crawling node 22 communicating with the dispatching end 21;
  • the dispatching end 21 includes:
  • the task request command receiving module 211 is configured to receive a task request command sent by the crawling node;
  • the priority bucket task generating module 212 is configured to acquire a secondary download link address generation task from the priority bucket that saves the secondary download link address and join the task list, and if the task list is allowed to join the priority bucket number, execute the task list return module. Otherwise, the dynamic bucket task generating module is executed, and the secondary download link address is obtained by the crawling node according to the task in the task list, and the crawling page is obtained, and the link that needs to be downloaded twice is analyzed from the crawling page. address;
  • the dynamic bucket task generating module 213 is configured to save a dynamic bucket of the keyword multi-page code link address to obtain a keyword link address, and derive a derivative link address of the page number according to the number of page codes corresponding to the keyword link address, according to the The derived link address of the page number generates the task of the page number and joins the task list. If the number of dynamic buckets allowed to join the task list is reached, the task list return module is executed, otherwise the dynamic bucket task generation module is executed, and the keyword link is executed.
  • the address is in the target website according to the keyword a search result page of the generated search result page, and the search result page corresponding to the keyword link address in the dynamic bucket includes a page number greater than or equal to a preset page number threshold, and the page number threshold is greater than or equal to 2;
  • the basic bucket task generating module 214 is configured to obtain a keyword link address generation task from the basic bucket storing the keyword link address and join the task list. If the task list is allowed to join the basic bucket number, the task list return module is executed.
  • the keyword link address is a link address of a search result page generated according to the keyword in the target website, and the search result page corresponding to the keyword link address in the basic bucket includes a page number smaller than a preset page number threshold.
  • the page number threshold is greater than or equal to 2;
  • the task list returning module 215 is configured to return a task list to the crawling node
  • the crawling node 22 includes:
  • the task request command sending module 221 is configured to send a task request command to the dispatching end;
  • the task execution module 222 is configured to execute a task in the task list according to the received task list.
  • the priority bucket task generating module 212 is specifically configured to: acquire a secondary download link address generation task from the priority bucket that saves the secondary download link address, join the task list, and delete the generated from the priority bucket.
  • the secondary download link address of the task if the priority bucket number of the task list is allowed to be added, the task list returning module 215 is executed. Otherwise, if the secondary download link address is also stored in the priority bucket, the priority bucket task generating module 212 is executed. If all the secondary download link addresses in the priority bucket have been deleted, the dynamic bucket task generation module 213 is executed.
  • the dynamic bucket task generating module 213 is specifically configured to: obtain an unscheduled keyword link address from a dynamic bucket that stores a keyword link address, and derive the page number according to the number of page codes corresponding to the keyword link address. Deriving a link address of the page number, generating a task of the page number according to the derived link address of the page number, adding a task list, and setting a state of the keyword link address of the generated task to be scheduled, if the task has been reached If the number of dynamic buckets is allowed to be added to the list, the task list returning module 215 is executed, and the state of all the keyword link addresses in the dynamic bucket is set to be unscheduled, otherwise If the unscheduled keyword link address is also stored in the dynamic bucket, the dynamic bucket task generation module 213 is executed. If the unscheduled keyword link address is not stored in the dynamic bucket, the basic bucket task generation module 214 is executed.
  • the basic bucket includes an active bucket and a suspended bucket
  • the basic bucket task generating module 214 is specifically configured to acquire a keyword link address generation with the highest scheduling time from the active bucket that saves the keyword link address.
  • the task is added to the task list, and the scheduling time of the keyword link address of the generated task is increased by the preset scheduling time increase amount, and then moved to the suspended bucket. If the task list is allowed to join the basic bucket number, the task list is executed. Returning to the module 215, if the keyword link address is still stored in the activity bucket, the basic bucket task generation module 214 is executed. If the keyword link address is not saved in the activity bucket, the task list return module 215 is executed.
  • the task execution module is specifically configured to: perform a capture to the crawling page according to the task in the task list, and analyze the captured page to obtain analysis data including a secondary download link address, information details, or page number, and analyze the data. Data is sent to the dispatcher;
  • the dispatching end further includes an analysis data receiving module for receiving the analysis data:
  • the analysis data is a secondary download link address
  • the secondary download link address is placed in the priority bucket
  • analysis data is information details
  • the information details are put into the data bucket
  • the keyword link address corresponding to the number of page numbers is adjusted in the dynamic bucket and the basic bucket.
  • the analyzing the data receiving module to adjust the keyword link address corresponding to the page number in the dynamic bucket and the basic bucket includes:
  • the number of page codes corresponding to the corresponding keyword link address is modified to be a new page number
  • the corresponding keyword link address is moved to the dynamic bucket.
  • the task execution module is specifically configured to: perform a capture to the crawling page according to the task in the task list, and analyze the captured page to obtain analysis data including a secondary download link address, information details, or page number, and analyze the data. Data is sent to the dispatcher;
  • the dispatching end further includes an analysis data receiving module for receiving the analysis data:
  • the analysis data is a secondary download link address
  • the secondary download link address is placed in the priority bucket
  • analysis data is information details
  • the information details are put into the data bucket
  • the analysis data is the number of page numbers, set the number of page numbers currently received to be the number of new page numbers, and set the number of page numbers received by the same keyword link address as the number of old page numbers, and if the number of old page numbers does not match the number of new page numbers, :
  • the number of page numbers corresponding to the corresponding keyword link address is modified to be a new page number
  • the suspended bucket is retrieved, and the keyword link address in the suspended bucket that reaches the current time in the scheduled time is moved to the active bucket.
  • the number of allowed dynamic buckets is greater than the allowed addition. Enter the number of basic buckets.
  • FIG. 3 is a schematic structural diagram of a system according to an example of the present invention, including:
  • the priority bucket 321 located at the dispatching end 32, the dynamic bucket 322, the basic bucket 323, the active bucket 3231, the suspended bucket 3232, the bucket updater 324, the data bucket 325, and the database 326.
  • the bucket scheduling management module (BucketScheduler) includes a priority bucket 321, a dynamic bucket 322, and a basic bucket 323.
  • the priority bucket 321 stores a secondary download link address returned by the crawling node, and the URLs are deleted from the bucket after the scheduling.
  • the dynamic bucket 322 stores URLs composed of keywords of each website. These URLs will generate new URLs according to the results returned by the statistics module. These derived URLs will not be saved, but will be scheduled together with the URLs in the dynamic bucket.
  • the regular bucket also stores URLs composed of keywords of each website, but these URLs do not derive new URLs.
  • the regular bucket 323 is divided into two parts: the active bucket 3231 and the suspended bucket 3232, and the URL in the active bucket 3231 is scheduled.
  • the bucket scheduling management module extracts the corresponding URL from the bucket and constructs the task as a task and sends the task to the crawling node.
  • the URL is obtained from the regular bucket. If the number of URLs of the active bucket in the regular bucket is less than or equal to 200, all the URLs are built into the task, and all the URLs are added to the suspended bucket; if the number of URLs in the active bucket is greater than 200, then The top 200 URLs are scheduled to build tasks, and these URLs are added to the suspended bucket.
  • FIG. 4 is a flow chart of a bucket scheduling management module. Its In the priorityBucket is allowed to join the number of priority buckets, dynamicBucket ⁇ the average number of page numbers per URL is allowed to join the dynamic bucket number, the task size size minus the priorityBucket minus the dynamicBucket ⁇ the average number of pages per URL, then the basic number is allowed to join.
  • the number of barrels As shown in Figure 4, it includes:
  • Step S401 obtaining a URL from the priority bucket, while deleting from the bucket, if the task is full, return, otherwise step S402;
  • Step S402 obtaining a URL from the dynamic bucket, and deriving a new URL through the page number, if the task is full, returning, otherwise performing step S403;
  • Step S403 obtaining a URL from the activity bucket (conventional bucket), modifying the next scheduling time of the URLs, and inserting the suspended bucket.
  • the statistics module is located at the capture end. After the crawling node requests the data, the number of information of the page and the time interval between the first information and the last information of the page are extracted and sent to the statistics module, and the statistics module calculates the keyword according to the data. The number of page numbers that can be generated in 30 minutes and pass the data to the dispatcher.
  • the bucket updater 324 (BucketUpdater) is located at the dispatcher 32, and its main function is to update the URLs in the priority bucket 321, the dynamic bucket 322, and the basic bucket 323.
  • the data extracted by the crawling node 31, including the secondary download link address, the information details, and the page number, are transmitted to the bucket updater 324 of the dispatcher 32.
  • the bucket updater 324 updates different according to the data type.
  • Bucket the specific process, as shown in Figure 5, includes:
  • step S510 the data type is checked. If the analyzed data is the secondary download link address, step S520 is performed. If the analyzed data is the information details, step S540 is performed, and if the analyzed data is the number of page numbers, Then executing step S540;
  • Step S520 inserting the data into a priority bucket (PriorityBucket) for scheduling by the dispatcher;
  • step S530 the data is inserted into the data bucket (DataBucket) for the service party to take data.
  • DataBucket data bucket
  • Step S540 comparing with the number of page numbers obtained last time, if the number of page numbers is Wait, then do nothing and return. If the number of page numbers is not equal, then:
  • step S541 the URL is currently located in the dynamic bucket, and there is no need to exchange the bucket, and only the number of page numbers of the barcode is modified.
  • step S542 the URL is currently located in the dynamic bucket and needs to be moved to the active bucket.
  • step S543 the URL is currently located in the regular bucket and needs to be moved to the dynamic bucket.
  • step S544 the URL is currently located in the regular bucket and is likely to be a suspended bucket.
  • the suspended bucket needs to be scanned, and the URL of the suspended bucket to reach the scheduling time is added to the active bucket.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Information Transfer Between Computers (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种基于关键词检索的网络爬虫调度方法及系统，方法包括：调度端接收到抓取节点发送的任务请求命令；调度端从优先桶获取二次下载链接地址生成任务并加入任务列表，从动态桶获取关键词链接地址，并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址，根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表，从基本桶获取关键词链接地址生成任务并加入任务列表，调度端向抓取节点返回任务列表。通过调整各虚拟桶的允许加入任务数量，从而灵活地调整不同类型的链接地址的调用数量。同时更频繁地抓取热门关键词，避免数据遗漏，减少对冷门关键词的重复抓取。

Description

一种基于关键词检索的网络爬虫调度方法及系统 技术领域
本发明涉及网络爬虫相关技术领域，特别是一种基于关键词检索的网络爬虫调度方法及系统。
背景技术
基于关键词检索的网络爬虫是获取关键词信息的基础，但是由于信息的更新会导致网络爬虫的重复抓取或抓不全等情况的发生。特别是目标网站信息变更非常频繁的情况，比如新浪微博等微博类网站，百度等搜索类网站。对于热门关键词，由于信息更新很快，导致网络爬虫很难抓全这些数据；而对于冷门的关键词，由于信息更新比较慢，会导致信息的重复抓取。
现有的做法是给关键词设置不同的热度，然后根据关键词热度对关键词进行抓取，热度大的关键词抓取的更加频繁。
然而，现有的做法存在如下缺陷：
(1)需要知道每个关键词的热度，然后根据热度来设置抓取频率。
(2)在首次请求的过程中会涉及很多二次下载链接地址的请求，现有方案并没有对此进行区分。
发明内容
基于此，有必要针对现有技术通过设置关键词热度来设置抓取频率，需要预先确定关键词热度，从而导致抓取频率不准确的技术问题，提供一种基于关键词检索的网络爬虫调度方法及系统。
一种基于关键词检索的网络爬虫调度方法，包括：
步骤12，调度端从抓取节点接收任务请求命令；
步骤13，调度端从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表，如果已达到任务列表的允许加入优先桶数量，则执行步骤16，否则执行步骤14，所述二次下载链接地址为所述抓取节点根据任务列表中的任务进行抓取得到抓取页面，从抓取页面中分析得到需要进行二次下载的链接地址；
步骤14，调度端从保存关键词多页码链接地址的动态桶获取关键词链接地址，并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址，根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表，如果已达到任务列表的允许加入动态桶数量，则执行步骤16，否则执行步骤15，所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址，且动态桶中的关键词链接地址对应的搜索结果页面包括大于或等于预设页码数量阈值的页码数量，所述页码数量阈值大于或等于2；
步骤15，调度端从保存关键词链接地址的基本桶获取关键词链接地址生成任务并加入任务列表，如果已达到任务列表的允许加入基本桶数量，则执行步骤16，所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址，且基本桶中的关键词链接地址对应的搜索结果页面包括小于预设页码数量阈值的页码数量，所述页码数量阈值大于或等于2；
步骤16，调度端向抓取节点返回任务列表，所述抓取节点根据所接收到的任务列表执行任务列表中的任务。
一种基于关键词检索的网络爬虫调度系统，包括：调度端以及与调度端通讯的至少一个抓取节点；
所述调度端包括：
任务请求命令接收模块，用于接收到抓取节点发送的任务请求命令；
优先桶任务生成模块，用于从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表，如果已达到任务列表 的允许加入优先桶数量，则执行任务列表返回模块，否则执行动态桶任务生成模块，所述二次下载链接地址为所述抓取节点根据任务列表中的任务进行抓取得到抓取页面，从抓取页面中分析得到需要进行二次下载的链接地址；
动态桶任务生成模块，用于保存关键词多页码链接地址的动态桶获取关键词链接地址，并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址，根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表，如果已达到任务列表的允许加入动态桶数量，则执行任务列表返回模块，否则执行动态桶任务生成模块，所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址，且动态桶中的关键词链接地址对应的搜索结果页面包括大于或等于预设页码数量阈值的页码数量，所述页码数量阈值大于或等于2；
基本桶任务生成模块，用于从保存关键词链接地址的基本桶获取关键词链接地址生成任务并加入任务列表，如果已达到任务列表的允许加入基本桶数量，则执行任务列表返回模块，所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址，且基本桶中的关键词链接地址对应的搜索结果页面包括小于预设页码数量阈值的页码数量，所述页码数量阈值大于或等于2；
任务列表返回模块，用于向抓取节点返回任务列表；
所述抓取节点包括：
任务请求命令发送模块，用于向调度端发送任务请求命令；
任务执行模块，用于根据所接收到的任务列表执行任务列表中的任务。
本发明中优先桶、动态桶和基本桶均为虚拟桶，通过设置具有不同优先级的虚拟桶，在虚拟桶中分别保存不同类型的链接地址，对于较高热度的链接地址放置于较高优先级的虚拟桶中。通过调整各虚拟桶的允许加入任务数量，从而灵活地调整不同类型的链接地址的调用数量。同时，链接地址的热度并不是由人工设置，而是根据链接地址 所抓取到的页码的实际情况设置，因此使得更频繁地抓取热门关键词，避免数据遗漏，同时减少对冷门关键词的重复抓取。
附图说明
图1为本发明一种基于关键词检索的网络爬虫调度方法的工作流程图；
图2为本发明一种基于关键词检索的网络爬虫调度系统的模块示意图；
图3为本发明一个例子的系统结构示意图；
图4为本发明一个例子桶调度管理模块的工作流程图；
图5为本发明一个例子桶更新器的工作流程图。
具体实施方式
下面结合附图和具体实施例对本发明做进一步详细的说明。
如图1所示为本发明一种基于关键词检索的网络爬虫调度方法的工作流程图，包括：
步骤11，抓取节点向调度端发送任务请求命令；
步骤12，调度端接收到抓取节点发送的任务请求命令；
步骤13，调度端从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表，如果已达到任务列表的允许加入优先桶数量，则执行步骤16，否则执行步骤14，所述二次下载链接地址为所述抓取节点根据任务列表中的任务进行抓取得到抓取页面，从抓取页面中分析得到需要进行二次下载的链接地址；
步骤14，调度端从保存关键词多页码链接地址的动态桶获取关键词链接地址，并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址，根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表，如果已达到任务列表的允许加入动态桶数量，则执行步骤16，否则执行步骤15，所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址，且 动态桶中的关键词链接地址对应的搜索结果页面包括大于或等于预设页码数量阈值的页码数量，所述页码数量阈值大于或等于2；
步骤15，调度端从保存关键词链接地址的基本桶获取关键词链接地址生成任务并加入任务列表，如果已达到任务列表的允许加入基本桶数量，则执行步骤16，所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址，且基本桶中的关键词链接地址对应的搜索结果页面包括小于预设页码数量阈值的页码数量，所述页码数量阈值大于或等于2；
步骤16，调度端向抓取节点返回任务列表，所述抓取节点根据所接收到的任务列表执行任务列表中的任务。
其中，步骤13中，专门针对二次下载链接地址进行处理，二次下载链接地址，是指抓取节点根据任务列表中的任务进行抓取得到抓取页面，从抓取页面中分析得到需要进行二次下载的链接地址，例如，在百度中检索“京东”，会出现一个包括多条检索结果的搜索结果页面，每条检索结果都对应一个链接地址，检索结果所对应的链接地址即为二次下载链接地址。从抓取页面中分析得到需要进行二次下载的链接地址，现有已经有较为成熟的技术进行分析，在此不予赘述。
其中，步骤14和15中的关键词链接地址，指的是在目标网站中根据所述关键词所生成的搜索结果页面的链接地址，例如，在百度中检索“京东”，会出现一个根据关键词“京东”生成的搜索结果页面，如果关键词为热门关键词，则一个搜索结果页面并不能包括所有的搜索结果，则在搜索页面上会有一个页码数量。该页码数量用于表示有页码数量的搜索结果页面包括所述关键词。因此，页码数量越多，则表示该关键词越为热门，因此，将页码数量超过预设页码数量阈值的关键词链接地址放置在动态桶中，在步骤14中优先处理。由于是根据实际的页码数量进行判断，因此，能够更真实地反映关键词的热门程度。
其中，允许加入优先桶数量用于表示优先桶中的二次下载链接地址允许加入任务列表中的数量，允许加入动态桶数量用于表示动态桶中的关键词链接地址允许加入任务列表中的数量，允许加入基本桶数量用于表示基本桶中的关键词链接地址允许加入任务列表中的数量。
在其中一个实施例中，所述步骤13具体包括：从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表，并从优先桶中删除已生成任务的二次下载链接地址，如果已达到任务列表的允许加入优先桶数量，则执行步骤16，否则如果优先桶中还保存有二次下载链接地址，则执行步骤13，如果优先桶中所有的二次下载链接地址均已删除，则执行步骤14。
本实施例优先执行二次下载链接地址，且该二次下载链接地址仅执行一次后删除，避免重复抓取。
在其中一个实施例中，所述步骤14具体包括：从保存关键词链接地址的动态桶获取未调度的关键词链接地址，并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址，根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表，并设置已生成任务的关键词链接地址的状态为已调度，如果已达到任务列表的允许加入动态桶数量，则执行步骤16，并设置动态桶中所有的关键词链接地址的状态为未调度，否则如果动态桶中还保存有未调度的关键词链接地址，则执行步骤14，如果动态桶中未保存有未调度的关键词链接地址，则执行步骤15。
本实施例中，由于动态桶中的关键词链接地址为热门关键词对应的链接地址，因此会被多次调用。从而避免热门关键词抓取的遗漏。
在其中一个实施例中，所述基本桶包括活动桶和挂起桶，所述步骤15具体包括，从保存关键词链接地址的活动桶获取调度时间最前的 关键词链接地址生成任务并加入任务列表，并将已生成任务的关键词链接地址的调度时间增加预设的调度时间增加量后移动到挂起桶，如果已达到任务列表的允许加入基本桶数量，则执行步骤16，否则如果活动桶中还保存有关键词链接地址，则执行步骤15，如果活动桶中未保存关键词链接地址，则执行步骤16。
本实施例中，由于基本桶中的关键词链接地址为冷门关键词对应的链接地址，因此对基本桶进一步设定为活动桶和挂起桶，活动桶中的链接地址执行一次后会移动到挂起桶中，间隔一段调度时间增加量之后再执行，从而避免对冷门关键词的重复抓取。
在其中一个实施例中，还包括：
抓取节点根据任务列表中的任务进行抓取得到抓取页面，对所述抓取页面进行分析得到包括二次下载链接地址、信息详情或页码数量的分析数据，将分析数据发送到调度端；
调度端接收到分析数据：
如果分析数据为二次下载链接地址，则将二次下载链接地址放入优先桶；
如果分析数据为信息详情，则将信息详情放入数据桶；
如果分析数据为页码数量，则在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址。
将不同的分析数据放置于不同的虚拟桶中，实现对虚拟桶的链接地址的更新。
在其中一个实施例中，调度端在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址具体包括：
设置当前接收到的页码数量为新页码数量，设置上一次同一关键词链接地址接收到的页码数量为旧页码数量；
如果旧页码数量与新页码数量不一致，则：
如果旧页码数量大于或等于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则修改对应的关键词链接地址对应的页码数量为新页码数量；
如果旧页码数量大于或等于所述页码数量阈值，且新页码数量小于所述页码数量阈值，则将对应的关键词链接地址移动到基本桶；
如果旧页码数量小于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则将对应的关键词链接地址移动到动态桶。
本实施例，进一步限定根据旧页码数量和新页码数量的比较在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址。
在其中一个实施例中：
抓取节点根据任务列表中的任务进行抓取得到抓取页面，对所述抓取页面进行分析得到包括二次下载链接地址、信息详情或页码数量的分析数据，将分析数据发送到调度端；
调度端接收到分析数据：
如果分析数据为二次下载链接地址，则将二次下载链接地址放入优先桶；
如果分析数据为信息详情，则将信息详情放入数据桶；
如果分析数据为页码数量，则设置当前接收到的页码数量为新页码数量，设置上一次同一关键词链接地址接收到的页码数量为旧页码数量，且如果旧页码数量与新页码数量不一致，则：
如果旧页码数量大于或等于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则修改对应的关键词链接地址对应的页码数量为新页码数量；
如果旧页码数量大于或等于所述页码数量阈值，且新页码数量小于所述页码数量阈值，则将对应的关键词链接地址移动到活动桶；
如果旧页码数量小于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则将对应的关键词链接地址移动到动态桶
如果旧页码数量小于所述页码数量阈值，且新页码数量小于所述 页码数量阈值，则对挂起桶进行检索，将挂起桶中调度时间到达当前时间的关键词链接地址移动到活动桶。
本实施例，进一步限定根据旧页码数量和新页码数量的比较在动态桶和活动桶中调整与所述页码数量对应的关键词链接地址，以及在适当的时候对挂起桶进行检索，避免挂起桶中的链接地址长时间被搁置。
在其中一个实施例中，所述允许加入动态桶数量大于所述允许加入基本桶数量。通过设置允许加入动态桶数量大于允许加入基本桶数量，从而使得更多的资源用于抓取热门关键词。
如图2所示为本发明一种基于关键词检索的网络爬虫调度系统的模块示意图，其特征在于，包括：调度端21以及与调度端21通讯的至少一个抓取节点22；
所述调度端21包括：
任务请求命令接收模块211，用于接收到抓取节点发送的任务请求命令；
优先桶任务生成模块212，用于从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表，如果已达到任务列表的允许加入优先桶数量，则执行任务列表返回模块，否则执行动态桶任务生成模块，所述二次下载链接地址为所述抓取节点根据任务列表中的任务进行抓取得到抓取页面，从抓取页面中分析得到需要进行二次下载的链接地址；
动态桶任务生成模块213，用于保存关键词多页码链接地址的动态桶获取关键词链接地址，并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址，根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表，如果已达到任务列表的允许加入动态桶数量，则执行任务列表返回模块，否则执行动态桶任务生成模块，所述关键词链接地址为在目标网站中根据所述关键词所 生成的搜索结果页面的链接地址，且动态桶中的关键词链接地址对应的搜索结果页面包括大于或等于预设页码数量阈值的页码数量，所述页码数量阈值大于或等于2；
基本桶任务生成模块214，用于从保存关键词链接地址的基本桶获取关键词链接地址生成任务并加入任务列表，如果已达到任务列表的允许加入基本桶数量，则执行任务列表返回模块，所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址，且基本桶中的关键词链接地址对应的搜索结果页面包括小于预设页码数量阈值的页码数量，所述页码数量阈值大于或等于2；
任务列表返回模块215，用于向抓取节点返回任务列表；
所述抓取节点22包括：
任务请求命令发送模块221，用于向调度端发送任务请求命令；
任务执行模块222，用于根据所接收到的任务列表执行任务列表中的任务。
在其中一个实施例中，所述优先桶任务生成模块212具体用于：从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表，并从优先桶中删除已生成任务的二次下载链接地址，如果已达到任务列表的允许加入优先桶数量，则执行任务列表返回模块215，否则如果优先桶中还保存有二次下载链接地址，则执行优先桶任务生成模块212，如果优先桶中所有的二次下载链接地址均已删除，则执行动态桶任务生成模块213。
在其中一个实施例中，所述动态桶任务生成模块213具体用于：从保存关键词链接地址的动态桶获取未调度的关键词链接地址，并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址，根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表，并设置已生成任务的关键词链接地址的状态为已调度，如果已达到任务列表的允许加入动态桶数量，则执行任务列表返回模块215，并设置动态桶中所有的关键词链接地址的状态为未调度，否则 如果动态桶中还保存有未调度的关键词链接地址，则执行动态桶任务生成模块213，如果动态桶中未保存有未调度的关键词链接地址，则执行基本桶任务生成模块214。
在其中一个实施例中，所述基本桶包括活动桶和挂起桶，所述基本桶任务生成模块214具体用于，从保存关键词链接地址的活动桶获取调度时间最前的关键词链接地址生成任务并加入任务列表，并将已生成任务的关键词链接地址的调度时间增加预设的调度时间增加量后移动到挂起桶，如果已达到任务列表的允许加入基本桶数量，则执行任务列表返回模块215，否则如果活动桶中还保存有关键词链接地址，则执行基本桶任务生成模块214，如果活动桶中未保存关键词链接地址，则执行任务列表返回模块215。
在其中一个实施例中：
所述任务执行模块具体用于：根据任务列表中的任务进行抓取得到抓取页面，对所述抓取页面进行分析得到包括二次下载链接地址、信息详情或页码数量的分析数据，将分析数据发送到调度端；
调度端还包括分析数据接收模块，用于接收到分析数据：
如果分析数据为二次下载链接地址，则将二次下载链接地址放入优先桶；
如果分析数据为信息详情，则将信息详情放入数据桶；
如果分析数据为页码数量，则在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址。
在其中一个实施例中，分析数据接收模块在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址具体包括：
设置当前接收到的页码数量为新页码数量，设置上一次同一关键词链接地址接收到的页码数量为旧页码数量；
如果旧页码数量与新页码数量不一致，则：
如果旧页码数量大于或等于所述页码数量阈值，且新页码数量大 于或等于所述页码数量阈值，则修改对应的关键词链接地址对应的页码数量为新页码数量；
如果旧页码数量大于或等于所述页码数量阈值，且新页码数量小于所述页码数量阈值，则将对应的关键词链接地址移动到基本桶；
如果旧页码数量小于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则将对应的关键词链接地址移动到动态桶。
在其中一个实施例中：
所述任务执行模块具体用于：根据任务列表中的任务进行抓取得到抓取页面，对所述抓取页面进行分析得到包括二次下载链接地址、信息详情或页码数量的分析数据，将分析数据发送到调度端；
调度端还包括分析数据接收模块，用于接收到分析数据：
如果分析数据为二次下载链接地址，则将二次下载链接地址放入优先桶；
如果分析数据为信息详情，则将信息详情放入数据桶；
如果分析数据为页码数量，则设置当前接收到的页码数量为新页码数量，设置上一次同一关键词链接地址接收到的页码数量为旧页码数量，且如果旧页码数量与新页码数量不一致，则：
如果旧页码数量大于或等于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则修改对应的关键词链接地址对应的页码数量为新页码数量；
如果旧页码数量大于或等于所述页码数量阈值，且新页码数量小于所述页码数量阈值，则将对应的关键词链接地址移动到活动桶；
如果旧页码数量小于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则将对应的关键词链接地址移动到动态桶
如果旧页码数量小于所述页码数量阈值，且新页码数量小于所述页码数量阈值，则对挂起桶进行检索，将挂起桶中调度时间到达当前时间的关键词链接地址移动到活动桶。
在其中一个实施例中，所述允许加入动态桶数量大于所述允许加 入基本桶数量。
如图3所示为本发明一个例子的系统结构示意图，包括：
位于抓取节点31的线程请求模块311、任务获取模块312、网页下载器313、文档解析器314、统计模块315、数据存储模块316；
位于调度端32的优先桶321、动态桶322、基本桶323、活动桶3231、挂起桶3232、桶更新器324、数据桶325、数据库326。
其中，桶调度管理模块(BucketScheduler)包含优先桶321、动态桶322、基本桶323，优先桶321中保存抓取节点返回的二次下载链接地址，这些URL在调度完后会从桶中删除。动态桶322中保存各网站的关键词组成的URL，这些URL会根据统计模块返回的结果衍生出新的URL，这些衍生出的URL不会保存，但是会和动态桶中的URL一起被调度。常规桶中也保存各网站的关键词组成的URL，但是这些URL不会衍生新的URL，常规桶323分为两部分：活动桶3231和挂起桶3232，活动桶3231中的URL会按调度时间进行排序，调度时间越早的会越早调度，而挂起桶3232中的URL则不会被调度，这两个桶中的URL会相互交换。一旦调度端接收到抓取节点的任务请求，桶调度管理模块则从桶中提取相应的URL构建成任务发给抓取节点。构建任务的规则如下：假如任务大小size＝1000。优先桶(PriorityBucket)中的URL数为priorityBucket＝300。动态桶(DynamicBucket)中的URL数为dynamicBucket＝100且每个URL的平均页码数量page＝5。常规桶(CommonBucket)中的URL数为commonBucket＝500。则首先从优先桶中获取URL，剩余的任务大小为remainSize＝700。接着从动态桶获取URL，此时剩余的任务大小为remainSize＝700-100*5＝200。最后从常规桶获取URL，如果常规桶中活动桶的URL数小于等于200，则把这些URL全部构建任务，同时把这些URL全部加入挂起桶；如果活动桶中的URL数大于200，则取调度时间靠前的前200个URL构建任务，并把这些URL加入挂起桶。如果挂起桶中的URL到了可以调度的时间，则把它们移入活动桶。图4是桶调度管理模块的流程图。其 中，priorityBucket为允许加入优先桶数量，dynamicBucket×每个URL的平均页码数量为允许加入动态桶数量，任务大小size减去priorityBucket再减去dynamicBucket×每个URL的平均页码数量，则为允许加入基本桶数量。如图4所示，包括：
步骤S401，从优先桶中获取URL，同时从桶中删除，如果任务已满，返回，否则执行步骤S402；
步骤S402，从动态桶中获取URL，并通过页码衍生出新的URL，如果任务已满，返回，否则执行步骤S403；
步骤S403，从活动桶(常规桶)获取URL，修改这些URL的下次调度时间，插入挂起桶。
统计模块位于抓取端，抓取节点请求到数据后，提取页面的信息数以及该页第一条信息和最后一条信息的时间间隔，传给统计模块，统计模块根据这些数据计算该关键词在30分钟内可以产生的页码数量，并把这些数据传给调度端。
桶更新器324(BucketUpdater)位于调度端32，它的主要功能是更新优先桶321、动态桶322、基本桶323中的URL。抓取节点31抽取到的数据，包括二次下载链接地址、信息详情和页码数量都会传给调度端32的桶更新器324，桶更新器324接收到这些数据后，会根据数据类型更新不同的桶，具体的过程，如图5所示，包括：
步骤S510，对数据类型进行检查，如果分析出的数据是二次下载链接地址，则执行步骤S520，如果分析出的数据是信息详情，则执行步骤S540，如果分析分析出的数据是页码数量，则执行步骤S540；
步骤S520，把这些数据插入优先桶(PriorityBucket)，供调度机调度；
步骤S530，把这些数据插入数据桶(DataBucket)，供业务方取数据。
步骤S540，与上次获取到的页码数量进行比较，如果页码数量相 等，则不进行任何操作，返回。如果页码数量不等，则：
如果老的页码数量(oldPages)大于等于阈值(阈值threshold可以自己设定，在本方案中设置为3)即oldPages＞＝threshold，且新的页码数量(newPages)也大于等于阈值即newPages＞＝threshold。则执行步骤S541，如果oldPages＞＝threshold，但newPages＜threshold，则执行步骤S542，如果oldPages＜threshold，但newPages＞＝threshold，则执行步骤S543，如果oldPages＜threshold，但newPages＜threshold，则执行步骤S544；
步骤S541，该URL目前位于动态桶，无需进行桶的交换，只需修改它的页码数量即可。
步骤S542，该URL目前位于动态桶，需要把它移到活动桶。
步骤S543，该URL目前位于常规桶，需要把它移到动态桶。
步骤S544，该URL目前位于常规桶并且很有可能是挂起桶，需要扫描挂起桶，把挂起桶中到达调度时间的URL加入活动桶。
以上所述实施例仅表达了本发明的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对本发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本发明构思的前提下，还可以做出若干变形和改进，这些都属于本发明的保护范围。因此，本发明专利的保护范围应以所附权利要求为准。

Claims (16)

  1. 一种基于关键词检索的网络爬虫调度方法，其特征在于，包括：
    步骤(12)，调度端从抓取节点接收任务请求命令；
    步骤(13)，调度端从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表，如果已达到任务列表的允许加入优先桶数量，则执行步骤(16)，否则执行步骤(14)，所述二次下载链接地址为所述抓取节点根据任务列表中的任务进行抓取得到抓取页面，从抓取页面中分析得到需要进行二次下载的链接地址；
    步骤(14)，调度端从保存关键词多页码链接地址的动态桶获取关键词链接地址，并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址，根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表，如果已达到任务列表的允许加入动态桶数量，则执行步骤(16)，否则执行步骤(15)，所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址，且动态桶中的关键词链接地址对应的搜索结果页面包括大于或等于预设页码数量阈值的页码数量，所述页码数量阈值大于或等于2；
    步骤(15)，调度端从保存关键词链接地址的基本桶获取关键词链接地址生成任务并加入任务列表，如果已达到任务列表的允许加入基本桶数量，则执行步骤(16)，所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址，且基本桶中的关键词链接地址对应的搜索结果页面包括小于预设页码数量阈值的页码数量，所述页码数量阈值大于或等于2；
    步骤(16)，调度端向抓取节点返回任务列表，所述抓取节点根据所接收到的任务列表执行任务列表中的任务。
  2. 根据权利要求1所述的基于关键词检索的网络爬虫调度方法，其特征在于，所述步骤(13)具体包括：从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表，并从优先桶 中删除已生成任务的二次下载链接地址，如果已达到任务列表的允许加入优先桶数量，则执行步骤(16)，否则如果优先桶中还保存有二次下载链接地址，则执行步骤(13)，如果优先桶中所有的二次下载链接地址均已删除，则执行步骤(14)。
  3. 根据权利要求1所述的基于关键词检索的网络爬虫调度方法，其特征在于，所述步骤(14)具体包括：从保存关键词链接地址的动态桶获取未调度的关键词链接地址，并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址，根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表，并设置已生成任务的关键词链接地址的状态为已调度，如果已达到任务列表的允许加入动态桶数量，则执行步骤(16)，并设置动态桶中所有的关键词链接地址的状态为未调度，否则如果动态桶中还保存有未调度的关键词链接地址，则执行步骤(14)，如果动态桶中未保存有未调度的关键词链接地址，则执行步骤(15)。
  4. 根据权利要求1所述的基于关键词检索的网络爬虫调度方法，其特征在于，所述基本桶包括活动桶和挂起桶，所述步骤(15)具体包括，从保存关键词链接地址的活动桶获取调度时间最前的关键词链接地址生成任务并加入任务列表，并将已生成任务的关键词链接地址的调度时间增加预设的调度时间增加量后移动到挂起桶，如果已达到任务列表的允许加入基本桶数量，则执行步骤(16)，否则如果活动桶中还保存有关键词链接地址，则执行步骤(15)，如果活动桶中未保存关键词链接地址，则执行步骤(16)。
  5. 根据权利要求1所述的基于关键词检索的网络爬虫调度方法，其特征在于，还包括：
    抓取节点根据任务列表中的任务进行抓取得到抓取页面，对所述抓取页面进行分析得到包括二次下载链接地址、信息详情或页码数量的分析数据，将分析数据发送到调度端；
    调度端接收到分析数据：
    如果分析数据为二次下载链接地址，则将二次下载链接地址放入优先桶；
    如果分析数据为信息详情，则将信息详情放入数据桶；
    如果分析数据为页码数量，则在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址。
  6. 根据权利要求5所述的基于关键词检索的网络爬虫调度方法，其特征在于，调度端在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址具体包括：
    设置当前接收到的页码数量为新页码数量，设置上一次同一关键词链接地址接收到的页码数量为旧页码数量；
    如果旧页码数量与新页码数量不一致，则：
    如果旧页码数量大于或等于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则修改对应的关键词链接地址对应的页码数量为新页码数量；
    如果旧页码数量大于或等于所述页码数量阈值，且新页码数量小于所述页码数量阈值，则将对应的关键词链接地址移动到基本桶；
    如果旧页码数量小于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则将对应的关键词链接地址移动到动态桶。
  7. 根据权利要求4所述的基于关键词检索的网络爬虫调度方法，其特征在于：
    抓取节点根据任务列表中的任务进行抓取得到抓取页面，对所述抓取页面进行分析得到包括二次下载链接地址、信息详情或页码数量的分析数据，将分析数据发送到调度端；
    调度端接收到分析数据：
    如果分析数据为二次下载链接地址，则将二次下载链接地址放入优先桶；
    如果分析数据为信息详情，则将信息详情放入数据桶；
    如果分析数据为页码数量，则设置当前接收到的页码数量为新页码数量，设置上一次同一关键词链接地址接收到的页码数量为旧页码数量，且如果旧页码数量与新页码数量不一致，则：
    如果旧页码数量大于或等于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则修改对应的关键词链接地址对应的页码数量为新页码数量；
    如果旧页码数量大于或等于所述页码数量阈值，且新页码数量小于所述页码数量阈值，则将对应的关键词链接地址移动到活动桶；
    如果旧页码数量小于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则将对应的关键词链接地址移动到动态桶
    如果旧页码数量小于所述页码数量阈值，且新页码数量小于所述页码数量阈值，则对挂起桶进行检索，将挂起桶中调度时间到达当前时间的关键词链接地址移动到活动桶。
  8. 根据权利要求1所述的基于关键词检索的网络爬虫调度方法，其特征在于，所述允许加入动态桶数量大于允许加入基本桶数量。
  9. 一种基于关键词检索的网络爬虫调度系统，其特征在于，包括：调度端以及与调度端通讯的至少一个抓取节点；
    所述调度端包括：
    任务请求命令接收模块，用于接收抓取节点发送的任务请求命令；
    优先桶任务生成模块，用于从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表，如果已达到任务列表的允许加入优先桶数量，则执行任务列表返回模块，否则执行动态桶任务生成模块，所述二次下载链接地址为所述抓取节点根据任务列表中的任务进行抓取得到抓取页面，从抓取页面中分析得到需要进行二次下载的链接地址；
    动态桶任务生成模块，用于保存关键词多页码链接地址的动态桶获取关键词链接地址，并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址，根据所述页码数量的衍生链接地址 生成所述页码数量的任务并加入任务列表，如果已达到任务列表的允许加入动态桶数量，则执行任务列表返回模块，否则执行动态桶任务生成模块，所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址，且动态桶中的关键词链接地址对应的搜索结果页面包括大于或等于预设页码数量阈值的页码数量，所述页码数量阈值大于或等于2；
    基本桶任务生成模块，用于从保存关键词链接地址的基本桶获取关键词链接地址生成任务并加入任务列表，如果已达到任务列表的允许加入基本桶数量，则执行任务列表返回模块，所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址，且基本桶中的关键词链接地址对应的搜索结果页面包括小于预设页码数量阈值的页码数量，所述页码数量阈值大于或等于2；
    任务列表返回模块，用于向抓取节点返回任务列表；
    所述抓取节点包括：
    任务请求命令发送模块，用于向调度端发送任务请求命令；
    任务执行模块，用于根据所接收到的任务列表执行任务列表中的任务。
  10. 根据权利要求9所述的基于关键词检索的网络爬虫调度系统，其特征在于，所述优先桶任务生成模块具体用于：从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表，并从优先桶中删除已生成任务的二次下载链接地址，如果已达到任务列表的允许加入优先桶数量，则执行任务列表返回模块，否则如果优先桶中还保存有二次下载链接地址，则执行优先桶任务生成模块，如果优先桶中所有的二次下载链接地址均已删除，则执行动态桶任务生成模块。
  11. 根据权利要求9所述的基于关键词检索的网络爬虫调度系统，其特征在于，所述动态桶任务生成模块具体用于：从保存关键词链接地址的动态桶获取未调度的关键词链接地址，并根据与关键词链接地 址对应的页码数量衍生出所述页码数量的衍生链接地址，根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表，并设置已生成任务的关键词链接地址的状态为已调度，如果已达到任务列表的允许加入动态桶数量，则执行任务列表返回模块，并设置动态桶中所有的关键词链接地址的状态为未调度，否则如果动态桶中还保存有未调度的关键词链接地址，则执行动态桶任务生成模块，如果动态桶中未保存有未调度的关键词链接地址，则执行基本桶任务生成模块。
  12. 根据权利要求9所述的基于关键词检索的网络爬虫调度系统，其特征在于，所述基本桶包括活动桶和挂起桶，所述基本桶任务生成模块具体用于，从保存关键词链接地址的活动桶获取调度时间最前的关键词链接地址生成任务并加入任务列表，并将已生成任务的关键词链接地址的调度时间增加预设的调度时间增加量后移动到挂起桶，如果已达到任务列表的允许加入基本桶数量，则执行任务列表返回模块，否则如果活动桶中还保存有关键词链接地址，则执行基本桶任务生成模块，如果活动桶中未保存关键词链接地址，则执行任务列表返回模块。
  13. 根据权利要求9所述的基于关键词检索的网络爬虫调度系统，其特征在于：
    所述任务执行模块具体用于：根据任务列表中的任务进行抓取得到抓取页面，对所述抓取页面进行分析得到包括二次下载链接地址、信息详情或页码数量的分析数据，将分析数据发送到调度端；
    调度端还包括分析数据接收模块，用于接收到分析数据：
    如果分析数据为二次下载链接地址，则将二次下载链接地址放入优先桶；
    如果分析数据为信息详情，则将信息详情放入数据桶；
    如果分析数据为页码数量，则在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址。
  14. 根据权利要求13所述的基于关键词检索的网络爬虫调度系统，其特征在于，分析数据接收模块在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址具体包括：
    设置当前接收到的页码数量为新页码数量，设置上一次同一关键词链接地址接收到的页码数量为旧页码数量；
    如果旧页码数量与新页码数量不一致，则：
    如果旧页码数量大于或等于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则修改对应的关键词链接地址对应的页码数量为新页码数量；
    如果旧页码数量大于或等于所述页码数量阈值，且新页码数量小于所述页码数量阈值，则将对应的关键词链接地址移动到基本桶；
    如果旧页码数量小于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则将对应的关键词链接地址移动到动态桶。
  15. 根据权利要求12所述的基于关键词检索的网络爬虫调度系统，其特征在于：
    所述任务执行模块具体用于：根据任务列表中的任务进行抓取得到抓取页面，对所述抓取页面进行分析得到包括二次下载链接地址、信息详情或页码数量的分析数据，将分析数据发送到调度端；
    调度端还包括分析数据接收模块，用于接收到分析数据：
    如果分析数据为二次下载链接地址，则将二次下载链接地址放入优先桶；
    如果分析数据为信息详情，则将信息详情放入数据桶；
    如果分析数据为页码数量，则设置当前接收到的页码数量为新页码数量，设置上一次同一关键词链接地址接收到的页码数量为旧页码数量，且如果旧页码数量与新页码数量不一致，则：
    如果旧页码数量大于或等于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则修改对应的关键词链接地址对应的页码数量为新页码数量；
    如果旧页码数量大于或等于所述页码数量阈值，且新页码数量小于所述页码数量阈值，则将对应的关键词链接地址移动到活动桶；
    如果旧页码数量小于所述页码数量阈值，且新页码数量大于或等于所述页码数量阈值，则将对应的关键词链接地址移动到动态桶
    如果旧页码数量小于所述页码数量阈值，且新页码数量小于所述页码数量阈值，则对挂起桶进行检索，将挂起桶中调度时间到达当前时间的关键词链接地址移动到活动桶。
  16. 根据权利要求9所述的基于关键词检索的网络爬虫调度系统，其特征在于，所述允许加入动态桶数量大于允许加入基本桶数量。
PCT/CN2015/070475 2014-01-09 2015-01-09 一种基于关键词检索的网络爬虫调度方法及系统 Ceased WO2015103997A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US15/110,564 US10185771B2 (en) 2014-01-09 2015-01-09 Method and system for scheduling web crawlers according to keyword search
RU2016132573A RU2645266C1 (ru) 2014-01-09 2015-01-09 Способ и устройство для планирования web-обходчиков в соответствии с поиском по ключевым словам

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201410011079.8 2014-01-09
CN201410011079.8A CN103761279B (zh) 2014-01-09 2014-01-09 一种基于关键词检索的网络爬虫调度方法及系统

Publications (1)

Publication Number Publication Date
WO2015103997A1 true WO2015103997A1 (zh) 2015-07-16

Family

ID=50528516

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2015/070475 Ceased WO2015103997A1 (zh) 2014-01-09 2015-01-09 一种基于关键词检索的网络爬虫调度方法及系统

Country Status (4)

Country Link
US (1) US10185771B2 (zh)
CN (1) CN103761279B (zh)
RU (1) RU2645266C1 (zh)
WO (1) WO2015103997A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111460254A (zh) * 2020-03-24 2020-07-28 南阳柯丽尔科技有限公司 一种基于多线程的网页爬取方法、装置、存储介质及设备
CN113965555A (zh) * 2021-10-21 2022-01-21 北京值得买科技股份有限公司 一种参数化爬虫下载方法、装置、设备及存储介质

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103761279B (zh) * 2014-01-09 2017-02-08 北京京东尚科信息技术有限公司 一种基于关键词检索的网络爬虫调度方法及系统
CN105989151B (zh) * 2015-03-02 2019-09-06 阿里巴巴集团控股有限公司 网页抓取方法及装置
CN107784036A (zh) * 2016-08-31 2018-03-09 北京国双科技有限公司 网络爬虫系统和基于网络爬虫系统的数据处理方法
US20180189409A1 (en) * 2016-12-30 2018-07-05 Caavo Inc Targeted crawler to develop and/or maintain a searchable database of media content across multiple content providers
CN107480297A (zh) * 2017-08-30 2017-12-15 福建中金在线信息科技有限公司 一种文章收录方法及装置
CN110309403B (zh) * 2018-03-05 2022-11-04 百度在线网络技术（北京）有限公司 用于抓取数据的方法和装置
GB2572544A (en) * 2018-03-27 2019-10-09 Innoplexus Ag System and method of crawling a wide area computer network for retrieving contextual information
CN108647342A (zh) * 2018-05-14 2018-10-12 佛山市真觉网络科技有限公司 一种优化百度蜘蛛抓取的方法
CN111371705B (zh) * 2020-02-24 2023-09-12 维沃移动通信有限公司 一种下载任务执行方法及电子设备
CN114238732B (zh) * 2021-11-08 2025-11-07 国投智能(厦门)信息股份有限公司 一种爬取贴文过程中的数据排重方法及终端
CN116701742B (zh) * 2023-06-25 2026-04-28 中储粮成都储藏研究院有限公司 分布式爬虫方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6691108B2 (en) * 1999-12-14 2004-02-10 Nec Corporation Focused search engine and method
US7454430B1 (en) * 2004-06-18 2008-11-18 Glenbrook Networks System and method for facts extraction and domain knowledge repository creation from unstructured and semi-structured documents
CN102646129A (zh) * 2012-03-09 2012-08-22 武汉大学 一种主题相关的分布式网络爬虫系统
CN103279492A (zh) * 2013-04-28 2013-09-04 乐视网信息技术（北京）股份有限公司 一种抓取网页的方法和装置
CN103761279A (zh) * 2014-01-09 2014-04-30 北京京东尚科信息技术有限公司 一种基于关键词检索的网络爬虫调度方法及系统

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7305610B1 (en) * 2000-04-06 2007-12-04 Google, Inc. Distributed crawling of hyperlinked documents
US7512878B2 (en) * 2004-04-30 2009-03-31 Microsoft Corporation Modular document format
US20150066895A1 (en) * 2004-06-18 2015-03-05 Glenbrook Networks System and method for automatic fact extraction from images of domain-specific documents with further web verification
US7769742B1 (en) * 2005-05-31 2010-08-03 Google Inc. Web crawler scheduler that utilizes sitemaps from websites
US8875249B2 (en) * 2006-03-01 2014-10-28 Oracle International Corporation Minimum lifespan credentials for crawling data repositories
CN100520778C (zh) * 2006-07-25 2009-07-29 腾讯科技（深圳）有限公司 一种互联网主题文件搜索方法、爬虫系统和搜索引擎
US20090254512A1 (en) * 2008-04-03 2009-10-08 Yahoo! Inc. Ad matching by augmenting a search query with knowledge obtained through search engine results
US8352260B2 (en) * 2008-09-10 2013-01-08 Jun Hyung Sung Multimodal unification of articulation for device interfacing
TW201013433A (en) * 2008-09-19 2010-04-01 Esobi Inc Filtering method for the same or similar documents
WO2011135456A2 (en) * 2010-04-28 2011-11-03 Patrick Frankham Systems and methods for using online resources to design a clinical study and recruit participants
US9246932B2 (en) * 2010-07-19 2016-01-26 Sitelock, Llc Selective website vulnerability and infection testing
US8868541B2 (en) * 2011-01-21 2014-10-21 Google Inc. Scheduling resource crawls
EP2812843A1 (en) * 2012-02-09 2014-12-17 AOL Inc. Systems and methods for testing online systems and content

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6691108B2 (en) * 1999-12-14 2004-02-10 Nec Corporation Focused search engine and method
US7454430B1 (en) * 2004-06-18 2008-11-18 Glenbrook Networks System and method for facts extraction and domain knowledge repository creation from unstructured and semi-structured documents
CN102646129A (zh) * 2012-03-09 2012-08-22 武汉大学 一种主题相关的分布式网络爬虫系统
CN103279492A (zh) * 2013-04-28 2013-09-04 乐视网信息技术（北京）股份有限公司 一种抓取网页的方法和装置
CN103761279A (zh) * 2014-01-09 2014-04-30 北京京东尚科信息技术有限公司 一种基于关键词检索的网络爬虫调度方法及系统

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111460254A (zh) * 2020-03-24 2020-07-28 南阳柯丽尔科技有限公司 一种基于多线程的网页爬取方法、装置、存储介质及设备
CN111460254B (zh) * 2020-03-24 2023-05-05 南阳柯丽尔科技有限公司 一种基于多线程的网页爬取方法、装置、存储介质及设备
CN113965555A (zh) * 2021-10-21 2022-01-21 北京值得买科技股份有限公司 一种参数化爬虫下载方法、装置、设备及存储介质
CN113965555B (zh) * 2021-10-21 2024-04-12 北京值得买科技股份有限公司 一种参数化爬虫下载方法、装置、设备及存储介质

Also Published As

Publication number Publication date
RU2645266C1 (ru) 2018-02-19
CN103761279A (zh) 2014-04-30
CN103761279B (zh) 2017-02-08
US20160328475A1 (en) 2016-11-10
US10185771B2 (en) 2019-01-22

Similar Documents

Publication Publication Date Title
WO2015103997A1 (zh) 一种基于关键词检索的网络爬虫调度方法及系统
KR102133951B1 (ko) 짧은 링크 처리 방법, 디바이스, 및 서버
CN107590188B (zh) 一种自动化垂直细分领域的爬虫爬取方法及其管理系统
US10089143B2 (en) Dynamic scheduling of tasks for collecting and processing data using job configuration data
US9064013B1 (en) Application of resource limits to request processing
US8676782B2 (en) Information collection apparatus, search engine, information collection method, and program
WO2018095351A1 (zh) 搜索处理方法及装置
KR20160030381A (ko) 웹 페이지 액세스 방법, 장치, 라우터, 프로그램 및 기록매체
CN105279272A (zh) 一种基于分布式网络爬虫的内容聚合方法
CN110069526A (zh) 用于分布式数据库查询引擎的系统和方法
CN106033428B (zh) 统一资源定位符的选择方法和统一资源定位符的选择装置
CN107885777A (zh) 一种基于协作式爬虫的抓取网页数据的控制方法及系统
CN102355488A (zh) 爬虫种子获取方法与设备及爬虫爬取方法与设备
CN109600385B (zh) 一种访问控制方法及装置
CN106776983B (zh) 搜索引擎优化装置和方法
CN105718307B (zh) 进程管理方法及进程管理装置
WO2018035799A1 (zh) 数据查询方法、应用和数据库服务器、中间件及系统
CN107580052B (zh) 自演进的网络自适应爬虫方法及系统
CN108520052B (zh) 慢查询信息检索方法、装置、服务器及可读存储介质
CN108664493B (zh) 统计url是否有效的方法、装置、电子设备和存储介质
CN104008213A (zh) 一种网页信息更新发现与统计的方法和装置
Hurst et al. Social streams blog crawler
JP2009211514A (ja) 関係情報取得システム、関係情報取得方法および関係情報取得用プログラム
RU2660593C2 (ru) Способ и сервер определения исходной ссылки на исходный объект
CN103957252A (zh) 云储存系统的日志获取方法及其系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15735407

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 15110564

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: IDP00201604531

Country of ref document: ID

ENP Entry into the national phase

Ref document number: 2016132573

Country of ref document: RU

Kind code of ref document: A

122 Ep: pct application non-entry in european phase

Ref document number: 15735407

Country of ref document: EP

Kind code of ref document: A1