WO2020006909A1 - 一种url去重方法及装置 - Google Patents

一种url去重方法及装置 Download PDF

Info

Publication number
WO2020006909A1
WO2020006909A1 PCT/CN2018/108715 CN2018108715W WO2020006909A1 WO 2020006909 A1 WO2020006909 A1 WO 2020006909A1 CN 2018108715 W CN2018108715 W CN 2018108715W WO 2020006909 A1 WO2020006909 A1 WO 2020006909A1
Authority
WO
WIPO (PCT)
Prior art keywords
url
list
downloaded
hash value
threshold
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/108715
Other languages
English (en)
French (fr)
Inventor
熊庆昌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Publication of WO2020006909A1 publication Critical patent/WO2020006909A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions

Definitions

  • the present application relates to the field of Internet technologies, and in particular, to a URL deduplication method and device.
  • the URL crawler refers to that the crawler system first carefully selects a part of web pages from Internet pages, uses the link addresses of these web pages as seed URLs, and places these seeds in the queue of URLs to be crawled. The crawler reads from the queue of URLs to be crawled in turn, and The URL is resolved by the domain name system (DNS), and the link address is converted into the IP address corresponding to the website server. Then give it the relative path name of the webpage to the webpage downloader, and the webpage downloader is responsible for downloading the page.
  • DNS domain name system
  • the web pages downloaded to the local area are stored in the page library, waiting for subsequent processing such as indexing; on the other hand, the URL of the downloaded web page is placed in a crawl queue, and this queue records the crawler system has downloaded URL of the webpage to avoid repeated crawling of the system.
  • URL deduplication refers to removing duplicate crawled URLs to avoid crawling the same web page multiple times. For example, for each given URL, map it to a physical address. When it is necessary to detect whether the given URL is duplicated, it is only necessary to determine whether the physical address corresponding to the given URL already exists. If it exists, it means that it has been downloaded, then the download is abandoned, otherwise the given URL is put into the crawl. Queue, waiting for download. Because many URLs in a website only have different parameter parts, and these URLs with only different parameter parts are likely to have been downloaded, but the physical addresses obtained after mapping these URLs are not the same. In this case, only the URL Corresponding physical addresses to determine whether these URLs are duplicates will lead to crawling many duplicate URLs, affecting the scanning efficiency of the web vulnerability scanning system.
  • the embodiments of the present application provide a URL deduplication method and device, which can reduce the number of repeated URLs downloaded, and improve the scanning efficiency of the WEB vulnerability scanning system while ensuring the deduplication accuracy rate.
  • An embodiment of the present application provides a URL deduplication method.
  • the method includes:
  • the parameter part of the first URL is removed to obtain the second URL, and the target generalization process is used to convert the same type of URLs.
  • the first list is used to store the URLs that have been downloaded in the history after the target is generalized;
  • the first occurrence number of the second URL in the second list is obtained, and the second list is used to store the downloaded URL in the history record after removing the parameter part. URL;
  • the URL to be downloaded is downloaded.
  • the embodiment of the present application can reduce the number of duplicate URLs downloaded by hierarchically deduplicating URLs, and improve the scanning efficiency of the WEB vulnerability scanning system while ensuring the accuracy of deduplication.
  • FIG. 1 is a schematic flowchart of a URL deduplication method according to an embodiment of the present application
  • FIG. 2a is a schematic diagram of a relationship between a first list and occurrences
  • FIG. 2b is a schematic diagram of the relationship between the second list and the number of occurrences
  • FIG. 3 is another schematic flowchart of a URL deduplication method according to an embodiment of the present application.
  • FIG. 4 is a schematic block diagram of a URL deduplication device according to an embodiment of the present application.
  • FIG. 5 is a schematic block diagram of a terminal according to an embodiment of the present application.
  • the question mark "?” Character is used to separate the path and parameter in the URL.
  • the logical and "&" character indicates the separator between the parameters specified in the URL.
  • the URL deduplication method may include steps:
  • the terminal may sort the parameter fields of the crawled URL to be downloaded according to the parameter names, and perform target generalization processing on the parameter values of the parameter fields of the URL to be downloaded to obtain the first URL.
  • the terminal may detect whether the same URL as the first URL1 exists in the first list. If it does not exist, it indicates that the first URL is not in the first list, and then the parameter part of the first URL is removed to obtain the second URL.
  • the target generalization process can be used to replace multiple consecutive characters of the same type in a URL with a single character. For example, consecutive numbers such as 145 can be replaced with the number 1, consecutive letters such as FK, aj, and dgA are replaced with the letter A, and special characters are replaced with the percent sign%.
  • the first list can be used to store URLs that have been downloaded in the historical record after the target generalization process. Special characters refer to characters other than numbers and letters, such as a question mark "?", An exclamation point "!, And so on. Because the first URL obtained by the target generalization process only reduces some variables in the parameter values of the URL to be downloaded, when the first URL is not in the first list, it means that the URL to be downloaded has not been downloaded, and the terminal further proceeds Repeat processing ensures the accuracy of deduplication.
  • the terminal detects whether the same URL as URL1 exists in the first list (URL4, URL6, and URL7), that is, whether URL1 exists in the first list because the first list URL1 does not exist, indicating that URL1 is not in the first list, the terminal deletes the parameter field in URL1 and obtains the second URL as http://xxx.pingan.com/cgi-bin/index1.html.
  • the terminal may sort the parameter fields of the crawled URL to be downloaded according to the parameter names, and perform target generalization processing on the parameter values of the parameter fields of the URL to be downloaded to obtain the first URL.
  • the terminal may use the message-digest algorithm 5 (MD5) to calculate the hash value of the first URL, and may detect whether the hash value of the first URL is in the first list. If the hash value of the first URL is not in the first list, the terminal may remove the parameter part of the first URL to obtain the second URL.
  • MD5 message-digest algorithm 5
  • the target generalization process can be used to replace multiple consecutive characters of the same type in a URL with a single character.
  • consecutive numbers such as 145 can be replaced with the number 1
  • consecutive letters such as FK, aj, and dgA are replaced with the letter A
  • special characters are replaced with the percent sign%.
  • the first list can be used to store the URLs that have been downloaded in the historical records after the target generalization processing, and then the hash values obtained after MD5 calculation.
  • Special characters refer to characters other than numbers and letters, such as a question mark "?", An exclamation point "!, And so on.
  • the hash function is a function that converts data of any size into data of a specific size
  • the hash value of the URL is stored in the downloaded list (the first list) instead of the full URL, which can reduce the storage space (because the full URL There are many characters, and the hash value is a fixed size data).
  • the terminal can use MD5 to calculate the hash value of URL1 as 03. Assuming that the first list includes hash values 01, 04, and 07, the terminal detects whether the hash value 03 of URL1 is in the first list (01, 04, and 07) because the first The hash value 03 does not exist in the list, the terminal deletes the parameter field in URL1 and obtains the second URL as http://xxx.pingan.com/cgi-bin/index1.html.
  • the terminal may sort the parameter fields of the crawled URL to be downloaded according to the parameter names, and perform target generalization processing on the parameter values of the parameter fields of the URL to be downloaded to obtain the first URL.
  • the terminal can detect whether the same URL as the first URL1 exists in the first list. If it exists, it indicates that the first URL is in the first list, and the terminal can add the number of occurrences of the first URL in the first list by adding 1. Get the second occurrence of the first URL in the first list.
  • the terminal may compare the magnitude relationship between the second number of occurrences and a preset second threshold. If the second number of occurrences is less than or equal to the second threshold, the terminal may download the URL to be downloaded.
  • the terminal may discard the URL to be downloaded, that is, the URL to be downloaded is not downloaded.
  • FIG. 2a it is a schematic diagram of the relationship between the first list and the number of occurrences. It is assumed that the elements in the first list are URL1, URL4, and URL7, the first URL is URL1, and the second threshold is 7.
  • the terminal detects whether the same URL as URL1 exists in the first list (URL1, URL4, and URL7), that is, whether URL1 exists in the first list, because URL1 exists in the first list, and the terminal counts the number of occurrences of URL1 in the first list. Add 1 to 1 to get the second occurrence 2 of URL1 in the first list. Because the second occurrence number 2 is less than the second threshold value 7, the terminal can download the URL to be downloaded.
  • the first list may be empty (because there is no downloaded URL at this time).
  • the terminal may detect whether the second URL is in the second list, and if so, may increase the number of times that the second URL appears in the second list by 1 to obtain the second URL in the second list. The first occurrence in the second list. If not, that is, the second URL is not in the second list, the terminal may directly download the URL to be downloaded.
  • the second list is used to store the URLs obtained after removing the parameter part from the downloaded URLs in the history.
  • the terminal may use MD5 to calculate the hash value of the second URL, and may detect whether the hash value of the second URL is in the second list, and if so, may hash the second URL.
  • the number of occurrences of the value in the second list is increased by 1 to obtain the first number of occurrences of the hash value of the second URL in the second list. If not, that is, the hash value of the second URL is not in the second list, the terminal can directly download the URL to be downloaded.
  • FIG. 2b it is a schematic diagram of the relationship between the second list and the number of occurrences.
  • the elements in the second list are 02 and 06
  • the second URL is URL2
  • the terminal calculates the hash value of URL2 as 02
  • the terminal detects that the hash value of URL2 is in the first list
  • the terminal sets the hash value of URL2 as 02
  • the number of occurrences 2 in the first list is increased by 1 to obtain the first occurrence 3 of the hash value 02 of URL2 in the second list.
  • the terminal when the terminal detects that the second URL is not in the second list, the terminal may add the second URL to the second list to form a new second list, that is, update the second list At the same time, the number of times that the second URL appears in the second list can be set to 1, and the URL to be downloaded can be downloaded. After the terminal downloads the URL to be downloaded or discards the URL to be downloaded, the terminal uses the latest second list to detect whether the next URL to be downloaded has been downloaded. By continuously updating the second list, the terminal can more accurately filter duplicate URLs, and can further improve scanning efficiency.
  • the second list may be empty (because there is no downloaded URL at this time).
  • the terminal may compare the magnitude relationship between the acquired first occurrences and the preset first threshold. If the first number of occurrences is less than or equal to the first threshold, the terminal may directly download the URL to be downloaded. If the first occurrence number is greater than the first threshold, the terminal may directly discard the URL to be downloaded, that is, the URL to be downloaded is not downloaded.
  • the first threshold may be smaller than the second threshold, and the first threshold may be an integer greater than or equal to zero.
  • the terminal determines whether the second URL obtained by removing the parameter part of the first URL has been downloaded, thereby determining whether the URL to be downloaded needs to be downloaded, and the URLs to be downloaded with different parameter parts can be filtered out, that is, the URLs to be downloaded are reduced. In the variable, there are many URLs to be downloaded that are discarded, thereby reducing the number of repeated URLs to be downloaded, and further improving scanning efficiency and processing efficiency while ensuring accuracy.
  • the second URL is obtained by removing the parameter part of the first URL, and the target generalization processing is used for Replace multiple consecutive characters of the same type in the URL with a single character.
  • the first list is used to store the URLs that have been downloaded in the history after the target is generalized. After detecting that the second URL is in the In the second list, the first occurrence number of the second URL in the second list is obtained.
  • the second list is used to store the URL obtained by removing the parameter part of the downloaded URL in the history record. If the URL to be downloaded is less than or equal to the first threshold, the number of repeated URLs can be reduced, and the scanning efficiency of the WEB vulnerability scanning system can be improved while ensuring the accuracy of deduplication.
  • the URL deduplication method may include steps:
  • S301 Perform first generalization processing on the URL to be downloaded to obtain URL1 after the first generalization processing.
  • URL1 may represent the first URL in the embodiment of the present application.
  • URL1 and URL2 represent URLs that have undergone different processing.
  • the terminal may sort the parameter part of the URL to be downloaded according to the parameter name, and perform a first generalization process on the parameter value of the parameter part of the URL to be downloaded to obtain URL1.
  • the terminal may detect whether the same URL as the URL1 exists in the first list. If it does not exist, it indicates that the URL1 is not in the first list, and then the parameter value in the URL1 parameter part may be deleted to obtain the URL2.
  • the terminal can add 1 to the number of times that URL1 appears in the first list to obtain the second number of times that URL1 appears in the first list, and can judge the Whether the second number of occurrences is less than or equal to the second threshold, if yes (that is, the second number of occurrences is less than or equal to the second threshold), download the URL to be downloaded; if not (that is, the second number of occurrences is greater than the second threshold) ), The URL to be downloaded is discarded, that is, the URL to be downloaded is not downloaded.
  • the terminal determines whether the URL1 obtained through the first generalization process has been downloaded, thereby determining whether to download the URL to be downloaded, and can filter out the URLs to be downloaded with the same data format in the parameter values, reducing the number of downloads to duplicate URLs.
  • the first generalization process is used to replace multiple consecutive characters of the same type in the URL to be downloaded with a single character. For example, consecutive numbers such as 145 can be replaced with the number 1, consecutive letters such as FK, aj, and dgA are replaced with the letter A, and special characters are replaced with the symbol%.
  • the first list can be used to store the URLs obtained from the downloaded URLs in the history record after the first generalization process.
  • the terminal may use MD5 to calculate the hash value of the URL1, and may detect whether the hash value of the URL1 is in the first list, and if so, may delete the URL1.
  • the parameter value in the parameter section gets URL2.
  • the number of occurrences of the hash value of URL1 in the first list may be increased by 1 to obtain the second number of occurrences of the hash value of URL1 in the first list, and whether the second number of occurrences is less than Or equal to the second threshold, if yes (that is, the second occurrence is less than or equal to the second threshold), download the URL to be downloaded; if not (that is, the second occurrence is greater than the second threshold), discard the above Download URL, that is, the URL to be downloaded is not downloaded.
  • the first list may be used to store a hash value of the URL obtained after the URL of the history record is downloaded through the first generalization process.
  • the second threshold is an integer greater than 0.
  • the hash function is a function that converts data of any size into data of a specific size
  • the hash value of the URL is stored in the downloaded list (the first list) instead of the full URL, which can reduce the storage space (because the full URL There are many characters, and the hash value is a fixed-size data).
  • the processing efficiency can be improved.
  • the terminal when the terminal detects that the URL1 is not in the first list, the terminal may add the URL1 to the first list to form a new first list, that is, update the first list, and at the same time, may The number of occurrences of the URL1 in the first list is set to 1, and the parameter value in the parameter part of the URL1 can be deleted to obtain URL2.
  • the terminal After the terminal downloads the URL to be downloaded or discards the URL to be downloaded, the terminal uses the latest first list to detect whether the next URL to be downloaded has been downloaded. By constantly updating the first list, the terminal can more accurately filter out duplicate URLs, and can further improve scanning efficiency.
  • the first list includes URL5, URL6, and URL7.
  • the terminal may add the URL1 to the first list.
  • the updated first list includes URL1, URL5, URL6, and URL7.
  • the terminal may also set the number of occurrences of URL1 in the first list to 1.
  • URL3 may represent a second URL in the embodiment of the present application.
  • URL2 and URL3 represent URLs that have undergone different processing.
  • the terminal can detect whether the same URL as the above URL2 exists in the second list, and if it exists, it indicates that the URL2 is in the second list, then the number of occurrences of the URL2 in the second list can be increased by 1 to obtain the URL2 in the The first number of occurrences in the second list, and it can be detected whether the first number of occurrences is less than or equal to the first threshold, and if it is (ie, the first number of occurrences is less than or equal to the first threshold), the URL to be downloaded is downloaded If not (that is, the first occurrence number is greater than the first threshold), the URL to be downloaded is discarded, that is, the URL to be downloaded is not downloaded.
  • the terminal detects that the URL2 does not exist in the second list, it indicates that the URL2 is not in the second list, and then the parameter part in the URL2 can be deleted to obtain the URL3.
  • the second list can be used to store the URLs that have been downloaded in the historical record after removing the parameter part.
  • the terminal judges whether the URL2 obtained by removing the parameter value in the parameter part of the URL1 has been downloaded, thereby determining whether it is necessary to download the URL to be downloaded, and can filter out URLs to be downloaded with different parameter values, further reducing downloading to duplicate URLs.
  • the number of scans improves scanning efficiency while ensuring accuracy.
  • the first threshold is less than or equal to the second threshold, and the first threshold may be an integer greater than 0.
  • the first generalization process only reduces the variables in the parameter value of the URL to be downloaded, at this time there are more variables in URL1, there are fewer filtered URLs, and there are more downloaded URLs.
  • URL1 is in the first list Occurs more times. Therefore, the first threshold value is smaller than the second threshold value, which can ensure that the URLs that are not filtered out in steps S301 to S302 can be filtered out after removing the parameter values in the parameter part to achieve the purpose of hierarchical deduplication.
  • the terminal may use MD5 to calculate the hash value of the URL2, and may detect whether the hash value of URL2 is in the second list. If so, the hash value of URL2 may be in the second list. Plus 1 to the number of occurrences of the URL to obtain the first occurrence of the hash value of the URL2 in the second list, and to determine whether the first occurrence is less than or equal to the first threshold, and if so (ie, the first occurrence is less than or Equal to the first threshold), download the URL to be downloaded; if not (that is, the first number of occurrences is greater than the first threshold), discard the URL to be downloaded, ie, do not download the URL to be downloaded.
  • the second list can be used to store the URLs that have been downloaded in the historical record, excluding the parameter values, and the hash values obtained after MD5 calculations. Because the hash function is a function that converts data of any size into data of a specific size, and the hash value of the URL is stored in the downloaded list (second list), rather than the complete URL, the storage space can be further reduced. When detecting whether URL2 is in the second list, the processing efficiency can be further improved.
  • URL2 is: http://xxx.pingan.com/cgi-bin/index1.html? param1 & param2
  • the terminal calculates the hash value of URL2 as 04
  • the terminal detects that the hash value 04 of URL2 is not in the second list
  • the terminal directly deletes the parameter part in URL2 to get URL3
  • URL3 is: http://xxx.pingan.com/ cgi-bin / index1.html.
  • the terminal when the terminal detects that the URL2 is not in the second list, the terminal may add the URL2 to the second list to form a new second list, that is, update the second list, and at the same time, may The number of occurrences of URL2 in the second list is set to 1, and the parameter part of the URL2 can be removed to obtain URL3.
  • the terminal After the terminal downloads the URL to be downloaded or discards the URL to be downloaded, the terminal uses the latest second list to detect whether the next URL to be downloaded has been downloaded. By continuously updating the second list, the terminal can more accurately filter duplicate URLs, and can further improve scanning efficiency.
  • the second list may be empty (because there is no downloaded URL at this time).
  • URL3 may represent a second URL in the embodiment of the present application.
  • URL3 and URL4 represent URLs that have undergone different processing.
  • the terminal can detect whether the same URL as the above URL3 exists in the third list. If it exists, it indicates that the URL3 is in the third list, and then the number of occurrences of the URL3 in the third list can be increased by 1 to obtain the URL3 in the third list.
  • the third number of occurrences in the third list and it can be detected whether the third number of occurrences is less than or equal to the third threshold, and if it is (ie, the third number of occurrences is less than or equal to the third threshold), the URL to be downloaded is downloaded If not (that is, the third occurrence number is greater than the third threshold), the URL to be downloaded is discarded, that is, the URL to be downloaded is not downloaded. If the terminal detects that the URL3 does not exist in the third list, it indicates that the URL3 is not in the third list, and then a second generalization process may be performed on the path part of the URL3 to obtain the URL4.
  • the second generalization process is used to replace at least one character of a target type in a path part of URL3 with a target character, such as replacing one or more numbers in the path part of URL3 with a preset single number "1".
  • the third list can be used to store the URLs that have been downloaded in the historical records, except for the parameter part.
  • the terminal judges whether the URL3 obtained by removing the parameter part of URL2 has been downloaded, thereby determining whether the URL to be downloaded needs to be downloaded, and the URLs to be downloaded with different parameter parts can be filtered out, that is, the variables in the URL to be downloaded are reduced, and discarded There are many URLs to be downloaded, which reduces the number of duplicate URLs downloaded, and further improves scanning efficiency while ensuring accuracy.
  • the third threshold value may be less than or equal to the first threshold value, the first threshold value is less than or equal to the second threshold value, and the third threshold value may be an integer greater than 0. Because removing the parameter value in the parameter part only reduces the variables in the parameter part of the URL to be downloaded. At this time, there are more variables in URL2, fewer URLs are filtered out, and more URLs are downloaded. URL2 is in the second list. Appears more often. Therefore, the third threshold value is smaller than the first threshold value, which can ensure that the URLs not filtered out in step S303 to step S304 can be filtered out after removing the parameter part, thereby achieving the purpose of hierarchical deduplication.
  • the terminal may use MD5 to calculate the hash value of the URL3, and may detect whether the hash value of the URL3 is in the third list. If so, the terminal may calculate the hash value of the URL3 in the third list. Plus 1 to the number of occurrences of the URL, to get the third occurrence of the hash value of the URL3 in the third list, and to determine whether the third occurrence is less than or equal to the third threshold, and if so (that is, the third occurrence is less than or equal to Equal to the third threshold), download the URL to be downloaded; if not (that is, the third occurrence number is greater than the third threshold), discard the URL to be downloaded, ie, do not download the URL to be downloaded.
  • a second generalization process may be performed on the URL3 to obtain the URL4.
  • the second generalization process is used to replace at least one character of a target type in a path part of URL3 with a target character, such as replacing one or more numbers in the path part of URL3 with a preset single number "1".
  • the third list can be used to store the URLs that have been downloaded in the historical record, excluding the parameter part, and the hash values obtained after MD5 calculation.
  • the hash function is a function that converts data of any size into data of a specific size, and the hash value of the URL is stored in the downloaded list (third list) instead of the complete URL, which can further reduce the storage space.
  • the terminal when the terminal detects that the URL3 is not in the third list, the terminal may add the URL3 to the third list to form a new third list, that is, update the third list, and at the same time, may The number of occurrences of the URL3 in the third list is set to 1, and a second generalization process may be performed on the URL3 to obtain the URL4.
  • the terminal After the terminal downloads the URL to be downloaded or discards the URL to be downloaded, the terminal uses the latest third list to detect whether the next URL to be downloaded has been downloaded. By continuously updating the third list, the terminal can filter out the duplicate URLs more accurately and further improve the scanning efficiency.
  • the third list may be empty (because there is no downloaded URL at this time).
  • the terminal may detect whether the same URL4 as the above URL4 exists in the fourth list. If it exists, it indicates that the URL4 is in the fourth list, and then the URL4 may appear in the fourth list. Add 1 to the number of times to get the fourth occurrence of URL4 in the fourth list, and you can detect whether the fourth occurrence is less than or equal to the fourth threshold, and if so (that is, the fourth occurrence is less than or equal to the fourth threshold ), Download the URL to be downloaded, if not (that is, the fourth occurrence number is greater than the fourth threshold), discard the URL to be downloaded, that is, do not download the URL to be downloaded.
  • the terminal detects that the URL4 does not exist in the fourth list, it indicates that the URL4 is not in the fourth list, and can directly download the URL to be downloaded.
  • the fourth list may be used to store the URLs obtained by removing the parameter part of the downloaded URLs in the history and performing the second generalization process.
  • the terminal determines whether the URL 4 obtained after removing the parameter part and undergoing the second generalization process has been downloaded, thereby determining whether the URL to be downloaded needs to be downloaded, and the URLs to be downloaded that are different in the path part can be filtered, thereby reducing the URLs to be downloaded.
  • the number of URLs to be downloaded is discarded, and the number of duplicate URLs downloaded is small, which improves the scanning efficiency while ensuring accuracy.
  • the fourth threshold is less than or equal to the third threshold
  • the third threshold may be less than or equal to the first threshold
  • the first threshold is less than or equal to the second threshold
  • the fourth threshold is greater than or equal to An integer of 0.
  • the fourth threshold value is smaller than the third threshold value, which can ensure that the URLs that are not filtered out in steps S305 to S306 can be filtered out after the second generalization process, thereby achieving the purpose of hierarchical deduplication.
  • the terminal may use MD5 to calculate the hash value of the URL4, and may detect whether the hash value of the URL4 is in the fourth list. If so, the terminal may calculate the hash value of the URL4 in the fourth list. The number of occurrences of the URL4 is increased by 1 to obtain the fourth occurrence of the hash value of the URL4 in the fourth list, and it can be determined whether the fourth occurrence is less than or equal to the fourth threshold. If so (that is, the fourth occurrence is less than or equal to Equal to the fourth threshold), download the URL to be downloaded; if not (that is, the fourth occurrence is greater than the fourth threshold), discard the URL to be downloaded, ie, do not download the URL to be downloaded.
  • the fourth list can be used to store the URLs that have been downloaded in the historical records, excluding the parameter part, and obtained after the second generalization process, and then the hash value obtained by MD5 calculation.
  • the hash function is a function that converts data of any size into data of a specific size, and the hash value of the URL is stored in the downloaded list (fourth list) instead of the complete URL, which can further reduce the storage space.
  • the terminal may add the URL4 to the fourth list to form a new fourth list, that is, update the fourth list, and at the same time,
  • the number of occurrences of URL4 in the fourth list is set to 1, and the URL to be downloaded can be downloaded.
  • the terminal uses the latest fourth list to detect whether the next URL to be downloaded has been downloaded.
  • the fourth list may be empty (because there is no downloaded URL at this time).
  • the URLs to be downloaded are hierarchically deduplicated, and the variables in the URLs to be downloaded are gradually reduced, and whether the URLs to be downloaded are downloaded is determined according to the URL after the variables are gradually reduced. If at this level, the URLs to be downloaded are determined If it has been downloaded, the URL to be downloaded is discarded. If the level determines that the URL to be downloaded is not downloaded, the URL to be downloaded is downloaded. If the level cannot determine whether the URL to be downloaded has been downloaded, it proceeds to the next level until it is determined The download URL is downloaded or not downloaded. Through a more detailed hierarchical deduplication scheme, it can not only reduce the number of downloads to duplicate URLs, improve the scanning efficiency of the WEB vulnerability scanning system, but also improve the accuracy of deduplication.
  • FIG. 4 it is a schematic block diagram of a URL deduplication device according to an embodiment of the present application.
  • the URL deduplication device provided in the embodiment of the present application includes:
  • the processing module 10 is configured to: when it is detected that the first URL obtained after the target generalization process is not in the first list, remove the parameter part of the first URL to obtain the second URL.
  • the target generalization process is used to replace multiple consecutive characters of the same type in the URL with a single character, and the first list is used to store the URLs that have been downloaded in the historical record after the target generalization process.
  • the first obtaining module 20 is configured to obtain the first occurrence number of the second URL in the second list when the second URL is detected in the second list.
  • the second list is used to store the URLs obtained by removing the parameter part from the downloaded URLs in the history.
  • the download module 30 is configured to download the URL to be downloaded when the first number of occurrences is less than or equal to a first threshold.
  • the device further includes:
  • the second obtaining module 40 is configured to obtain a second URL of the first URL in the first list when detecting that the first URL obtained after the URL to be downloaded is processed by the target generalization is in the first list.
  • the number of occurrences; the downloading module 30 is further configured to download the URL to be downloaded when the second number of occurrences is less than or equal to a second threshold.
  • the second threshold is greater than or equal to the first threshold, and the first threshold is an integer greater than or equal to zero.
  • the processing module 10 is specifically configured to perform target generalization processing on the URL to be downloaded to obtain a first URL; use a target hash function to calculate a hash value of the first URL; and detect the first URL. Whether the hash value of the URL is in the first list; when the hash value of the first URL is not in the first list, removing a parameter part of the first URL to obtain a second URL.
  • the first list is used to store the hash value of the URL obtained after the downloaded URL in the historical record is processed by the target generalization.
  • the first URL includes a protocol, a server name, a path, and a parameter field.
  • the processing module is specifically configured to detect that the first URL obtained after the URL to be downloaded is processed by the target is not in the first list. In the middle, the parameter field in the first URL is deleted to obtain the second URL.
  • the first obtaining module 20 is specifically configured to: use a target hash function to calculate a hash value of the second URL; and detect whether the hash value of the second URL is in the second list; If the hash value of the second URL is in the second list, the first occurrence number of the hash value of the second URL in the second list is obtained.
  • the second list is used to store the hash value of the URL obtained after removing the parameter part of the downloaded URL in the history record.
  • the downloading module 30 is further configured to download the URL to be downloaded when the second URL is not in the second list.
  • the URL deduplication device may implement the implementation manner provided by each step in the implementation manner provided in FIG. 1 or FIG. 3 through the foregoing modules to implement the functions implemented in the foregoing embodiments.
  • the URL deduplication device may implement the implementation manner provided by each step in the implementation manner provided in FIG. 1 or FIG. 3 through the foregoing modules to implement the functions implemented in the foregoing embodiments.
  • the corresponding description provided by each step in the method embodiment shown in FIG. 1 or FIG. 3 is not described herein again.
  • the URL deduplication device may obtain the second URL by removing a parameter part of the first URL when it is detected that the first URL obtained after the target URL is generalized is not in the first list.
  • the target generalization process is used to replace multiple consecutive characters of the same type in the URL with a single character.
  • the first list is used to store the URLs that have been downloaded in the historical record after the target generalization process.
  • the second URL When the second URL is in the second list, obtain the first occurrence number of the second URL in the second list, and the second list is used to store the URL obtained by removing the parameter part of the downloaded URL in the history record
  • the first occurrence number is less than or equal to the first threshold
  • downloading the URL to be downloaded can reduce the number of duplicate URLs downloaded, and improve the scanning efficiency of the WEB vulnerability scanning system while ensuring the accuracy of deduplication.
  • the terminal in the embodiment of the present application may include: one or more processors 501 and a memory 502.
  • the processor 501 and the memory 502 are connected via a bus 503.
  • the memory 502 is configured to store a computer program, where the computer program includes program instructions, and the processor 501 is configured to execute the program instructions stored in the memory 502.
  • the processor 501 is configured to call the program instruction to execute:
  • the parameter part of the first URL is removed to obtain the second URL, and the target generalization process is used to convert the same type of URLs.
  • the first list is used to store the URLs that have been downloaded in the history after the target is generalized;
  • the first occurrence number of the second URL in the second list is obtained, and the second list is used to store the downloaded URL in the history record after removing the parameter part. URL;
  • the URL to be downloaded is downloaded.
  • the processor 501 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs) ), Application specific integrated circuit (ASIC), ready-made programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • DSPs digital signal processors
  • ASIC Application specific integrated circuit
  • FPGA ready-made programmable gate array
  • FPGA field-programmable gate array
  • a general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the memory 502 may include a read-only memory and a random access memory, and provide instructions and data to the processor 501. A part of the memory 502 may further include a non-volatile random access memory. For example, the memory 502 may also store information of a device type.
  • the processor 501 described in the embodiment of the present application may perform the implementation manner described in the uniform resource locator URL deduplication method provided in the embodiment of the present application, and may also perform the URL delineation described in the embodiment of the present application.
  • the implementation manner of the heavy device is not repeated here.
  • An embodiment of the present application further provides a computer-readable storage medium.
  • the computer-readable storage medium stores a computer program.
  • the computer program includes program instructions. When the program instructions are executed by a processor, the unity shown in FIG. 1 or FIG. 3 is implemented.
  • the resource locator URL deduplication method For details, please refer to the description of the embodiment shown in FIG. 1 or FIG. 3, and details are not described herein again.
  • the computer-readable storage medium may be the uniform resource locator URL deduplication device or an internal storage unit of an electronic device, such as a hard disk or a memory of the electronic device.
  • the computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card provided on the electronic device, Flash card, etc.
  • the computer-readable storage medium may include both an internal storage unit of the electronic device and an external storage device.
  • the computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device.
  • the computer-readable storage medium may also be used to temporarily store data that has been or will be output.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Transfer Between Computers (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请实施例公开了一种统一资源定位符URL去重及装置，其中方法包括：在检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，除去该第一URL中的参数部分得到第二URL，又在检测到该第二URL在该第二列表中，获取该第二URL在该第二列表中的第一出现次数，当该第一出现次数小于或等于第一阈值，就下载该待下载URL。采用本申请实施例，可以减少下载到重复URL的数量，在保证去重准确率的同时提高WEB漏洞扫描系统的扫描效率。

Description

一种URL去重方法及装置
本申请要求于2018年7月5日提交中国专利局、申请号为2018107337160、申请名称为“一种统一资源定位符URL去重方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及互联网技术领域，尤其涉及一种URL去重方法及装置。
背景技术
目前业内很多万维网(world wide web，WEB)漏洞扫描系统，都有自己的统一资源定位符(uniform resource locator，URL)爬虫，也有自己的URL去重方法。URL爬虫指爬虫系统首先从互联网页面中精心选择一部分网页，以这些网页的链接地址作为种子URL，将这些种子放入待抓取URL队列中，爬虫从待抓取URL队列依次读取，并将URL通过域名系统(domain name system，DNS)解析，把链接地址转换为网站服务器对应的IP地址。然后将其和网页相对路径名称交给网页下载器，网页下载器负责页面的下载。对于下载到本地的网页，一方面将其存储到页面库中，等待建立索引等后续处理；另一方面将下载网页的URL放入已抓取队列中，这个队列记录了爬虫系统已经下载过的网页URL，以避免系统的重复抓取。
URL去重指的是将重复抓取的URL去除，避免多次抓取同一网页。例如，对每一个给定的URL，将其映射到某个物理地址上。当需要检测该给定URL是否重复时，只需判断该给定URL对应的物理地址是否已经存在，若存在，说明已经被下载过，则放弃下载，否则将该给定URL放入待抓取队列，等待下载。由于一个网站中的很多URL仅是参数部分不同，而这些仅是参数部分不同的URL很大可能已经被下载过，但这些URL经过映射后得到的物理地址是不相同的，这时仅通过URL对应的物理地址来判断这些URL是否重复将导致爬取到很多重复的URL，影响WEB漏洞扫描系统的扫描效率。
发明内容
本申请实施例提供一种URL去重方法及装置，可减少下载到重复URL的数量，在保证去重准确率的同时提高WEB漏洞扫描系统的扫描效率。
本申请实施例提供了一种URL去重方法，该方法包括：
当检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，除去该第一URL中的参数部分得到第二URL，该目标泛化处理用于将URL中相同类型的多个连续字符替换为单个字符，该第一列表用于存储历史记录中已下载的URL经过该目标泛化处理后得到的URL；
若检测到该第二URL在该第二列表中，获取该第二URL在该第二列表中的第一出现次数，该第二列表用于存储历史记录中已下载的URL除去参数部分后得到的URL；
若该第一出现次数小于或等于第一阈值，则下载该待下载URL。
本申请实施例通过对URL进行分级去重，可以减少下载到重复URL的数量，在保证去重准确率的同时提高WEB漏洞扫描系统的扫描效率。
附图说明
图1是本申请实施例提供的URL去重方法的一示意流程图；
图2a是第一列表和出现次数的关系示意图；
图2b是第二列表和出现次数的关系示意图；
图3是本申请实施例提供的URL去重方法的另一示意流程图；
图4是本申请实施例提供的URL去重装置的一示意性框图；
图5是本申请实施例提供的终端的一示意性框图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都在本申请保护的范围。
在介绍本申请实施例之前，先介绍URL的数据结构。一个URL的结构通常可以为“协议://服务器名称(IP地址)/路径？参数”，例如：http://xxx.pingan.com/cgi-bin/index1.html？param1＝value1&param2＝value2，其中param1＝value1&param2＝value2表示这个URL的参数字段(参数部分)，URL的参数字段由参数名和参数值组成，param1和param2表示参数名，value1和value2表示参数值，参数值可为数字、字母(包括大小写)、特殊字符(指除数字、字母之外的字符)和/或它们的组合。问号“？”字符用于分隔URL中的路径和参数，逻辑与“&”字符表示URL中指定的参数间的分隔符。
下面将结合图1至图5，对本申请实施例提供的URL去重方法及装置进行说明。
参见图1，是本申请实施例提供的URL去重方法的一示意流程图。如图1所示，该URL去重方法可包括步骤：
S101，当检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，除去第一URL中的参数部分得到第二URL。
在一些可行的实施方式中，终端可以将爬取到的待下载URL的参数字段按照参数名进行排序，并对该待下载URL参数字段的参数值进行目标泛化处理，得到第一URL。终端可以检测第一列表中是否存在与该第一URL1相同的URL，若不存在，说明该第一URL不在该第一列表中，则除去该第一URL中的参数部分得到第二URL。其中，该目标泛化处理可以用于将URL中相同类型的多个连续字符替换为单个字符。例如，可将连续的数字如145替换为数字1，连续的字母如FK、aj、dgA都替换为字母A，特殊字符都替换为百分号％。该第一列表可以用于存储历史记录中已下载的URL经过该目标泛化处理后得到的URL。特殊字符指除数字和字母外的字符，如问号“？”、感叹号“！”等。因为经过目标泛化处理得到的第一URL只减少了待下载URL参数值中的部分变量，所以当第一URL不在第一列表中时，说明待下载URL未被下载过，终端再进行进一步去重处理，保证了去重的准确率。
例如，终端爬取到的待下载URL为http://xxx.pingan.com/cgi-bin/index1.html？param1＝v167！ABD&param2＝val_ue2，终端可以将待下载URL的参数部分按照参数名进行 排序，以保证待下载URL的参数部分顺序排列，再将待下载URL的参数值中连续的数字替换为预设的单个数字“1”、连续的字母替换为预设的单个字母“A”、特殊字符替换为预设的单个特殊字符“％”，得到URL1(第一URL)为http://xxx.pingan.com/cgi-bin/index1.html？param1＝v1％A&param2＝A％A2。假设第一列表中包括URL4、URL6以及URL7这三个URL，终端检测第一列表(URL4、URL6以及URL7)中是否存在与URL1相同的URL，即第一列表中是否存在URL1，因为第一列表中不存在URL1，说明URL1不在第一列表中，终端就删除URL1中的参数字段得到第二URL为http://xxx.pingan.com/cgi-bin/index1.html。
在一些可行的实施方式中，终端可以将爬取到的待下载URL的参数字段按照参数名进行排序，并对该待下载URL参数字段的参数值进行目标泛化处理，得到第一URL。终端可以利用消息-摘要算法5(message-digest algorithm 5，MD5)计算上述第一URL的哈希(hash)值，并可以检测该第一URL的hash值是否在该第一列表中。若该第一URL的hash值不在第一列表中，终端可以除去该第一URL中的参数部分得到第二URL。其中，该目标泛化处理可以用于将URL中相同类型的多个连续字符替换为单个字符。例如，可将连续的数字如145替换为数字1，连续的字母如FK、aj、dgA都替换为字母A，特殊字符都替换为百分号％。该第一列表可以用于存储历史记录中已下载的URL经过该目标泛化处理后得到的URL，再经过MD5计算后得到的hash值。特殊字符指除数字和字母外的字符，如问号“？”、感叹号“！”等。因为hash函数是将任意大小的数据转换成特定大小的数据的函数，且已下载列表(第一列表)中存储的是URL的hash值，而不是完整的URL，可以减少存储空间(因为完整URL字符较多，而hash值是固定大小的数据)。
例如，终端爬取到的待下载URL为http://xxx.pingan.com/cgi-bin/index1.html？param1＝v167！ABD&param2＝val_ue2，终端可以将待下载URL的参数部分按照参数名进行排序，以保证待下载URL的参数部分顺序排列，再将待下载URL的参数值中连续的数字替换为预设的单个数字“1”、连续的字母替换为预设的单个字母“A”、特殊字符替换为预设的单个特殊字符“％”，得到URL1(第一URL)为http://xxx.pingan.com/cgi-bin/index1.html？param1＝v1％A&param2＝A％A2。终端可以利用MD5计算URL1的hash值为03，假设第一列表中包括hash值01、04以及07，终端检测URL1的hash值03是否在第一列表(01、04以及07)中，因为第一列表中不存在hash值03，终端就删除URL1中的参数字段得到第二URL为http://xxx.pingan.com/cgi-bin/index1.html。
在一些可行的实施方式中，终端可以将爬取到的待下载URL的参数字段按照参数名进行排序，并对该待下载URL参数字段的参数值进行目标泛化处理，得到第一URL。终端可以检测第一列表中是否存在与该第一URL1相同的URL，若存在，说明该第一URL在该第一列表中，终端可以将该第一URL在该第一列表中的出现次数加1，得到该第一URL在该第一列表中的第二出现次数。终端可以比较该第二出现次数与预设的第二阈值之间的大小关系，若该第二出现次数小于或等于该第二阈值，则终端可以下载该待下载URL。若该第二出现次数大于该第二阈值，则终端可以丢弃该待下载URL，即不下载该待下载URL。例如，如图2a所示，是第一列表与出现次数的关系示意图。其中，假设第一列表中的元素 为URL1、URL4以及URL7，第一URL为URL1，第二阈值为7。终端检测第一列表(URL1、URL4以及URL7)中是否存在与URL1相同的URL，即第一列表中是否存在URL1，因为第一列表中存在URL1，终端就将URL1在第一列表中的出现次数1加1，得到URL1在第一列表中的第二出现次数2。因为第二出现次数2小于第二阈值7，故终端可以下载该待下载URL。
在一些可行的实施方式中，若上述待下载URL为某个网站中爬取到的第一个URL，那么第一列表可以为空(因为此时没有已下载的URL)。
S102，若检测到第二URL在第二列表中，获取第二URL在第二列表中的第一出现次数。
在一些可行的实施方式中，终端可以检测上述第二URL是否在第二列表中，若是，则可以将该第二URL在该第二列表中的出现次数加1，得到该第二URL在该第二列表中的第一出现次数。若不是，即上述第二URL不在第二列表中，终端可以直接下载上述待下载URL。其中，该第二列表用于存储历史记录中已下载的URL除去参数部分后得到的URL。
在一些可行的实施方式中，终端可以利用MD5计算上述第二URL的hash值，并可以检测该第二URL的hash值是否在该第二列表中，若是，则可以将该第二URL的hash值在该第二列表中的出现次数加1，得到该第二URL的hash值在该第二列表中的第一出现次数。若不是，即上述第二URL的hash值不在第二列表中，终端可以直接下载上述待下载URL。
例如，如图2b所示，是第二列表与出现次数的关系示意图。其中，假设第二列表中的元素为02和06，第二URL为URL2，终端计算URL2的hash值为02，终端检测到URL2的hash值02在第一列表中，终端将URL2的hash值02在第一列表中的出现次数2加1，得到URL2的hash值02在第二列表中的第一出现次数3。
在一些可行的实施方式中，终端在检测出该第二URL不在该第二列表中时，终端可以将该第二URL加入该第二列表以形成新的第二列表，即更新该第二列表，同时可以将该第二URL在该第二列表中的出现次数置为1，并可以下载上述待下载URL。终端在下载上述待下载URL或丢弃上述待下载URL之后，终端利用最新的第二列表检测下一个待下载URL是否被下载过。终端通过不断更新第二列表，可以更准确地过滤掉重复的URL，并且可以进一步提高扫描效率。
在一些可行的实施方式中，若上述待下载URL为某个网站中爬取到的第一个URL，那么第二列表可以为空(因为此时没有已下载的URL)。
S103，若第一出现次数小于或等于第一阈值，则下载待下载URL。
在一些可行的实施方式中，终端可以比较上述获取到的第一出现次数与预设的第一阈值之间的大小关系。若该第一出现次数小于或等于该第一阈值，则终端可以直接下载上述待下载URL。若该第一出现次数大于该第一阈值，则终端可以直接丢弃上述待下载URL，即不下载该待下载URL。其中，该第一阈值可以小于上述第二阈值，该第一阈值可以为大于或等于0的整数。终端通过判断除去第一URL中的参数部分得到的第二URL是否已经被下载过，从而判断出是否需要下载待下载URL，可以过滤掉参数部分不相同的待下载URL，即减少了待下载URL中的变量，丢弃的待下载URL多，进而减少了下载到重复URL的数量，在保证准确率的同时进一步提高了扫描效率，处理效率更高。
本申请实施例通过在检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，除去该第一URL中的参数部分得到第二URL，该目标泛化处理用于将URL中相同类型的多个连续字符替换为单个字符，该第一列表用于存储历史记录中已下载的URL经过该目标泛化处理后得到的URL，又在检测到该第二URL在该第二列表中，获取该第二URL在该第二列表中的第一出现次数，该第二列表用于存储历史记录中已下载的URL除去参数部分后得到的URL，当该第一出现次数小于或等于第一阈值，就下载该待下载URL，可以减少下载到重复URL的数量，在保证去重准确率的同时提高WEB漏洞扫描系统的扫描效率。
参见图3，是本申请实施例提供的URL去重方法的另一示意流程图。如图3所示，该URL去重方法可包括步骤：
S301，对待下载URL进行第一泛化处理得到第一泛化处理后的URL1。
S302，当第一URL不在第一列表中时，除去第一URL参数部分中的参数值得到除去参数值的URL2。
在一些可行的实施方式中，为便于描述，URL1可以表示本申请实施例中的第一URL。其中URL1、URL2表示经过不同处理后的URL。终端可以将待下载URL的参数部分按照参数名进行排序，并对该待下载URL参数部分的参数值进行第一泛化处理，得到URL1。终端可以检测第一列表中是否存在与上述URL1相同的URL，若不存在，说明上述URL1不在第一列表，则可以删除URL1参数部分中的参数值得到URL2。若存在，说明该URL1在该第一列表中，则终端可以将该URL1在该第一列表中的出现次数加1，得到该URL1在该第一列表中的第二出现次数，并可以判断该第二出现次数是否小于或等于第二阈值，若是(即该第二出现次数小于或等于该第二阈值)，则下载上述待下载URL，若否(即该第二出现次数大于该第二阈值)，则丢弃上述待下载URL，即不下载该待下载URL。终端通过判断经过第一泛化处理得到的URL1是否已经被下载过，从而判断出是否需要下载待下载URL，可以过滤掉参数值中具有相同数据格式的待下载URL，减少了下载到重复URL的数量，又因为只减少了待下载URL参数值中的部分变量，所以保证了去重的准确率。其中，该第一泛化处理用于将该待下载URL中相同类型的多个连续字符替换为单个字符。例如，可将连续的数字如145替换为数字1，连续的字母如FK、aj、dgA都替换为字母A，特殊字符都替换为符号％。其中，特殊字符指除数字和字母外的字符，如问号“？”、感叹号“！”等。该第一列表可以用于存储历史记录中已下载的URL经过上述第一泛化处理后得到的URL。
在一些可行的实施方式中，终端在得到上述URL1之后，终端可以利用MD5计算上述URL1的hash值，并可以检测该URL1的hash值是否在该第一列表中，若是，则可以删除该URL1的参数部分中的参数值得到URL2。若否，可以将该URL1的hash值在该第一列表中的出现次数加1，得到该URL1的hash值在该第一列表中的第二出现次数，并可以判断该第二出现次数是否小于或等于第二阈值，若是(即该第二出现次数小于或等于该第二阈值)，则下载上述待下载URL，若否(即该第二出现次数大于该第二阈值)，则丢弃上述待下载URL，即不下载该待下载URL。其中，第一列表可以用于存储历史记录已下载的 URL经过上述第一泛化处理后得到的URL的哈希值。该第二阈值为大于0的整数。因为hash函数是将任意大小的数据转换成特定大小的数据的函数，且已下载列表(第一列表)中存储的是URL的hash值，而不是完整的URL，可以减少存储空间(因为完整URL字符较多，而hash值是固定大小的数据)，同时在检测URL1是否在第一列表中时，可以提高处理效率。
在一些可行的实施方式中，终端在检测出该URL1不在该第一列表中时，终端可以将该URL1加入该第一列表以形成新的第一列表，即更新该第一列表，同时可以将该URL1在该第一列表中的出现次数置为1，并可以删除该URL1的参数部分中的参数值得到URL2。终端在下载上述待下载URL或丢弃上述待下载URL之后，终端利用最新的第一列表检测下一个待下载URL是否被下载过。终端通过不断更新第一列表，可以更准确地过滤掉重复的URL，并且可以进一步提高扫描效率。例如，第一列表包括URL5、URL6以及URL7，此时，URL1不在第一列表中，终端可以将该URL1加入第一列表，此时更新后的第一列表包括URL1、URL5、URL6以及URL7。同时，终端也可以将URL1在第一列表中的出现次数也置为1。
S303，当检测到URL2在第二列表时，检测URL2在第二列表中的第一出现次数是否小于或等于第一阈值，若是，则下载待下载URL。
S304，当检测到URL2不在第二列表中时，除去URL2中的参数部分得到除去参数部分的URL3。
在一些可行的实施方式中，为便于描述，URL3可以表示本申请实施例中的第二URL。其中URL2、URL3表示经过不同处理后的URL。终端可以检测第二列表中是否存在与上述URL2相同的URL，若存在，说明该URL2在该第二列表中，则可以将该URL2在该第二列表中的出现次数加1，得到该URL2在该第二列表中的第一出现次数，并可以检测该第一出现次数是否小于或等于第一阈值，若是(即该第一出现次数小于或等于该第一阈值)，则下载上述待下载URL，若否(即第一出现次数大于该第一阈值)，则丢弃上述待下载URL，即不下载该待下载URL。若终端检测到该第二列表中不存在该URL2，说明该URL2不在该第二列表中，则可以删除该URL2中的参数部分得到URL3。其中，第二列表可以用于存储历史记录中已下载的URL除去参数部分后得到的URL。终端通过判断除去URL1的参数部分中的参数值得到的URL2是否已经被下载过，从而判断出是否需要下载待下载URL，可以过滤掉参数值不相同的待下载URL，进一步减少了下载到重复URL的数量，在保证准确率的同时提高了扫描效率。需要说明的是，该第一阈值小于或等于上述第二阈值，该第一阈值可以为大于0的整数。因为第一泛化处理仅是减少了待下载URL的参数值中的变量，此时URL1中的变量还较多，过滤掉的URL就少，下载的URL就较多，URL1在第一列表中出现的次数就多。因此第一阈值小于第二阈值可以保证步骤S301-步骤S302未过滤掉的URL，除去参数部分中的参数值之后可以被过滤掉，达到分级去重的目的。
在一些可行的实施方式中，终端可以利用MD5计算上述URL2的hash值，并可以检测该URL2的hash值是否在该第二列表中，若是，可以将该URL2的hash值在该第二列表中的出现次数加1，得到该URL2的hash值在该第二列表中的第一出现次数，并可以判断该第一出现次数是否小于或等于第一阈值，若是(即该第一出现次数小于或等于该第一阈 值)，则下载上述待下载URL，若否(即该第一出现次数大于该第一阈值)，则丢弃上述待下载URL，即不下载该待下载URL。若否，则可以删除URL2中的参数部分(包括参数值和参数名)得到URL3。其中，第二列表可以用于存储历史记录中已下载的URL除去参数值后得到的URL，再经过MD5计算后得到的hash值。因为hash函数是将任意大小的数据转换成特定大小的数据的函数，且已下载列表(第二列表)中存储的是URL的hash值，而不是完整的URL，可以进一步减少存储空间，同时在检测URL2是否在第二列表时，可以进一步提高处理效率。
例如，假设第二列表中的hash值为07和09，URL2为：http://xxx.pingan.com/cgi-bin/index1.html？param1&param2，终端计算URL2的hash值为04，终端检测到URL2的hash值04不在第二列表中，终端就直接删除URL2中的参数部分得到URL3，URL3为：http://xxx.pingan.com/cgi-bin/index1.html。
在一些可行的实施方式中，终端在检测出该URL2不在该第二列表中时，终端可以将该URL2加入该第二列表形成新的第二列表，即更新该第二列表，同时可以将该URL2在该第二列表中的出现次数置为1，并可以除去该URL2中的参数部分得到URL3。终端在下载上述待下载URL或丢弃上述待下载URL之后，终端利用最新的第二列表检测下一个待下载URL是否被下载过。终端通过不断更新第二列表，可以更准确地过滤掉重复的URL，并且可以进一步提高扫描效率。
在一些可行的实施方式中，若上述待下载URL为某个网站中爬取到的第一个URL，那么第二列表可以为空(因为此时没有已下载的URL)。
S305，当检测到URL3在第三列表中时，检测URL3在第三列表中的第三出现次数是否小于或等于第三阈值，若是，则下载待下载URL。
S306，当检测到URL3不在第三列表中时，对URL3进行第二泛化处理得到第二泛化处理后的URL4。
在一些可行的实施方式中，为便于描述，URL3可以表示本申请实施例中的第二URL。其中URL3、URL4表示经过不同处理后的URL。终端可以检测第三列表中是否存在与上述URL3相同的URL，若存在，说明该URL3在该第三列表中，则可以将该URL3在该第三列表中的出现次数加1，得到该URL3在该第三列表中的第三出现次数，并可以检测该第三出现次数是否小于或等于第三阈值，若是(即该第三出现次数小于或等于该第三阈值)，则下载上述待下载URL，若否(即第三出现次数大于该第三阈值)，则丢弃上述待下载URL，即不下载该待下载URL。若终端检测到该第三列表中不存在该URL3，说明该URL3不在该第三列表中，则可以对该URL3的路径部分进行第二泛化处理得到URL4。该第二泛化处理用于将URL3的路径部分中目标类型的至少一个字符替换为目标字符，如将该URL3的路径部分中的一个或多个数字替换为预设的单个数字“1”。其中，第三列表可以用于存储历史记录中已下载的URL除去参数部分后得到的URL。终端通过判断除去URL2参数部分得到的URL3是否已经被下载过，从而判断出是否需要下载待下载URL，可以过滤掉参数部分不相同的待下载URL，即减少了待下载URL中的变量，丢弃的待下载URL多，进而减少了下载到重复URL的数量，在保证准确率的同时进一步提高了扫描效率。需要说明的是，该第三阈值可以小于或等于上述第一阈值，上述第一阈值小于或等于上述第二阈 值，该第三阈值可以为大于0的整数。因为除去参数部分中的参数值仅是减少了待下载URL参数部分的变量，此时URL2中的变量还较多，过滤掉的URL就较少，下载的URL就较多，URL2在第二列表中出现的次数就较多。因此第三阈值小于第一阈值可以保证步骤S303-步骤S304未过滤掉的URL，除去参数部分后可以被过滤掉，达到分级去重的目的。
在一些可行的实施方式中，终端可以利用MD5计算上述URL3的hash值，并可以检测该URL3的hash值是否在该第三列表中，若是，可以将该URL3的hash值在该第三列表中的出现次数加1，得到该URL3的hash值在该第三列表中的第三出现次数，并可以判断该第三出现次数是否小于或等于第三阈值，若是(即该第三出现次数小于或等于该第三阈值)，则下载上述待下载URL，若否(即该第三出现次数大于该第三阈值)，则丢弃上述待下载URL，即不下载该待下载URL。若否，则可以对该URL3进行第二泛化处理得到URL4。该第二泛化处理用于将URL3的路径部分中目标类型的至少一个字符替换为目标字符，如将该URL3的路径部分中的一个或多个数字替换为预设的单个数字“1”。其中，第三列表可以用于存储历史记录中已下载的URL除去参数部分后得到的URL，再经过MD5计算后得到的hash值。因为hash函数是将任意大小的数据转换成特定大小的数据的函数，且已下载列表(第三列表)中存储的是URL的hash值，而不是完整的URL，可以进一步减少存储空间，同时在检测URL3是否在第三列表中时，可以进一步提高处理效率。
在一些可行的实施方式中，终端在检测出该URL3不在该第三列表中时，终端可以将该URL3加入该第三列表以形成新的第三列表，即更新该第三列表，同时可以将该URL3在该第三列表中的出现次数置为1，并可以对该URL3进行第二泛化处理得到URL4。终端在下载上述待下载URL或丢弃上述待下载URL之后，终端利用最新的第三列表检测下一个待下载URL是否被下载过。终端通过不断更新第三列表，可以更准确地分级过滤掉重复的URL，并且可以进一步提高扫描效率。
在一些可行的实施方式中，若上述待下载URL为某个网站中爬取到的第一个URL，那么第三列表可以为空(因为此时没有已下载的URL)。
S307，当检测到URL4在第四列表中时，检测URL4在第四列表中的第四出现次数是否小于或等于第四阈值，若是，则下载待下载URL。
S308，当检测到URL4不在第四列表中时，下载待下载URL。
在一些可行的实施方式中，终端可以检测第四列表中是否存在与上述URL4相同的URL，若存在，说明该URL4在该第四列表中，则可以将该URL4在该第四列表中的出现次数加1，得到该URL4在该第四列表中的第四出现次数，并可以检测该第四出现次数是否小于或等于第四阈值，若是(即该第四出现次数小于或等于该第四阈值)，则下载上述待下载URL，若否(即第四出现次数大于该第四阈值)，则丢弃上述待下载URL，即不下载该待下载URL。若终端检测到该第四列表中不存在该URL4，说明该URL4不在该第四列表中，则可以直接下载上述待下载URL。其中，第四列表可以用于存储历史记录中已下载URL除去参数部分并经过上述第二泛化处理后得到的URL。终端通过判断除去参数部分且经过第二泛化处理后得到的URL4是否已经被下载过，从而判断出是否需要下载待下载URL，可以过滤掉路径部分不同的待下载URL，减少了待下载URL中的变量，进而丢弃的待下载URL多，下载到的重复URL少，在保证准确率的同时进一步提高了扫描效率。 需要说明的是，该第四阈值小于或等于上述第三阈值，上述第三阈值可以小于或等于上述第一阈值，上述第一阈值小于或等于上述第二阈值，该第四阈值为大于或等于0的整数。第四阈值小于第三阈值可以保证步骤S305-步骤S306未过滤掉的URL，经过第二泛化处理后可以被过滤掉，达到分级去重的目的。
在一些可行的实施方式中，终端可以利用MD5计算上述URL4的hash值，并可以检测该URL4的hash值是否在该第四列表中，若是，可以将该URL4的hash值在该第四列表中的出现次数加1，得到该URL4的hash值在该第四列表中的第四出现次数，并可以判断该第四出现次数是否小于或等于第四阈值，若是(即该第四出现次数小于或等于该第四阈值)，则下载上述待下载URL，若否(即该第四出现次数大于该第四阈值)，则丢弃上述待下载URL，即不下载该待下载URL。若否，则可以直接下载上述待下载URL。其中，第四列表可以用于存储历史记录中已下载的URL除去参数部分且经过上述第二泛化处理后得到的URL，再经过MD5计算后得到的hash值。因为hash函数是将任意大小的数据转换成特定大小的数据的函数，且已下载列表(第四列表)中存储的是URL的hash值，而不是完整的URL，可以进一步减少存储空间，同时在检测URL4是否在第四列表时，可以进一步提高处理效率。
在一些可行的实施方式中，终端在检测出该URL4不在该第四列表中时，终端可以将该URL4加入该第四列表以形成新的第四列表，即更新该第四列表，同时可以将URL4在第四列表中的出现次数置为1，并可以下载上述待下载URL。终端在下载上述待下载URL或丢弃上述待下载URL之后，终端利用最新的第四列表检测下一个待下载URL是否被下载过。终端通过不断更新第四列表，可以更准确地过滤掉重复的URL，并且可以进一步提高扫描效率。
在一些可行的实施方式中，若上述待下载URL为某个网站中爬取到的第一个URL，那么第四列表可以为空(因为此时没有已下载的URL)。
本申请实施例通过对待下载URL进行分级去重，逐级减少待下载URL中的变量，并根据逐级减少变量后的URL判断待下载URL是否已下载，若在该级就判断出待下载URL已下载，则丢弃待下载URL，若该级就确定待下载URL未下载，则下载待下载URL，若该级无法判断出待下载URL是否已下载，则进入下一级判断，直至判断出待下载URL已下载或未下载。通过更细化的分级去重方案，不仅能够减少下载到重复URL的数量，提高WEB漏洞扫描系统的扫描效率，还可以提高去重的准确率。
参见图4，是本申请实施例提供的URL去重装置的一示意性框图。本申请实施例提供的URL去重装置包括：
处理模块10，用于当检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，除去该第一URL中的参数部分得到第二URL。其中，该目标泛化处理用于将URL中相同类型的多个连续字符替换为单个字符，该第一列表用于存储历史记录中已下载的URL经过该目标泛化处理后得到的URL。
第一获取模块20，用于当检测到该第二URL在该第二列表中时，获取该第二URL在该第二列表中的第一出现次数。其中，该第二列表用于存储历史记录中已下载的URL除去 参数部分后得到的URL。
下载模块30，用于当该第一出现次数小于或等于第一阈值时，则下载该待下载URL。
在一些可行的实施方式中，该装置还包括：
第二获取模块40，用于当检测到该待下载URL经过该目标泛化处理后得到的该第一URL在该第一列表中时，获取该第一URL在该第一列表中的第二出现次数；上述下载模块30，还用于当该第二出现次数小于或等于第二阈值时，下载该待下载URL。其中，该第二阈值大于或等于该第一阈值，该第一阈值为大于或等于0的整数。
在一些可行的实施方式中，上述处理模块10具体用于：对该待下载URL进行目标泛化处理得到第一URL；利用目标哈希函数计算该第一URL的哈希值；检测该第一URL的哈希值是否在第一列表中；当该第一URL的哈希值不在该第一列表中时，除去该第一URL中的参数部分得到第二URL。其中，该第一列表用于存储历史记录中已下载的URL经过目标泛化处理后得到的URL的哈希值。
在一些可行的实施方式中，该第一URL包括协议、服务器名称、路径以及参数字段；上述处理模块具体用于当检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，删除该第一URL中的参数字段得到该第二URL。
在一些可行的实施方式中，上述第一获取模块20具体用于：利用目标哈希函数计算该第二URL的哈希值；检测该第二URL的哈希值是否在该第二列表中；若该第二URL的哈希值在该第二列表中，获取该第二URL的哈希值在该第二列表中的该第一出现次数。其中，该第二列表用于存储历史记录中已下载的URL除去参数部分后得到的URL的哈希值。
在一些可行的实施方式中，上述下载模块30还用于当该第二URL不在该第二列表中时，下载该待下载URL。
具体实现中，上述URL去重装置可通过上述各个模块执行上述图1或者图3所提供的实现方式中各个步骤所提供的实现方式，实现上述各实施例中所实现的功能，具体可参见上述图1或图3所示的方法实施例中各个步骤提供的相应描述，在此不再赘述。
本申请实施例中，URL去重装置可通过在检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，除去该第一URL中的参数部分得到第二URL，该目标泛化处理用于将URL中相同类型的多个连续字符替换为单个字符，该第一列表用于存储历史记录中已下载的URL经过该目标泛化处理后得到的URL，又在检测到该第二URL在该第二列表中，获取该第二URL在该第二列表中的第一出现次数，该第二列表用于存储历史记录中已下载的URL除去参数部分后得到的URL，当该第一出现次数小于或等于第一阈值，就下载该待下载URL，可以减少下载到重复URL的数量，在保证去重准确率的同时提高WEB漏洞扫描系统的扫描效率。
参见图5，是本申请实施例提供的一种终端的示意性框图。如图5所示，本申请实施例中的终端可以包括：一个或多个处理器501和存储器502。上述处理器501和存储器502通过总线503连接。存储器502用于存储计算机程序，所述计算机程序包括程序指令，处理器501用于执行存储器502存储的程序指令。其中，处理器501被配置用于调用该程序指令执行：
当检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，除去该第一URL中的参数部分得到第二URL，该目标泛化处理用于将URL中相同类型的多个连续字符替换为单个字符，该第一列表用于存储历史记录中已下载的URL经过该目标泛化处理后得到的URL；
若检测到该第二URL在该第二列表中，获取该第二URL在该第二列表中的第一出现次数，该第二列表用于存储历史记录中已下载的URL除去参数部分后得到的URL；
若该第一出现次数小于或等于第一阈值，则下载该待下载URL。
应当理解，在一些可行的实施方式中，所称处理器501可以是中央处理单元(central processing unit，CPU)，该处理器还可以是其他通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现成可编程门阵列(field-programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
该存储器502可以包括只读存储器和随机存取存储器，并向处理器501提供指令和数据。存储器502的一部分还可以包括非易失性随机存取存储器。例如，存储器502还可以存储设备类型的信息。
具体实现中，本申请实施例中所描述的处理器501可执行本申请实施例提供的统一资源定位符URL去重方法中所描述的实现方式，也可执行本申请实施例所描述的URL去重装置的实现方式，在此不再赘述。
本申请实施例还提供一种计算机可读存储介质，该计算机可读存储介质存储有计算机程序，该计算机程序包括程序指令，该程序指令被处理器执行时实现图1或图3所示的统一资源定位符URL去重方法，具体细节请参照图1或图3所示实施例的描述，在此不再赘述。
上述计算机可读存储介质可以是前述任一实施例所述的统一资源定位符URL去重装置或电子设备的内部存储单元，例如电子设备的硬盘或内存。该计算机可读存储介质也可以是该电子设备的外部存储设备，例如该电子设备上配备的插接式硬盘，智能存储卡(smart media card,SMC)，安全数字(secure digital,SD)卡，闪存卡(flash card)等。进一步地，该计算机可读存储介质还可以既包括该电子设备的内部存储单元也包括外部存储设备。该计算机可读存储介质用于存储该计算机程序以及该电子设备所需的其他程序和数据。该计算机可读存储介质还可以用于暂时地存储已经输出或者将要输出的数据。
以上所述，仅为本发明的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本发明的保护范围之内。因此，本发明的保护范围应以所述权利要求的保护范围为准。

Claims (20)

  1. 一种统一资源定位符URL去重方法，其特征在于，包括：
    当检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，除去所述第一URL中的参数部分得到第二URL，所述目标泛化处理用于将URL中相同类型的多个连续字符替换为单个字符，所述第一列表用于存储历史记录中已下载的URL经过所述目标泛化处理后得到的URL；
    若检测到所述第二URL在所述第二列表中，获取所述第二URL在所述第二列表中的第一出现次数，所述第二列表用于存储历史记录中已下载的URL除去参数部分后得到的URL；
    若所述第一出现次数小于或等于第一阈值，则下载所述待下载URL。
  2. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    当检测到所述待下载URL经过所述目标泛化处理后得到的所述第一URL在所述第一列表中时，获取所述第一URL在所述第一列表中的第二出现次数；
    若所述第二出现次数小于或等于第二阈值，下载所述待下载URL；
    其中，所述第二阈值大于或等于所述第一阈值，所述第一阈值为大于或等于0的整数。
  3. 根据权利要求1或2所述的方法，其特征在于，所述当检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，除去所述第一URL中的参数部分得到第二URL，包括：
    对所述待下载URL进行目标泛化处理得到第一URL；
    利用目标哈希函数计算所述第一URL的哈希值；
    检测所述第一URL的哈希值是否在第一列表中，所述第一列表用于存储历史记录中已下载的URL经过目标泛化处理后得到的URL的哈希值；
    当所述第一URL的哈希值不在所述第一列表中时，除去所述第一URL中的参数部分得到第二URL。
  4. 根据权利要求1-3任意一项所述的方法，其特征在于，所述第一URL包括协议、服务器名称、路径以及参数字段；所述除去所述第一URL中的参数部分得到第二URL，包括：
    删除所述第一URL中的参数字段得到所述第二URL。
  5. 根据权利要求1-4任意一项所述的方法，其特征在于，所述若检测到所述第二URL在所述第二列表中，获取所述第二URL在所述第二列表中的第一出现次数，包括：
    利用目标哈希函数计算所述第二URL的哈希值；
    检测所述第二URL的哈希值是否在所述第二列表中，所述第二列表用于存储历史记录中已下载的URL除去参数部分后得到的URL的哈希值；
    若所述第二URL的哈希值在所述第二列表中，获取所述第二URL的哈希值在所述第二列表中的所述第一出现次数。
  6. 根据权利要求1-4任意一项所述的方法，其特征在于，所述方法还包括：
    若所述第二URL不在所述第二列表中，则下载所述待下载URL。
  7. 根据权利要求1-4任意一项所述的方法，其特征在于，所述获取所述第二URL在所述第二列表中的第一出现次数，包括：
    将所述第二URL在所述第二列表中的出现次数加1得到第一出现次数。
  8. 一种URL去重装置，其特征在于，包括：
    处理模块，用于当检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，除去所述第一URL中的参数部分得到第二URL，所述目标泛化处理用于将URL中相同类型的多个连续字符替换为单个字符，所述第一列表用于存储历史记录中已下载的URL经过所述目标泛化处理后得到的URL；
    第一获取模块，用于当检测到所述第二URL在所述第二列表中时，获取所述第二URL在所述第二列表中的第一出现次数，所述第二列表用于存储历史记录中已下载的URL除去参数部分后得到的URL；
    下载模块，用于当所述第一出现次数小于或等于第一阈值时，则下载所述待下载URL。
  9. 根据权利要求8所述的装置，其特征在于，所述装置还包括：
    第二获取模块，用于当检测到所述待下载URL经过所述目标泛化处理后得到的所述第一URL在所述第一列表中时，获取所述第一URL在所述第一列表中的第二出现次数；
    所述下载模块，还用于当所述第二出现次数小于或等于第二阈值时，下载所述待下载URL；
    其中，所述第二阈值大于或等于所述第一阈值，所述第一阈值为大于或等于0的整数。
  10. 根据权利要求8或9所述的装置，其特征在于，所述处理模块具体用于：
    对所述待下载URL进行目标泛化处理得到第一URL；
    利用目标哈希函数计算所述第一URL的哈希值；
    检测所述第一URL的哈希值是否在第一列表中，所述第一列表用于存储历史记录中已下载的URL经过目标泛化处理后得到的URL的哈希值；
    当所述第一URL的哈希值不在所述第一列表中时，除去所述第一URL中的参数部分得到第二URL。
  11. 根据权利要求8-10所述的装置，其特征在于，所述第一URL包括协议、服务器名称、路径以及参数字段；
    所述处理模块具体用于当检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，删除该第一URL中的参数字段得到该第二URL。
  12. 根据权利要求8-11所述的装置，其特征在于，所述第一获取模块具体用于：利用目标哈希函数计算该第二URL的哈希值；检测该第二URL的哈希值是否在该第二列表中；若该第二URL的哈希值在该第二列表中，获取该第二URL的哈希值在该第二列表中的该第一出现次数。其中，该第二列表用于存储历史记录中已下载的URL除去参数部分后得到的URL的哈希值。
  13. 根据权利要求8-11所述的装置，其特征在于，所述下载模块还用于当该第二URL不在该第二列表中时，下载该待下载URL。
  14. 根据权利要求8-11所述的装置，其特征在于，所述第一获取模块具体用于：
    当检测到所述第二URL在所述第二列表中时，将所述第二URL在所述第二列表中的 出现次数加1得到第一出现次数。
  15. 一种终端，其特征在于，包括处理器和存储器，所述处理器和存储器相互连接，其中，所述存储器用于存储计算机程序，所述计算机程序包括程序指令，所述处理器被配置用于调用所述程序指令，执行：
    当检测到待下载URL经过目标泛化处理后得到的第一URL不在第一列表中时，除去所述第一URL中的参数部分得到第二URL，所述目标泛化处理用于将URL中相同类型的多个连续字符替换为单个字符，所述第一列表用于存储历史记录中已下载的URL经过所述目标泛化处理后得到的URL；
    若检测到所述第二URL在所述第二列表中，获取所述第二URL在所述第二列表中的第一出现次数，所述第二列表用于存储历史记录中已下载的URL除去参数部分后得到的URL；
    若所述第一出现次数小于或等于第一阈值，则下载所述待下载URL。
  16. 根据权利要求15所述的终端，其特征在于，所述处理器还用于：
    当检测到所述待下载URL经过所述目标泛化处理后得到的所述第一URL在所述第一列表中时，获取所述第一URL在所述第一列表中的第二出现次数；
    若所述第二出现次数小于或等于第二阈值，下载所述待下载URL；
    其中，所述第二阈值大于或等于所述第一阈值，所述第一阈值为大于或等于0的整数。
  17. 根据权利要求15或16所述的终端，其特征在于，所述处理器具体用于：
    对所述待下载URL进行目标泛化处理得到第一URL；
    利用目标哈希函数计算所述第一URL的哈希值；
    检测所述第一URL的哈希值是否在第一列表中，所述第一列表用于存储历史记录中已下载的URL经过目标泛化处理后得到的URL的哈希值；
    当所述第一URL的哈希值不在所述第一列表中时，除去所述第一URL中的参数部分得到第二URL。
  18. 根据权利要求15-17任意一项所述的终端，其特征在于，所述第一URL包括协议、服务器名称、路径以及参数字段；
    所述处理器具体用于：删除所述第一URL中的参数字段得到所述第二URL。
  19. 根据权利要求15-18任意一项所述的终端，其特征在于，所述处理器具体用于：
    利用目标哈希函数计算所述第二URL的哈希值；
    检测所述第二URL的哈希值是否在所述第二列表中，所述第二列表用于存储历史记录中已下载的URL除去参数部分后得到的URL的哈希值；
    若所述第二URL的哈希值在所述第二列表中，获取所述第二URL的哈希值在所述第二列表中的所述第一出现次数。
  20. 一种计算机可读存储介质，其特征在于，所述计算机存储介质存储有计算机程序，所述计算机程序包括程序指令，所述程序指令当被处理器执行时使所述处理器执行如权利要求1-7任一项所述的方法。
PCT/CN2018/108715 2018-07-05 2018-09-29 一种url去重方法及装置 Ceased WO2020006909A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810733716.0 2018-07-05
CN201810733716.0A CN108920668B (zh) 2018-07-05 2018-07-05 一种统一资源定位符url去重方法及装置

Publications (1)

Publication Number Publication Date
WO2020006909A1 true WO2020006909A1 (zh) 2020-01-09

Family

ID=64425420

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/108715 Ceased WO2020006909A1 (zh) 2018-07-05 2018-09-29 一种url去重方法及装置

Country Status (2)

Country Link
CN (1) CN108920668B (zh)
WO (1) WO2020006909A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113449232A (zh) * 2020-03-27 2021-09-28 北京沃东天骏信息技术有限公司 一种数据处理方法、装置、设备和存储介质
CN116432190A (zh) * 2023-06-15 2023-07-14 杭州美创科技股份有限公司 接口未授权访问检测方法、装置、计算机设备及存储介质

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110855676B (zh) * 2019-11-15 2021-08-31 腾讯科技（深圳）有限公司 网络攻击的处理方法、装置及存储介质
CN111143648A (zh) * 2019-12-02 2020-05-12 西安交大捷普网络科技有限公司 统一资源定位符url的去重方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013143363A1 (en) * 2012-03-29 2013-10-03 Tencent Technology (Shenzhen) Company Limited A method and apparatus for data storage and downloading
CN104933056A (zh) * 2014-03-18 2015-09-23 腾讯科技（深圳）有限公司 统一资源定位符去重方法及装置
CN106547764A (zh) * 2015-09-18 2017-03-29 北京国双科技有限公司 网页数据去重的方法及装置
CN106919570A (zh) * 2015-12-24 2017-07-04 国家新闻出版广电总局广播科学研究院 一种面向网络新媒体的页面链接去重扫描方法及装置
CN107885820A (zh) * 2017-11-07 2018-04-06 北京小度互娱科技有限公司 基于爬虫系统的广度遍历定向抓取方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102185741B (zh) * 2011-06-10 2013-06-26 浙江大学 多层架构下估算事务的处理器需求的方法
CA2824977C (en) * 2012-08-30 2019-03-19 Accenture Global Services Limited Online content collection
CN106815247B (zh) * 2015-11-30 2020-05-22 北京国双科技有限公司 统一资源定位符获取方法及装置
CN106844389B (zh) * 2015-12-07 2021-05-04 阿里巴巴集团控股有限公司 网络资源地址url的处理方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013143363A1 (en) * 2012-03-29 2013-10-03 Tencent Technology (Shenzhen) Company Limited A method and apparatus for data storage and downloading
CN104933056A (zh) * 2014-03-18 2015-09-23 腾讯科技（深圳）有限公司 统一资源定位符去重方法及装置
CN106547764A (zh) * 2015-09-18 2017-03-29 北京国双科技有限公司 网页数据去重的方法及装置
CN106919570A (zh) * 2015-12-24 2017-07-04 国家新闻出版广电总局广播科学研究院 一种面向网络新媒体的页面链接去重扫描方法及装置
CN107885820A (zh) * 2017-11-07 2018-04-06 北京小度互娱科技有限公司 基于爬虫系统的广度遍历定向抓取方法

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113449232A (zh) * 2020-03-27 2021-09-28 北京沃东天骏信息技术有限公司 一种数据处理方法、装置、设备和存储介质
CN116432190A (zh) * 2023-06-15 2023-07-14 杭州美创科技股份有限公司 接口未授权访问检测方法、装置、计算机设备及存储介质
CN116432190B (zh) * 2023-06-15 2023-09-08 杭州美创科技股份有限公司 接口未授权访问检测方法、装置、计算机设备及存储介质

Also Published As

Publication number Publication date
CN108920668B (zh) 2023-04-18
CN108920668A (zh) 2018-11-30

Similar Documents

Publication Publication Date Title
CN108206802B (zh) 检测网页后门的方法和装置
CN110855661B (zh) 一种WebShell检测方法、装置、设备及介质
CN112771525B (zh) 松散耦合的分布式系统中个人数据的模糊和删除
WO2020006908A1 (zh) 一种url去重方法及装置
CN111259282B (zh) Url去重方法、装置、电子设备及计算机可读存储介质
CN107241296B (zh) 一种Webshell的检测方法及装置
CN109768992B (zh) 网页恶意扫描处理方法及装置、终端设备、可读存储介质
WO2020006909A1 (zh) 一种url去重方法及装置
CN107547671A (zh) 一种url匹配方法及装置
WO2012089005A1 (zh) 钓鱼网页检测方法及设备
CN103699585A (zh) 文件的元数据存储以及文件恢复的方法、装置和系统
US9749295B2 (en) Systems and methods for internet traffic analysis
CN105791273A (zh) web漏洞扫描系统
CN105100084A (zh) 一种防止跨站请求伪造攻击的方法及系统
CN106104550A (zh) 网站信息提取装置、系统、网站信息提取方法以及网站信息提取程序
CN105868234A (zh) 缓存数据的更新方法及装置
CN112445771A (zh) 网络流量的数据处理方法、装置、设备及存储介质
CN114679306B (zh) 一种攻击检测方法及装置
CN106911636B (zh) 一种检测网站是否存在后门程序的方法及装置
CN106250476A (zh) 一种更新和同步白名单的方法、装置和系统
CN109788050B (zh) 一种获取源站ip地址方法、系统、电子设备和介质
WO2025076898A1 (zh) Dns解析方法、dns服务器、电子设备及存储介质
US8694659B1 (en) Systems and methods for enhancing domain-name-server responses
WO2020000748A1 (zh) 一种文件检测方法及装置
CN110717036B (zh) 一种统一资源定位符的去重方法、装置及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18925424

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 13.04.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 18925424

Country of ref document: EP

Kind code of ref document: A1