WO2016110273A1 - 一种对访问请求进行限制的系统和方法 - Google Patents

一种对访问请求进行限制的系统和方法 Download PDF

Info

Publication number
WO2016110273A1
WO2016110273A1 PCT/CN2016/070522 CN2016070522W WO2016110273A1 WO 2016110273 A1 WO2016110273 A1 WO 2016110273A1 CN 2016070522 W CN2016070522 W CN 2016070522W WO 2016110273 A1 WO2016110273 A1 WO 2016110273A1
Authority
WO
WIPO (PCT)
Prior art keywords
aggregator
blacklist
load balancer
access request
statistical data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/070522
Other languages
English (en)
French (fr)
Inventor
翁志
肖思兴
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Jingdong Shangke Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to US15/542,086 priority Critical patent/US10735501B2/en
Priority to RU2017128207A priority patent/RU2666289C1/ru
Priority to JP2017554634A priority patent/JP2018508166A/ja
Publication of WO2016110273A1 publication Critical patent/WO2016110273A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
    • H04L63/0227—Filtering policies
    • H04L63/0236—Filtering by address, protocol, port number or service, e.g. IP-address or URL
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/10—Network architectures or network communication protocols for network security for controlling access to devices or network resources
    • H04L63/101—Access control lists [ACL]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L12/00—Data switching networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/40—Support for services or applications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1004—Server selection for load balancing
    • H04L67/1023—Server selection for load balancing based on a hash applied to IP addresses or costs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/563—Data redirection of data network streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/564—Enhancement of application control based on intercepted application data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/565—Conversion or adaptation of application format or content
    • H04L67/5651—Reducing the amount or size of exchanged application data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/566—Grouping or aggregating service requests, e.g. for unified processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
    • H04L69/164—Adaptation or special uses of UDP protocol
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1441—Countermeasures against malicious traffic
    • H04L63/1458—Denial of Service

Definitions

  • the present invention relates to a system for restricting access requests from a network and a corresponding method thereof.
  • the traditional load balancer such as the configuration of HaProxy under Linux, is used to read the cookies or URL interpretation contained in every HTTP request from the network, and rewrite the header based on this information and send the HTTP request.
  • Load Balancer To the back-end server group, each server in the back-end server group achieves a balanced state of traffic and resource usage.
  • the traditional load balancer does not automatically filter the traffic of the network, and cannot limit or discard the attack traffic.
  • One idea is to analyze access to URLs and restrict access requests based on unit time visits such as QPS. Analysis of access to URLs for a large website typically consumes a lot of memory. In general, for any combination of data item IP address, user identification (USERID), uniform resource locator (URL), etc., a timestamp of each access associated with the combination needs to be recorded. When QPS needs to be calculated, filtering or sorting each time point is time consuming and consumes memory space.
  • the invention is particularly suitable for solving DOS attacks (denial of service attacks) and DDOS attacks (distributed denial of service attacks) in the field of networks.
  • This technique is especially suitable for the prevention of HTTP flooding (http flooding).
  • the invention protects the back-end server by collecting and analyzing the attack traffic matching the pattern matching and limiting or discarding the traffic of the user traffic.
  • a system for restricting access requests comprising a load balancer, an aggregator and an aggregator.
  • the load balancer is configured to receive each access request from the network, parse the received access request into a UDP message and transmit the UDP message to the aggregator;
  • the aggregator is configured to Accumulating a plurality of UDP messages received from the load balancer according to a predefined data item combination and a predefined duration to generate statistical data, and transmitting the accumulated statistical data thereto in response to a request from the aggregator;
  • the aggregator is configured to request statistical data from the aggregator according to a predetermined request time interval, receive statistical data from the aggregator, generate a blacklist of access requests according to the received statistical data and a predefined rule, and
  • the blacklist is sent to the load balancer in response to a request from the load balancer, wherein the blacklist specifies processing actions to be made for each of one or more particular access requests.
  • the load balancer is further configured to determine
  • a computer-implemented method for restricting an access request comprising: receiving, by a load balancer, each access request from a network, parsing the received access request into a UDP Transmitting, and transmitting the UDP message to the aggregator; determining, by the load balancer, a processing action on the received access request according to a current blacklist, wherein the blacklist specifies one or more specific Access to each of the processing actions to be made.
  • the concentrator receives the UDP packet sent by the load balancer according to a predefined data item combination and a predefined duration.
  • the aggregator requests statistics from the aggregator according to a predetermined request time interval Data, receiving statistical data from the aggregator, generating a blacklist of access requests based on the received statistics and predefined rules, and transmitting the blacklist to the request in response to a request from the load balancer Load balancer.
  • the statistical data comprises the number of UDP messages containing the predefined data item combination accumulated over the predefined duration.
  • the network is the Internet and the access request is an HTTP request from the Internet.
  • the aggregator comprises a plurality of aggregation units operating independently of each other, the load balancer transmitting each UDP message to one of the plurality of aggregation units, the plurality of aggregation units Each of the aggregation units performs the accumulation separately.
  • the aggregator receives statistical data from the plurality of aggregation units separately, and generates aggregated statistical data.
  • the blacklist is generated according to the summarized statistical data and the predefined rules.
  • the load balancer also includes a predefined whitelist that includes one or more records regarding access requests to be released. And the load balancer determines, for each access request received, a processing action of the access request: when the access request matches one of the records in the whitelist, or does not match the blacklist When any one of the records matches, the access request is released; when the access request does not match any of the records in the whitelist and matches one of the records in the blacklist, the access request The operation is performed according to the processing action specified by the matching record in the blacklist.
  • the predefined whitelist in the load balancer may be obtained from the aggregator along with the blacklist.
  • processing action specified in said blacklist comprises one of: interception, redirection, and delay.
  • FIG. 1 is a structural diagram of a system for restricting an access request in accordance with the present invention
  • FIG. 2 is a timing diagram of a load equalizer processing received access request in accordance with the present invention
  • FIG. 3 is a block diagram of an exemplary composition of a summarizer in accordance with the present invention.
  • the present invention provides systems and methods for limiting access requests from a network using statistical data.
  • An access request such as an HTTP request, includes at least the following data fields: the destination domain to which the query is to be accessed (host); the universal resource identifier (uri) requested by the user; the username (uid); the client IP Address (cip), and so on.
  • the present invention provides a system for restricting access requests from a network, the system including a load balancer, an aggregator, and an aggregator.
  • the load balancer receives an access request from the network, parses each received access request into a UDP message, and transmits the UDP message to the aggregator.
  • the aggregator accumulates the received plurality of UDP messages according to a predefined data item combination and a predefined duration to generate statistical data, and transmits statistical data thereto in response to the request from the aggregator.
  • the statistical data includes the number of UDP messages containing the predefined data item combination accumulated over the current predefined duration.
  • the aggregator requests statistics about the access request from the aggregator according to a predetermined request time interval, and receives statistics from the aggregator according to the received system. Generating data and predefined rules to generate a blacklist of access requests, and transmitting the blacklist to the load balancer according to the request of the load balancer, wherein the blacklist specifies one or more specific access requests Process actions.
  • the load balancer determines a processing action for each received access request based on the current blacklist.
  • FIG. 1 illustrates a system 100 including three modules in accordance with an embodiment of the present invention:
  • the load balancer 101 receives an access request from the network, such as an HTTP request from the Internet, and parses each received access request into a UDP message and transmits the UDP message to the aggregator 102.
  • the UDP protocol is not a connection type protocol, so there will be a certain packet loss phenomenon. However, this loss is within the controllable error range and the impact on the robustness of the system is negligible.
  • the aggregator 102 can include a plurality of aggregation units. Multiple aggregation units can be scaled independently horizontally, ie each aggregation unit is deployed and operated independently, so they have no influence or dependency on each other. If the processing power of the current aggregator does not meet the needs, it is convenient to add server resources to deploy new aggregation units. This can better meet the needs of large traffic flows.
  • the load balancer 101 transmits each UDP message to a selected aggregation unit.
  • Each aggregation unit separately accumulates the received UDP packets and generates statistical data.
  • the load balancer 101 sends UDP packets to a plurality of aggregation units in a round-by-round manner.
  • the first UDP packet is sent to the first aggregation unit
  • the second UDP packet is sent to the second aggregation unit
  • the generated UDP packets are sequentially sent to the aggregation units in sequence.
  • Each aggregating unit analyzes and accumulates the received UDP packets according to predefined rules or conditions, such as a predefined combination of data items, a predefined time interval, and the like.
  • Each UDP packet contains information about the user's HTTP access request, such as the destination domain name (host) accessed by the user, the universal resource identifier (uri) requested by the user, the user name (uid), and the client IP address (cip). Wait.
  • the aggregation unit counts the number of access requests with the same destination domain name (same host), the same source (same uid, the same cip), and thus the number of access requests from this source.
  • the number of access requests per unit time from this source can be derived.
  • the value per second query rate (QPS) for the source can be obtained by dividing the total number of access requests from the source received during the defined time period by the length of the time period.
  • the aggregation unit transmits to it the latest statistical data about the access request generated at the aggregation unit according to the request of the aggregator 103.
  • the aggregator 103 may request "statistics" once to the aggregator or for each of the aggregation units at regular intervals, for example every 10 seconds, such as in the form of HTTP.
  • each aggregation unit of the aggregator When each aggregation unit of the aggregator receives the request from the aggregator 103, it constructs the latest statistical data about the access generated by it into a response message, such as a response message in the form of HTTP, and contains the latest information.
  • the response message of the statistical data is sent to the aggregator 103.
  • the aggregator 103 After receiving the response message from each aggregation unit, the aggregator 103 processes the statistical data contained in the messages according to the specific classification, thereby generating aggregated statistical data, and generating a blacklist according to the predefined rules.
  • a blacklist record in the blacklist may include the following four parameters:
  • the meaning of the blacklist record is understood to be that for a request sent to the destination domain, if the username and client IP contained in the request are equal to the corresponding value in the record (source username, source client) IP), the processing action specified in the record is taken.
  • the processing action may be, for example, intercepting, redirecting, or delaying the request.
  • the access request is said to match the blacklist.
  • a threshold for the number of requests per unit time for a specific destination domain name and a corresponding processing action when the threshold is exceeded are set in the configuration file 1036 of the aggregator 103.
  • Aggregator 103 summarizes statistics based on aggregated access requests with the same data item combination
  • the data is used to determine that when the unit time visit amount (QPS) value of the access request exceeds the threshold of the unit time access amount corresponding to the destination domain name set in the configuration file 1036, a corresponding blacklist record is generated.
  • the record includes the destination domain name, username and client IP of the request and the corresponding processing action.
  • the generated blacklist record is blacklisted to generate the current blacklist.
  • a whitelist can also be maintained in the aggregator 103.
  • the self-list can have the same structure as the blacklist, but the whitelist can implement a priority processing policy, and the access request matching the whitelist (that is, when the access request matches a whitelist record in the whitelist)
  • the request is directly released, that is, the access request is directed to its destination backend server (host), and the backend server directly responds to the access request.
  • the aggregator 103 can asynchronously load the current blacklist and the predefined whitelist to the load balancer 101.
  • the listening thread of the aggregator 103 can provide a blacklist to the load balancer 101.
  • the listener thread acquires the current blacklist at the aggregator 103 and sends the blacklist along with the whitelist to the load balancer 101.
  • the load balancer 101 is the actual performer of the "processing action" specified by each record in the blacklist.
  • the whitelist does not have to be maintained in the aggregator 103, for example, it can also be maintained in the load balancer 101.
  • the load balancer 101 will release the access request regardless of whether the access request matches the blacklist. If the access request does not match the whitelist but matches the blacklist, the load balancer 101 takes a corresponding processing action according to the specific record matched in the blacklist. If the access request does not match the whitelist or the blacklist, then the access request is released.
  • the load balancer 101 performs the following processing for each access request from the network according to the blacklist and the whitelist:
  • the access request matches the whitelist or does not match the blacklist, the access request is released, and the corresponding UDP packet is constructed and sent to the aggregator (because only such release traffic is actually sent to the corresponding backend server). Traffic, the aggregator will continue to calculate the "statistics" of such access requests);
  • the access request When the access request does not match the whitelist but matches the blacklist, the access request is operated according to the processing action specified by the matching record in the blacklist.
  • FIG. 2 illustrates a method in which a load balancer in accordance with the present invention limits access requests from a network.
  • step S11 the load balancer receives an access request from the network and parses the received access request into a UDP message.
  • step S12 the UDP message is transmitted to the aggregator.
  • the load balancer determines a processing action for the received access request based on a current whitelist and a current blacklist at the load balancer, wherein the blacklist specifies processing actions for certain access requests.
  • the load balancer determines whether the access request matches the whitelist. If it matches, then go to S14, the access request is released, and sent to the corresponding background server for further processing. Otherwise, it proceeds to step S15.
  • step S15 the load balancer determines whether the access request matches the blacklist. If there is no match, then go to S14, the access request is released, and sent to the corresponding background server for further processing. Otherwise, it proceeds to step S16.
  • the load balancer processes the access request according to the processing action specified by the matching record in the blacklist.
  • the concentrator receives the UDP packet sent by the load balancer, accumulates the received UDP packet according to the predefined data item combination and the predefined duration to generate statistical data, and according to the request of the aggregator Send statistics to it.
  • the statistical data includes the number of UDP messages including the predefined data item combination accumulated in the current time period of a predetermined length of time.
  • the aggregator requests statistical data from the aggregator according to a predetermined request time interval, receives statistical data from the aggregator, generates a blacklist of access requests according to the received statistical data and a predefined rule, and according to the The load balancer's request sends the current blacklist at the aggregator to the load balancer.
  • the aggregator can include multiple aggregation units.
  • the process by which each aggregation unit performs accumulation to generate statistical data is described in detail below.
  • the aggregation unit generates statistical data for a specific data item combination based on the UDP message received from the load balancer.
  • the aggregation unit comprises:
  • a receiving part configured to receive a UDP packet sent by the load balancer, where the UDP packet is generated by a load balancer parsing an access request from a network;
  • the calculating part accumulates the received UDP packets according to the predefined data item combination and the predefined duration to generate statistical data
  • the sending part sends the current statistics to the aggregator based on the request.
  • Each HTTP request from the network can only be parsed into a UDP message by the load balancer 101 and transmitted to one of the aggregation unit clusters of the aggregator 102.
  • Each aggregation unit is combined according to predefined conditions, such as a specific data item according to a user name userid, an IP address ip, and a universal resource identifier uri in a UDP message (ie, a combination of specific values contained in these specific data items) ), and the specified length of time, visit Ask for the cumulative number of requests.
  • the aggregation unit For the received UDP packet, the aggregation unit extracts data items such as userid, ip, uri, etc. from the UDP packet, with the specified combination of the data items (there may be various combinations) as the data item combination for accumulation, and Accumulate the number of requests according to the set duration (which can include short and long periods).
  • data items such as userid, ip, uri, etc.
  • This combination of data items can be one or more of the following:
  • this combination of data items can also be a combination of other data items.
  • the aggregation unit After the accumulation is completed, the aggregation unit generates statistical data, for example, in the form of a web page in the JSONS data format. Upon receiving the request from the aggregator 103, the aggregation unit transmits the statistical data to the aggregator 103.
  • the corresponding variable is initialized, and the current time of receiving the "first UDP packet" is taken as the effective start calculation time. , the number of visits is set to 1, the start time is set to the current time, and the duration is set to 0:
  • lasting_length is the duration from the start of the calculation of the time start_time to the duration of the current time current_time.
  • the reception time of the message falls within the first preset time period from the start_time (effective start calculation time)
  • the number of accesses is incremented by 1
  • the duration is the difference between the current time and the start_time ( Assuming that the predetermined duration is set to 60 seconds, then one hour can be divided into 60 time periods of a preset length, which is the first 60 seconds from the effective start calculation time. time).
  • start_time (effective start calculation time) is updated, wherein
  • the receiving time of the message is taken as the end point of a new time period whose length is equal to the preset duration, and the starting time of the new time period (ie, the time from the receiving time of the message minus the preset duration) It is then recorded as valid to start the calculation time.
  • the parameter total_count ie, the number of accesses
  • the duration is updated to the preset duration.
  • the current receiving time will be set as the effective start calculation time, the number of accesses is reset to 1 and the duration is 0, which is equivalent to At initialization, the traffic will be recalculated at this time.
  • the pseudo code of the corresponding algorithm is as follows:
  • Total_count total_count-total_count/60/X*(diff-60*X)+1;
  • Lasting_length 60*X
  • the preset duration can be set to two types, short time period and long time period. Set to a short period of time to prevent dos fast attacks, that is, a large number of connection requests impact the server in a short period of time, so that all available resources are exhausted, and finally the computer can no longer handle legitimate user requests.
  • the growth period is set to prevent dos slow attacks.
  • the aggregator 103 reads statistical data on the number of requests from all the aggregation units according to a specific classification, summarizes the statistics, generates a blacklist according to the predefined policy, and asynchronously loads the blacklist to the load balancer 101.
  • composition and specific working principle of the aggregator 103 are as shown in FIG.
  • the aggregator 103 includes a receiving portion 1031, a generating portion 1032, a listening portion 1033, a configuration file 1036, and a database 1037.
  • the receiving portion 1031 may create a receiving thread for each aggregation unit, and the receiving thread reads the combination of different data items from the web page output of each aggregation unit at intervals (eg, one minute).
  • the statistical data is, for example, data in the JSONSG format.
  • the generating portion 1032 of the aggregator 103 aggregates the statistical data received from the respective clustering units in accordance with a predetermined combination of data items.
  • These predetermined combinations of data items may be userid+ip+uri, ip+uri, userid+uri, and the like.
  • Hashmap is an implementation of the Map interface based on the hash table. It is used for storage, searching and other functions in computer programming.
  • the predefined blacklist threshold for the combination of data items in the configuration file such as 0.5QPS (query rate per second)
  • userid or ip is added to the blacklist , thus generating the current blacklist.
  • the inbound thread in the generation portion 1032 stores the summary data in the hashmap into the database 1037.
  • a dual hashmap mechanism is used in order to not lock the shared hashmap between the inbound thread and the receiving thread.
  • the monitoring terminal 200 of the system can directly acquire the data of the database 1037 through the HTTP protocol, that is, the data can be directly exported in the JSONS format without passing through the aggregator 103.
  • the listening portion 1033 of the aggregator 103 is for providing the load balancer 101 with a blacklist.
  • the listening part 1033 acquires the current blacklist and sends the current blacklist together with the whitelist to the load balancer 101.
  • the aggregator 103 can employ a mode in which the master works from two servers, wherein the master server can function as a normal working machine and the slave server can act as a data backup machine.
  • the secondary server can automatically take over all the work of the primary server, thus ensuring high reliability of the system.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Computing Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Computer And Data Communications (AREA)

Abstract

本发明涉及一种对访问请求进行限制的系统和方法，在该系统中，负载均衡器接收访问请求，把所接收的访问请求解析成UDP报文，并且把UDP报文传送给聚集器；聚集器根据预定义数据项组合和预定义时长对接收到的UDP报文进行累计以产生统计数据，并且响应于汇总器的请求而向其发送统计数据；汇总器按照预定的请求时间间隔向聚集器请求统计数据，接收来自所述聚集器的统计数据，根据所接收的统计数据和预定义规则生成访问请求的黑名单，并且响应于来自所述负载均衡器的请求而把黑名单发送给所述负载均衡器。所述负载均衡器根据当前的黑名单确定对所接收的每个访问请求的处理动作。

Description

一种对访问请求进行限制的系统和方法 技术领域
本发明涉及一种对来自网络的访问请求进行限制的系统及其相应的方法。
背景技术
传统的负载平衡器(Load Balancer)，如采用linux下HaProxy的配置，用于读取来自网络的每一个HTTP请求中所包含的cookies或URL解释，并且基于这些信息重写报头并将HTTP请求发往后端服务器群，使得后端服务器群中各服务器达到流量、资源占用的平衡状态。但是，传统的负载平衡器不会自动对网络的流量进行滤过检查，不能对攻击流量进行限流或丢弃。
在现有技术中，已知有基于流量清洗技术通过侦测TCP分组来实现针对ICMP/TCP/UDP流攻击进行防护的，通过重发TCP/UDP分组来实现。这种已知方案仅对TCP/UDP层的分组有效，它对于预防需解密的开放式系统互联参考模型(OSI)第7层应用层的HTTP流攻击无能为力。
一种想法是分析对URL的访问，根据单位时间访问量如QPS来限制访问请求。针对一个大型网站的URL的访问的分析通常要消耗很多内存。一般来说，对于数据项IP地址、用户标识(USERID)、统一资源定位符(URL)等的任何一个组合，需要记录与该组合相关联的每一个访问的时间戳(timestamp)。当需要计算QPS时，将各个时间点过滤或排序，既费时又耗内存空间。
在现有技术中，当需要计算QPS时，将各个时间点过滤或排序，既费时又耗内存空间。
发明内容
本发明的目的是提供至少部分解决上述问题的系统和方法。本发明特别适合于解决网络的DOS攻击(拒绝服务攻击)和DDOS攻击(分布式的拒绝服务攻击)领域。该技术尤其适合于针对HTTP流攻击(http flooding)的防范。本发明通过对用户流量的累计和分析，找到符合模式匹配的攻击流量并加以限流或丢弃，从而保护其后端服务器。
根据本发明的一个方面，提供一种一种对访问请求进行限制的系统，该系统包括负载均衡器、聚集器和汇总器。所述负载均衡器被配置为接收来自网络的每一个访问请求，把所接收的该访问请求解析成一个UDP报文并且把该UDP报文传送给所述聚集器；所述聚集器被配置为根据预定义数据项组合和预定义时长对从所述负载均衡器接收的多个UDP报文进行累计以产生统计数据，并且响应于来自所述汇总器的请求向其发送所累计的统计数据；所述汇总器被配置为按照预定的请求时间间隔向所述聚集器请求统计数据，接收来自所述聚集器的统计数据，根据所接收的统计数据和预定义规则生成访问请求的黑名单，并且响应于来自所述负载均衡器的请求而把所述黑名单发送给所述负载均衡器，其中所述黑名单指定了对一个或多个特定访问请求中的每一个要作出的处理动作。其中，所述负载均衡器进一步被配置为根据当前的黑名单确定对所接收的每个访问请求的处理动作。
根据本发明的另一方面，提供一种对访问请求进行限制的计算机实现的方法，所述方法包括：由负载均衡器接收来自网络的每一个访问请求，把所接收的访问请求解析成一个UDP报文、并且把该UDP报文传送给聚集器；由所述负载均衡器根据当前的黑名单确定对所接收的该访问请求的处理动作，其中所述黑名单指定了对一个或多个特定访问请求中的每一个要作出的处理动作。其中，所述聚集器接收所述负载均衡器发送的UDP报文，根据预定义数据项组合和预定义时长 对所接收的多个UDP报文进行累计以产生统计数据，并且响应于来自汇总器的请求向其发送所累计的统计数据；所述汇总器按照预定的请求时间间隔向所述聚集器请求统计数据，接收来自所述聚集器的统计数据，根据所接收的统计数据和预定义规则生成访问请求的黑名单，并且响应于来自所述负载均衡器的请求而把所述黑名单发送给所述负载均衡器。
根据本发明，所述统计数据包括在所述预定义时长内累计的含有所述预定义数据项组合的UDP报文的数量。
根据本发明，所述网络是因特网，所述访问请求是来自因特网的HTTP请求。
根据本发明，其中所述聚集器包括多个彼此独立运行的聚集单元，所述负载均衡器把每个UDP报文传输至所述多个聚集单元中的一个聚集单元，所述多个聚集单元中的每一个聚集单元分别进行所述累计。所述汇总器从所述多个聚集单元分别接收统计数据，产生汇总的统计数据。并且所述黑名单根据所述汇总的统计数据和所述预定义规则生成。
根据本发明，所述负载均衡器还包括预定义的白名单，所述白名单包括关于要被放行的访问请求的一个或多个记录。并且所述负载均衡器对所接收的每个访问请求按如下操作确定对该访问请求的处理动作：当该访问请求与所述白名单中的一个记录相匹配、或者不与所述黑名单中的任何一个记录相匹配时，对该访问请求放行；当该访问请求不与所述白名单中的任何一个记录相匹配、并且与所述黑名单中的一个记录相匹配时，对该访问请求按照黑名单中的匹配记录所指定的处理动作来进行操作。此外，所述负载均衡器中的所述预定义的白名单可以是从所述汇总器连同所述黑名单一起获得的。
根据本发明的系统，其中所述黑名单中规定的所述处理动作包括下述之一：拦截、重定向和延时。
附图说明
下而将参考附图详细地描述本发明的实施例，其中：
图1是根据本发明的用于对访问请求进行限制的系统的结构图；
图2是根据本发明的负载均衡器处理接收的访问请求的时序图；
图3是根据本发明的汇总器的实例性组成的框图。
具体实施方式
本发明提供利用统计数据对来自网络的访问请求进行限制的系统和方法。
一个访问请求，例如HTTP请求，至少包括如下数据项(data fields)：该查询用户要访问的目的域名(host)；用户请求的通用资源标示符(uri)；用户名(uid)；客户端IP地址(cip)，等等。
根据本发明，本发明提供一种对来自网络的访问请求进行限制的系统，该系统包括负载均衡器、聚集器和汇总器。
所述负载均衡器接收来自所述网络的访问请求，把所接收的每个访问请求解析成一个UDP报文并且把该UDP报文传送给聚集器。
该聚集器根据预定义数据项组合和预定义时长对接收到的多个UDP报文进行累计以产生统计数据，并且响应于来自汇总器的请求向其发送统计数据。所述统计数据包括在当前的预定义时长内累计的含有所述预定义数据项组合的UDP报文的数量。
所述汇总器按照预定的请求时间间隔向所述聚集器请求关于访问请求的统计数据，接收来自所述聚集器的统计数据，根据所接收的统 计数据和预定义规则生成访问请求的黑名单，并且根据所述负载均衡器的请求把该黑名单发送给所述负载均衡器，其中所述黑名单指定了对一个或多个特定访问请求的处理动作。
所述负载均衡器根据当前黑名单确定对所接收的每个访问请求的处理动作。
下面结合具体实施例详细描述根据本发明实施例的系统。但应该理解，具体实施例仅用于理解本发明的精神和具体实现，但本发明并不受限于特定的实施例。
图1示出根据本发明实施例的系统100，该系统100包括3个模块：
负载均衡器101，
聚集器102，和
汇总器103。
负载均衡器101接收来自网络的访问请求，如来自因特网的HTTP请求，并将接收的每个访问请求解析成UDP报文、并将该UDP报文传输给聚集器102。
由于处理UDP报文消耗的资源少、并且处理速度快，因此可以大大减小负载均衡器的额外开销。此外，UDP协议不属于连接型协议，因此会存在一定的报文丢失现象。但是，这种丢失在可控误差范围之内，对系统的健硕性(robustness)影响可忽略。
聚集器102可包括多个聚集单元。多个聚集单元可以水平地独立扩展，即每个聚集单元是独立部署和运行的，因此它们相互之间没有影响和依赖。如果当前的聚集器的处理能力不能满足需要，可以方便地增加服务器资源以部署新的聚集单元。这样可以较好地满足大交通流量的需要。
负载均衡器101把每个UDP报文传输至一个选定的聚集单元。而各个聚集单元对所接收的UDP报文分别进行累计并产生统计数据。
根据本发明的一个实施例，负载均衡器101用逐个轮流的方式向多个聚集单元发UDP报文。例如，第一个UDP报文发给第一个聚集单元，第二个UDP报文发给第二个聚集单元，依次把所生成的UDP报文按顺序发给各个聚集单元。在向最后一个聚集单元发送了一个UDP报文之后，再重新向第一个聚集单元发送UDP报文。
每个聚集单元按照预定义的规则或条件，如预定义的数据项组合、预定义的时间间隔等，对接收到的UDP报文进行分析并累计。每个UDP报文中包含了用户的HTTP访问请求的信息，比如用户访问的目的域名(host)、用户请求的通用资源标示符(uri)、用户名(uid)、客户端IP地址(cip)等。聚集单元针对具有同一个目的域名(相同的host)、同一来源(相同的uid，相同的cip)的访问请求的数量进行计数，进而获得来自此来源的访问请求的数量。
根据在限定的时间段内(如最近60秒或最近5分钟)接收到的来自某一来源的访问请求的数量，可以得出来自此来源的单位时间内的访问请求的数量。比如，可以通过将在该限定的时间段内接收到的来自该来源的访问请求的总数量除以该时间段的时长，来获得关于该来源的每秒查询率(QPS)的值。
然后，聚集单元根据汇总器103的请求向其发送在该聚集单元处产生的有关访问请求的最新的统计数据。
汇总器103可以每隔一定时间，例如每隔10秒，向聚集器或者向其中的每一个聚集单元请求一次“统计数据”，这种请求例如可以是以HTTP的形式。
当聚集器的每个聚集单元收到来自汇总器103的请求后，则把其所产生的有关访问的最新的统计数据构造成一个响应报文，例如HTTP形式的响应报文，并且把包含最新统计数据的响应报文发给汇总器103。
汇总器103在接收来自各聚集单元的响应报文后，根据特定分类来处理这些报文中包含的统计数据，进而产生汇总的统计数据，并根据预定义规则来生成黑名单。
根据本发明的一个实施例，黑名单中的一条黑名单记录可以包含以下4个参数：
目的域名(host)，
源用户名(uid)，
源客户端IP(cip)，和
处理动作(action)。
该黑名单记录的含义可理解为，对于发送给该目的域名(host)的一个请求，如果该请求中包含的用户名和客户端IP等同于该记录中对应的值(源用户名、源客户端IP)，则采取该记录中指定的处理动作。该处理动作例如可以是对该请求进行拦截、重定向或延时。
当一访问请求中包含的目的域名、用户名、客户端IP地址与黑名单中的一条记录中的目的域名、用户名、客户端IP地址完全相同时，则称该访问请求匹配该黑名单。
在汇总器103的配置文件1036中设置针对具体目的域名的在单位时间内的请求数量的阀值和当超过该阈值时的相应的处理动作。
汇总器103根据具有同一种数据项组合的访问请求的汇总的统计 数据来进行判断，当这种访问请求的单位时间访问量(QPS)值超过了配置文件1036中所设置的其目的域名对应的单位时间访问量的阀值时，则产生相应的黑名单记录，该记录包括该请求的目的域名、用户名和客户端IP以及相应的处理动作。所产生的黑名单记录被加入黑名单，从而产生当前黑名单。
另外，在汇总器103中还可以维护一个白名单。该自名单可以具有与黑名单同样的结构，但该白名单可以实现一种优先处理策略，对匹配白名单的访问请求(即在该访问请求与该白名单中的一个白名单记录相匹配时)，直接放行该访问请求，即把该访问请求引导至其目的后端服务器(host)，并由该后端服务器直接对该访问请求作出响应。
汇总器103可以将当前的黑名单和预定义的白名单异步加载至负载均衡器101。根据本发明的一个实施例，汇总器103的监听线程可向负载均衡器101提供黑名单列表。当负载均衡器101向汇总器103发起请求时，监听线程获取在汇总器103处的当前黑名单，并把该黑名单连同白名单一起发送给负载均衡器101。
负载均衡器101是黑名单中每条记录所指定的“处理动作”的实际执行者。
当然白名单并不一定要在汇总器103中维护，例如也可保持在负载均衡器101中。
如果一访问请求匹配了白名单，由于负载均衡器101的处理原则是白名单优先，因此不论该访问请求是否匹配黑名单，负载均衡器101都将放行该访问请求。如果该访问请求不匹配白名单但是匹配了黑名单，则负载均衡器101根据黑名单中所匹配的具体记录采取相应的处理动作。如果该访问请求既不匹配白名单也不匹配黑名单，则对该访问请求放行。
根据本发明，负载均衡器101根据黑名单和白名单，对来自网络的每个访问请求做出如下处理：
当该访问请求匹配白名单或者不匹配黑名单时，则对该访问请求放行，并且构造相应的UDP报文并发给聚集器(因只有这种放行流量才是真正发到相应的后端服务器的流量，聚集器要继续计算这种访问请求的“统计数据”)；
当该访问请求不匹配白名单但匹配黑名单时，对该访问请求按照黑名单中的匹配记录所指定的处理动作来进行操作。
图2示出了根据本发明的负载均衡器对来自网络的访问请求进行限制的方法。
在步骤S11，所述负载均衡器接收来自所述网络的访问请求，并把所接收的该访问请求解析成一个UDP报文。
在步骤S12，把该UDP报文传送给聚集器。
然后，所述负载均衡器根据在所述负载均衡器处的当前白名单和当前黑名单确定对所接收的该访问请求的处理动作，其中所述黑名单指定对某些访问请求的处理动作。
在步骤S13，负载均衡器判断该访问请求是否与白名单匹配。如果匹配，则转S14，即放行该访问请求，将其发往相应的后台服务器以进一步处理。否则，进至步骤S15。
在步骤S15，负载均衡器判断该访问请求是否与黑名单匹配。如果不匹配，则转至S14，即放行该访问请求，将其发往相应的后台服务器以进一步处理。否则，进至步骤S16。
在步骤S16，负载均衡器按照黑名单中的匹配记录所指定的处理动作来处理该访问请求。
根据本发明，所述聚集器接收所述负载均衡器发送的UDP报文，根据预定义数据项组合和预定义时长对接收到的UDP报文进行累计以产生统计数据，并且根据汇总器的请求向其发送统计数据。所述统计数据包括在长度为预定义时长的当前时间段内累计的包含有所述预定义数据项组合的UDP报文的数量。
所述汇总器按照预定的请求时间间隔向所述聚集器请求统计数据，接收来自所述聚集器的统计数据，根据所接收的统计数据和预定义规则生成访问请求的黑名单，并且根据所述负载均衡器的请求把在汇总器处的当前黑名单发送给所述负载均衡器。
如前所述，聚集器可包括多个聚集单元。下面详细描述每个聚集单元执行累计来产生统计数据的过程。
聚集单元根据从负载均衡器接收的UDP报文产生针对特定数据项组合的统计数据。根据本发明的一个实施例，聚集单元包括：
接收部分，用于接收所述负载均衡器发送的UDP报文，所述UDP报文是通过负载均衡器解析来自网络的访问请求而产生的；
计算部分，根据预定义数据项组合和预定义时长对所接收的UDP报文进行累计以产生统计数据；
发送部分，根据汇总器的请求向其发送当前的统计数据。
对于来自网络的每个HTTP请求，它只能被负载均衡器101解析成一个UDP报文并传输至聚集器102的聚集单元集群中的某一个聚集单元上。每个聚集单元根据预定义的条件，如根据UDP报文中的如用户名userid、IP地址ip和通用资源标示符uri的特定数据项组合(即这些特定数据项中含有的特定的值的组合)、以及指定的时长，进行访 问请求的次数的累计。
对接收的UDP报文，聚集单元从该UDP报文中抽取数据项如userid、ip、uri等，以这些数据项的指定组合(可以存在多种组合)作为用于累计的数据项组合，并且按照设定的时长(可包括短时间段和长时间段)进行请求次数的累计。
这种数据项组合可以是以下的一种或多种：
userid+ip+uri；
userid+uri；
ip+uri。
当然这种数据项组合还可以是包括其他数据项的组合。
累计完成后，聚集单元生成统计数据，例如以JSONS数据格式的web页形式。在接收到汇总器103的请求时，聚集单元把统计数据传送给汇总器103。
一个聚集单元计算流量统计的详细过程如下：
在启动累计过程之后，当接收到具有给定数据项组合的第一个UDP报文时，对相应变量进行初始化，将接收到该“第一个UDP报文”的当前时刻作为有效开始计算时刻，访问次数设为1，开始时间设为该当前时间，持续时间设为0：
total_count＝1；
start_time＝time(NULL)；
lasting_length＝0；
在后续的累计计算中，lasting_length是从开始计算时刻start_time到当前时刻current_time的持续时间的时长。
对于后续接收的具有该数据项组合的每一个报文，根据该报文的 接收时间与预设时长的关系，访问的计数分为三种情况：
(1)当该报文的接收时间落在自start_time(有效开始计算时刻)起的第一个预设时间段内时，访问次数(total_count)递增1，持续时间为当前时间与start_time之差(假设预定时长设置为60秒，那么一小时就可分为60个长度为预设时长的时间段，该“第一个预设时间段”即指自有效开始计算时刻起的第一段60秒的时间)。
(2)当该报文的接收时间落在紧随所述第一个预设时间段之后的第二个预设时间段中时，start_time(有效开始计算时间)将被更新，其中，使得该报文的接收时间被作为长度等于所述预设时长的一个新时间段的末点，而该新时间段的起点时刻(即从该报文的接收时间减去所述预设时长的时刻)则被记为有效开始计算时间。此时参数total_count(即访问次数)将通过下述方式被更新：先计算所述第一个时间段内的QPS，将QPS乘以该新时间段与第一个时间段重叠的那部分时间的长度，再递增1。而持续时间则更新为所述预设时长。
(3)当该报文的接收时间落在第二个时间段之后时，当前的接收时间将被定为有效开始计算时间，访问次数重新设置为1，持续时间为0，即这种情况等同于初始化，此时流量将被重新计算。
以预设时长为60秒为例，相应算法伪代码如下：
diff＝current_time-start_time；
if(diff＜60*X){#当接收时间落在第一个时间段内时
total_count++；lasting_length＝diff；
}else if(diff＞2*60*X){#当接收时间落在第二个时间段之后
total_count＝1；
start_time＝current_time；
lasting_length＝0；
}else{#当接收时间落在第二个时间段之内
start_time＝current_time-60*x；
total_count＝total_count-total_count/60/X*(diff-60*X)+1；
lasting_length＝60*X；
}
预设时长可被设置为短时间段和长时间段两种。设置成短时间段是为了防范dos快攻击，即短时间内大量的连接请求冲击服务器，使得所有可用的资源都被消耗殆尽，最终计算机无法再处理合法用户的请求。而设置成长时间段是为了防范dos慢攻击。
下面详细描述汇总器103的汇总处理和黑名单的生成。
汇总器103根据特定分类读取来自所有聚集单元的关于请求次数的统计数据，汇总这些统计数据，根据预定义策略生成黑名单列表，并把黑名单异步加载至负载均衡器101。
汇总器103的构成和具体工作原理如图3所示。
汇总器103包括接收部分1031、生成部分1032、监听部分1033、配置文件1036和数据库1037。
根据本发明的实施例，接收部分1031可为每个聚集单元创建一个接收线程，接收线程每隔一段时间(如一分钟)从每个聚集单元的Web页输出中读取针对不同的数据项组合的请求数的统计数据。该统计数据例如是JSONSG格式的数据。
汇总器103的生成部分1032把从各个聚集单元器接收的统计数据根据预定的数据项组合进行汇总。
这些预定的数据项组合可以是userid+ip+uri、ip+uri、userid+uri等。
针对不同的数据项组合的请求的数量会分别被存储到对应的 hashmap中。Hashmap是基于哈希表的Map接口的实现，在计算机编程中用于存储、查找等功能。
当具有某个数据项组合的请求的数量超过配置文件中预定义的关于数据项组合的黑名单阀值时，如0.5QPS(每秒查询率)，userid或ip便被添加到黑名单列表中，从而产生当前的黑名单。
根据一个实施例，生成部分1032中的入库线程将hashmap里的汇总数据存入数据库1037。根据本发明的实施例，为了在入库线程和接收线程之间不对共享的hashmap加锁，使用双hashmap的机制。
另外，系统的监控终端200可通过HTTP协议直接获取数据库1037的数据，即，该数据可以直接以JSONS格式被导出而不通过汇总器103。
汇总器103的监听部分1033用于给负载均衡器101提供黑名单列表。当负载均衡器101发起请求时，监听部分1033获取当前的黑名单，并把该当前的黑名单连同白名单一起发送给负载均衡器101。
根据本发明的一个实施例，汇总器103可采用主从两个服务器一起工作的模式，其中的主服务器可以作为正常工作机器，而从服务器可以作为数据备份机器。当主服务器出现故障以后，从服务器能够自动接管主服务器的所有工作，从而保证系统的高可靠性。
尽管结合附图和特定实施例描述了本发明。但附图和说明书中的实施例仅仅是本发明的实例，而不是用于限制本发明。本发明的范围由权利要求书来限定。

Claims (12)

  1. 一种对访问请求进行限制的系统，该系统包括负载均衡器、聚集器和汇总器，其中
    所述负载均衡器被配置为接收来自网络的每一个访问请求，把所接收的该访问请求解析成一个UDP报文并且把该UDP报文传送给所述聚集器；
    所述聚集器被配置为根据预定义数据项组合和预定义时长对从所述负载均衡器接收的多个UDP报文进行累计以产生统计数据，并且响应于来自所述汇总器的请求向其发送所累计的统计数据；
    所述汇总器被配置为按照预定的请求时问问隔向所述聚集器请求统计数据，接收来自所述聚集器的统计数据，根据所接收的统计数据和预定义规则生成访问请求的黑名单，并且响应于来自所述负载均衡器的请求而把所述黑名单发送给所述负载均衡器，其中所述黑名单指定了对一个或多个特定访问请求中的每一个要作出的处理动作，并且
    所述负载均衡器进一步被配置为根据当前的黑名单确定对所接收的每个访问请求的处理动作。
  2. 根据权利要求1所述的系统，所述统计数据包括在所述预定义时长内累计的包含有所述预定义数据项组合的UDP报文的数量。
  3. 根据权利要求1所述的系统，其中
    所述聚集器包括多个彼此独立运行的聚集单元，所述负载均衡器把每个UDP报文传输至所述多个聚集单元中的一个聚集单元，所述多个聚集单元中的每一个聚集单元分别进行所述累计，
    所述汇总器从所述多个聚集单元分别接收统计数据，并产生汇总的统计数据，并且
    所述黑名单根据所述汇总的统计数据和所述预定义规则生成。
  4. 根据权利要求1所述的系统，其中
    所述负载均衡器还包括预定义的白名单，所述白名单包括关于要被放行的访问请求的一个或多个记录，并且
    所述负载均衡器对所接收的每个访问请求按如下操作确定对该访问请求的处理动作：
    当该访问请求与所述白名单中的一个记录相匹配、或者不与所述黑名单中的任何一个记录相匹配时，对该访问请求放行，
    当该访问请求不与所述白名单中的任何一个记录相匹配、并且与所述黑名单中的一个记录相匹配时，对该访问请求按照所述黑名单中的匹配记录所指定的处理动作来进行操作。
  5. 根据权利要求4所述的系统，其中，所述负载均衡器中的所述预定义的白名单是从所述汇总器连同所述黑名单一起获得的。
  6. 根据权利要求1所述的系统，其中所述黑名单中规定的所述处理动作包括下述之一：拦截、重定向和延时。
  7. 一种对访问请求进行限制的方法，所述方法包括：
    由负载均衡器接收来自网络的每一个访问请求，把所接收的访问请求解析成一个UDP报文、并且把该UDP报文传送给聚集器；
    由所述负载均衡器根据当前的黑名单确定对所接收的该访问请求的处理动作，其中所述黑名单指定了对一个或多个特定访问请求中的每一个要作出的处理动作，
    其中，
    所述聚集器接收所述负载均衡器发送的UDP报文，根据预定义数据项组合和预定义时长对所接收的多个UDP报文进行累计以产生统计数据，并且响应于来自汇总器的请求向其发送所累计的统计数据；
    所述汇总器按照预定的请求时间间隔向所述聚集器请求统计数据，接收来自所述聚集器的统计数据，根据所接收的统计数据和预定义规则生成访问请求的黑名单，并且响应于来自所述负载均衡器的请求而把所述黑名单发送给所述负载均衡器。
  8. 根据权利要求7所述的方法，所述统计数据包括在所述预定义时长内累计的包含有所述预定义数据项组合的UDP报文的数量。
  9. 根据权利要求7所述的方法，其中
    所述聚集器包括多个彼此独立运行的聚集单元，所述负载均衡器把每个UDP报文传输至所述多个聚集单元中的一个聚集单元，所述多个聚集单元中的每一个聚集单元分别进行所述累计，
    所述汇总器从所述多个聚集单元分别接收统计数据，并产生汇总的统计数据，并且
    所述黑名单根据所述汇总的统计数据和所述预定义规则生成。
  10. 根据权利要求7所述的方法，其中
    所述负载均衡器还包括预定义的白名单，所述白名单包括关于要被放行的访问请求的一个或多个记录，并且
    所述负载均衡器对所接收的每个访问请求按如下操作确定对该访问请求的处理动作：
    当该访问请求与所述白名单中的一个记录相匹配、或者不与所述黑名单中的任何一个记录相匹配时，对该访问请求放行，
    当该访问请求不与所述白名单中的任何一个记录相匹配、并且与所述黑名单中的一个记录相匹配时，对该访问请求按照所述黑名单中的匹配记录所指定的处理动作来进行操作。
  11. 根据权利要求10所述的方法，其中，所述负载均衡器中的所述预定义的白名单是从所述汇总器连同所述黑名单一起获得的。
  12. 根据权利要求7所述的方法，其中所述黑名单中规定的所述处理动作包括下述之一：拦截、重定向和延时。
PCT/CN2016/070522 2015-01-09 2016-01-08 一种对访问请求进行限制的系统和方法 Ceased WO2016110273A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US15/542,086 US10735501B2 (en) 2015-01-09 2016-01-08 System and method for limiting access request
RU2017128207A RU2666289C1 (ru) 2015-01-09 2016-01-08 Система и способ для ограничения запросов доступа
JP2017554634A JP2018508166A (ja) 2015-01-09 2016-01-08 アクセス要求を規制するシステムおよび方法

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510011598.9A CN104580216B (zh) 2015-01-09 2015-01-09 一种对访问请求进行限制的系统和方法
CN201510011598.9 2015-01-09

Publications (1)

Publication Number Publication Date
WO2016110273A1 true WO2016110273A1 (zh) 2016-07-14

Family

ID=53095397

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/070522 Ceased WO2016110273A1 (zh) 2015-01-09 2016-01-08 一种对访问请求进行限制的系统和方法

Country Status (5)

Country Link
US (1) US10735501B2 (zh)
JP (2) JP2018508166A (zh)
CN (1) CN104580216B (zh)
RU (1) RU2666289C1 (zh)
WO (1) WO2016110273A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115913664A (zh) * 2022-10-31 2023-04-04 深圳市欧瑞博科技股份有限公司 一种控制设备智能接入的方法、系统及设备

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104580216B (zh) 2015-01-09 2017-10-03 北京京东尚科信息技术有限公司 一种对访问请求进行限制的系统和方法
CN104580228A (zh) * 2015-01-16 2015-04-29 北京京东尚科信息技术有限公司 对来自网络的访问请求产生黑名单的系统和方法
CN105939320A (zh) * 2015-12-02 2016-09-14 杭州迪普科技有限公司 处理报文的方法及装置
CN107454120A (zh) * 2016-05-30 2017-12-08 北京京东尚科信息技术有限公司 网络攻击防御系统和防御网络攻击的方法
CN110830496B (zh) * 2018-03-30 2021-08-17 厦门白山耘科技有限公司 一种防止扫描权限文件的系统的使用方法及作业方法
CN109241458A (zh) * 2018-07-11 2019-01-18 上海斐讯数据通信技术有限公司 一种基于路由器的广告拦截方法和路由器
CN109617932B (zh) * 2019-02-21 2021-07-06 北京百度网讯科技有限公司 用于处理数据的方法和装置
US11237270B2 (en) 2019-06-20 2022-02-01 Cilag Gmbh International Hyperspectral, fluorescence, and laser mapping imaging with fixed pattern noise cancellation
CN112953985B (zh) * 2019-12-10 2023-04-07 贵州白山云科技股份有限公司 请求数据处理方法、装置、介质及系统
US12519792B2 (en) * 2021-03-15 2026-01-06 Circle Media Labs Inc. Internet access filtering systems and methods
CN113179317B (zh) * 2021-04-27 2023-02-07 杭州迪普科技股份有限公司 内容重写设备的测试系统及方法
CN113726683B (zh) * 2021-09-09 2023-08-15 海尔数字科技(青岛)有限公司 访问限流方法、装置、设备、存储介质及计算机程序产品
CN113904839A (zh) * 2021-09-30 2022-01-07 杭州数梦工场科技有限公司 访问请求管理方法及装置
CN115174249B (zh) * 2022-07-18 2024-09-24 湖北天融信网络安全技术有限公司 安全日志的处理方法及电子设备、存储介质
CN115396376A (zh) * 2022-08-22 2022-11-25 平安科技（深圳）有限公司 负载均衡方法、装置、设备及存储介质
CN115484109A (zh) * 2022-09-23 2022-12-16 时照智能科技(上海)有限公司 一种针对异常访问ip的限流方法、系统以及存储介质
CN115694996A (zh) * 2022-10-31 2023-02-03 深圳供电局有限公司 增强网络服务安全的方法及其装置、存储介质、服务器

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101437030A (zh) * 2008-11-29 2009-05-20 成都市华为赛门铁克科技有限公司 一种防止服务器被攻击的方法、检测装置及监控设备
CN103746982A (zh) * 2013-12-30 2014-04-23 中国科学院计算技术研究所 一种http网络特征码自动生成方法及其系统
CN104104669A (zh) * 2014-06-17 2014-10-15 上海地面通信息网络有限公司 适用于因特网数据中心领域的抗DDoS攻击防护系统
CN104580216A (zh) * 2015-01-09 2015-04-29 北京京东尚科信息技术有限公司 一种对访问请求进行限制的系统和方法
CN104580228A (zh) * 2015-01-16 2015-04-29 北京京东尚科信息技术有限公司 对来自网络的访问请求产生黑名单的系统和方法
CN104579841A (zh) * 2015-01-09 2015-04-29 北京京东尚科信息技术有限公司 根据接收的udp报文产生对特定统计数据项的统计结果的系统

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020133603A1 (en) * 2001-03-13 2002-09-19 Fujitsu Limited Method of and apparatus for filtering access, and computer product
US7463590B2 (en) * 2003-07-25 2008-12-09 Reflex Security, Inc. System and method for threat detection and response
FR2872983A1 (fr) * 2004-07-09 2006-01-13 Thomson Licensing Sa Systeme de pare-feu protegeant une communaute d'appareils, appareil participant au systeme et methode de mise a jour des regles de pare-feu au sein du systeme
US7478429B2 (en) * 2004-10-01 2009-01-13 Prolexic Technologies, Inc. Network overload detection and mitigation system and method
US8089871B2 (en) * 2005-03-25 2012-01-03 At&T Intellectual Property Ii, L.P. Method and apparatus for traffic control of dynamic denial of service attacks within a communications network
JP4602158B2 (ja) * 2005-05-25 2010-12-22 三菱電機株式会社 サーバ装置保護システム
US9794272B2 (en) * 2006-01-03 2017-10-17 Alcatel Lucent Method and apparatus for monitoring malicious traffic in communication networks
JP2008135871A (ja) * 2006-11-27 2008-06-12 Oki Electric Ind Co Ltd ネットワーク監視システム、ネットワーク監視方法及びネットワーク監視プログラム
JP4900119B2 (ja) * 2007-08-01 2012-03-21 ヤマハ株式会社 ネットワーク機器
JP4319246B2 (ja) * 2007-12-12 2009-08-26 デュアキシズ株式会社 通信制御装置及び通信制御方法
JP5142956B2 (ja) * 2008-11-20 2013-02-13 日本電信電話株式会社 トラフィック情報管理サーバ及びトラフィック情報管理方法
JP2011049794A (ja) * 2009-08-27 2011-03-10 Alaxala Networks Corp パケットフロー統計値取得システム及びパケットフロー統計値取得方法
US20110083179A1 (en) * 2009-10-07 2011-04-07 Jeffrey Lawson System and method for mitigating a denial of service attack using cloud computing
US9197600B2 (en) * 2011-09-29 2015-11-24 Israel L'Heureux Smart router
CN103491053A (zh) * 2012-06-08 2014-01-01 北京百度网讯科技有限公司 Udp负载均衡方法、系统及装置
US9553809B2 (en) * 2013-04-16 2017-01-24 Amazon Technologies, Inc. Asymmetric packet flow in a distributed load balancer
RU133954U1 (ru) * 2013-04-29 2013-10-27 Федеральное государственное образовательное бюджетное учреждение высшего профессионального образования "Санкт-Петербургский государственный университет телекоммуникаций им. проф. М.А. Бонч-Бруевича" (СПбГУТ) Устройство защиты сети
US9055095B2 (en) * 2013-06-14 2015-06-09 Microsoft Technology Licensing, Llc DOS detection and mitigation in a load balancer

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101437030A (zh) * 2008-11-29 2009-05-20 成都市华为赛门铁克科技有限公司 一种防止服务器被攻击的方法、检测装置及监控设备
CN103746982A (zh) * 2013-12-30 2014-04-23 中国科学院计算技术研究所 一种http网络特征码自动生成方法及其系统
CN104104669A (zh) * 2014-06-17 2014-10-15 上海地面通信息网络有限公司 适用于因特网数据中心领域的抗DDoS攻击防护系统
CN104580216A (zh) * 2015-01-09 2015-04-29 北京京东尚科信息技术有限公司 一种对访问请求进行限制的系统和方法
CN104579841A (zh) * 2015-01-09 2015-04-29 北京京东尚科信息技术有限公司 根据接收的udp报文产生对特定统计数据项的统计结果的系统
CN104580228A (zh) * 2015-01-16 2015-04-29 北京京东尚科信息技术有限公司 对来自网络的访问请求产生黑名单的系统和方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115913664A (zh) * 2022-10-31 2023-04-04 深圳市欧瑞博科技股份有限公司 一种控制设备智能接入的方法、系统及设备

Also Published As

Publication number Publication date
JP2018508166A (ja) 2018-03-22
HK1204726A1 (zh) 2015-11-27
CN104580216A (zh) 2015-04-29
JP6726331B2 (ja) 2020-07-22
US10735501B2 (en) 2020-08-04
JP2019134484A (ja) 2019-08-08
US20180278678A1 (en) 2018-09-27
RU2666289C1 (ru) 2018-09-06
CN104580216B (zh) 2017-10-03

Similar Documents

Publication Publication Date Title
JP6726331B2 (ja) アクセス要求を規制するシステムおよび方法
US8661544B2 (en) Detecting botnets
EP2289221B1 (en) Network intrusion protection
US9584531B2 (en) Out-of band IP traceback using IP packets
US7623466B2 (en) Symmetric connection detection
CN105791213B (zh) 一种策略优化装置及方法
US10091198B2 (en) Rule-based fingerprint generation methods and apparatus
US10498618B2 (en) Attributing network address translation device processed traffic to individual hosts
KR20130014226A (ko) 공격 트래픽 형태별 특성에 따른 ｄｎｓ 플러딩 공격 탐지 방법
CA2947325A1 (en) Protocol type identification method and apparatus
CN104243408A (zh) 域名解析服务dns系统中监控报文的方法、装置及系统
HK1204728A1 (zh) 对来自网络的访问请求产生黑名单的系统和方法
Li et al. A case study of ipv6 network performance: Packet delay, loss, and reordering
Mohammadi et al. Practical extensions to countermeasure dos attacks in software defined networking
CN104579841B (zh) 根据接收的udp报文产生对特定统计数据项的统计结果的系统
Bellaïche et al. SYN flooding attack detection by TCP handshake anomalies
HK1204726B (zh) 一种对访问请求进行限制的系统和方法
HK1204727B (zh) 根据接收的udp报文产生对特定统计数据项的统计结果的系统
JP4489714B2 (ja) パケット集約方法、装置、およびプログラム
WO2024119923A1 (zh) 一种应用识别方法以及相关设备
Korn Defense mechanisms against network attacks and worms
Žádník Network monitoring based on ip data flows
Limmer Efficient Network Monitoring for Attack Detection

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16734933

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2017554634

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 15542086

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017128207

Country of ref document: RU

Kind code of ref document: A

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 07/11/2017)

122 Ep: pct application non-entry in european phase

Ref document number: 16734933

Country of ref document: EP

Kind code of ref document: A1