CN110944000A - OpenResty gateway feature anti-brushing method based on multi-Agent cluster - Google Patents
OpenResty gateway feature anti-brushing method based on multi-Agent cluster Download PDFInfo
- Publication number
- CN110944000A CN110944000A CN201911236035.4A CN201911236035A CN110944000A CN 110944000 A CN110944000 A CN 110944000A CN 201911236035 A CN201911236035 A CN 201911236035A CN 110944000 A CN110944000 A CN 110944000A
- Authority
- CN
- China
- Prior art keywords
- matrix
- gateway
- rule
- cluster
- openresty
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000004364 calculation method Methods 0.000 claims abstract description 34
- 230000004044 response Effects 0.000 claims abstract description 32
- 238000001514 detection method Methods 0.000 claims abstract description 23
- 238000011144 upstream manufacturing Methods 0.000 claims abstract description 8
- 238000012216 screening Methods 0.000 claims abstract description 4
- 239000011159 matrix material Substances 0.000 claims description 134
- 239000003795 chemical substances by application Substances 0.000 claims description 89
- 239000013598 vector Substances 0.000 claims description 28
- 238000012545 processing Methods 0.000 claims description 19
- 238000010200 validation analysis Methods 0.000 claims description 18
- 239000006185 dispersion Substances 0.000 claims description 5
- 238000012805 post-processing Methods 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 4
- 230000002265 prevention Effects 0.000 claims description 3
- 238000005201 scrubbing Methods 0.000 claims 2
- 238000013461 design Methods 0.000 abstract description 10
- 230000006870 function Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 230000001680 brushing effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000012938 design process Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
Abstract
The invention discloses a multi-Agent cluster OpenResty gateway feature-based anti-brush method, which comprises the following steps: sending an API request to the OpenResty gateway cluster; the gateway management system configures a detection rule of a gateway; after receiving the API request, the OpenResty gateway cluster records a log generated by the API request and filters the log according to a detection rule, wherein the log comprises a source IP and an environment of the API request, a URL (uniform resource locator) of the API request, a request parameter and a response parameter; the OpenResty gateway cluster transmits the filtered log to an Agent cluster; and the Agent cluster calculates the filtered log data stream according to the detection rule to obtain an execution rule, and sends the processed API request meeting the screening requirement to an upstream server cluster. The invention designs a multi-Agent cluster, analyzes the log in real time, supports multi-service rule calculation, quantifies the income value through the request parameter and the returned distribution of the IP address to perform anti-brushing and access control, is more accurate and simultaneously prevents misjudgment.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a multi-Agent cluster OpenResty gateway feature brushing prevention method.
Background
In the era of rapid popularization of micro services, back-end applications are increasing day by day and are also more and more independent and dispersed, a plurality of services all comprise a plurality of background applications, each application also has a plurality of domain names, in order to uniformly manage a plurality of services, the outside and the inside are isolated, gateways are adopted in the prior art, an OpenResty framework is adopted in most of the gateways, the framework is based on Nginx, the performance is excellent, a relatively friendly programming interface is provided for controlling API requests, so that a plurality of functions of API current limiting, access statistics, reverse proxy, firewall and the like are realized, and the upstream services of the gateways are effectively protected from attack and resource waste.
The OpenResty framework is, after all, a tool, and provides an interface for operating each stage of the life cycle of the API, but it does not implement the feature anti-brushing function, and the feature anti-brushing function and the anti-crawler function are different rules for different services, and need to be configured and analyzed separately. Therefore, the OpenResty framework designs a customized gateway management system with necessary requirements.
Disclosure of Invention
In view of this, the present invention provides a multi-Agent cluster OpenResty gateway feature-based anti-refresh method, which includes the steps of:
sending an API request to the OpenResty gateway cluster;
the gateway management system configures a detection rule of a gateway;
after receiving the API request, the OpenResty gateway cluster records a log generated by the API request and filters the log according to the detection rule, wherein the log comprises a source IP and an environment of the API request, a URL (uniform resource locator) of the API request, a request parameter and a response parameter;
the OpenResty gateway cluster transmits the filtered log to an Agent cluster;
the Agent cluster calculates the filtered log data stream according to the detection rule to obtain an execution rule, wherein the detection rule comprises the following steps: { rule number; the effective time; an effective object; numbering the agents; [ judgment of request parameter ]; [ judgment of response parameter ]; [ rule number of pre-dependency ]; [ rule number of post-processing ]; a rank; processing item }; the execution rule includes: { executing an object, whether to pull black, forwarding an address, and requesting parameter dispersion; limiting the frequency; processing items; [ calculation rule ], pass or not }; the calculation method is as follows:
step 1): the gateway management system filters all rules of the current time period to generate vectorsThe vector includes E(i)、E(u)、E(p)And E(q)In which E(i)For IP matrix [ IP address in all validation objects ]]、E(u)For URL matrix [ URL in all validation objects]、E(p)For determination of all request parameters in the request matrix [ all rules ]]And E(q)Response matrix [ judgment of all response parameters in all rules]；
Step 2): each Agent in the Agent cluster acquires a rule configured by the gateway management system, and the vector value is generated for the rule in each Agent:
step 3): performing vector analysis on the log data stream filtered by the OpenResty gateway cluster by adopting the methods in the step 1) and the step 2), generating matrix streams and transmitting the matrix streams to the multi-Agent cluster, wherein each matrix stream comprises the following vector E(i)’、E(u)’、E(p)', and E(q)’：
E(i)' is IP matrix [ IP addresses in all validation objects]、E(u)' is a URL matrix [ all URLs in validation object]、E(p)' determination of all request parameters in request matrix [ all rules]And E(q)' response matrix [ judgment of all response parameters in all rules]，
The matrix flow is:
step 4): multiplying each single-row matrix in the step 3) by the transposed matrix in the step 2), when the value in the single-row matrix in the step 3) is consistent with the value in the vector value in the step 2), the calculated value is 1, when the value in the single-row matrix is inconsistent with the value in the vector value, the calculated value is 0, and converting the single-row matrix stream in the gateway cluster into a result matrix with the item of 1 or 0;
step 5): generating a full matrix which has the same number of terms as the result matrix in the step 4) and each term is 1, multiplying the result matrix obtained in the step 4) by a transposed matrix of the full matrix by each Agent, wherein if the obtained numerical value is less than the number of terms of the matrix, the term in the result matrix is 0, which indicates that the condition is not passed, and if the obtained numerical value is equal to the number of terms of the matrix, the term in the result matrix is 1, which indicates that all the conditions are passed;
step 6): calculating the four matrixes respectively according to the step 5) to obtain four one-dimensional result matrixes, wherein if 0 item exists in the four matrixes, the row in the matrix flow is not detected, and an execution rule of the row is generated, and after the execution rule is obtained by the OpenResty gateway, the API request corresponding to the row in the matrix flow is processed;
and sending the processed API request meeting the screening requirement to an upstream server cluster.
Optionally, the OpenResty gateway cluster includes Lua logic.
Optionally, the Lua logic includes a log parsing plug-in, which is used to filter the log generated by the API request.
Optionally, the Lua logic further includes a feature anti-flush plug-in for adjusting the current limiting parameter and the route forwarding rule according to the detection rule.
Optionally, the number of OpenResty gateway clusters is at least one.
Optionally, a plurality of agents perform calculation corresponding to one API request. .
The invention designs a multi-Agent cluster for multi-service collaborative computing, supports collaborative computing of various rules, and provides good support for cross-service anti-brush and crawler analysis;
the multi-Agent cooperation designed by the invention can quantify the request parameter distribution and result distribution of the IP address, perform characteristic calculation on certain columns of the matrix and quantify the income value of the IP address, thereby more accurately judging the crawler and invalid access and reducing misjudgment;
the invention adopts a calculation method for converting parameters into matrixes, can calculate the characteristics of IP addresses in parallel and quickly obtain the judgment of the characteristics;
the invention adopts a matrix flow mode, is convenient for counting the period rules, and has higher efficiency under the condition of large-scale data flow.
Compared with the prior art, the multi-Agent cluster OpenResty gateway feature-based anti-brush method provided by the invention at least realizes the following beneficial effects:
most gateways in the prior art only consider current limiting, feature anti-brushing, and firewall for independent services. The current limit is not based on the CPU and memory dynamic generation parameters of the upstream service, the characteristic of the anti-brush is simply to match the request parameters, remove the request with obviously abnormal parameters, and is ineffective for the customized crawler, and the firewall is simply a black and white list. The invention designs a multi-Agent cluster, analyzes the OpenResty log in real time, supports multi-service rule calculation, quantifies the profit value to perform anti-brushing and access control through the request parameter and the returned distribution of the IP address, is more accurate, and simultaneously prevents misjudgment.
Of course, it is not necessary for any product in which the present invention is practiced to achieve all of the above-described technical effects simultaneously.
Other features of the present invention and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.
Fig. 1 is a flowchart of a multi-Agent cluster OpenResty gateway feature-based anti-refresh method provided in this embodiment 1;
fig. 2 is a physical architecture diagram of an OpenResty gateway in embodiment 2;
fig. 3 is a workflow diagram of an OpenResty gateway in embodiment 2;
FIG. 4 is a flow chart of gateway rules and detection in embodiment 2;
fig. 5 is an Agent cluster operation flow in embodiment 2.
Detailed Description
Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
Example 1:
with reference to fig. 1, the present invention provides a multi-Agent cluster openresistance gateway feature-based anti-brush method, which includes the following steps:
s1: sending an API request to the OpenResty gateway cluster;
it should be noted that the API request herein refers to a plurality of API requests.
S2: the gateway management system configures a detection rule of a gateway;
it is understood that the sequence of S1 and S2 can be reversed.
S3: after receiving the API request, the OpenResty gateway cluster records a log generated by the API request and filters the log according to the detection rule, wherein the log comprises a source IP and an environment of the API request, a URL (uniform resource locator) of the API request, a request parameter and a response parameter;
s4: the OpenResty gateway cluster transmits the filtered log to an Agent cluster;
s5: the Agent cluster calculates the filtered log data stream according to the detection rule to obtain an execution rule, wherein the detection rule comprises the following steps: { rule number; the effective time; an effective object; numbering the agents; [ judgment of request parameter ]; [ judgment of response parameter ]; [ rule number of pre-dependency ]; [ rule number of post-processing ]; a rank; processing item }; the execution rule includes: { executing an object, whether to pull black, forwarding an address, and requesting parameter dispersion; limiting the frequency; processing items; [ calculation rule ], pass or not }; the calculation method is as follows:
step 1): the gateway management system filters all rules of the current time period to generate a vector, wherein the vector comprises E(i)、E(u)、E(p)And E(q)In which E(i)For IP matrix [ IP address in all validation objects ]]、E(u)For URL matrix [ URL in all validation objects]、E(p)For determination of all request parameters in the request matrix [ all rules ]]And E(q)Response matrix [ judgment of all response parameters in all rules]；
Step 2): each Agent in the Agent cluster acquires a rule configured by the gateway management system, and the vector value is generated for the rule in each Agent:
step 3): performing vector analysis on the log data stream filtered by the OpenResty gateway cluster by adopting the methods in the step 1) and the step 2), generating matrix streams and transmitting the matrix streams to the multi-Agent cluster, wherein each matrix stream comprises the following vector E(i)’、E(u)’、E(p)', and E(q)’：
E(i)' is IP matrix [ IP addresses in all validation objects]、E(u)' is a URL matrix [ all URLs in validation object]、E(p)' determination of all request parameters in request matrix [ all rules]And E(q)' response matrix [ determination of all response parameters in all rulesBreak-off]，
The matrix flow is:
step 4): multiplying each single-row matrix in the step 3) by the transposed matrix in the step 2), when the value in the single-row matrix in the step 3) is consistent with the value in the vector value in the step 2), the calculated value is 1, when the value in the single-row matrix is inconsistent with the value in the vector value, the calculated value is 0, and converting the single-row matrix stream in the gateway cluster into a result matrix with the item of 1 or 0;
step 5): generating a full matrix which has the same number of terms as the result matrix in the step 4) and each term is 1, multiplying the result matrix obtained in the step 4) by a transposed matrix of the full matrix by each Agent, wherein if the obtained numerical value is less than the number of terms of the matrix, the term in the result matrix is 0, which indicates that the condition is not passed, and if the obtained numerical value is equal to the number of terms of the matrix, the term in the result matrix is 1, which indicates that all the conditions are passed;
step 6): calculating the four matrixes respectively according to the step 5) to obtain four one-dimensional result matrixes, wherein if 0 item exists in the four matrixes, the row in the matrix flow is not detected, and an execution rule of the row is generated, and after the execution rule is obtained by the OpenResty gateway, the API request corresponding to the row in the matrix flow is processed;
s6: and sending the processed API request meeting the screening requirement to an upstream server cluster.
In another embodiment provided by the present invention, the OpenResty gateway cluster includes Lua logic therein.
The Lua logic comprises a log analysis plug-in used for filtering logs generated by API requests.
The Lua logic also comprises a characteristic anti-brush plug-in unit which is used for adjusting the current limiting parameter and the route forwarding rule according to the detection rule.
The number of the OpenResty gateway clusters is at least one.
According to the multi-Agent cluster OpenResty gateway feature-based anti-brush method, a plurality of agents correspond to one API request to perform calculation.
Example 2:
on the basis of embodiment 1, this embodiment is an application example.
The invention relates to an OpenResty-oriented gateway solution, which comprises a gateway cluster, a gateway management system and a multi-Agent cluster, wherein the gateway cluster is an OpenResty cluster and is responsible for forwarding and routing an API (application program interface), and performing feature anti-brush interception on the API which does not accord with a rule according to a gateway execution rule. The gateway management system comprises a gateway rule configuration management module, a gateway machine module and an upstream service management module. The multi-Agent cluster is used for calculating the data flow in the gateway cluster according to the detection rule and generating a gateway execution rule.
The invention modifies the OpenResty gateway to support multi-service deployment. Under the multi-service mode, the method designs a data acquisition plug-in to acquire and filter the logs in OpenResty, and delivers the filtered fields to a multi-Agent cluster for data operation so as to meet the characteristic anti-brush requirement of each service.
The gateway operation architecture in this embodiment is shown in fig. 2. The OpenResty gateway related by the invention is also a cluster and supports multiple services, the gateway performs API routing and distribution according to rules in a gateway database, and service data streams generated in the API distribution process are sent to a multi-Agent computing cluster for computation, so that rules related to some service parties are obtained.
The gateway cluster records logs during operation, and the contents include the source IP and environment of the API, and the URL, request parameters and response parameters of the requested API. And the gateway cluster transmits the log to the Agent cluster after filtering the log. In the prior art, whether the request is rejected or forwarded is obtained by analyzing the log in the gateway cluster, and the complex log causes resource consumption of the gateway cluster and results in slow request, so that the method only simply analyzes a plurality of parameters, does not correlate a plurality of services, and cannot analyze based on historical data. The method and the system use the independent Agent cluster for analysis, and have no influence on the service of the gateway. The gateway only receives the rule of the Agent cluster for judgment, and has no resource consumption.
The detection rule of the gateway is configured through the gateway management system, the gateway cluster filters logs according to the rule, and the Agent cluster calculates according to the detection rule.
The OpenResty framework is a gateway framework combining the Nginx and the Lua, supports the ngnx, and has strong performance, but because the difficulty of customizing the Nginx is higher, OpenResty integrates the Nginx and the Lua, and provides some interfaces for controlling the routing, forwarding and monitoring of the API in the Nginx by adopting the Lua language, thereby reducing the cost of customizing the gateway.
The invention designs a gateway management system and a multi-Agent computing machine for customizing logic for a Lua interface provided by OpenResty, calculates and generates each rule of the gateway in real time, and stores the rule into a gateway database. And acquiring rules in a gateway database from an API (application program interface) provided by OpenResty and a timer provided by OpenResty so as to control the operation of the gateway.
And dynamically generating nginx.conf files in the OpenResty compiling period to realize a multi-service gateway, wherein each service independently manages an API set of the service. The rules of each service come from the same gateway rule base, and the log of each service is also filtered and then output to the multi-Agent cluster for calculation. The overall scheme is shown in figure 3. The names of OpenResty gateway clusters, such as queuing service gateway, C-side service gateway, and reservation service gateway, are only schematically shown in fig. 3, although it is understood that the invention is not limited thereto.
The logs generated during the operation of each service are filtered by the log analysis plug-in the Lua logic in fig. 3 to obtain fields included in the rules, and then the fields are input into the Agent cluster in a vector format for operation and recording.
And the Agent cluster calculates according to the initial rule configured by the gateway system to obtain the rule executed by the gateway. And adjusting the current limiting parameters and the routing forwarding rules by a characteristic anti-brush plug-in the Lua logic in the OpenResty of each service according to the gateway rules.
The gateway has the main functions of protecting and supporting background services, filtering out a large number of invalid requests such as crawler and attack requests, and effectively reducing the consumption of the server.
The gateway rule design and validation process in this embodiment is described with reference to figure 4,
whether the webpage or the mobile APP is used for acquiring data, a Request is sent to a background service, a Request URL is an address of an API, Request parameters are Request Headers, and a Response is Response.
For example, Request Headers parameter, User-Agent of real User is related to Request environment, low-end crawlers are the same or empty, frequency of low-end crawler requests is fixed, and many anti-crawler gateways adopt a method of judging User-Agent to judge, which is also a rule.
The crawler is also developed, the current crawler User-Agent is dynamic, the request frequency is also dynamic, and the common rule can not identify whether the crawler is a crawler. Therefore, the invention designs multi-Agent collaborative analysis, analyzes the parameters and the rules of the cross-API and cross-service IP, and obtains the gateway rules.
The parameters of current limiting, feature anti-brushing and access control of different services are different and need to be calculated independently. The access control of some businesses also needs to rely on the precondition, and needs to calculate a plurality of services at the same time and collaboratively generate the gateway rule. The Agent is a program capable of running independently, and the invention designs a rule of a multi-Agent cluster for rapidly calculating the gateway.
Every several agents correspond to a service, the service calculates what fields in the API get what values, and the dependency relationship between the agents can be configured through the gateway management system. Meanwhile, the gateway configures the characteristics of an API to require the calculation result of which condition is satisfied. For example, the IP requesting the coupon must be an interface that has accessed a member login, and the IP obtaining details of the dish must be an interface that has accessed store information or an interface that has accessed a dish order. The gateway rules and detection flow are shown in fig. 4.
Configuring a calculation rule of the Agent in a gateway background, wherein the calculation rule comprises a { rule number; the effective time; an effective object; numbering the agents; [ judgment of request parameter ]; [ judgment of response parameter ]; [ rule number of pre-dependency ]; [ rule number of post-processing ]; a rank; transaction }.
The rule number is the number of the rule in the gateway and is automatically generated and unique;
the effective time is as the name implies.
The validation object indicates whether the rule is for an IP address, or for a URL, or for a request parameter or for a response parameter.
The Agent is a process which can run independently, and the number is the PID of the process, which indicates that the rule is operated by the PID.
The judgment of the Request parameter is that under what condition a certain parameter of the Request Load of the upper graph is true or false, and the specific data structure is { must/optional, URL, parameter name, condition, true/false }. For example, the true condition configured with the requestId is! &! Then the rule for the request is passed. The request parameter is more than one, which is an array.
The judgment of the response parameter is that if a certain condition is included in the response parameter, it is detected whether the condition is true or false. The specific data structure is { must/optional, parameter name, condition, true/false }.
The rule number of the pre-dependency is that which rule calculation needs to be called after the rule calculation is finished, and the result of the rule is put in front.
The rule number of the post-processing is that which rule calculation needs to be called after the rule calculation is completed. And put the result of the rule behind.
The level, i.e., the risk level, is an integer value. In the present invention, it can be used to define importance and also can be used to express the level of such rules.
The handling items refer to how to handle after the rule is satisfied, and include not handling, black IP pulling, forwarding to a certain interface, and frequency limitation. This is put into the gateway rule, and when a plurality of calculation rules conflict, the highest level is taken as the standard for the last time. For example, it is not conflicting with the black IP pull and forward, but the processing of rule 1 is forwarded to request a and the processing of rule 2 is forwarded to request B, subject to the processing transaction of the last rule with the highest rank.
The execution rule obtained by calculating the rule comprises the following steps: { executing an object, whether to pull black, forwarding an address, and requesting parameter dispersion; limiting the frequency; processing items; [ calculation rule ], pass or not }. Wherein the attributes of the execution object include the type, key, and value of the execution object in the rule; the forwarding address is the original forwarding address of the request; the dispersion of the request is the proportion of the times of request judgment in the parameter hit rule to the total times within the effective time; the limit frequency is how many times only requests are allowed, how many times requests are allowed per minute, how many times requests are allowed per day, which is configured for a single interface in the gateway system. The processing items are processing items obtained by calculating the rules; the calculation rule is the gateway rule through which the calculation rule has been passed.
For example, the menu is limited from being viewed at the order IP address. The rules are set as follows:
{ number 1; 12:00-14: 00; an IP address; agent 1; { must pick, order URL, reqestId! &! "true } ]; [] (ii) a [] (ii) a [] (ii) a 1; }
{ number 2; 12:00-14: 00; an IP address; agent 2; { must choose, order URL, reqestId! &! "true } ]; [] (ii) a [ number 1 ]; [] (ii) a 1; forward to the order URL }
When the IP address 10.0.231.1 requests a dish-ordering URL, the rule numbered 1 is triggered if reqestId! &! "means that the rule passes, and the rule is not processed if no processing item is disposed in the rule. If the URL for ordering is requested, the rule is not passed, and a gateway rule { { IP address, IP:10.0.231.1}, No, 1 is generated; (ii) a (ii) a [ number 1], fail }. The gateway will reject this request for this IP.
When the IP address 10.0.231.2 requests the order-placing URL, the rule numbered 1 is triggered, and the rule is judged to be false, so that the rule does not pass, and the processing item is forwarded to the order-placing URL. If reqestId! &! If yes, triggering the post-dependent rule number 1, if no order is placed, the number 1 does not pass, the rule still does not hold, and the processing item is to forward to the order placing URL. If the order has been placed, the number 1 passes, the present rule is established, and no gateway rule is generated.
When the calculation rule is hit, the Agent stores the calculation result, wherein the object is { the unique identification of the current judgment, the rule object, the identification of the current judgment and the judgment result } in a Redis database of the Agent cluster, and notifies other agents configured in the previous and next rules. The identifier of the current determination is a value after the determination of the "validation object" in the rule, if the object is an IP address, the IP address is the IP address of the current request, for example, 10.0.231.1 described above, and if the object is a value of a parameter, for example, a value of a user name, the parameter is a specific user name.
And after the related Agent is informed, the calculation results of the preposed rule and the postpositional rule of the identification of the Agent for the current judgment of the rule are taken out from the Redis, if the calculation results cannot be taken out, the fact that the Agent does not calculate the rule of the current judgment and configures the rule for the Agent is shown, and the Agent starts to calculate. If the effective object can be taken out, whether all the hit rules of the effective object at the stage are established or not is further judged, if not, the processing items are obtained, and the gateway rules are generated. The gateway rules are stored in a database of gateway rules.
For example, the case IP address 10.0.231.2 is used to send the IP address 10.0.231.2 to Agent1 after Agent2 is executed, and if 10.0.231.2 does not order a dish, Agent1 does not perform 10.0.231.2 determination, and cannot take out the content, and starts to trigger the execution of rule 1. If 10.0.231.2 has ordered dishes, when Agent2 executes, Agent1 has executed judgment, directly takes out the dishes to judge the result, generates a gateway rule, calls an interface of the gateway cluster, inserts the gateway cluster into the Mysql database and Redis of the gateway rule respectively, and cleans the Lua memory.
The access design mode can avoid redundant operation and improve the efficiency of the rule calculation of the multi-service gateway. And the characteristic anti-brush plug-in the gateway cluster can be taken from Redis if the rule can not be obtained because the Lua memory is cleaned, and can be taken from the Mysql database if the rule can not be obtained from the Redis. The method needs a certain cold start time, but has no consumption on the gateway cluster and has high interception accuracy.
The embodiment also provides a multi-Agent cluster:
the Agent can continuously calculate data input by the gateway, analyze the request parameters and the response parameters of each interface of each IP, obtain the distribution of the request parameters and the distribution of the result of the IP and analyze the distribution, thereby obtaining the information quantity and the income value of the IP. For IP that gets information for a long period without revenue, suspected to be a crawler, access speed limits and degradations are given. For example, a store may be taken for a long time without viewing the dishes, a dish may be requested for a long time without ordering and paying, or an IP address may be viewed asynchronously with the store, perhaps by a competitor viewing the information. Giving access control and reducing the return or prompting of the data volume.
In connection with fig. 5, the Agent cluster work logic is as follows:
the parameters and responses of the interface are one-dimensional vectors, the IP accesses the interface continuously, a matrix formed by the one-dimensional vectors is obtained, the density of the matrix (the number of non-0 values in each m x n matrix, wherein m represents the number of columns, namely the number of calculation fields in the parameters or request results, and n represents the number of rows and is configured by a gateway system) is calculated, the distribution of the request parameters and the result of the IP is obtained, certain columns of the matrix are subjected to characteristic calculation, and the profit value of the IP address is obtained according to the existence or nonexistence of the columns.
When multiple agents rely on the matrix, matrix data are interacted between the agents, and the dependency relationship is calculated in a mode of calculating a reachable matrix. For example, when all request matrices in the request of Agent1 (login) are 0, the request is multiplied by the matrix of Agent2 (coupon), even if the matrix of Agent2 has a value, the result will be 0, and thus the IP of the coupon is directly checked if no login is found.
The specific operation flow is as follows:
step1 the gateway management system filters out all rules for the current time period based on the validation time to generate a vector. The vector includes four types, an IP matrix [ IP addresses in all validation objects ], a URL matrix [ URLs in all validation objects ], a request matrix [ judgment of all request parameters in all rules ], and a response matrix [ judgment of all response parameters in all rules ]. The four methods respectively correspond to the following four methods:
E(i)、E(u)、E(p)and E(q)。
step 2A single Agent gets the rules assigned to it, converting all rules into the values of the four vectors described above. For an IP address, the value of the entry is the address of the IP address, and for a URL, the value of the entry is the URL itself. For request and response parameters, the value of the entry is the configured value of the parameter.
step3, the gateway cluster analyzes the corresponding data flow according to the four vectors to generate a matrix flow and transmits the matrix flow to the multi-Agent cluster; the resolution rule is shown as step 2.
Each of the matrix streams includes the following vector E(i)’、E(u)’、E(p)', and E(q)’；
The matrix flow is:
step4: Agent "multiplies" a single row of matrix in the matrix stream obtained in step3 by the transposed matrix of the matrix in step2 to obtain a new matrix stream, each entry in this matrix stream being a detected value. The multiplication in this step is a matrix multiplication step, but since the value of the matrix entry is not an integer or floating point type, direct calculation cannot be performed, the matrix entry is an executable rule, the value in the matrix stream is replaced in the executable rule, and if the condition is satisfied, the multiplied value is 1, and if not, the value is 0. This step converts the matrix flow in the gateway cluster into a matrix with entries of 1 or 0.
step5, the Agent multiplies the result matrix of the single-row matrix stream obtained in step4 by the transpose matrix of the all-one matrix with the same number of terms, if the obtained numerical value is less than the number of terms of the matrix, the term in the result matrix is 0, which indicates that the condition fails, and if the obtained numerical value is equal to the number of terms of the matrix, the term in the result matrix is 1, which indicates that all the conditions pass.
step6, calculating the four matrixes according to the rule of step5 respectively to obtain four one-dimensional result matrixes, wherein if 0 item exists in the four matrixes, the four matrixes indicate that the row in the matrix stream fails to be detected. An execution rule for the row is generated. And after the rule is acquired by the gateway, processing the API request corresponding to the row in the matrix flow.
The matrix adopted in the steps can quickly carry out parallel processing on large-scale data, and the parallel processing is faster than direct judgment.
The matrix stream obtained in step5 includes the detailed characteristics of the request, the number of distribution rows of 1 entries in the matrix is counted to obtain the distribution of the request parameters and the distribution of the result of the whole system, the matrix stream of a certain IP address is filtered out, the distribution of the request parameters and the distribution of the result of the IP address can be obtained, characteristic calculation is performed on a certain column, for example, a certain entry of a response matrix, and the profit value of the IP address is obtained according to the existence or nonexistence of the number of 1 entries of the column.
The number of rows of the matrix stream of the filtered IP address per unit time obtains the request frequency of the IP address. The Agent cluster defines the API with fixed access parameters, excessive request frequency and data page number and no income for a long time as the illegal access of the crawler, and directly blackens the IP address. Avoiding the impact on the subsequent service.
According to the embodiment, the method for preventing the characteristic of the OpenResty gateway based on the multi-Agent cluster at least has the following beneficial effects that:
most gateways in the prior art only consider current limiting, feature anti-brushing, and firewall for independent services. The current limit is not based on the CPU and memory dynamic generation parameters of the upstream service, the characteristic of the anti-brush is simply to match the request parameters, remove the request with obviously abnormal parameters, and is ineffective for the customized crawler, and the firewall is simply a black and white list. The invention designs a multi-Agent cluster, analyzes the OpenResty log in real time, supports multi-service rule calculation, quantifies the profit value to perform anti-brushing and access control through the request parameter and the returned distribution of the IP address, is more accurate, and simultaneously prevents misjudgment.
Although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustrative purposes only and are not intended to limit the scope of the present invention. It will be appreciated by those skilled in the art that modifications may be made to the above embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.
Claims (6)
1. A multi-Agent cluster OpenResty gateway feature-based anti-brush method is characterized by comprising the following steps:
sending an API request to the OpenResty gateway cluster;
the gateway management system configures a detection rule of a gateway;
after receiving the API request, the OpenResty gateway cluster records a log generated by the API request and filters the log according to the detection rule, wherein the log comprises a source IP and an environment of the API request, a URL (uniform resource locator) of the API request, a request parameter and a response parameter;
the OpenResty gateway cluster transmits the filtered log to an Agent cluster;
the Agent cluster calculates the filtered log data stream according to the detection rule to obtain an execution rule, wherein the detection rule comprises the following steps: { rule number; the effective time; an effective object; numbering the agents; [ judgment of request parameter ]; [ judgment of response parameter ]; [ rule number of pre-dependency ]; [ rule number of post-processing ]; a rank; processing item }; the execution rule includes: { executing an object, whether to pull black, forwarding an address, and requesting parameter dispersion; limiting the frequency; processing items; [ calculation rule ], pass or not }; the calculation method is as follows:
step 1): the gateway management system filters all rules of the current time period to generate a vector, wherein the vector comprises E(i)、E(u)、E(p)And E(q)In which E(i)For IP matrix [ IP address in all validation objects ]]、E(u)For URL matrix [ URL in all validation objects]、E(p)For determination of all request parameters in the request matrix [ all rules ]]And E(q)Response matrix [ judgment of all response parameters in all rules]；
Step 2): each Agent in the Agent cluster acquires a rule configured by the gateway management system, and the vector value is generated for the rule in each Agent:
step 3): performing vector analysis on the log data stream filtered by the OpenResty gateway cluster by adopting the methods in the step 1) and the step 2), generating matrix streams and transmitting the matrix streams to the multi-Agent cluster, wherein each matrix stream comprises the following vector E(i)’、E(u)’、E(p)', and E(q)’：
E(i)' is IP matrix [ IP addresses in all validation objects]、E(u)' is a URL matrix [ all URLs in validation object]、E(p)' determination of all request parameters in request matrix [ all rules]And E(q)' response matrix [ judgment of all response parameters in all rules]，
The matrix flow is:
step 4): multiplying each single-row matrix in the step 3) by the transposed matrix in the step 2), when the value in the single-row matrix in the step 3) is consistent with the value in the vector value in the step 2), the calculated value is 1, when the value in the single-row matrix is inconsistent with the value in the vector value, the calculated value is 0, and converting the single-row matrix stream in the gateway cluster into a result matrix with the item of 1 or 0;
step 5): generating a full matrix which has the same number of terms as the result matrix in the step 4) and each term is 1, multiplying the result matrix obtained in the step 4) by a transposed matrix of the full matrix by each Agent, wherein if the obtained numerical value is less than the number of terms of the matrix, the term in the result matrix is 0, which indicates that the condition is not passed, and if the obtained numerical value is equal to the number of terms of the matrix, the term in the result matrix is 1, which indicates that all the conditions are passed;
step 6): calculating the four matrixes respectively according to the step 5) to obtain four one-dimensional result matrixes, wherein if 0 item exists in the four matrixes, the row in the matrix flow is not detected, and an execution rule of the row is generated, and after the execution rule is obtained by the OpenResty gateway, the API request corresponding to the row in the matrix flow is processed;
and sending the processed API request meeting the screening requirement to an upstream server cluster.
2. The multi-Agent cluster-based OpenResty gateway feature scrubbing prevention method of claim 1, wherein the OpenResty gateway cluster includes Lua logic.
3. The multi-Agent cluster OpenResty gateway feature-based anti-brush method of claim 2, wherein the Lua logic includes a log parsing plug-in for filtering logs generated by API requests.
4. The multi-Agent cluster OpenResty gateway feature-based anti-brush method of claim 2, wherein the Lua logic further comprises a feature anti-brush plug-in for adjusting a current-limiting parameter and a routing forwarding rule according to the detection rule.
5. The multi-Agent cluster-based OpenResty gateway feature scrubbing prevention method of claim 1, wherein the number of OpenResty gateway clusters is at least one.
6. The multi-Agent cluster OpenResty gateway feature-based anti-brush method of claim 1, wherein multiple agents compute corresponding to one API request.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911236035.4A CN110944000B (en) | 2019-12-05 | 2019-12-05 | OpenResty gateway feature anti-brushing method based on multi-Agent cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911236035.4A CN110944000B (en) | 2019-12-05 | 2019-12-05 | OpenResty gateway feature anti-brushing method based on multi-Agent cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110944000A true CN110944000A (en) | 2020-03-31 |
| CN110944000B CN110944000B (en) | 2021-09-28 |
Family
ID=69909669
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911236035.4A Active CN110944000B (en) | 2019-12-05 | 2019-12-05 | OpenResty gateway feature anti-brushing method based on multi-Agent cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110944000B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112202598A (en) * | 2020-09-10 | 2021-01-08 | 青岛海信网络科技股份有限公司 | Log recording method and device |
| CN114389900A (en) * | 2022-03-23 | 2022-04-22 | 广东睿江云计算股份有限公司 | OpenResty-based abnormal traffic capturing and intercepting method and system |
| CN114866457A (en) * | 2022-04-27 | 2022-08-05 | 猪八戒股份有限公司 | High-performance dynamic route forwarding method, system and equipment based on Nginx and Lua |
| CN115296959A (en) * | 2022-07-25 | 2022-11-04 | 紫光云技术有限公司 | Method for replacing SpringCloudGateway gateway by using Nginx + Lua script |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140366118A1 (en) * | 2013-06-05 | 2014-12-11 | Fortinet, Inc. | Cloud based logging service |
| CN108322502A (en) * | 2017-12-22 | 2018-07-24 | 杭州大搜车汽车服务有限公司 | Method, gateway system and storage medium for equalization server load |
| CN109672612A (en) * | 2018-12-13 | 2019-04-23 | 中国电子科技集团公司电子科学研究院 | API gateway system |
-
2019
- 2019-12-05 CN CN201911236035.4A patent/CN110944000B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140366118A1 (en) * | 2013-06-05 | 2014-12-11 | Fortinet, Inc. | Cloud based logging service |
| CN108322502A (en) * | 2017-12-22 | 2018-07-24 | 杭州大搜车汽车服务有限公司 | Method, gateway system and storage medium for equalization server load |
| CN109672612A (en) * | 2018-12-13 | 2019-04-23 | 中国电子科技集团公司电子科学研究院 | API gateway system |
Non-Patent Citations (1)
| Title |
|---|
| 熊智等: "Web集群中文档组织分布的优化策略", 《计算机科学》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112202598A (en) * | 2020-09-10 | 2021-01-08 | 青岛海信网络科技股份有限公司 | Log recording method and device |
| CN114389900A (en) * | 2022-03-23 | 2022-04-22 | 广东睿江云计算股份有限公司 | OpenResty-based abnormal traffic capturing and intercepting method and system |
| CN114866457A (en) * | 2022-04-27 | 2022-08-05 | 猪八戒股份有限公司 | High-performance dynamic route forwarding method, system and equipment based on Nginx and Lua |
| CN114866457B (en) * | 2022-04-27 | 2024-01-16 | 猪八戒股份有限公司 | High-performance dynamic route forwarding method, system and equipment based on Nginx and Lua |
| CN115296959A (en) * | 2022-07-25 | 2022-11-04 | 紫光云技术有限公司 | Method for replacing SpringCloudGateway gateway by using Nginx + Lua script |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110944000B (en) | 2021-09-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110944000B (en) | OpenResty gateway feature anti-brushing method based on multi-Agent cluster | |
| CN103902646B (en) | Distributed task managing system and method | |
| CN106682097B (en) | Method and device for processing log data | |
| US10997192B2 (en) | Data source correlation user interface | |
| US8335838B2 (en) | Web page load time prediction and simulation | |
| US8402131B2 (en) | Hierarchy for characterizing interactions with an application | |
| US8656006B2 (en) | Integrating traffic monitoring data and application runtime data | |
| CN105283849B (en) | For the Parallel Tracking of performance and details | |
| US9288124B1 (en) | Systems and methods of classifying sessions | |
| US8751184B2 (en) | Transaction based workload modeling for effective performance test strategies | |
| US9740991B2 (en) | Calculating in-flight metrics for non-interruptible business transactions | |
| US20140173744A1 (en) | System and methods for scalably identifying and characterizing structural differences between document object models | |
| CN107463641A (en) | System and method for improving the access to search result | |
| US20210385251A1 (en) | System and methods for integrating datasets and automating transformation workflows using a distributed computational graph | |
| JP2007073024A (en) | Macro information generation system, macro information generation device, macro information generation method and macro information generation program | |
| US20180341681A1 (en) | Search results based on a search history | |
| US20210136121A1 (en) | System and method for creation and implementation of data processing workflows using a distributed computational graph | |
| CN110390584A (en) | A kind of recognition methods of abnormal user, identification device and readable storage medium storing program for executing | |
| US11663172B2 (en) | Cascading payload replication | |
| CN100566260C (en) | A kind of method for monitoring network service quality and system thereof | |
| US11714683B1 (en) | Information technology and security application automation architecture | |
| Bocciarelli et al. | Automated performance analysis of business processes | |
| CN112383513A (en) | Crawler behavior detection method and device based on proxy IP address pool and storage medium | |
| CN107798085B (en) | Service processing method and device based on plug-in | |
| US20220247784A1 (en) | Extension framework for an information technology and security operations application |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231219 Address after: 200331, Room 515, No. 788 Zhenbei Road, Putuo District, Shanghai Patentee after: Meizhiwei (Shanghai) Information Technology Co.,Ltd. Address before: 201207 Room 302, building 88, Lane 887, Zuchongzhi Road, pilot Free Trade Zone, Pudong New Area, Shanghai Patentee before: DELICIOUS NOWAIT (SHANGHAI) INFORMATION TECHNOLOGY CO.,LTD. |
|
| TR01 | Transfer of patent right |