CN111162930A - Delay response control method - Google Patents
Delay response control method Download PDFInfo
- Publication number
- CN111162930A CN111162930A CN201911248863.XA CN201911248863A CN111162930A CN 111162930 A CN111162930 A CN 111162930A CN 201911248863 A CN201911248863 A CN 201911248863A CN 111162930 A CN111162930 A CN 111162930A
- Authority
- CN
- China
- Prior art keywords
- response
- time
- request
- crawling
- url
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0823—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0604—Management of faults, events, alarms or notifications using filtering, e.g. reduction of information by using priority, element types, position or time
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
- Debugging And Monitoring (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a delay response control method, which comprises the following steps: configuring an application service IP and a port; crawling a module and a url in the application service by a crawler according to the IP and the port configured in the step 1; crawling a module and url in the application service according to the step 2 to obtain a crawler result; filtering the crawling result obtained in the step 3, removing the crawler result with the access type as the request, and obtaining a response request; generating a response time threshold T of the response request according to the response request and the crawling time acquired in the step 4: setting a response rule according to the response time threshold T obtained in the step 5 to obtain response floating time; and 6, obtaining page indexes according to the response rule and the response floating time obtained in the step 6: and performing response control according to the response time threshold T, the response floating time and the page index. The invention configures reasonable delay response time for abnormal or failed requests without influencing user experience.
Description
Technical Field
The present invention relates to a network technology, and in particular, to a delay response control method.
Background
The Internet brings convenience to life of people, and meanwhile, some application services bringing convenience to people are particularly deployed on the network, or the application cannot normally provide services for users due to some special requirements or due to continuous requests of some attackers. For example, many crawlers on the network continuously grab data or abnormally access the url of the website, so that network congestion is caused and normal use is influenced. Such as data theft, especially requests for large downloads of large files; such as brute force, large requests to log on. These abnormal uses are increasingly sophisticated and difficult to identify.
The prior art has the following problems:
1. at present, the alarm is carried out after the detection by adopting monitoring and detecting means, which are usually methods of post-discovery and post-processing.
The method has the following defects: abnormal use is increasingly sophisticated and difficult to identify, and can still cause gambling, rendering applications incapable of providing normal services.
2. And blocking after the strategy firewall is successfully matched.
The method has the following defects: misjudgment may be caused, and the user experience is affected.
Accordingly, there is a need for improvements in the art.
Disclosure of Invention
The invention aims to provide an efficient delay response control method.
In order to solve the above technical problem, the present invention provides a delay response control method, including the following steps:
1) configuring an application service IP and a port;
2) crawling a module and a url in the application service by a crawler according to the IP and the port configured in the step 1;
3) crawling a module and url in the application service according to the step 2 to obtain a crawler result;
4) filtering the crawling result obtained in the step 3, removing the crawler result with the access type as the request, and acquiring a response request;
5) and generating a response time threshold T of the response type request according to the response type request and the crawling time acquired in the step 4:
6) setting a response rule according to the response time threshold T obtained in the step 5 to obtain response floating time;
7) and according to the response rule and the response floating time obtained in the step 6, obtaining page indexes:
8) and performing response control according to the response time threshold T, the response floating time and the page index.
As an improvement of the delay response control method of the present invention:
in step 3, the crawler results include, but are not limited to, url, access type, crawl time, details; the access type includes, but is not limited to, request/response.
As a further improvement to the delay response control method of the present invention:
the step 5 comprises the following steps:
5.1) obtaining a distribution map by using the response request as an abscissa and the crawling time as an ordinate through the response request and the crawling time obtained in the step 4; by observing a distribution map, taking the maximum value in the crawling time in the response request obtained in the step 4 as the maximum response time t, and obtaining the maximum response time t;
5.2) repeatedly executing the steps 2-4 to obtain n maximum response times t;
5.3) averaging the n maximum response times T to obtain a response time threshold T.
As a further improvement to the delay response control method of the present invention:
the step 6 comprises the following steps:
6.1), configuring URL rules: setting a matching rule for url of the response request obtained in the step 4;
6.2), configuration response time: taking a response time threshold T as a request response time reference;
6.3), configuration response float time: a corresponding response float time is set.
As a further improvement to the delay response control method of the present invention:
in step 7, a page index is obtained according to the URL rule and the response floating time configured in the step 6.1; the page indicators include, but are not limited to, the sensitivity of the data contained in the response page, and the amount of data contained in the response page.
As a further improvement to the delay response control method of the present invention:
the URL rule is: the response float time is proportional or inversely proportional to the page index.
The delay response control method has the technical advantages that:
the invention configures reasonable delay response time for abnormal or failed requests without influencing user experience. By reasonably configuring and controlling response time, the whole network flow is reduced, and abnormal flow is prevented from blocking services.
Detailed Description
The invention will be further described with reference to specific examples, but the scope of the invention is not limited thereto.
Embodiment 1, a delay response control method, specifically including the steps of:
1. and configuring an application service IP and a port.
The application service refers to a web application providing service for the outside, and is needed by the invention to be deployed.
2. And (4) crawling the module and url in the application service through a crawler according to the IP and the port configured in the step (1).
Crawlers include, but are not limited to, open source crawler tools, and the present invention employs an autonomously developed web scanner.
3. And (3) obtaining a crawler result after the step 2, wherein the crawler result comprises, but is not limited to url, access type, crawling time and details.
The access type includes, but is not limited to, request/response.
4. And filtering the response request: and (4) filtering the crawling result obtained in the step (3), removing the crawler result with the access type as the request, and acquiring the response request.
5. Generating a response time threshold T:
5.1) obtaining a distribution map by taking the response request as an abscissa and the crawling time as an ordinate through the response request and the crawling time obtained in the step 4. By observing a distribution map, taking the maximum value in the crawling time in the response request obtained in the step 4 as the maximum response time t, and obtaining the maximum response time t;
5.2) through crawling for n times (repeatedly executing the steps 2-4), n maximum response times t can be obtained;
5.3) averaging the n maximum response times T to obtain a response time threshold T;
6. and the protective equipment of the linkage application service sets a response rule.
6.1), configuring URL rules: i.e. the url of the response class request obtained in step 4, a matching rule is set (the ratio between the response float time and the page index is in direct proportion or in inverse proportion).
6.2), configuration response time: and taking the response time threshold T as a request response time reference.
6.3), configuration response float time: and responding to the configuration rule of the URL and setting corresponding response floating time.
7. Delay response control:
7.1, according to the URL rule configured in the step 6.1, the response floating time is in direct proportion or inverse proportion to the page index.
7.2, the page index includes but is not limited to the sensitivity of the data contained in the response page and the data amount of the data contained in the response page.
8. And performing response control according to the response time threshold T, the response floating time and the page index.
The rules, indicators, and response float times can be customized empirically.
Finally, it is also noted that the above-mentioned lists merely illustrate a few specific embodiments of the invention. It is obvious that the invention is not limited to the above embodiments, but that many variations are possible. All modifications which can be derived or suggested by a person skilled in the art from the disclosure of the present invention are to be considered within the scope of the invention.
Claims (6)
1. A delayed response control method, characterized by: the method comprises the following steps:
1) configuring an application service IP and a port;
2) crawling a module and a url in the application service by a crawler according to the IP and the port configured in the step 1;
3) crawling a module and url in the application service according to the step 2 to obtain a crawler result;
4) filtering the crawling result obtained in the step 3, removing the crawler result with the access type as the request, and acquiring a response request;
5) and generating a response time threshold T of the response type request according to the response type request and the crawling time acquired in the step 4:
6) setting a response rule according to the response time threshold T obtained in the step 5 to obtain response floating time;
7) and according to the response rule and the response floating time obtained in the step 6, obtaining page indexes:
8) and performing response control according to the response time threshold T, the response floating time and the page index.
2. The delay response control method according to claim 1, characterized in that:
in step 3, the crawler results include, but are not limited to, url, access type, crawl time, details; the access type includes, but is not limited to, request/response.
3. The delay response control method according to claim 2, characterized in that:
the step 5 comprises the following steps:
5.1) obtaining a distribution map by using the response request as an abscissa and the crawling time as an ordinate through the response request and the crawling time obtained in the step 4; by observing a distribution map, taking the maximum value in the crawling time in the response request obtained in the step 4 as the maximum response time t, and obtaining the maximum response time t;
5.2) repeatedly executing the steps 2-4 to obtain n maximum response times t;
5.3) averaging the n maximum response times T to obtain a response time threshold T.
4. The delay response control method of claim 3, wherein:
the step 6 comprises the following steps:
6.1), configuring URL rules: setting a matching rule for url of the response request obtained in the step 4;
6.2), configuration response time: taking a response time threshold T as a request response time reference;
6.3), configuration response float time: a corresponding response float time is set.
5. The delay response control method of claim 4, wherein:
in step 7, a page index is obtained according to the URL rule and the response floating time configured in the step 6.1; the page indicators include, but are not limited to, the sensitivity of the data contained in the response page, and the amount of data contained in the response page.
6. The delay response control method of claim 5, wherein:
the URL rule is: the response float time is proportional or inversely proportional to the page index.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911248863.XA CN111162930B (en) | 2019-12-09 | 2019-12-09 | Delay response control method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911248863.XA CN111162930B (en) | 2019-12-09 | 2019-12-09 | Delay response control method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111162930A true CN111162930A (en) | 2020-05-15 |
| CN111162930B CN111162930B (en) | 2022-11-11 |
Family
ID=70555784
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911248863.XA Active CN111162930B (en) | 2019-12-09 | 2019-12-09 | Delay response control method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111162930B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113746790A (en) * | 2020-07-22 | 2021-12-03 | 北京沃东天骏信息技术有限公司 | Abnormal flow management method, electronic device and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160127262A1 (en) * | 2014-10-31 | 2016-05-05 | The Nielsen Company (Us), Llc | Method and apparatus to throttle media access by web crawlers |
| CN105610728A (en) * | 2015-12-28 | 2016-05-25 | 湖南蚁坊软件有限公司 | Web crawler flow control automatic degradation method based on time window |
| CN106331108A (en) * | 2016-08-25 | 2017-01-11 | 北京量科邦信息技术有限公司 | Crawler realization method and system capable of breaking through IP limit |
-
2019
- 2019-12-09 CN CN201911248863.XA patent/CN111162930B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160127262A1 (en) * | 2014-10-31 | 2016-05-05 | The Nielsen Company (Us), Llc | Method and apparatus to throttle media access by web crawlers |
| CN105610728A (en) * | 2015-12-28 | 2016-05-25 | 湖南蚁坊软件有限公司 | Web crawler flow control automatic degradation method based on time window |
| CN106331108A (en) * | 2016-08-25 | 2017-01-11 | 北京量科邦信息技术有限公司 | Crawler realization method and system capable of breaking through IP limit |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113746790A (en) * | 2020-07-22 | 2021-12-03 | 北京沃东天骏信息技术有限公司 | Abnormal flow management method, electronic device and storage medium |
| CN113746790B (en) * | 2020-07-22 | 2023-09-05 | 北京沃东天骏信息技术有限公司 | Abnormal flow management method, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111162930B (en) | 2022-11-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10637888B2 (en) | Automated lifecycle system operations for threat mitigation | |
| US10021127B2 (en) | Threat indicator analytics system | |
| US9426125B2 (en) | Real-time reconfigurable web application firewall for a distributed platform | |
| US9807120B2 (en) | Method and system for automated incident response | |
| US11729193B2 (en) | Intrusion detection system enrichment based on system lifecycle | |
| EP2955894B1 (en) | Deception network system | |
| US8302198B2 (en) | System and method for enabling remote registry service security audits | |
| US9781133B2 (en) | Automatic stability determination and deployment of discrete parts of a profile representing normal behavior to provide fast protection of web applications | |
| US7996896B2 (en) | System for regulating host security configuration | |
| US8505092B2 (en) | Dynamic provisioning of protection software in a host intrusion prevention system | |
| US20200128038A1 (en) | Network security system with enhanced traffic analysis based on feedback loop and low-risk domain identification | |
| CN104065657A (en) | Method for dynamically controlling user behavior based on IP access and system thereof | |
| CN112534432A (en) | Real-time mitigation of unfamiliar threat scenarios | |
| US20190042736A1 (en) | Iintrusion detection system enrichment based on system lifecycle | |
| KR101282297B1 (en) | The apparatus and method of unity security with transaction pattern analysis and monitoring in network | |
| KR101951730B1 (en) | Total security system in advanced persistent threat | |
| US20170318037A1 (en) | Distributed anomaly management | |
| US20230119649A1 (en) | Intrusion detection and prevention system rule automation and optimization | |
| CN106506559A (en) | Access Behavior- Based control method and device | |
| DE112021004808T5 (en) | DETECTING MALWARE THROUGH ANALYSIS OF DISTRIBUTED TELEMETRY DATA | |
| CN111162930B (en) | Delay response control method | |
| CN107911232A (en) | A kind of method and device of definite business operation rule | |
| RU2481633C2 (en) | System and method for automatic investigation of safety incidents | |
| CN109981573B (en) | Security event response method and device | |
| CN105763365B (en) | Exception handling method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |