CN112291225A - Big data abnormal flow detection method and system applied to integral system - Google Patents
Big data abnormal flow detection method and system applied to integral system Download PDFInfo
- Publication number
- CN112291225A CN112291225A CN202011148805.2A CN202011148805A CN112291225A CN 112291225 A CN112291225 A CN 112291225A CN 202011148805 A CN202011148805 A CN 202011148805A CN 112291225 A CN112291225 A CN 112291225A
- Authority
- CN
- China
- Prior art keywords
- flow
- abnormal
- log
- module
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000002159 abnormal effect Effects 0.000 title claims abstract description 55
- 238000001514 detection method Methods 0.000 title claims abstract description 35
- 238000004458 analytical method Methods 0.000 claims abstract description 20
- 238000007405 data analysis Methods 0.000 claims abstract description 14
- 238000012544 monitoring process Methods 0.000 claims abstract description 13
- 238000000034 method Methods 0.000 claims abstract description 12
- 238000007726 management method Methods 0.000 claims abstract description 8
- 238000010223 real-time analysis Methods 0.000 claims abstract description 3
- 238000005516 engineering process Methods 0.000 claims description 8
- 238000004140 cleaning Methods 0.000 claims description 6
- 238000011897 real-time detection Methods 0.000 claims description 5
- 238000004220 aggregation Methods 0.000 claims description 2
- 230000002776 aggregation Effects 0.000 claims description 2
- 230000004907 flux Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000002265 prevention Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/145—Network analysis or design involving simulating, designing, planning or modelling of a network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
Abstract
The method is oriented to the integral service system, obtains service access flow information (such as access flow logs) of each service system in real time, performs real-time analysis processing through big data, performs behavior classification, analysis and other processing on the flow logs, then performs matching by combining an abnormal flow detection model, and performs judgment, identification and alarm on abnormal flow. The system comprises a flow log collection module, a flow big data analysis processing module, an abnormal flow detection module, an abnormal flow monitoring management module and a main service processing flow.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a big data abnormal flow detection method and system applied to an integral system.
Background
With the continuous development of internet technology and services, network security problems occur frequently, which cause more and more service failures and economic losses, and network security has become a more and more concern of people. In the scoring system, because there are many business systems of various categories, the corresponding security level and the requirements of security countermeasure are different, and the conventional firewall type conformance protocol, the conventional precautionary measures such as ports are not enough to cope with more and more new hacker technologies. Especially, in the abnormal access mixed with the normal service flow, the detection technology needs to be improved, and deep real-time detection, identification and prevention are performed on the basis of the global service flow. With the rise of big data technologies and applications, real-time processing capability for a large amount of streaming services has been provided. Therefore, in the integral business system, a big data abnormal flow detection method and a big data abnormal flow detection system applied to the integral system can be invented, and the abnormal flow is detected and identified in real time in the business application process by utilizing a big data analysis processing technology, so that the network safety problem and the risk are reduced to the minimum.
Disclosure of Invention
The invention provides a big data abnormal flow detection method and system applied to an integral system. The method is oriented to the integral service system, obtains service access flow information (such as access flow logs) of each service system in real time, performs real-time analysis processing through big data, performs behavior classification, analysis and other processing on the flow logs, then performs matching by combining an abnormal flow detection model, and performs judgment, identification and alarm on abnormal flow. The system comprises a flow log collection module, a flow big data analysis processing module, an abnormal flow detection module, an abnormal flow monitoring management module and a main service processing flow.
1. A flow log collection module: and the system is oriented to the integral service system and is used for collecting and storing global flow log information in real time. The flow log information comes from the front and back end buried points of the service system and comprises service codes, client access IP, client equipment type (mobile end or PC end), browser type, access time, request packet size, return data packet size and the like. The log aggregation processing can be carried out by a common log collection tool, namely, the common log collection tool, wherein the common log collection tool is used for deploying the flux agent at each service node and sending log data to the flow log collection module
2. The flow big data analysis and processing module: and analyzing and processing the collected and stored flow logs by using a big data platform and technology, wherein the analysis comprises data cleaning, access log classification/behavior classification, behavior trend analysis and the like.
3. An abnormal flow detection module: and based on the configured abnormal flow detection model, carrying out real-time detection on flow log analysis data, judging and identifying abnormal access flow, and then carrying out processing such as alarming according to business rules.
4. Abnormal flow monitoring and managing module: the method comprises the steps of configuring parameters and storage of a flow log collection module, configuring classification and behavior definition of a flow big data analysis processing module, and setting an abnormal flow model of a legacy flow detection module.
5. The main business processing flow is as follows: firstly, in an integral service system, according to the monitoring requirement of abnormal service flow, flow logs are buried, and a detection model and other modules are set through an abnormal flow monitoring management module; in the service operation process, the access log data after point burying are continuously converged to a flow log collection module in real time, and the converged log data are stored by the flow log collection module; the flow big data analysis processing module is used for analyzing and processing the stored access log data, and the analysis processing comprises data cleaning, access log classification/behavior classification, behavior trend analysis and the like; and the abnormal flow detection module is used for detecting the flow log analysis data in real time based on the configured abnormal flow detection model, judging and identifying abnormal access flow, and then performing processing such as alarming according to business rules.
Drawings
Fig. 1 is a diagram of a big data abnormal flow detection method and system structure applied to an integral system.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, in an embodiment of the present invention, a big data abnormal traffic detection method and a system structure diagram applied to a scoring system include a traffic log collection module (1), a traffic big data analysis processing module (2), an abnormal traffic detection module (3), an abnormal traffic monitoring management module (4), and a main service processing flow.
1. Flow log collection module (1): and the system is oriented to the integral service system and is used for collecting and storing global flow log information in real time. The flow log information comes from the front and back end buried points of the service system and comprises service codes, client access IP, client equipment type (mobile end or PC end), browser type, access time, request packet size, return data packet size and the like.
2. The flow big data analysis and processing module (2): and analyzing and processing the collected and stored flow logs by using a big data platform and technology, wherein the analysis comprises data cleaning, access log classification/behavior classification, behavior trend analysis and the like.
3. Abnormal flow detection module (3): and based on the configured abnormal flow detection model, carrying out real-time detection on flow log analysis data, judging and identifying abnormal access flow, and then carrying out processing such as alarming according to business rules.
4. An abnormal flow monitoring management module (4): the method comprises the steps of configuring parameters and storage of a flow log collection module, configuring classification and behavior definition of a flow big data analysis processing module, and setting an abnormal flow model of a legacy flow detection module.
5. The main business processing flow is as follows: firstly, in an integral service system, according to the monitoring requirement of abnormal service flow, flow logs are buried, and a detection model and other modules are set through an abnormal flow monitoring management module (4); in the service operation process, the access log data after point burying are continuously converged to a flow log collection module (1) in real time, and the converged log data are stored by the flow log collection module (1); the flow big data analysis processing module (2) analyzes and processes the stored access log data, including data cleaning, access log classification/behavior classification, behavior trend analysis and the like; and the abnormal flow detection module (3) carries out real-time detection on flow log analysis data based on the configured abnormal flow detection model, and carries out processing such as alarming according to business rules after judging and identifying abnormal access flow.
Claims (8)
1. A big data abnormal flow detection method and system applied to an integral system are characterized by comprising the following steps: the method is oriented to the integral service system, obtains service access flow information (such as access flow logs) of each service system in real time, performs real-time analysis processing through big data, performs behavior classification, analysis and other processing on the flow logs, then performs matching by combining an abnormal flow detection model, and performs judgment, identification and alarm on abnormal flow.
2. The system comprises a flow log collection module, a flow big data analysis processing module, an abnormal flow detection module, an abnormal flow monitoring management module and a main service processing flow.
3. The method of claim 1, wherein the traffic log collection module: and the system is oriented to the integral service system and is used for collecting and storing global flow log information in real time.
4. The flow log information comes from the front and rear end buried points of the service system and comprises service codes, client access IP, client equipment type (mobile end or PC end), browser type, access time, request packet size, return data packet size and the like; the log aggregation processing can be performed by a common log collection tool, namely, the common log collection tool, deploys the flux agent at each service node and sends log data to the traffic log collection module.
5. The flow big data analysis and processing module: and analyzing and processing the collected and stored flow logs by using a big data platform and technology, wherein the analysis comprises data cleaning, access log classification/behavior classification, behavior trend analysis and the like.
6. An abnormal flow detection module: and based on the configured abnormal flow detection model, carrying out real-time detection on flow log analysis data, judging and identifying abnormal access flow, and then carrying out processing such as alarming according to business rules.
7. Abnormal flow monitoring and managing module: the method comprises the steps of configuring parameters and storage of a flow log collection module, configuring classification and behavior definition of a flow big data analysis processing module, and setting an abnormal flow model of a legacy flow detection module.
8. The main business processing flow is as follows: firstly, in an integral service system, according to the monitoring requirement of abnormal service flow, flow logs are buried, and a detection model and other modules are set through an abnormal flow monitoring management module; in the service operation process, the access log data after point burying are continuously converged to a flow log collection module in real time, and the converged log data are stored by the flow log collection module; the flow big data analysis processing module is used for analyzing and processing the stored access log data, and the analysis processing comprises data cleaning, access log classification/behavior classification, behavior trend analysis and the like; and the abnormal flow detection module is used for detecting the flow log analysis data in real time based on the configured abnormal flow detection model, judging and identifying abnormal access flow, and then performing processing such as alarming according to business rules.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011148805.2A CN112291225A (en) | 2020-10-23 | 2020-10-23 | Big data abnormal flow detection method and system applied to integral system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011148805.2A CN112291225A (en) | 2020-10-23 | 2020-10-23 | Big data abnormal flow detection method and system applied to integral system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112291225A true CN112291225A (en) | 2021-01-29 |
Family
ID=74423819
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011148805.2A Pending CN112291225A (en) | 2020-10-23 | 2020-10-23 | Big data abnormal flow detection method and system applied to integral system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112291225A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113034178A (en) * | 2021-03-15 | 2021-06-25 | 深圳市麦谷科技有限公司 | Multi-system integral calculation method and device, terminal equipment and storage medium |
| CN114900356A (en) * | 2022-05-06 | 2022-08-12 | 联云(山东)大数据有限公司 | Malicious user behavior detection method and device and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104618343A (en) * | 2015-01-06 | 2015-05-13 | 中国科学院信息工程研究所 | Method and system for detecting website threat based on real-time log |
| CN106487596A (en) * | 2016-10-26 | 2017-03-08 | 宜人恒业科技发展（北京）有限公司 | Distributed Services follow the tracks of implementation method |
| US20190188046A1 (en) * | 2015-04-06 | 2019-06-20 | EMC IP Holding Company LLC | Blockchain integration for scalable distributed computations |
| CN110086649A (en) * | 2019-03-19 | 2019-08-02 | 深圳壹账通智能科技有限公司 | Detection method, device, computer equipment and the storage medium of abnormal flow |
| CN111507742A (en) * | 2019-01-31 | 2020-08-07 | 何成 | Hyperchain integrating system |
-
2020
- 2020-10-23 CN CN202011148805.2A patent/CN112291225A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104618343A (en) * | 2015-01-06 | 2015-05-13 | 中国科学院信息工程研究所 | Method and system for detecting website threat based on real-time log |
| US20190188046A1 (en) * | 2015-04-06 | 2019-06-20 | EMC IP Holding Company LLC | Blockchain integration for scalable distributed computations |
| CN106487596A (en) * | 2016-10-26 | 2017-03-08 | 宜人恒业科技发展（北京）有限公司 | Distributed Services follow the tracks of implementation method |
| CN111507742A (en) * | 2019-01-31 | 2020-08-07 | 何成 | Hyperchain integrating system |
| CN110086649A (en) * | 2019-03-19 | 2019-08-02 | 深圳壹账通智能科技有限公司 | Detection method, device, computer equipment and the storage medium of abnormal flow |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113034178A (en) * | 2021-03-15 | 2021-06-25 | 深圳市麦谷科技有限公司 | Multi-system integral calculation method and device, terminal equipment and storage medium |
| CN114900356A (en) * | 2022-05-06 | 2022-08-12 | 联云(山东)大数据有限公司 | Malicious user behavior detection method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100463409C (en) | Network security system and method | |
| CN114584401B (en) | Tracing system and method for large-scale network attack | |
| US20030159069A1 (en) | Network-based attack tracing system and method using distributed agent and manager system | |
| CN106357685A (en) | Method and device for defending distributed denial of service attack | |
| CN112291225A (en) | Big data abnormal flow detection method and system applied to integral system | |
| CN109150869A (en) | A kind of exchanger information acquisition analysis system and method | |
| EP2747365A1 (en) | Network security management | |
| CN101668012A (en) | Method and device for detecting security event | |
| CN111970233B (en) | Analysis and identification method for network violation external connection scene | |
| KR100846835B1 (en) | Method and apparatus for Security Event Correlation Analysis based on Context Language | |
| CN115941317A (en) | Network security comprehensive analysis and situation awareness platform | |
| CN111786986A (en) | Numerical control system network intrusion prevention system and method | |
| CN113259367B (en) | Industrial control network flow multistage anomaly detection method and device | |
| CN112217777A (en) | Attack backtracking method and equipment | |
| CN113132370A (en) | Universal integrated safety pipe center system | |
| CN112217826A (en) | Network asset association analysis and dynamic supervision method based on flow perception | |
| CN115567258B (en) | Network security situation awareness method, system, electronic equipment and storage medium | |
| CN104717188A (en) | Asset object security protection system and method in industrial control firewall | |
| CN111049853A (en) | Security authentication system based on computer network | |
| CN114513342B (en) | Intelligent substation communication data safety monitoring method and system | |
| CN116015925A (en) | Data transmission method, device, equipment and medium | |
| CN115208690A (en) | Screening processing system based on data classification and classification | |
| CN106993005A (en) | The method for early warning and system of a kind of webserver | |
| CN112769847A (en) | Safety protection method, device, equipment and storage medium for Internet of things equipment | |
| CN112887288B (en) | Internet-based E-commerce platform intrusion detection front-end computer scanning system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| CB02 | Change of applicant information |
Address after: 200060 5th floor, 1207 Jiangning Road, Putuo District, Shanghai Applicant after: Yijifen (Shanghai) Digital Technology Co.,Ltd. Address before: 200060 5th floor, 1207 Jiangning Road, Putuo District, Shanghai Applicant before: Yijifen e-commerce (Shanghai) Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |