CN110958251A - Method and device for detecting and backtracking lost host based on real-time stream processing - Google Patents
Method and device for detecting and backtracking lost host based on real-time stream processing Download PDFInfo
- Publication number
- CN110958251A CN110958251A CN201911227773.2A CN201911227773A CN110958251A CN 110958251 A CN110958251 A CN 110958251A CN 201911227773 A CN201911227773 A CN 201911227773A CN 110958251 A CN110958251 A CN 110958251A
- Authority
- CN
- China
- Prior art keywords
- host
- data
- real
- lost
- backtracking
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1416—Event detection, e.g. attack signature detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Medical Informatics (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a method and a device for detecting and backtracking a lost host based on real-time stream processing, and relates to a technical scheme and a device for discovering the lost host through an anomaly detection algorithm. The invention takes the network connection behavior of the host as an analysis foothold, adopts the real-time stream processing technology to realize the machine learning anomaly detection algorithm, and brings mass historical data into the behavior pattern recognition of the host, so that the detection has more real-time property and landing property. Meanwhile, the abnormal host flow is retained, flow backtracking analysis can be carried out, accuracy of detection results is improved, meanwhile, threat information is updated, and the subsequent judging rate of the abnormal host is accelerated.
Description
Technical Field
The invention relates to the technical field of network security, in particular to a method and a device for detecting and backtracking a lost host based on real-time stream processing.
Background
The ckc (cyber Kill Chain) model proposed by Lockheed Martin decomposes the high-level threat process, and as the attack process goes deeper, the target host locked by the attacker will go through several stages, such as suffering from invasion, being controlled, and initiating malicious behaviors. After the target host is lost, the target host establishes connection with a remote C & C server, is continuously controlled by an attacker, and is often used as a springboard to perform a series of activities such as scanning attack, service attack denial, malicious website access, vulnerability intrusion, data stealing and the like on a new target of an internal network or an external network.
The existing host security is mostly identified by anti-virus software deployed on the host or an anti-virus gateway deployed on a network side, and the basic principle is mostly carried out on a feature signature based on a file hash; or the detection is carried out by flow analysis, and the network flow-based fail-safe host analysis mainly determines which flows belong to the abnormity based on rules. The attack detection of the lost host is realized based on the feature signature, and novel features cannot be early warned, for example, the attack behavior based on new generation and not supplemented in a feature library cannot be detected; the behavior of the lost host is more and more hidden, and the identification based on the statistical law is more and more difficult; the identification of a lost host purely based on threat intelligence is limited by the quality of the threat intelligence; the behavior process of the host computer for the defect lacks an effective history backtracking means, and the method has the limitation that the behavior process must be based on a certain specific scene.
Disclosure of Invention
The invention aims to provide a method and a device for detecting and backtracking a lost host based on real-time stream processing. Meanwhile, the abnormal host flow is retained, flow backtracking analysis can be carried out, accuracy of detection results is improved, meanwhile, threat information is updated, and the subsequent judging rate of the abnormal host is accelerated.
The technical scheme adopted by the invention is as follows:
a method for detecting and backtracking a lost host based on real-time stream processing comprises the following steps:
step 1, acquiring network data packets from each node of a network, retaining corresponding original network data packets based on configuration requirements, and extracting behavior data of a host from the network data packets based on a DPI technology through an analytic probe;
step 2, comparing the current real-time behavior data with historical data, and performing anomaly detection to judge whether a difference exists; if yes, executing step 3; otherwise, executing step 1;
step 3, storing an original network data packet of the host with abnormal detection data, recording network behavior data of the host in detail, and judging whether the host is an abnormal host by analyzing a corresponding data file; if yes, judging the host computer is suspected to be lost and executing the step 4; otherwise, executing step 1;
step 4, matching the suspected defect host information with threat information of a threat information library; when the matching is successful, the host is a lost host, the information of the lost host is output, and the step 1 is executed; otherwise, executing step 5;
step 5, analyzing the original network data packet of the host with the abnormal detection data, acquiring the original access details of the host, and performing host flow backtracking analysis to judge whether the host is a lost host; if yes, updating the host information to a threat information base; otherwise, step 1 is executed.
Further, the configuration requirement of retaining the original network data in step 1 is as follows: abnormal flow data that is not matched to threat intelligence or abnormal flow data that is matched to threat intelligence, or other custom conditions to aid in analysis and evaluation.
Further, the behavior data of the host in step 1 includes network interconnection behavior data, application access data and file transfer data.
Further, in step 2, the host is uniquely marked as an index based on real-time stream processing to construct a host behavior record, and then the behavior record is stored.
Further, in the step 2, a big data platform is used for carrying out anomaly detection on the real-time behavior data and the historical data together by adopting a machine learning algorithm.
A device for detecting a lost host and backtracking based on real-time stream processing comprises the following modules:
the host flow analysis module: acquiring network data packets from each node of a network, retaining corresponding original network data packets based on configuration requirements, and extracting behavior data of a host from the network data packets based on a DPI technology through an analytic probe;
the abnormal host detection module: by utilizing a real-time stream processing technology, various historical data and real-time data reported by a host flow analysis module are analyzed and judged in a big data engine based on an anomaly detection algorithm based on a big data analysis platform, and a suspected defect host is found;
threat intelligence matching module: matching the suspected lost host information with threat information, and outputting the lost host information when the matching is successful; when the matching is not successful, the host behavior backtracks;
the host flow backtracking module: further analyzing based on the reserved host flow message information, acquiring the original access details of the host, and studying and judging whether the host is lost;
threat intelligence update module: aiming at the condition that the host is determined to be a lost host based on the original network message research and judgment, the information is supplemented and updated to a threat information library for being called by a threat information matching module.
Further, the host traffic analysis module extracts the behavior data of the host from the network data packet by using a DPI technology.
Further, the behavior data of the host includes network interconnection behavior data, application access data, and file transfer data.
Further, the abnormal host detection module uses the unique host mark as an index to construct a host behavior record, and further stores the behavior record.
Further, the abnormal host detection module utilizes a big data platform to perform abnormal detection on the real-time behavior data and the historical data together by adopting a machine learning algorithm.
The invention adopts the technical scheme, is based on an open source big data calculation engine, adopts a real-time stream processing technology with strong practicability, is very suitable for real-time security analysis, simultaneously supports batch processing and stream processing, and ensures flexible window and accurate semantics to make the analysis more in line with the requirements of actual business analysis; by adopting a machine learning algorithm, the studying, judging and predicting are more accurately realized by taking historical data into calculation, and meanwhile, the problem host which is controlled and can not be identified through a statistical rule is identified; and by adopting an original flow message retention technology, further data support is provided for the behavior analysis of the abnormal host by accurately positioning and reproducing the historical behavior.
Drawings
The invention is described in further detail below with reference to the accompanying drawings and the detailed description;
FIG. 1 is a schematic view of a flow structure of a method for detecting and backtracking a failed host based on real-time stream processing according to the present invention;
fig. 2 is a schematic structural diagram of a device for detecting and backtracking a failed host based on real-time stream processing according to the present invention.
Detailed Description
As shown in fig. 1 or fig. 2, the present invention discloses a method for detecting and backtracking a failed host based on real-time stream processing, which comprises the following steps:
step 1, acquiring network data packets from each node of a network, retaining corresponding original network data packets based on configuration requirements, and extracting behavior data of a host from the network data packets based on a DPI technology through an analytic probe;
step 2, comparing the current real-time behavior data with historical data, and performing anomaly detection to judge whether a difference exists; if yes, executing step 3; otherwise, executing step 1;
step 3, storing an original network data packet of the host with abnormal detection data, recording network behavior data of the host in detail, and judging whether the host is an abnormal host by analyzing a corresponding data file; if yes, judging the host computer is suspected to be lost and executing the step 4; otherwise, executing step 1;
step 4, matching the suspected defect host information with threat information of a threat information library; when the matching is successful, the host is a lost host, the information of the lost host is output, and the step 1 is executed; otherwise, executing step 5;
step 5, analyzing the original network data packet of the host with the abnormal detection data, acquiring the original access details of the host, and performing host flow backtracking analysis to judge whether the host is a lost host; if yes, updating the host information to a threat information base; otherwise, step 1 is executed.
Further, the configuration requirement of retaining the original network data in step 1 is as follows: abnormal flow data that is not matched to threat intelligence or abnormal flow data that is matched to threat intelligence, or other custom conditions to aid in analysis and evaluation.
Further, the behavior data of the host in step 1 includes network interconnection behavior data, application access data and file transfer data.
Further, in step 2, the host is uniquely marked as an index based on real-time stream processing to construct a host behavior record, and then the behavior record is stored.
Further, in the step 2, a big data platform is used for carrying out anomaly detection on the real-time behavior data and the historical data together by adopting a machine learning algorithm. The details of the logic realized by the anomaly detection by adopting a machine learning algorithm are as follows:
a device for detecting a lost host and backtracking based on real-time stream processing comprises the following modules:
the host flow analysis module: acquiring network data packets from each node of a network, retaining corresponding original network data packets based on configuration requirements, and extracting behavior data of a host from the network data packets based on a DPI technology through an analytic probe;
the abnormal host detection module: by utilizing a real-time stream processing technology, various historical data and real-time data reported by a host flow analysis module are analyzed and judged in a big data engine based on an anomaly detection algorithm based on a big data analysis platform, and a suspected defect host is found;
threat intelligence matching module: matching the suspected lost host information with threat information, and outputting the lost host information when the matching is successful; when the matching is not successful, the host behavior backtracks;
the host flow backtracking module: further analyzing based on the reserved host flow message information, acquiring the original access details of the host, and studying and judging whether the host is lost;
threat intelligence update module: aiming at the condition that the host is determined to be a lost host based on the original network message research and judgment, the information is supplemented and updated to a threat information library for being called by a threat information matching module.
Further, the host traffic analysis module extracts the behavior data of the host from the network data packet by using a DPI technology.
Further, the behavior data of the host includes network interconnection behavior data, application access data, and file transfer data.
The invention adopts the technical scheme, is based on an open source big data calculation engine, adopts a real-time stream processing technology with strong practicability, is very suitable for real-time security analysis, simultaneously supports batch processing and stream processing, and ensures flexible window and accurate semantics to make the analysis more in line with the requirements of actual business analysis; by adopting a machine learning algorithm, the studying, judging and predicting are more accurately realized by taking historical data into calculation, and meanwhile, the problem host which is controlled and can not be identified through a statistical rule is identified; and by adopting an original flow message retention technology, further data support is provided for the behavior analysis of the abnormal host by accurately positioning and reproducing the historical behavior.
Claims (10)
1. A method for detecting and backtracking a lost host based on real-time stream processing is characterized in that: which comprises the following steps:
step 1, acquiring network data packets from each node of a network, retaining corresponding original network data packets based on configuration requirements, and extracting behavior data of a host from the network data packets based on a DPI technology through an analytic probe;
step 2, comparing the current real-time behavior data with historical data, and performing anomaly detection to judge whether a difference exists; if yes, executing step 3; otherwise, executing step 1;
step 3, storing an original network data packet of the host with abnormal detection data, recording network behavior data of the host in detail, and judging whether the host is an abnormal host by analyzing a corresponding data file; if yes, judging the host computer is suspected to be lost and executing the step 4; otherwise, executing step 1;
step 4, matching the suspected defect host information with threat information of a threat information library; when the matching is successful, the host is a lost host, the information of the lost host is output, and the step 1 is executed; otherwise, executing step 5;
step 5, analyzing the original network data packet of the host with the abnormal detection data, acquiring the original access details of the host, and performing host flow backtracking analysis to judge whether the host is a lost host; if yes, updating the host information to a threat information base; otherwise, step 1 is executed.
2. The method for detecting and backtracking a failed host based on real-time stream processing according to claim 1, wherein: the configuration requirement for retaining the original network data in the step 1 is as follows: abnormal flow data that is not matched to threat intelligence or abnormal flow data that is matched to threat intelligence, or other custom conditions to aid in analysis and evaluation.
3. The method for detecting and backtracking a failed host based on real-time stream processing according to claim 1, wherein: the behavior data of the host in the step 1 comprises network interconnection behavior data, application access data and file transmission data.
4. The method for detecting and backtracking a failed host based on real-time stream processing according to claim 1, wherein: and 2, constructing a host behavior record by using the unique host mark as an index based on real-time stream processing, and further storing the behavior record.
5. The method for detecting and backtracking a failed host based on real-time stream processing according to claim 1, wherein: and 2, carrying out anomaly detection on the real-time behavior data and the historical data by using a machine learning algorithm through a big data platform.
6. An apparatus for detecting and backtracking a lost host based on real-time stream processing, which applies any one of claims 1 to 5, and is characterized in that: the device comprises the following modules:
the host flow analysis module: acquiring network data packets from each node of the network, retaining corresponding original network data packets based on configuration requirements, extracting behavior data of a host from the network data packets by an analytic probe based on a DPI technology,
the abnormal host detection module: by utilizing a real-time stream processing technology, various historical data and real-time data reported by a host flow analysis module are analyzed and judged in a big data engine based on an anomaly detection algorithm based on a big data analysis platform, and a suspected defect host is found;
threat intelligence matching module: matching the suspected lost host information with threat information, and outputting the lost host information when the matching is successful; when the matching is not successful, the host behavior backtracks;
the host flow backtracking module: further analyzing based on the reserved host flow message information, acquiring the original access details of the host, and studying and judging whether the host is lost;
threat intelligence update module: aiming at the condition that the host is determined to be a lost host based on the original network message research and judgment, the information is supplemented and updated to a threat information library for being called by a threat information matching module.
7. The device according to claim 6, wherein the device for detecting and backtracking the lost host based on real-time stream processing comprises: and the host flow analysis module extracts the behavior data of the host from the network data packet by adopting a DPI technology.
8. The device according to claim 6, wherein the device for detecting and backtracking the lost host based on real-time stream processing comprises: the behavior data of the host computer comprises network interconnection behavior data, application access data and file transfer data.
9. The device according to claim 6, wherein the device for detecting and backtracking the lost host based on real-time stream processing comprises: the abnormal host detection module takes the unique host mark as an index to construct a host behavior record and further stores the behavior record.
10. The device according to claim 6, wherein the device for detecting and backtracking the lost host based on real-time stream processing comprises: and the abnormal host detection module utilizes the big data platform to carry out abnormal detection on the real-time behavior data and the historical data together by adopting a machine learning algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911227773.2A CN110958251A (en) | 2019-12-04 | 2019-12-04 | Method and device for detecting and backtracking lost host based on real-time stream processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911227773.2A CN110958251A (en) | 2019-12-04 | 2019-12-04 | Method and device for detecting and backtracking lost host based on real-time stream processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110958251A true CN110958251A (en) | 2020-04-03 |
Family
ID=69979694
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911227773.2A Pending CN110958251A (en) | 2019-12-04 | 2019-12-04 | Method and device for detecting and backtracking lost host based on real-time stream processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110958251A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112422554A (en) * | 2020-11-17 | 2021-02-26 | 杭州安恒信息技术股份有限公司 | Method, device, equipment and storage medium for detecting abnormal traffic external connection |
| CN113157652A (en) * | 2021-05-12 | 2021-07-23 | 中电福富信息科技有限公司 | User line image and abnormal behavior detection method based on user operation audit |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170099310A1 (en) * | 2015-10-05 | 2017-04-06 | Cisco Technology, Inc. | Dynamic deep packet inspection for anomaly detection |
| CN108965346A (en) * | 2018-10-10 | 2018-12-07 | 上海工程技术大学 | One kind is fallen Host Detection method |
| CN109391602A (en) * | 2017-08-11 | 2019-02-26 | 北京金睛云华科技有限公司 | A kind of zombie host detection method |
| CN109660539A (en) * | 2018-12-20 | 2019-04-19 | 北京神州绿盟信息安全科技股份有限公司 | It falls device identification method, device, electronic equipment and storage medium |
| CN109688092A (en) * | 2018-04-25 | 2019-04-26 | 北京微步在线科技有限公司 | It falls equipment detection method and device |
-
2019
- 2019-12-04 CN CN201911227773.2A patent/CN110958251A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170099310A1 (en) * | 2015-10-05 | 2017-04-06 | Cisco Technology, Inc. | Dynamic deep packet inspection for anomaly detection |
| CN109391602A (en) * | 2017-08-11 | 2019-02-26 | 北京金睛云华科技有限公司 | A kind of zombie host detection method |
| CN109688092A (en) * | 2018-04-25 | 2019-04-26 | 北京微步在线科技有限公司 | It falls equipment detection method and device |
| CN108965346A (en) * | 2018-10-10 | 2018-12-07 | 上海工程技术大学 | One kind is fallen Host Detection method |
| CN109660539A (en) * | 2018-12-20 | 2019-04-19 | 北京神州绿盟信息安全科技股份有限公司 | It falls device identification method, device, electronic equipment and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112422554A (en) * | 2020-11-17 | 2021-02-26 | 杭州安恒信息技术股份有限公司 | Method, device, equipment and storage medium for detecting abnormal traffic external connection |
| CN112422554B (en) * | 2020-11-17 | 2023-04-07 | 杭州安恒信息技术股份有限公司 | Method, device, equipment and storage medium for detecting abnormal traffic external connection |
| CN113157652A (en) * | 2021-05-12 | 2021-07-23 | 中电福富信息科技有限公司 | User line image and abnormal behavior detection method based on user operation audit |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Farhadi et al. | Alert correlation and prediction using data mining and HMM. | |
| CN105704103B (en) | Modbus TCP communication behavior abnormity detection method based on OCSVM double-contour model | |
| EP2040435B1 (en) | Intrusion detection method and system | |
| CN111698260B (en) | DNS hijacking detection method and system based on message analysis | |
| CN113645232B (en) | Intelligent flow monitoring method, system and storage medium for industrial Internet | |
| CN109660518B (en) | Communication data detection method and device of network and machine-readable storage medium | |
| US10462170B1 (en) | Systems and methods for log and snort synchronized threat detection | |
| Vidal et al. | Alert correlation framework for malware detection by anomaly-based packet payload analysis | |
| CN112492059A (en) | DGA domain name detection model training method, DGA domain name detection device and storage medium | |
| Dhakar et al. | A novel data mining based hybrid intrusion detection framework | |
| US11115823B1 (en) | Internet-of-things device classifier | |
| CN112333128B (en) | Web attack behavior detection system based on self-encoder | |
| CN114422184A (en) | Network security attack type and threat level prediction method based on machine learning | |
| CN110958251A (en) | Method and device for detecting and backtracking lost host based on real-time stream processing | |
| CN113904795A (en) | Rapid and accurate flow detection method based on network security probe | |
| Kozik et al. | Pattern extraction algorithm for NetFlow‐based botnet activities detection | |
| CN117220920A (en) | Firewall policy management method based on artificial intelligence | |
| CN114785563A (en) | Encrypted malicious flow detection method for soft voting strategy | |
| JP7086230B2 (en) | Protocol-independent anomaly detection | |
| CN109547496B (en) | Host malicious behavior detection method based on deep learning | |
| CN111464510A (en) | Network real-time intrusion detection method based on rapid gradient lifting tree model | |
| CN115211075A (en) | Network attack identification in a network environment | |
| Thanthrige | Hidden markov model based intrusion alert prediction | |
| Protic et al. | WK-FNN design for detection of anomalies in the computer network traffic | |
| Sulaiman et al. | Big data analytic of intrusion detection system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200403 |