CN108076019B - Abnormal flow detection method and device based on flow mirror image - Google Patents
Abnormal flow detection method and device based on flow mirror image Download PDFInfo
- Publication number
- CN108076019B CN108076019B CN201611012012.1A CN201611012012A CN108076019B CN 108076019 B CN108076019 B CN 108076019B CN 201611012012 A CN201611012012 A CN 201611012012A CN 108076019 B CN108076019 B CN 108076019B
- Authority
- CN
- China
- Prior art keywords
- flow
- tenant
- traffic
- abnormal
- mirror
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/20—Support for services
- H04L49/208—Port mirroring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Abstract
The invention discloses an abnormal flow detection method and device based on flow mirror images. The abnormal flow detection method comprises the following steps: the method comprises the steps of acquiring the mirror image flow of the cloud IDC in real time by adopting a flow mirror image method, and distributing the acquired mirror image flow according to tenants to obtain the respective mirror image flow of each tenant; performing feature extraction on the mirror image flow of each tenant to obtain and store feature data of the mirror image flow; and for each tenant, judging whether the current mirror flow is abnormal flow or not by combining a Poisson distribution and outlier detection method according to the characteristic data of the current mirror flow and the characteristic data of the historical mirror flow. The abnormal flow detection device comprises a flow mirror image unit, a shunting unit, a feature extraction unit and an abnormal detection unit, which are respectively used for realizing the processing and the function corresponding to the corresponding steps in the detection method. The abnormal traffic detection method and the abnormal traffic detection device can process mass traffic and accurately find network attacks or system abnormalities.
Description
Technical Field
The present invention relates to network technologies, and in particular, to a method and an apparatus for detecting abnormal traffic based on a traffic mirror.
Background
With the development of computer and network technologies, network traffic anomaly detection and massive data processing technologies gradually become research hotspots.
Currently, the existing abnormal traffic detection technology adopts a traffic sampling mode to perform analysis, and has insufficient support for traffic analysis of a cloud Internet Data Center (IDC), and is more focused on traffic analysis of the internet or certain services. Under the condition of determining the service, the characteristics of the traffic are relatively determined, but the traffic complexity is much higher in a cloud environment, and a larger challenge is met.
Therefore, the existing abnormal flow detection technology has insufficient capacity of analyzing the large flow of a cloud service provider, and especially has low accuracy of abnormal flow detection when analyzing and processing the flow of the cloud IDC.
Disclosure of Invention
The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. It should be understood that this summary is not an exhaustive overview of the invention. It is not intended to determine the key or critical elements of the present invention, nor is it intended to limit the scope of the present invention. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is discussed later.
In order to solve the technical problems, the invention provides an abnormal traffic detection method and device based on traffic mirroring, so as to at least solve the problem that the accuracy of abnormal traffic detection is low when the existing abnormal traffic detection technology is used for analyzing and processing cloud IDC traffic.
In order to achieve the object of the present invention, according to an aspect of the present invention, there is provided an abnormal traffic detection method based on a traffic mirror, the abnormal traffic detection method including: the method comprises the steps of acquiring the mirror image flow of the cloud IDC in real time by adopting a flow mirror image method, and distributing the acquired mirror image flow according to tenants to obtain the respective mirror image flow of each tenant; performing feature extraction on the mirror image flow of each tenant to obtain and store feature data of the mirror image flow; and for each tenant, judging whether the current mirror flow is abnormal flow or not by combining a Poisson distribution and outlier detection method according to the characteristic data of the current mirror flow and the characteristic data of the historical mirror flow.
Further, the outlier detection method includes one or more of a Density-Based Clustering method with Noise (DBSCAN), a Support Vector Machine (SVM), a logistic regression, and a linear fitting.
Further, the step of determining whether the current mirror traffic is abnormal traffic by combining the poisson distribution and outlier detection method includes: judging whether the characteristic data of the historical mirror image flow of the tenant corresponds to a stable service type or not according to the characteristic data of the historical mirror image flow of the tenant; under the condition that the tenant is judged to correspond to the stable service type, a chi-square calibration method and a least square method are adopted to carry out linear fitting to obtain a characteristic predicted value corresponding to the current mirror flow of the tenant, and whether the current mirror flow is abnormal flow is determined according to the difference between the characteristic predicted value corresponding to the current mirror flow of the tenant and actual characteristic data.
Further, the step of determining whether the current mirror traffic is abnormal traffic by combining the poisson distribution and outlier detection method further includes: determining historical mirror flow in a preset time period before the current mirror flow, obtaining a corresponding characteristic predicted value of the current mirror flow by using the characteristic data of the historical mirror flow in the preset time period and adopting a logistic regression method, and determining whether the current mirror flow is abnormal flow according to the difference between the characteristic predicted value corresponding to the current mirror flow of the tenant and the actual characteristic data.
Further, the step of extracting features of the mirror traffic of each tenant includes: for the mirror traffic of each tenant, a plurality of features of the mirror traffic are extracted, and at least part of the features are selected from the plurality of features as a feature extraction result by calculating correlations between the plurality of features.
Further, the characteristic data of the mirrored flow comprises at least part of the following characteristics: a current flow value; network protocol data; application layer data; a transmission rate; a packet forwarding rate; source IP distribution; IP characteristics; a protocol of the load; the number of connections; URL access frequency.
Further, the abnormal traffic detection method further includes: for each tenant, judging whether the tenant meets a preset condition or not based on the characteristic data of the historical mirror image flow of the tenant, and judging that the tenant has service abnormality under the condition that the tenant meets the preset condition; the preset conditions are as follows: the number of connections, URL access frequency and traffic of the tenant are increased, and the source IP reputation value of the tenant is higher than a preset value.
According to another aspect of the present invention, there is also provided an abnormal traffic detection apparatus based on traffic mirroring, the abnormal traffic detection apparatus including: the flow mirroring unit is used for acquiring the mirroring flow of the cloud IDC in real time by adopting a flow mirroring method; the flow distribution unit is used for distributing the acquired mirror image flow according to the tenants to obtain the respective mirror image flow of each tenant; the characteristic extraction unit is used for extracting the characteristics of the mirror image flow of each tenant to obtain and store the characteristic data of the mirror image flow; and the anomaly detection unit is used for judging whether the current mirror flow is the abnormal flow or not according to the feature data of the current mirror flow and the feature data of the historical mirror flow of each tenant by combining a Poisson distribution and outlier detection method.
Further, the outlier detection method adopted by the anomaly detection unit comprises one or more of DBSCAN, SVM, logistic regression and linear fitting.
Further, the abnormality detection unit is further configured to: for each tenant, judging whether the tenant meets a preset condition or not based on the characteristic data of the historical mirror image flow of the tenant, and judging that the tenant has service abnormality under the condition that the tenant meets the preset condition; the preset conditions are as follows: the number of connections, URL access frequency and traffic of the tenant are increased, and the source IP reputation value of the tenant is higher than a preset value.
The abnormal flow detection method and device based on the flow mirror image acquire the cloud IDC mirror image flow in real time by adopting the flow mirror image method, and the characteristic data of the mirror image flow is obtained by preprocessing the high-performance data packet of the mirror image flow of each tenant, so that the abnormal flow is detected based on the characteristic data of the current mirror image flow and the characteristic data of the historical mirror image flow.
The abnormal flow detection method and the abnormal flow detection device of the invention process the flow data of the cloud IDC by adopting the flow mirroring technology, and have the following beneficial effects: (1) the method can process massive traffic, can accurately find network attacks or system abnormalities, and can solve the problems that the prior art is insufficient in analysis capability of processing massive network traffic and data and cannot accurately detect abnormal traffic during cloud IDC traffic processing and analysis; (2) compared with the prior art, the abnormal flow detection method and the abnormal flow detection device can provide real-time detection and quickly locate the abnormal object, so that the problem that the analysis and detection real-time performance of the prior art is insufficient is solved; (3) the method is suitable for being implemented in the cloud computing IDC machine room and is convenient to expand.
In addition, in some embodiments of the present invention, DPDK may be used for high performance packet processing, and a streaming computing platform may be used for low latency detection logic, so that the latency of detection may be reduced.
In addition, in some embodiments of the invention, by optimizing the detection method and adopting a plurality of detection methods, the limitation caused by a single method is avoided, and the probability of false alarm is reduced.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the example serve to explain the principles of the invention and not to limit the invention.
FIG. 1A is a flow diagram of an exemplary process for a traffic mirror based abnormal traffic detection method of the present invention;
FIG. 1B is a diagram of an abnormal traffic detection method based on traffic mirroring according to the present invention
FIG. 1C is a network topology diagram of the abnormal traffic detection method based on traffic mirroring according to the present invention;
FIG. 2 is a flow chart of a first sub-process flow;
FIG. 3 is a flow chart of a second sub-process flow;
FIG. 4 is a flow diagram of another exemplary process for a traffic mirroring-based abnormal traffic detection method of the present invention;
fig. 5 is a block diagram illustrating an exemplary configuration of an abnormal traffic detection apparatus based on traffic mirroring according to the present invention.
Skilled artisans appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help improve the understanding of the embodiments of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict.
The steps illustrated in the flow charts of the figures may be performed in a computer system such as a set of computer-executable instructions. Also, while a logical order is shown in the flow diagrams, in some cases, the steps shown or described may be performed in an order different than here.
The embodiment of the invention provides an abnormal flow detection method based on flow mirror image, which comprises the following steps: the method comprises the steps of acquiring the mirror image flow of the cloud IDC in real time by adopting a flow mirror image method, and distributing the acquired mirror image flow according to tenants to obtain the respective mirror image flow of each tenant; performing feature extraction on the mirror image flow of each tenant to obtain and store feature data of the mirror image flow; and for each tenant, judging whether the current mirror flow is abnormal flow or not by combining a Poisson distribution and outlier detection method according to the characteristic data of the current mirror flow and the characteristic data of the historical mirror flow.
The processing flow of an example of the abnormal traffic detection method based on traffic mirroring is described below with reference to fig. 1A.
As shown in fig. 1A, after the method starts, step S110 is performed. In step S110, a traffic mirroring method is used to obtain mirror traffic of the cloud IDC in real time, and the obtained mirror traffic is distributed according to tenants, so as to obtain respective mirror traffic of each tenant. Then, step S120 is performed.
The traffic mirroring technique is to forward data traffic of one or more source ports to a certain designated port on a switch or a router, for example, to monitor the network, where the designated port is called a "mirror port" and the traffic of the network can be monitored and analyzed through the mirror port without seriously affecting the normal throughput of the source ports.
In addition, real-time computation is generally performed on massive data, and generally requires the second level.
In step S120, feature data of the mirror traffic of each tenant is obtained and stored by performing feature extraction on the mirror traffic of each tenant. Then, step S130 is performed.
Step S120 is a high-performance packet preprocessing step, in which high-performance packet processing may be implemented by using DPDK (a packet forwarding processing suite published by Intel corporation), each feature of the obtained mirror image traffic is analyzed, and the mirror image traffic preprocessing is accelerated. For example, the HTTP protocol, which includes some characteristic fields, can establish a state machine based on this, and perform protocol parsing by matching the payload content of the packet. Parallel matching is required due to the need for performance enhancement.
According to one implementation, the step of performing feature extraction on the mirror traffic of each tenant in step S120 may be implemented by: for the mirror traffic of each tenant, a plurality of features of the mirror traffic are extracted, and at least part of the features are selected from the plurality of features as a feature extraction result by calculating correlations between the plurality of features.
For example, the characteristic data of the mirrored traffic may include at least part of the following characteristics: a current flow value; network protocol (TCP/UDP) data; application layer (e.g., HTTP) data; transmission rate (in bps); packet forwarding rate (in pps); source IP distribution; IP characteristics; a protocol of the load; the number of connections; URL access frequency.
In step S130, for each tenant, it is determined whether the current mirror traffic is an abnormal traffic (such as an attack traffic) according to the feature data of the current mirror traffic and the feature data of the historical mirror traffic, in combination with a poisson distribution and outlier detection method. Thus, the process may be ended after step S130 is executed. For example, the anomaly may be found by matching the flow distribution, analyzing the correlation of each feature (euclidean distance may be used to describe the correlation, using a machine learning algorithm to find the degree of correlation between features, such as bayesian formula), finding outliers, and the like.
According to one implementation, the outlier detection method may include one or more of DBSCAN, SVM, logistic regression, linear fitting.
In one example, the process of step S130 can be implemented by a flow as shown in fig. 1B, for example. As shown in fig. 1B, step S131 is first performed. In step S131, the bps and pps values of the current mirror traffic are counted by hour as the feature data of the current mirror traffic, and the corresponding feature data of the historical mirror traffic is the bps and pps values of the same time period every day in a past period of time. For example, the bps and pps values of 20: 00-21: 00 of the current day are obtained through statistics, and the characteristic data of the corresponding historical mirror flow is the bps and pps values of 20: 00-21: 00 of each day of the past 30 days (or other time periods).
In this way, in step S132, the probability of occurrence of the feature data of the current mirror flow is calculated according to the poisson algorithm using the feature data of the corresponding historical mirror flow and the feature data of the current mirror flow, and the probability is denoted as P.
Then, in step S133, it is determined whether or not the probability P is less than 80%: if yes, go to step S134; otherwise, it is determined that the current mirror traffic is the normal traffic (step S135).
In step S134, it is determined that there is an abnormality in the current mirror traffic, and further calculation is required for determination at this time.
Then, in step 136, an outlier calculation is performed by using the feature data of the historical mirror traffic and the feature data of the current mirror traffic to obtain an outlier factor of each segment of the current mirror traffic and the historical mirror traffic.
For example, the feature data of the current mirror flow is included in the feature data of the historical mirror flow, and then a Local Outlier Factor (LOF) detection method is used to perform distance calculation on the data (i.e., the feature data of the historical mirror flow and the feature data of the current mirror flow) to obtain an Outlier Factor of each point. Each point referred to herein means each segment of mirror traffic, such as current mirror traffic and each segment of historical mirror traffic.
Then, in step S137, all the outlier factors obtained in step S136 are sorted from large to small, and k (for example, k may be 3 or 5) outlier factors ranked at the top are taken.
In this way, in step S138, it is determined whether the outlier factor of the current mirror traffic is included in the k outlier factors: if yes, judging that the current mirror flow is abnormal flow, and alarming (step S139); otherwise, it is determined that the current mirror traffic is the normal traffic (step S135).
Fig. 1C shows a network topology for implementing the above abnormal traffic detection method based on traffic mirroring in one example of the present invention.
As shown in fig. 1C, the mirror traffic (corresponding to step S110 shown in fig. 1A) obtained from the cloud IDC by the traffic mirror technique is transferred to "traffic preprocessing", and the traffic deep analysis result is derived after the "traffic preprocessing" (corresponding to step S120 shown in fig. 1A), and then is sent to the "computing cluster". The deep traffic analysis can detect the application protocols of the data packets, such as QQ, BT, WeChat and the like, increase the characteristics of the abnormal detection algorithm and improve the accuracy. The "computing cluster" is used to execute an anomaly detection program (corresponding to step S130 shown in fig. 1A), and the "computing cluster" is linked with a firewall to improve the security thereof.
According to one implementation, the step of determining whether the current mirror traffic is abnormal traffic in combination with the poisson distribution and outlier detection method in step S130 may include a first sub-process flow as shown in fig. 2. As shown in fig. 2, after the first sub-process flow is started, step S210 is executed.
In step S210, it is determined whether the tenant corresponds to a stable service type according to the characteristic data of the historical mirror traffic of the tenant. If the tenant is determined to correspond to the stable service type, step S220 is executed; otherwise, the first sub-processing flow is ended.
In step S220, a chi-square calibration method and a least square method are used to perform linear fitting to obtain a feature prediction value corresponding to the current mirror flow of the tenant. Then, step S230 is performed.
In step S230, it is determined whether the current mirror traffic is an abnormal traffic according to a difference between a feature prediction value corresponding to the current mirror traffic of the tenant and actual feature data. The first sub-process flow is ended.
In addition, according to another implementation manner, the step of determining whether the current mirror traffic is an abnormal traffic in combination with the poisson distribution and outlier detection method in step S130 may also include a second sub-process flow shown in fig. 3. As shown in fig. 3, after the second sub-process flow is started, step S310 is executed.
In step S310, a historical mirror traffic within a predetermined time period before the current mirror traffic is determined. Then, step S320 is performed.
In step S320, a logistic regression method is used to obtain a corresponding feature prediction value of the current mirror flow rate by using the feature data of the historical mirror flow rate in the predetermined time period. Then, step S330 is performed.
In step S330, it is determined whether the current mirror traffic is an abnormal traffic according to a difference between a feature prediction value corresponding to the current mirror traffic of the tenant and actual feature data. The second sub-process flow is ended.
For example, in step S330, a set of thresholds, such as a first threshold S1, a second threshold S2, and an nth threshold SN of …, may be preset, where N is the number of feature types (the number of feature types for feature extraction of the mirror traffic of each tenant described in step S120). In this way, each feature corresponds to a threshold, the difference between the predicted feature value and the actual feature value corresponding to each feature is calculated, and if the difference between the predicted feature value and the actual feature value corresponding to any feature is greater than the corresponding threshold in the feature data corresponding to the current mirror flow, whether the current mirror flow is abnormal or not is determined.
In summary, since there are a large number of tenants in the cloud IDC and the service has a large uncertainty, in the embodiment of the present invention, a plurality of ways of comprehensive inspection are adopted to discover the abnormal traffic. The features extracted from the flow mirror image are analyzed, and for stable services, the flow distribution generally conforms to the poisson distribution, so that whether the flow features of the source meet the distribution or not can be verified. For example, the flow of a certain IP in a week satisfies poisson distribution, or satisfies distribution at a certain time every day, so the chi-square method can be used for verification, and the least square method is used for linear fitting to predict. The flow generally accords with linear characteristics in a small time interval, and the characteristic data of the next time point is predicted by adopting a logistic regression mode.
The processing flow of another example of the abnormal traffic detection method based on traffic mirroring is described below with reference to fig. 4.
As shown in fig. 4, the process flow includes steps S410 to S460. Steps S410 to S430 may be the same as steps S110 to S130 described above with reference to fig. 1A, respectively, and may achieve similar functions and effects, which are not described again here.
After steps S410-S430 are performed, step S440 is performed.
In step S440, for each tenant, it is determined whether the tenant meets a preset condition based on the feature data of the historical mirror traffic of the tenant. If the preset condition is satisfied, executing step S450; otherwise, step S460 is performed.
In step S450, it is determined that the tenant has a service exception, and the process is ended.
In step S460, it is determined that the tenant does not have a service exception, and then the process ends.
Wherein the preset conditions are as follows: the number of connections, URL access frequency and traffic of the tenant are increased, and the source IP reputation value of the tenant is higher than a preset value.
In one example, a traffic type anomaly (e.g., kill of seconds, sudden increase of traffic, code error, etc.) can be detected by calculating Round-Trip Time (RTT) of a tenant, monitoring connection data thereof in real Time, locating a CC (meaning black hole Challenge) or a tenant system overload. And (3) carrying out protocol analysis on the flow of the client, establishing a user flow model according to historical data, and if the client connection number, URL access frequency and flow increase are found and the source IP credit value is higher, estimating that the service is abnormal.
In the prior art, the service is monitored at a host end, so that the requirement can be met, but the monitoring system is difficult to develop and deploy and needs high technical quality, so that the implementation of small and medium-sized users in the cloud is difficult. The invention adopts the mirror image flow data analysis capability, and can help to monitor and find problems in time.
As can be seen from the above diagram, we detect both traffic type anomalies and business anomalies, and give users more comprehensive anomaly traffic detection.
In summary, abnormal traffic detection is a systematic project, and needs to be improved in terms of processing capability, scalability, and algorithm to implement a cloud computing environment.
In addition, an embodiment of the present invention further provides an abnormal traffic detection apparatus based on a traffic mirror, where the abnormal traffic detection apparatus includes: the flow mirroring unit is used for acquiring the mirroring flow of the cloud IDC in real time by adopting a flow mirroring method; the flow distribution unit is used for distributing the acquired mirror image flow according to the tenants to obtain the respective mirror image flow of each tenant; the characteristic extraction unit is used for extracting the characteristics of the mirror image flow of each tenant to obtain and store the characteristic data of the mirror image flow; and the anomaly detection unit is used for judging whether the current mirror flow is the abnormal flow or not according to the feature data of the current mirror flow and the feature data of the historical mirror flow of each tenant by combining a Poisson distribution and outlier detection method.
The structure of one example of the abnormal flow rate detecting apparatus based on the flow rate mirror of the present invention is described below with reference to fig. 5.
As shown in fig. 5, the abnormal flow rate detection apparatus 500 based on flow rate mirroring of the present invention includes a flow rate mirroring unit 501, a flow dividing unit 502, a feature extraction unit 503, and an abnormal detection unit 504.
The traffic mirroring unit 501 is configured to obtain a mirroring traffic of the cloud IDC in real time by using a traffic mirroring method.
The offloading unit 502 is configured to offload the obtained mirror traffic according to tenants, so as to obtain respective mirror traffic of each tenant.
The feature extraction unit 503 is configured to perform feature extraction on the mirror traffic of each tenant to obtain and store feature data of the mirror traffic. For example, the feature extraction unit 503 may be provided with a storage module for storing feature data of the mirror traffic of each tenant.
The anomaly detection unit 504 is configured to determine, for each tenant, whether the current mirror traffic is an anomalous traffic by combining poisson distribution and an outlier detection method according to the feature data of the current mirror traffic and the feature data of the historical mirror traffic.
For example, the anomaly detection unit 504 may adopt one or more outlier detection methods of DBSCAN, SVM, logistic regression, and linear fitting.
According to one implementation, the anomaly detection unit 504 may be further configured to: for each tenant, judging whether the tenant meets a preset condition or not based on the characteristic data of the historical mirror image flow of the tenant, and judging that the tenant has service abnormality under the condition that the tenant meets the preset condition; the preset conditions are as follows: the number of connections, URL access frequency and traffic of the tenant are increased, and the source IP reputation value of the tenant is higher than a preset value.
Although the embodiments of the present invention have been described above, the above description is only for the convenience of understanding the present invention, and is not intended to limit the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. The abnormal flow detection method based on the flow mirror image is characterized by comprising the following steps:
the method comprises the steps of acquiring the mirror image flow of the cloud IDC in real time by adopting a flow mirror image method, and distributing the acquired mirror image flow according to tenants to obtain the respective mirror image flow of each tenant;
performing feature extraction on the mirror image flow of each tenant to obtain and store feature data of the mirror image flow;
and for each tenant, judging whether the current mirror flow is abnormal flow or not by combining a Poisson distribution and outlier detection method according to the characteristic data of the current mirror flow and the characteristic data of the historical mirror flow.
2. The abnormal traffic detection method based on traffic mirroring according to claim 1, wherein the outlier detection method comprises one or more of DBSCAN, SVM, logistic regression, linear fitting.
3. The abnormal traffic detection method based on traffic mirroring according to claim 1, wherein the step of determining whether the current mirrored traffic is the abnormal traffic by combining a poisson distribution and outlier detection method comprises:
judging whether the characteristic data of the historical mirror image flow of the tenant corresponds to a stable service type or not according to the characteristic data of the historical mirror image flow of the tenant;
under the condition that the tenant is judged to correspond to the stable service type, a chi-square calibration method and a least square method are adopted to carry out linear fitting to obtain a characteristic predicted value corresponding to the current mirror flow of the tenant, and whether the current mirror flow is abnormal flow is determined according to the difference between the characteristic predicted value corresponding to the current mirror flow of the tenant and actual characteristic data.
4. The abnormal traffic detection method based on traffic mirroring according to claim 3, wherein the step of determining whether the current mirrored traffic is the abnormal traffic by combining the Poisson distribution and outlier detection method further comprises:
determining historical mirror flow in a preset time period before the current mirror flow, obtaining a corresponding characteristic predicted value of the current mirror flow by using the characteristic data of the historical mirror flow in the preset time period and adopting a logistic regression method, and determining whether the current mirror flow is abnormal flow according to the difference between the characteristic predicted value corresponding to the current mirror flow of the tenant and the actual characteristic data.
5. The abnormal traffic detection method based on traffic mirroring according to any one of claims 1 to 4, wherein the step of performing feature extraction on the mirrored traffic of each tenant comprises:
for the mirror traffic of each tenant, a plurality of features of the mirror traffic are extracted, and at least part of the features are selected from the plurality of features as a feature extraction result by calculating correlations between the plurality of features.
6. The abnormal traffic detection method based on traffic mirroring according to any one of claims 1 to 4, characterized in that the characteristic data of the mirrored traffic comprises at least part of the following characteristics:
a current flow value; network protocol data; application layer data; a transmission rate; a packet forwarding rate; source IP distribution; IP characteristics; a protocol of the load; the number of connections; URL access frequency.
7. The abnormal traffic detection method based on traffic mirroring according to any one of claims 1 to 4, characterized in that the abnormal traffic detection method further comprises:
for each tenant, judging whether the tenant meets a preset condition or not based on the characteristic data of the historical mirror image flow of the tenant, and judging that the tenant has service abnormality under the condition that the tenant meets the preset condition;
the preset conditions are as follows:
the number of connections, URL access frequency and traffic of the tenant are increased, and the source IP reputation value of the tenant is higher than a preset value.
8. An abnormal flow rate detection device based on a flow rate mirror image, characterized in that the abnormal flow rate detection device comprises:
the flow mirroring unit is used for acquiring the mirroring flow of the cloud IDC in real time by adopting a flow mirroring method;
the flow distribution unit is used for distributing the acquired mirror image flow according to the tenants to obtain the respective mirror image flow of each tenant;
the characteristic extraction unit is used for extracting the characteristics of the mirror image flow of each tenant to obtain and store the characteristic data of the mirror image flow;
and the anomaly detection unit is used for judging whether the current mirror flow is the abnormal flow or not according to the feature data of the current mirror flow and the feature data of the historical mirror flow of each tenant by combining a Poisson distribution and outlier detection method.
9. The abnormal traffic detection device based on traffic mirroring according to claim 8, wherein the outlier detection method adopted by the abnormal detection unit comprises one or more of DBSCAN, SVM, logistic regression, and linear fitting.
10. The abnormal flow detection device based on the flow mirror according to claim 8 or 9, wherein the abnormal detection unit is further configured to:
for each tenant, judging whether the tenant meets a preset condition or not based on the characteristic data of the historical mirror image flow of the tenant, and judging that the tenant has service abnormality under the condition that the tenant meets the preset condition; the preset conditions are as follows: the number of connections, URL access frequency and traffic of the tenant are increased, and the source IP reputation value of the tenant is higher than a preset value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611012012.1A CN108076019B (en) | 2016-11-17 | 2016-11-17 | Abnormal flow detection method and device based on flow mirror image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611012012.1A CN108076019B (en) | 2016-11-17 | 2016-11-17 | Abnormal flow detection method and device based on flow mirror image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108076019A CN108076019A (en) | 2018-05-25 |
| CN108076019B true CN108076019B (en) | 2021-04-09 |
Family
ID=62163448
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611012012.1A Active CN108076019B (en) | 2016-11-17 | 2016-11-17 | Abnormal flow detection method and device based on flow mirror image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108076019B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108880931B (en) * | 2018-05-29 | 2020-10-30 | 北京百度网讯科技有限公司 | Method and apparatus for outputting information |
| CN108429651B (en) * | 2018-06-06 | 2022-02-25 | 腾讯科技（深圳）有限公司 | Flow data detection method and device, electronic equipment and computer readable medium |
| CN109245913A (en) * | 2018-08-02 | 2019-01-18 | 浙江口碑网络技术有限公司 | A kind of cloud Physical Examination System and method |
| CN109040127B (en) * | 2018-09-18 | 2020-11-03 | 中国人民解放军战略支援部队信息工程大学 | Detection device and method for Diameter flooding attack |
| TWI704784B (en) | 2018-12-25 | 2020-09-11 | 安華聯網科技股份有限公司 | Device, method and non-transitory tangible machine-readable medium for traffic monitoring |
| CN109889552A (en) * | 2019-04-18 | 2019-06-14 | 南瑞集团有限公司 | Power marketing terminal abnormal flux monitoring method, system and Electric Power Marketing System |
| CN110209560B (en) * | 2019-05-09 | 2023-05-12 | 北京百度网讯科技有限公司 | Data anomaly detection method and detection device |
| CN110602046B (en) * | 2019-08-13 | 2022-04-26 | 未鲲(上海)科技服务有限公司 | Data monitoring processing method and device, computer equipment and storage medium |
| CN110519290B (en) * | 2019-09-03 | 2021-01-26 | 南京中孚信息技术有限公司 | Abnormal flow detection method and device and electronic equipment |
| CN111049762A (en) * | 2019-12-23 | 2020-04-21 | 上海金仕达软件科技有限公司 | Data acquisition method and device, storage medium and switch |
| CN111555919B (en) * | 2020-04-28 | 2022-12-09 | 山石网科通信技术股份有限公司 | Method and device for forwarding flow based on virtualized cloud platform and storage medium |
| CN111884881B (en) * | 2020-07-28 | 2022-02-18 | 苏州浪潮智能科技有限公司 | Monitoring method, device and system for Ethernet switching network and switch |
| CN112291276B (en) * | 2020-12-28 | 2021-03-23 | 金锐同创(北京)科技股份有限公司 | Flow alarm method and device and electronic equipment |
| CN113114691B (en) * | 2021-04-15 | 2022-02-22 | 西安交通大学 | Network intrusion detection method, system, equipment and readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101431449A (en) * | 2008-11-04 | 2009-05-13 | 中国科学院计算技术研究所 | Network flux cleaning system |
| CN103441982A (en) * | 2013-06-24 | 2013-12-11 | 杭州师范大学 | Intrusion alarm analyzing method based on relative entropy |
| CN103973481A (en) * | 2014-04-21 | 2014-08-06 | 蓝盾信息安全技术股份有限公司 | System and method for auditing cloud computing data center based on SDN |
| CN105007282A (en) * | 2015-08-10 | 2015-10-28 | 济南大学 | Malicious software network behavior detection method specific to network service provider and system thereof |
| KR101623071B1 (en) * | 2015-01-28 | 2016-05-31 | 한국인터넷진흥원 | System for detecting attack suspected anomal event |
| CN105871847A (en) * | 2016-04-01 | 2016-08-17 | 国网江苏省电力公司电力科学研究院 | Intelligent substation network abnormal flow detection method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8965334B2 (en) * | 2005-12-19 | 2015-02-24 | Alcatel Lucent | Methods and devices for defending a 3G wireless network against malicious attacks |
| WO2014006692A1 (en) * | 2012-07-03 | 2014-01-09 | 富士通株式会社 | Program for identifying flow to be controlled, method for identifying flow to be controlled, and device for identifying flow to be controlled |
-
2016
- 2016-11-17 CN CN201611012012.1A patent/CN108076019B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101431449A (en) * | 2008-11-04 | 2009-05-13 | 中国科学院计算技术研究所 | Network flux cleaning system |
| CN103441982A (en) * | 2013-06-24 | 2013-12-11 | 杭州师范大学 | Intrusion alarm analyzing method based on relative entropy |
| CN103973481A (en) * | 2014-04-21 | 2014-08-06 | 蓝盾信息安全技术股份有限公司 | System and method for auditing cloud computing data center based on SDN |
| KR101623071B1 (en) * | 2015-01-28 | 2016-05-31 | 한국인터넷진흥원 | System for detecting attack suspected anomal event |
| CN105007282A (en) * | 2015-08-10 | 2015-10-28 | 济南大学 | Malicious software network behavior detection method specific to network service provider and system thereof |
| CN105871847A (en) * | 2016-04-01 | 2016-08-17 | 国网江苏省电力公司电力科学研究院 | Intelligent substation network abnormal flow detection method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108076019A (en) | 2018-05-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108076019B (en) | Abnormal flow detection method and device based on flow mirror image | |
| US11818025B2 (en) | Methods, systems, and apparatus to generate information transmission performance alerts | |
| US10505819B2 (en) | Method and apparatus for computing cell density based rareness for use in anomaly detection | |
| US8954550B2 (en) | Service dependency discovery in enterprise networks | |
| US10027694B1 (en) | Detecting denial of service attacks on communication networks | |
| CN108965347B (en) | Distributed denial of service attack detection method, device and server | |
| US10355949B2 (en) | Behavioral network intelligence system and method thereof | |
| CN111277570A (en) | Data security monitoring method and device, electronic equipment and readable medium | |
| CN111935170A (en) | Network abnormal flow detection method, device and equipment | |
| US11621971B2 (en) | Low-complexity detection of potential network anomalies using intermediate-stage processing | |
| US10742672B2 (en) | Comparing metrics from different data flows to detect flaws in network data collection for anomaly detection | |
| US20080186876A1 (en) | Method for classifying applications and detecting network abnormality by statistical information of packets and apparatus therefor | |
| CN112583797B (en) | Multi-protocol data processing method, device, equipment and computer readable storage medium | |
| WO2017000536A1 (en) | Bfd method and apparatus | |
| WO2017039506A1 (en) | Method and network node for localizing a fault causing performance degradation of service | |
| CN115348092A (en) | Industrial control network abnormal flow detection method and device and electronic equipment | |
| KR20180130295A (en) | Apparatus for predicting failure of communication network and method thereof | |
| US11711310B2 (en) | System and method for determining a network performance property in at least one network | |
| US10834110B1 (en) | Methods for preventing DDoS attack based on adaptive self learning of session and transport layers and devices thereof | |
| CN110995606B (en) | Congestion analysis method and device | |
| EP4084410A1 (en) | Method, apparatus and system for training fault detection model | |
| EP4084408A1 (en) | Fault detection method, apparatus and system | |
| CN111510443B (en) | Terminal monitoring method and terminal monitoring device based on equipment portrait | |
| US20220353169A1 (en) | Method for detecting systematic communications in a communications network, corresponding device, and computer program product | |
| CN117768165A (en) | Network anomaly detection method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |