TW202017337A - Method and system for backbone network flow anomaly detection - Google Patents

Abstract

The present invention discloses a method for backbone network flow anomaly detection, which includes the steps of: using the source Internet Protocol (IP) address to filter the network flow in a backbone network; evenly distributing the filtered network flow into a plurality of working nodes of a distributed big data processing system; performing a parallel processing on the plurality of working nodes, and generating a plurality of flow feature data sets on each of the working nodes; using a plurality of abnormal flow identification models and the flow feature data sets to determine whether the network flow is abnormal or not on each of the working nodes; and, when determining that the network flow is abnormal, using a plurality of attack-type identification models and the flow feature data sets to determine the attack type and generate an analysis result.

Description

Backbone network abnormal flow detection method and system

The invention discloses a method and system for detecting network traffic, in particular to a method and system for detecting abnormal traffic in a backbone network.

With the development of the Internet, network traffic has grown rapidly, and the Internet has become an indispensable information carrier. At the same time, network traffic often has abnormal traffic that deviates from the normal range, mainly spread by worms. (Worms), distributed denial of service (DDoS) attacks, botnet (Botnet) and other malicious network attacks, as well as network configuration errors or occasional line interruptions. These abnormal traffic will often lead to a sharp decline in the quality of the entire network service, directly paralyzing the victim host and the network. Therefore, how to perform network anomaly detection and provide warning information in a backbone network environment is of great significance to ensure the normal operation of the backbone network.

At the same time, with the continuous improvement of network bandwidth, network traffic anomaly detection faces new problems: on the one hand, the network transmission rate has been greatly improved. The same network attack is very obvious in the regional network, but in the backbone It may not be easy to find on the network, and requires a high-accuracy network traffic anomaly detection method. On the other hand, the increase in network bandwidth also speeds up the speed of network attacks. Take the outbreak of network worms as an example It can infect most vulnerable hosts in the Internet in a shorter period of time, which requires the anomaly detection system to identify anomalous traffic more quickly and efficiently, so that subsequent blocking can be implemented in real time.

In addition, the current method for detecting backbone network traffic is usually to directly analyze the original network traffic. To quickly and efficiently identify abnormal traffic, it will require a lot of bandwidth and a lot of computing resources, which will increase computing. Hardware costs.

In addition, there are currently load balance switches (Load Balance Switch) to split the backbone network traffic for abnormal traffic detection, but because the load balance switch is only a general Netcom device, it is limited by the computing power of the device and cannot Simultaneously perform multiple calculation tasks (that is, parallel processing), so it is not possible to efficiently identify abnormal flows.

Therefore, the conventional technology needs an improved solution for improving the abnormal traffic detection performance of the backbone network, which can identify abnormal traffic more quickly and efficiently in response to the bandwidth requirements of the backbone network.

The purpose of the present invention is to provide a method and system for detecting backbone network traffic, which can respond to the bandwidth requirements of the backbone network and utilize the concepts of "layering" and "distribution" and the two stages of "offline" and "real-time" The process achieves rapid and efficient identification of abnormal traffic.

In order to achieve the purpose of the invention, the present invention discloses a method for detecting abnormal traffic in a backbone network, which includes the following steps: filtering the network traffic in the backbone network by using the source Internet Protocol (IP) address; the filtered network Road traffic is equally divided into the plural working nodes of the distributed big data processing system; parallel processing is performed on the aforementioned plural working nodes, and a complex flow characteristic data set is generated on each of the aforementioned working nodes; on each of the aforementioned working nodes, the plural anomalies are used The traffic identification model and the above-mentioned traffic characteristic data set to determine whether the network traffic is abnormal; and when the network traffic is determined to be abnormal, the multiple attack type identification model and the above-mentioned traffic characteristic data set are used to determine the attack type and Produce analysis results.

In a specific embodiment, the backbone network abnormal traffic detection method further includes sending the analysis result to an analysis database for storage and display on the display interface.

In a specific embodiment, the step of using the IP address to filter the network traffic in the backbone network further includes: creating a white list and a black and white list, wherein the black and white list stores a plurality of abnormal source IP addresses, the A plurality of trusted source IP addresses are stored in the white list; determine whether a source IP address of a packet in the backbone network is in the white list or the black and white list; and when the source IP address of the packet is in the white When the list or the black-and-white list is included, the above packet is discarded.

In a specific embodiment, when it is determined that the network traffic is abnormal, the analysis result is sent to a large number of abnormal traffic analysis modules; analyzing the network traffic to obtain the source IP address of the network traffic; and Add the source IP address to the blacklist.

In a specific embodiment, the step of distributing the filtered network traffic to the above-mentioned working nodes of the distributed big data processing system further includes: distributing the filtered network traffic to the Apache storm (Apache Storm) In the plural working nodes of the system, the traffic transmission statistics are output after conversion processing.

In a specific embodiment, the step of generating the flow characteristic data set at each of the working nodes further includes: analyzing the flow transmission statistical data using a flow characteristic algorithm to generate the flow characteristic data set.

In a specific embodiment, the aforementioned flow characteristic data set includes at least one basic flow characteristic, at least one original flow characteristic and at least one additional flow characteristic.

In a specific embodiment, the step of determining whether the network traffic is abnormal using the aforementioned abnormal traffic identification model and the traffic characteristic data set further includes: selecting at least one behavior characteristic from at least one known intrusion detection data set; A machine learning algorithm and a recognition result perform benefit analysis to generate at least one selected machine learning algorithm; and the above-mentioned behavioral characteristics and the selected machine learning algorithm are used to train the abnormal traffic identification model offline.

In a specific embodiment, the step of using the attack type identification model and the traffic feature data set to determine the attack type further includes: selecting at least one behavior feature from at least one known intrusion detection data set; learning at least one machine An algorithm and an identification result are used for benefit analysis to generate at least one selected machine learning algorithm; and the attack type identification model is trained offline with the behavior characteristics and the selected machine learning algorithm.

Accordingly, the present invention also provides a backbone network abnormal traffic detection system that executes the aforementioned backbone network abnormal traffic detection method.

These and other viewpoints and embodiments will become clear to those of ordinary skill in the related art with reference to the subsequent detailed description and accompanying drawings.

The embodiments will now be described in detail with reference to the accompanying drawings of the present invention. In the accompanying drawings, the same and/or corresponding elements are denoted by the same reference symbols.

Various embodiments will be disclosed here; however, it should be understood that the disclosed embodiments are only used as examples that can be embodied in various forms. In addition, each example given in connection with the various embodiments is intended to be illustrative, not limiting. Further, the drawings do not necessarily conform to the size ratio, and some features are enlarged to show the details of specific elements (and any dimensions, materials, and similar details shown in the drawings are intended to be illustrative and not limiting) . Therefore, the specific structural and functional details disclosed herein are not to be construed as limitations, but are merely used to teach those skilled in the relevant art to implement the disclosed embodiments.

In the following detailed description of a number of example specific embodiments, reference is made to the accompanying drawings, which form part of the present invention. It is shown by way of example description, and the described specific embodiments can be implemented by this example. Provide sufficient details to enable those skilled in the art to implement the described specific embodiments, and understand that other specific embodiments can be used and other changes can be made without departing from the spirit or scope thereof. In addition, although this may be the case, reference to "one embodiment" does not need to belong to the same or singular specific embodiment. Therefore, the following detailed description does not have a limiting idea, and the scope of the specific embodiments of the description is only defined by the scope of the additional patent applications.

The first figure is a flowchart of a backbone network abnormal traffic detection method according to an embodiment of the invention, the second figure is a flowchart of an offline training abnormal traffic identification model according to an embodiment of the invention, and the third figure is based on A flowchart of an offline training attack type identification model according to an embodiment of the present invention. In the following, please refer to the first figure to the third figure to explain the abnormal traffic detection method of the backbone network according to an embodiment of the present invention.

The first figure is a flowchart of a method for detecting abnormal traffic in a backbone network according to an embodiment of the present invention. Although the steps shown in the figure are sequential, those with ordinary knowledge in the field of the present invention should be able to understand that in other implementations For example, some steps can be exchanged or executed simultaneously.

In step S102, the source Internet Protocol (IP) address is used to filter the network traffic in the backbone network. In this embodiment, the pre-established blacklist (not shown) and whitelist (not shown) are used to filter the network traffic in the backbone network as the first layer filtering mechanism, and the known abnormal traffic source IP is stored in the blacklist Address, the trusted source IP address is stored in the white list. When a source IP address has a large amount of abnormal traffic in a short time, the source IP address will be stored in the black list, and the white list is Users join in advance, usually will join the IP addresses of large network service providers (such as: Google, Facebook, YouTube, etc.), when the packets from the IP addresses in the blacklist or whitelist arrive, they will be discarded directly .

In step S104, the filtered network traffic is evenly distributed to a plurality of working nodes of the distributed big data processing system. In this embodiment, the filtered network traffic is evenly distributed to the working nodes of the decentralized big data processing system as the first layer of distribution mechanism, and the decentralized big data processing system is the Apache Storm system. Each working node is a server with parallel processing capability. Each server can be a physical server or a virtual machine (Virtual Machine). In this embodiment, instead of sending the original network traffic to the plural working nodes, the converted traffic transmission statistics are sent to the plural working nodes, so a lot of bandwidth will be saved, and the specific implementation method It can be realized by a conventional packet parser, for example: the load balance setting of the built-in "PacketX Grism" product of Rockwell Digital Co., Ltd., and if it is realized by this product, the traffic sent to multiple working nodes The format of the transmission statistics is Cisco Netflow V9.

In step S106, parallel processing is performed on the plural working nodes, and plural flow characteristic data sets (Datasets) are generated on each of the working nodes. In this embodiment, parallel processing is performed on each working node and a flow characteristic data set is generated on each working node as a second-level distribution mechanism and data preprocessing program. The data format sent to each working node will be Cisco Netflow V9. This format already includes the basic flow characteristics. Then, the flow characteristics calculation algorithm is used to analyze the flow transmission statistics to generate the above flow characteristic data set. For example, use Conventional network traffic monitors: For example, the tool provided by Argus converts network traffic to Argus traffic, and then obtains the original traffic characteristics, and then generates it according to the algorithm defined by UNSW-NB15 (University of New South Wale Network Based 2015) Additional flow characteristics. The flow characteristics data set includes basic flow characteristics, original flow characteristics, and additional flow characteristics. It should be noted that since UNSW-NB15 is an intrusion detection data set released by the Australian Defense College (ADFA) at the University of New South Wales (UNSW) in 2015, it is currently the most widely used intrusion detection data set. Therefore, detailed implementation will be omitted.

In step S108, at each of the working nodes, a plurality of abnormal traffic identification models and the above-mentioned traffic characteristic data set are used to determine whether the network traffic is abnormal. In this embodiment, the offline abnormal multiple traffic identification model and the above-mentioned traffic feature data set are used to determine whether the network traffic is abnormal is the second layer filtering mechanism. In order to improve the overall analysis performance, the offline trained abnormal The traffic identification model identifies whether the traffic has attack behavior characteristics. When the attack behavior characteristics appear, it proceeds to step S110. When it is determined that the network traffic is normal, the process of the backbone network abnormal traffic detection method of the present invention will be ended (not Icon).

In this embodiment, an abnormal traffic identification model will be pre-established in an offline manner. Please also refer to the second figure, which is a flowchart of training an abnormal traffic identification model offline according to an embodiment of the present invention.

In step S202, at least one behavior feature is selected from at least one known intrusion detection data set. In the conventional technology, using all the features of the data set to train the machine learning knowledge model does not necessarily guarantee the best performance (performance), and will increase the calculation cost and increase the recognition error rate. In the example, feature selection will be done first. Its purpose is to improve the classification speed of machine learning without losing accuracy. The feature selection methods used include the following:

CfsSubsetEval: Generates a set of features that have a high correlation with the class (Class) but low correlation between features.

CorrelationAttributeEval: Calculate the correlation between category and feature (Feature), the value range is 1 to -1.

InfoGainAttributeEval: Based on the entropy (Entropy) as the benchmark, the information gain (Information Gain) is calculated. The larger the value, the better this feature is used to classify the data.

GainRatioAttributeEval: According to the information gain and split information, the value of Gain Ratio is calculated. The larger the value, the more important this feature is.

OneRAttributeEval: According to a classifier called OneR (Classifier) method, calculate the feature false positive rate, the lower the false positive rate, the better.

ReliefFAttributeEval: Calculate feature weights based on near hits and near misses. The nearest neighbor in the guess refers to the closest value of the same category, and the wrong neighbor is the closest value in different categories.

SymmetricalUncertAttributeEval: measures the relationship between categories and features.

WrapperSubsetEval: Set the classification method, filter the features to select a set of feature sets, and bring in the classification to confirm whether the feature set is the best combination, otherwise re-filter the features.

In step S204, a benefit analysis is performed on at least one machine learning algorithm and the recognition result to generate at least one selected machine learning algorithm. In this embodiment, in order to train the machine, the pre-collected packet data samples with attack behaviors are used as training data (Training Data), and the features of the data are extracted from the training data to help the system interpret the target, for example: source IP , Use agreement, etc., and then tell the machine the answer corresponding to each attack, the data label (Label) of the packet with the attack behavior is 1, the general packet label is 0, thus letting the machine know that those packets have the attack behavior, those There is no packet. As the amount of training data becomes larger, when a new piece of data is entered into the machine, for example, the feature agreement is User Datagram Protocol (UDP), the system will determine whether the packet has attack behavior or the probability of attack behavior. Since this method tells the machine that the above-mentioned "labeled" data is in the training process of the machine, this is "supervised learning".

In this embodiment, "Unsupervised Learning" can also be used. There is no standard answer to the training data and there is no need to input tags in advance. The machine does not know whether the classification result is correct when learning. During training, you only need to provide input examples to the machine, and it will automatically find potential rules from these examples.

In this embodiment, the machine learning algorithm used includes the following:

Bayesian network learning method (BayesNet): Bayesian network learning method uses various search algorithms and quality measurement methods, based on Bayesian network classifier, provides data structure (network structure, conditional probability distribution, etc.) and Use common tools for Bayesian network learning algorithms.

Simple Bayesian model (NaiveBayes): The simple Bayesian model directly assumes that all random variables are conditionally independent, so you can directly use the method of conditional probability multiplication to calculate the joint probability distribution. p(X|C) = P(X1|C)P(X2|C)...P(Xd|C), where X=[X1,X2,...,Xd] is a feature vector, and C Represents a specific category. The simple naive Bayes classifier produced by this assumption is quite practical, and its recognition performance is often not lost to other more complex classifiers.

JRip taxonomy: This taxonomy implements the rule-based classifier (Rule-Based Classifier), which mainly uses the "If...Then" method to classify records. The classification method is proposed by William W. Cohen, and it is pruned by repeated increments to reduce errors.

PART taxonomy: This taxonomy implements the rule-based classifier (Rule-Based Classifier), unique to Weka (Weka is a software based on Java-based data exploration and machine learning), and constructs a partial C4.5 by means of individual breakdown Decision tree, and make the best leaves into rules.

J48 algorithm: This algorithm is a decision tree using C4.5, and its core algorithm is the ID3 algorithm, which improves ID3, uses the information gain ratio (Gain Ratio) to select attributes, and overcomes the bias selection when using information gain to select attributes Insufficient attributes with many values. In Weka, you can set parameters to use pruned or unpruned decision trees. If you choose pruning, the default is Pessimistic Error Pruning (PER). Use error 率來进行 pruning. This algorithm first determines this The empirical error rate of the leaves is (E+0.5)/N, and 0.5 is an adjustment factor. For a subtree with L leaves, the error number and the number of instances of the subtree are the result of the sum of the number of leaf errors and the number of instances. J48 also presets to use the Subtree Raising method by selecting a subtree, raising its hierarchy to replace and replace the root of the subtree with its internal node or 葉node. It can also be changed to Subtree Replacement by setting parameters by selecting a subtree and replacing it with a single tree 葉來.

Random Tree (Random Tree): This algorithm is unique to Weka. Some of the other libraries are called random trees. The algorithm is actually a random forest, but Weka is different. This refers to a tree with randomly selected attributes. .

Random Forest (Random Forest): This algorithm will extract some features (Feature) and some data from the training data to generate a tree (usually using the CART algorithm), repeatedly constructing a few trees without pruning After random trees, each tree is predicted, and finally each prediction result is voted. The one with the most votes is the prediction result of the entire forest. In Weka, several random trees are constructed to build a forest and make predictions.

In step S206, the abnormal traffic identification model is trained offline with the behavior characteristics and the selected machine learning algorithm.

Returning to the first figure, in step S108, the abnormal traffic identification model trained using the above method determines whether the network traffic is abnormal. When it is determined that the network traffic is abnormal, the process proceeds to step S110.

In step S110, when it is determined that the network traffic is abnormal, the multiple attack type identification model and the aforementioned traffic feature data set are used to determine the attack type and generate an analysis result. In this embodiment, the attack type identification model and the traffic characteristic data set trained offline are used to determine the attack type and generate the analysis result as the third layer filtering mechanism. After identifying the attack type, the analysis results can be sent to the analysis database for storage and graphically displayed in the display interface to display the analysis results and issue real-time warning information. After identifying the attack type, the analysis results can also be sent to a large number of abnormal traffic analysis modules to analyze the network traffic to obtain the source IP address of the network traffic, and the source IP address is added to the blacklist to update the black real-time List.

In this embodiment, an attack type identification model is pre-established in an offline manner. Please also refer to the third figure. The third figure is a flowchart of training an attack type identification model according to an embodiment of the present invention.

In step S302, at least one behavior feature is selected from at least one known intrusion detection data set. In this embodiment, for the feature selection method, please refer to the description of step S202 in the second figure above.

In step S304, at least one machine learning algorithm and a recognition result are analyzed for benefit to generate at least one selected machine learning algorithm. In this embodiment, please refer to the description of step S204 in the second figure above for the machine learning algorithm used.

In step S306, the attack type identification model is trained offline with the behavior characteristics and the selected machine learning algorithm.

The fourth figure is a schematic diagram of a specific structure of a backbone network abnormal traffic detection system according to an embodiment of the present invention. As shown in the fourth figure, the backbone network abnormal traffic detection system 400 includes a source filtering module 410, a data distribution module 420, a distributed big data processing system 430, a large number of abnormal traffic analysis modules 480, and an analysis database 490. The decentralized big data processing system 430 includes n working nodes 432_1~432_n, and m flow characteristic processing modules 440_1~440_m, m abnormal flow recognition modules 450_1~450_m and attack types are arranged in the working node 432_1 Identification modules 460_1~460_m, where n is a natural number of at least 1 and m is a natural number of at least 2.

The source filtering module 410 uses the source Internet Protocol (IP) address to filter the network traffic in the backbone network. In this embodiment, the source filtering module 410 performs a first-level filtering mechanism, pre-establishing a black list (not shown) and a white list (not shown) in the source filtering module 410, and the known exceptions are stored in the black list Traffic source IP address, the trusted source IP address is stored in the white list. When a source IP address has a large amount of abnormal traffic in a short time, the source IP address will be stored in the black list, and The white list is manually added by the system administrator, and usually will be added to the IP addresses of large network service providers (for example: Google, Facebook, YouTube, etc.). When the packets from the IP addresses in the black list or white list reach the source filtering module In group 410, the source filtering module 410 will directly discard these packets.

In this embodiment, the source filter module 410 can be implemented using a general packet parser, for example: the "PacketX Grism" product of Realtek Digital Co., Ltd. The white list can be easily changed using the built-in settings of the product Trust source IP address is added, and the blacklist uses the application provided by the product to connect with the large number of abnormal source modules 480 of the present invention. When a large number of abnormal behavior packets occur, they can be blocked at the first time to avoid End device is attacked.

The data distribution module 420 divides the network traffic filtered by the source filtering module 410 into n working nodes 432_1~432_n of the distributed big data processing system 430. In this embodiment, the data distribution module 420 performs With a layered distribution mechanism, the data distribution module 420 outputs to each working node 432_1~432_n the converted traffic transmission statistics, so a large amount of bandwidth will be saved.

In this embodiment, the specific implementation of the data distribution module 420 can be implemented using a general packet parser, for example: the load balancing setting built into the "PacketX Grism" product of Realtek Digital Co., Ltd., data distribution mode The output format of group 420 is Cisco Netflow V9.

Each working node 432_1~432_n is a server with parallel processing capability. Each server can be a physical server or a virtual machine. Since each working node 432_1~432_n is a server with parallel processing capabilities, a plurality of traffic feature processing modules, abnormal traffic identification modules, and attack type defense can be configured on each working node 432_1~432_n Recognition module. In this embodiment, m working flow nodes 432_1 are equipped with m flow characteristic processing modules 440_1~440_m, m abnormal flow recognition modules 450_1~450_m and attack type recognition modules 460_1~460_m as Examples.

The flow characteristic processing modules 440_1~440_m perform parallel processing on the working node 432_1 and generate flow characteristic data sets respectively. In this embodiment, the traffic feature processing modules 440_1~440_m perform the second layer of traffic distribution mechanism and data preprocessing procedures. The format of the data traffic distribution module 420 output is Cisco Netflow V9. This format already includes the basic traffic characteristics, and then , And then use the traffic characteristic algorithm to analyze the traffic transmission statistics to generate the above traffic characteristic data set. For example, using a conventional network traffic monitor such as: Argus provides tools to convert network traffic to Argus traffic, first Obtain the original flow characteristics, and then generate additional flow characteristics according to the algorithm defined by UNSW-NB15. The flow characteristic data set includes basic flow characteristics, original flow characteristics, and additional flow characteristics.

The abnormal traffic identification modules 450_1~450_m use a plurality of abnormal traffic identification models trained offline and the above-mentioned traffic characteristic data set to determine whether the network traffic is abnormal. In this embodiment, the abnormal traffic identification modules 450_1~450_m perform the second layer filtering mechanism. For the overall analysis performance of the system, the abnormal traffic identification modules 450_1~450_m only identify whether the traffic has attack behavior characteristics. Only when the traffic is sent to the next layer for attack type identification, if the identification result is normal, the analysis results 474_1~474_m will be output.

The attack type identification modules 460_1~460_m use the offline multiple attack type identification model and the traffic feature data set to determine the attack type and generate analysis results 472_1~472_m. In this embodiment, the attack type identification modules 460_1~460_m perform a third layer filtering mechanism. If the traffic is determined to have an attack behavior, the attack type identification modules 460_1~460_m identify the attack type and generate an analysis result 472_1 ~472_m.

In this embodiment, the analysis results 470 of all working nodes 432_1~432_n, including the analysis results 472_1~472_m and the analysis results 474_1~474_m, can be sent to the analysis database 490 for storage and graphically displayed in the display interface (Not shown). In this embodiment, when the analysis results 472_1~472_m are displayed in the display interface, real-time warning information is also issued at the same time. In this embodiment, the analysis results 470 of all working nodes 432_1~432_n can also be sent to a large number of abnormal traffic analysis modules 480 to analyze the network traffic to obtain the source IP address of the network traffic, and the source IP address Join the blacklist to update the blacklist instantly.

In this embodiment, the decentralized big data processing system 430 is a decentralized real-time computing system, so the Apache storm system will be used as an embodiment of the decentralized big data processing system 430 in more detail below. Please also refer to the Figure 5 is a schematic diagram of the Apache storm system architecture.

As shown in the fifth figure, Apache Storm System 500 is a decentralized, reliable, and fault-tolerant system and processes a large amount of data in a stream. It is currently widely used in real-time data analysis or processing. Apache Storm System 500 There are three types of nodes: working nodes 432_1~432_n, master node 510, and temporary nodes 520_1~520_t, where t is a natural number of at least 3.

The master node 510, also known as Nimbus, is mainly responsible for managing, coordinating, and monitoring the topology running in the entire system, including topology deployment, task allocation, and task redistribution in the event of a failure.

Temporary nodes 520_1~520_t are also known as ZooKeeper. In conventional distributed applications, various workflows need to coordinate with each other and share some information. Temporary nodes 520_1~520_t play the role of master node 510 and working nodes 432_1~432_n Communication bridge. The master node 510 and the working nodes 432_1~432_n store all the data in the temporary nodes 520_1~520_t. If the master node 510 and the working nodes 432_1~432_n are terminated abruptly, it will not affect the operation of the entire system.

Work nodes 432_1~432_n are also called Supervisors. Each work node 432_1~432_n has a work flow, which is mainly responsible for creating, starting, and stopping the work flow to perform the assigned tasks.

The sixth figure is a flowchart of the Apache storm system of the fifth figure for the backbone network abnormal traffic detection method. In this embodiment, in order to use real-time computing on the Apache Storm System 500, a topology needs to be established and deployed on the cluster to achieve real-time data processing. The topology consists of the root node (Spout) 600 and the child nodes (Bolt ) Nodes 610_1~610_6 are formed, and data is transferred between the root node 600 and the child nodes 610_1~610_6 through a data structure such as a tuple.

The root node 600 mainly receives the network traffic represented by the traffic characteristic data set, and flows the network traffic to the child node 610_1 in a value group format.

The child node 610_1 starts multiple traffic abnormality identification models generated offline to identify network traffic behavior in real time. If the model considers the network traffic to be normal, it is marked as 0, and if it is abnormal, it is marked as 1. The network represented by the flow characteristic data set The identification results of the road flow and multiple flow anomaly identification models flow to the child node 610_2 in the value group format.

The child node 610_2 integrates the recognition results of multiple traffic anomaly recognition models. If any recognition result is abnormal, the network traffic represented by the traffic characteristic data set flows to the child node 610_3 in the value group format. If multiple recognition results are normal , The final result is marked as 0, and the network traffic represented by the traffic characteristic data set, multiple recognition results, and the final recognition result are flowed to the child node 610_4 in a value group format.

The child node 610_3 starts multiple attack type identification models generated offline to identify the type of network attack. If the model considers that the network traffic is not the attack type of the model itself, it is marked as 0, and if it is the model itself attack type, it is marked as 1, and The network traffic represented by the traffic characteristic data set and the multiple recognition results flow to the child node 610_5 in a value group format.

The child node 610_5 integrates multiple recognition results. If any recognition result is abnormal, the final recognition result is marked as 1, if each recognition result is normal, the final result is marked as 0. Finally, the network traffic represented by the traffic characteristic data set, multiple recognition results, and the final recognition result flow to the child node 610_6 in the value group format.

The child node 610_4 records the data transmitted by the child node 610_2 in the form of log (Log), that is, the identification result of the abnormal traffic, and generates a log file at the rate of every second.

The child node 610_6 records the data transmitted by the child node 610_5 as a result of attack type identification in a log (Log) mode, and generates a log record file at a rate of every second.

The seventh figure is a schematic diagram of a backbone network abnormal traffic detection system according to another embodiment of the present invention. As shown in the figure, in this embodiment, the backbone network abnormal traffic detection system 700 is divided into an offline part 702 and a real-time part 704.

In the offline part 702, a plurality of abnormal traffic identification models 706 and attack type identification models 708 are pre-established. In this embodiment, multiple abnormal traffic identification models 706 and attack type identification models 708 are established using the processes in the second and third figures, respectively. For the method used, please refer to the descriptions in the second and third figures, respectively , This content will not repeat.

In the real-time part 704, the source filtering module 710 uses the source internet protocol (IP) address to filter the network traffic in the backbone network, and the source filtering module 710 performs the first layer filtering mechanism to extract the real network Traffic and filter known harmless network traffic to reduce the load of subsequent intrusion detection

The data distribution module 720 divides the network traffic filtered by the source filtering module 710 into n working nodes 732_1~732_n of the distributed big data processing system 730. The data distribution module 720 performs the first layer distribution mechanism. It will be divided into a stream type that conforms to the behavior characteristics of the abnormal traffic identification model and the attack type identification model, and then performs distributed abnormal behavior detection on each working node 732_1~732_n.

Each working node 732_1~732_n performs parallel processing, that is, the second-layer diversion mechanism, and quickly distinguishes normal and abnormal network traffic at the same time according to multiple abnormal traffic identification models 706 and multiple attack type identification models 708 established in advance And generate analysis results 740 to warn about abnormal network traffic. In addition, in this embodiment, the working nodes 732_1 to 732_n and the working node 432_1 of the fourth figure have the same working content. Please refer to the related description of the fourth figure, which will not be repeated here.

In summary, the feature of the present invention is to provide a backbone network traffic detection method and system, which can respond to the bandwidth requirements of the backbone network, and utilize the concepts of "layering" and "shunting" and "offline" With the "real-time" two-stage processing, abnormal and rapid traffic can be identified quickly and efficiently. Based on this, the performance of the detection method and system is greatly improved. In addition, the present invention is divided into two parts: offline and real-time. It first captures real traffic, uses the first layer filtering mechanism to filter known harmless network traffic to reduce the load of subsequent intrusion detection, and then conforms to intrusion detection. Streaming patterns that measure the behavior characteristics of the knowledge base to perform distributed anomaly behavior detection. Each anomaly behavior classifier quickly distinguishes normal and abnormal network traffic based on the intrusion detection knowledge base loaded in advance. Warn about abnormal network traffic.

S102: Step S104: Step S106: Step S108: Step S110: Step S202: Step S204: Step S206: Step S302: Step S304: Step S306: Step 400: backbone network abnormal traffic detection system 410: Source filter module 420: data distribution module 430: Decentralized big data processing system 432_1 to 432_n: working node 440_1 to 440_m: flow characteristic processing module 450_1 to 450_m: abnormal flow identification module 460_1 to 460_m: Attack type identification module 470: Analysis results 472_1 to 472_m: analysis results 474_1 to 474_m: analysis results 480: A large number of abnormal traffic analysis modules 490: Analysis database 500: Apache Storm System 510: master node 520_1 to 520_t: temporary node 600: root node 610_1 to 610_6: child nodes 700: Backbone network abnormal traffic detection system 702: Offline part 704: real-time part 706: Multiple abnormal traffic identification models 708: Multiple attack type identification model 710: Source filter module 720: data distribution module 730: Decentralized big data processing system 732_1 to 732_n: working node 740: Analysis results

The invention can be further understood with reference to the following drawings and description. Non-limiting and non-exhaustive examples are described with reference to the following drawings. The components in the drawings do not have to be actual sizes; the emphasis is on explaining the structure and principles. The first figure is a flowchart of a method for detecting abnormal traffic in a backbone network according to an embodiment of the present invention. The second figure is a flowchart of an offline training abnormal traffic identification model according to an embodiment of the present invention. The third figure is a flowchart of an offline training attack type identification model according to an embodiment of the invention. The fourth figure is a schematic diagram of a specific structure of a backbone network abnormal traffic detection system according to an embodiment of the present invention. The fifth figure is a schematic structural diagram of a distributed real-time computing system according to an embodiment of the invention. Figure 6 is a flow chart of a method for detecting abnormal traffic on the backbone network using the distributed real-time computing system of Figure 5. The seventh figure is a schematic diagram of a backbone network abnormal traffic detection system according to another embodiment of the present invention.

S102: Step

S104: Step

S106: Step

S108: Step

S110: Step

Claims (10)

A method for detecting abnormal traffic in a backbone network, which includes the following steps: filtering a network traffic in a backbone network using a source Internet Protocol (IP) address; dividing the filtered network traffic into equal parts In the plural working nodes of a decentralized big data processing system; parallel processing is performed on the plural working nodes, and a complex flow characteristic data set is generated on each of the working nodes; on each of the working nodes, the plural abnormal flow identification is used The model and the above-mentioned traffic feature data set to determine whether the network traffic is abnormal; and when the network traffic is determined to be abnormal, use the multiple attack type identification model and the above-mentioned traffic feature data set to determine an attack type and generate a Analyze the results. The method for detecting abnormal traffic in the backbone network as described in the first patent application scope further includes: sending the analysis result to an analysis database for storage and display in a display interface. The method for detecting abnormal traffic in the backbone network as described in the first patent application, wherein the step of using the IP address to filter the network traffic in the backbone network further includes: creating a white list and a black and white list, wherein A plurality of abnormal source IP addresses are stored in the black and white list, and a plurality of trusted source IP addresses are stored in the white list; determine whether a source IP address of a packet in the backbone network is in the white list or the black and white list; And when the source IP address of the above packet is in the white list or the black and white list, the above packet is discarded. The method for detecting abnormal traffic in the backbone network as described in the third patent application scope further includes: when it is determined that the network traffic is abnormal, sending the analysis result to a large number of abnormal traffic analysis modules; analyzing the network traffic To obtain the source IP address of the network traffic; and add the source IP address to the blacklist. The method for detecting abnormal traffic in the backbone network as described in claim 1 of the patent scope, wherein the step of dividing the filtered network traffic into the above-mentioned working nodes of the distributed big data processing system further includes: filtering The network traffic is divided into a plurality of working nodes of an Apache Storm system, and a flow transmission statistics is output after a conversion process. The abnormal traffic detection method of the backbone network as described in claim 5 of the patent application, wherein the step of generating the traffic characteristic data set at each of the working nodes further includes: analyzing the traffic transmission statistics using a traffic characteristic algorithm to Generate the above flow characteristic data set. The abnormal traffic detection method of the backbone network as described in the first patent application, wherein the traffic characteristic data set includes at least one basic traffic characteristic, at least one original traffic characteristic, and at least one additional traffic characteristic. The method for detecting abnormal traffic in the backbone network as described in claim 1 of the patent application, wherein the step of using the abnormal traffic identification model and the flow characteristic data set to determine whether the network traffic is abnormal further includes: from at least one Knowing intrusion detection data sets select at least one behavior feature; perform benefit analysis on at least one machine learning algorithm and a recognition result to generate at least one selected machine learning algorithm; using the above behavior characteristics and the selected machine learning algorithm Method to train the above abnormal traffic identification model offline. The method for detecting abnormal traffic in the backbone network as described in the first patent application, wherein the step of using the attack type identification model and the traffic characteristic data set to determine the attack type further includes: from at least one known intrusion detection data Select at least one behavior feature; perform benefit analysis on at least one machine learning algorithm and a recognition result to generate at least one selected machine learning algorithm; train the above offline with the above behavior characteristics and the selected machine learning algorithm Attack type identification model. A backbone network abnormal traffic detection system executes the backbone network abnormal traffic detection method as described in one of the patent application items 1 to 10.

Images