CN117501655A - Anomaly detection for a network - Google Patents

Abstract

An anomaly detection device for detecting anomalies in network traffic includes a statistics generator that receives characteristics of packets in network traffic and generates statistics for the network traffic. The statistics include distribution statistics about respective distributions over time of respective characteristics of packets in the network traffic. An anomaly detection processor detects a deviation of the distribution statistics from distribution statistics of normal network traffic and detects anomalies with respect to the network traffic based on the deviation of the distribution statistics from the distribution statistics of the normal network traffic.

Description

Anomaly detection for a network

Cross Reference to Related Applications

The present application claims the benefit of U.S. provisional patent application No.63/170,944 entitled "feature extractor Novel Feature Extractor for an Ensemble of Autoencoders (novel feature extractor for auto encoder set)" filed on month 5 of 2021, and claims the benefit of U.S. provisional patent application No. 63/208,879 entitled "feature extractor Online Feature Extractor & Ensemble of Autoencoders for High-Rate Anomaly Detectionin Networking (online feature extractor and auto encoder set for high rate anomaly detection in a network)" filed on month 9 of 2021. Both applications cited above are incorporated herein by reference in their entirety for all purposes.

Technical Field

The present disclosure relates generally to network communications, and more particularly, to detecting anomalies in network traffic.

Background

Anomaly detection systems are used to detect anomalies in network traffic that may be caused by, for example, malicious network intrusions, network equipment failures or malfunctions, new traffic patterns, and the like. Some anomaly detection systems use machine learning techniques to detect anomalies in network traffic. However, the packet rate of modern networks is high and increasing, and thus it is challenging to implement a commercially viable anomaly detection system that can operate at the necessary speeds.

The network anomaly detection system may be located within a network device (e.g., switch, router, bridge, network Interface Card (NIC), etc.), or at a central location that serves many network devices.

Some network anomaly detection systems use machine learning (e.g., artificial neural networks). However, if machine learning algorithms/hardware are processing new data and/or determining whether anomalies are detected at the rate of packets transmitted in the network, it is challenging to detect anomalies while keeping the system cost at a commercially viable level.

Disclosure of Invention

In an embodiment, an anomaly detection apparatus for detecting anomalies in network traffic, the anomaly detection apparatus comprising: a statistics generator configured to receive characteristics of packets in the network traffic and to generate statistics for the network traffic, the statistics including distribution statistics for respective distributions over time of respective characteristics of packets in the network traffic; and an anomaly detection processor configured to detect anomalies with respect to network traffic based at least on the statistics generated by the statistics generator, including detecting deviations of the distribution statistics from the distribution statistics of normal network traffic, and detecting anomalies with respect to network traffic based on deviations of the distribution statistics from the distribution statistics of normal network traffic.

In another embodiment, a method for detecting anomalies in network traffic includes: receiving, at the feature extraction circuitry, characteristics of packets in the network traffic; generating, at the feature extraction circuitry, statistics of the network traffic, the statistics including distribution statistics regarding respective distributions over time of respective characteristics of packets in the network traffic; and detecting, at the anomaly detection processor, anomalies with respect to network traffic based at least on the statistics generated by the statistics generator, including detecting deviations of the distribution statistics from the distribution statistics of normal network traffic, and detecting anomalies with respect to network traffic based on deviations of the distribution statistics from the distribution statistics of normal network traffic.

Drawings

FIG. 1 is a simplified diagram of an anomaly detection system in an example network traffic including a feature extraction system and an anomaly detection processor, according to an embodiment.

FIG. 2A is a simplified block diagram of the anomaly detection processor of FIG. 1, according to an embodiment.

FIG. 2B is a simplified block diagram of the anomaly detection processor of FIG. 1 according to another embodiment.

FIG. 2C is a simplified block diagram of the anomaly detection processor of FIG. 1 according to another embodiment.

Fig. 3 is a flowchart of an example method for detecting anomalies in network traffic, according to an embodiment.

FIG. 4 is a simplified block diagram of an example network device incorporating the feature extraction system and anomaly detection processor of FIG. 1, according to an embodiment.

FIG. 5 is a simplified block diagram of an example system including a plurality of network devices, each incorporating a respective feature extraction system of FIG. 1, and the anomaly detection processor of FIG. 1, in accordance with an embodiment.

Detailed Description

Various embodiments of an anomaly detection system in network traffic are described below. In some embodiments, the anomaly detector of the anomaly detection system in the network traffic is configured to i) operate at a lower rate than the packet rate, and ii) use network traffic statistics that provide information about the plurality of packets. For example, the rate at which the anomaly detector operates corresponds to a period of time having a duration at least as long as the total time of transmission of the plurality of packets. In some embodiments, such anomaly detectors are significantly less costly than anomaly detectors operating at the packet rate, e.g., processing new data and/or determining whether anomalies are detected at the rate of transmission packets in the network.

In other embodiments, the network anomaly detection system is additionally or alternatively configured to i) generate distribution statistics (particular types of network traffic statistics) regarding the distribution over time of respective characteristics of packets in the network traffic (e.g., packet length, duration of inter-packet gaps, etc.), and ii) use the distribution statistics to detect anomalies in the network traffic. As an illustrative example only, the network anomaly detection system uses a distribution of packet sizes in traffic flows over time and detects anomalies based at least on significant deviations of the traffic flows from the packet size distribution. In some embodiments, such distribution statistics are generated at a rate lower than the packet rate, which facilitates the anomaly detector to operate at a rate lower than the packet rate.

Fig. 1 is a simplified diagram of an anomaly detection system 100 in an example network traffic, according to an embodiment. The anomaly detection system 100 in network traffic detects anomalies in network traffic corresponding to malicious network intrusions, network equipment failures or malfunctions, new traffic patterns, and the like.

Anomaly detection system 100 in network traffic includes a packet parser 104, a feature extraction system 108 coupled to packet parser 104, and an anomaly detection processor 112 coupled to feature extraction system 108. Packet parser 104 typically extracts information from packets in network traffic and provides the extracted information to feature extraction system 108. The feature extraction system 108 generally uses information extracted from the packets and timing information associated with the packets to generate statistical data information about the packets. The anomaly detection processor 112 generally uses statistical data information from the feature extraction system 108 to detect anomalies in network traffic.

As briefly discussed above, the packet parser extracts information from packets in network traffic. More particularly, according to some embodiments, packet parser 104 is configured to receive packet data (i.e., network traffic) corresponding to packets transmitted in a network, and extract information from the packets. As an example, packet parser 104 is configured to extract header information from the packet, such as an Internet Protocol (IP) source address, an IP destination address, a layer 2 source address (as an example, a Media Access Control (MAC) source address), a layer 2 destination address (as an example, a MAC destination address), a Transmission Control Protocol (TCP) source port Identifier (ID), a TCP destination port ID, a User Datagram Protocol (UDP) source port ID, a UDP destination port ID, an IP version identifier, a packet length, and the like.

Feature extraction system 108 is configured to receive at least some of the information extracted by packet parser 104. In some embodiments, feature extraction system 108 is further configured to receive packet metadata that includes timing information about packets in network traffic. For example, the feature extraction system 108 is configured to receive timing information that indicates one or more of: i) The time at which a network device (e.g., a network device including packet parser 104) begins receiving packets (i.e., time of arrival), ii) the time at which packets are transmitted (i.e., time of transmission), iii) the time at which packet reception ends at the network device, iv) the duration of packet transmission, v) the duration of gaps between packets, vi) the length of packets, etc. In some embodiments, the metadata includes other information about the packet, such as the port (or interface) from which the packet was received, the port (or interface) via which the packet was transmitted, an error code generated by a packet processor (of the network device) that processes the packet, and so forth.

The metadata is generated by a network device associated with the anomaly detection system 100 in the network traffic and provided by the network device to the anomaly detection system 100 in the network traffic. In some embodiments, feature extraction system 108 is included in a network device (such as a switch, router, etc.) configured to receive packets via a plurality of network links and forward the packets via the plurality of network links, and metadata is generated by the network device. In some embodiments where feature extraction system 108 is included in a network device, such as a switch, router, or the like, packet parser 104 is a component of the network device, and packet information generated by packet parser 104 is also used by the network device to process the packet (e.g., determine which ports of the network device to forward the packet received by the network device via, determine how to modify the packet (e.g., whether to add a tunnel header to the packet, whether to remove a tunnel header from the packet, whether to update a next hop address in the packet, etc.)).

As will be described further below, feature extraction system 108 uses i) information extracted from packets by packet parser 104 and/or ii) packet metadata to generate statistics about network traffic corresponding to packets processed by packet parser 104. Examples of statistical data generated by feature extraction system 108 are further described below. In some embodiments, the statistics generated by feature extraction system 108 include distribution statistics regarding the distribution of the respective characteristics of packets in the network traffic. Examples of distribution statistics (described further below) include a distribution of packet sizes in network traffic during a time period (or during transmission of a set of N packets, where N is a suitable integer greater than 1), and a distribution of inter-packet gap sizes during the time period (or during the set of N packets). Illustrative and non-limiting examples of N include 100, 200, 300, etc.

Feature extraction system 108 also generates respective sets of information (sometimes referred to as "feature vectors") that provide information about network traffic during respective time periods or during transmission of respective sets of N packets. The respective sets of information generated by feature extraction system 108 include at least statistics (including distributed statistics in some embodiments) for network traffic during respective time periods or during transmission of respective sets of N packets.

The corresponding set of information (or feature vectors) generated by feature extraction system 108 is provided to anomaly detection processor 112. The anomaly detection processor 112 is configured to process the feature vectors to detect anomalies with respect to network traffic. In some embodiments, the anomaly detection processor 112 is configured to generate an indicator as to whether an anomaly in network traffic is detected based on processing the feature vector. In some embodiments, the indicator of whether an anomaly is detected includes a score indicating a degree of deviation from normal network traffic behavior.

In some embodiments, anomaly detection processor 112 includes a machine learning engine that is trained to detect anomalies in network traffic based on feature vectors. For example, the anomaly detection processor 112 is trained on network traffic that is assumed to be normal, and thus the anomaly detection processor 112 learns the statistical data patterns of normal network traffic. After training, if the statistics monitored by anomaly detection processor 112 deviate to a large extent from the statistics of normal network traffic, the output generated by anomaly detection processor 112 may be indicative of anomalies in network traffic.

In some embodiments, the anomaly detection processor 112 includes a support vector machine. In some embodiments, the anomaly detection processor 112 includes a Bayesian network.

Referring to fig. 2A, in some embodiments, the anomaly detection processor 112 includes an artificial neural network 150. Referring to FIG. 2B, in some embodiments, the anomaly detection processor 112 includes an automatic encoder 160, such as a single automatic encoder 160.

Referring to FIG. 2C, according to an embodiment, anomaly detection processor 112 includes a plurality of auto encoders disposed in a collection layer 174 and an output layer 178. The aggregation layer 174 includes a plurality of auto encoders 182. Feature mapper 186 is coupled to aggregation layer 174. Feature mapper 186 receives feature vectors from feature extractor 140 and provides each auto-encoder with a respective feature subset (a respective subspace) from each feature vector. Each auto-encoder 182 is configured to process a respective subspace to generate a respective subspace score indicative of a degree to which the subspace deviates from normal behavior.

According to an embodiment, the output layer comprises an automatic encoder 190, e.g. a single automatic encoder 190. The auto-encoder 190 receives the subspace scores generated by the plurality of auto-encoders 182 and is configured to use the subspace scores to generate a final score that is indicative of a degree of deviation from normal network traffic behavior. In an embodiment, the final score corresponds to an anomaly indicator.

Referring again to fig. 1, in some embodiments, anomaly detection processor 112 includes a statistical-based detection engine that implements a suitable algorithm (such as a standard score algorithm, tukey's range test, grubb's test, etc.) on the feature vectors to detect anomalies with respect to network traffic.

Feature extraction system 108 includes a flow classifier 124, flow classifier 124 configured to process header information extracted from packets by packet parser 104 to determine which flow the packets belong to. In an embodiment, the flow classifier 124 defines a flow as a packet sharing the same set of header information. In some embodiments, the same set of header information includes a network source address (e.g., a source IP address, a source MAC address, or another suitable network address) and a network destination address (e.g., a destination IP address, a destination MAC address, or another suitable network address). In the illustrative embodiment, the same set of header information includes a source IP address, a source TCP/UDP port, a destination IP address, a destination TCP/UDP port, and an IP version identifier. In an embodiment, the same set of information includes a source IP address, a source TCP/UDP port, a destination IP address, a destination TCP/UDP port, and an IP version identifier. In other embodiments, the flows identified by the flow classifier 124 correspond to another suitable same set of header information, such as packets destined for the same endpoint, packets to be forwarded to the same intermediate device (e.g., the same switch, router, bridge, etc.), and so forth.

In some embodiments, flow classifier 124 generates flow classification information indicating the flow to which the packet belongs. In an embodiment, the flow classification information includes a flow Identifier (ID) identifying the flow to which the packet belongs.

In other embodiments, such as those in which feature extraction system 108 is incorporated in a network device (such as a switch, router, bridge, etc.) configured to process packets and make forwarding decisions for the packets (e.g., determine one or more ports of the network device via which the packets are to be transmitted), flow classifier 124 is omitted from feature extraction system 108, and feature extraction system 108 essentially treats packets being transmitted via the same port of the network device (and/or queued for transmission in the same queue of the network device) as belonging to the same flow. In some such embodiments, determining that multiple packets are to be transmitted via the same port (or queuing the packets in the same queue) may be considered as classifying the packets as belonging to the same flow by the network device. In some embodiments, the plurality of queues of the network device may correspond to a same network link, and wherein respective ones of the plurality of queues correspond to different transmission priorities.

Thus, the term "flow" as used herein refers to a set of packets having the same set of header information, and/or packets determined by a network device to be transmitted via the same port of the network device, and/or packets queued in the same queue by the network device for transmission by the network device.

The statistics generator 128 receives header information extracted from the packets by the packet parser 104, flow classification information from the flow classifier 124, and packet metadata. The statistics generator 128 is configured to generate statistics on packet data using at least the flow classification information and packet metadata from the flow classifier 124. In embodiments where the flow classifier 124 is omitted (e.g., embodiments where the feature extraction system 108 processes packets queued by network devices), the statistics generator 128 does not receive flow classification information and does not use the flow classification information to generate statistics.

More particularly, the statistics generator 128 is configured to generate statistics regarding characteristics of network traffic in first time windows, each first time window corresponding to transmission of a plurality of packets. In some embodiments, the first time window is a non-overlapping time window that does not overlap in time with other time windows.

In other embodiments, the first time window is a sliding window that overlaps in time with other first time windows.

In an embodiment, each first time window corresponds to a predetermined amount of time. For illustrative example only, each first time window has a duration of 200 microseconds, 500 microseconds, 1 second, etc., or any other suitable duration. In another embodiment, each first time window corresponds to a predetermined number of packets in the network traffic. By way of illustrative example only, each first time window corresponds to 200 packets, 300 packets, 500 packets, 1000 packets, etc., or any other suitable number of packets. In some embodiments, the predetermined number of packets is a predetermined number of packets in a stream for which statistics are being generated.

Examples of statistics generated by statistics generator 128 regarding network traffic characteristics in a time window include: i) Packet rate during the time window (e.g., number of packets divided by duration of the time window), ii) data rate during the time window (e.g., total number of bits divided by duration of the time window), iii) average packet size during the time window, iv) minimum packet size during the time window, v) maximum packet size during the time window, vi) minimum inter-packet gap (IPG) size during the time window, vii) maximum IPG size during the time window, viii) average IPG size during the time window. In various embodiments, the statistics generator 128 is configured to generate one or any suitable combination of the two or more statistics described above.

In some embodiments, the statistics generator 128 includes a distribution statistics generator 132, the distribution statistics generator 132 being configured to generate distribution statistics regarding respective distributions over time of respective characteristics of packets in the network traffic. In an embodiment, the distribution statistics generator 132 is configured to generate distribution statistics about the distribution of packet sizes over each first time window. For example, a plurality of packet size ranges (sometimes referred to herein as "packet size bins") are defined, and the distribution statistics generator 132 records a respective number of packets corresponding to the respective packet size ranges (or bins) during the first time window. In the illustrative embodiment, the number of packet size bins is eight. In other embodiments, the number of packet size bins is a suitable number other than eight.

In various other examples, the distribution statistics generator 132 generates any suitable combination of one or two or more of the following: average deviation of the packet size from the average packet size during the first time window, mean square deviation of the packet size from the average packet size during the first time window, etc.

In another embodiment, the distribution statistics generator 132 is additionally or alternatively configured to generate distribution statistics about the distribution of IPG sizes over each first time window. For example, a plurality of IPG size ranges (sometimes referred to herein as "IPG size bins") are defined, and the distribution statistics generator 132 records a respective number of IPGs corresponding to the respective IPG size ranges (or bins) during a first time window. In the illustrative embodiment, the number of IPG sized bins is eight. In other embodiments, the number of IPG-sized bins is a suitable number other than eight.

In various other examples, the distribution statistics generator 132 generates one of, or any suitable combination of two or more of: average deviation of IPG size from average IPG size during the first time window, mean square deviation of IPG size from average IPG size during the first time window, etc.

In some embodiments, the statistics generator 128 omits the distribution statistics generator 132 and does not generate the distribution statistics as described above.

In some embodiments, statistics generator 128 generates some or all of the statistics described above, including distribution statistics for each stream. In some embodiments, the first time window for generating statistics for a stream corresponds to a particular number of packets in the stream, e.g., 100 packets in the stream, 200 packets in the stream, 300 packets in the stream, etc. In other embodiments, the first time window for generating statistics for a flow corresponds to a particular number of packets, regardless of which flow the packets belong to. In other embodiments, the first time window for generating statistics for the stream corresponds to a particular duration, e.g., 200 microseconds, 300 microseconds, 1 second, etc.

In various embodiments, the statistics generator 128 generates any suitable combination of one or two or more of the following: i) The packet rate of packets belonging to the flow during the time window (e.g., the number of packets divided by the duration of the window), ii) the data rate of packets belonging to the flow during the time window (e.g., the total number of bits in the flow divided by the duration of the window), iii) the average packet size of packets belonging to the flow during the time window, iv) the minimum packet size of packets belonging to the flow during the time window, v) the maximum packet size of packets belonging to the flow during the time window, vi) the minimum IPG size between packets belonging to the flow during the time window, vii) the maximum IPG size between packets belonging to the flow during the time window, viii) the average IPG size between packets belonging to the flow during the time window, etc.

In some embodiments in which the statistics generator 128 includes a distribution statistics generator 132, the distribution statistics generator 132 is configured to generate distribution statistics for a respective distribution of respective characteristics of packets for each flow, i.e., for packets having the same set of header information (e.g., the same set of source addresses, destination addresses, etc.). For example, in various embodiments, the distribution statistics generator 132 is configured to generate any suitable combination of one or two or more of the following: i) Distribution statistics regarding packet size distribution in the flow within each time window (e.g., distribution statistics generator 132 records a respective number of packets in the flow corresponding to a respective packet size range of packets in the flow during the time window), ii) average deviation of packet size from average packet size during the time window for packets in the flow, iii) average deviation of packet size from average packet size during the time window for packets in the flow, iv) distribution statistics regarding IPG size distribution of the flow over each time window (e.g., distribution statistics generator 132 records a respective number of IPGs between packets in the flow corresponding to a respective IPG size range during the time window); v) average deviation of IPG size for packets from average IPG size in the flow during the time window; vi) mean square deviation of IPG size for packets from average IPG size in the stream during the time window, etc.

The statistics generator 128 is coupled to the memory 136 and uses the memory 136 to generate and store statistics as described above.

Feature extractor 140 is coupled to statistics generator 128. Feature extractor 140 generates feature vectors based on the statistics generated by statistics generator 128. For example, in some embodiments, feature extractor 140 generates new statistics by mathematically combining the plurality of statistics generated by statistics generator 128, compiles a plurality of distribution statistics generated by statistics generator 128 for a plurality of first time windows to generate distribution statistics for a second, longer time window, and so on. As an illustrative example, feature extractor 140 mathematically combines the plurality of average packet size statistics for the plurality of first time windows to generate an average packet size for a second, longer time window corresponding to the plurality of first time windows. As another illustrative example, feature extractor 140 mathematically combines the plurality of average IPG size statistics for the plurality of first time windows to generate an average IPG size for a longer second time window corresponding to the plurality of first time windows. As another illustrative example, feature extractor 140 mathematically combines a plurality of average deviations of the average packet size statistics for a plurality of first time windows to generate an average deviation of the average packet size for a longer second time window corresponding to the plurality of first time windows. As another illustrative example, feature extractor 140 mathematically combines the plurality of average deviations of the average IPG size statistics for the plurality of first time windows to generate an average deviation of the average IPG size for a longer second time window corresponding to the plurality of first time windows.

As another illustrative example, feature extractor 140 compiles records of the number of packets falling within various size ranges during a plurality of first time windows to generate records of the number of packets falling within various size ranges during a longer second time window corresponding to the plurality of first time windows. As another illustrative example, feature extractor 140 compiles records of the number of IPGs that fall within various size ranges during a plurality of first time windows to generate records of the number of IPGs that fall within various size ranges during a longer second time window corresponding to the plurality of first time windows.

In general, the feature extractor 140 generates statistics for a second time window that is longer than the first time window according to which the statistics generator 128 operates. For example, each feature vector corresponds to a second, longer time window (e.g., a longer time window than the first time window upon which the statistics generator 128 operates), and the feature vector includes statistics generated by the feature extractor 140 for the second, longer time window, and the statistics are generated for a plurality of first time windows corresponding to the second, longer time window based on the statistics from the statistics generator 128. In some embodiments in which the statistics generator 128 generates statistics for each stream, the feature vector includes information about the stream and statistics corresponding to the stream and for a longer second time window. The information about the flow includes any suitable combination of one or two or more of the following: an identifier of a port of the network device via which the packet from which the statistics were generated is to be transmitted; an identifier of a queue of the network device, storing packets from which the statistics were generated; a stream identifier; one or more source addresses (e.g., source IP address, source MAC address, etc.), one or more destination addresses (e.g., destination IP address, destination MAC address, etc.), one or more source port identifiers (e.g., source TCP port, source UDP port, etc.), one or more destination port identifiers (e.g., destination TCP port, destination UDP port, etc.), a protocol identifier (e.g., IP version identifier), an Internet Control Message Protocol (ICMP) type, ICMP code, address Resolution Protocol (ARP) opcode, ARP source MAC address, ARP source IPv4 address, ARP destination MAC address, etc.

In some embodiments, feature extractor 140 generates feature vectors at a rate corresponding to a longer second time window interval, and thus lower than the packet rate. In other embodiments, feature extractor 140 generates feature vectors at a rate that corresponds to a relatively long second time window interval that is short, but still lower than the time interval of the packet rate.

The rate at which feature extractor 140 generates feature vectors is less than the packet rate of the network traffic, thus reducing the cost of feature extractor 140 compared to feature extractors that must generate feature vectors at that packet rate. Additionally, because the rate at which statistics are generated by statistics generator 128 is less than the packet rate of the network traffic, anomaly detection processor 112 can operate at a lower rate (rather than the packet rate), thereby reducing the cost of anomaly detection processor 112 as compared to anomaly detectors that must process statistics at the packet rate.

Feature extractor 140 is coupled to memory 144 and uses memory 144 to generate/compile and store statistical data such as described above.

In embodiments in which the statistics generator 128 includes a distribution statistics generator 132, the anomaly detection processor 112 is configured to detect anomalies in network traffic using distribution statistics (e.g., packet size distribution, IPG size distribution, etc.) such as described above. For example, normal operation of the flow may have a relatively consistent packet size distribution over time, which is known by anomaly detection processor 112 during training. Thus, according to an embodiment, when the distribution of packet sizes in the stream deviates significantly from a uniform packet size distribution, the output of the anomaly detection processor 112 may indicate an anomaly. As another example, the flow may have a relatively consistent IPG size distribution over time, which is known by anomaly detection processor 112 during training. Thus, according to an embodiment, the output of the anomaly detection processor 112 may indicate an anomaly when the distribution of IPG sizes in the stream deviates significantly from a consistent IPG size distribution.

In some embodiments where feature extractor 140 provides feature vectors at a rate lower than the packet rate, anomaly detection processor 112 operates at a rate lower than the packet rate. In some embodiments where feature extractor 140 provides feature vectors at a packet rate, anomaly detection processor 112 samples the feature vectors at a rate lower than the packet rate and operates at a rate lower than the packet rate. In other embodiments where feature extractor 140 provides feature vectors at a packet rate, anomaly detection processor 112 operates at the packet rate.

In an embodiment, the packet parser 104 and feature extraction system 108 are implemented using hardware circuitry. For example, the stream classifier 124, the statistics generator 128, and the feature extractor 140 are implemented using corresponding hardware circuitry. In another embodiment, one or more components of packet parser 104 and/or feature extraction system 108 are implemented using a processor executing machine readable instructions stored in memory.

In an embodiment, the anomaly detection processor 112 is implemented using hardware circuitry. In another embodiment, the anomaly detection processor 112 is implemented using a processor executing machine-readable instructions stored in a memory.

Fig. 3 is a flowchart of an example method 200 for detecting anomalies in network traffic, according to an embodiment. In an embodiment, the anomaly detection system 100 (fig. 1) in example network traffic implements a method 200, and for purposes of explanation, the method 200 is discussed with reference to fig. 1. In other embodiments, the method 200 is implemented by another suitable anomaly detection system in network traffic.

At block 204, characteristics of packets in network traffic are received. For example, the statistics generator 128 receives characteristics of packets in network traffic, such as header information and packet metadata extracted from the packets by the packet parser 104. In some embodiments, the metadata includes timing information about the packet, such as described above.

At block 208, statistics for network traffic are generated. In some embodiments, the statistics generated at block 208 include distribution statistics regarding respective distributions over time of respective characteristics of packets in the network traffic. For example, as discussed above, the statistics generator 128 (and optional distribution statistics generator 132) generates statistics for network traffic.

In some embodiments, distribution statistics are generated at block 208, the distribution statistics including statistics of a distribution of packet sizes in network traffic over time. In some embodiments, where the distribution statistics are generated at block 208, the distribution statistics include a respective distribution over time of packet sizes in respective packet flows in the network traffic, each packet flow including packets having a respective set of common packet header information.

In some embodiments, distribution statistics are generated at block 208, the distribution statistics including distribution statistics of IPG size over time in network traffic. In some embodiments, where the distribution statistics are generated at block 208, the distribution statistics include statistics of the distribution over time of IPG sizes in respective packet flows in the network traffic, each packet flow including packets having a respective set of common packet header information.

At block 212, anomalies with respect to network traffic are detected using the statistics generated at block 208. For example, feature extractor 140 generates feature vectors using the statistics generated at block 208, and anomaly detection processor 112 detects anomalies using the feature vectors generated by feature extractor 140. In some embodiments, the statistics generated at block 208 include statistics of a respective distribution of packet sizes, and detecting anomalies at block 212 includes using statistics of a respective distribution of packet sizes. In some embodiments, the statistics generated at block 208 include statistics of respective distributions of packet sizes, and detecting anomalies at block 212 includes using statistics of respective distributions of packet sizes in respective packet flows.

In some embodiments, the anomaly detection processor 112 is trained to learn statistics of network traffic that is assumed to be normal (e.g., corresponding to the statistics generated at block 208), and detecting anomalies at block 212 includes the anomaly detection processor 112 determining a degree of deviation of the statistics generated at block 208 from the statistics of network traffic that is assumed to be normal.

In some embodiments where the statistics generated at block 208 include statistics of a corresponding distribution of IPG sizes, detecting anomalies at block 212 includes using statistics of a corresponding distribution of IPG sizes. In some embodiments, wherein the statistics generated at block 208 include statistics of a respective distribution of IPG sizes, detecting anomalies at block 212 includes using statistics of a respective distribution of IPGs of packets in a respective packet stream.

In some embodiments, detecting the anomaly at block 212 includes performing, by the anomaly detection processor 112, a process of detecting the anomaly at a rate corresponding to a time interval at least as long as the aggregate duration of the plurality of packets.

In some embodiments, generating statistics for network traffic at block 208 includes providing updated statistics of network traffic to anomaly detection processor 112 at a rate corresponding to a time interval at least as long as a total duration of the plurality of packets, including updated distribution statistics regarding a distribution of respective characteristics of packets in the network traffic over time.

In some embodiments, generating distribution statistics at block 208 includes generating distribution statistics for respective distributions of respective characteristics of packets in network traffic over a predetermined time interval; and detecting an anomaly in the network traffic at block 212 includes detecting an anomaly in the network traffic that occurred during the time interval.

In some embodiments, generating the distribution statistics at block 208 includes generating distribution statistics for respective distributions of respective characteristics of packets in the network traffic over a time interval corresponding to a predetermined number of packets in the network traffic; and detecting an anomaly in the network traffic at block 212 includes detecting an anomaly in the network traffic that occurred during the time interval.

FIG. 4 is a simplified block diagram of an example network device 400 including feature extraction system 108 and anomaly detection processor 112, according to an embodiment. In various embodiments, network device 400 is a layer 2 switch, router, bridge, or the like.

In some embodiments, network device 400 includes a plurality of ports (not shown) coupled to a plurality of network links (not shown). The network device 400 includes a packet processor 404, which packet processor 404 is configured to process packets received by the network device 400 and make forwarding decisions for the packets (e.g., determine one or more ports of the network device 400 via which the packets are to be transmitted). The processing of the packet by the packet processor 404 includes generating and/or compiling metadata as described above, parsing a header of the packet as described above, and the like. For example, the packet processor 404 includes a packet parser (not shown), such as the packet parser 104 of fig. 1. Feature extraction system 108 of network device 400 receives the metadata (including timing information) and parsed packet header data and generates statistics (including distribution statistics in some embodiments) such as described above. Additionally, the feature extraction system 108 uses statistics (including distribution statistics in some embodiments) to generate feature vectors such as described above. The feature vectors provide information about network traffic (e.g., in some embodiments, statistical data information including distribution statistics) during respective time periods or during transmission of respective sets of N packets received by the network device 400. The anomaly detection processor 112 processes the feature vectors and uses the processing of the feature vectors to detect anomalies in network traffic received by the network device 404.

FIG. 5 is a simplified block diagram of an example system 500 including a plurality of network devices 504 and an anomaly detection processor 112, according to an embodiment. In various embodiments, each network device 504 is a layer 2 switch, router, bridge, or the like. Each network device 504 includes a respective feature extraction system 108, the respective feature extraction system 108 generating feature vectors such as described above for packets received at the network device 504. In an embodiment, each network device 504 is similar to network device 400 of fig. 4, but does not include an anomaly detection system. Each network device 504 transmits the feature vector to anomaly detection system 112 via a communication path (not shown) in system 500.

The anomaly detection processor 112 processes the feature vectors received from the network devices 504 and uses the processing of the feature vectors to detect anomalies in network traffic received by the network devices 504.

Example 1: an anomaly detection device for detecting anomalies in network traffic, the anomaly detection device comprising: a statistics generator configured to receive characteristics of packets in the network traffic and to generate statistics for the network traffic, the statistics including distribution statistics for respective distributions over time of respective characteristics of packets in the network traffic; and an anomaly detection processor configured to detect anomalies with respect to network traffic based at least on the statistics generated by the statistics generator, including detecting deviations of the distribution statistics from the distribution statistics for normal network traffic, and detecting anomalies with respect to network traffic based on deviations of the distribution statistics from the distribution statistics for normal network traffic.

Example 2: the abnormality detection device according to embodiment 1, wherein: the statistics generator is configured to generate statistics of a distribution of packet sizes in network traffic over time; and the anomaly detection processor is configured to detect anomalies with respect to network traffic based on detecting deviations of the statistics of packet size distribution in network traffic compared to the statistics of packet size distribution in normal network traffic.

Example 3: the abnormality detection device according to embodiment 2, wherein: the statistics generator is configured to generate statistics of respective distributions of packet sizes over time in respective packet flows in the network traffic; and the anomaly detection processor is configured to detect anomalies with respect to respective ones of the network traffic based on detecting deviations of statistics of respective distributions of packet sizes in the respective packet flows from statistics of respective distributions of packet sizes in normal network traffic in the respective packet flows.

Example 4: the abnormality detection device according to any one of embodiments 1 to 3, wherein: the statistics generator is configured to generate statistics of a distribution of sizes of inter-packet gaps (IPGs) in network traffic over time; and the anomaly detection processor is configured to detect anomalies with respect to network traffic based on detecting deviations of the statistics of the IPG size distribution from the statistics of the IPG size distribution in normal network traffic.

Example 5: the abnormality detection device according to embodiment 4, wherein: the statistics generator is configured to generate statistics of respective distributions of the IPGs in respective packet flows in the network traffic over time; and the anomaly detection processor is configured to detect anomalies with respect to respective ones of the network traffic based on detecting deviations of statistics of the respective distributions of IPG sizes in the respective packet flows from statistics of the respective distributions of IPG sizes in normal network traffic in the respective packet flows.

Example 6: the abnormality detection device according to any one of embodiments 1 to 5, wherein: the anomaly detection processor is configured to perform the process for detecting anomalies at a rate corresponding to a time interval at least as long as a total duration of the plurality of packets.

Example 7: the abnormality detection device according to embodiment 6, further comprising: a feature extractor is coupled to the statistics generator, the feature extractor is configured to generate compiled distribution statistics regarding a distribution of respective characteristics of packets in the network traffic over time, and to provide the compiled distribution statistics to the anomaly detection processor at a rate corresponding to a time interval at least as long as a total duration of the plurality of packets.

Example 8: the abnormality detection device according to embodiment 7, wherein: the feature extractor is configured to generate compiled distribution statistics for respective distributions of respective characteristics of packets in the network traffic over a predetermined time interval; and the anomaly detection processor is configured to detect anomalies in network traffic that occur during the time interval.

Example 9: the abnormality detection device according to embodiment 7, wherein: the feature extractor is configured to generate compiled distribution statistics regarding respective distributions of respective characteristics of packets in the network traffic over time intervals corresponding to a predetermined number of packets in the network traffic; and the anomaly detection processor is configured to detect anomalies in network traffic that occur during the time interval.

Example 10: a method for detecting anomalies in network traffic, the method comprising: receiving, at the feature extraction circuitry, characteristics of packets in the network traffic; generating, at the feature extraction circuitry, statistics for the network traffic, the statistics including distribution statistics for respective distributions over time of respective characteristics of packets in the network traffic; and detecting, at the anomaly detection processor, anomalies with respect to network traffic based at least on the statistics generated by the statistics generator, including detecting deviations of the distribution statistics from the distribution statistics for normal network traffic, and detecting anomalies with respect to network traffic based on deviations of the distribution statistics from the distribution statistics for normal network traffic.

Example 11: the method of embodiment 10, wherein: generating distribution statistics includes generating distribution statistics of packet sizes in network traffic over time; and detecting anomalies with respect to network traffic includes: the anomaly is detected based on detecting statistics of packet size distribution in network traffic, deviations from statistics of packet size distribution for normal network traffic.

Example 12: the method of embodiment 11, wherein: generating statistics of the packet size distribution includes generating statistics of respective distributions of packet sizes in respective packet flows in the network traffic over time, each packet flow including packets having a respective set of common packet header information; and detecting anomalies with respect to network traffic: including detecting anomalies based on detecting deviations in statistics of respective distributions of packet sizes in respective packet flows as compared to statistics of packet size distributions for normal network traffic in the respective packet flows.

Example 13: the method of any one of embodiments 10 to 12, wherein: generating distribution statistics includes generating distribution statistics of the size of inter-packet gaps (IPGs) in network traffic over time; and detecting anomalies with respect to network traffic includes detecting anomalies based on detecting deviations in the statistics of the IPG size distribution compared to the statistics of the IPG size distribution for normal network traffic.

Example 14: the method of embodiment 13, wherein: generating statistics of IPG size distribution includes generating statistics of size distribution of IPGs over time in respective packet flows in network traffic, each packet flow including packets having a respective set of common packet header information; and detecting anomalies with respect to corresponding packet flows in network traffic includes: an anomaly is detected based on detecting a deviation of statistics of a respective distribution of IPG sizes in the respective packet flows from statistics of a distribution of IPG sizes of normal network traffic in the respective packet flows.

Example 15: the method of any one of embodiments 10 to 14, wherein: detecting anomalies with respect to network traffic includes performing, by an anomaly detection processor, a process of detecting anomalies at a rate corresponding to a time interval at least as long as a total duration of the plurality of packets.

Example 16: the method of embodiment 15, further comprising: generating, by the feature extraction circuitry, compiled distribution statistics regarding a distribution over time of respective characteristics of packets in the network traffic; and providing the compiled distribution statistics to the anomaly detection processor at a rate corresponding to a time interval at least as long as a total duration of the plurality of packets.

Example 17: the method of embodiment 16, wherein: generating compiled distribution statistics includes generating compiled distribution statistics for respective distributions of respective characteristics of packets in network traffic over a predetermined time interval; and detecting an anomaly in the network traffic includes detecting an anomaly in the network traffic that occurred during the time interval.

Example 18: the method of embodiment 16, wherein: generating compiled distribution statistics includes generating compiled distribution statistics for respective distributions of respective characteristics of packets in the network traffic over time intervals corresponding to a predetermined number of packets in the network traffic; and detecting an anomaly in the network traffic includes detecting an anomaly in the network traffic that occurred during the time interval.

At least some of the various blocks, operations, and techniques described above may be implemented with hardware, a processor executing firmware instructions, a processor executing software instructions, or any combination thereof. When implemented with a processor executing software or firmware instructions, the software or firmware instructions may be stored in any suitable computer-readable memory, such as Random Access Memory (RAM), read-only memory (ROM), flash memory, and the like. The software or firmware instructions may include machine-readable instructions that, when executed by one or more processors, cause the one or more processors to perform various actions.

When implemented in hardware, the hardware may include one or more of discrete components, integrated circuits, application Specific Integrated Circuits (ASICs), programmable Logic Devices (PLDs), and the like.

Although the present invention has been described with reference to particular embodiments, these embodiments are illustrative only and not limiting of the invention, as changes, additions and/or deletions may be made to the disclosed embodiments without departing from the scope of the invention.

Claims (18)

1. An anomaly detection device for detecting anomalies in network traffic, the anomaly detection device comprising:

a statistics generator configured to receive characteristics of packets in network traffic and to generate statistics for the network traffic, the statistics including distribution statistics regarding respective distributions over time of respective characteristics of packets in the network traffic; and

an anomaly detection processor configured to detect anomalies with respect to the network traffic based at least on the statistics generated by the statistics generator, including detecting deviations of the distribution statistics from distribution statistics for normal network traffic, and detecting anomalies with respect to the network traffic based on the deviations of the distribution statistics from the distribution statistics for the normal network traffic.

2. The abnormality detection device according to claim 1, wherein:

the statistics generator is configured to generate statistics of a distribution of packet sizes in the network traffic over time; and is also provided with

The anomaly detection processor is configured to detect anomalies with respect to the network traffic based on detecting the deviation of the statistical data of the packet size distribution in the network traffic compared to the statistical data of the packet size distribution in the normal network traffic.

3. The abnormality detection device according to claim 2, wherein:

the statistics generator is configured to generate statistics of respective distributions of packet sizes over time in respective packet flows in the network traffic; and is also provided with

The anomaly detection processor is configured to detect anomalies with respect to the respective ones of the network traffic based on detecting the deviation of the statistics of the respective distributions of packet sizes in the respective packet flows from statistics of the respective distributions of packet sizes in the normal ones of the respective packet flows.

4. The abnormality detection device according to claim 1, wherein:

The statistics generator is configured to generate statistics of a distribution of sizes of inter-packet gaps (IPGs) in the network traffic over time; and is also provided with

The anomaly detection processor is configured to detect anomalies with respect to the network traffic based on detecting the deviation of the statistical data of the IPG size distribution from the statistical data of the IPG size distribution in the normal network traffic.

5. The abnormality detection device according to claim 4, wherein:

the statistics generator is configured to generate statistics of respective distributions of IPGs over time in respective packet flows in the network traffic; and is also provided with

The anomaly detection processor is configured to detect anomalies with respect to the respective ones of the network traffic based on detecting the statistics of the respective distributions of IPG sizes in the respective packet flows, the deviations compared to the statistics of the respective distributions of IPG sizes in the normal network traffic in the respective packet flows.

6. The abnormality detection device according to claim 1, wherein:

the anomaly detection processor is configured to perform a process for detecting anomalies at a rate corresponding to a time interval at least as long as a total duration of the plurality of packets.

7. The abnormality detection device according to claim 6, further comprising:

a feature extractor coupled to the statistics generator, the feature extractor configured to generate compiled distribution statistics regarding the distribution of respective characteristics of packets in the network traffic over time, and to provide the compiled distribution statistics to the anomaly detection processor at the rate corresponding to the time interval that is at least as long as a total duration of the plurality of packets.

8. The abnormality detection device according to claim 7, wherein:

the feature extractor is configured to generate compiled said distribution statistics for respective distributions of respective characteristics of packets in the network traffic over a predetermined time interval; and is also provided with

The anomaly detection processor is configured to detect anomalies in the network traffic that occur during the time interval.

9. The abnormality detection device according to claim 7, wherein:

the feature extractor is configured to generate compiled said distribution statistics regarding respective distributions of respective characteristics of packets in the network traffic over time intervals corresponding to a predetermined number of packets in the network traffic; and is also provided with

The anomaly detection processor is configured to detect anomalies in the network traffic that occur during the time interval.

10. A method for detecting anomalies in network traffic, the method comprising:

receiving, at the feature extraction circuitry, characteristics of packets in the network traffic;

generating, at the feature extraction circuitry, statistics for the network traffic, the statistics comprising distribution statistics for respective distributions over time of respective characteristics of packets in the network traffic; and

at an anomaly detection processor, detecting anomalies with respect to the network traffic based at least on the statistics generated by the statistics generator includes detecting deviations of the distribution statistics from the distribution statistics for normal network traffic, and detecting anomalies with respect to the network traffic based on deviations of the distribution statistics from the distribution statistics for the normal network traffic.

11. The method according to claim 10, wherein:

generating distribution statistics includes generating distribution statistics of packet sizes in the network traffic over time; and is also provided with

Detecting anomalies with respect to the network traffic includes: an anomaly is detected based on detecting the statistics of packet size distribution in the network traffic, a deviation from the statistics of packet size distribution for the normal network traffic.

12. The method according to claim 11, wherein:

generating statistics of packet size distribution includes generating statistics of respective distributions of packet sizes over time in respective packet flows in the network traffic, each packet flow including packets having a respective set of common packet header information; and is also provided with

Detecting anomalies with respect to network traffic includes: an anomaly is detected based on the statistics detecting a respective distribution of packet sizes in a respective packet stream, a deviation from the statistics of the distribution of packet sizes for the normal network traffic in the respective packet stream.

13. The method according to claim 10, wherein:

generating distribution statistics includes generating distribution statistics of the size of inter-packet gaps (IPGs) in the network traffic over time; and is also provided with

Detecting anomalies with respect to the network traffic includes: an anomaly is detected based on detecting the statistical data of the IPG size distribution, a deviation from the statistical data of the IPG size distribution for the normal network traffic.

14. The method according to claim 13, wherein:

generating statistics of IPG size distribution includes generating statistics of a distribution over time of sizes of IPGs in respective packet flows in the network traffic, each packet flow including packets having a respective set of common packet header information; and is also provided with

Detecting anomalies with respect to corresponding packet flows in the network traffic includes: an anomaly is detected based on detecting a deviation of the statistics of the respective distribution of IPG sizes in the respective packet flows from the statistics of the distribution of IPG sizes of the normal network traffic in the respective packet flows.

15. The method according to claim 10, wherein:

detecting anomalies with respect to the network traffic includes performing, by the anomaly detection processor, a process of detecting anomalies at a rate corresponding to a time interval at least as long as a total duration of a plurality of packets.

16. The method of claim 15, further comprising:

generating, by the feature extraction circuitry, compiled distribution statistics regarding a distribution over time of the respective characteristics of packets in the network traffic; and

The compiled distribution statistics are provided to the anomaly detection processor at the rate corresponding to the time interval that is at least as long as the aggregate duration of a plurality of packets.

17. The method according to claim 16, wherein:

generating the compiled distribution statistics includes generating the compiled distribution statistics for respective distributions of respective characteristics of packets in the network traffic over a predetermined time interval; and is also provided with

Detecting anomalies in the network traffic includes detecting anomalies in the network traffic that occur during the time interval.

18. The method according to claim 16, wherein:

generating the compiled distribution statistics includes: generating compiled said distribution statistics for respective distributions of respective characteristics of packets in said network traffic over time intervals corresponding to a predetermined number of packets in said network traffic; and is also provided with

Detecting anomalies in the network traffic includes detecting anomalies in the network traffic that occur during the time interval.