EP4736369A1 - Network anomaly detection - Google Patents

Abstract

A method comprising accessing network traffic data collected from a node in a core of a communications network; computing an encoding of the network traffic data by processing the network traffic data using an adversarially trained encoder; concatenating the encoding with the network traffic data to form an input; classifying the input as benign or anomalous; and in response to a classification of anomalous, triggering a security action.

Description

NETWORK ANOMALY DETECTION

[0001] The present disclosure relates to network anomaly detection for use in communications networks.

BACKGROUND

[0002] There has been a sharp increase in communications devices such as smart phones, smartwatches and other devices connected to communications networks. There has also been an increase in the number of network intrusion and cyberattack incidents across the globe. Undetected threats on a communications network have severe impact and consequences such as, data loss, decline in an organisation’s reputation, and potential loss of essential or emergency communications network services.

[0003] The early detection and identification of anomalies, which are often threats, is key to keeping a communications network protected since early detection and identifications enables action to be taken reducing the impact of any attacks or avoiding them completely. Recent technical advancements and access to technologies by cyber criminals has made it easier to orchestrate a devastating attack on a vulnerable communications network. Threats such as advanced persistent threats and denial of service attacks (DDoS) can easily propagate through traditional security systems and remain undetected.

[0004] The examples described herein are not limited to examples which solve problems mentioned in this background section.

SUMMARY

[0005] Examples of preferred aspects and embodiments of the invention are as set out in the accompanying independent and dependent claims.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0007] A first aspect of the disclosed technology comprises a computer implemented method comprising: accessing network traffic data collected from a node in a core of a communications network; computing an encoding of the network traffic data by processing the network traffic data using an adversarially trained encoder; concatenating the encoding with the network traffic data to form an input; classifying the input as benign or anomalous; and in response to a classification of anomalous, triggering a security action. The combination of using the adversarially trained encoder and the classifying of the concatenation is found to give results which are more accurate than using the classifier alone, or than using a combination of the classifier and outlier mining algorithms. The combination of using the adversarially trained encoder and the classifying of the concatenation is found to be more resilient to a decrease in an amount of labelled data available for training than using the classifier alone. The combination of using the adversarially trained encoder and the classifying of the concatenation is found to have a low inference time.

[0008] In some examples, the encoding of the network traffic data is computed by processing the network traffic data using an adversarially trained encoder and bypassing a decoder and a critic that was used to adversarially train the encoder. As a result the deployment of an anomaly and threat detector is more compact than deployments which include a decoder. In addition, inference time is reduced since a decoder is absent in an inference process.

[0009] In various examples, the network traffic data excludes raw packet data. Excluding raw packet data improves security and also improves efficiency.

[0010] In various examples, the network traffic data comprises internet protocol traffic statistics. Using internet protocol traffic statistics is found to give accurate performance and thus improves security of a communications network.

[0011] In examples, classifying the input comprises using a trained decision tree. In various examples, classifying the input comprises using a decision tree based classifier. Using a trained decision tree based classifier is found to give accurate results in an efficient manner as compared with other types of classifiers such as support vector machines, neural networks or other classifiers.

[0012] In examples, the trained decision tree based classifier implements extreme gradient boosting. Using extreme gradient boosting in combination with the adversarially trained encoder and the concatenated input is found to give particularly accurate results.

[0013] In examples, triggering a security action comprises one or more of: automatically isolating a node of the communications network, sending an alert message to an operator, automatically deactivating a node of the communications network. Thus by early and accurate detection of anomalies, which are often threats, it is possible to take automated mitigating action and so improve security.

[0014] In various examples, training a classifier using supervised learning and using the trained classifier to carry out the classifying enables accurate performance. Because of the combination of the adversarially trained encoder and the classifier trained with supervised learning, the amount of labelled training data needed for training the classifier is reduced as compared with using a classifier alone.

[0015] In examples training the encoder comprises using a loss function that takes into account a critic score and a reconstruction error. In this way, generalization ability of the encoder is reduced thus reducing the ability of the encoder to encode anomalies.

[0016] In examples training the encoder comprises using training data comprising only benign network traffic data. Since benign network traffic data is plentiful whereas threat traffic data is not there is significant benefit in training the encoder using only benign network traffic data. [0017] In another aspect there is an apparatus comprising: a processor; a memory storing network traffic data collected from a node in a core of a communications network; an adversarially trained encoder; a classifier; wherein the memory stores instructions which when executed on a processor: encode the network traffic data using the adversarially trained encoder to produce an encoding; concatenate the encoding with the network traffic data to form an input; classify the input as benign or anomalous using the classifier; and in response to a classification of anomalous, trigger a security action.

[0018] In examples the apparatus is absent a decoder and the network traffic data excludes raw packet data. Thus the apparatus is compact and inference time is reduced as compared with using a decoder.

[0019] In some examples the classifier is a decision tree based classifier implementing extreme gradient boosting. Optionally, the instructions comprise training the encoder using only benign network traffic data.

[0020] In another aspect there is a communications network comprising the apparatus mentioned above. The resulting communications network has improved security since anomalies and threats are detected early and accurately using the apparatus.

[0021] It will also be apparent to anyone of ordinary skill in the art, that some of the preferred features indicated above as preferable in the context of one of the aspects of the disclosed technology indicated may replace one or more preferred features of other ones of the preferred aspects of the disclosed technology. Such apparent combinations are not explicitly listed above under each such possible additional aspect for the sake of conciseness.

[0022] Other examples will become apparent from the following detailed description, which, when taken in conjunction with the drawings, illustrate by way of example the principles of the disclosed technology.

BRIEF DESCRIPTION OF THE DRAWINGS

[0001] FIG. 1 of the accompanying drawings shows an anomaly and threat detector deployed with a network traffic data exporter in a communications network;

[0002] FIG. 2A shows an anomaly and threat detector such as that of FIG. 1;

[0003] FIG. 2B shows a method of operation of an anomaly and threat detector such as that of FIG. 2A;

[0004] FIG. 3 illustrates a method of training an encoder using adversarial training; [0005] FIG. 4 illustrates a method of operation of an adversarially trained encoder as part of an anomaly and threat detector;

[0006] FIG. 5 is a flow diagram of a method of training a decision tree based classifier implementing extreme gradient boosting for use in a classifier of an anomaly and threat detector;

[0007] FIG. 6 is a flow diagram of a method of using a trained decision tree based classifier implementing extreme gradient boosting for detecting anomalies and threats;

[0008] FIG. 7 is a schematic diagram of a computing device in which an anomaly and threat detector is implemented.

[0009] The accompanying drawings illustrate various examples. The skilled person will appreciate that the illustrated element boundaries (e.g., boxes, groups of boxes, or other shapes) in the drawings represent one example of the boundaries. It may be that in some examples, one element may be designed as multiple elements or that multiple elements may be designed as one element. Common reference numerals are used throughout the figures, where appropriate, to indicate similar features.

DETAILED DESCRIPTION

[0010] The following description is made for the purpose of illustrating the general principles of the present technology and is not meant to limit the inventive concepts claimed herein. As will be apparent to anyone of ordinary skill in the art, one or more or all of the particular features described herein in the context of one embodiment are also present in some other embodiment(s) and/or can be used in combination with other described features in various possible combinations and permutations in some other embodiment(s).

[0011] As mentioned above, the early detection and identification of anomalies and threats to a communications network is key to keeping a communications network protected. Network threat detection technology seeks to understand traffic patterns on a communications network and monitor traffic within and between trusted networks, as well as to the internet for any suspicious or malicious activities. Network threat detection technologies enable early identification of potential threats or anomalies on a communications network, which improves the response times to threats compared to other types of threat detection methods. Network threat detection technology is well suited for countering communications network-wide attacks.

[0012] The inventors have recognized that supervised machine learning approaches rely on huge quantities of labelled data which is difficult to obtain. They also find that machine learning anomaly and threat detection approaches are highly vulnerable to evolution of threats and anomalies, especially malware type threats, unless they are updated. However, it is typically not practical to update machine learning anomaly and threat detection systems where large quantities of labelled data are unavailable or where the time taken to carry out updates is prohibitive.

[0013] The inventors have developed an efficient and accurate way of detecting anomalies and threats in network traffic data. In various examples, network traffic data is network metadata. Network metadata is data that is used in order to deliver internet protocol IP packets and which is anonymized so that data identifying individuals is omitted. Network metadata may comprise internet protocol traffic statistics collected from nodes in a core of a communications network. A non-limiting example of network metadata is NetFlow (trade mark) data. NetFlow is a communications network protocol developed by Cisco (trade mark) for collecting IP traffic information and monitoring network flow. The present technology is workable with network metadata collected by other network flow monitoring technology and is not limited to the use of NetFlow. A non-exhaustive list of other types of network monitoring technology is: Argus (audit record generation and utilization system), Jflow, cflowd, NetStream, Cflowd, Rflow, Appflow, sFlow.

[0014] The inventors have found that by using a combination of an adversarially trained encoder and a classifier it is possible to achieve extremely accurate anomaly and threat detection in an efficient manner.

[0015] FIG. 1 of the accompanying drawings shows an anomaly and threat detector 106 deployed with a network traffic data exporter 100 in a communications network 104. In various examples, an anomaly corresponds to a threat. In various examples, an anomaly is benign. In this example the communications network comprises a local area network 118, an intranet 116, the internet 120 and a network traffic data exporter 100. The intranet 116 may be at an enterprise site and the local area network 118 may be at another geographical site However, the example of FIG. 1 is not intended to be limiting as other arrangements of communications network 104 are possible spanning one or more geographical sites. The network metadata exporter 100 is in a core of the communications network 104 and is able to observe traffic flows between trusted parts of the communications network 104 such as intranet 116 and LAN 118 as well as between the internet 120 and trusted parts of the communications network 104. The network traffic data exporter 100 sends network traffic data to a network traffic data collector 102 which stored the network traffic data in store 110. Anomaly and threat detector 106 is computer implemented and has access to network traffic data in store 110 and in network traffic data collector 102. The anomaly and threat detector 106 is able to trigger a security action such as by one or more of: sending an alert to an operator, automatically isolating a node of the communications network 104 such as a node within LAN 118 or the whole of LAN 118, automatically shutting down a node of the communications network 104. The anomaly and threat detector 106 uses a combination of an adversarially trained encoder and a classifier to achieve high accuracy. [0016] FIG. 2A shows an anomaly and threat detector 106 such as that of FIG. 1. The anomaly and threat detector 106 is computer implemented such as using one or more of software, firmware, hardware of a server or compute node. The anomaly and threat detector 106 comprises an adversarially trained encoder 200 and a supervised classifier 202.

[0017] The adversarially trained encoder 200 is a neural network which takes as input network traffic data about packet exchanges between a source and a destination in the communications network 104. In various examples, network traffic data is network metadata. A non-exhaustive, non-limiting list of examples of the network metadata is: IPv4 source address, Layer 4 source port, IPv4 destination address, layer 4 destination port, protocol identifier, layer 7 protocol, number of bytes in, number of bytes out, number of packets in, number of packets out, number of transport control protocol TCP flags, duration of the flow in milli seconds.

[0018] The adversarially trained encoder takes the input network traffic data and computes an encoding of the input network traffic data. The encoding is a vector (also referred to as an embedding) denoting a location in a multi-dimensional embedding space. Due to the adversarial training, flows which are functionally or behaviourally similar have embeddings which tend to be closer together in the multi-dimensional space. Thus an adversarially trained encoder is functionality for computing an embedding of network traffic data describing a flow, where an embedding is a vector in a multi-dimensional space where flows which are functionally or behaviourally similar tend to be closer together in the multi-dimensional space. A flow is an exchange of packets between a source and a destination in a packet-based communications network. The adversarially trained encoder is trained using only benign network traffic data in some cases, that is, only network traffic data about flows which are known to be benign and not anomalous. Note that the vectors are also referred to as latent features and the multi-dimensional space is also referred to as a latent space. Adversarial training means using a critic in training an autoencoder comprising an encoder which produces embeddings which when decoded by a decoder reconstruct the original network traffic data accurately, as determined by the critic.

[0019] The supervised classifier 202 is any machine learning classifier which has been trained using supervised training. A non-exhaustive list of example classifiers which may be used is: support vector machine, decision tree, random decision forest, neural network. Supervised training uses labelled training examples. Each training example is network traffic data for a flow and a label indicating whether the flow is benign or is an anomaly. In various examples, each training example is network traffic data for a flow and a label indicating whether the flow is benign, a threat or an anomaly. In various examples, each training example is network traffic data for a flow and a label indicating whether the flow is benign or a threat, and does not indicate an anomaly. The labelled training examples are obtained from historical network traffic data which is manually annotated as being benign or anomalous, or has been automatically annotated as being benign or anomalous. [0020] FIG. 2B shows a method of operation of an anomaly and threat detector such as that of FIG. 2A. Network traffic data in this example is NetFlow data 260 although the method of FIG. 2A also works with other types of network traffic data. The network traffic data for a flow is processed by an adversarially trained encoder 200 to produce enriched features 262 which are a vector in a multi-dimensional space as mentioned above. The enriched features 262 are concatenated 264 with the network traffic data to produce enriched data 266. The enriched data 266 is input to a supervised classifier 202 which classifies the enriched data 266 as being benign 268 or anomalous 270. If the enriched data 266 is classified as benign the communications network proceeds. If the enriched data is classified as anomalous 270 a security action is triggered. In various examples a security action is triggered such as to send an alert to an operator, terminate the flow, isolate the flow or another security action. In various examples, enriched data classified as anomalous 270 is malicious. In various examples, enriched data classified as anomalous 270 is benign. In various examples, the supervised classifier 202 classifies the enriched data 266 as being malicious, benign 268 or anomalous 270. In various examples, a classification of anomalous is determined when a confidence interval or uncertainty associated with a classification of enriched data as either benign or malicious reflects an uncertainty higher than a pre-defined threshold.

[0021] FIG. 3 illustrates a method of training an encoder 300 using adversarial training. During adversarial training a decoder 302 is used alongside a critic 304. Each of the encoder 300, decoder 302 and critic 304 are neural networks. The training data is network traffic data about flows which are assumed to be benign. Each training data example is network traffic data about one benign flow and is expressed as a vector of numerical values 306. The training data is collected from a network traffic data collector 102 during healthy operation of the communications network 104. The training data does not have to be manually labelled by a person. Since the training data was collected during healthy operation of the communications network 104 it is assumed to be benign. Thus there is plentiful training data available for the adversarial training.

[0022] During training a training example 306 is encoded using the encoder 300 into a vector. The vector is input to the decoder 302 as indicated in FIG. 3. The decoder reconstructs the training example. The reconstructed training example 308 is compared with the training example 306 by computing a reconstruction error 312 (such as a difference between vector 306 and vector 308). The reconstruction error 312 along with a critic score 310 are used to determine what updates to make to values of weights of nodes of the encoder, decoder and critic neural networks during a backpropagation process. The weights are updated so as to reduce the reconstruction error 312 and maximize the difference between the critic score on reconstructed and on network traffic data.

[0023] Because the encoder and decoder are trained using only benign network traffic data the decoder is assumed to be poor at reconstructing threat network traffic data.

[0024] A critic 304 is further used as an adversary during training. Using a critic addresses a known generalization problem where the encoder and decoder have good generalization ability and are able to encode anomalous or threat traffic data so that it can be reconstructed well by the decoder. The critic 304 is a neural network that takes as input the reconstructed data 308 from the decoder 302 as well as the input network traffic data. The critic computes a score on each of these inputs called a critic score 310 which is a numerical value indicating whether the input data is reconstructed or whether it is the original network traffic data. Where a critic 304 is used the weights of the encoder and decoder are updated along with those of the critic so as to both reduce the reconstruction error 312 and make the critic score distinguish between original and reconstructed data. This way of training the autoencoder is called adversarial training where a critic network functions as an adversary to the autoencoder. Thus the trained encoder is more tightly bound to the data it is trained on (normal traffic in this case) and reduces the generic nature of the encoder thereby addressing the generalization problem to an extent. Additionally, the use of a critic is helpful in the present context as the more tightly bound encoder to normal traffic improves the detection of anomalies (i.e. traffic that is different beyond a threshold to normal traffic) which may indicate a threat. This increases the chance of detecting a threat that may have evolved over time beyond known threats and that may not be identifiable if a network had been trained only on previously known threats.

[0025] In a non-limiting example the encoder 200 is 4-layer deep convolutional autoencoder that “encodes” the input data 402 by learning latent features at each layer to output an encoding of the original data. This encoding captures a summary of the data in a way as it is of a smaller dimension than the original data and can be used to reproduce the original data by the decoder 302. The decoder 302 may have an architecture similar to the encoder 200 but uses transpose-convolutions to expand the encoding in each step to produce a reconstructed data that is ideally 100% similar to the input data 306. The critic is identical to the encoder in terms of layers except with the replacement of the final layer to output critic score instead of the encoding.

[0026] The encoder 300, decoder 302 and critic 304 are trained in tandem on solely normal (benign) traffic flows with the objective of minimizing the reconstruction error (difference between the original and reconstructed data) and maximizing the critic score difference between original and reconstructed network traffic data. This ensures that the encoder and decoder only learn to reconstruct normal traffic and not anomalies thus the reconstruction error for anomalous flows will be high and the value can be used to distinguish between real and anomalous flows after a threshold is placed. Note that other architectures may be used for the encoder and decoder. The encoder 300, decoder 302 and the critic 304 are trained using any suitable backpropagation process such as the Adam process or other gradient descent process.

[0027] In an example the loss function for training the encoder and decoder is a mean- squared distance between the network traffic data and the reconstructed network traffic data. The loss function for training the critic is a constant times the critic score on the reconstructed traffic data minus the mean-squared difference between critic score on the network traffic data and the critic score on the reconstruction of the network traffic data. The two loss functions are added together to form one loss function to train the encoder, decoder and critic in tandem. Note this example is non-limiting as other loss functions using other distance measures and other aggregations of the critic score and the reconstructed traffic data or the critic score and the network traffic data are used in some cases. Rather than adding the two loss functions they may be aggregated in other ways such as a weighted sum.

[0028] FIG. 4 illustrates a method of operation of an adversarially trained encoder as part of an anomaly and threat detector. Input data 402 comprising a vector of network traffic data for a single flow is received at an adversarially trained encoder 200. The encoder 200 encodes the input data 402 to produce encoding 404 which is more compact than the input data 402. Because the encoder was trained adversarially using benign network traffic data the encoding 404 encodes benign network traffic data 402 well and encodes threat network traffic data poorly. The encoding 404 is concatenated to the input data 402 to create enriched features 266. In the case of benign network traffic data the encoding 404 represents the benign data strongly and the input data 402 is also benign. In the case of threat or anomalous network traffic data the encoding 404 is weak and the input data 402 represents threat or anomalous network traffic data respectively.

[0029] The enriched features 266 are input to a classifier such as a decision tree implementing extreme gradient boosting which predicts a classification of either benign 268 or anomalous 270. In various examples, the classifier predicts a classification of benign 268, threat 272 (i.e. malicious), or anomalous 270. In various examples, a classification of anomalous 270 is determined when an uncertainty associated with a classification of benign 268 or threat 272 reflects an uncertainty above a pre-defined threshold.

[0030] Note that although FIG. 4 shows a decision tree implementing extreme gradient boosting it is also possible to use other classifiers such as other types of decision tree including a random decision forest, support vector machines or neural networks. [0031] FIG. 5 is a flow diagram of a method of training a decision tree that implements extreme gradient boosting, for use in a classifier of an anomaly and threat detector 106. Supervised training is used. The training data is a plurality of labelled training examples. Each training example is network traffic data concatenated with an embedding from the encoder 200 which has already been trained. Each training example is known to denote a threat or a benign flow. In a non-limiting example there are 8848 training examples of which 88.5% were benign and 11.5% were known to be anomalous, corresponding to threats. The training examples are obtained empirically by collecting data from a real communications network or by synthetic generation from a simulation of a communications network.

[0032] A decision tree comprises a root node. The root node branches into two split nodes. Split nodes, also referred to as internal nodes have further branches. Split nodes terminate at leaf nodes.

[0033] A number of trees is set 500 for use in a forest. In an example there are around 100 trees although other numbers of trees can be used. During training the structure of a tree is grown and data is stored at the leaf nodes. The stored data is labelled training examples which are network traffic data labelled as benign or anomalous. In various examples, the network traffic data is labelled as benign, threat or anomalous. In various examples, anomalous corresponds to a label of benign or threat but with an associated uncertainty that reflects an uncertainty in the label greater than a pre-defined threshold. The stored data may be statistics describing the labelled training examples at the leaf node, such as a histogram or a mean or mode. During training criteria are learnt for making decisions at each split node.

[0034] A root node of a first tree is selected 502. A set of all possible criteria are generated 504 for use at the root node, where each criteria of the set of all possible criteria is tested on training examples and selected 506 or not as the criteria to use at the root node according to an objective. In an example the objective is to maximise an information gain whilst achieving a gain above a predefined threshold, where any suitable measure of information gain is used such as variance of a distribution of the training examples. If the objective at the root node is met 508 by a training example (i.e. the root node is not going to be a leaf node) the process moves to operation 510 where the root node is set to be a split node and training examples which pass the criteria test of the root node are passed to the split node as a child node. The child node to which the training example will be passed is determined by a binary test 522 using the criteria generated for use at the node 504. The process of operations 504-510 are recursively executed for the subset of training example data directed to the child node, where reference above to the root node is replaced by reference to a current node. If the criteria are not met at operation 508, the root node is set to be a leaf node 518. The process waits 520 until all branches of the first tree are trained, such that the training data is partitioned to a predefined degree by the first tree into groups corresponding to the known labels of the training data, where a trained first decision tree predicts the label of a training example. In various examples, training data that reaches a leaf node is stored as a histogram or other statistics.

[0035] A residual is calculated 524 for each training example based on the difference between the known label of the training example and the label of the training example output by the first tree. The method then iterates 526 to a subsequent tree of the forest, ending 528 only when the method has iterated through all trees in the forest, where the number of trees was set 500 previously. When iterating 526 to a subsequent tree of the forest, the subsequent tree is trained in the same way as the first tree, but instead of being trained to predict the label of each training example as in the first tree, a subsequent tree is trained to predict a residual of all prior trees for each training example, such that the residual is based on the difference between the known label of the training example and the sum of the outputs of all prior trees. After training of the subsequent tree, a new residual is then calculated 524 for each training example, and the method iterates 526 to a subsequent tree unless the method has iterated through all trees in the forest.

[0036] FIG. 6 is a flow diagram of a method of using a trained decision tree implementing extreme gradient boosting for detecting anomalies and threats. Unseen original network traffic data 600 is received, where original network traffic data is generated from core network infrastructure. Aggregation and/or pre-processing of the original network traffic data is then performed, to produce aggregated (pre-processed) network traffic data 602. Encoded data which is produced by processing network traffic data through an encoder such as described herein, for example in FIG. 4, and aggregated network traffic data 602, are then concatenated 604 into concatenated data, and the concatenated data is processed through a trained decision tree classifier such as a classifier as described above. Processing the concatenated network traffic data through a trained decision tree classifier implementing extreme gradient boosting as described above comprises processing each data item of the data through every decision tree of the decision tree classifier 608 to determine a leaf node i.e. a termination point of every decision tree, and computing the sum of all leaf node outputs 610. Computing the sum of all leaf node outputs 610 corresponds to obtaining a prediction of the label of the concatenated network traffic data from the decision tree classifier implementing extreme gradient boosting. This prediction corresponds to a classification of the concatenated network traffic data as benign or anomalous. In various examples, the concatenated network traffic data is classified as benign, threat or anomalous. In various examples, anomalous corresponds to a predicted classification of benign or threat but with an associated uncertainty that reflects an uncertainty in the classification that is greater than a pre-defined threshold. If the prediction is anomalous then a security action is triggered 514. In various examples, the security action functions to remedy or counter a consequence or a potential consequence associated with the anomalous prediction. In various examples, if the prediction is threat then a security action is triggered 514.

[0037] In examples where the classifier is a decision tree implementing extreme gradient boosting the XGBoost (trade mark) open source software library may be used. However, any suitable decision tree implementing gradient boosting may be used which implements a Newton-Raphson in function space and uses a second order Taylor approximation in a loss function.

[0038] The inventors have tested the technology in a simulated communications network. There are various kinds of network traffic data that are generated from core network infrastructure, including NetFlow, domain name system DNS, dynamic host configuration protocol DHCP and others. The simulated communications network tested by the inventors used NetFlow network traffic data. The technology which was tested comprised the 4 layer encoder architecture mentioned above combined with the decision tree implementing XGBoost and is referred to as “Model X”.

[0039] The Model X was trained on 8846 flows overall with 2011 flows used for testing. Both the autoencoder and the classifier of the Model X were trained on the flows, however only the benign training data was used to train the autoencoder. Each NetFlow data row contained 17 features which were extracted from simulated network traffic data in the simulated communications network tested by the inventors. A port-scan attack is used by malicious parties to identify vulnerabilities in a communications network prior to launching an attack. A port scan attack helps cyber criminals find open ports and figure out whether they are receiving or sending data. It can also reveal whether active security devices like firewalls are being used by an organization. When hackers send a message to a port, the response they receive determines whether the port is being used and if there are any potential weaknesses that could be exploited. In the simulated communications network port-scan attacks were simulated.

[0040] It was found that the inference time for Model X was less than for other approaches where the encoder is replaced by outlier mining processes.

[0041] It was found that the accuracy of Model X was greater than when only a decision tree utilising extreme gradient boosting is used, or when using other approaches where the encoder is replaced by outlier mining processes. For Model X precision was 0.87, recall was 0.83 and F1 score was 0.85. In contrast when using a random forest alone, precision was 0.85, recall was 0.80 and F1 score was 0.82. When using other approaches where the encoder is replaced by an ensemble algorithm comprising 41 outlier mining processes, precision was 0.88, recall was 0.79 and F1 score was 0.83.

[0042] It was found that Model X is more resilient to a decrease in the amount of labelled data available. Where only 30% of the training data examples were labelled the precision, recall and F1 score of Model X outperformed a supervised approach using only a decision tree implementing extreme gradient boosting.

[0043] FIG. 7 is a schematic diagram of a computing device in which an anomaly and threat detector is implemented. FIG. 7 illustrates various components of an example computing device 706 in which example anomaly and threat detectors are implemented in some examples. The computing device is of any suitable form such as compute node in a data centre, a communications network management node, a node comprised in a cloud computing environment, a node comprised in an edge deployment, or other node.

[0044] The computing device 706 comprises one or more processors 702 which are microprocessors, controllers or any other suitable type of processors for processing computer executable instructions to control the operation of the device in order to perform the methods of FIGs. 2B, 3, 4, 5 and 6. In some examples, for example where a system on a chip architecture is used, the processors 702 include one or more fixed function blocks (also referred to as accelerators) which implement a part of the method of FIGs. 2B, 3, 4, 5 and 6 in hardware (rather than software or firmware). That is, the methods described herein are implemented in any one or more of software, firmware, hardware. The computing device has an encoder 200 and a classifier 202. Platform software comprising an operating system 710 or any other suitable platform software is provided at the computing-based device to enable application software 712 to be executed on the device. Although the computer storage media (memory 700) is shown within the computing-based device 706 it will be appreciated that the storage is, in some examples, distributed or located remotely and accessed via a network or other communication link (e.g. using network interface 704).

[0045] Any reference to 'an' item refers to one or more of those items. The term 'comprising' is used herein to mean including the method blocks or elements identified, but that such blocks or elements do not comprise an exclusive list and an apparatus may contain additional blocks or elements and a method may contain additional operations or elements. Furthermore, the blocks, elements and operations are themselves not impliedly closed.

[0046] The steps of the methods described herein may be carried out in any suitable order, or simultaneously where appropriate. The arrows between boxes in the figures show one example sequence of method steps but are not intended to exclude other sequences or the performance of multiple steps in parallel. Additionally, individual blocks may be deleted from any of the methods without departing from the spirit and scope of the subject matter described herein. Aspects of any of the examples described above may be combined with aspects of any of the other examples described to form further examples without losing the effect sought. Where elements of the figures are shown connected by arrows, it will be appreciated that these arrows show just one example flow of communications (including data and control messages) between elements. The flow between elements may be in either direction or in both directions.

[0047] Where the description has explicitly disclosed in isolation some individual features, any apparent combination of two or more such features is considered also to be disclosed, to the extent that such features or combinations are apparent and capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

1. A computer implemented method comprising: accessing network traffic data collected from a node in a core of a communications network; computing an encoding of the network traffic data by processing the network traffic data using an adversarially trained encoder; concatenating the encoding with the network traffic data to form an input; classifying the input as benign or anomalous; and in response to a classification of anomalous, triggering a security action.

2. The method of claim 1 comprising computing the encoding of the network traffic data by processing the network traffic data using an adversarially trained encoder and bypassing a decoder and a critic that was used to adversarially train the encoder.

3. The method of any preceding claim wherein the network traffic data excludes raw packet data.

4. The method of any preceding claim wherein the network traffic data comprises internet protocol traffic statistics.

5. The method of any preceding claim wherein classifying the input comprises using a trained decision tree.

6. The method of any preceding claim wherein classifying the input comprises using extreme gradient boosting.

7. The method of any preceding claim comprising triggering a security action by one or more of: automatically isolating a node of the communications network, sending an alert message to an operator, automatically deactivating a node of the communications network.

8. The method of any preceding claim comprising training a classifier using supervised learning and using the trained classifier to carry out the classifying.

9. The method of any preceding claim comprising training the encoder using a loss function that takes into account a critic score and a reconstruction error.

10. The method of any preceding claim comprising training the encoder using training data comprising only benign network traffic data.

11. An apparatus comprising: a processor; a memory storing network traffic data collected from a node in a core of a communications network; an adversarially trained encoder; a classifier; wherein the memory stores instructions which when executed on a processor: encode the network traffic data using the adversarially trained encoder to produce an encoding; concatenate the encoding with the network traffic data to form an input; classify the input as benign or anomalous using the classifier; and in response to a classification of anomalous, trigger a security action.

12. The apparatus of claim 11 absent a decoder and wherein the network traffic data excludes raw packet data.

13. The apparatus of claim 11 wherein the classifier is a decision tree implementing extreme gradient boosting.

14. The apparatus of claim 11 wherein the instructions comprise training the encoder using only benign network traffic data.

15. A communications network comprising the apparatus of any of claims 11 to 14.