ES2577143T3 - Method and system to detect malicious software - Google Patents

Abstract

Method to detect malicious software, said detection being performed in an anomaly detection system, or ADS, analyzing the behavior of a network and looking for deviations from a normality, indicating said normality the common behavior of users of said network and defining before said detection, said method comprising: - constructing a plurality of detection models for each of a plurality of different entities of said network, each of said plurality of detection models adapted to said different entities of said network and to different algorithms, implementing said different algorithms different detection strategies and representing said plurality of detection models said normality, and - representing said plurality of detection models in a two-dimensional matrix, a dimension of said matrix corresponding to a number of said different entities of said network and corresponding the other dimension of said matrix to a number of said different algorithms used.

Description

5

10

fifteen

twenty

25

30

35

40

Four. Five

fifty

55

60

Method and system to detect malicious software DESCRIPTION

Technique Field

The present invention relates, in general, in a first aspect, to a method for detecting malicious software, said detection being performed in an anomaly detection system, or ADS, analyzing the behavior of a network and looking for deviations from normality. , said normality indicating the common behavior of users of said network and being defined before said detection, and more particularly to a method comprising constructing a plurality of detection models, each of said plurality of detection models being adapted to different entities of said network and different algorithms, implementing said different algorithms different detection strategies and representing said plurality of detection models said normality.

A second aspect of the invention relates to a system arranged to implement the method of the first aspect.

State of the prior art

Malicious software (malware) detection can be classified in many ways. One of them is the classic categorization that distinguishes between host-based and network-based detection. The first type tries to find evidence of the existence of viruses, Trojans, etc. through processes, physical memory and several other host analyzes, while the second focuses on communications made by such viruses or Trojans.

Several strategies emerged at the beginning of network based detection. The most basic was to block certain communications that go through a special network node, a firewall. This solution was useful until malware, in order to hide, began using widely used protocols such as HTTP, SMTP that cannot be blocked without stopping the business of ISPs and operators.

Then, it was obvious that it was not enough to analyze a few fields of TCP / IP packets (protocol, source and destination ports and IP addresses), and it was necessary to extend the monitoring process to the full TCP / IP header and even The payload of the packages. This is how network intrusion detection systems (NIDS or IDS) are born [15]. IDSs are based on traffic signatures, that is, special chains that, when they appear within network packages, indicate the presence of specific malware. The widely known IDS are Snort [1] and Bro [2].

Although IDS currently continues to play an important role in the monitoring processes of all operators around the world, since they deal with a significant percentage of malware detection, the researchers found that some malware remains undetectable to firewalls and IDS. These viruses, worms, etc. undetectable used both popular protocols and normal content a priori within the payload of packages. Some examples are: spammers [3], service-oriented denial (DoS) bots [4] or scanners [5]. Then, it was observed that the only way to detect these threats was to analyze the behavior of the network in order to find deviations from normality, that is, anomalies. Normality is defined through a mathematical representation of common reality, that is, a model, which is constructed at a stage prior to detection. Network anomaly detection systems (NADS) [14] or simply anomaly detection systems (ADS) such as Proventia [6] quickly proved useful in covering the gap mentioned in the monitoring field. And also in many other scenarios, such as zero-day malware detection (new existing malware for which IDS does not have a valid signature) or encrypted traffic (signatures are not important when the payload cannot be seen).

The detection of anomalies is still an interesting research area that can provide many solutions for security administrators. Innovative artificial intelligence (AI) algorithms are appearing, and the objective of modeling, which is sometimes more important than the detection algorithm itself, is among the most general granularity, the entire network, to a minimum, the end user individual.

Examples of anomaly detection systems can be found in documents US2007 / 289013 and US8015133.

Two types of problems can be distinguished when talking about existing ADS, those related to effectiveness and those related to efficiency.

The problems of effectiveness are evident on the day after day of all the comparuas. Their already implemented systems do not detect everything, and the things they must detect are not properly found due to strategies and

5

10

fifteen

twenty

25

30

35

40

Four. Five

fifty

55

60

obsolete mechanisms

- Network based behavior modeling

Models with respect to the behavior of monitored networks are always based on aggregation, counting the number of packets / flows / bytes per unit of time and thus defining a reference. Although it is true that this approach is sufficient to face important threats such as DoS attacks [4], in which a large amount of data is generated, it has nothing to do with more discrete and sophisticated attacks based on very low variations of certain characteristics. of traffic that can only be seen at the level of individual entity.

- Rudimentary detection algorithms

The condition-consequence and volume / threshold rules are almost all the technologies implemented in the majority of successful NADS. These are very basic algorithms that are difficult to maintain (new conditions must be considered when a threat evolves), showing very low flexibility (a threshold of 10,000,000 bytes exceeded when 100,000,000 bytes are reached) and poor self-adaptation.

Most of the problems related to efficiency with respect to existing solutions are due to the lack of a common framework for monitoring purposes. Each solution is proprietary and is very closed to third parties, which causes organizations to install a new solution pool every time a new threat arises.

- Level of zero integration between suppliers

Except SIEM [9], a special class of systems not responsible for direct network monitoring, but responsible for the aggregation and correlation of events from different sources, integration between current monitoring systems (including ADS) is not allowed, and not only monitoring systems: sniffing tools, registration applications, etc. For example, it is very common for IDS / IPS / ADS to sniff through network traffic, while many snooping systems perform the same task.

- Low level of customization / extension

One of the main problems that the monitoring and ADS systems have in particular with respect to their architecture is the lack of flexibility. The solutions of detection of anomaftas in research and current commercials are very closed and are proprietary, which makes almost any kind of customization / extension impossible. This low level of customization may seem normal in any other kind of software system, but not in the field of monitoring, in which security operators need to face the continuous evolution of malware through new algorithms, strategies, sources of data, etc.

- Multiple monitoring points

In addition, if an organization expects to use several NADS (because each of these targets a different threat) then each individual NADS will require an exclusive monitoring point from which to obtain the unprocessed traffic. This may seem an insignificant problem if providing monitoring points is not expensive. On the one hand, the physical bypass solutions must cut the cable for a few seconds, but enough to stop critical applications, and the optical power division must be done very carefully in order to allow the end nodes to continue negotiating the link. . On the other hand, logical derivations such as port duplication consume a high amount of processing capabilities in network nodes; Additionally, each monitoring application needs an exclusive port on the node.

- Monolithic applications

Finally, existing solutions are designed to run on a single computer, avoiding a desirable distribution of processing characteristics. Monolithic applications usually require large amounts of resources such as CPU, RAM and disk to deal with high-speed networks.

Description of the invention

It is necessary to offer an alternative to the state of the art that covers the gaps found therein, particularly in relation to the lack of proposals that actually allow detecting all possible malicious software and implement a common framework for monitoring purposes so that Organizations do not need to install a new solution pool every time a new threat arises.

5

10

fifteen

twenty

25

30

35

40

Four. Five

fifty

55

60

For this, the present invention provides, in a first aspect, a method for detecting malicious software, said detection being performed in an anomaly detection system, or ADS, analyzing the behavior of a network and looking for deviations from normality, indicating said normality the common behavior of the users of said network and being defined before said detection.

Unlike the known proposals, the method of the invention, in a characteristic way, also comprises constructing a plurality of detection models, each of said plurality of detection models being adapted to different entities of said network and to different algorithms, implementing said different algorithms different detection strategies and representing said plurality of detection models said normality.

Other embodiments of the method of the first aspect of the invention are described according to the appended claims 1 to 15, and in a subsequent section relative to the detailed description of various embodiments.

A second aspect of the present invention relates to a system for detecting malicious software, said detection being carried out in an anomaly detection system, or ADS, analyzing the behavior of a network and looking for deviations from a normality, said normality indicating common reality of said network and defining itself before said detection.

In the system of the second aspect of the invention, unlike the known systems mentioned in the prior art section of the prior art, and in a characteristic way, this comprises a probe module for monitoring traffic of said network connected to a controller module responsible for performing said detection, wherein said controller module is provided with a plurality of detection models constructed by means of a compiler module, each of said plurality of detection models being adapted to different entities of said network and to different algorithms, implementing said different algorithms different detection strategies and representing said plurality of detection models said normality.

The system of the second aspect of the invention is adapted to implement the method of the first aspect.

Other embodiments of the system of the second aspect of the invention are described according to the appended claims 16 to 22, and in a subsequent section relating to the detailed description of various embodiments.

Brief description of the drawings

The foregoing and other advantages and features will be more fully understood from the following detailed description of embodiments, with reference to the attached drawings, which should be considered in an illustrative and non-limiting manner, in which:

Figure 1 shows the components and the interaction between them, said components and interactions defining a possible architecture of the proposed system, according to an embodiment of the present invention.

Figure 2 shows the probe module, according to an embodiment of the present invention.

Figure 3 shows the monitoring module and the detector library, according to an embodiment of the present invention.

Figure 4 shows the compiler module and the compiler library, according to an embodiment of the present invention.

Figure 5 shows the registrar module and the registrar library, according to an embodiment of the present invention.

Figure 6 shows a possible sequence diagram between the monitoring module and the compiler module, in which the monitoring module and the compiler module make use of the compiler-monitor interface, according to an embodiment of the present invention.

Figure 7 shows the flow chart for the DNS-based domain flow detector, according to an embodiment of the present invention.

Figure 8 shows a graphical representation of clusters when using suspicious traffic clustering, in which normal vectors are represented by "0", anomalies by "x" and clustering by circles, according to an embodiment of the present invention.

Figure 9 shows a graphical representation of 2 characteristics of the rapid group membership algorithm, according to an embodiment of the present invention.

Figure 10 shows an example of distributed physical implantation of the system components, according to an embodiment of the present invention.

Detailed description of various embodiments

The proposed invention relates to hardware and software equipment (or set of equipment if its components are

5

10

fifteen

twenty

25

30

35

40

Four. Five

fifty

55

60

finally distribute) which acts as a platform that allows much more efficiency and effectiveness in the field of network anomaly detection.

The efficiency gain is obtained thanks to the combination of two innovative approaches:

- The use of detection models in one mode per user. Instead of creating a single model for the entire network, each entity within it is modeled.

- The use of detection models in one mode per algorithm. Instead of having a unique detection strategy, multiple detector examples run in parallel.

The two previous approaches can be seen as a model matrix in which N entities (Ei ... En) and M algorithms (Ai ... Am) are considered, obtaining NxM different models (Mii ... Mnm).

The model and its respective detectors can be either ADS of the state of the art (they can be considered simple 1x1 matrices, since a single modeled entity and only one algorithm are used) or innovative detection algorithms, such as the two explained below based on artificial intelligence.

In addition, other architectural features available in the proposed invention allow much more efficiency in terms of:

- The distribution of the processing effort among several physical nodes.

- Easy integration of software related to existing monitoring, such as network traffic processors, through architecture design.

- A single monitoring point in the network.

• Architecture components

The system components and the interaction between them are shown in Figure 1. The light gray boxes represent SoA components, the dark gray represents the modified SoA and the white boxes, which include the libraries of detectors and compilers, are innovative components within the architecture. Next, the system components will be detailed:

-PROBE

It is the only monitoring point that provides traces of network traffic to the rest of the system components.

The detection of anomalies performed by ADS applications of multiple algorithms is based on network traffic, specifically looking for deviations from normality in the TCP / IP packets that cross the monitored network. To do this it is necessary (1) to capture the physical media packets and (2) to prepare the captured packets for the detection algorithms, which usually work with aggregate flows (a flow is defined, at least, by the transport protocol, TCP or UDP, source and destination IP addresses and source and destination ports). The PROBE is the component responsible for packet capture and flow processing, as shown in Figure 2.

The PROBES can also be inherited, for which it is necessary to simply implant an adapter in order to achieve the interface between the PROBE and the MONITOR.

- MONITORING AND DETECTORS LIBRARY

The MONITORIZER is the main controller of the system, receives network traces from the PROBE and is responsible (1) for the storage of traces if it works in training mode and (2) the invocation of DETECTORS, passing them a copy of the traces, if it works in detection mode. The DETECTORS LIBRARY is a collection of DETECTORS that each implement an NADS algorithm.

Once the network traffic is added in flows, it is ready to be analyzed using a wide variety of anomaly detection algorithms. However, it should be remembered that the traffic can either be stored (in order to build normal models) or used by the detection algorithms. Thus, some entity must (1) know if the system is currently operating in detection or storage mode and (2) forward the traffic to the storage component or to the detection entities depending on the operating mode. The proposed invention implements this through the MONITORIZER.

With regard to detection, since the algorithm folder cannot be complete (impossibility of implementing all existing algorithms, algorithms that have not yet been discovered) it will be very convenient

5

10

fifteen

twenty

25

30

35

40

Four. Five

fifty

55

60

have a mechanism to adjust the detection system (add, delete or modify algorithms). The system architecture provides such a docking mechanism through the DETECTORS LIBRARY. The DETECTORS LIBRARY is a discrete collection of detection algorithms, the current collection, which has a defined interface with the MONITOR. A new detection algorithm, formerly a DETECTOR, can be included in the collection if such DETECTOR respects the interface defined with the MONITOR. All DETECTORS within the library do not always have to be used, and if there is a possibility to list the desired subset of algorithms for the MONITOR. This can be done, for example, by means of configuration files.

The MONITORIZER then simply resends, as previously stated, through the defined interface, the flows received to all DETECTORS within the DETECTORS LIBRARY. Finally, DETECTORS compare incoming flows with stored normal behavior models.

The DETECTORS LIBRARY is originally conceived, precisely, as a library that is natively incorporated into the MONITORING software (static or dynamic libraries in C / C ++, JAR files in Java, etc.). The purpose of this decision is to improve the delivery of flows to the DETECTORS, but the developer can implement the library as a set of processes that are executed independently of the MONITOR, for example; even, DETECTORS can be placed in separate physical devices.

- COMPILER and COMPILER LIBRARY

The COMPILER is responsible for invoking COMPILERS when the system operates in training mode. It runs once the trail storage phase ends. The COMPILERS LIBRARY is a compilation of COMPILERS that each implement a behavior modeling mechanism.

The COMPILER is the module of the ADS architecture of multiple algorithms involved in the generation of normal behavior models and, therefore, is the central component of the proposed invention. As is known, anomaly detection algorithms generally compare the observed traffic with a normal model, looking for deviations between them. Since the detection algorithms can be very different, then their reference models will also be very different despite the fact that the data used to generate them are the same (the traffic captured and aggregated). The COMPILER is responsible for generating this generation of differentiated models through the COMPILER LIBRARY, which contains a model generator, previously a COMPILER, by detection algorithm in the system. COMPILERS within the library can be enabled or disabled.

However, the COMPILER is an optional component since some DETECTORS may not need a custom model (a default model is always used independently of the specific behavior of the users); or the models may have been generated by other means; or even the DETECTOR does not need a model.

- REGISTER AND REGISTER LIBRARY

The REGISTER receives alerts from the MONITOR and invokes the REGISTERS. The REGISTER LIBRARY is a compilation of REGISTERS each implementing a registration facility.

Finally, the generated collaborative alarms can be registered, a task for which the REGISTER component is introduced into the architecture (once again, an exhaustive registration folder is unfeasible, so a REGISTER LIBRARY is used to dynamically add or delete facilities Registration: files, databases, syslog, etc.). Existing REGISTERS can be enabled or disabled in the configuration.

• Architecture interfaces

Next, the interfaces between the system components will be described:

- PROBE - MONITORING

This interface defines how the flows generated in the PROBE component are sent to the MONITORING component. Communications, basically, can implement two different schemes proactively depending on the PROBE:

1. Proactive exchange: the PROBE sends the flows at the moment they are ready to share.

5

10

fifteen

twenty

25

30

35

40

Four. Five

fifty

55

60

2. Exchange on request: flows are sent when requested by the MONITOR.

The use of one or the other scheme has an important impact on performance in real time. On the one hand, it is obvious that the exchange on request is not compatible with real time since data forwarding depends on the availability of the MONITOR. On the other hand, it can be thought that proactive exchange is always compatible with real time; but this may not be true if the MONITORING component again buffers proactively received data.

In any case, the format of the exchanged flows must be established in order to guarantee the correct interaction between the PROBES and the MONITOR. If legacy data sources are used then an adapter must be implemented in order to ensure its availability. A non-exhaustive list of fields that can be exchanged between the PROBES and the MONITOR is:

- Layer 4 protocol (TCP, UDP, etc.).

- Date and time stamps of the first and last package.

- IP addresses of origin and destination in the flow.

- TCP or UDP ports of origin and destination in the flow.

- Number of packages sent and received.

- Number of bytes sent and received.

- TCP status or at least TCP flags involved in the flow.

- Certain application header data, such as DNS, HTTP, SMTP, SIP and others.

- MONITORIZER - COMPILER

As indicated in the specific MONITORIZER and COMPILER sections, communication between these two components of the proposed invention architecture is done through databases. There are two reasons for this:

1. The high amount of captured streams does not allow simple buffer storage.

2. The majority of anomaly detection algorithms need ample time windows to perform their normal behavior models. Naturally, a real-time model construction is possible in some anomaly detection algorithms, but, since this task is not critical, it will always be done in a non-real-time mode.

The implemented database must allow the MONITORIZER to store the flows built in the exchange format agreed between the SONDa and the MONITORIZER (see previous section). These stored flows can be consulted by the COMPILER as well, which will generate the models and store them in another database.

- MONITORING - REGISTER

The alarms generated by the MONITORIZER are sent to the REGISTER component in a specific format, for which the candidate fields could be the following:

- Identifier of the attacker.

- Identifier of the attacked host.

- Protocol involved.

- Date and time stamp for the alert.

- Type of attack / type of anomaly.

- DETECTOR identifier.

- Feedback information such as original data that causes the alarm to be set in the DETECTOR, extended target list (if there is not one), etc.

• Detectors

The system provides for the use of advanced algorithms to improve the efficiency of anomaly detection systems. These advanced algorithms come from the field of artificial intelligence, with neural networks and clustering algorithms the main candidates for use.

Neural networks and other supervised machine learning algorithms [10] have demonstrated several times their potency in classification problems, due to their ability to generalize solutions through a few training examples; its adaptability; and its low rate of false positives.

However, it is not always possible to use supervised algorithms, especially when there is no expert who

5

10

fifteen

twenty

25

30

35

40

Four. Five

fifty

55

can label or classify the training examples beforehand. When this occurs, unsupervised algorithms [11] such as clustering are needed. Clustering is very useful in auto-generation of traffic classes, behaviors, etc.

In this case, a couple of DETECTORS that are part of the DETECTORS LIBRARY are detailed. These DETECTORS are examples of the advanced detection algorithms contemplated for the proposed multi-algorithm ADS. Other simple algorithms such as volumetric traffic monitoring, absolute packet counts or flows between nodes, and periodic flow detection are not described since they are not innovative algorithms although they can be perfectly developed on the proposed invention.

Additional DETECTORS not documented in this document may be added to the ADS of multiple algorithms in the form of extensions of the current patent.

- DNS based domain flow detector

This DETECTOR attempts to detect domain flow activities [12] in the monitored DNS traffic, which may indicate the presence of a bot [7].

Bots within a botnet usually implement DNS queries in order to discover their command and control (C&C) server; This allows bot owners to change the actual location (IP address) of the server without reconfiguring their bots. While the fixed FQDNs are easy to detect and filter, the bots implement the domain flow technique, which dynamically generates a high amount of FQDN for the C&C server in order to synchronize with the owner. These dynamically generated FQDNs can be based on the current date and time stamp, or they can be a set generated in a pseudorandom manner, being only one of the possibilities valid. In any case, this technique generates many responses from NX_DOMAIN (and others) when querying the DNS server. This detector analyzes these anomalous responses.

Therefore, DNS responses are analyzed and a set of features is extracted within a time interval, the following being an example of a set of features:

- Number of responses of NX_DOMAIN.

- Number of responses from FORMAT_ERROR.

- Number of REFUSED responses.

- Number of responses from SERVER_FAILURE.

- Number of different FQDN consulted.

- Number of FQDN of a layer

- Number of two layer FQDN.

- FQDN number of three layers or more.

- Number of suspicious top-level domains (TLD).

- Number of layer two domains with length less than 6.

- Number of layer two domains with length greater than 5 and less than 21.

- Number of layer two domains with length greater than 20.

- Number of layer two domains with number of vowels less than 0.3%

- Number of layer two domains with number of digits greater than 0.5%

These features make up a feature vector that is evaluated using a neural network. This neural network is previously trained (monitored) using examples of vectors that contain values for all the previous features and providing an additional field, a label. This label will provide information regarding the convenience of setting an alarm or not when a similar vector is found in the traffic.

The DETECTOR therefore uses a unique default model, previously constructed, the specific neural network configuration that results after the training phase.

A flow chart was provided in Figure 7 to illustrate the proposed DETECTOR, in which some variables are described in the following table:

 Variable name

 Description

 F

 Flow

 sid

 Subscriber Identifier

 m

 Model for the subscriber identifier (currently one by default)

 td

 Is the current flow related to a higher domain?

5

10

fifteen

twenty

25

30

35

40

Four. Five

fifty

 ts

 Date and time stamp for current flow

 tw

 Start date and time stamp of the current time window

 tw '

 updated tw

 s

 Time window size

 ctw

 Is the current flow in the current time window?

 to

 Alert

 v

 Feature vector obtained from the current flow

 gv

 Global cumulative vector with respect to the current time window

 gv '

 updated gv

• Grouping of suspicious traffic

The fundamentals of this DETECTOR are based on the construction of a customized model with respect to the values of certain characteristics within an interval period. Unlike the previous DETECTOR, no expert is used to train it, that is, an unsupervised algorithm will be used. Specifically, a clustering algorithm for maximizing hope (MS) [13] will define the traffic classes in which each user is commonly involved. Then, in a second stage, anomalies will be detected if a different traffic pattern appears than normal. Graphically, this process is shown in the simplified 2-character image of Figure 8, in which the circles represent clusters of data, the "o" represents a normal vector and an "x" is an anomaly.

However, the key point is not the algorithm but the characteristics of the vectors that will feed the EM implementation. Features such as the number of packets / bytes / streams sent and received appear to be valid candidates, but these characteristics are not statistically stable over time. On the contrary, stable characteristics are needed; Those related precisely to abnormal behaviors are better. This DETECTOR considers at least the following (counts with respect to a time interval):

- Number of flows that have the destination located in an unusual country.

- Number of flows that have the destination included in a blacklist.

- Number of flows that use an unusual protocol.

- Number of suspicious TCP flows (SYN flows only, SYN + ACK only flows, FIN only flows, FIN + ACK only flows).

- Number of flows of more than 10 KB in length.

- Number of flows of more than 50 KB in length.

- Number of very small flows (less than half of MTU).

- Number of flows not generated by the modeled entity.

The majority of the above features must have values other than zero, but very close to zero, in the normal case. This is not relevant since a user can access one or two legitimate servers located in an unusual country, for example; The important thing is to find abnormal values.

Unusual countries and unusual protocols are defined by performing a prior statistic with respect to countries that are most accessed and most used protocols during a certain time interval.

The flowchart for the suspect traffic pool DETECTOR is the same as in the DNS-based domain flow DETECTOR, but the evaluation function is a different one. In this case, the evaluation function verifies whether the accumulated vector correlates with any grouping within the model (it is normal behavior) or not (it is an anomaly). In this sense a fast algorithm of belonging to the grouping is proposed.

Group membership functions can be complex, especially when large feature vectors are used. In this case, a rapid evaluation function is presented, based on a simplification of the groupings obtained: for each grouping and for each dimension or characteristic its limits are calculated, that is, the minimum and maximum values that the characteristic can adopt with respect to current grouping. Then, the function of belonging to the grouping is as simple as verifying if each characteristic within the evaluated vector is within the limits of the same characteristic of the grouping. The idea can be easily observed in a 2-character space, as shown in Figure 9.

■ Advantages of the invention

1. Advantages of efficiency

5

10

fifteen

twenty

25

30

35

40

Four. Five

fifty

55

60

- Modeling behavior based on entity and based on algorithm

The proposed invention allows modeling behavior based on entity and based on algorithm, statistically addressing few sophisticated attacks that are not detected based on network. However, this ADS with multiple algorithms also allows the definition of global behavior models.

- Advanced detection algorithms

The algorithms detailed above are based on complex artificial intelligence and machine learning algorithms that provide flexible, self-adaptation, self-learning and precision mechanisms to detect both general anomalies and specific traffic anomaly behaviors, such as domain flow.

2. Advantages of efficiency

- A common framework for the development of monitoring applications

The architecture shown in this document aims to be a reference model when implementing collaborative detection systems, specifically collaborative anomaly detection applications. As can be seen in the state of the art section, many attempts to define such applications have led to a confusion of concepts that does not help to know if an architecture has emerged from an algorithm or vice versa, and worse, it does Difficult to integrate both new algorithms and new architecture requirements. The proposed system gives developers a common framework to design new anomaly detection applications, but also add those new applications to existing ones in an aseptic way. This is clearly an advantage for network administrators (mainly telecommunications operators) who wish to include in their portfolio of detection systems a new detection strategy or algorithm without interfering with existing ones.

- Processing distribution

All components of this architecture can be easily distributed among several physical devices; The only requirement is that the NADS developer implements the interfaces between the modules using one of the many communications technologies such as sockets, web services, RPC, RMI, etc. Figure 10 shows an example of such an implementation distributed in four different servers.

The distribution of processing also helps in the scalability of the developed system, since the inclusion of an intense processing algorithm can be resolved, for example, by including a specialized server for that detection algorithm and implementing a TCP / IP based interface with the MONITORIZER .

- A single monitoring point for a wide variety of applications

The reader should keep in mind that monitoring points are limited due to technical constraints (fiber shunts make the optical signal weaker when splitting; duplication of ports on routers or switches consumes enough resources that can be subtracted from the processing of traffic; or directly the business cannot be stopped by cutting the line, after all, a physical derivation or a logical duplication must be made). As can be seen, the ADS with multiple algorithms only needs a single monitoring point common to all the applications implemented.

- Integration of inherited probes

The PROBE component does not need to be developed from scratch. Many other widely known flow generation systems such as Netflow [8] can be used. The only requirement to perform such integration is to include a standardization stage.

A person skilled in the art can make changes and modifications to the described embodiments without departing from the scope of the invention as defined in the appended claims.

Acronym

ADS Anomaly Detection System; anomaly detection system

AI Artificial Intelligence; artificial intelligence

CPU Central Processing Unit; central processing unit

C&C Command and Control; command and control

5

10

fifteen

twenty

25

30

35

40

Four. Five

fifty

DNS Domain Name System; domain name system

DoS Denial of Service; denial of service

EM Expectation-Maximization; maximization of hope

FQDN Fully Qualified Domain Name; fully qualified domain name

HTTP Hyper-Text Transfer Protocol; hypertext transfer protocol

IP Internet Protocol; internet protocol

IDS Intrusion Detection System; intrusion detection system

ISP Internet Service Provider; internet service provider

KB Kilo Bytes; kilobytes

MTU Maximum Transfer Unit; maximum transfer unit

NADS Network Anomaly Detection System; network anomaly detection system

NIDS Network Intrusion Detection System; network intrusion detection system

RAM Random Access Memory; random access memory

SIEM Security Information and Events Management; event management and security information

SMTP Simple Mail Transfer Protocol; simple mail transfer protocol

SOA Service Oriented Architecture; service oriented architecture

TCP Transport Control Protocol; transport control protocol

TLD Top Layer Domain; top layer domain

UDP User Datagram Protocol; user datagram protocol

Bibliography

[1] Snort IDS.
http://www.snort.org/

[2] Bro IDS.
http://bro-ids.org/

[3] "Spambot" on Wikipedia.
http://en.wikipedia.org/wiki/Spambot

[4] "Denial-of-service attack" on Wikipedia.
http://en.wikipedia.org/wiki/Denial-of-service_attack

[5] "Port scanner" on Wikipedia.
http://en.wikipedia.org/wiki/Port_scanner

[6] Proventia ADS by IBM.

http://www-935.ibm.com/services/uk/index.wss/offering/iss/y1026942

[7] "Botnets" on Wikipedia.
http://en.wikipedia.org/wiki/Botnets

[8] Cisco IOS NetFlow.
http://www.cisco.com/en/US/products/ps6601/products_ios_protocol_group_home.html

[9] "SEM" on Wikipedia.
http://en.wikipedia.org/wiki/Security_event_manager

[10] "Supervised learning" on Wikipedia.
http://en.wikipedia.org/wiki/Supervised_learning

[11] "Unsupervised learning" on Wikipedia.
http://en.wikipedia.org/wiki/Unsupervised_learning

[12] "Fast-flux" on Wikipedia.
http://en.wikipedia.org/wiki/Fast_flux

[13] “Expectation-maximization algorithm” on Wikipedia
http://en.wikipedia.org/wiki/Expectation- maximization_algorithm

[14] "Anomaly detection" on Wikipedia
http://en.wikipedia.org/wiki/Anomaly_detection

[15] "Intrusion detection system" on Wikipedia
http://en.wikipedia.org/wiki/Intrusion-detection_system

Claims (21)

5 10 fifteen twenty 25 30 35 40 Four. Five fifty 55 60 1. Method to detect malicious software, said detection being carried out in an anomalous detection system, or ADS, analyzing the behavior of a network and looking for deviations from a normality, indicating said normality the common behavior of users of said network and defining before said detection, said method comprising: - construct a plurality of detection models for each of a plurality of different entities of said network, each of said plurality of detection models adapted to said different entities of said network and to different algorithms, implementing said different algorithms different detection strategies and representing said plurality of detection models said normality, and - representing said plurality of detection models in a two-dimensional matrix, one dimension of said matrix corresponding to a number of said different entities of said network and the other dimension of said matrix corresponding to a number of said different algorithms employed. 2. Method according to claim 1, which comprises monitoring the traffic of said network, said traffic comprising packages that cross said network, and preparing said packages for said different algorithms, or for said different detection strategies, adding said packages in flows. 3. Method according to claim 2, which comprises storing said traffic in order to construct said plurality of detection models when said ADS is operating in storage mode. 4. Method according to claim 2 or 3, comprising: - processing said flows according to at least part of said different algorithms when said ADS is operating in detection mode, said different algorithms being defined in a detector library, allowing said detector library to at least add, delete or modify algorithms; Y - comparing said processed flows with at least part of said plurality of detection models. 5. Method according to claim 4 when it depends on claim 3, which comprises constructing said plurality of detection models according to a compiler library containing a model generator by algorithm, each model generator defining a compiler to be used with said traffic stored when operating in such storage mode. 6. Method according to claim 5, which comprises having a default model generator contained in said compiler library to construct at least part of said plurality of detection models. 7. Method according to claim 4, 5 or 6, comprising recording alarms generated when said malicious software is detected according to said comparison between said flows processed with said at least part of said different algorithms and said at least part of said plurality of models of detection. 8. Method according to claim 7, wherein said alarms contain at least part of the information in the following unclosed list: attacker identifier, host identifier attacked, protocol involved, date and time stamp for the alert, type of attack or type of anomaly, algorithm identifier and feedback information. 9. Method according to claim 8, wherein said different algorithms are defined according to neural networks, supervised machine learning algorithms, grouping algorithms and / or simple algorithms of the following unclosed list: volumetric traffic monitoring, absolute counts of packets or flows between nodes and periodic detection of flows. 10. Method according to claim 9, comprising implementing a given algorithm according to a domain flow detector based on the domain name system, or DNS, detecting said algorithm given domain flow activities in a monitored DNS traffic analyzing the DNS responses, said DNS response analysis comprising: - extract a plurality of features from said DNS responses; - construct a feature vector with said plurality of features; - evaluating said feature vector using a neural network, previously training said neural network with vector examples; Y - provide, by means of said neural network, a label indicating the convenience of establishing an alarm according to said evaluation. 11. Method according to claim 10, wherein at least part of said plurality of features are 5 10 fifteen twenty 25 30 35 40 Four. Five fifty 55 60 contained in the following unclosed list: number of NX_DOMAIN responses, number of FORMAT_ERROR responses, number of REFUSED responses, number of SERVER_FAILUrEs responses, number of different FQDNs consulted, number of FQDN of one layer, number of FQDN of two layers, number of FQDN of three layers or more, number of suspicious top-level domains, number of layer two domains with length less than 6, number of layer two domains with length greater than 5 and less than 21, number of domains of layer two with length greater than 20, number of domains of layer two with number of vowels less than 0.3% and number of domains of layer two with number of digits greater than 0.5%. 12. Method according to claim 8, which comprises implementing a specific algorithm according to a clustering algorithm of maximization of hope, wherein said specific algorithm identifies the traffic classes, groups said traffic classes into clusters and detects an anomaly if a traffic pattern that does not belong to one of said groupings. 13. Method according to claim 12, which comprises feedback said concrete algorithm with a vector of features of said flows, said features being stable over time and being contained in the following unclosed list: number of flows destined for an unusual country, number of flows that are blacklisted, number of flows using an unusual protocol, number of suspicious TCP flows, number of flows more than 10 KB in length, number of flows more than 50 KB in length, number of flows less than half of the maximum transmission unit and number of flows not generated by a modeled entity, in which said suspicious TCP flows comprise SYN only flows, SYN + ACK only flows, END only flows and flows only FIN + ACK. 14. Method according to claim 13, which comprises calculating the limits of each of said groupings or of each characteristic of said feature vector, and detecting an anomaly if a value of a characteristic of each characteristic vector is outside the limits of the corresponding grouping or corresponding feature of said feature vector. 15. System to detect malicious software, said detection being performed in an anomaly detection system, or ADS, analyzing the behavior of a network and looking for deviations from a normality, indicating said normality the common behavior of the users of said network and defined before said detection, said system comprising a probe module for monitoring traffic of said network connected to a controller module responsible for performing said detection, in which said controller module is provided with a plurality of detection models constructed by means of of a compiler module for each of a plurality of different entities of said network, each of said plurality of detection models adapted to said different entities of said network and to different algorithms, implementing said different algorithms different detection strategies and representing said plurality of detection models said normal d. 16. System according to claim 15, wherein a first interface between said probe module and said controller module allows traffic traces to be sent in the form of flows from said probe monitor to said controller module each time said flows are ready to be shared or when said controller module requests said flows. 17. System according to claim 16, wherein said probe module adapts said traffic trails to the flows and allows the availability in said flows of at least part of the following fields: layer 4 protocol, date and time stamps of the first and last packet, IP addresses of origin and destination in the flow, number of packets sent and received, number of bytes sent and received, TCP flags involved in the flow and application header data. 18. System according to claim 16 or 17 wherein said controller module is provided with a database of flows in which at least said flows are stored and said compiler module is provided with a database of models in which they store at least said plurality of detection models. 19. System according to claim 18, wherein a second interface between said controller module and said compiler module is implemented in order to allow communication between said controller module and said flow database, between said flow database and said compiler module and between said compiler module and said model database. 20. System according to any of the preceding claims 15 to 19, wherein a register module connected to said controller module is provided to record alarms generated by said controller module when said malicious software is detected, said controller module and said register module communicated by Middle of a third interface. 21. System according to claim 20, wherein said probe module, said controller module, said module compiler and / or said registrar module are distributed in different physical servers or devices, using said first interface, said second interface and / or said third interface at least one of the communication technologies of the following unclosed list: sockets, web services, RPC commands and remote method invocation. 5