IT201600091521A1 - METHOD FOR THE EXPLORATION OF PASSIVE TRAFFIC TRACKS AND GROUPING OF SIMILAR URLS. - Google Patents

Description

METHOD FOR EXPLORING PASSIVE TRAFFIC TRACKS AND GROUPING OF SIMILAR URLs.

DESCRIPTION

The present invention refers to a computer security method for analyzing traces of HTTP traffic on the Internet (HyperText Transfer Protocol - standard application protocol used as the main system for transmitting information on the Web), aimed at extracting and grouping of similar web transactions generated automatically by malware, malicious services, unwanted advertising or other. Web transactions are understood as HTTP and HTTPS requests and responses containing URLs (Uniform Resource Locator - unique address of a resource on the Internet, through which transactions are identified).

In the current state of the art there are some prior documents, US7680858, US7962487, US7376752, EP2291812, WO2013009713, but none of these documents uses the innovative features of the present invention described below, which allow for better performance and greater advantages.

Specifically, US7680858: performs a normalization of the URLs (unique address of a resource on the Internet) dividing them into "levels" of information; the measure of the variation between two URLs is calculated on the basis of the "differences" of keywords (search keywords); it also uses information about the "content" of the page.

US7962487: is oriented only to the improvement of search engines; is based on the clustering of tokens (categorized text blocks) associated with search queries (queries).

US7376752: divides the URL into two parts; the distance between URLs is calibrated to recognize typing errors.

EP2291812: is based on the "content" of the page; creates a set of features from each page, on which it calculates the "distance" between URLs.

WO2013009713: aims at recognizing phishing pages; searches for “relationships” between phishing page files to determine their similarity.

In scientific literature there are therefore two types of works relating to the subject matter of the present invention, the first of which includes all those works that aim to classify a page by processing only the "content" present in it or the web address of a page (URL). In this case, therefore, only "text recognition" algorithms are used, which represent only a part of the present invention. The methodologies present in this type of work, however, require a high computational cost to process the text of billions of Web pages and also aim to recognize the "theme" of each page, therefore they have totally different objectives from the present invention.

The second type, on the other hand, includes all those works that apply data-mining techniques (data extraction and processing) to URLs to detect only "certain types" of cyber attacks, such as phishing or spam.

The present invention is therefore much more complete and universal than the current state of the art. In fact, using different algorithms, suitably adapted / modified, of "text recognition" and "clustering" algorithms (unsupervised techniques developed in the field of data-mining to extract knowledge from large amounts of data), it will be possible to detect a decidedly higher than "artificial" and / or "malicious" traffic.

The present invention was therefore created to help network administrators and / or computer security analysts to extract information from Web traffic generated by networks with thousands of computers. Without tools that can help analysts, it becomes very difficult to detect problems or anomalies by looking at blocks of data made up of billions of web transactions.

This method inspects traces of web traffic generated by real users or automated bots. For each pair of network transactions present in a trace, the "degree of lexical similarity" is then calculated and the "similar" transactions are then "grouped" together to form homogeneous groups that are presented to the network analyst or expert in safety, sorted by "importance".

This method, in particular, makes it possible to automatically detect and make easily visible all that traffic that is not generated by human users but by "automatic systems", also known in jargon as bots (robots). This type of traffic, in fact, is often generated by malware or other malicious services, so such a methodology can be essential to reduce the time that passes between a cyber attack and its discovery (on average about 150-180 days) or to recognize anomalies that cause malfunctions in networks.

The present invention differs from the prior art for the following reasons:

- it is based solely on the analysis of URLs and their syntax (address of an Internet resource), ignoring the "content" of the page or other information;

- does not analyze and does not use particular structural characteristics of URLs, but maintains a neutral point of observation, verifying only the "similarity" between pairs of URLs;

- uses techniques based on "unsupervised algorithms" and therefore does not need to use, a priori, any type of knowledge or information;

- is based exclusively on the calculation of the "syntactic similarity" between the different URLs, avoiding having to have a set of pre-labeled elements, and thus also preventing problems of excessive adaptation of the algorithm used.

Inspired by text-mining algorithms (text extraction and processing), the concept of "distance" between URLs is introduced, used to compose "groups" of URLs using the well-known clustering algorithm DBSCAN (Density-Based Spatial Clustering of Applications with Noise) based on “density” because it connects regions of points with sufficiently high density.

To better illustrate how “density” -based clustering algorithms work, consider a set of points in a sample space to be clustered. Let D (x1, x2) be the distance between two points x1 and x2. Let us now consider the sphere of radius E centered at x1. If at least a minimum number of points (minPoints) are within the distance E from x1, the point x1 is classified as a "central point". Formally, a given point x1 is a "central point" if at least a minimum number of points (minPoints) are within the distance E from it. These points are defined as "directly reachable" by x1. A generic point xk is "reachable" from x1 if there is a path x1, x2, ..., xk so that xi + 1 is directly reachable from xi. The points reachable from x1 form a “cluster”, ie a “dense” region. Points that are not reachable from x1 are called "outliers" and can either form a separate cluster, if they belong to another dense region, or fall into the c.d. region of "noise". The minPoints and E parameters are adjustable and can be set by a domain expert. The minPoints parameter defines the minimum size of a cluster and has little impact on the final results. The parameter E, on the other hand, is a fundamental parameter. If it is set too small, it leads to a large number of small groups and to many points that are not clusterable / groupable. If, on the other hand, it is set too large, it leads us to a few groups with a multitude of heterogeneous points. A sensitivity analysis is therefore essential to correctly choose the value of the radius E.

The groupings thus generated are subsequently sorted to help the network administrator or security expert view them. The ordering is carried out considering the degree of cohesion of the elements within each grouping.

The present invention therefore solves the problem of processing the incoming data, aggregating them syntactically and semantically, and showing them to the analyst in a cohesive and coherent way and ordered by importance.

The method in question of the present invention is also able to offer a tool for the aggregate analysis of Web traffic, allowing to identify in a simple and direct way Web transactions related to malicious services, or provided by automatic systems such as those for generations of advertisements, systems tracking, or, in general, of interest to the network administrator or security expert.

The aforementioned and other purposes and advantages of the invention, which will emerge from the following description, are achieved with the method described in claim 1.

Preferred embodiments and non-trivial variants of the present invention form the subject of the dependent claims.

It is understood that all the attached claims form an integral part of the present description.

It will be immediately obvious that innumerable variations and modifications can be made to what has been described without departing from the scope of the invention, as appears from the attached claims.

The invention relates to a computer security method for analyzing traces of HTTP and HTTPS traffic on the Internet, aimed at extracting and grouping "similar" Web transactions generated in an "automatic" manner by malware, malicious services, unwanted advertising or otherwise.

The main objectives of this method are essentially:

- reduce the number of elements that the analyst has to view and process, from hundreds of millions of single transactions to a few hundred clusters (groups with similar / homogeneous elements within them);

- identify transactions generated "automatically", for example transactions generated by advertising platforms, polymorphic malware and / or wiki-like systems.

Specifically, the method in question includes at least the following processing and control phases:

a) extraction of transactions from an operating network, by exploration of HTTP and HTTPS traffic data and subsequent collection in batches (groups of elements) of the extracted transactions;

b) identification of "similar" transactions, by means of calculation metrics based on the "similarity" between pairs of transactions, or based on a measure of the degree of "diversity" between the pairs of character strings of which the URLs are composed;

c) activation of one or more "clustering" algorithms, used to group transactions based on similarity metrics, thus obtaining, within each group of transactions, elements with similar / homogeneous characteristics, which can therefore be analyzed as a "single" entity, significantly reducing the number of elements to be analyzed, facilitating and speeding up the work of analyzing and searching for malicious and / or unwanted internet traffic generated artificially / automatically;

d) ordering of the transaction groups based on their importance, or the degree of cohesion of the transactions contained in the groupings.

The extraction of transactions takes place via a passive traffic extraction and filtering network / probe, located in a specific link, which processes the data packets in real time, extracts the transactions and then groups them into specific batches for subsequent processing.

Once a batch of transactions has been formed, the "distance" between all pairs of transactions is then calculated, i.e. the level of similarity / similarity, and this distance is calculated considering the entire URL as a single character string, consisting of both " hostname ”(identification name of a device within a computer network), and from“ path ”.

To detect URLs similar to each other, a distance between pairs of strings is used, belonging to the "edit-distance" class, suitable for calculating the dissimilarity of pairs of character strings making up the URLs, considering the "distance" between pairs of strings of characters as the minimum number of steps required to convert one of the two strings into the other.

In the state of the art, the most popular technique is the so-called Levenshtein distance which assigns a unitary value to all editing operations, i.e. insertion, deletion and replacement of a character. It calculates an absolute distance between pairs of strings which at most is equal to the length of the longest string. This, however, makes the Levenshtein distance technique inconvenient when comparing a short and a long URL (in this case the length of the URL can extend from a few to hundreds of characters). Unlike the various known techniques, in this method, to calculate the "distance" between character strings making up the URLs, the following conditions apply:

- “insertion” of a character has a value equal to 1;

- “cancellation” of a character has a value equal to 1;

- "substitution" of a character has a value equal to 2, equivalent, the substitution, to a cancellation plus an insertion;

- the value obtained is normalized in the interval between 0 and 1 by adding all the previous operations necessary to make the two strings coincide (i.e. insertions, deletions and / or replacements) and dividing this value by the sum of the lengths of the two strings.

- the similarity between two strings of URL characters therefore varies in a normalized range of values between 0 and 1, thus obtaining that a pair of identical strings has a distance equal to 0 and a pair of completely different strings has a distance equal to 1.

A pair of similar URLs has a small distance, while a pair of different URLs has a large distance.

Said one or more "clustering" algorithms, used to group URLs based on similarity metrics, they group URLs into the same set when they have a high similarity value (ie low distance).

For the purposes of the present invention, the known clustering algorithm called DBSCAN is preferably used, based on the calculation of the "density" of elements present within a certain area.

Subsequently, the network administrator or security expert is provided with a view of these groupings of transactions, sorted according to the degree of cohesion, starting with the most cohesive grouping.

In detail, an analysis tool called the “silhouette coefficient” was used for this task. This coefficient, which is based on the concepts of cohesion and separation, provides that a cluster is identified as cohesive if the elements within it are very close to each other. Furthermore, a cluster is well separated if its points are distant from those of other clusters. The silhouette coefficient then evaluates how well each point is included in a cluster.

Given a point i, let a (i) be the average distance between that point and all the other points of the cluster to which it belongs. In this way it is calculated how well the point i is included in its grouping. With b (i) instead we define the average of the lowest distances between i and all the other points of the remaining clusters. The silhouette is therefore defined as the ratio between the difference between b (i) and a (i) and the maximum value between a (i) and b (i), thus obtaining values between 0 and 1. The more s ( i) is high, the more similar i is to its cluster. In particular, if the silhouette value is> 0, it means that the average distance between i and the other objects in its grouping is lower than the minimum average distance compared to the elements of all the other clusters. For s (i) <0 the opposite of the above is true.

The method relating to the present invention is therefore based solely and advantageously on the "syntax" of the URLs, ignoring the "content" of the pages or other information.

Claims (12)

CLAIMS 1) IT security method for the analysis of passive traces of HTTP and HTTPS traffic on the Internet, with extraction and grouping of similar web transactions generated automatically by malware, malicious services, unwanted advertising or other, characterized by the fact of include at least the following processing and control phases: a) extraction of URLs from an operational network, by passive exploration of traffic data and subsequent collection in batches of the extracted URLs; b) identification of similar URLs, by means of calculation metrics based on the similarity between URLs, or based on a measure of the degree of diversity between pairs of character strings of which the URLs are composed; c) activation of one or more clustering algorithms used to group URLs based on similarity metrics, and to obtain, within each group of URLs, elements with similar / homogeneous characteristics capable of being analyzed as a single entity; d) ordering of said groups of URLs based on their importance, or the degree of cohesion of the URLs contained in said groupings. 2) Method according to claim 1, characterized by the fact that said URL extraction takes place via a passive exploration and filtering network / probe, located in a specific link, suitable for processing data packets in real time, extracting the URLs and download them in specific batches for subsequent processing. 3) Method according to claim 2, characterized by the fact that when an HTTP and HTTPS transaction is found, the URL contained is recorded in a specific file. 4) Method according to claims 2 and 3, characterized by the fact that, once a batch of URLs has been formed, the distance between all the pairs of the various URLs is then calculated, i.e. the level of similarity / similarity, and said distance is calculated considering the 'integer URL as a single character string, consisting of both hostname and path. 5) Method according to one or more of the preceding claims from 1 to 4, characterized in that to detect URLs similar to each other, a similarity metric between pairs of strings is used, suitable for calculating the dissimilarity of pairs of character strings making up the URLs, considering the distance between pairs of character strings as the minimum number of steps necessary to convert one of the two strings into the other. 6) Method according to one of the preceding claims 1 to 5, characterized in that, for the calculation of the distance between pairs of character strings making up the URLs, the following conditions apply: - insertion of a character has a value equal to 1; - deletion of a character has a value equal to 1; - substitution of a character has a value equal to 2, equivalent, the substitution, to a cancellation plus an insertion; - normalization between 0 and 1 of the previous value obtained of the sum of the operations to make the two strings coincide divided by the sum of the lengths of the two strings; - the similarity of a pair of URL character strings varying in a normalized range of values between 0 and 1, thus obtaining that a pair of identical strings has a distance equal to 0 and a completely different pair of strings has a distance equal to 1. 7) Method according to one of the preceding claims 1 to 6, characterized in that a pair of similar URLs has a small distance, while a pair of different URLs has a large distance. 8) Method according to claim 1, characterized in that said one or more clustering algorithms are adapted to be used to group the URLs on the basis of similarity metrics. 9) Method according to claim 8, characterized in that a DBSCAN clustering algorithm is preferably used, based on the calculation of the density of elements present within a certain area. 10) Method according to claim 9, characterized by the fact that said groupings generated using the DBSCAN clustering algorithm are ordered according to the degree of cohesion between the URLs contained therein. 11) Method according to claim 10, characterized in that the silhouette coefficient is preferably used, based on the calculation of the cohesion and the degree of separation for all the elements of each grouping. 12) Method according to one or more of the preceding claims from 1 to 11, characterized in that it is based solely on the syntax of the URLs.