RU105758U1 - ANALYSIS AND FILTRATION SYSTEM FOR INTERNET TRAFFIC BASED ON THE CLASSIFICATION METHODS OF MULTI-DIMENSIONAL DOCUMENTS - Google Patents

Abstract

 A system for analyzing and filtering Internet traffic based on training methods for classifying the contents of hypertext documents, including a parsing and classification module that converts into internal representation and classification of hypertext documents based on machine learning methods with the possibility of further training to determine the subjects of documents and take into account the reference structure of documents by including topics of documents from the hypertext environment in the presentation of the document; a decision-making module that allows or blocks access to resources based on user and group filtering policies, as well as identified topics of requested hypertext documents; the core of the system, coordinating all operations in the system; a cache proxy server that intercepts requests from the local network and redirects them to the core of the system, and a robot that downloads the contents of resources from the Internet to create sets of documents for training and further training the system.

Description

The utility model relates to the field of computing, in particular, to a system for analyzing and filtering Internet traffic with the possibility of adaptive filtering by content and hypertext environment of documents based on trained classification methods.

The system is intended for analysis and filtering of Internet graphics based on the use of machine learning methods with the possibility of further training for setting and recognizing the topics of hypertext web pages, taking into account the reference information contained in them. The use of such methods allows the system to possess the following properties:

- filtering based on the content and hypertext environment of Internet resources, which can dynamically change over time;

- analysis of incoming and outgoing traffic in real time;

- the ability to dynamically change the list of topics for filtering Internet traffic (adding new ones and deleting existing ones);

- autonomy - independence from external knowledge bases and experts.

Known systems that can be used to analyze and filter Internet traffic [1, 2, 3, 4].

The first of the systems [1] is mainly based on static rules for filtering resources. The system supports a user profile. The solution is to store an approved approved list, which is not necessary to filter. All resources that do not fall into the specified list are considered prohibited. The list does not have to be local and can be stored on some remote server and used by several users. The proposed utility model uses adaptive methods of content analysis based on machine learning along with static methods of traffic analysis.

The filtering system [2] is based on classification, but offers classification not by software, but by the online content providers themselves, and then to be certified by the appropriate certification bodies, which is fundamentally different from the proposed utility model due to the lack of adaptability and autonomy.

Although the system [3] uses clear concepts of a categorized site and categorized content, it does not describe any methods for automatically classifying documents, it also offers algorithms for comparing new resources with those available in the database (local and remote), which also does not imply the autonomy of the system and does not provide high accuracy of categorization.

System [4] describes a traffic filtering system based on resource categorization. The patent proposes to use existing databases of classified resources, and the main attention is paid to the coverage of the problems of bandwidth load control by the traffic filtering system and user identification issues.

As a result of a search in the databases of the Federal Service for Intellectual Property, Patents and Trademarks (Rospatent), Russian patents on a given topic were not found.

The purpose of the invention is to solve the problem of analyzing and filtering Internet traffic with the possibility of adaptive filtering by content and hypertext environment of documents based on trained classification methods.

This goal is achieved by creating a system for analyzing and filtering Internet traffic (figure 1), containing the following modules:

- Cache proxy server 1 - a module that intercepts requests from the local network and redirects them to the core of the traffic filtering system.

- Kernel 2 is the central module of the traffic filtering system through which all operations within the system are performed.

- Decision making module 3 - a module that makes decisions about allowing or blocking access to resources.

- Parsing and classification module 4 - a module that performs lexical analysis of the contents of a resource, converts it into an internal representation and classification.

- Robot 5 - a module that downloads the contents of links from the Internet to create sets of documents for training and further training the system.

Cache proxy server module 1 is designed as a computing unit that analyzes the HTTP network traffic of a local network with the goal of caching to increase the efficiency of local network users access to the Internet by reducing the average access time to Internet resources. ICAP is used for the interaction of the proxy server and the traffic filtering system because it is a logical extension of the HTTP protocol and adds a minimal amount of redundant information to the analyzed HTTP requests and responses. The main idea of interaction between the cache proxy server and the traffic filtering system using ICAP is as follows: ICAP cache proxy server contains an integrated ICAP client that redirects new HTTP requests and user responses to the ICAP server integrated into the filtering system core traffic. The cache proxy server does not distinguish between the types of HTTP traffic being analyzed and intercepts both incoming and outgoing traffic. Outgoing traffic is intercepted during the filtering phase of the user request. In this case, the system performs filtering based on the IP address or domain of the machine to which the request is addressed, or based on the contents of the request, using classification methods.

The module 2 core is the central element of the system and is made in the form of a computing unit that implements:

1. Monitoring the filtering process of incoming and outgoing traffic, namely, identification of the one who requests the information; storage of each request in the knowledge base; transfer of requests to the decision-making module, saving the classification results and the decision-making module in the knowledge base.

2. Providing an API for other modules, namely, an API for storing links received through analysis by a resource content classifier; An API for a decision module that can request additional information about resources, users, or statistics.

3. Organization of work with the knowledge base and the provision of a knowledge base interface that allows users and system administrators to view statistics and configure the system.

4. Identification of the one who requests information, namely, identification by IP address or identification using LDAP and other protocols.

5. Storage of white lists of allowed domains and IP addresses, black lists of prohibited domains and IP addresses, storage of information about users of the system and their rights for various resource categories.

Each user can belong to one or several groups. Each user or group is assigned a white and black list of allowed and forbidden domains and IP addresses, as well as a list of allowed and forbidden resource categories. To identify a resource, its URL is used. Therefore, each request is uniquely identified by the time of the request, the user who requested it, and the URL of the resource. For compatibility with other components, the XML-RPC protocol is used to write components in different languages and place them on different physical machines. An ICAP server is built into the kernel, which receives and filters requests from the cache proxy server.

Module 3 of the decision-making module is made in the form of a computing unit that analyzes the data entering the kernel and makes a decision to allow or block access to the requested Internet resource for one or another user. The decision module works in two stages:

1. Analysis and filtering of requests from users. At this stage, the kernel transmits the following information to the decision-making module: IP-address of the machine from which the resource is requested, resource URL and meta-information about the resource, i.e. All headers received from the HTTP request. Using this information, the module is trying to make a decision. A decision at this stage can be made if, for example, the domain of the requested resource is in the white or black list for the current user or if the resource categories have been previously determined.

2. If this information is not enough to make a decision, the decision module requests the content of the resource. The kernel redirects this request to the cache proxy server, which downloads the resource from the Internet and transfers the contents to the kernel. The kernel calls the decision module method, which is responsible for filtering content. Along with the content, information is transmitted about the user, the resource’s site, additional metadata, such as the type of resource’s content, the date of the last modification, and other metadata received from the HTTP response. To obtain information about the categories of the resource, the decision-making module may refer to the parsing and classification module.

Figure 1 shows the information inputs 1 and 2, which are fed the current analyzed document and a training set of documents, tuning input 3 for the administrator to set the system operation parameters, as well as information outputs 4 and 5 with the classification results and the decision to block or allow the resource.

Module 4 parsing and classification of multi-dark hypertext documents is made in the form of a computing unit and consists of three main components (figure 2):

1. lexical analysis component (parser) 6 - parses, highlights features and converts hypertext documents into an internal representation;

2. component for calculating the similarity measure 7 - determines the proximity values between documents based on the presentation issued by the parser and caches these values;

3. Classifier 8 - builds a retraining classification model and, on its basis, classifies multi-dark hypertext documents.

The parser parses hypertext documents to a stream of tokens. Tokens found in the training collection of documents are stored in the dictionary. It sets the mapping of the string representation of the attribute to the attribute number in the vector representation, as well as statistics on the frequency of occurrences of words in the training set. The lexical analysis component also filters stop words, selects hyperlinks in documents and converts them into special tokens. The parser outputs the identifiers of the tokens found in the document (the correspondence between the tokens and their identifiers is established in the dictionary), and also replaces each encountered hyperlink with a list of identifiers corresponding to the topics of the document located at this link.

The parser architecture consists of five main modules (Fig. 3), corresponding to the stages of document conversion: encoding converter, lexical analyzer, stop-word filter, feature extraction module and aggregator. The parser works on the principle of a pipeline: each document sequentially "passes" through all its modules and as a result is converted into an internal representation.

The encoding converter recognizes the encoding of the document and converts it to UTF-8 encoding. At the input, it receives the source HTML document in the form of a byte stream; at the output, it issues the document in the form of a byte stream, but already converted to the desired encoding.

The lexical analyzer parses the structure of HTML documents (highlighting tags, links, special markup) in order to obtain a representation of the document as a stream of words that are displayed in the browser. Receiving a UTF-8 encoded byte stream from the encoding converter, the lexical analyzer recognizes meeting tags and selects BASE tags (which contain the full path to the files), as well as links to other documents. These tags are removed from the stream, processed separately, and then written back to the stream. The lexical analyzer also removes other markup elements of HTML documents and punctuation marks. The result of this component is the transformation of the document as a stream of words.

The task of the stop word filter is to remove all stop words from the data stream obtained after the lexical analyzer works.

In the feature extraction module, the word stream is converted to the corresponding token stream. As a method for distinguishing features, the basic stemming based on stemming and the N-gram method are used. Words are selected in turn from the stream, converted accordingly into a set of tokens, then the correspondence of these tokens with the identifiers in the dictionary is established and the numbers of tokens (identifiers) are recorded in the output stream with preservation of the sequence. As a result, an identifier stream is obtained at the output of the feature extraction module. Each feature in the dictionary is allocated its own number, which subsequently plays the role of the index of this feature in the feature vector.

The aggregator receives converted hyperlinks highlighted by the lexical analysis module (conversion takes place outside this module). At this stage, hyperlinks are presented in the form of sets of special identifiers corresponding to the topics of this link. These identifiers are inserted into the main stream, and they are inserted into those places of the transformed text that occupied the corresponding links in the source document. The stream of attribute identifiers received at the output of the aggregator is the result of the parser, and this stream is then used to calculate the measure of similarity between documents.

The component for calculating the similarity measure (Fig. 4) determines the proximity values between documents based on the presentation issued by the parser (as a sequence of identifiers of attributes) and caches the calculated proximity values in order to increase the learning speed and classification. The algorithm for calculating the measure of similarity is based on a representation model in the form of vectors of frequency characteristics of features. The method based on the extension of the traditional vector representation model by adding frequent combinations of features is used as a method for distinguishing features of documents. The proximity between the documents is estimated based on the scalar product between the corresponding vectors of the frequency characteristics of the attributes. When calculating the frequency characteristics of signs, not only the frequency of occurrence of the characteristic in this document is taken into account, but also the frequency of occurrence of the characteristic in the entire collection of documents.

The component for calculating the similarity measure has two operating modes: a training mode and a classification mode.

In the training mode, the component receives a training set for the classification algorithm, previously transformed by the HTML document parser. During training, the construction of feature vectors for training set documents and the calculation of feature weights based on the TF-IDF method are performed. After training, all the information necessary for the subsequent classification of documents (namely, the computed vectors of training set attributes) is stored in the context of training. The similarity measure component also supports caching to improve learning speed.

When working in the classification mode, the component uses the saved training set vectors to “compare” incoming documents with them. At the same time, for each incoming document, the module calculates the proximity values only once and saves these values before submitting it to the classifier, thereby achieving a gain in the classification speed.

The work of the classifier is based on the method of classifying multi-dark documents on the basis of decomposition into a set of binary problems of the “every-against-every” type. The classifier builds a retraining classification model and, on its basis, classifies multi-dark hypertext documents. The classifier uses the results of calculating the measure of similarity. The classifier has four main external interfaces: training, further training, classification and removal of topics.

The classification model that the classifier builds consists of the following elements:

- model of multi-label decomposition (multi-label model);

- a set of models of binary classification (the number of models depends on the type of decomposition);

- threshold function model.

The classifier has a composition / decomposition module that decomposes the multi-label problems of learning into binary subproblems in accordance with the approach of pairwise comparisons for substantially overlapping classes. Information on decomposition into binary subproblems is stored in a multi-label model. For each binary subproblem, a binary classification model is trained. When constructing binary models, the classifier uses data on measures of similarity between the documents of the training set. The classifier also provides training for the threshold function and stores its coefficients in the model.

When predicting the topics of new documents, a measure of similarity of the current classified document to all documents from the training set is passed to the classifier. The classifier applies the constructed binary models to the current classified document (using the results of the similarity measure), and then composes the predicted results to assess the degrees of belonging of the document to the classes. Then, on the basis of the threshold function model, the classifier selects classes relevant to the document.

Module 5, the robot is made in the form of a computing unit that downloads the contents of hyperlinks from the Internet to form sets of documents for training and further training the system. A typical robot operation scenario is as follows. At the beginning of the work, the robot adds the URLs to the queue. The next URL is taken from the queue. The scheduler makes a decision about the time when this resource can be downloaded, and when this time arrives, it transfers the download task to the multithreaded bootloader. A multi-threaded downloader downloads content from the Internet, writes the necessary information to the knowledge base of resources, selects the necessary links and adds them to the queue, and the process repeats.

Sources of information taken into account when drawing up the description of the application:

1. US patent No. 6745367.

2. Patent WO 2002/005148.

3. Patent WO 2006/036170.

4. US patent No. 6 947 985.

Brief Description of the Drawings:

1. Figure 1. The architecture of the traffic filtering system.

2. Figure 2. Classification module architecture.

3. Figure 3. The general scheme of the parser.

4. Figure 4. The general scheme of the component calculation measure.

Claims (1)

A system for analyzing and filtering Internet traffic based on training methods for classifying the contents of hypertext documents, including a parsing and classification module that converts into internal representation and classification of hypertext documents based on machine learning methods with the possibility of further training to determine the subjects of documents and take into account the reference structure of documents by including topics of documents from the hypertext environment in the presentation of the document; decision-making module, allowing or blocking access to resources based on user and group filtering policies, as well as identified topics of requested hypertext documents; the core of the system, coordinating all operations in the system; a cache proxy server that intercepts requests from the local network and redirects them to the core of the system, and a robot that downloads the contents of resources from the Internet to create sets of documents for training and further training the system.