EA041004B1 - TRAFFIC CLASSIFICATION SYSTEM - Google Patents

Description

The present invention relates to the field of data processing, in particular to systems for syntactic processing of a text data stream used in telecommunications, and is intended for processing messages transmitted both between subscribers and between subscribers and information systems, and can be used in a2p, p2a, p2p , a2a services.

In data processing problems, the question almost always arises of consistently refining the results as new data arrives on the fly. The growth of traffic of a2p, p2a, p2p, a2a services is well known. Banks, social networks, mobile operators generate a large volume of messages to inform their customers, including automatically. SMS and USSD technologies, standardized within the SMPP protocol, are still in demand within these services. In this regard, the problem of traffic classification arises. The task of classifying traffic according to the type of protocol or application - the type of traffic SMPP for SMS and USSD, SMTP for mail, etc., is solved at the present level of technology due to the well-known network packet analysis technology. The task of recognizing the type of traffic by message semantics (advertising, transactional, service, international, fraudulent traffic, etc.) for making business decisions is a complex technical task. From the prior art, two main methods of text classification can be distinguished - semantic text analysis based on regular expressions and text classification by heuristic methods, which are based on machine learning methods. Each method has its own advantages and disadvantages. The specifics of a2p, p2p, p2a, a2a traffic in the general case is that a large array of text data is a set of short messages. In terms of machine recognition, this presents difficulties because in a small text, there are significantly fewer useful features, and the influence of word order is greater in comparison with the classification of large blocks of text. This is exacerbated by the fact that this traffic is highly subject to changes in both the syntactic and semantic parts, and for the traffic recognizer, these changes occur continuously (online).

From the prior art technologies of semantic analysis are known, mainly represented by software and hardware systems made according to the client-server architecture. IBM LanguageWare platform, which is a set of Java, C ++ libraries, provides developers with a set of tools for natural language processing (Natural Language Processing, NLP). IBM AlchemyAPI is a cloud service that provides NLP technology on a SaaS model through its APIs, there is a development kit (SDK) supported by IBM. Both of these tools provide embeddable API clients, direct word processing is performed on remote servers of the developer company. PalitrumLab EUREKA ENGINE is a domestic representative of linguistic analysis platforms, provided both via an external API and as an embedded system for business applications. These systems have their advantages and disadvantages, as LanguageWare and AlchemyAPI are more adapted to process large volumes of text and show less efficiency when processing a stream of heterogeneous messages. PalitrumLab, while doing well with the probabilistic processing of text data streams, does not support their clear classification. There are other technologies that are generally of the same type in appearance and architecture.

A significant drawback that combines the above and other well-known solutions, from the point of view of the authors, is the inability to apply custom settings on an already running system. Analysis settings are written by developers in code when integrating clients or independent systems into business applications and are applied to data after compilation. Those. configuring known systems is available only to specialists developing client software for integration into business applications. Users who maintain business applications cannot configure analysis rules. The growth in the number of heterogeneous nodes in the network is well known, the data exchange is increasing, requiring heterogeneous analysis. The prior art does not allow to formalize, i.e. put on the logic of the algorithm, all current and future applied problems of semantic analysis. At the same time, engineers and users of business applications need a universal tool that allows them to quickly change and apply the rules for processing text and other data without a laborious change in the algorithm when the schema of text data (syntax and message semantics) is significantly changed. For the same reason, the growth of heterogeneous nodes in the network is of great importance for the performance of equipment and its control algorithms.

The authors propose a universal solution for classifying traffic into categories, i.e. semantic type of traffic - advertising, transactional, service, international, fraudulent, etc. The solution combines accurate text parsing by semantic attributes and probabilistic text classification based on machine learning. The algorithm of the classification method and the system architecture that implements it provides system administrators with real-time settings for message classification categories, rules for its analysis, and correct the separability results (separability of classification results) of the neural network. Moreover, the proposed settings are provided to a wide range of users and do not require deep training in semantic analysis and/or machine learning. Additionally, the method integrates into heterogeneous hardware and software platforms (network nodes), and scales its performance in accordance with the online streaming load.

- 1 041004

Known patent No. US 8494987 dated July 23, 2013, representing a system and methods for generating hypotheses, in which a local or distributed system interacts with a database and contains a hypothesis generation module, a module for storing concepts based on which hypotheses are generated, as well as user input modules and displaying the results. And the way to generate hypotheses is to pre-calculate hypotheses based on pre-established concepts (knowledge topic) and database data, and after receiving user input, the system generates at least one hypothesis based on jointly occurring concepts (knowledge topic) related to entities extracted from user input (suggestions, statements).

The method requires a user input of a knowledge base or an indication of the database by belonging to the analyzed entities or text processing instructions, which excludes its use in the mode of constantly arriving large amounts of online data.

The disadvantage of the hardware-software architecture is the vertical, non-scalable modularity of the software architecture, in which the modules are rigidly configured to sequentially perform the functions of analysis-loading data from the database, pre-computation, receiving user data, calculating, output.

Patent No. PCT/US2007/063983 dated March 14, 2007 describes a software product, a system and a method for generating hypotheses from a database, according to which various semantic relationships between concepts and concepts and/or concepts and concepts are built and entered into structures according to certain rules of relations and based on parsing multiple documents in a pre-installed repository. Phrases are assigned by one of a plurality of concepts, or an identifier of a relationship linking one concept to another, or an assignment of one concept to a semantic category, or an arrangement of concepts in a hierarchical relationship.

The disadvantages of this solution are also its inapplicability in the processing of constantly incoming streaming data and the impossibility of scaling the load.

PCT Application No. WO 2013/135474 dated February 22, 2013 claims a method for semantic text analysis, which consists in constructing a matrix of vectors of word occurrences in a training sample of documents, for example, Wikipedia articles, training 1 neural network (a self-organizing Kohonen map) with this sample of documents and vectors , matching documents containing the target word and a set of coordinates of the map (area) of occurrences of these documents, determining the semantic context of the word and storing the match in a dictionary, template map. Training a second neural network on documents of the same language, which consists of constructing a sequence of sentences from the documents of the training sample, calculating the coefficient of document complexity (frequency of a word in a document, etc.) and sorting the words in the sequence. And training the network with sample documents with an increase in the coefficient of semantic complexity, matching the map and words of the second sample, training the second (usually hierarchical) neural network with template maps.

The solution is based only on heuristic methods of analysis, does not imply a clear text classification, is not adapted to the operational setting of analysis rules, and is not applicable to the analysis of streaming data.

Patent RU 2615632 dated March 20, 2015, offering a method for recognizing communication messages in terms of determining the sender's name and assuming the presence of special characters in the message highlighting the sender's name or a request to a server with a pre-installed database of subscriber names and determining the sender's name based on the results of comparing the contents of a given identifier and username in the database. In this case, the semantic analysis according to the invention can be implemented in hardware by modifying the mobile terminal with text processing modules and communication with the database server.

In this solution, the obvious disadvantage is the predetermination of a set of semantic features of the sender, the forced maintenance of a database with user names, the solution provides for hardware modification of the user's end device by a recognition device. The absence of fuzzy logic algorithms (machine learning), the impossibility of setting the rules is also noted as a disadvantage.

The closest in technical essence and adopted as a prototype is patent RU 2429533 dated June 29, 2009, which describes a scheme-based dynamic parsing/assembly mechanism for multi parsing, which receives a message in various formats, converts it to a common format, defines the grammatical data structure of the message format and, based on it, an indication of the appropriate handler. Moreover, the handlers are made by modules with separate compilation and can be added to the architecture as new message formats appear. By loading individual processing modules, a gain in the performance of the analysis system as a whole is achieved. An additional performance gain is achieved by indexing the internal message format. Message routing to handlers, processing rules and new specifications are loaded dynamically into the database when a new service is registered, thus achieving a universal solution.

This solution is applicable for processing data coming in an online stream, but it is obvious

- 2 041004 A major disadvantage is the process of lexical and syntactic parsing of messages only according to the rules for analyzing the description of the syntax of formal languages (parsing). It seems that this is not enough for semantic analysis, understanding natural language text messages for making business decisions. Although the American authors indicated that handlers are loaded dynamically into the database, and new handlers do not affect the operation of the system as a whole, this does not relieve the authors of the solution from the need to develop, compile and test each new handler for loading.

The authors note an interesting solution for indexing the message format arising from the architecture of such a system, however, they also note the disadvantages of the prototype:

1) The impossibility of manual hot customization of analysis rules online without the need to develop, test and connect a new semantic analysis processor module for a new grammar in advance.

2) The absence of a probabilistic assessment of the message.

3) Poor hardware scalability - the lack of a solution for automatically adjusting system performance when the load increases - a significant increase in incoming messages.

The technical task of the proposed traffic classification system according to the a2p, p2a, p2p, a2a traffic classification method according to predefined semantic categories of traffic is:

1) increasing the flexibility of message classification due to manual user definition of intuitive informal string patterns applied to text analysis and determining the probability coefficients for classifying a message to a certain category;

2) improving the reliability of message classification by combining clear logic algorithms based on the use of regular expressions, heuristic algorithms of neural networks and additional flexible customization of these algorithms in accordance with user definitions of string templates and probability coefficients;

3) improving the performance of message analysis by dynamically generating (reproducing) and distributing the configuration of analysis rules over a set of nodes that analyze the message.

The introduction of the above user settings into the process of analytical text processing brings it to a new universal level, at which the processing process is not only subject to ordinary administrators (users) of the system who are not specialists in the field of semantic analysis, languages and dialects of regular expressions, but is also unlimited by existing and future applications of analysis due to changes in the message schema. Moreover, their combined application to the analyzed messages provides a significant increase in the reliability of the analysis. The interaction of user settings on the classification result is predetermined by the sequence of execution of the method, which first involves the application of informal string templates to the message text by clear logic algorithms, and the subsequent application of probability coefficients to the results of heuristic algorithms of the neural network. In this case, the method provides for the independence of the definition of user settings.

The technical result of the traffic classification method is achieved through a traffic classification system, according to which a natural language message is received, the message scheme is determined, the message is divided into meaningful syntactic (semantic) fields, and a plurality of nodes are provided for analyzing meaningful message fields, to which text analysis rules are applied. depending on the application protocol of the message, and in order to classify the message into certain categories, an informal pattern-string of the informal language for searching and processing a substring in a string and a probability coefficient for referring the message to a certain category are determined, the informal pattern-string is converted into a regular expression of the standard language for searching and processing a substring in a string, convert a standard language regular expression into a text parsing rule configuration plug-in containing string processing data and methods, dynamically create a plurality of AN rule configuration plug-ins analysis and distribute them over a plurality of nodes to analyze the meaningful fields of the message, apply the configuration of the analysis rules to the meaningful fields of the message, feed the message to the input of the neural network, and the result of classifying the message by the neural network is corrected in accordance with a certain probability coefficient of classifying the message to a certain category. Determination of the informal template-string of the informal language for searching and processing a substring in a string and the probability coefficient of referring a message to a certain category is carried out during message processing. A plurality of parsing rule configuration software modules are generated and distributed to a plurality of nodes for parsing meaningful message fields in proportion to the network load due to the number of parsed messages. And the classification result of the neural network is corrected in accordance with a certain probability coefficient of assigning a message to a certain category, the transformation of the threshold value of the classification of the neural network or the establishment of the reliability limit of the output elements of the neural network.

The technical task of the a2p, p2a, p2p, a2a traffic classification system device for implementing the traffic classification method is:

- 3 041004

1) improvement of operational properties, due to the inclusion in the architecture of the traffic classification system of module 4 of the user interface, made with the possibility of manual user determination of the semantic categories of message classification, informal string templates applied to the analysis of the message to determine its category and the probability coefficients for classifying the message to a certain categories;

2) increasing the reliability of classification, due to the inclusion in the architecture of the system module 2 regex, the memory of which contains and the processor of which executes the algorithm of the informal language of regular expressions irregex (irregular expressions) and the algorithm of one or more formal languages of regular expressions regex (regular expressions). Moreover, the irregex algorithm makes it possible to obtain informal user string patterns and convert them into regular expressions of standard regex languages, and the neural network module 3 neural net is configured to algorithmically convert user probability coefficients into the transformation of the threshold value of the neural network and / or establish the confidence limit for the values of its output elements;

3) improvement of operational properties due to the possibility of using a variety of semantic analysis tools, which is achieved by encapsulating the software architecture, the user of the system is provided with only module 4 of the user interface, and polymorphic work (integration with a wide range of standard regex libraries and neurosimulator programs) is achieved by defining a set of software APIs interfaces for matching the irregex language with various regex libraries and finalizing the results of the work of neurosimulator programs by programmatically setting the confidence limit in accordance with a user-defined probability coefficient for classifying a message to a certain category;

4) improving classification performance by including in the architecture of the system module 7 a configuration broker configured to dynamically reproduce - generating a plurality of program modules for the configuration of analysis rules and distributing them over a plurality of nodes to apply analysis rules to the analyzed text.

The technical problem is solved by a traffic classification system for implementing a traffic classification method that includes a coordinating module, a module for determining analysis rules, a machine learning module, at least one module for applying analysis rules to meaningful fields of a message, and additionally including an administrator (user) interface of the system that is executed with the ability to determine message classification categories, informal string templates applied to message analysis, probability coefficients for classifying a message to a certain category, and additionally includes an analysis rules compiler module designed to create a program analysis rules configuration module and consisting of a regular expression language module and a machine learning module , wherein the regular expression language module contains and executes the algorithms of the informal regular expression language and at least one formal regular expression language and is executed with the possibility of obtaining an informal pattern-string and converting it into a regular expression of a formal regular expression language, and the neural network module is configured to obtain a probability coefficient for assigning a message to a certain category and adjusting the result of the neural network operation based on it, and the coordinating module is configured to receive a software configuration module parsing rules, creating a plurality of parsing rules configuration program modules, and distributing them across a plurality of modules applying parsing rules to meaningful message fields. At the same time, the machine learning module contains in memory and executes the algorithm for transforming the threshold value of the neural network and establishing the reliability boundary of the output elements of the neural network in accordance with the obtained coefficient of probability of classifying the message to a certain category.

The invention is illustrated by drawings.

In FIG. 1 shows a functional diagram of a traffic classification system;

in fig. 2 - general scheme of a2p service;

in fig. 2a shows a generalized sequence of SMPP TCP / IP calls to the traffic classification method within the a2p service;

in fig. 2b is a generalized sequence of SMPP TCP/IP calls to a method for classifying traffic within a p2a service;

in fig. 3a is a generalized algorithm for the serial-parallel operation of a traffic classification system;

in fig. 3b - generalized algorithm for parallel operation of the traffic classification system;

in fig. 4 is a generalized data conversion process by a traffic classification system.

The functional diagram of the device that implements the proposed method of network semantic analysis is shown in Fig. 1. Module 1 of the parse rule compiler includes module 2 regex and module 3 neural net. The main function of module 1 is to assemble the analysis rules into a program module of the final configuration of the analysis rules and transfer it to the module 7 of the configuration broker for further reproduction of the specified number of program modules and transfer them to the analytical nodes 8 for application to the message.

- 4 041004

The memory of module 2 regex stores data and machine instructions with instructions to the processors of the compiled informal regular expression language - irregex and dynamic and / or static translation units containing commands and data of at least one standard regular expression language known from the prior art - regex. As regex languages, memory can contain several standard languages - these can be well-known regex libraries of Perl, Java, C ++ and other known from the prior art. Data consistency between the irrigex language and regex is achieved by redefining the programming interfaces (polymorphism) of the irregex language and the programming logic for converting generated irregex structures to regex structures. The programming logic for converting irrigex structures to regex structures is included in the functionality of the informal irrigex language developed by the applicant and being his know-how. A generalized method for converting irrigex structures to regex structures is to harmonize the data interface APIs and data processing methods of the irrigex language and the regex languages available in the memory of the module while maintaining the semantics of the informal template-string of the irrigex language. The choice of a specific regex language can be done by the system administrator in module 5 of the regex configuration.

The neural network module 3 stores data of the neural network structure (neural network). There are a large number of software products (neurosimulator programs) that provide tools for designing neural network structures for text analysis - Keras platforms, SyntaxNet, Microsoft Cognitive Toolkit (CNTK) and others known from the prior art. The neural network can be designed independently in known programming languages, this option is preferred and used by the applicant, because provides the ability to programmatically transform (transform) the threshold value of the neural network in accordance with the user probability coefficient Pf obtained from module 6 of the neural net settings, because not all neurosimulator programs provide the possibility of such a transformation. In the context of the proposed classification method, the threshold value is a value that determines the division of the set of input data of the traffic classification system into subsets, each of which is a semantic category of message classification defined by the system administrator (advertising, transactional, service, international, fraudulent, etc.) category! fig. 4(1b). As is known, the division of a set into classes (subsets) must obey the rule that an input data element falls into only one subset, and the union of all subsets must coincide with the input data set. Depending on the number of predefined classification categories or the dimension of the classification space, the output elements of the neural network, in accordance with the activation function, form the threshold value of the division of this space. For example, in the case of two classification categories, an advertisement and a service message, the category space is a plane, and the threshold is a line separating this plane and forming subsets of the classification categories. The values of the output neurons determine the coordinates of the distribution of messages on the category plane (for example, advertising traffic 0.5, service traffic 0.3). If the system administrator assumes that, in accordance with the string template, the message should refer to advertising traffic, then he can set for it the prevailing probability coefficient P _f of FIG. 4 (1s). The software algorithm for transforming the threshold value performs line transformation, for example, a shift in the space of two-dimensional coordinates (classification categories) taking into account the predominant influence of the classification category for which the system administrator has set a higher probability (confidence) coefficient P _f . After the shift is performed, more messages fall into a subset of the advertising category. A larger number of classification categories implies a more complex transformation logic in the general case similar to that described above. The transformation of the threshold value is possible by known methods: transformation of the activation function Fa (X, P _f ) fig. 4 (1c, 5a) and/or changing the values of weights of bias neurons, other parameters of the neural network in accordance with P _f , involves complex logic developed by the authors, and is the know-how of the applicant. The threshold value conversion algorithm provides the system administrator with the ability to quickly refine the classification results online without the need for retraining and testing of the neural network. In another implementation, for example, when using neural networks of third-party vendors, the architecture of which cannot be influenced in terms of the transformation of the threshold value, the correction of the classification results of the neural network is implemented by software setting the allowable classification confidence limit of Fig. 4 (1c, 5b). In this case, the results exceeding the established confidence limit P _f fall under the confidence interval. The training of the neural network is carried out according to the sets of training data, which are also transmitted to the input of the neural network from module 6.

The user interface module 4 is designed to interact with the administrator (user) of the system and in various versions may have a command line interface and / or an interface executed in the form of graphic images, includes visual interactive controls for various traffic channels (SMPP, SMTP, HTTP, etc.). d.). The user interface module 4 includes a regex setting module 5 allowing the system administrator to define message classification categories of FIG. 4(1b), e.g. transactional category for detecting a login/password request SMS, advertising category for detecting promotional SMS

- 5 041004 niami (alerts), service category for SMS user interaction with the operator SPRS, international, etc. The regex setting module 5 allows the administrator to define simplified informal user template strings of FIG. 4(1a), which he prefers to apply to the message, taking into account the current syntax format and message semantics for each defined category. The regex setting module 5 allows the administrator to determine the probability factor for a message to be assigned to a particular category user factor Pf _FIG . 4 (1s). The system administrator defines the informal user template strings of FIG. 4 (1a) is outside the rules for specifying the format of regular expressions of regex languages. The probability coefficient P _f is not strictly tied to the informal string template, because in post-processing, the user template is converted to the regex language format of FIG. 4(3), a P _f is converted to the threshold value transformation of FIG. 4 (5a, 5b). The setting data is applied to the message regardless of FIG. 4 (8, 10). The indirect connection of their mutual influence on the final classification of the message is determined by the fact that the processing of the message in module 3 neural net of Fig. 4 (9) is made after applying the rules of module 2 regex of FIG. 4 (8). Those. after classifying the message by regex crisp logic algorithms according to the user pattern of FIG. 3a (6-10), fig. 3b (5-9), the subsequent neural net classification result is refined in FIG. 3a (11), fig. 3b (10) according to the _Pf confidence factor of FIG. 3a (15-18), fig. 3b (14-17) of the user to apply their template. The ability to enter an arbitrary format of string templates intuitively at the discretion of the system administrator, relieves him of the need to delve into the rules and dialects of regex languages, and also provides the ability to quickly change the classification parameter (string template) and establish trust in it P _f with a little predictable change message format (syntax, semantics, message scheme). Subsequently, after transformations in regex module 2, the modified string template is applied to the meaningful field fields (attributes) of the message of FIG. 4 (8). Such fields can be, for example, data of the sender, recipient, message text and may include rarely used phrases, expressions, symbols, etc. The neural net 6 configuration module additionally includes an interface for setting training and testing settings for the neural network, as well as a data store for storing a historical array of messages and training data. Template strings and probability coefficients are transferred by the interface module 4 respectively to modules 2, 3 of module 1 of the analysis compiler for further processing.

Module 7 - configuration broker, which performs the functions of a coordinating module, receives the analysis rules configuration software module generated by the analysis rules compiler 1, reproduces the required set of configurations and distributes them among the analytical nodes 8. The software and hardware logic of the configuration broker creates and maintains the required number of analytical nodes, depending on of the type of traffic analyzed by the system (SMS, USSD, mail, etc.) and distributes analytical nodes depending on the load in addition to the load balancer 9, distributes software modules for configuring analysis rules between analytical nodes 8, writes/rewrites them into the memory of analytical nodes. The hardware-software logic of the configuration broker allows the processing of a message within one network session (sender-receiver) by several analytical nodes and/or processing by one analytical node of messages of different established network sessions. In the general case, the logic for distributing the configuration of the analysis rules among the analytical nodes 8 is determined by the type of traffic being analyzed, the load (the number of incoming messages) for each type of traffic, and the syntactic and semantic affiliation of the generated configuration. For example, the less changeable service structure of SMS traffic (one-time password direction-service traffic category) can be analyzed by one analytical node 8 within different network sessions.

The analytical node 8 processes messages in accordance with the configuration of the analysis rules, provides interfaces for receiving a certain type of traffic in accordance with the protocol (SMPP for SMS, USSD, SMTP for mail, etc.). Collecting a message from network packets, parsing by meaningful fields, and applying the analysis rules configuration is performed using commands and data from the handler process. The analytical node contains a pool of such processes, the logic of their creation, placement in memory, and deletion. A handler process is a set of executable machine instructions that are stored in memory and are part of the set of instructions for the processing unit of the analytical node. The handler process pool additionally provides load balancing to the measures applied by the logic of the load balancer 9 and the configuration module 7. After processing the message, the analytic node classifies the message according to whether it belongs to a predefined category (advertising, transactional, service, international, fraudulent, etc.) and forwards message to the recipient.

Load balancer 9 provides network load distribution (forwarding client network packets) between analytical nodes 8. Forwarding can be implemented dynamically at the transport level by the well-known DNS balancing technology that involves assigning several IP addresses to one domain name and selecting a specific IP address depending on the rules balancing (for example, an algorithm for round robin of IP addresses.). In another implementation, load balancer 9 statically assigns client IP addresses to specific analytics IP addresses.

- 6 041004 cal nodes 8 (example - Tanenbaum E., Weatherall D. Computer networks. 5th ed. - St. Petersburg: Peter, 2012. pp. 783, 787.). In the preferred implementation, the analytical nodes 8 are spaced across different network nodes to ensure the fault tolerance of the service using the HA (High Availability) technology of high availability clusters. In this case, the load balancer 9 can be built on the basis of proxy servers (routers) that establish redundant network connections to analytical nodes 8 to ensure a uniform distribution of client requests. Additionally, the load balancer 9 implements message forwarding in the manner of the application protocol of FIG. 1 - SMTP protocol network packets are routed to analytical nodes 8 processing SMS, USSD messages, SMPP packets are addressed to analytical nodes processing e-mail messages. Client network nodes route messages to load balancer network address 9 through a generalized gateway (SMS aggregator for a2p service or PCEF operator gateway for p2a). Physically, the load balancer 9 is implemented by the network topology by including additional routers with software logic for routing network packets. For the purpose of simplifying FIG. 1 this architecture is functionally included in module 9 and in FIG. 1 is not disclosed.

Hardware modules 1, 4, 7, 8, 9 may be executed on the same network node or different, may be present in the core network of the SPRS and/or outside it. The modules themselves 1, 4, 7, 8, 9 represent a complex of software and hardware (computers) of the von Neumann architecture of a universal or specialized type and generally contain: a network card controller implemented in the form of a board containing a processor, memory and auxiliary logic, made with the ability to process incoming and outgoing network packets with the speed of their transmission. The central processing unit, which can be implemented in a single/multi-processor version, controls the operation of the node, executes instructions and processes data stored in memory. The memory of the corresponding nodes contains (compiled) blocks of source code converted into an internal machine representation (compilation units): an interface module, a module for informal and formal languages of regular expressions regex, a neural network neural network, a configuration broker, analytical nodes. In addition to this, the interface module 4 contains a video controller - a microcircuit that forms a video image on the monitor of the system administrator. The monitor is configured to display an image that includes a graphical interface and / or command line interface that provides tools for managing traffic channels (SMPP, SMTP, others known from the prior art), defining message classification categories, setting and displaying simplified custom strings for substring search patterns in a string, displaying the input by the administrator of the system of probabilistic coefficients, visual control by training and testing the neural network. Input devices (keyboard, mouse, etc.) and an input device controller responsible for processing data entered by the administrator from input devices. The data storage (hard disk) of the interface module 4 is designed to store historical data for the purpose of their post-processing by the neural net module 3, training samples and other data, and the hard disk controller is designed to receive, transmit and process data from the data warehouse. A bus is a collection of conductors connecting the CPU to memory, storage devices, and I/O.

Fig. 2 illustrates the general scheme of an a2p service as understood by one skilled in the art, without being limited to FIG. 2 the invention can be used in p2a, a2a, p2p services. In FIG. 2 SMS sent to the subscriber can be bank SMS mailings, promotional SMS offers, SMS notifications from social networks and any other known from the prior art. TSC (Traffic Classification System) - the traffic classification system monitors all SMS messages sent to the network by programs generating SMS distribution and running on network nodes - ESME1 - ESME _n _1 Servers of various business entities - banks, shops, etc. can act as network nodes ( ESMEn).

In FIG. Figure 2a shows a generalized sequence of SMPP TCP/IP calls to a2p traffic for receiving, parsing and sending SMS messages of the SMPP protocol (3GPP specification 3G TS 23.039). The example illustrates SMS messages in the direction from the external network node ESME to the MS subscriber of the mobile radio system (MRS).

The dotted line shows signaling calls, the solid line shows TCP/IP calls.

At stages 1-2, a standard signaling dialog for establishing a connection occurs between the ESME and the SMSC of the SPRS subscriber; signaling messages of the SMPP protocol, according to known rules, fit into the OSI protocol stack and are transmitted via the TCP / IP protocol to the network. After the connection is established, the external ESME node that generates the SMS message puts the deliversm, datasm SMS message into the OSI protocol stack according to the same rules, fragments it into TCP/IP packets, and sends it to the network. By TCP/IP definition, packets are directed to the IP address of the load balancer of the traffic classification system of FIG. 19). Typically, redirection of SMS messages from different ESMEs to the IP address of the load balancer can be implemented through a network intermediary that aggregates SMS messages from different ESMEs (the so-called SMS aggregators, not shown in the figure). The load balancer, in accordance with the forwarding algorithm, directs the packets to the IP address of the corresponding analytic node of FIG. 18). The process of the analytical TCS handler applies the analysis algorithm to the SMS message (SMS PDU analysis and processing) and then sends the SMS to the network to the SMSC (SMS Center) SPRS subscriber 4, and

- 7 041004 message classification mark sends to the interested DMNS node, for example, to the SCP billing node 5. Next, the standard signaling exchange of message delivery 6-7 and delivery confirmation 8 takes place. . In another implementation, TCS stores all received SMS messages in the user interface data store 4 and post-processes them with modules 2 regex and 3 neural net 3, after which it transmits the classification results to the interested operator node. This mode can be used in addition to online classification to refine the classification results obtained in real time in FIG. 2a is not shown.

In FIG. Figure 2b shows an example of SMPP TCP/IP calls p2a traffic for receiving, parsing and sending SMS messages of the SMPP protocol (3GPP 3G TS 23.039 specification).

On the example of an SMS message in the direction from the SMSC of the DSS operator to the external ESME node.

The dotted line shows signaling calls, the solid line shows TCP/IP calls.

The clients of the traffic classification system in the SMS message classification variant are mainly SPRS operators, and the network nodes are mobile stations of subscribers (MS Mobil Station), short message service nodes (SMSC Short Message Service Center) of the SPRS operator, services or applications VAS (VAS Value Added Services ) SPRS operators or other external nodes (ESME External Short Message Entities). Clients send SMS messages to the network IP address of the load balancer 9, which executes the network packet routing algorithm for load balancing (not shown in Figs. 3a, 3b). Forwarding of client packets may be implemented at the PCEF gateway node of the SPRS operator.

The SPRS subscriber sends an SMS message from his MS, a number of calls 1-2 are made using signaling protocols to deliver the outgoing SMS to the SMSC. The SMSC starts a typical SMPP request/response sequence as an SMS sender to an external ESME 3-4 node. After the connection is established 3-4, the SMSC puts the deliversm, datasm SMS message into the OSI protocol stack, fragments it into TCP / IP packets and sends it to the network through the PCEF 5 PCEF gateway. The PCEF at the transport layer replaces the recipient address of the IP TCP packet with the IP address of the load balancer 9 traffic classification systems 6. After the packets are routed by the load balancer 9, the packets arrive at the network interface of the TCS analytical node 8 where they are collected in an SMS PDU and further analyzed and processed (SMS PDU analysis and processing). After classifying the message, the analytic node forwards the message to the ESME 7 IP address, and sets the PCEF SPRS gateway address as the sender IP address. After the ESME SMS message is received, a signaling dialog confirms the receipt of SMS 8, and the traffic classification system TCS (Traffic Classification System) sends the SMS message classification mark 8 to the address of the interested node (for example, to the SCP billing node). p2a traffic can be carried out in the same way as described above for a2p traffic.

Fig. 3a illustrates a generalized algorithm for the operation of a traffic classification system using the example of a serial-parallel operation of a traffic classification system. At stage 1, the analytical node 8 assigned to process SMS messages receives an SMS message - network packets are captured, packets are aggregated into streams, packets (streams) are classified according to the application layer protocol, data is extracted and other standard DPI traffic analysis operations are performed. At stage 2, the process pool manager initiates a handler process that checks the size of the message, parses the message into meaningful fields for analysis (text, sender, recipient, service data, etc.), forms the appropriate data structure divided into meaningful fields, and places it in the memory of the analytical node 8. At stage 3, the program logic of the analytical node provides for the transfer of the message structure to the module 4 of the interface for writing to the data store 4 in order to accumulate historical data and post-processing. Step 5 prompts the administrator to select a classification algorithm, Regex crisp logic algorithms or Neural net heuristic algorithm. At step 6, the system administrator is given the option to change the message parsing rules. For example, if in messages in a random order that is impossible for formalization (expressions in the program logic of the algorithm), the syntax or semantics of the message changes, characters of a different alphabet appear and / or new words and phrases appear borrowed from other languages (for example, deadline, PR, zoom, etc.). In this case, at stage 7, the system administrator, at his own discretion, introduces simplified templates-strings that are intuitively understandable to him, which, from his point of view, are acceptable for recognizing the changed message semantics. The input of informal template strings is carried out by the technical means of module 5 of the regex settings of the user interface - the administrator defines the informal template strings of the user template. At the same time, it can determine the probability coefficients user factor P _f for assigning messages to a predefined category - the type of traffic (category1), fig. 3a, 3b. The administrator controls the correctness of the input on the monitor using visual control tools (graphical interface) rendered by the video controller of the user interface module 4. At stage 8, the data of user templates converted into a machine representation is transmitted and placed in the memory of the module 2 regex and then by a set of machine instructions of the informal regular expression language irrigex module 2 converts custom

- 8 041004 patterns into regular expressions of the already formal regex language. The transformation involves programmatically generating the irrigex struct data structure of the informal irrigex language and converting it into a regex struct structure corresponding to patterns (regular expressions) of the already formal regex regular expression language. At step 9, from the regex struct structure, the parsing rules compiler 1 generates the final configuration of the parsing rules config - a program module that is an object code file with instructions for parsing the meaningful fields of the message and passes it to the configuration broker 7 for writing into memory, which reproduces the assigned set of copies of the object file configuration code and transmits it to the assigned analytical nodes 8. At step 10, the analytical node 8 places the configuration in the memory of the corresponding handler process, which uses it to process the text of the meaningful fields of the message and then generates a sign that the message belongs to a predefined category, type of traffic - step 11. Further, at step 12, the meaningful fields of the string message can be transmitted to the neural net module 3 at the input of the neural network (steps 14-18) to refine the classification (step 11) or the message is transmitted to the network to be sent to the recipient 13. Such a loop allows you to refine the category of the message online by means neura l net after applying the regex rules. In the case of transferring significant fields of the string message to the input of the neural network, the neural net module 3 uses the machine representation of the probability coefficient Pf obtained from the neural net setup module 6, step 15. network by the coefficient P _f - stage 18. Further, according to the classification feature generated by neural net module 3, the handler process specifies the category of the message and then transfers it to the network for sending - stages 11, 12. Within one handler process, when performing stages 14-18, stages 1- 10 are applied to the next message, this achieves serial-parallel processing of messages.

Fig. 3b illustrates a block diagram of the parallel message processing. In this variant, after step 3, clear classification algorithms based on Regex rules - steps 5-9 and heuristic Neural net classification algorithms steps 13-17 are applied in parallel to the meaningful fields of the string message. At step 10, the program algorithm of the handler process waits for the completion of both branches of the algorithm, steps 13-17 involve transferring (copying) the meaningful fields of the string message to the neural net module 3 and waiting for the neural network classification algorithm to execute. After that, a sign of message classification is formed (the program logic of steps 5-9 and 13-17 are performed similarly to the variant of serial-parallel processing). Next, the message and the classification flag are sent to the recipients - steps 10-12.

Fig. 4 illustrates the data processing scheme of the traffic classification system. The SMS pdu flow message flow, after processing the logic of network load balancing by module 9 (not shown in Fig. 4), arrives at the network interface of the analytical node 8. The analytical node receives a network packet 6 via the socket program interface (address of the network node and assigned port of the handler process). performs standard operations (DPI Deep Packet Inspection) of classifying packets, aggregating them into streams and, according to the software algorithm of the handler process, generates an SMS struct 7 data structure, which is a machine representation of meaningful SMS message data - length length, sender sender, receiver receiver, message text -string string, which includes semantic fields fieldn significant for analysis and classification. The user interface module 4 contains in memory the data defined by the system administrator: message classification categories - category1 1b, informal string template - user template 1a, probability coefficient of assigning a message to a certain category - user factor P _f 1c. Module 2 regex, based on the informal pattern-string 1a, using the program algorithm of the informal regular expression language irrigex, generates the irrigex struct 2 structure, which includes data and methods for various operations such as searching for a pattern occurrence, performing a string replacement, etc. and saves it in the module memory. In accordance with the user settings for the selected library of the standard language regex, the program algorithm of module 2 converts the data of the structure of the informal language irrigex struct 2 into the data structure of the corresponding already standard regular expression language regex struct 3, which defines the rules of text analysis. The program algorithm of the analysis rules compiler 1 collects the config file 4 - the program module for configuring the analysis rules that includes data and methods for processing them methodsk (..) of the regex struct structure 3. The configuration broker 7 reproduces (copies) the number of config configuration files specified by the program algorithm and distributes them over analytical nodes 8 (not shown in Fig. 4). The handler process applies the rules for parsing the configuration file config to the meaningful fields of the message string 8 according to the program algorithm and, based on the result, generates a classification sign, i.e. refers the message to one of the preset categories category1 1b. To classify a message by a neural network, the handler process passes the meaningful fields of the string message to module 3 neural net. The meaningful fields of the message fieldn are input to the input elements of the neural network 9, after the classification logic is executed, the output elements X1 of the neural network generate the corresponding probability coefficients Fa1 (X1) for classifying the message to a certain category 1. The software algorithm for transforming the threshold value of neural net module 3 transforms the function of the output elements X1 (threshold value) taking into account the coefficient P _f 1c - Fa1 (X1, P _f ) 5a. This ensures the influence of the user

- 9 041004 Tel probability coefficient Pf of classifying a message in a certain category on the classification result. After that, a sign of classifying a message is formed, referring it to a certain category 10. In another version of work 5b, the program algorithm of module 3 neural net assigns a sign of classifying a message of that category category ₁ , the threshold value of which P1-1 of the corresponding output neuron X ₁ prevails and exceeds the confidence interval , set by the user coefficient P _f .

The presented architecture of the traffic classification system assumes the separation of modules across different network nodes. System speed is achieved by processing messages with regex rules only on the analytic node 8. Performance scaling, along with known measures taken by the load balancer, is achieved by reproducing and redistributing the configuration of the configuration broker parsing rules among the analytic nodes and maintaining their required number. The system architecture allows the administrator to perform steps 6-10, 14-18 of FIG. 3a and 5-9, 13-17 of FIG. 3b in series-parallel and parallel modes, respectively. The architecture of the system allows you to apply user settings - changing the operating mode, disabling / enabling regex and / or neural net processing, entering patterns and probability coefficients, without interrupting the analysis of the online message flow.

The main feature and advantage of the system proposed by the authors is the flexibility and versatility of customizing the classification by ordinary network node administrators through the use of the author's informal regular expression language irregex, which performs its logic in module 3 regex. The functionality of the simplified input of text recognition templates allows you to remove restrictions on the maintenance of the device by narrowly specialized highly qualified specialists and provide the opportunity for its operation to ordinary telecommunications specialists. Currently, in known text analysis solutions, when changing the message scheme, the developers of semantic analysis solutions are inevitably forced to deal with time-consuming, not always possible refinement of the architecture of the semantic analysis solution. An accidental change in the scheme and structure of a message in the initial settings of the server cannot be determined by default, and it is not possible to provide for formalizing all possible message formats for obvious reasons. The functionality of the user interface module 4, available for configuration, transfers the server proposed by the authors from the category of specialized equipment to the category of telecommunications network nodes available for configuration by ordinary specialists in telecommunications. Additionally, the inclusion of the informal irregex language in the system architecture made it possible to interact with any of the various standard regex languages, connect them for analysis in accordance with user preferences. Moreover, the inclusion of custom probability coefficients in the classification process, and the implementation of the neural net module with the ability to adjust the results of the neural network (of its own architecture or third-party tools) allows the system administrator to quickly respond to changes in the scheme and semantics of the message and quickly correct the results of neural classification. Moreover, the classification method, both in serial-parallel mode and in parallel, provides a significant increase in classification accuracy due to the synergy of the regex and neural net algorithms. Most importantly, the way and architecture of the traffic classification system provides indirect mutual influence of user settings, while the system administrator can configure them independently.

The proposed traffic classification system is demonstrated above on the example of SMPP traffic within the a2p, p2a interaction. As specialists understand, the proposed solution can be used for semantic analysis of the message flow of p2p, a2a services, mostly small amounts of text. As well as other standards and applications (including, for example, the SMPT specification for mail services), without going beyond the legal protection of the proposed method of traffic classification and device architecture for its implementation.

The proposed invention was tested in PJSC Megafon and PJSC VimpelCom, as a result of the work, the claimed technical effect was confirmed.

Claims (8)

CLAIM 1. A traffic classification method, according to which a natural language message is received, a message scheme is determined, a message is divided into meaningful syntactic and semantic fields, a plurality of nodes are provided for analyzing meaningful message fields, to which text analysis rules are applied depending on the message application protocol, characterized in that in order to classify a message according to certain categories, an informal template-string of the informal language for searching and processing a substring in a string and a probability coefficient for referring the message to a certain category are determined, the informal template-string is converted into a regular expression of the standard language for searching and processing a substring in a string, and regular expression of the standard language into the text parsing rule configuration plug-in containing string processing data and methods, dynamically create a plurality of rule configuration plug-ins - 10 041004 analysis and distribute them over a plurality of nodes to analyze the meaningful fields of the message, apply the configuration of the analysis rules to the meaningful fields of the message, feed the message to the input of the neural network, and the result of the classification of the message by the neural network is corrected in accordance with a certain probability coefficient for classifying the message to a certain category . 2. A method for classifying traffic according to claim 1, characterized in that the informal template-string of the informal language for searching and processing a substring in a string and the probability coefficient for classifying a message to a certain category are determined during message processing. 3. The traffic classification method according to claim 1, characterized in that a plurality of analysis rule configuration software modules are generated and distributed over a plurality of nodes to analyze the significant fields of the message in proportion to the network load due to the number of messages to be analyzed. 4. The traffic classification method according to claim 1, characterized in that the classification result of the neural network is corrected in accordance with a certain probability coefficient for classifying a message to a certain category, a transformation of the neural network classification threshold. 5. The method of classifying traffic according to claim 1, characterized in that the classification result of the neural network is corrected in accordance with a certain probability coefficient for classifying a message to a certain category, establishing a confidence limit for the output elements of the neural network. 6. A traffic classification system, including a coordinating module, a module for determining analysis rules, a machine learning module, at least one module for applying analysis rules to meaningful fields of a message, characterized in that it additionally includes an administrator or user interface of the system, configured to determine classification categories message, informal string templates applied to the analysis of the message, probability coefficients for classifying the message to a certain category, and additionally includes an analysis rules compiler module, configured to create a software analysis rules configuration module and consisting of a regular expression language module and a neural network module, moreover, the language module regular expression contains in memory and executes the algorithms of the informal regular expression language and at least one formal regular expression language and is configured to obtain an informal template-string and convert it o into a regular expression of a formal regular expression language, and the neural network module is configured to obtain a probabilistic coefficient for classifying a message to a certain category and adjust the result of the neural network operation based on it, and the coordinating module is configured to obtain a software module for configuring analysis rules, creating a plurality of software configuration modules parsing rules and distributing them over a plurality of modules for applying parsing rules to meaningful message fields. 7. The traffic classification system according to claim 6, characterized in that the neural network module contains in memory and performs an algorithm for transforming the threshold value of the neural network in accordance with the obtained coefficient of probability of classifying the message to a certain category. 8. The traffic classification system according to claim 6, characterized in that the neural network module contains in memory and executes an algorithm for establishing the reliability boundary of the output elements of the neural network in accordance with the received coefficient of classifying the message to a certain category.