WO2020071962A1 - Système de classification de trafic - Google Patents

Système de classification de trafic

Info

Publication number
WO2020071962A1
WO2020071962A1 PCT/RU2019/000715 RU2019000715W WO2020071962A1 WO 2020071962 A1 WO2020071962 A1 WO 2020071962A1 RU 2019000715 W RU2019000715 W RU 2019000715W WO 2020071962 A1 WO2020071962 A1 WO 2020071962A1
Authority
WO
WIPO (PCT)
Prior art keywords
message
classification
module
neural network
analysis
Prior art date
Application number
PCT/RU2019/000715
Other languages
English (en)
Russian (ru)
Inventor
Мария Давидовна ГОРЬКОВА
Original Assignee
Общество с ограниченной ответственностью "Алгоритм"
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Общество с ограниченной ответственностью "Алгоритм" filed Critical Общество с ограниченной ответственностью "Алгоритм"
Publication of WO2020071962A1 publication Critical patent/WO2020071962A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/12—Applying verification of the received information
    • H04L63/126—Applying verification of the received information the source of the received data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/22—Parsing or analysis of headers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/12—Messaging; Mailboxes; Announcements
    • H04W4/14—Short messaging services, e.g. short message services [SMS] or unstructured supplementary service data [USSD]

Definitions

  • the present invention relates to the field of data processing, in particular to systems for syntactic processing of a text data stream used in telecommunications, and is intended for processing messages transmitted both between subscribers and between subscribers and information systems and can be used in a2p, p2a, p2p, A2a services.
  • IBM LanguageWare platform which is a set of Java, C ++ libraries, provides developers with a set of tools for processing natural language (Natural Language Processing, NLP).
  • IBM Alchemy API - a cloud service that provides NLP technology for SaaS models through its APIs; there is a set of development tools (SDKs) supported by IBM. Both of these tools provide embedded API clients; direct text processing is performed on the remote servers of the development company.
  • PalitrumLab EUREKA ENGINE is a domestic representative of linguistic analysis platforms, it is provided both by an external API and as an embedded system for business applications.
  • the authors offer a universal solution for classifying traffic into categories, i.e. semantic type of traffic - advertising, transactional, service, international, fraudulent, etc.
  • the solution combines accurate parsing of text by semantic attributes and probabilistic classification of text based on machine learning.
  • the algorithm of the classification method and the architecture of the system that implements it provides the system administrators in real time with the option to configure the classification categories of the message, the rules for its analysis and adjust the separability (separability of classification results) of the neural network.
  • the proposed settings are provided to a wide range of users and do not require in-depth training in issues of semantic analysis and / or machine learning.
  • the method integrates into heterogeneous software and hardware platforms (network nodes), and scales its performance in accordance with the online streaming load.
  • Known patent N ° US8494987 from 07.23.13g. representing a system and methods for generating hypotheses, in which a local or distributed system interacts with a database and contains a module for generating hypotheses, a module for storing concepts on the basis of which hypotheses are generated, as well as modules for user input and output of results.
  • a method for generating hypotheses is to pre-calculate hypotheses based on predefined concepts (the topic of knowledge) and database data, and after receiving user input, the system generates at least one hypothesis based on jointly occurring concepts (topic of knowledge) related to entities extracted from user input (sentences, statements).
  • the method requires user input of the knowledge base or an indication of the database by belonging to the analyzed entities or instructions for processing text, which excludes its use in the mode of constantly arriving large volumes of online data.
  • the disadvantage of the software and hardware architecture is the “vertical”, non-scalable modularity of the software architecture, in which the modules are rigidly configured for sequentially performing the functions of analysis — loading data from the database, preliminary calculation, receiving user data, computing, output.
  • Patent N ° PCT / US2007 / 063983 dated March 14, 2007. describes a software product, system and method for generating hypotheses from a database, by which various semantic relationships between concepts and concepts are built and entered into structures and / or concepts and concepts according to certain rules of relations and based on the analysis of many documents in a predefined repository. Phrases are assigned according to one of the many concepts, or the identifier of the relationship connecting one concept with the second, or the assignment of one concept of a semantic category or the arrangement of concepts in a hierarchical relationship.
  • PCT Application N2WO2013 / 135474 of 02.22.2013 claims a method of semantic analysis of the text, which consists in constructing a matrix of vectors of occurrences of words in the training sample of documents, for example Wikipedia articles, learning 1 neural network (self-organizing Kohonen map) with this sample of documents and vectors, matching documents containing the target word and the set of map coordinates (region ) occurrences of these documents, the definition of the semantic context of the word and the preservation of correspondence in the dictionary, map templates Learning a second neural network using documents of the same language, consisting of constructing a sequence of sentences from documents of a training sample, calculating the document's complexity factor (word frequency in a document, etc.) and sorting words in a sequence. And training the network with sample documents with an increase in the coefficient of semantic complexity, comparing the map and the words of the second sample, training the second (usually hierarchical) neural network with template cards.
  • the solution is based only on heuristic methods of analysis, does not imply a clear classification of the text, is not adapted to the operational configuration of the analysis rules, and is not applicable to the analysis of streaming data.
  • Patent RU2615632 dated 03/20/2015 offers a method for recognizing communication messages in terms of determining the name of the sender and assuming the presence of special characters in the message highlighting the name of the sender or a request to the server with a predefined database of subscriber names and determining the name of the sender by comparing the contents of the specified identifier and user name in the database.
  • the semantic analysis according to the invention can be implemented in hardware by finalizing the mobile terminal with text processing and communication modules with the database server.
  • the technical task of the proposed traffic classification system according to the classification method of a2p, p2a, p2p, a2a traffic according to predefined semantic categories of traffic is: to increase the flexibility of message classification due to the manual user-defined intuitive informal line patterns that are used to analyze the text and determine the probabilistic attribution of messages to a certain category;
  • the introduction of the above user settings into the process of analytical text processing takes it to a new universal level, at which the processing process is not only subject to ordinary administrators (users) of the system, who are not experts in the field of semantic analysis, languages and dialects of regular expressions, but also unlimited by existing and future analysis applications due to changes in the message scheme. Moreover, their combined application to the analyzed messages provides a significant increase in the reliability of the analysis.
  • the interaction of user settings on the classification result is predetermined by the sequence of the method, which involves first applying informal line patterns to the message text with clear logic algorithms, and then applying probability coefficients to the results of heuristic algorithms of the neural network. In this case, the method provides for no dependence on determining user settings.
  • the technical result of the traffic classification method is achieved by the traffic classification system, according to which a natural language message is received, a message scheme is determined, the message is divided into meaningful syntax (semantic) fields, provide many nodes for the analysis of significant message fields, to which the text analysis rules are applied depending on the application message protocol, and for classifying messages into certain categories, an informal template-string of an informal language for searching and processing substring in a string and a probability coefficient are determined assigning a message to a certain category converts an informal string template into a regular expression of the standard language for searching and processing a substring in a string, they convert the regular expression of the standard language into a text analysis rule configuration program module containing data and string processing methods, dynamically create many analysis rule configuration software modules and distribute them across many nodes to analyze significant message fields, apply the analysis rule configuration to significant message fields, send a message to the input of the neural network, and the result of the classification of messages by the neural network is adjusted in accordance with a certain probabilistic coefficient The attribute of a message to a certain category
  • the definition of an informal template-string of an informal language for searching and processing a substring in a string and the probabilistic coefficient of assigning a message to a certain category is carried out during message processing.
  • Many software modules for the configuration of analysis rules are generated and distributed across many nodes for the analysis of significant message fields in proportion to the network load due to the number of messages analyzed.
  • the classification result of a neural network is adjusted in accordance with a certain probabilistic coefficient of classifying a message to a certain category, transforming a threshold value for classifying a neural network, or setting a confidence limit for output elements of a neural network.
  • the technical task of the device classification system a2p, p2a, p2p, a2a traffic to implement the method of traffic classification is to improve operational properties, by including in the architecture of the traffic classification system module 4 of the user interface, made with the possibility of manual user definition of semantic categories of message classification, informal string patterns applied to the analysis of a message to determine its category and probabilistic coefficients of classifying a message as a definition flax category;
  • module 2 regex in the architecture of the system, the memory of which contains and the processor whose algorithm is executed the informal regular expression language irregex (irregular expressions); and the algorithm of one or more formal regular expression languages regex (regular expressions).
  • the irregex algorithm allows you to get informal custom string patterns and convert them into regular expressions of the standard regex languages
  • the neural network module 3 neural net is configured to algorithmically convert user probability coefficients to transform the threshold value of the neural network and / or establish a confidence boundary for the values of its output elements;
  • module 7 of a configuration broker in the architecture of the system configured to dynamically reproduce - generate many software modules for configuration of analysis rules and distribute them across many nodes to apply analysis rules to the text being analyzed.
  • a traffic classification system for implementing a traffic classification method including a coordinating module, an analysis rule definition module, a machine learning module, at least one module for applying the analysis rules to significant message fields, and additionally including a system administrator (user) interface that is implemented with the ability to determine message classification categories, informal string patterns applied to message analysis, probability coefficients to classify a message into a certain category and additionally includes an analysis rule compiler module configured to create a program analysis analysis module configuration module and consisting of a regular expression language module and a machine learning module, the regular expression language module contains in memory and executes algorithms of the informal regular expression language and at least one formal language of regular expressions and is configured to obtain an informal string pattern and convert it into a regular expression of the formal regular expression language, and the neural network module is configured to obtain a probabilistic referral coefficient to a certain category and adjustments based on it the result of the operation of the neural network, and the coordinating module is configured to receive the software module configuration of the analysis of the rules, creating a plurality of software modules configuration rules of analysis and their
  • Figure 1 shows a functional diagram of a traffic classification system.
  • FIG. 2 shows the General scheme A2P service.
  • Figure 2a shows a generalized sequence of SMPP TCP / IP calls of a method for classifying traffic within an A2p service.
  • Fig.2b shows a generalized sequence of SMPP TCP / IP calls of a method for classifying traffic within a P2A service.
  • Fig. 3a shows a generalized algorithm of a series-parallel mode of operation of a traffic classification system.
  • On Fig shows a generalized algorithm for the parallel mode of operation of the traffic classification system.
  • Figure 4 shows a generalized data conversion process by the traffic classification system.
  • Analysis rule compiler module 1 includes regex module 2 and neural net module 3.
  • the main function of module 1 is the assembly of analysis rules into a software module of the final configuration of analysis rules and transferring it to module 7 of the configuration broker for further playback of a given number of software modules and transferring them to analytical nodes 8 for application to the message.
  • the memory of module 2 regex stores data and machine instructions with instructions to the processors of the compiled informal regular expression language - irregex and dynamic and / or static translation units containing the commands and data of at least one standard regular expression language known from the prior art - regex.
  • memory can contain several standard languages - these can be well-known regex libraries of Perl, Java, C ++ and other known from the prior art.
  • Data coordination between the irrigex and regex languages is achieved by redefining the irregex programming interfaces (polymorphism) and the programming logic for converting the generated irregex structures to regex structures.
  • the program logic for converting irrigex structures to regex structures is included in the functionality of the informal language irrigex developed by the applicant and his know-how.
  • a generalized method for converting irrigex structures to regex structures is to coordinate the data interfaces API and irrigex language processing methods and regex languages available in the memory while maintaining the semantics of the irrigex informal string template.
  • the choice of a specific regex language can be carried out by the system administrator in module 5 of the regex settings.
  • the memory of module 3 of the neural net contains the data of the neural network structure (neural network).
  • neural network There are a large number of software products (neuroimitator programs) that provide tools for designing neural network structures for text analysis - the Keras, SyntaxNet, Microsoft Cognitive Toolkit (CNTK) platforms and other known in the art.
  • a neural network can be designed independently in well-known programming languages, this option is preferred and is used by the applicant since provides the ability to programmatically convert (transform) a threshold value of a neural network in accordance with a user probability coefficient Pf obtained from module 6 of setting up neural net since not all neuroimitators provide the possibility of such a conversion.
  • the threshold value is a value that determines the division of the set of input data from the traffic classification system into subsets, each of which is a semantic category of message classification defined by the system administrator (advertising, transactional, service, international, fraudulent, etc.) categoryi of FIG. 4 lb.
  • the division of the set into classes (subsets) must obey the rule that the input data element falls into only one subset, and the union of all subsets must coincide with the set of input data.
  • the output elements of the neural network in accordance with the activation function, form a threshold value for the separation of this space.
  • the category space is a plane
  • the threshold value is a line separating this plane and forming a subset of the classification categories.
  • the values of the output neurons determine the coordinates of the distribution of messages on the category plane (for example, advertising traffic 0.5, service 0.3). If the system administrator assumes that in accordance with the line template the message should be related to advertising traffic, then he can set the prevailing probability coefficient P f for it . 4 1s.
  • a software threshold transformation algorithm performs line transformation - for example, a shift in the space of two-dimensional coordinates (classification categories), taking into account the prevailing influence of the classification category for which the system administrator has set a higher probability (confidence) coefficient Pf.
  • a larger number of classification categories suggests a more complex transformation logic in the general case similar to that described above.
  • the transformation of the threshold value is possible by known methods: by converting the activation function Fa (X, Pf) of Fig. 4 1c, 5a and / or by changing the weights of the bias neurons, other parameters of the neural network in accordance with Pf, includes complex logic developed by the authors, and is know-how how the applicant.
  • the threshold conversion algorithm allows the system administrator to quickly refine the classification results online without the need for retraining and testing the neural network.
  • the adjustment of the results of the classification of the neural network is implemented by programmatically setting the acceptable boundary of the classification classification reliability figure 4 lc, 5b. In this case, results that exceed the established confidence limit Pf fall under the confidence interval.
  • Neural network training is carried out using training data sets that are also transmitted to the input of the neural network from module 6.
  • the user interface module 4 is intended for interaction with the system administrator (user) and, in various versions, can have a command line interface and / or an interface executed in the form of graphic images, includes visual interactive tools for monitoring various traffic channels (SMPP, SMTP, HTTP, etc. d.).
  • the user interface module 4 includes a regex configuration module 5 that allows the system administrator to determine the message classification categories of Fig. 4 lb, for example: transaction category for identifying SMS request for login / password, advertising category for identifying SMS with advertising offers (notifications), service category for SMS interaction user with operator SPRS, international and others.
  • Module 5 settings regex provides the administrator to define simplified informal templates-strings user template figure 4 la ko orye he preferred to apply to the message with the current format syntax and semantics of the messages for each specific category.
  • the regex configuration module 5 allows the administrator to determine the probability coefficient of assigning a message to a certain category of user factor Pf of FIG. 4 1c.
  • the system administrator defines informal templates - user template strings in Fig.
  • the probability coefficient Pf is not rigidly “tied” to an informal template string, because during subsequent processing, the user template is converted to the regex language format of FIG. 4 3., and Pf is converted to a threshold value transformation of FIGS. 4 5a, 5b.
  • the setting data is applied to the message independently of FIG. 4 8.10.
  • the indirect relationship of their influence on the final classification of the message is determined by the fact that the message is processed in module 3 neural net of Fig. 4 9 after applying the rules of module 2 of regex of Fig. 4 8. That is, after the message is classified by the regex clear logic algorithms according to the user template of Figs. 6-10, 36 5-9, the subsequent neural net classification result is refined by fig.
  • the neural net 6 configuration module additionally includes an interface for setting training settings and testing the neural network, as well as a data warehouse for storing a historical array of messages and training data. Template strings and probabilistic coefficients the interface module 4 transmits, respectively, to modules 2, 3 of module 1 of the analysis compiler for further processing.
  • Module 7 - the configuration broker acting as the coordinating module receives the software module for the configuration of analysis rules generated by the analysis rules compiler 1, reproduces the required set of configurations and distributes them to analytical nodes 8.
  • the hardware and software logic of the configuration broker creates and maintains the required number of analytical nodes depending on the analyzed traffic type system (SMS, USSD, mail, etc.) and distributes analytical nodes depending on the load in addition to the balance ovschiku load 9, distributes between nodes 8 analytical program modules configuration analysis rules, carries them recording / rewriting memory analytical nodes.
  • SMS analyzed traffic type system
  • USSD USSD
  • mail etc.
  • the hardware and software logic of the configuration broker allows the processing of a message within a single network session (sender-receiver) by several analytical nodes and / or processing by a single analytical node of messages of different established network sessions.
  • the logic of the distribution of the configuration of the analysis rules by analytical nodes 8 is determined by the type of traffic being analyzed, the load (number of incoming messages) for each type of traffic, and the syntactic and semantic affiliation of the generated configuration.
  • the service structure of SMS traffic that is less susceptible to changes can be analyzed by one analytical node 8 within different network sessions.
  • Analytical node 8 processes messages in accordance with the configuration of the analysis rules, provides interfaces for receiving a certain type of traffic in accordance with the protocol (SMPP for SMS, USSD, SMTP for mail, etc.).
  • the message is collected from network packets, parsed by significant fields, and the configuration of the analysis rules is applied according to the commands and data of the processor process.
  • the analytical node contains a pool of such processes, the logic of their creation, placement in memory and deletion.
  • the processor process is a set of executed machine instructions stored in memory and which is part of a set of instructions processor unit of the analytical node.
  • the processor process pool additionally provides load balancing to the measures applied by the load balancer 9 logic and configuration module 7.
  • the analytical node classifies the message according to its predefined category (advertising, transactional, service, international, fraudulent, etc.) and directs message to the recipient.
  • Load balancer 9 provides distribution of network load (forwarding client network packets) between analytic nodes 8. Forwarding can be implemented dynamically at the transport level by the well-known DNS-balancing technology, which provides for the assignment of several IP addresses to one domain name and the choice of a specific IP address depending on the rules balancing (for example, an algorithm for cyclic bypass of IP addresses.) ". In another implementation, the load balancer 9 statically assigns client IP addresses the specific IP addresses of the analytic nodes 8 (for example, Tanenbaum E., Weatheroll D. Computer networks. 5th ed. - St. Beach: Peter, 2012. p. 783, 787.) .
  • the analytical nodes 8 are spaced across different network nodes to ensure fault tolerance of the service using HA technology (High Availability) of high availability clusters.
  • the load balancer 9 can be built on the basis of proxy servers (routers) that establish redundant network connections to analytical nodes 8 to ensure uniform distribution of client requests. Additionally, the load balancer 9 implements message forwarding according to the type of application protocol of FIG. 1 - network packets of the SMTP protocol are routed to analytical nodes 8 that process SMS, USSD messages, SMPP packets are addressed to analytical nodes that process email messages.
  • Client network nodes send messages to the network address of load balancer 9 through a generalized gateway (SMS aggregator for a2p service or a PCEF operator's gateway for P2a).
  • load balancer 9 is implemented by the network topology by including additional routers with programmatic logic for routing network packets. In order to simplify figure 1, this architecture is functionally included in module 9 and is not disclosed in figure 1.
  • Hardware modules 1, 4, 7, 8, 9 can be executed on one network node or different, can be present in the basic network of SPRS and / or outside it.
  • the modules 1,4, 7, 8, 9 themselves are a complex of software and hardware (computers) of the von Neumann architecture of a universal or specialized form and in the general case contain:
  • the network card controller is implemented in the form of a board containing a processor memory and auxiliary logic, configured to process incoming and outgoing network packets with a transmission speed.
  • the central processing unit can be implemented in a single / multiprocessor version, controls the operation of the unit, executes instructions and processes the data stored in memory.
  • the memory of the corresponding nodes contains (compiled) blocks of the source code (compilation units) converted to the internal machine representation: interface module, informal and formal module of regex regular expression languages, neural network neural net, configuration broker, analytical nodes.
  • the interface module 4 contains a video controller - a microcircuit that forms a video image on the monitor of the system administrator.
  • the monitor is configured to display an image including a graphical interface and / or command line interface providing traffic channel management tools (SMPP, SMTP, other known from the prior art), defining message classification categories, defining and displaying simplified user strings - substring search patterns in a string, displaying the administrator’s input of the system of probabilistic coefficients, visual control by training and testing the neural network.
  • SMPP traffic channel management tools
  • Input devices keyboard, mouse, etc.
  • the data storage (hard disk) of the interface module 4 is intended for storing historical data for the purpose of post-processing by the neural net module 3, training samples and other data, and the hard disk controller for receiving, transmitting and processing data from the data storage.
  • Figure 2 illustrates the general scheme of a2p service, as is clear to a specialist, not limited to Fig.2, the invention can be used in P2A, A2A, P2P services.
  • SMS sent to the subscriber can be a bank SMS distribution, SMS advertising offers, SMS notifications of social networks and any other known from the prior art.
  • TSC Traffic Classification System
  • TSC Traffic Classification System
  • ESMEn Traffic Classification System
  • On figa shows a generalized sequence of SMPP TCP / IP calls A2P traffic to receive, analyze and send SMS messages SMPP protocol (specification 3GPP 3G TS 23.039).
  • the example illustrates SMS messages in the direction from the external network node ESME to the MS of the subscriber of the mobile radio communication system (SPRS).
  • SPRS mobile radio communication system
  • the dashed line shows the signal calls, the solid calls of the TSRNR.
  • a standard signaling dialog for establishing a connection between the ESME and SMSC of the SPRS subscriber takes place.
  • Signals of the SMPP protocol according to well-known rules, are stacked in the OSI protocol stack and transmitted via TCP / IP to the network.
  • the external ESME node After the connection is established, the external ESME node generating an SMS message by the same rules puts the SMS message deliver sm, data sm on the OSI protocol stack, fragmentes the TCP / IP packets and sends it to the network.
  • TCP / IP packets are sent to the IP address of the load balancer of the traffic classification system of Figure 1 9.
  • redirection of SMS messages from various ESMEs to the IP address of the load balancer can be implemented through a network intermediary that aggregates SMS messages from various ESMEs (so-called SMS aggregators, not shown in Fig.).
  • the load balancer in accordance with the forwarding algorithm, forwards packets to the IP address of the corresponding analytic node in Fig. 1. 8.
  • the analytic TCS processor process applies the analysis algorithm to SMS messages (SMS PDU analysis and processing) and then sends SMS to the network to SMSC (SMS Center) SPRS subscriber 4, and the message classification symptom is directed to the interested SPRS node, for example, to SCP billing node 5.
  • SMSC SMS Center
  • TCS stores all received SMS messages in the data store of user interface 4 and carries out their post-processing with modules 2 regex and 3 neural net 3, after which it transfers the results of the classification to the interested operator node.
  • This mode can be used in addition to online classification to clarify the classification results obtained in real time in Fig.2A is not shown.
  • Figure 2b shows npnMepSMPP TCP / IP traffic P2a calls for receiving, analyzing and sending SMS messages of the SMPP protocol (3GPP 3G TS 23.039 specification).
  • SMS as an example, from the SMSC of the SPRS operator to the external ESME node.
  • the dashed line shows signaling calls, continuous TCP / IP calls.
  • the clients of the traffic classification system in the SMS message classification option are mainly SPRS operators, and the network nodes are mobile subscriber stations (MS Mobil Station), short message service centers (SMSC Short Message Service Center) of the SPRS operator, service or application VAS Value Added Services ) SPRS operators or other external nodes (ESME External Short Message Entities).
  • Clients send SMS messages to the network IP address of the load balancer 9, which implements the routing algorithm of network packets for load balancing (not shown in FIG. Za, 36).
  • Client packet redirection can be implemented on the PCEF gateway node of the SPRS operator.
  • the SPRS subscriber sends an SMS message from his MS, a number of calls 1-2 are made using signaling protocols for delivering outgoing SMS to SMSC.
  • SMSC starts a typical SMPP request / response sequence as an SMS sender to an external ESME 3-4 node. After the connection is established, 3-4 SMSC puts the SMS message deliver sm, data sm on the OSI protocol stack, fragmentes it onto TCP / IP packets and sends PCEF 5 to the network through the SPRS gateway.
  • PCEF at the transport level replaces the recipient IP address of the TCP packet with the balancer's IP address load 9 of the traffic classification system 6.
  • the packets After routing the packets by the load balancer 9, the packets arrive at the network interface of the analytical node 8 TCS where they are collected in the SMS PDU and then the analysis and processing (SMS PDU analysis and processing) is performed.
  • the analytic node After classifying the message, the analytic node sends the message to the IP address of ESME 7, and sets the sender IPEF gateway address PCEF as the sender's IP address.
  • an SMS 8 confirmation message dialog is generated, and the TCS (Traffic Classification System) traffic classification system sends to the address of the interested node (for example, the SCP billing node) an SMS classification symptom of classification mark 8.
  • Post-processing of the saved message array within P2a traffic can be carried out similarly as described above for a2p traffic.
  • Fig. 3a illustrates the generalized algorithm of the traffic classification system by the example of a series-parallel mode of operation of the traffic classification system.
  • analytic node 8 receives an SMS message — network packets are captured, packets are aggregated into streams, packets (streams) are classified according to the application level protocol, data extraction and other standard DPI traffic analysis operations are performed.
  • the process pool manager initiates a process-handler that checks the message size, parses the message into significant fields for analysis (text, sender, recipient, service data, etc.) forms the corresponding data structure with separation into significant fields and places it in the analytical memory node 8.
  • the program logic of the analytical node provides for the transfer of the message structure to the interface module 4 for recording in the data warehouse 4 in order to accumulate historical data and post-processing.
  • Stage 5 offers the administrator the choice of a classification algorithm with Regex clear logic algorithms or the Neural net heuristic algorithm.
  • the system administrator is given the option to change the rules for analyzing the message. For example, if in messages in a random order, not possible for formalization (expression in the program logic of the algorithm), the syntax or semantics of the message changes, symbols of another alphabet appear and / or new words and phrases borrowed from other languages appear (for example, deadline, public relations , zoom, etc.).
  • the system administrator at his discretion, introduces simplified string patterns that are intuitive to him, which, from his point of view, are acceptable for recognizing the changed semantics of the message.
  • the input of informal templates-strings is carried out by the technical means of the regex module of the user interface configuration module 5 - the administrator defines informal user-template strings-templates.
  • the administrator can determine the probabilistic coefficients user factor Pf for assigning messages to a predefined category — type of traffic (category i), FIG. For, 36.
  • the administrator checks the correctness of the input on the monitor using the visual controls (graphical interface) drawn by the video controller of the user interface module 4.
  • user template data converted to a machine view is transmitted and stored in the memory of regex module 2 and then according to a set of machine instructions
  • the informal regular expression language irrigex module 2 converts custom patterns into regular expressions of the already formal regex language.
  • the transformation includes the programmatic generation of the irrigex struct data structure of the informal language irrigex and its transformation into the regex struct structure corresponding to the templates (regular expressions) of the already formal regex regular expression language.
  • the analysis rule compiler 1 forms the final configuration of analysis rules config - a program module that is an object code file with instructions for analyzing significant message fields and transfers it to memory for configuration broker 7, which reproduces the assigned set of copies of the configuration object code file and transfers it to the assigned analytical nodes 8.
  • the analytical node 8 places the configuration in the memory of the corresponding processor process, which uses it to process the text of the significant fields of the message and then generates a sign assigning a message to a predefined category, type of traffic - step 11.
  • the significant fields of the string message can be transmitted to the neural net module 3 to the input of the neural network (steps 14-18) to clarify the classification (step 11) or the message is transmitted to the network for sending to the recipient 13.
  • This "loop" allows you to specify the category of messages online using neural net after applying the regex rules.
  • module 3 neural net uses the machine representation of the probability coefficient Pf obtained from module neural net 6, step 15. After working out the classification algorithm for the neural network, steps 16, 17, the program algorithm for converting the threshold value of the neural net network according to the coefficient Pf - step 18.
  • the process the processor determines the category of the message and then passes to the network for sending - steps 11,12.
  • steps 14-18 steps 1-10 are applied to the next message, this achieves sequentially-parallel processing of messages.
  • Fig. Zb illustrates a block diagram of a parallel message processing.
  • clear classification algorithms based on the Regex rules — steps 5–9 and heuristic neural network classification algorithms Neural net — steps 13–17 are simultaneously applied to the significant fields of the string message.
  • the program algorithm of the processor process waits for the completion of the execution of both branches of the algorithm, steps 13-17 assume the transfer (copying) of the significant fields of the string message to module 3 neural net and the execution of the classification algorithm for the neural network.
  • a message classification symptom is formed (the program logic of steps 5–9 and 13–17 are performed similarly to the version of parallel-parallel processing). Further, the message and the classification attribute are sent to the recipients - steps 10-12.
  • FIG. 4 illustrates a data processing flowchart for a traffic classification system.
  • SMS pdu flow after network load balancing logic module 9 arrives at the network interface of the analytic node 8.
  • the analytical node receives the network packet 6 performs standard operations (DPI Deep Packet Inspection) of packet classification , aggregating them into streams, and according to the software algorithm of the processor process, generates an SMS struct 7 data structure that is a machine representation of the significant data of an SMS message - length length, sender sender, receiver, text message ia - a string string that includes semantic fields fieldn that are significant for analysis and classification.
  • DPI Deep Packet Inspection standard operations
  • SMS struct 7 data structure that is a machine representation of the significant data of an SMS message - length length, sender sender, receiver, text message ia - a string string that includes semantic fields fieldn that are significant for analysis and classification.
  • the user interface module 4 contains in memory the data defined by the system administrator: message classification categories - categoryi lb, informal line template - user template la, probability coefficient for assigning a message to a certain category - user factor Pf 1zie.
  • the regex module 2 based on the informal template-string 1a, according to the program algorithm of the informal regular expression language irrigex generates an irrigex struct 2 structure that includes data and methods for various operations such as searching for the occurrence of a template, performing a string replacement, etc., and saves it in the module memory.
  • the program algorithm of module 2 converts the data of the structure of the informal language irrigex struct 2 into the data structure of the corresponding standard language of regular expressions regex struct 3 that defines the rules for text analysis.
  • the program algorithm of the analysis rules compiler 1 collects the config file 4 - the software module for the analysis rules configuration that includes the data and methods for processing them to methods (..) of the regex struct 3.
  • Configuration broker 7 reproduces (copies) the number of config files specified by the program algorithm and distributes them by analytical nodes 8 (not shown in Fig. 4).
  • the processor process uses a software algorithm, applies the rules for analyzing the configuration file config to the meaningful fields of the string 8 message and generates a classification attribute based on the result, i.e. assigns a message to one of the predefined categoriesi 16 categories.
  • the process handler transfers the significant fields of the string message to module 3 neural net.
  • Significant fields of the message field n are fed to the input elements of the neural network 9, after the classification logic is completed, the output elements Ci of the neural network generate the corresponding probability coefficients Fai (Ci) to classify the message into a certain category 1.
  • the program algorithm for converting the threshold value of module 3 neural net converts the function of the output elements Ci (threshold value) taking into account the coefficient Pf 1 victim - Fai (Xi, Pf) 5 réelle. This provides an impact. a user probability coefficient Pf that a message is categorized as a message to a classification result. After that, a sign is formed for classifying the message, assigning it to a certain category 10.
  • the program algorithm of module 3 neural net assigns a sign for classifying messages of that category categoryi, the threshold value of which ⁇ fensiv of the corresponding output neuron Ci prevails and exceeds the confidence interval set by the user coefficient Pf.
  • the presented architecture of the traffic classification system assumes module spacing on different network nodes.
  • the system performance is achieved by processing messages with regex rules only on the analytical node 8.
  • Performance scaling along with the well-known measures taken by the load balancer, is achieved by reproducing and redistributing the configuration of the analysis rules by the broker of the configurations to the analytical nodes and maintaining their required number.
  • the system architecture allows the administrator to perform steps 6-10, 14-18 of FIG. 3a and 5-9, 13-17 of FIG. 3b in sequentially-parallel and parallel mode, respectively.
  • the system architecture allows you to apply custom settings - changing the operating mode, disabling / enabling regex and / or neural net processing, entering templates and probability coefficients, without interrupting the analysis of the online message flow.
  • the main property and advantage of the system proposed by the authors is the flexibility and universality of the classification classification by ordinary administrators of network nodes through the use of the author's informal regular expression language irregex, which performs its logic in the 3 regex module.
  • the functionality of the simplified input of text recognition templates allows you to remove restrictions on the maintenance of the device by highly specialized highly qualified specialists and provide the opportunity for its operation to ordinary telecommunication specialists.
  • the developers of semantic analysis solutions are inevitably forced to engage in the time-consuming, not always possible refinement of the architecture of the semantic analysis solution.
  • An accidental change in the message scheme and structure in the initial server settings by default cannot be determined, and it is not possible to foresee formalizing all possible message formats for obvious reasons.
  • module 4 of the user interface available for configuration translates the server proposed by the authors from the category of specialized equipment in the category of telecommunication network nodes available for configuration by ordinary experts in telecommunications.
  • the inclusion of the informal language irregex in the system architecture made it possible to interact with any various standard regex languages and connect them for analysis in accordance with the user's preferences.
  • the inclusion of user probability coefficients in the classification process, and the implementation of the neural net module with the ability to adjust the results of a neural network allows the system administrator to quickly respond to changes in the scheme and semantics of the message and promptly "tweak" the results of neural classification.
  • the classification method both in serial-parallel mode and in parallel, provides a significant increase in the classification accuracy due to the synergy of the regex and neural net algorithms.
  • the method and architecture of the traffic classification system provides indirect interaction of user settings, while the system administrator can configure them independently.
  • the proposed traffic classification system is demonstrated above by the example of SMPP traffic in the framework of the interaction of a2p, p2a.
  • the proposed solution can be used for semantic analysis of the message flow of P2P, A2A services, mainly small amounts of text.
  • other standards and applications including, for example, the SMPT specification for mail e-mail services, without going beyond the scope of legal protection of the proposed method for classifying traffic and device architecture for its implementation.
  • the present invention was tested in PJSC Megafon and PJSC VimpelCom, as a result of the work, the claimed technical effect is confirmed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)
  • Machine Translation (AREA)

Abstract

L'invention concerne un procédé et un système reposant sur celui-ci de classification de trafic, qui représentent un ajustement flexible et universel d'une classification par des administrateurs habituels de noeuds de réseau du fait de l'utilisation d'un langage d'auteur informel d'images régulières "irregex". La mise en oeuvre du système de classification en utilisant un langage d'auteur informel d'images régulières "irregex" permet de déterminer des limites de maintenance du système par des spécialistes de haut niveau à orientation étroite, et d'offrir la possibilité de l'exploiter par des spécialistes de télécommunication usuels. L'intégration dans le processus de la classification de coefficients de probabiblité d'utilisateurs et la mise en oeuvre d'un module de système de type réseau neuronal de classification avec la posssibilité de corriger les résultats de fonctionnement du réseau neuronal (architecture propre ou outils connexes) permet à l'administrateur du système de réagir rapidement à un changement du schéma et de la sématique de communication et de "corriger" les résultats de la classification neuronale. Le procédé de classification, par exemple en mode série-parallèle ainsi qu'en mode parallèle, assure une croissance importante de la précision de classification du fait de la synergie des algorythme De manière importante, le procédé permet, grâce à une architecture système de classification de trafic, une influence réciproque indirecte des ajustements d'utilisateur, et l'administrateur système peut les ajuster de manière indépendante.
PCT/RU2019/000715 2018-10-05 2019-10-07 Système de classification de trafic WO2020071962A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
RU2018135235 2018-10-05
RU2018135235A RU2697648C2 (ru) 2018-10-05 2018-10-05 Система классификации трафика

Publications (1)

Publication Number Publication Date
WO2020071962A1 true WO2020071962A1 (fr) 2020-04-09

Family

ID=64317094

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/RU2019/000715 WO2020071962A1 (fr) 2018-10-05 2019-10-07 Système de classification de trafic

Country Status (3)

Country Link
EA (1) EA201900470A3 (fr)
RU (1) RU2697648C2 (fr)
WO (1) WO2020071962A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112668340A (zh) * 2020-12-28 2021-04-16 北京捷通华声科技股份有限公司 一种信息处理方法及装置
CN113872918A (zh) * 2020-06-30 2021-12-31 苏州三六零智能安全科技有限公司 网络流量分类方法、设备、存储介质及装置

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110489102B (zh) * 2019-07-29 2021-06-18 东北大学 一种从自然语言自动生成Python代码的方法
CN110781950B (zh) * 2019-10-23 2023-06-30 新华三信息安全技术有限公司 一种报文处理方法及装置
WO2023033684A1 (fr) * 2021-09-04 2023-03-09 Акционерное Общество "Квантум А Рус" Procédé d'information mobile d'abonnés de réseau cellulaire

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050060295A1 (en) * 2003-09-12 2005-03-17 Sensory Networks, Inc. Statistical classification of high-speed network data through content inspection
US20160065510A1 (en) * 2005-06-29 2016-03-03 Mark Carlson Schema-based dynamic parse/build engine for parsing multi-format messages
US20180083903A1 (en) * 2016-09-21 2018-03-22 King Fahd University Of Petroleum And Minerals Spam filtering in multimodal mobile communication

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7096498B2 (en) * 2002-03-08 2006-08-22 Cipher Trust, Inc. Systems and methods for message threat management
US8364766B2 (en) * 2008-12-04 2013-01-29 Yahoo! Inc. Spam filtering based on statistics and token frequency modeling

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050060295A1 (en) * 2003-09-12 2005-03-17 Sensory Networks, Inc. Statistical classification of high-speed network data through content inspection
US20160065510A1 (en) * 2005-06-29 2016-03-03 Mark Carlson Schema-based dynamic parse/build engine for parsing multi-format messages
US20180083903A1 (en) * 2016-09-21 2018-03-22 King Fahd University Of Petroleum And Minerals Spam filtering in multimodal mobile communication

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113872918A (zh) * 2020-06-30 2021-12-31 苏州三六零智能安全科技有限公司 网络流量分类方法、设备、存储介质及装置
CN112668340A (zh) * 2020-12-28 2021-04-16 北京捷通华声科技股份有限公司 一种信息处理方法及装置

Also Published As

Publication number Publication date
EA201900470A3 (ru) 2020-10-30
RU2018135235A (ru) 2018-11-19
EA201900470A2 (ru) 2020-06-30
RU2018135235A3 (fr) 2019-05-22
RU2697648C2 (ru) 2019-08-15

Similar Documents

Publication Publication Date Title
RU2697648C2 (ru) Система классификации трафика
US10795992B2 (en) Self-adaptive application programming interface level security monitoring
US8676965B2 (en) Tracking high-level network transactions
US20200327444A1 (en) System and method for customer journey event representation learning and outcome prediction using neural sequence models
US8782223B2 (en) Systems and methods for categorizing network traffic content
US20220237567A1 (en) Chatbot system and method for applying for opportunities
US11269665B1 (en) Method and system for user experience personalization in data management systems using machine learning
JP2008211835A (ja) ペイロード検査を介したパケット・ルーティング、及びパブリッシュ／サブスクライブ型ネットワークにおけるサブスクリプション処理
AU2020229706B2 (en) Intent-driven contact center
CN103138981A (zh) 一种社交网络分析方法和装置
CN103248677A (zh) 互联网行为分析系统及其工作方法
CN102090039B (zh) 执行数据中间处理的方法、数据中间处理设备和信息系统
CN112015374B (zh) 一种基于自然语言的跨编程语言微服务集成系统
CN111343201A (zh) 一种快速适配协议变更的解析方法、装置及系统
CN116668520A (zh) 一种基于网关的服务编排方法、系统、设备及存储介质
CN116346660A (zh) 基于依赖替换服务的数据处理方法、装置、设备及介质
CN114006831B (zh) 报文数据处理方法及装置
KR20210000041A (ko) 로그 데이터의 실시간 분석 방법 및 그 장치
US20060034303A1 (en) System and method for managing transactions related to messages transmitted in a communication network
EA041004B1 (ru) Система классификации трафика
KR100522440B1 (ko) 대화형 인터페이스를 이용한 사용자 정황 수집방법
CN108809900B (zh) 一种统一资源访问的框架及方法
US12093162B1 (en) Block anchors for online log parsing
CN115250254B (zh) Netflow报文分发处理方法及装置
CN118555263A (zh) 消息处理系统和方法、消息更新方法及订阅请求处理方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19869929

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19869929

Country of ref document: EP

Kind code of ref document: A1