CN112134898A

CN112134898A - Network flow judgment method and system

Info

Publication number: CN112134898A
Application number: CN202011038845.1A
Authority: CN
Inventors: 刘敬良; 魏泽; 陶志豪
Original assignee: Beijing Didi Infinity Technology and Development Co Ltd
Current assignee: Beijing Didi Infinity Technology and Development Co Ltd
Priority date: 2020-09-28
Filing date: 2020-09-28
Publication date: 2020-12-25

Abstract

The embodiment of the application discloses a method and a system for judging network traffic. The network flow judging method comprises the following steps: acquiring a first flow rate; judging the flow type of the first flow according to a flow judgment engine, and dividing the first flow into judged flow or flow to be measured based on the judgment result of the flow judgment engine; obtaining a plurality of judged flows with the same flow type; and updating the knowledge base corresponding to the flow judgment engine based on the characteristic data of at least part of the plurality of judged flows.

Description

Network flow judgment method and system

Technical Field

The present application relates to the field of network security, and in particular, to a method and a system for determining network traffic.

Background

With the rapid development of network technologies, data security issues are becoming more and more concerned in the internet and related applications. The accurate and effective judgment of the encrypted traffic has important significance for network criminal behavior analysis, public opinion analysis, national information security and the like. Along with the increasing variety and the increasing scale of encrypted traffic on the internet, a new challenge is brought to traffic determination.

Therefore, how to better determine various types of encrypted traffic and detect abnormal traffic therein becomes an urgent problem to be solved.

Disclosure of Invention

One of embodiments of the present application provides a method for determining network traffic, including: acquiring a first flow rate; judging the flow type of the first flow according to a flow judgment engine, and dividing the first flow into judged flow or flow to be measured based on the judgment result of the flow judgment engine; obtaining a plurality of judged flows with the same flow type; and updating the knowledge base corresponding to the flow judgment engine based on the characteristic data of at least part of the plurality of judged flows.

One of embodiments of the present application provides a network traffic determination system, including: the device comprises an acquisition module, a flow type judgment module and an updating module; the acquisition module is used for acquiring a first flow; the flow type judging module is used for judging the flow type of the first flow according to a flow judging engine and dividing the first flow into judged flow or flow to be measured based on the judgment result of the flow judging engine; the updating module is used for acquiring a plurality of judged flows with the same flow type; the updating module is further configured to update the knowledge base corresponding to the traffic determination engine based on the feature data of at least part of the multiple determined flows.

One of the embodiments of the present application provides a network traffic determination device, including a processor, where the processor is configured to execute the network traffic determination method according to any embodiment of the present application.

One of the embodiments of the present application provides a computer-readable storage medium, where the storage medium stores computer instructions, and after a computer reads the computer instructions in the storage medium, the computer executes the network traffic determination method according to any embodiment of the present application.

Drawings

The present application will be further explained by way of exemplary embodiments, which will be described in detail by way of the accompanying drawings. These embodiments are not intended to be limiting, in which like numerals refer to like structures or operations, and wherein:

FIG. 1 is a block diagram of a network traffic decision system according to some embodiments of the present application;

FIG. 2 is a schematic diagram of an update module according to some embodiments of the present application;

FIG. 3 is a schematic diagram of a training module according to some embodiments of the present application;

FIG. 4 is an exemplary flow chart of a network traffic decision method according to some embodiments of the present application;

FIG. 5 is an exemplary flow chart of a method of determining flow to measure according to some embodiments of the present application;

FIG. 6 is an exemplary flow chart of a method of determining flow to measure according to some embodiments of the present application;

FIG. 7 is an exemplary flow diagram of a traffic classification model training method according to some embodiments of the present application;

FIG. 8 is an exemplary flow chart of a rule establishment method according to some embodiments of the present application;

FIG. 9 is an exemplary flow chart of a network traffic decision method according to some embodiments of the present application;

FIG. 10 is an exemplary flow chart of a network traffic determination method according to some embodiments of the present application

FIG. 11 is an exemplary flow chart of a network traffic decision method according to some embodiments of the present application;

fig. 12 is a schematic view of an application scenario of a network service system according to some embodiments of the present application.

Detailed Description

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings used in the description of the embodiments will be briefly introduced below. It is obvious that the drawings in the following description are only examples or embodiments of the application, from which the application can also be applied to other similar scenarios without inventive effort for a person skilled in the art. Unless otherwise apparent from the context, or otherwise indicated, like reference numbers in the figures refer to the same structure or operation.

It should be understood that "system", "device", "unit" and/or "module" as used herein is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, other words may be substituted by other expressions if they accomplish the same purpose.

As used in this application and the appended claims, the terms "a," "an," "the," and/or "the" are not intended to be inclusive in the singular, but rather are intended to be inclusive in the plural unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that steps and elements are included which are explicitly identified, that the steps and elements do not form an exclusive list, and that a method or apparatus may include other steps or elements.

Flow charts are used herein to illustrate operations performed by systems according to embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in the exact order in which they are performed. Rather, the various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to the processes, or a certain step or several steps of operations may be removed from the processes.

The embodiment of the specification relates to a network flow judgment method and a network flow judgment system. The network flow judgment method and the system are an overall scheme judgment design aiming at the flow of the internet data center, cross over the judgment of a protocol, establish a judgment mode based on the flow and construct a judgment thinking taking the flow as the center of gravity. The network traffic judging method can efficiently and accurately judge mass traffic and encrypted traffic by establishing a judging mode based on a model and/or a rule, establishing an automatic closed-loop knowledge base iteration mechanism, applying a traffic judging mode of a deep fit service mode and the like, and provides a premise and guarantee for network safety. In addition, the network traffic determination method and the network traffic determination system can also be applied to service mode security protection of the internet data center, and can solve the determination of private protocol traffic, encrypted traffic, abstract visual traffic class and the like of the internet data center.

Fig. 1 is a block diagram of a network traffic determination system according to some embodiments. As shown in fig. 1, the network traffic determination system 100 may include an obtaining module 110, a feature extracting module 120, a feature determining module 130, a traffic type determining module 140, a policy executing module 150, a traffic pattern determining module 112, a knowledge base invoking module 114, a traffic to be measured determining module 160, an updating module 170, a training module 180, and a rule determining module 190.

The acquisition module 110 may be used to acquire information/data in the network traffic decision process.

In some embodiments, the acquisition module 110 may be configured to acquire the first traffic. In some embodiments, the traffic may be interactive data packets in the network. The first traffic may be any one or more interactive data packets. In some embodiments, the disclosed network traffic determination system 100 may be applied to an Internet Data Center (IDC), and the obtaining module 110 may obtain the interactive Data packet entering an IDC border gateway as a first traffic. For example, when a user accesses an application service mode in the IDC and/or uses a service mode provided by the IDC, the user sends traffic (interactive data packet) to the IDC through the terminal, and the obtaining module 110 may obtain the interactive data packet sent by the user terminal as the first traffic. The network traffic determination system 100 disclosed in the embodiment can be applied not only to IDC but also to a server, a cloud platform, and other scenarios.

In some embodiments, the first traffic may be a previous packet or packets in a data stream (e.g., a data stream resulting from a single access). For example, the first traffic may be a header of a data packet of the data stream, a few first data packets, or all data packets of the data stream. The large number of data packets taken contributes to the improvement of the accuracy of the subsequent determination, but generally, the amount of calculation and the amount of data stored in the system are also increased. The number of the data packets to be fetched can be flexibly adjusted according to the property of the data stream and the requirement of a subsequent judgment strategy. This number can be dynamically adjusted according to system feedback, or periodically adjusted as a basic parameter of the system. In some embodiments, the first traffic is the first three packets of the data flow. By judging the previous one or more data packets of the data stream, the detection performance can be considered on the basis of effectively ensuring the judgment precision. In some embodiments, for a data stream, the fetch module 110 may fetch its packets in order to make the determination. For example, in some embodiments, the obtaining module 110 may obtain a first packet of the data flow, and when the traffic type of the first packet cannot be determined by the system 100, the obtaining module 110 obtains a second packet of the data flow. And so on until the system 100 determines the first packets of the data stream (e.g., the first three packets, the first five packets, etc.). If a packet is determined by the system 100 to be of a traffic type in the process, the acquisition of packets of the data stream may be stopped. In some embodiments, the system may set a threshold for obtaining an upper limit of the number of packets of the data stream, and when the number of packets of the data stream obtained by the obtaining module 110 reaches the threshold, and the system 100 cannot determine the traffic type of the packets, no more packets are obtained, but special processing is performed on the data stream, such as marking the data stream as undetermined traffic or traffic to be measured. In some embodiments, the threshold may be preset by the system, or may be dynamically adjusted according to system feedback; the threshold may be updated periodically or aperiodically; the updating manner of the threshold may include optimization updating based on a machine learning model, updating based on system feedback, updating based on manual input, etc., and the specific updating manner may refer to the related description of the updating module 170. In some embodiments, the first traffic may also include acknowledgement packets.

In some embodiments, the acquisition module 110 includes an abnormal flow determination sub-module. The abnormal flow determination submodule may process the flow through an abnormal flow determination model (e.g., the abnormal flow determination model 205) to determine whether the flow is a non-abnormal flow or an abnormal flow defined by the system. After traffic enters the IDC border gateway, the system 100 may first determine the traffic using an abnormal traffic determination model. When the abnormal traffic determination model determines that the traffic belongs to the abnormal traffic type, the system 100 may directly apply a corresponding policy (e.g., disallow access, report, alarm, special flag, abnormal report, etc.) to the abnormal traffic. When the abnormal flow determination model determines that the flow belongs to the non-abnormal flow type or cannot determine whether the flow is abnormal, the flow can be regarded as the non-abnormal flow. The obtaining module 110 may selectively obtain a certain type of flow after being determined by the abnormal flow determination model as the first flow.

In some embodiments, the abnormal traffic determination model may be a trained machine learning model. For example only, the abnormal flow determination model may include, but is not limited to, a combination of one or more of a neural network model, a support vector machine model, a k-nearest neighbor model, a decision tree model, and the like. The neural network model may include one or more of a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a multilayer neural network (MLP), a ballistic neural network (GAN), and the like. In some embodiments, the system 100 may obtain a plurality of corresponding sets of samples, including traffic and its corresponding anomaly/normal labels. The label can be automatically labeled according to the historical performance of the flow (such as whether abnormal influence is generated) or can be obtained by manual labeling. The system 100 may train a machine learning model using the corresponding set of multiple samples to obtain a trained abnormal flow determination model. For example only, the abnormal traffic determined by the abnormal traffic determination model may include, but is not limited to, one or more combinations of aggressive traffic, traffic for flushing data, traffic for obtaining data through abnormal steps, and the like.

By setting the abnormal flow determination model, the system 100 can perform quick preliminary screening on the flow, and is helpful to find the abnormal flow in time and take corresponding measures. Meanwhile, the abnormal flow judgment model is arranged, so that the calculation amount of the subsequent steps can be reduced. The abnormal flow rate determination model is provided as only one example, and some embodiments may not have the abnormal flow rate determination model.

In some embodiments, the acquisition module 110 includes a normal flow determination sub-module. The normal flow judgment submodule can process the flow through the normal flow judgment model and judge whether the flow is the normal flow defined by the system. When the normal traffic determination model determines that the traffic belongs to normal traffic, the system 100 may directly apply a corresponding policy (e.g., allow access) to the normal traffic. When the normal flow judgment model judges that the flow does not belong to the normal flow or cannot judge whether the flow is normal or not, the flow can be regarded as abnormal flow. The obtaining module 110 may selectively obtain a certain type of flow rate after being determined by the normal flow rate determination model as the first flow rate.

In some embodiments, the normal traffic decision model may be a trained machine learning model. For example only, the normal flow determination model may include, but is not limited to, a combination of one or more of a neural network model, a support vector machine model, a k-nearest neighbor model, a decision tree model, and the like. The neural network model may include one or more of a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a multilayer neural network (MLP), a ballistic neural network (GAN), and the like. In some embodiments, the system 100 may obtain a plurality of corresponding sets of samples, including traffic and its corresponding normal/abnormal labels. The label can be automatically labeled according to the historical performance of the flow (such as whether abnormal influence is generated) or can be obtained by manual labeling. The system 100 may train a machine learning model using the corresponding set of multiple samples to obtain a trained normal flow decision model.

By setting the normal flow determination model, the system 100 can perform quick preliminary screening on the flow, and is helpful to find the normal flow in time and take corresponding measures. Meanwhile, the normal flow judgment model is set, so that the calculation amount of the subsequent steps can be reduced. The setting of the normal flow rate determination model is only one example, and some examples may not have the normal flow rate determination model.

In some embodiments, the first flow rate may be a certain type of flow rate determined by a normal flow rate determination model and/or an abnormal flow rate determination model, and for example, may be a flow rate that is neither normal nor abnormal.

The feature extraction module 120 may be used to extract feature data of the traffic.

In some embodiments, the feature extraction module 120 may obtain feature data of the first traffic at a preset position of the first traffic according to the feature extraction rule. In some embodiments, the feature extraction rules may specify the number of preset positions, the starting position of a certain preset position, and the corresponding bit fetch length. The feature extraction rules may also specify the number of preset positions, the starting position and the ending position of a certain preset position. The feature extraction module 120 extracts a preset amount of feature data according to the feature extraction rule at a first flow preset position according to the feature extraction rule. The characteristic data of the first flow rate may be a data characteristic of the first flow rate at a preset position. The feature extraction module 120 may extract data of a specified length at one or more preset positions of the first traffic as feature data of the first traffic according to a feature extraction rule. The characteristic data length of each preset position can be the same or different. For example, the preset positions may include two, and the characteristic data length at each preset position is 2 bytes. In some embodiments, the feature extraction rule may be expressed as "offset: 0; depth:2 "(representing a start position of 0 th byte, a bit length of 2 bytes) and" offset: 6; depth:4 "(indicating the starting position is byte 6, and the bit length is 4 bytes). The feature extraction module 120 may obtain two bytes of data of the first flow at the two preset positions as feature data of the first flow. For example, the first traffic is at a preset position "offset: 0; the data at depth:2 "is" a 12 f "(in hexadecimal characters), the first traffic is at the preset position" offset: 6; the character at depth:4 "is" 5c 5e 1602 "(in hexadecimal notation). In some embodiments, the feature extraction rules may be stored in the knowledge base 290. The feature extraction module 120 may invoke corresponding feature extraction rules 294 from the knowledge base 290. The feature extraction rules may differ depending on the traffic type or traffic type. In some embodiments, the feature extraction rule may be preset by the system, or may be dynamically adjusted according to system feedback; the feature extraction rules can be updated regularly or irregularly; the updating manner of the feature extraction rule may include optimization updating based on a machine learning model, updating based on system feedback, updating based on manual input, and the like, and the specific updating manner may refer to the relevant description of the updating module 170.

The feature determination module 130 may be used to determine the type of feature to which the traffic belongs.

In some embodiments, the feature determination module 130 may determine the feature type of the first traffic according to a feature determination rule. The feature type may be a protocol (e.g., a standard network protocol, a custom protocol, etc.), or may be traffic with a certain feature specified by a given rule. One traffic type may contain one or more feature types, and in general, one feature type belongs to only one traffic type, but if it can be distinguished, more than two traffic types may also contain the same feature type.

In some embodiments, the feature determination module 130 may determine the feature type of the first traffic according to a feature determination rule based on the feature data of the first traffic. In some embodiments, the characteristic determination rule may determine the characteristic type of the first flow rate by determining whether the characteristic data of the first flow rate at one or more preset positions meets the determination rule. The feature data rule corresponding to the preset position in the feature determination rule may be determined when the feature determination rule is established or updated. For example only, the characteristic data of the first traffic includes data characteristics (e.g., characters) of the first traffic at two preset positions (e.g., "offset: 0; depth: 2" and "offset: 6; depth: 4"). When the feature data of the first flow rate at the first preset position matches the feature data feature of the feature type a at the preset position in the feature determination rule (for example, "a 12 f"), the feature determination module 130 may determine that the first flow rate belongs to the feature type a. In some embodiments, the feature decision rules may be stored in knowledge base 290. The feature decision module 130 may invoke the feature decision rule 296 from the knowledge base 290. In some embodiments, the characteristic determination rule may be preset by the system, or may be dynamically adjusted according to the system feedback; the feature decision rule may be updated periodically or aperiodically. The updating manner of the feature determination rule may include optimization updating based on a machine learning model, updating based on system feedback, updating based on manual input, and the like, and the specific updating manner may refer to the relevant description of the updating module 170 and the rule determination module 190. In some embodiments, the system 100 (e.g., the rule decision module 190) may establish initial feature extraction rules and feature decision rules via the rule establishment method 600. For more details on the rule establishment method 600, reference may be made to the rule decision module 190, FIG. 8, and related description thereof.

The traffic type determination module 140 may be used to determine the traffic type to which the traffic belongs.

In some embodiments, the traffic type determination module 140 may determine the traffic type of the first traffic based on the traffic classification model.

The traffic type may be a classification based on traffic. Multiple flows under the same flow type may have the same or different feature types. In some embodiments, a traffic type may contain one or more network protocols. In some embodiments, a traffic type may contain only one network protocol; in this case, the network protocol of all traffic under this traffic type is the same. In some embodiments, one traffic type may contain more than two different network protocols; in this case, there are two or more different network protocols of data traffic under this traffic type. In some embodiments, there may be both traffic types that contain only one network protocol and traffic types that contain more than two network protocols. In some embodiments, the network protocols may include a plaintext protocol and an encrypted protocol. The plaintext protocol is transmitted by adopting plaintext, and the encryption protocol is transmitted in an encryption mode. In some embodiments, at least one traffic type includes more than two different network protocols. Furthermore, at least one of the two different network protocols included in at least one traffic type requires encrypted transmission. In some embodiments, the one or more network protocols encompassed by the traffic type may include standard network protocols (e.g., HTTPS, HTTP, FTP, SMTP, SSH, NTP, SDP, POP3, etc.); custom network protocols may also be included. In some embodiments, the network protocols involved differ from one traffic type to another, in which case the number of traffic types is less than or equal to the number of protocol types. In some embodiments, the traffic type may be further subdivided; for example, a traffic type may include two or more sub-traffic types, each of which may contain one or more network protocols.

In some embodiments, the traffic classification model may be a trained machine learning model. For example only, the traffic classification model may include, but is not limited to, a combination of one or more of a neural network model, a support vector machine model, a k-nearest neighbor model, a decision tree model, and the like. The neural network model may include one or more of a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a multilayer neural network (MLP), a ballistic neural network (GAN), and the like. In some embodiments, the traffic classification model may be stored in knowledge base 290. Traffic type determination module 140 may invoke traffic classification model 292 from knowledge base 290. In some embodiments, the system 100 (e.g., the training module 180) may train to obtain an initial traffic classification model through the traffic classification model training method 500. For more details on the traffic classification model training method 500, reference may be made to fig. 7 and its associated description.

In some embodiments, the traffic type of the first traffic may include one or more. That is, the first traffic may belong to one or more traffic types simultaneously. For example only, the traffic type determination module 140 may determine probabilities that the first traffic belongs to the plurality of traffic types, respectively, according to a traffic classification model. In some embodiments, the traffic type determination module 140 may determine that the first traffic belongs to a certain traffic type when the probability that the first traffic belongs to the traffic type is greater than a set threshold (e.g., 0.5, 0.6, 0.8, etc.).

In some embodiments, the traffic type determination module 140 may determine the traffic type of the first traffic based on the characteristic type of the first traffic determined by the characteristic determination rule. For example, the traffic type determination module 140 may take the traffic type to which the feature type corresponds (e.g., belongs) as the traffic type of the first traffic.

In some embodiments, the traffic type of the first traffic determined according to the traffic classification model may be a first preliminary type; and judging the characteristic type of the first flow according to the characteristic judgment rule. The traffic type determination module 140 may determine the traffic type of the first traffic based on the first preliminary type and the characteristic type.

In some embodiments, the traffic type determination module 140 may determine the traffic type of the first traffic by comparing the first preliminary type and the characteristic type. Specifically, the traffic type determination module 140 may determine that the traffic type of the first traffic is a traffic type common to the first preliminary type and the feature type. When there is no common traffic type in the first preliminary type and the feature type, the traffic type determination module 140 may determine that the first traffic is the traffic to be measured 270. For example, if the first preliminary type of the first traffic is a and B and the characteristic type of the first traffic is a and C, the traffic type determination module 140 may determine that the traffic type of the first traffic is a. For another example, if the first preliminary type of the first flow rate is a and B, and the characteristic type of the first flow rate is C, the flow rate type determination module 140 may determine that the first flow rate is the flow rate to be measured 270. In some embodiments, when the traffic type determination module 140 determines the first traffic as the traffic to be determined 270, the system 100 may report a message that the first traffic is to be determined or is not determined; and meanwhile, the first flow can be stored in a data area of the flow to be measured.

In some embodiments, the traffic type determination module 140 may further determine the traffic type of the first traffic according to a traffic determination engine, and divide the first traffic into the determined traffic or the traffic to be measured based on the determination result of the traffic determination engine. In some embodiments, the traffic determination engine may determine the traffic type of the first traffic through a traffic classification model in the knowledge base, and/or a feature extraction rule and a feature determination rule.

In some embodiments, when the traffic type is excessive; for example, when the number of flow types is greater than a set threshold (e.g., 20, 30, 50, etc.), the system 100 may integrate multiple flow types into one; for example, multiple traffic types are integrated into a sub-traffic pattern, and the sub-traffic pattern is treated as a traffic type.

The policy enforcement module 150 may be used to take policies on traffic.

In some embodiments, the policy enforcement module 150 may take a corresponding policy on the first traffic based on the traffic type of the first traffic.

In some embodiments, the policy taken on the first traffic may include, but is not limited to: one or more of allowed access, disallowed access, bandwidth limitations, security level limitations, network traffic prioritization, reporting, alarms, special flags, and exception reporting. In some embodiments, the policy enforcement module 150 may adopt a corresponding policy for the first traffic according to a preset corresponding rule based on the traffic type of the first traffic. For example, traffic type a may correspond to a first policy, traffic type B may correspond to a second policy, and so on.

In some embodiments, the policy enforcement module 150 may take a corresponding policy on the first traffic based on the traffic pattern and/or the traffic type of the first traffic. In some embodiments, the policy enforcement module 150 may take a corresponding policy on the first traffic based only on the traffic pattern to which the first traffic belongs (regardless of the traffic type of the first traffic). For example, when the first traffic belongs to traffic pattern a, the policy enforcement module 150 may take a first policy on the first traffic; when the first traffic belongs to traffic pattern b, the policy enforcement module 150 may adopt a second policy on the first traffic. In some embodiments, the policy enforcement module 150 may apply a corresponding policy to the first traffic based on the traffic pattern to which the first traffic belongs and the traffic type of the first traffic. When the first traffic belongs to different traffic types in the same service mode, the same traffic type in different service modes, and different traffic types in different service modes, the policies corresponding to the first traffic may be the same, partially the same, or different. For example, the traffic patterns may include a traffic pattern a and a traffic pattern b; the traffic pattern a may contain traffic types a and B; traffic pattern b may contain traffic types a and C; when the first traffic belongs to the traffic pattern a and the traffic type a, the policy enforcement module 150 may adopt a first policy on the first traffic; when the first traffic belongs to the traffic pattern a and the traffic type B, the policy enforcement module 150 may adopt a second policy on the first traffic; when the first traffic belongs to the traffic pattern b and the traffic type a, the policy enforcement module 150 may adopt a third policy on the first traffic; when the first traffic belongs to traffic pattern b and traffic type C, the policy enforcement module 150 may adopt a fourth policy on the first traffic.

In some embodiments, the policy enforcement module 150 may also take a corresponding policy on the first traffic based on the traffic type of the first traffic and the data size of the first traffic. The data size of the first traffic may be the size of the data stream corresponding to the first traffic. For example, the first traffic belongs to a traffic type a (e.g., high risk traffic), and when data transmission of the first traffic is higher than a first threshold (e.g., 1MB, 3MB, etc.), the policy enforcement module 150 may apply a first policy (e.g., alarm) to the first traffic; when the data transmission of the first traffic exceeds a second threshold (e.g., 8MB, 10MB, 15MB, etc.), the policy enforcement module 150 may apply a second policy to the first traffic (e.g., flush the first traffic, disallow access, etc.). For another example, the first traffic belongs to a traffic type B (e.g., a medium-risk traffic), and when data transmission of the first traffic is higher than a third threshold (e.g., 20MB, 30MB, etc.), the policy enforcement module 150 may apply a third policy (e.g., report) to the first traffic. In some embodiments, the policy enforcement module 150 may also take a corresponding policy for the first traffic based on the traffic pattern to which the first traffic belongs, the traffic type of the first traffic, and the data size of the first traffic.

In some embodiments, the policy enforcement module 150 may also adopt policies such as data extraction, data detection, etc. for first traffic belonging to a particular traffic type (e.g., high risk traffic, sensitive traffic, etc.). For example, the policy enforcement module 150 may extract various fields of the first traffic, perform data-sensitive detection on the fields, and perform data security-related scanning on the first traffic through the data security detection engine.

The traffic pattern determination module 112 may be configured to determine a traffic pattern to which the traffic belongs.

In some embodiments, the traffic pattern determination module 112 may determine a traffic pattern corresponding to the first traffic volume.

The business model can be used for enterprises to produce products and services which can be delivered to users by applying scientific methods and production processes. By way of example only, the business model may include, but is not limited to, a financial business model, a network appointment business model, a shared-bicycle business model, a designated-drive business model, a public transportation business model, a navigation business model, a vehicle sales business model, an express business model, a network security business model, and the like. The traffic pattern corresponding to the first traffic may be a traffic pattern accessed by the first traffic. In some embodiments, a traffic pattern may contain one or more traffic types. In some embodiments, the traffic patterns may be further subdivided; for example, a traffic pattern may include two or more sub-traffic patterns, each of which may contain one or more traffic types. The traffic pattern determination module 112 may determine a traffic pattern and/or a sub-traffic pattern corresponding to the first traffic. In some embodiments, the types of traffic not involved between traffic patterns may be the same, partially the same, or completely different.

In some embodiments, the traffic pattern determination module 112 may determine the traffic pattern corresponding to the first traffic according to a traffic pattern determination model. In some embodiments, the traffic pattern classification model may be a trained machine learning model. By way of example only, the traffic pattern classification model may include, but is not limited to, a combination of one or more of a neural network model, a support vector machine model, a k-nearest neighbor model, a decision tree model, and the like. The neural network model may include one or more of a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a multilayer neural network (MLP), a ballistic neural network (GAN), and the like. In some embodiments, the system 100 may obtain a plurality of corresponding sets of samples, the corresponding sets of samples including traffic and its corresponding traffic pattern labels. The service mode label may be automatically labeled according to the historical representation of the traffic (e.g., which service mode the traffic finally visits), or may be obtained by manual labeling. The system 100 may train a machine learning model using the corresponding set of multiple samples to obtain a trained traffic pattern decision model.

In some embodiments, the traffic pattern determination module 112 may determine the traffic pattern corresponding to the first traffic volume in other manners. For example, the traffic pattern determination module 112 may determine, according to the similarity between the first traffic and the multiple pieces of traffic in different traffic patterns, that the traffic pattern with the highest similarity to the first traffic is the traffic pattern corresponding to the first traffic. For another example, the traffic pattern determination module 112 may determine the traffic pattern corresponding to the first traffic according to a traffic pattern determination rule. The traffic pattern determination rule may determine the traffic pattern corresponding to the first traffic by determining data characteristics of the first traffic at one or more locations. In some embodiments, the traffic pattern determination module 112 may also determine the traffic pattern corresponding to the first traffic volume by combining two or more manners, such as a traffic pattern determination model, a similarity determination, a rule determination, and the like.

The knowledge base retrieval module 114 may be used to retrieve models, rules, etc. in the knowledge base.

In some embodiments, the knowledge base invoking module 114 may invoke different traffic classification models, feature extraction rules, feature determination rules, business pattern determination models, business pattern determination rules, policy enforcement rules, update methods, training methods, rule determination methods, and the like, according to the system settings.

In some embodiments, the knowledge base invocation module 114 may invoke traffic classification models, feature extraction rules, and feature decision rules based on traffic patterns.

In some embodiments, the traffic classification model and the feature extraction rule and the feature determination rule corresponding to different traffic patterns may be the same or different. In some embodiments, the system 100 (e.g., the training module 180) may respectively train corresponding traffic classification models (e.g., the method 500 according to the traffic classification models) for different traffic patterns. The system 100 (e.g., the rule decision module 190) may also establish corresponding feature decision rules and feature decision rules for different traffic patterns (e.g., via the rule establishment method 600). The models and rules corresponding to different business models may be stored in knowledge base 290; the knowledge base retrieval module 114 may retrieve the corresponding traffic classification model 292, feature extraction rules 294, and feature decision rules 296 from the knowledge base 290 based on the traffic patterns.

In some embodiments, each traffic pattern may include one or more traffic types due to differences in the importance, focus, etc. of the different traffic patterns. For example, for a certain type of traffic pattern (e.g., a traffic pattern with a lower importance level), the traffic pattern may include only one traffic type, and all traffic in the traffic pattern belongs to the traffic type, and the system 100 will apply the same policy (e.g., allow access) to all traffic in the traffic pattern. For another example, for another type of traffic pattern (e.g., a traffic pattern with a higher importance level), the traffic pattern may include a plurality of traffic types (e.g., 5, 10, 20, etc.), and the system 100 may adopt different policies for the traffic of different traffic types in the traffic pattern.

In some embodiments, the knowledge base invoking module 114 may invoke the traffic classification model, the feature extraction rules, and the feature decision rules according to the level of the traffic pattern. In particular, each traffic pattern may have a corresponding rank. For example, the service mode classes may be classified into a high-level service mode, a middle-level service mode, and a low-level service mode, divided by the importance of the service mode. For another example, the traffic patterns may be classified into class 1 traffic patterns, class 2 traffic patterns, class 3 traffic patterns, class …, and class 10 traffic patterns according to the degree of interest of the traffic patterns. In some embodiments, the traffic classification model, the feature extraction rule, and the feature determination rule corresponding to each traffic pattern level may be the same. In some embodiments, the higher the importance degree or attention degree of the service pattern level is, the more traffic types corresponding to the service pattern level are, and the more complicated the corresponding traffic classification model, the feature extraction rule and the feature determination rule are.

The to-be-measured flow determination module 160 may be configured to determine a flow type of the to-be-measured flow.

In some embodiments, the flow rate to be measured determination module 160 may obtain a plurality of flow rates to be measured.

In some embodiments, the flow 270 to be determined may include a first flow for which a first preliminary type, a characteristic type, and/or a flow type cannot be determined. Specifically, when the flow type determination module 140 cannot determine the first preliminary type of the first flow according to the flow classification model, the first flow may be determined as the flow to be measured 270; when the feature determination module 130 cannot determine the feature type of the first flow rate according to the feature determination rule, the first flow rate may be determined as the flow rate to be measured 270; when there is no common traffic type in the first preliminary type and the feature type, the traffic type determination module 140 may determine the first traffic as the traffic to be measured 270. The flow to be measured differentiated in different stages can have the same or different labels.

In some embodiments, the flow determination module to be measured 160 may obtain a plurality of flow rates to be measured accumulated over a period of time (e.g., one day, one week, one month, etc.). In some embodiments, the to-be-measured flow determination module 160 may determine whether the plurality of to-be-measured flows are of known flow types or determined to be determined flows or unknown flows based on the clustering model. Specifically, the unknown flow rate may be a flow rate to be measured that is determined not to be a known flow rate type.

In some embodiments, the to-be-measured flow determination module 160 may cluster a plurality of to-be-measured flows by using an unsupervised clustering model to obtain at least one to-be-measured flow group; and judging whether the undetermined flow group is of a known flow type according to the feature extraction rule and the feature judgment rule. In some embodiments, the flow rate determination module to be determined 160 may obtain a plurality of flow rate types, and based on the plurality of flow rate types, determine whether the plurality of flow rates to be determined belong to the plurality of flow rate types by using a supervised clustering model; and determines that the flow of the plurality of flows to be measured does not belong to any flow type as unknown flow 276. For more details on the flow determination method to be measured, reference may be made to fig. 5-6 and their associated description.

In some embodiments, the flow to be measured 270 may include a first flow for which a first preliminary type, a characteristic type, and/or a flow type has been determined. In some embodiments, the first traffic for which the first preliminary type, characteristic type, and/or traffic type has been determined may be processed as the traffic to be measured 270, and the traffic type determination rule, characteristic determination rule, and/or various other types of related rules may be optimized, refined, or verified by processing the traffic to be measured (which may include only the determined type of traffic or a mixture of both the determined type and the undetermined type). In some embodiments, the flow to be measured 270 may be blended into a portion of the first flow for which the first preliminary type, the signature type, and/or the flow type has been determined, according to a ratio, to enable monitoring, optimization, verification, or comparison of the model and the training method.

The update module 170 may be used to update a knowledge base (e.g., models, rules, policies, etc. in the knowledge base). FIG. 2 is a schematic diagram of an update module according to some embodiments of the present application. As shown in fig. 2, the updating module 170 may further include a traffic classification model updating unit 172, a feature extraction rule updating unit 174, a feature determination rule updating unit 176, and a policy updating unit 178.

Specifically, the traffic classification model updating unit 172, the feature extraction rule updating unit 174, the feature determination rule updating unit 176, and the policy updating unit 178 may be respectively configured to update the traffic classification model, the feature extraction rule, the feature determination rule, and the policy in the knowledge base.

In some embodiments, for unknown traffic 276, the system 100 may notify an operator (e.g., an expert in the field of network traffic) for intervention, and the operator may perform feature extraction and classification on the unknown traffic. For example, an operator may classify one or more unknown flows into an existing flow type. For another example, the operator may determine one or more unknown flows as a new flow type. As another example, an operator may manually extract feature data for one or more unknown flows, which may include data features of the one or more unknown flows at one or more locations.

In some embodiments, the update module 170 may obtain the feature extraction result and the flow classification result of the unknown flow, and update the flow classification model 292, the feature extraction rule 294, and/or the feature determination rule 296 in the knowledge base 290 according to the feature extraction result and the flow classification result of the unknown flow. In some embodiments, the feature extraction result and the flow classification result of the unknown flow may be the result of manually performing feature extraction and flow classification on the unknown flow. Some embodiments can effectively supplement and/or correct the flow classification model 292, the feature extraction rule 294 and/or the feature determination rule 296 in the knowledge base by acquiring the feature extraction result and the classification result of the unknown flow manually and updating the knowledge base, thereby solving the uncertainty of the flow determination of the system 100 and ensuring the accuracy and the continuity of the future determination. In some embodiments, the feature extraction result and the flow classification result of the unknown flow may also be obtained by a machine (e.g., a corresponding model).

In some embodiments, the update module 170 (e.g., the classification model update unit 172) may update the traffic classification model 292 based on the classification result of the unknown traffic. For example, the update module 170 may retrain the traffic classification model 292 with a part or all of the corresponding training sample set of the original traffic classification model 292 by using the unknown traffic and the classification result thereof as a new corresponding training sample set. For another example, the update module 170 may update the parameters of the traffic classification model 292 by validating and/or testing the traffic classification model 292 using the unknown traffic and its classification results as a validation set and/or a test set of the traffic classification model 292.

In some embodiments, the update module 170 (e.g., the feature extraction rule update unit 174 and the feature decision rule update unit 176) may update the feature extraction rules 294 and the feature decision rules 296 in the knowledge base 290 according to the feature extraction results and the traffic classification results of the unknown traffic. For example only, the updating module 170 may update the feature extraction rule 294 according to the feature extraction result of the unknown flow, including but not limited to the number of preset positions in the feature extraction rule 294, the start position and the fetch length (or the start position and the end position) of the preset positions, and the like. The update module 170 may further update the feature decision rule 296 based on the traffic classification result of the unknown traffic, including but not limited to the predetermined data features of one or more predetermined locations in the feature decision rule 296.

In some embodiments, the update module 170 may obtain a plurality of flows (e.g., determined flows) of the same type of flow. In some embodiments, the determined flow 280 may include: the first flow of the flow type is determined by the flow type determination module 140, and the determined flow determined by the flow to be determined determination module 160. In some embodiments, the determined traffic 280 may also include historical traffic corresponding to the traffic type. In some embodiments, the update module 170 may obtain a plurality of determined flows at each of one or more flow types.

In some embodiments, the update module 170 may cluster the plurality of determined flows and obtain at least some of the plurality of determined flows based on the similarity. In some embodiments, the update module 170 may cluster the plurality of determined flows through an unsupervised clustering model. For example, the update module 170 may group the plurality of determined flows into two or more classes through an unsupervised clustering model. The unsupervised clustering model may include, but is not limited to, one or more combinations of a K-means clustering model, a hierarchical clustering model, a gaussian clustering model, and the like. On this basis, the update module 170 may calculate the similarity between the determined flows in each class. Further, the updating module 170 may obtain one or more types of determined traffic (i.e., the at least partial traffic) with high similarity. For example, the update module 170 may obtain one or more types of determined flows with a similarity higher than a set threshold (e.g., 50%, 60%, 80%, etc.). For another example, the updating module 170 may obtain the determined traffic remaining after the class 1-2 with the lowest similarity is removed.

In some embodiments, the update module 170 may extract the characteristic data of the at least part of the traffic. In some embodiments, the update module 170 (e.g., the feature extraction rule update unit 174 and the feature decision rule update unit 176) may extract feature data from the at least part of the traffic and update the feature extraction rules 294 and/or the feature decision rules 296 in the knowledge base 290 based on the extracted feature data of the at least part of the traffic.

In some embodiments, the update module 170 may extract data characteristics of the at least part of the traffic at one or more locations as characteristic data of the at least part of the traffic. In particular, the data characteristics of all or a majority (e.g., 60%, 80%, 90%, etc.) of the at least some of the flows at the one or more locations are the same. At the same time, the data characteristics of the traffic of the other traffic types at the one or more locations are not identical. The extracted feature data of the at least part of the traffic from the updating module 170 can be used to determine the traffic type corresponding to the feature data. In some embodiments, the one or more locations may be the same, partially the same, or completely different from the preset locations in the original feature extraction rules.

After extracting the feature data of the at least partial traffic, the update module 170 may construct feature extraction rules and feature determination rules based on the extracted features and update the original feature extraction rules 294 and feature determination rules 296 in the knowledge base 290 (e.g., replace the original rules with new rules).

Because there may be misjudged flows in the judged flows 280 (for example, a flow belonging to the flow type a is judged as the flow type B), the misjudged flows in each flow type can be effectively eliminated by clustering the judged flows with the same flow type; meanwhile, the flow rate with high similarity is subjected to feature extraction, and the knowledge base 290 is updated, so that the accuracy of judging the flow rate by the feature extraction rule 294 and the feature judgment rule 296 can be effectively improved.

In some embodiments, policies enforced on traffic may also be included in the knowledge base. One policy may correspond to one or more traffic patterns and/or one or more traffic types. The update module 170 (e.g., policy update unit 178) may update the policy. In some embodiments, the update module 170 may obtain information such as a policy manually made for the new traffic pattern, a policy made for the new traffic type, a policy change made for the original traffic pattern, or a policy change made for the original traffic type, and update the policy in the knowledge base according to the information.

The training module 180 may be used to train the model. FIG. 3 is a schematic diagram of a training module according to some embodiments of the present application. As shown in fig. 3, the training module 180 may include a traffic classification model training unit 182, an abnormal traffic determination model training unit 184, and a traffic pattern determination model training unit 186.

Specifically, the traffic classification model training unit 182, the abnormal traffic determination model training unit 184, and the traffic pattern determination model training unit 186 may be respectively used for training the traffic classification model, the abnormal traffic determination model, and the traffic pattern determination model.

In some embodiments, the training module 180 (e.g., the traffic classification model training unit 182) may collect a plurality of sample traffic over a period of time. For example, training module 180 may collect traffic received by an internet data center (or server, cloud platform, etc.) over a period of time (e.g., a day, a week, a month, etc.) as sample traffic. In some embodiments, the traffic classification model training unit 182 may obtain a classification result (e.g., a result of manual classification) for the plurality of sample traffics, where the classification result includes traffic types corresponding to the plurality of sample traffics. Specifically, an operator (e.g., an expert in the field of network traffic) may classify a plurality of sample flows, and determine a corresponding traffic type for each sample flow. In this process, the operator can also determine the type and number of the traffic types. In some embodiments, the traffic classification model training unit 182 may train the original machine learning model with a plurality of sample traffics as input and a traffic type corresponding to each sample traffic as a label to obtain an initial traffic classification model. In some embodiments, the raw machine learning model may include, but is not limited to, a combination of one or more of a neural network model, a support vector machine model, a k-nearest neighbor model, a decision tree model, and the like. The neural network model may include one or more of a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a multilayer neural network (MLP), a ballistic neural network (GAN), and the like. In some embodiments, the training module 180 (e.g., the traffic classification model training unit 182) may train to obtain corresponding initial traffic classification models by using the traffic classification model training method 500 for a plurality of different traffic patterns, respectively.

In some embodiments, the training module 180 (e.g., the abnormal traffic determination model training unit 184) may obtain a plurality of corresponding sets of samples, where the corresponding sets of samples include traffic and its corresponding abnormal/normal labels. The label can be automatically labeled according to the historical performance of the flow (such as whether abnormal influence is generated) or can be obtained by manual labeling. The abnormal traffic determination model training unit 184 may train a machine learning model using the plurality of sample correspondence groups to obtain a trained abnormal traffic determination model.

In some embodiments, the training module 180 (e.g., the traffic pattern decision model training unit 186) may obtain a plurality of corresponding sets of samples, where the corresponding sets of samples include traffic and corresponding traffic pattern labels. The service mode label may be automatically labeled according to the historical representation of the traffic (e.g., which service mode the traffic finally visits), or may be obtained by manual labeling. The traffic pattern decision model training unit 186 may train a machine learning model using the plurality of corresponding sets of samples to obtain a trained traffic pattern decision model.

In some embodiments, training module 180 may also be used to train other models (e.g., normal traffic decision models, etc.).

The rule decision module 190 may be used to establish, update, or refine rules. For example, the rule decision module 190 may be used to establish, update, or refine feature extraction rules and/or feature decision rules.

In some embodiments, the rule decision module 190 may collect a plurality of sample flows over a period of time. For example, the rule determination module 190 may collect traffic received by an internet data center (or server, cloud platform, etc.) over a period of time (e.g., a day, a week, a month, etc.) as sample traffic. In some embodiments, the at least one sample traffic type includes more than two different network protocols. In some embodiments, at least one of the two or more different network protocols included in the at least one sample traffic type requires encrypted transmission.

In some embodiments, the rule decision module 190 may cluster a plurality of sample traffic to obtain a plurality of sample traffic types. In some embodiments, the rule determination module 190 may cluster the plurality of sample traffic using an unsupervised clustering model to obtain a plurality of sample traffic types. The unsupervised clustering model may include, but is not limited to, one or more combinations of a K-means clustering model, a hierarchical clustering model, a gaussian clustering model, and the like. In some embodiments, the rule determination module 190 may preset the number of sample traffic types when an unsupervised clustering model is used to cluster a plurality of sample traffic.

In some embodiments, the rule decision module 190 may establish initial feature extraction rules and initial feature decision rules based on a plurality of sample traffic types. In some embodiments, the data characteristics of the traffic, which generally have fixed positions before and after encryption, are the same, or the data characteristic distribution rules before and after encryption are the same, so that the traffic (whether plaintext or ciphertext) can be distinguished by using the data characteristics with distinction degree. The rule decision module 190 may process the flow according to the above theory to establish an initial feature extraction rule and an initial feature decision rule.

In some embodiments, the rule determination module 190 may determine the preset position in the initial feature extraction rule based on the number of the plurality of sample traffic types. For example, the rule determination module 190 may determine the number of preset positions in the initial feature extraction rule based on the number of the plurality of sample traffic types. In some embodiments, the number of preset positions satisfies the following determination condition: the plurality of sample flow types can be distinguished by whether the number of preset positions respectively satisfies a preset data characteristic. For example, when the flow types are 2, the number of the preset positions is ≧ 2. Specifically, assuming that the flow types are a and B, the preset positions include positions 1 and 2; when the data characteristic of a certain flow at the position 1 meets the preset data characteristic, the flow can be considered to belong to the type A; when the data characteristic of a certain flow at the position 2 meets the preset data characteristic, the flow can be considered to belong to the type B; when a certain flow rate meets the preset data characteristics at the positions 1 and 2, the flow rate can be considered to belong to the types A and B; and when a certain flow rate does not meet the preset data characteristics at the positions 1 and 2, the flow rate can be considered to belong to the flow rate to be measured. In some embodiments, in order to enable the constructed rule to make a better decision on the flow rate in consideration of the diversity of subsequent flows and the iterative update of the rule, a case where the data feature of the preset position is consistent with the preset data feature may be used for the decision, and a case where the data feature of the preset position is not consistent with the preset data feature may not be used for the decision.

In some embodiments, the number of preset positions may be a minimum number that can satisfy the determination condition. For example, when the flow types are 20, the number of the preset positions may be 6. Specifically, a combination of 3 positions can be selected from 6 positions to be used for determining the traffic type. According to the combination formula, 3 positions are arbitrarily selected from 6 positions, namely C (6,3) is 20 combinations, and 20 different preset position configurations can be formed by selecting 3 positions from the minimum 6 positions. In some embodiments, a threshold (e.g., 7, 8, etc.) may be set for the number of predetermined positions for a combination of computational speed, performance, and efficiency. In some embodiments, the number threshold of preset locations may be different based on the type of traffic; the number threshold of the preset positions can be preset by the system and can also be dynamically adjusted according to system feedback; the number threshold of the preset positions can be updated regularly or irregularly; the updating manner of the number threshold of the preset positions may include optimization updating based on a machine learning model, updating based on system feedback, updating based on manual input, and the like, and the specific updating manner may refer to the relevant description of the updating module 170. The combination of the plurality of preset positions in the preset positions may be provided by a knowledge base. For example, if there are 6 preset positions, the rule determination mode may select 1, 2, 3, or up to 6 positions of feature data from the 6 preset positions according to a predetermined algorithm to perform feature data extraction and determination; or different feature data combinations can be selected from 3 positions in the 6 preset positions according to a preset algorithm for extraction and judgment; when various combinations of the preset positions cannot meet the requirement for distinguishing various flow types, the number of the preset positions can be increased, and iterative updating is performed on the selection of the preset positions.

In some embodiments, the rule determination module 190 may further determine a start position and a fetch length (or a start position and an end position) of the preset positions based on the determined number of the preset positions, thereby determining the initial feature extraction rule. For example, the rule determination module 190 may extract data characteristics at one or more locations from a plurality of sample flows under each sample flow type, such as data characteristics of all or most (e.g., 60%, 80%, 90%, etc.) of the plurality of sample flows at the one or more locations are similar or even identical, or conform to a certain rule; while the data characteristics of the traffic at the one or more locations are more clearly distinguished for the other sample traffic types.

After determining the feature extraction rule, the rule determination module 190 may construct a feature determination rule based on features of a plurality of sample flows under each sample flow type. In some embodiments, the initial feature extraction rule includes a plurality of preset positions, and the feature data of at least one sample flow type corresponding to at least one preset position conforms to the feature determination rule. In some embodiments, a combination of the feature data corresponding to the at least two preset positions of the at least one sample flow type complies with the feature determination rule.

In some embodiments, the rule decision module 190 may establish initial feature extraction rules and initial feature decision rules according to the rule establishment method 600. In some embodiments, the rule decision module 190 may verify, optimize, update, or even refine the feature extraction rules and the feature decision rules. In some embodiments, the feature extraction rules and feature decision rules may be updated periodically or aperiodically; the updating modes of the feature extraction rule and the feature judgment rule may include optimization updating based on a machine learning model, updating based on system feedback, updating based on manual input, and the like, and the specific updating mode may refer to the relevant description of the updating module 170.

In some embodiments, the rule determination module 190 may respectively use the rule establishment method 600 to establish a corresponding feature extraction rule and a feature determination rule for a plurality of different business models.

It should be understood that the system and its modules shown in FIG. 1 may be implemented in a variety of ways. For example, in some embodiments, the system and its modules may be implemented in hardware, software, or a combination of software and hardware. Wherein the hardware portion may be implemented using dedicated logic; the software portions may be stored in a memory for execution by a suitable instruction execution system, such as a microprocessor or specially designed hardware. Those skilled in the art will appreciate that the methods and systems described above may be implemented using computer executable instructions and/or embodied in processor control code, such code being provided, for example, on a carrier medium such as a diskette, CD-or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The system and its modules of the present application may be implemented not only by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but also by software executed by various types of processors, for example, or by a combination of the above hardware circuits and software (e.g., firmware).

It should be noted that the above descriptions of the network traffic determination system and its modules are merely for convenience of description, and are not intended to limit the present application within the scope of the illustrated embodiments. It will be appreciated by those skilled in the art that, given the teachings of the present system, any combination of modules or sub-system configurations may be used to connect to other modules without departing from such teachings. For example, the feature extraction module 120, the feature determination module 130, and the flow type determination module 140 disclosed in fig. 1 may be different modules in a system, or may be a module that implements the functions of two or more modules described above. For example, each module may share one memory module, and each module may have its own memory module. Such variations are within the scope of the present application.

Fig. 4 illustrates an exemplary flow diagram of a network traffic determination method of some embodiments. The network traffic decision method 200 may be implemented by the network traffic decision system 100. As shown in fig. 4, the network traffic determination method 200 may include:

step 210, a first flow rate is obtained. Specifically, step 210 may be performed by the network traffic determination system 100 (e.g., the obtaining module 110).

In some embodiments, the traffic may be interactive data packets in the network. The first traffic may be any one or more interactive data packets. In some embodiments, the first traffic may be a previous packet or packets in a data stream (e.g., a data stream resulting from a single access). For example, the first traffic may be a header of a data packet of the data stream, a few first data packets, or all data packets of the data stream. By judging the previous one or more data packets of the data stream, the detection performance can be considered on the basis of effectively ensuring the judgment precision.

In some embodiments, the acquisition module 110 includes an abnormal flow determination sub-module. The abnormal flow determination submodule may process the flow through the abnormal flow determination model 205 to determine whether the flow is a non-abnormal flow or an abnormal flow defined by the system. After traffic enters the IDC border gateway, the system 100 may first determine the traffic using the abnormal traffic determination model 205. When the abnormal traffic determination model 205 determines that the traffic belongs to the abnormal traffic type, the system 100 may directly apply a corresponding policy (e.g., disallow access, report, alarm, special flag, abnormal report, etc.) to the abnormal traffic. When the abnormal traffic determination model 205 determines that the traffic is of a non-abnormal traffic type or cannot determine whether the traffic is abnormal, the traffic may be regarded as non-abnormal traffic. The obtaining module 110 may selectively obtain a certain type of traffic after being determined by the abnormal traffic determination model 205 as the first traffic. By setting the abnormal flow determination model 205, the system 100 can perform quick preliminary screening on the flow, and is helpful to find the abnormal flow in time and take corresponding measures. Meanwhile, the provision of the abnormal flow determination model 205 can reduce the amount of calculation in the subsequent steps. The abnormal traffic determination model 205 is provided as only one example, and some embodiments may not have the abnormal traffic determination model 205.

In some embodiments, the first flow rate may be a certain type of flow rate determined by a normal flow rate determination model and/or an abnormal flow rate determination model, and for example, may be a flow rate that is neither normal nor abnormal. For more details on acquiring the first traffic, reference may be made to the related description of the acquisition module 110.

Step 220, a first preliminary type of the first flow rate is determined according to the flow rate classification model. Specifically, step 220 may be performed by the network traffic determination system 100 (e.g., the traffic type determination module 140).

In some embodiments, the first preliminary type refers to a preliminary traffic type of the first traffic as determined by the traffic classification model. In some embodiments, the system 100 (e.g., the training module 180) may train to obtain an initial traffic classification model through the traffic classification model training method 500. For more details on the traffic classification model training method 500, reference may be made to fig. 7 and its associated description.

In some embodiments, the first preliminary type of first flow rate may include one or more. That is, the first traffic may belong to one or more traffic types simultaneously. For example only, the traffic type determination module 140 may determine probabilities that the first traffic belongs to the plurality of first preliminary types, respectively, according to the traffic classification model. When the probability that the first flow belongs to a certain flow type is greater than a set threshold (e.g., 0.5, 0.6, 0.8, etc.), the flow type determination module 140 may determine that the first flow belongs to the certain flow type. Further details regarding the determination of the first preliminary type of the first traffic may be found in relation to the description of the traffic type determination module 140.

In some embodiments, the system 100 performs step 230-240 after the traffic type determination module 140 determines the first preliminary type of the first traffic based on the traffic classification model. In some embodiments, when the traffic type determination module 140 cannot determine the first preliminary type of the first traffic according to the traffic classification model, the first traffic may be determined as the traffic to be measured 270.

And step 230, acquiring feature data of the first flow at a preset position according to the feature extraction rule. Specifically, step 230 may be performed by the network traffic decision system 100 (e.g., the feature extraction module 120).

In some embodiments, the feature extraction module 120 may obtain feature data of the first traffic at a preset position of the first traffic according to the feature extraction rule. The feature extraction rules may specify the number of preset positions, the starting position of a certain preset position, and the corresponding bit extraction length. The feature extraction rules may also specify the number of preset positions, the starting position and the ending position of a certain preset position. The characteristic data of the first flow rate may be a data characteristic of the first flow rate at a preset position. The feature extraction module 120 may extract data of a specified length at one or more preset positions of the first traffic as feature data of the first traffic according to a feature extraction rule. In some embodiments, the feature extraction rules may be stored in the knowledge base 290. The feature extraction module 120 may invoke feature extraction rules 294 from the knowledge base 290. For more details on extracting feature data of the first traffic, reference may be made to the relevant description of the feature extraction module 120.

And step 240, judging the characteristic type of the first flow according to the characteristic judgment rule based on the characteristic data of the first flow. Specifically, step 240 may be performed by network traffic decision system 100 (e.g., feature decision module 130).

In some embodiments, the feature determination rule may determine the feature type of the first traffic by determining whether the data features of the first traffic at the one or more predetermined locations meet predetermined data features. For more details on determining the feature type of the first flow, reference may be made to the associated description of the feature determination module 130.

In some embodiments, the system 100 may perform step 250 after the feature determination module 130 determines the feature type of the first flow according to the feature determination rule. In some embodiments, the system 100 may determine the first flow rate as the flow rate to be measured 270 when the feature determination module 130 cannot determine the feature type of the first flow rate according to the feature determination rule.

In some embodiments, the system 100 may also perform

steps

212 and 214 before performing

steps

220, 230, and 240. In some embodiments, each traffic pattern may include one or more traffic types due to differences in the importance, focus, etc. of the different traffic patterns. When determining network traffic, the system 100 may first determine a traffic pattern corresponding to the network traffic, and then call a traffic classification model 292, feature extraction rules 294, and/or feature determination rules 296 corresponding to the traffic pattern from the knowledge base 290; therefore, the system 100 can perform personalized flow judgment according to different service modes, and the flow judgment efficiency is effectively improved.

Step 212, determine the traffic pattern corresponding to the first traffic. Specifically, step 212 may be performed by the traffic determination system 100 (e.g., the traffic pattern determination module 112).

In some embodiments, the traffic pattern determination module 112 may determine the traffic pattern corresponding to the first traffic according to a traffic pattern determination model. In some embodiments, the traffic pattern classification model may be a trained machine learning model. In some embodiments, the traffic pattern determination module 112 may determine the traffic pattern corresponding to the first traffic volume in other manners. For example, the traffic pattern determination module 112 may determine, according to the similarity between the first traffic and the multiple pieces of traffic in different traffic patterns, that the traffic pattern with the highest similarity to the first traffic is the traffic pattern corresponding to the first traffic. For another example, the traffic pattern determination module 112 may determine the traffic pattern corresponding to the first traffic according to a traffic pattern determination rule. The traffic pattern determination rule may determine the traffic pattern corresponding to the first traffic by determining data characteristics of the first traffic at one or more locations. In some embodiments, the traffic pattern determination module 112 may also determine the traffic pattern corresponding to the first traffic volume by combining two or more manners, such as a traffic pattern determination model, a similarity determination, a rule determination, and the like. For more details on determining the traffic pattern corresponding to the first traffic flow, reference may be made to the relevant description of the traffic pattern determination module 112.

Step 214, based on the traffic pattern, a traffic classification model, a feature extraction rule and a feature determination rule are called. In particular, step 214 may be performed by the traffic determination system 100 (e.g., the knowledge base retrieval module 114).

In some embodiments, the traffic classification model and the feature extraction rule and the feature determination rule corresponding to different traffic patterns may be the same or different. In some embodiments, the system 100 (e.g., the training module 180) may respectively train corresponding traffic classification models (e.g., the method 500 according to the traffic classification models) for different traffic patterns. The system 100 (e.g., the rule decision module 190) may also establish corresponding feature decision rules and feature decision rules for different traffic patterns (e.g., via the rule establishment method 600). The models and rules corresponding to different business models may be stored in knowledge base 290; the knowledge base retrieval module 114 may retrieve the corresponding traffic classification model 292, feature extraction rules 294, and feature decision rules 296 from the knowledge base 290 based on the traffic patterns. In some embodiments, the knowledge base invoking module 114 may invoke the traffic classification model, the feature extraction rules, and the feature decision rules according to the level of the traffic pattern. More details about invoking the traffic classification model, the feature extraction rules, and the feature decision rules may refer to the relevant description of the knowledge base invocation module 114.

And step 250, judging the flow type of the first flow based on the first preliminary type and the characteristic type. Specifically, step 250 may be performed by the network traffic decision system 100 (e.g., the traffic type decision module 140).

In some embodiments, the traffic type determination module 140 may determine the traffic type of the first traffic by comparing the first preliminary type and the characteristic type. Specifically, the traffic type determination module 140 may determine that the traffic type of the first traffic is a traffic type common to the first preliminary type and the feature type. When there is no common traffic type in the first preliminary type and the feature type, the traffic type determination module 140 may determine that the first traffic is the traffic to be measured 270. More details regarding determining the traffic type of the first traffic may be found in relation to the description of the traffic type determination module 140.

The flow type of the first flow is judged through the flow classification model, the feature extraction rule and the feature judgment rule, and the accuracy of flow (especially encrypted flow) judgment can be effectively improved.

After the traffic type determination module 140 determines the traffic type of the first traffic, the system 100 may proceed to step 260. At the same time, the system 100 may allocate the first flow into the determined flow 280 (e.g., stored in a database of determined flows).

And step 260, adopting a corresponding strategy for the first flow based on the flow type of the first flow. In particular, step 260 may be performed by the network traffic decision system 100 (e.g., the policy enforcement module 150).

In some embodiments, the policy taken on the first traffic may include, but is not limited to: one or more of allowed access, disallowed access, bandwidth limitations, security level limitations, network traffic prioritization, reporting, alarms, special flags, and exception reporting. In some embodiments, the policy enforcement module 150 may adopt a corresponding policy for the first traffic according to a preset corresponding rule based on the traffic type of the first traffic. For example, traffic type a may correspond to a first policy, traffic type B may correspond to a second policy, and so on. In some embodiments, the policy enforcement module 150 may take a corresponding policy on the first traffic based on the traffic pattern and/or the traffic type of the first traffic. In some embodiments, the policy enforcement module 150 may also take a corresponding policy on the first traffic based on the traffic type of the first traffic and the data size of the first traffic. In some embodiments, the policy enforcement module 150 may also adopt policies such as data extraction, data detection, etc. for first traffic belonging to a particular traffic type (e.g., high risk traffic, sensitive traffic, etc.). Further details regarding taking a corresponding policy for the first traffic may be found in relation to the policy enforcement module 150.

In some embodiments, the steps 210-260 may be online execution steps, which can perform online determination on the first traffic. In some embodiments, after determining that the first flow rate is the flow rate to be measured 270 or the determined flow rate 280 through the above steps, the system 100 may further process the flow rate to be measured 270 and/or the determined flow rate 280 to update the flow rate classification model 292, the feature extraction rules 294, and/or the feature determination rules 296 in the knowledge base 290.

Step 272, obtain a plurality of flows to be measured. Specifically, step 272 may be performed by network traffic determination system 100 (e.g., to-be-determined traffic determination module 160).

In some embodiments, the flow 270 to be determined may include a first flow for which a first preliminary type, a characteristic type, and/or a flow type cannot be determined. In some embodiments, the flow determination module to be measured 160 may obtain a plurality of flow rates to be measured accumulated over a period of time (e.g., one day, one week, one month, etc.). In some embodiments, the flow to be measured 270 may include a first flow for which a first preliminary type, a characteristic type, and/or a flow type has been determined. For more details on obtaining the plurality of flows to be measured, reference may be made to the related description of the flow to be measured determination module 160.

Step 274, determining whether the plurality of predetermined flows to be measured are of known flow types based on the clustering model. Specifically, step 274 may be performed by network traffic determination system 100 (e.g., to-be-determined traffic determination module 160).

Through step 274, the system 100 can re-determine the plurality of flows to be measured, thereby classifying the flows to be measured as determined flows 280 or unknown flows 276, and effectively reducing the probability of system misdetermination. In addition, the system 100 may further update the knowledge base 290 through processing the unknown traffic, so as to optimize the traffic classification model 292, the feature extraction rule 294 and/or the feature determination rule 296 in the knowledge base 290, thereby improving the accuracy of the subsequent traffic determination.

Step 278, obtaining the feature extraction result and the flow classification result of the unknown flow. Specifically, step 278 may be performed by network traffic determination system 100 (e.g., update module 170).

In some embodiments, for unknown traffic 276, the system 100 may notify an operator (e.g., an expert in the field of network traffic) for intervention, and the operator may perform feature extraction and classification on the unknown traffic. In some embodiments, the update module 170 may obtain the feature extraction result and the flow classification result of the unknown flow, and update the flow classification model 292, the feature extraction rule 294, and/or the feature determination rule 296 in the knowledge base 290 according to the feature extraction result and the flow classification result of the unknown flow. In some embodiments, the feature extraction result and the flow classification result of the unknown flow may be the result of manually performing feature extraction and flow classification on the unknown flow. By acquiring the feature extraction result and classification result of the artificial unknown flow and updating the knowledge base, the flow classification model 292, the feature extraction rule 294 and/or the feature determination rule 296 in the knowledge base can be effectively supplemented and/or corrected, the uncertainty of the flow determination of the system 100 can be solved, and the future determination accuracy and continuity can be guaranteed. In some embodiments, the feature extraction result and the flow classification result of the unknown flow may also be obtained by a machine (e.g., a corresponding model). For more details on obtaining the feature extraction results and the flow classification results of the unknown flow and updating the knowledge base, reference may be made to the relevant description of the updating module 170.

In some embodiments, for the determined flow 280, the system 100 may further perform deep feature extraction on it and update the feature extraction rules 294 and the feature determination rules 296 in the knowledge base 290 with the deep extracted features. For details regarding the deep feature extraction and updating of the knowledge base 290 for the determined flow 280, see steps 282-286, described below.

In step 282, a plurality of determined flows of the same flow type are obtained. Specifically, step 282 may be performed by the network traffic decision system 100 (e.g., the update module 170).

In some embodiments, as shown in FIG. 4, the determined flow 280 may include: the first flow of the flow type is determined by the flow type determination module 140, and the determined flow determined by the flow to be determined determination module 160. In some embodiments, the determined traffic 280 may also include historical traffic corresponding to the traffic type. In some embodiments, the update module 170 may obtain a plurality of determined flows at each of one or more flow types.

Step 284, clustering the plurality of determined flows, and obtaining at least part of the plurality of determined flows based on the similarity. Specifically, step 284 may be performed by network traffic decision system 100 (e.g., update module 170).

In some embodiments, the update module 170 may cluster the plurality of determined flows through an unsupervised clustering model. On this basis, the update module 170 may calculate the similarity between the determined flows in each class. Further, the updating module 170 may obtain one or more types of determined traffic (i.e., the at least partial traffic) with high similarity to execute step 286.

At step 286, feature data of the at least partial flow is extracted. In particular, step 286 may be performed by network traffic determination system 100 (e.g., update module 170).

In some embodiments, the update module 170 (e.g., the feature extraction rule update unit 174 and the feature decision rule update unit 176) may extract feature data from the at least part of the traffic and update the feature extraction rules 294 and/or the feature decision rules 296 in the knowledge base 290 based on the extracted feature data of the at least part of the traffic. Further details regarding obtaining the determined flows, clustering the plurality of determined flows and obtaining at least a portion of the flows, extracting feature data of at least a portion of the flows, and updating the knowledge base 290 may be found in relation to the description of the update module 170.

It should be noted that the above description of the network traffic determination method 200 is for illustration and explanation only, and does not limit the application scope of the present application. Various modifications and changes may be made to the network traffic determination method 200 by those skilled in the art in light of the teachings herein. However, such modifications and variations are intended to be within the scope of the present application. For example, steps 220 and 230 may be two steps performed independently, and both may be performed sequentially or in parallel. For another example, steps 212 and 214 may be omitted and the system 100 may not determine the traffic pattern to which the first traffic belongs. For another example, for a traffic pattern that only includes one traffic type, the system 100 may skip the step of determining the traffic type of the first traffic (e.g., step 220 and step 250), and perform the policy on the first traffic directly according to the traffic pattern to which the first traffic belongs. For another example, for a partial traffic mode (e.g., a low-risk traffic mode), the accuracy requirement of the system 100 for determining the first traffic is not high, and at this time, the system 100 may omit the step 230 and the step 250, and determine the traffic type of the first traffic only according to the traffic classification model. In some embodiments, the determination rate of the first flow rate by the system 100 may be increased after the system 100 has been operating for a period of time, and the step 278 may be omitted when the determination rate of the first flow rate by the system 100 is greater than a set threshold (e.g., 95%, 98%, 99%, etc.). In some embodiments, when the determination rate of the first flow rate by the feature extraction rule 294 and the feature determination rule 296 is greater than a set threshold (e.g., 95%, 98%, 99%, etc.), the system 100 may collect only the flow rate to be measured without collecting the determined flow rate to update the knowledge base.

Fig. 5 is an exemplary flow chart of a method for determining a flow to be measured according to some embodiments. The method 300 for determining traffic to be measured can be implemented by the network traffic determination system 100 (e.g., the traffic to be measured determination module 160). As shown in fig. 5, the method 300 for determining a flow to be measured may include:

in step 272, a plurality of flows 270 to be measured are obtained.

In some embodiments, the flow 270 to be determined may include a first flow for which a first preliminary type, a characteristic type, and/or a flow type cannot be determined. In some embodiments, the flow determination module to be measured 160 may obtain a plurality of flow rates to be measured accumulated over a period of time (e.g., one day, one week, one month, etc.). For more details on obtaining the plurality of flows to be measured, reference may be made to the related description of the flow to be measured determination module 160.

And 310, clustering the plurality of flows to be measured by using an unsupervised clustering model to obtain at least one group of flows to be measured.

In some embodiments, the unsupervised clustering model may include, but is not limited to, a combination of one or more of a K-means clustering model, a hierarchical clustering model, a gaussian clustering model, and the like. Through step 310, the to-be-measured flow rate determination module 160 may group similar flows of the plurality of to-be-measured flows together to form a group of to-be-measured flows.

And step 320, judging whether the undetermined flow group is a known flow type according to the feature extraction rule and the feature judgment rule.

In some embodiments, the flow determination module to be determined 160 may extract feature data (e.g., characters of a preset position) of each flow in the flow group to be determined according to the feature extraction rule. Then, the to-be-measured flow determination module 160 may determine, based on the feature data of each flow, a flow type corresponding to each flow according to the feature determination rule. In some embodiments, when all flows in the pending flow group are determined to belong to a certain flow type (e.g., a first known flow type) according to the feature extraction rule and the feature determination rule, the to-be-measured flow determination module 160 may determine that the pending flow group is a determined flow. In some embodiments, when it is determined that at least two flows in the pending flow group belong to different flow types according to the feature extraction rule and the feature determination rule, the to-be-determined flow determination module 160 may determine that the pending flow group is an unknown flow. In some embodiments, when the traffic type of at least one traffic in the pending traffic group cannot be determined according to the feature extraction rule and the feature determination rule, the traffic to be determined determination module 160 may determine that the pending traffic group is an unknown traffic.

Since there may be some flows in the flow to be measured 270 that are missed in determination, the determined flows that can be determined in the flow to be measured can be effectively sorted out by performing unsupervised clustering on the flow to be measured and performing determination based on rules. Thereby improving the efficiency of processing the unknown flow 276 and the determined flow 280.

Fig. 6 is an exemplary flow chart of a method for determining a flow to be measured according to some embodiments. The traffic to be measured determination method 400 may be implemented by the network traffic determination system 100 (e.g., the traffic to be measured determination module 160). As shown in fig. 6, the method 400 for determining a flow to be measured may include:

step 272, obtain a plurality of flows to be measured.

At step 410, a plurality of traffic types are obtained.

In some embodiments, the traffic to be determined determination module 160 may obtain all traffic types in the knowledge base 290. In some embodiments, the to-be-measured traffic determination module 160 may also obtain all traffic types related to the service mode corresponding to the plurality of to-be-measured traffic in the knowledge base 290.

And step 420, based on the multiple traffic types, judging whether the multiple pieces of flow to be measured belong to the multiple traffic types by using a supervised clustering model.

In some embodiments, the to-be-measured flow determination module 160 may calculate the similarity between each of the to-be-measured flows and the plurality of flow types, and determine whether to classify the to-be-measured flow into one or more flow types based on the similarity. For example, the to-be-measured flow determination module 160 may calculate an average value of the similarity between the to-be-measured flow and a plurality of flows in a certain flow type, and use the average value as the similarity between the to-be-measured flow and the flow type. For another example, the to-be-measured flow determination module 160 may calculate the similarity between the feature data (such as characters of a preset position) of the to-be-measured flow and characters of a preset position (which may be determined from the feature extraction rule and the feature determination rule) in a certain flow type. In some embodiments, when the similarity between the flow to be measured and a certain flow type is greater than a set threshold (e.g., 70%, 80%, etc.), the flow to be measured determination module 160 may determine that the flow to be measured belongs to the flow type; at the same time, the flow to be measured is considered to be the determined flow 280. In some embodiments, the supervised clustering model may be a machine learning model that may classify a plurality of traffic flows to be measured (e.g., into one or more traffic types). The machine learning model may include, but is not limited to, a combination of one or more of a neural network model, a support vector machine model, a k-nearest neighbor model, a decision tree model, and the like.

And 430, judging the flow which does not belong to any flow type in the plurality of flows to be measured as unknown flow.

In some embodiments, the to-be-measured flow determination module 160 may determine that the to-be-measured flow is the unknown flow 276 when the to-be-measured flow does not belong to any one of the flow types.

FIG. 7 is an exemplary flow diagram illustrating a method for traffic classification model training in accordance with some embodiments. The traffic classification model training method may be implemented by the network traffic decision system 100 (e.g., the training module 180). As shown in fig. 7, the traffic classification model training method 500 may include:

multiple sample flows are collected over a period of time, step 510.

In some embodiments, training module 180 may collect traffic received by an internet data center (or server, cloud platform, etc.) over a period of time (e.g., a day, a week, a month, etc.) as sample traffic.

Step 520, obtaining a classification result of the plurality of sample flows, where the classification result includes flow types corresponding to the plurality of sample flows.

In some embodiments, an operator (e.g., an expert in the field of network traffic) may classify a plurality of sample traffic and determine a corresponding traffic type for each sample traffic. In this process, the operator can also determine the type and number of the traffic types.

Step 530, training the original machine learning model by taking a plurality of sample flows as input and taking the flow type corresponding to each sample flow as a label to obtain an initial flow classification model.

In some embodiments, the raw machine learning model may include, but is not limited to, a combination of one or more of a neural network model, a support vector machine model, a k-nearest neighbor model, a decision tree model, and the like. The neural network model may include one or more of a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a multilayer neural network (MLP), a ballistic neural network (GAN), and the like.

In some embodiments, for a plurality of different traffic patterns, the training module 180 may respectively train to obtain corresponding initial traffic classification models by using the traffic classification model training method 500. Specifically, in step 510, the training module 180 may collect a plurality of sample flows in a specific service mode within a period of time. In step 520, the training module 180 may obtain the classification result of the plurality of sample traffic, and the classification result may be related to the specific traffic pattern. For example, the plurality of sample traffic may be divided into two or more traffic types according to the degree of importance and the degree of attention of the traffic pattern. In step 530, the training module 180 may train the original machine learning model with a plurality of sample flows as input and a flow type corresponding to each sample flow as a label to obtain an initial flow classification model corresponding to the specific traffic pattern. For more details on the traffic classification model training, reference may be made to the associated description of training module 180.

FIG. 8 illustrates an exemplary flow diagram of a rule establishment methodology, in accordance with some embodiments. The rule establishing method may be implemented by the network traffic decision system 100 (e.g., the rule decision module 190). As shown in fig. 8, the rule establishing method 600 may include:

at step 610, a plurality of sample flows over a period of time are collected.

In some embodiments, the rule determination module 190 may collect traffic received by an internet data center (or server, cloud platform, etc.) over a period of time (e.g., a day, a week, a month, etc.) as sample traffic. In some embodiments, the at least one sample traffic type includes more than two different network protocols. In some embodiments, at least one of the two or more different network protocols included in the at least one sample traffic type requires encrypted transmission.

Step 620, clustering the plurality of sample flows to obtain a plurality of sample flow types.

In some embodiments, the rule determination module 190 may cluster the plurality of sample traffic using an unsupervised clustering model to obtain a plurality of sample traffic types. The unsupervised clustering model may include, but is not limited to, one or more combinations of a K-means clustering model, a hierarchical clustering model, a gaussian clustering model, and the like. In some embodiments, the rule determination module 190 may preset the number of sample traffic types when an unsupervised clustering model is used to cluster a plurality of sample traffic.

Step 630, based on the plurality of sample traffic types, an initial feature extraction rule and an initial feature decision rule are established.

In some embodiments, the rule determination module 190 may determine the number of preset positions in the initial feature extraction rule based on the number of the plurality of sample traffic types. In some embodiments, the number of preset positions satisfies the following determination condition: the plurality of sample flow types can be distinguished by the fact that these number of preset positions satisfy a preset data characteristic. For example, when the flow types are 2, the number of the preset positions is ≧ 2. Specifically, assuming that the flow types are a and B, the preset positions include positions 1 and 2; when the data characteristic of a certain flow at the position 1 meets the preset data characteristic, the flow can be considered to belong to the type A; when the data characteristic of a certain flow at the position 2 meets the preset data characteristic, the flow can be considered to belong to the type B; when a certain flow rate meets the preset data characteristics at the positions 1 and 2, the flow rate can be considered to belong to the types A and B; and when a certain flow rate does not meet the preset data characteristics at the positions 1 and 2, the flow rate can be considered to belong to the flow rate to be measured. In some embodiments, in order to enable the constructed rule to make a better decision on the flow rate in consideration of the diversity of subsequent flows and the iterative update of the rule, a case where the data feature of the preset position is consistent with the preset data feature may be used for the decision, and a case where the data feature of the preset position is not consistent with the preset data feature may not be used for the decision.

In some embodiments, the number of preset positions may be a minimum number that can satisfy the determination condition. For example, when the flow types are 20, the number of the preset positions may be 6. Specifically, a combination of 3 positions can be selected from 6 positions to be used for determining the traffic type. C (6,3) ═ 20, so 20 traffic types can be distinguished by a minimum of 6 positions. In this calculation process, it is considered that C (6,2) ═ C (6,4) ═ 15<20, that is, the number of positions (e.g., 2 bits and 4 bits) selected from the 6 positions is smaller than the number of traffic types that can be distinguished by selecting 3 bits. And from 6 positions it is not possible to select both 3 and 2 position combinations (otherwise duplication would result and traffic types could not be distinguished uniquely). It can be determined that the minimum preset number of positions capable of satisfying the determination condition is 6 when the flow rate types are 20.

In some embodiments, the rule determination module 190 may further determine a start position and a fetch length (or a start position and an end position) of the preset positions based on the determined number of the preset positions, thereby determining the initial feature extraction rule. For example, the rule determination module 190 may extract data characteristics at one or more locations from a plurality of sample flows under each sample flow type, such as data characteristics of all or a majority (e.g., 60%, 80%, 90%, etc.) of the plurality of sample flows at the one or more locations being the same; at the same time, the data characteristics of the flow at the one or more locations are not exactly the same for the other sample flow types.

After determining the initial feature extraction rule, the rule determination module 190 may further construct a feature determination rule based on features of a plurality of sample flows under each sample flow type. In some embodiments, the initial feature extraction rule includes a plurality of preset positions, and the feature data of at least one sample flow type corresponding to at least one preset position conforms to the feature determination rule. In some embodiments, a combination of the feature data corresponding to the at least two preset positions of the at least one sample flow type complies with the feature determination rule.

In some embodiments, for a plurality of different traffic patterns, the rule determination module 190 may respectively employ the rule establishment method 600 to determine the corresponding initial feature extraction rule and initial feature determination rule. For example, in step 610, the rule decision module 190 may collect a plurality of sample flows in a specific traffic pattern over a period of time. In step 620, the rule determining module 190 may set the number of the traffic types according to the importance degree and the attention degree of the traffic pattern, and cluster a plurality of sample traffics based on the set number of the traffic types to obtain a plurality of sample traffic types. In step 630, the rule determination module 190 may determine an initial feature extraction rule and an initial feature determination rule corresponding to the specific traffic pattern based on the plurality of sample traffic types.

Fig. 9 is an exemplary flow diagram of a network traffic decision method in accordance with some embodiments. The network traffic decision method 700 may be implemented by the network traffic decision system 100. As shown in fig. 9, a network traffic determination method 700 may include:

step 710, a first flow rate is obtained. Specifically, step 710 may be performed by the network traffic determination system 100 (e.g., the obtaining module 110). For details of acquiring the first traffic, reference may be made to the relevant description of the acquisition module 110.

And 720, judging the flow type of the first flow according to the flow judgment engine, and dividing the first flow into the judged flow or the flow to be measured based on the judgment result of the flow judgment engine. Specifically, step 720 may be performed by the network traffic determination system 100 (e.g., the traffic type determination module 140).

In some embodiments, the traffic decision engine may be a processing device (e.g., processor, module, unit, etc.) that makes decisions on traffic. In some embodiments, the traffic determination engine may determine the traffic type of the first traffic through a traffic classification model in the knowledge base, and/or a feature extraction rule and a feature determination rule. For example, the traffic type determination module 140 may retrieve the traffic classification model from the knowledge base, and/or the feature extraction rule and the feature determination rule, and determine the traffic type of the first traffic according to the traffic determination engine. In some embodiments, when a traffic type of the first traffic is determined according to the traffic determination engine, the traffic type determination module 140 may divide the first traffic into the determined traffic 750; the traffic type determination module 140 may divide the first traffic into the traffic to be measured 740 when the traffic type of the first traffic cannot be determined according to the traffic determination engine.

In some embodiments, the traffic determination engine may determine the traffic type of the first traffic through a traffic classification model. The traffic classification model may be a trained machine learning model.

In some embodiments, the traffic determination engine may extract feature data of the first traffic through the feature extraction rule and determine the feature type of the first traffic through the feature determination rule. In some embodiments, the traffic determination engine may determine the traffic type of the first traffic based on the characteristic type of the first traffic. For example, the traffic type of the first traffic may be the traffic type to which the characteristic type of the first traffic corresponds (e.g., belongs).

In some embodiments, the traffic type of the first traffic determined according to the traffic classification model may be a first preliminary type; and judging the characteristic type of the first flow according to the characteristic judgment rule. The traffic determination engine may determine a traffic type for the first traffic based on the first preliminary type and the feature type. In some embodiments, the traffic type determination module 140 may determine the traffic type of the first traffic by comparing the first preliminary type and the characteristic type. Specifically, the traffic type determination module 140 may determine that the traffic type of the first traffic is a traffic type common to the first preliminary type and the feature type. When there is no common traffic type in the first preliminary type and the feature type, the traffic type determination module 140 may determine the first traffic as the traffic to be measured 740. More details regarding determining the traffic type of the first traffic may be found in relation to the description of the traffic type determination module 140.

In some embodiments, the system 100 (e.g., the traffic pattern determination module 112) may determine the traffic pattern corresponding to the first traffic flow according to a traffic pattern determination model. The system 100 (e.g., the knowledge base invoking module 114) may invoke a traffic classification model corresponding to the traffic decision engine, and/or feature extraction rules and feature decision rules based on the traffic pattern corresponding to the first traffic.

Step 730, based on the traffic type of the first traffic, a corresponding policy is applied to the first traffic. Specifically, step 730 may be performed by the network traffic decision system 100 (e.g., the policy enforcement module 150). Details regarding taking a corresponding policy for the first traffic may be found in relation to the policy enforcement module 150.

And 741, acquiring a plurality of fixed flows to be detected. Specifically, step 741 may be executed by network traffic determination system 100 (e.g., to-be-determined traffic determination module 160).

In some embodiments, the flow to be determined 740 may include a first flow for which the type of flow cannot be determined. In some embodiments, the flow determination module to be measured 160 may obtain a plurality of flow rates to be measured accumulated over a period of time (e.g., one day, one week, one month, etc.). For more details on obtaining the plurality of flows to be measured, reference may be made to the related description of the flow to be measured determination module 160.

Step 742, based on the clustering model, determining whether the plurality of pieces of flow to be measured are of known flow types. Specifically, step 742 can be performed by the network traffic determination system 100 (e.g., the traffic determination module to be determined 160).

In some embodiments, the to-be-measured flow determination module 160 may cluster a plurality of to-be-measured flows by using an unsupervised clustering model to obtain at least one to-be-measured flow group; and judging whether the undetermined flow group is of a known flow type according to the feature extraction rule and the feature judgment rule. In some embodiments, the flow rate determination module to be determined 160 may obtain a plurality of flow rate types, and based on the plurality of flow rate types, determine whether the plurality of flow rates to be determined belong to the plurality of flow rate types by using a supervised clustering model; and determines the flow which does not belong to any flow type in the plurality of flows to be measured as unknown flow 743. For more details on the flow determination method to be measured, reference may be made to fig. 5-6 and their associated description.

Through step 742, the system 100 can re-determine the plurality of flows to be measured, thereby classifying the flows to be measured into determined flows 750 or unknown flows 743, and effectively reducing the probability of system misdetermination. In addition, the system 100 may further update the knowledge base by processing the unknown traffic, so as to optimize the traffic classification model, the feature extraction rule, and/or the feature determination rule in the knowledge base, thereby improving the accuracy of the subsequent traffic determination.

Step 744, obtaining the feature extraction result and the flow classification result of the unknown flow. Specifically, step 744 may be performed by the network traffic determination system 100 (e.g., the update module 170).

In some embodiments, the feature extraction result and the flow classification result of the unknown flow may be the result of manually performing feature extraction and flow classification on the unknown flow. In some embodiments, the feature extraction result and the flow classification result of the unknown flow may also be obtained by a machine (e.g., a corresponding model).

Step 745, updating the knowledge base according to the feature extraction result and the flow classification result of the unknown flow. In particular, step 745 can be performed by network traffic decision system 100 (e.g., update module 170).

By obtaining the feature extraction result and the classification result of the unknown flow and updating the knowledge base, the flow classification model, the feature extraction rule and/or the feature judgment rule in the knowledge base can be effectively supplemented and/or corrected, the uncertainty of the flow judgment of the system 100 can be solved, and the future judgment precision and the future judgment continuity can be guaranteed. For more details on obtaining the feature extraction results and the flow classification results of the unknown flow and updating the knowledge base, reference may be made to the relevant description of the updating module 170.

In step 751, a plurality of determined flows of the same flow type are obtained. Specifically, step 751 can be performed by network traffic decision system 100 (e.g., update module 170).

In some embodiments, the determined flow 750 may include: the first flow of the flow type is determined according to the flow determination engine, and the determined flow determined by the flow to be determined determination module 160. In some embodiments, the determined traffic 750 may further include historical traffic corresponding to the traffic type. In some embodiments, the update module 170 may obtain a plurality of determined flows at each of one or more flow types.

Step 752, updating the knowledge base corresponding to the traffic decision engine based on the feature data of at least part of the plurality of determined traffic. Specifically, step 752 can be performed by network traffic decision system 100 (e.g., update module 170).

In some embodiments, the update module 170 may cluster the plurality of determined flows and obtain at least some of the plurality of determined flows based on the similarity. The updating module 170 (e.g., the feature extraction rule updating unit 174 and the feature determination rule updating unit 176) may extract feature data from the at least part of the traffic, and update the knowledge base corresponding to the traffic determination engine based on the extracted feature data of the at least part of the traffic. For example, the update module 170 may update feature extraction rules and/or feature decision rules in the knowledge base. For more details on obtaining the determined traffic and updating the knowledge base, reference may be made to the relevant description of the update module 170.

Fig. 10 is an exemplary flow diagram of a network traffic decision method in accordance with some embodiments. The network traffic decision method 800 may be implemented by the network traffic decision system 100. As shown in fig. 10, a network traffic determination method 800 may include:

step 810, obtaining a first flow. Specifically, step 810 may be performed by the network traffic determination system 100 (e.g., the obtaining module 110). For details of acquiring the first traffic, reference may be made to the relevant description of the acquisition module 110.

Step 820, determining a traffic pattern corresponding to the first traffic according to a traffic pattern determination model. Specifically, step 820 may be performed by the traffic determination system 100 (e.g., the traffic pattern determination module 112). For details of determining the traffic pattern corresponding to the first traffic flow, reference may be made to the relevant description of the traffic pattern determination module 112.

Step 830, a traffic classification model is invoked based on the traffic pattern. Specifically, step 830 may be performed by the traffic determination system 100 (e.g., the knowledge base retrieval module 114).

In some embodiments, the traffic classification models corresponding to different traffic patterns may be the same or different. The flow classification models corresponding to different service modes can be stored in a knowledge base; the knowledge base retrieval module 114 may retrieve the corresponding traffic classification model from the knowledge base based on the traffic patterns. For more details on invoking the traffic classification model, reference may be made to the relevant description of the knowledge base invocation module 114.

In step 840, the traffic type of the first traffic is determined based on the traffic classification model. Specifically, step 840 may be performed by network traffic decision system 100 (e.g., traffic type decision module 140). Details regarding the determination of the traffic type of the first traffic may refer to the related description of the traffic type determination module 140.

Step 850, feature extraction rules and feature decision rules are invoked based on the traffic patterns. In particular, step 850 may be performed by the traffic determination system 100 (e.g., the knowledge base retrieval module 114).

In some embodiments, the feature extraction rule and the feature determination rule corresponding to different business modes may be the same or different. The feature extraction rules and the feature judgment rules corresponding to different business modes can be stored in a knowledge base; the knowledge base retrieving module 114 may retrieve corresponding feature extraction rules and feature determination rules from the knowledge base based on the business model. For more details on invoking the feature extraction rules and the feature decision rules, reference may be made to the relevant description of the knowledge base invocation module 114.

And step 860, judging the flow type of the first flow based on the feature extraction rule and the feature judgment rule. Specifically, step 860 may be performed by network traffic decision system 100 (e.g., traffic type decision module 140).

In some embodiments, the system 100 (e.g., the feature extraction module 120) may obtain the feature data of the first traffic at a preset position of the first traffic according to the feature extraction rule. The system 100 (e.g., the feature determination module 130) may determine the feature type of the first traffic according to the feature determination rule based on the feature data of the first traffic. The system 100 (e.g., the traffic type determination module 140) may determine the traffic type of the first traffic based on the feature type of the first traffic determined by the feature determination rule. For example, the traffic type determination module 140 may take the traffic type to which the feature type corresponds (e.g., belongs) as the traffic type of the first traffic. More details regarding determining the traffic type of the first traffic may be found in relation to the description of the traffic type determination module 140.

Based on the traffic pattern and/or the traffic type of the first traffic, a corresponding policy is applied to the first traffic, step 870. Specifically, step 870 may be performed by network traffic decision system 100 (e.g., policy enforcement module 150).

In some embodiments, the policy enforcement module 150 may take a corresponding policy on the first traffic based only on the traffic pattern to which the first traffic belongs (regardless of the traffic type of the first traffic). In some embodiments, the policy enforcement module 150 may apply a corresponding policy to the first traffic based on the traffic pattern to which the first traffic belongs and the traffic type of the first traffic. When the first traffic belongs to different traffic types in the same service mode, the same traffic type in different service modes, and different traffic types in different service modes, the policies corresponding to the first traffic may be the same, partially the same, or different. Further details regarding taking a corresponding policy for the first traffic may be found in relation to the policy enforcement module 150.

Fig. 11 is an exemplary flow diagram of a network traffic decision method according to some embodiments. The network traffic decision method 900 may be implemented by the network traffic decision system 100. As shown in fig. 11, a method 900 for determining network traffic may include:

at step 910, a first flow rate is obtained. Specifically, step 910 may be performed by the network traffic determination system 100 (e.g., the obtaining module 110). For details of acquiring the first traffic, reference may be made to the relevant description of the acquisition module 110.

Step 920, obtaining a feature extraction rule and a feature judgment rule. Specifically, step 920 may be performed by the network traffic determination system 100 (e.g., the knowledge base invoking module 114).

In some embodiments, the knowledge base retrieval module 114 may retrieve the feature extraction rules and the feature decision rules from the knowledge base. In some embodiments, the knowledge base invoking module 114 may invoke the corresponding feature extraction rule and the feature decision rule based on the traffic pattern of the first traffic. In some embodiments, the feature extraction rules and feature decision rules may be established according to the rule establishment method 600. For more details on the acquisition of the feature extraction rules and the feature determination rules, reference may be made to the relevant description of the knowledge base retrieval module 114.

And step 930, acquiring feature data of the first flow at a preset position of the first flow according to the feature extraction rule. Specifically, step 930 may be performed by the network traffic decision system 100 (e.g., the feature extraction module 120). For more details on extracting feature data of the first traffic, reference may be made to the relevant description of the feature extraction module 120.

And 940, judging the flow type of the first flow according to the characteristic judgment rule based on the characteristic data of the first flow. Specifically, step 940 may be performed by the network traffic determination system 100 (e.g., the feature determination module 130). In some embodiments, the feature determination module 130 may determine the feature type of the first traffic according to a feature determination rule. In some embodiments, the system 100 (e.g., the characteristic determination module 130 or the traffic type determination module 140) may determine a traffic type corresponding to the characteristic type of the first traffic as the traffic type of the first traffic. For more details regarding the determination of the traffic type of the first traffic according to the characteristic determination rule, reference may be made to the relevant description of the characteristic determination module 130.

Step 950, based on the traffic type of the first traffic, a corresponding policy is applied to the first traffic. Specifically, step 950 may be performed by the network traffic decision system 100 (e.g., the policy enforcement module 150). Details regarding taking a corresponding policy for the first traffic may be found in relation to the policy enforcement module 150.

Fig. 12 is a schematic application scenario diagram of a network service system according to some embodiments.

In some embodiments, the network service system 1000 may be applied to one or more of a traffic service system, a map service system, a navigation system, a transportation system, a financial service system, and the like. For example, the web service system 1000 may be applied to an online service platform providing internet services. For example, the network service system 1000 may be applied to an online transportation service platform providing transportation services. In some embodiments, the network service system 1000 may be applied to a network appointment service, such as a taxi call, a express call, a special call, a mini-bus call, a car pool, a bus service, a driver hiring and pick-up service, and the like. In some embodiments, the network service system 1000 may also be applied to designated driving services, express delivery, take-away, and the like. Specifically, the network service system 1000 may be an online service platform, which includes a server 1010, a network 1020, a terminal 1030, and a database 1040. The server 1010 may include a processing device 1012.

In some embodiments, server 1010 may be used to process information and/or data related to network services. The server 1010 may be a stand-alone server or a group of servers. The set of servers can be centralized or distributed (e.g., server 1010 can be a distributed system). In some embodiments, the server 1010 may be local or remote. The server(s) 1010 can access information and/or data stored in the terminal(s) 1030, database(s) 1040 via the network 1020. The server 1010 may be directly coupled to the terminal 1030 and the database 1040 to access information and/or material stored therein. The server 1010 may also receive traffic data that is accessed by the terminal 1030 via the network 1020. In some embodiments, the server 1010 may execute on a cloud platform. For example, the cloud platform may include one or any combination of a private cloud, a public cloud, a hybrid cloud, a community cloud, a decentralized cloud, an internal cloud, and the like. In some embodiments, the server 1010 may be an Internet Data Center (IDC).

In some embodiments, the server 1010 may include a processing device 1012. The processing device 1012 may process data and/or information related to network services. In some embodiments, the processing device 1012 may perform one or more of the functions described herein. For example, the processing device 1012 may perform one or more functions of the network traffic decision system 100. In some embodiments, the processing device 1012 may include one or more sub-processing devices (e.g., a single core processing device or a multi-core, multi-core processing device). By way of example only, the processing device 1012 may include a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), an Application Specific Instruction Processor (ASIP), a Graphics Processing Unit (GPU), a Physical Processing Unit (PPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), a programmable logic circuit (PLD), a controller, a micro-controller unit, a Reduced Instruction Set Computer (RISC), a microprocessor, or the like, or any combination thereof.

Network 1020 may facilitate the exchange of data and/or information. In some embodiments, one or more components in the network services system 1000 (e.g., the server 1010, the terminal 1030, the database 1040) may send data and/or information to other components in the network services system 1000 via the network 1020. In some embodiments, the network 1020 may be any type of wired or wireless network. For example, network 1020 may include a cable network, a wired network, a fiber optic network, a telecommunications network, an intranet, the internet, a Local Area Network (LAN), a Wide Area Network (WAN), a Wireless Local Area Network (WLAN), a Metropolitan Area Network (MAN), a Public Switched Telephone Network (PSTN), a bluetooth network, a ZigBee network, a Near Field Communication (NFC) network, a cellular network, the like, or any combination thereof. In some embodiments, network 1020 may include one or more network access points. For example, the network 1020 may include wired or wireless network access points, such as base stations and/or Internet switching points 1020-1, 1020-2, …, through which one or more components of the network services system 1000 may connect to the network 1020 to exchange data and/or information. In some embodiments, network 1020 may include one or more network devices. Network devices may include, but are not limited to, firewalls, routers, gateways, switches, hubs, bridges, reverse proxies, proxy servers, security devices, intrusion detection devices, load balancers, and the like, or similar devices. In some embodiments, traffic data sent by the terminal 1030 may be transmitted to the server 1010 through one or more network devices in the network 1020. In some embodiments, one or more network devices in network 1020 may perform certain operations on the traffic data (e.g., allow access, disallow access, flag, intercept, clear, etc.).

In some embodiments, the user of terminal 1030 may be any person or machine, etc. In some embodiments, the terminal 1030 may include one or any combination of a mobile device 1030-1, a tablet 1030-2, a laptop 1030-3, an automotive built-in device 1030-4, and the like. In some embodiments, mobile device 1030-1 may include a wearable device, a smart mobile device, a virtual reality device, an augmented reality device, and the like, or any combination thereof. In some embodiments, the wearable device may include a smart bracelet, smart footwear, smart glasses, smart helmet, smart watch, smart clothing, smart backpack, smart accessory, or the like, or any combination thereof. In some embodiments, the smart mobile device may comprise a smart phone, a Personal Digital Assistant (PDA), a gaming device, a navigation device, a POS device, and the like, or any combination thereof. In some embodiments, the metaverse device and/or the augmented reality device may include metaverse helmets, metaverse glasses, metaverse eyewear, augmented reality helmets, augmented reality glasses, augmented reality eyewear, and the like, or any combination thereof. In some embodiments, the in-vehicle device 1030-4 may include an in-vehicle navigator, an in-vehicle locator, a tachograph, and the like, or any combination thereof. In some embodiments, the terminal 1030 may include a location-enabled device to determine a location of the user and/or the terminal 1030. In some embodiments, the terminal 1030 may send traffic data to the server 1010.

Database 1040 may store data and/or instructions. In some embodiments, the database 1040 may store the profile obtained from the terminal 1030. In some embodiments, database 1040 may store information and/or instructions for execution or use by server 1010 to perform the example methods described herein. In some embodiments, database 1040 may store traffic classification models, feature extraction rules, feature decision rules, and the like. In some embodiments, database 1040 may also store flows to be determined, flows unknown, and the like. In some embodiments, database 1040 may include mass storage, removable storage, volatile read-write memory (e.g., random access memory RAM), read-only memory (ROM), the like, or any combination thereof. In some embodiments, database 1040 may be implemented on a cloud platform. For example, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a decentralized cloud, an internal cloud, and the like, or any combination thereof.

In some embodiments, the database 1040 may be connected to the network 1020 to communicate with one or more components of the network services system 1000 (e.g., the server 1010, the terminal 1030, etc.). One or more components of the network services system 1000 may access data or instructions stored in a database 1040 via the network 1020. For example, server 1010 may invoke traffic classification models, feature extraction rules, feature decision rules, etc. from database 1040. In some embodiments, the database 1040 may be directly connected to or in communication with one or more components (e.g., server 1010, terminal 1030) in the network services system 1000. In some embodiments, database 1040 may be part of server 1010.

The beneficial effects that may be brought by the embodiments of the present application include, but are not limited to: (1) the judgment rate and/or judgment accuracy of the encrypted flow can be improved; (2) mass flow can be efficiently judged; (3) different flow judgment methods can be adopted according to different service modes, so that the flow judgment has stronger applicability and higher efficiency; (4) the flow judgment system can automatically perform iterative updating, and the judgment rate and/or the judgment accuracy of the system are/is continuously improved; (5) the abnormal flow can be efficiently and accurately judged, so that the network safety is guaranteed. It is to be noted that different embodiments may produce different advantages, and in different embodiments, any one or combination of the above advantages may be produced, or any other advantages may be obtained.

Having thus described the basic concept, it will be apparent to those skilled in the art that the foregoing detailed disclosure is to be considered merely illustrative and not restrictive of the broad application. Various modifications, improvements and adaptations to the present application may occur to those skilled in the art, although not explicitly described herein. Such modifications, improvements and adaptations are proposed in the present application and thus fall within the spirit and scope of the exemplary embodiments of the present application.

Also, this application uses specific language to describe embodiments of the application. Reference throughout this specification to "one embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic described in connection with at least one embodiment of the present application is included in at least one embodiment of the present application. Therefore, it is emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, some features, structures, or characteristics of one or more embodiments of the present application may be combined as appropriate.

Moreover, those skilled in the art will appreciate that aspects of the present application may be illustrated and described in terms of several patentable species or situations, including any new and useful combination of processes, machines, manufacture, or materials, or any new and useful improvement thereon. Accordingly, various aspects of the present application may be embodied entirely in hardware, entirely in software (including firmware, resident software, micro-code, etc.) or in a combination of hardware and software. The above hardware or software may be referred to as "data block," module, "" engine, "" unit, "" component, "or" system. Furthermore, aspects of the present application may be represented as a computer product, including computer readable program code, embodied in one or more computer readable media.

The computer storage medium may comprise a propagated data signal with the computer program code embodied therewith, for example, on baseband or as part of a carrier wave. The propagated signal may take any of a variety of forms, including electromagnetic, optical, etc., or any suitable combination. A computer storage medium may be any computer-readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code located on a computer storage medium may be propagated over any suitable medium, including radio, cable, fiber optic cable, RF, or the like, or any combination of the preceding.

Computer program code required for the operation of various portions of the present application may be written in any one or more programming languages, including an object oriented programming language such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C + +, C #, VB.NET, Python, and the like, a conventional programming language such as C, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP, a dynamic programming language such as Python, Ruby, and Groovy, or other programming languages, and the like. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any network format, such as a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet), or in a cloud computing environment, or as a service, such as a software as a service (SaaS).

Additionally, the order in which elements and sequences of the processes described herein are processed, the use of alphanumeric characters, or the use of other designations, is not intended to limit the order of the processes and methods described herein, unless explicitly claimed. While various presently contemplated embodiments of the invention have been discussed in the foregoing disclosure by way of example, it is to be understood that such detail is solely for that purpose and that the appended claims are not limited to the disclosed embodiments, but, on the contrary, are intended to cover all modifications and equivalent arrangements that are within the spirit and scope of the embodiments herein. For example, although the system components described above may be implemented by hardware devices, they may also be implemented by software-only solutions, such as installing the described system on an existing server or mobile device.

Similarly, it should be noted that in the preceding description of embodiments of the application, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure aiding in the understanding of one or more of the embodiments. This method of disclosure, however, is not intended to require more features than are expressly recited in the claims. Indeed, the embodiments may be characterized as having less than all of the features of a single embodiment disclosed above.

Numerals describing the number of components, attributes, etc. are used in some embodiments, it being understood that such numerals used in the description of the embodiments are modified in some instances by the use of the modifier "about", "approximately" or "substantially". Unless otherwise indicated, "about", "approximately" or "substantially" indicates that the number allows a variation of ± 20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximations that may vary depending upon the desired properties of the individual embodiments. In some embodiments, the numerical parameter should take into account the specified significant digits and employ a general digit preserving approach. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the range are approximations, in the specific examples, such numerical values are set forth as precisely as possible within the scope of the application.

The entire contents of each patent, patent application publication, and other material cited in this application, such as articles, books, specifications, publications, documents, and the like, are hereby incorporated by reference into this application. Except where the application is filed in a manner inconsistent or contrary to the present disclosure, and except where the claim is filed in its broadest scope (whether present or later appended to the application) as well. It is noted that the descriptions, definitions and/or use of terms in this application shall control if they are inconsistent or contrary to the statements and/or uses of the present application in the material attached to this application.

Finally, it should be understood that the embodiments described herein are merely illustrative of the principles of the embodiments of the present application. Other variations are also possible within the scope of the present application. Thus, by way of example, and not limitation, alternative configurations of the embodiments of the present application can be viewed as being consistent with the teachings of the present application. Accordingly, the embodiments of the present application are not limited to only those embodiments explicitly described and depicted herein.

Claims

1. A method for determining network traffic, comprising:

acquiring a first flow rate;

judging the flow type of the first flow according to a flow judgment engine, and dividing the first flow into judged flow or flow to be measured based on the judgment result of the flow judgment engine;

obtaining a plurality of judged flows with the same flow type;

and updating the knowledge base corresponding to the flow judgment engine based on the characteristic data of at least part of the plurality of judged flows.

2. The method of claim 1, wherein at least one traffic type comprises more than two different network protocols.

3. The method of claim 2, wherein at least one of the two or more different network protocols included in the at least one traffic type requires encrypted transmission.

4. The network traffic decision method of claim 1, wherein the method further comprises:

and taking a corresponding strategy for the first flow based on the flow type of the first flow.

5. The network traffic decision method of claim 4, wherein the policy comprises at least one of: access allowed, access not allowed, reporting, alarms, special flags, and exception reporting.

6. The method for determining network traffic according to claim 1, wherein the updating the knowledge base corresponding to the traffic determination engine based on the feature data of at least some of the plurality of determined traffic comprises:

clustering the plurality of judged flows, and acquiring at least part of the plurality of judged flows based on the similarity between the clustered judged flows;

extracting feature data of the at least partial flow;

updating the knowledge base based on the characteristic data of the at least part of the traffic.

7. The network traffic decision method of claim 1, wherein the method further comprises:

acquiring a plurality of fixed flows to be detected, wherein the fixed flows to be detected comprise the first flows of which the flow types cannot be judged;

and judging whether the plurality of pieces of flow to be measured are of known flow types or not based on the clustering model.

8. The method according to claim 7, wherein the determining whether the plurality of predetermined flows to be measured are of known flow types based on the clustering model comprises:

clustering the plurality of to-be-measured flows by using an unsupervised clustering model to obtain at least one to-be-measured flow group;

and judging whether the undetermined flow group is a known flow type according to the feature extraction rule and the feature judgment rule.

9. The network traffic determination method of claim 8, wherein the determining whether the set of pending traffic is of a known traffic type based on the feature extraction rule and the feature determination rule comprises:

when all the flows in the undetermined flow group are judged to belong to a first known flow type according to the feature extraction rule and the feature judgment rule, judging the undetermined flow group to be a judged flow;

and when at least two flows in the flow group to be determined are judged to belong to different flow types according to the feature extraction rule and the feature judgment rule, judging that the flow group to be determined is unknown flow.

10. The method according to claim 7, wherein the determining whether the plurality of predetermined flows to be measured are of known flow types based on the clustering model comprises:

obtaining a plurality of flow types;

based on the multiple flow types, judging whether the multiple flows to be measured belong to the multiple flow types by using a supervised clustering model;

and judging the flow which does not belong to any flow type in the plurality of flows to be measured as unknown flow.

11. The network traffic decision method of claim 7, the method further comprising:

acquiring a feature extraction result and a flow classification result of the unknown flow, wherein the unknown flow is the flow to be measured which is judged not to be of the known flow type;

and updating the knowledge base according to the feature extraction result and the flow classification result of the unknown flow.

12. The network traffic decision method of claim 1, wherein the knowledge base comprises traffic classification models, and/or feature extraction rules and feature decision rules;

and the flow judgment engine judges the flow type of the first flow through a flow classification model in the knowledge base and/or a feature extraction rule and a feature judgment rule.

13. The network traffic decision method of claim 12, the method further comprising:

judging a business mode corresponding to the first flow according to a business mode judging model;

and calling a flow classification model corresponding to the flow judgment engine based on the business mode, and/or a feature extraction rule and a feature judgment rule.

14. A network traffic determination system, comprising: the device comprises an acquisition module, a flow type judgment module and an updating module;

the acquisition module is used for acquiring a first flow;

the flow type judging module is used for judging the flow type of the first flow according to a flow judging engine and dividing the first flow into judged flow or flow to be measured based on the judgment result of the flow judging engine;

the updating module is used for acquiring a plurality of judged flows with the same flow type;

the updating module is further configured to update the knowledge base corresponding to the traffic determination engine based on the feature data of at least part of the multiple determined flows.

15. The network traffic decision system of claim 14, wherein at least one traffic type comprises more than two different network protocols.

16. The network traffic determination system of claim 15, wherein at least one of the two or more different network protocols included in the at least one traffic type requires encrypted transmission.

17. The network traffic decision system of claim 14, wherein the method further comprises a policy enforcement module to:

18. The network traffic decision system of claim 17, wherein the policy comprises at least one of: access allowed, access not allowed, reporting, alarms, special flags, and exception reporting.

19. The network traffic decision system of claim 14, wherein the update module is further configured to:

extracting feature data of the at least partial flow;

20. The network traffic determination system of claim 14, wherein the system further comprises a to-be-measured traffic determination module configured to:

21. The network traffic determination system of claim 20, wherein the to-be-measured traffic determination module is further configured to:

22. The network traffic determination system of claim 21, wherein the to-be-measured traffic determination module is further configured to:

23. The network traffic determination system of claim 20, wherein the to-be-measured traffic determination module is further configured to:

obtaining a plurality of flow types;

24. The network traffic decision system of claim 20, wherein the system further comprises an update module to:

25. The network traffic decision system of claim 14, wherein the knowledge base comprises traffic classification models, and/or feature extraction rules and feature decision rules;

26. The network traffic decision system of claim 25, wherein the system further comprises a traffic pattern decision module and a knowledge base retrieval module:

the service mode judging module is used for judging a service mode corresponding to the first flow according to a service mode judging model;

the knowledge base calling module is used for calling a flow classification model corresponding to the flow judgment engine based on the service mode and/or a feature extraction rule and a feature judgment rule.

27. A network traffic determination apparatus comprising a processor, wherein the processor is configured to execute the network traffic determination method according to any one of claims 1 to 13.

28. A computer-readable storage medium storing computer instructions, wherein when the computer instructions in the storage medium are read by a computer, the computer executes the network traffic determination method according to any one of claims 1 to 13.