CN111464485A - Encrypted proxy flow detection method and device - Google Patents
Encrypted proxy flow detection method and device Download PDFInfo
- Publication number
- CN111464485A CN111464485A CN201910059354.6A CN201910059354A CN111464485A CN 111464485 A CN111464485 A CN 111464485A CN 201910059354 A CN201910059354 A CN 201910059354A CN 111464485 A CN111464485 A CN 111464485A
- Authority
- CN
- China
- Prior art keywords
- gait
- encryption
- traffic
- model
- proxy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/30—Network architectures or network communication protocols for network security for supporting lawful interception, monitoring or retaining of communications or communication related information
Abstract
The embodiment of the invention discloses a method and a device for detecting encrypted proxy flow, wherein the method comprises the following steps: extracting metadata from the real-time encryption proxy traffic through a traffic analysis engine; filtering the metadata according to a preset black data rule and/or a white data rule to acquire network traffic needing to be detected; extracting gait fingerprint characteristics of the network traffic needing to be detected; and detecting the network traffic by using the gait fingerprint characteristics and the created machine learning model and deep learning model, and judging the encryption agent type of the network traffic by adopting a preset judgment method. By the embodiment scheme, the encryption agent detection is effectively carried out and the situation that the encryption agent detection is bypassed by the anti-detection technology is avoided on the premise that the existing hardware equipment architecture is not changed and the encryption agent detection is deployed on various network security big data analysis platforms.
Description
Technical Field
The embodiment of the invention relates to the technical field of computer security and the field of machine learning, in particular to a method and a device for detecting encryption proxy flow.
Background
Traffic classification technology has wide application in the field of information security, but rapid development of internet services and wide application of encryption technology make it an open challenge. In recent years, various privacy-enhancing tools have adopted encryption technology, and at the same time, encryption traffic technology is also utilized by hackers to perform C & C transmission for the controlled machines in botnets. The current internet traffic is divided into seven common traffic types such as browse, Voip, Email, Chat, Streaming, File Transfer, P2P and the like according to types, and the encrypted traffic in the current internet traffic generally includes Tor traffic, Shadowclocks traffic, VPN traffic and the like, so that the encrypted traffic can be accurately detected, and the identification of botnet in network security is greatly facilitated.
The original design of VPN was an encrypted tunnel, but nowadays it has integrated multiple functions of access control, transmission management, encryption, routing, availability management, etc., and plays an important role in the global information security system. The tunneling technology is the basis of the vpn technology, and in the process of creating the tunnel, both the client and the server of the tunnel must use the same tunneling protocol. The tunneling technique can be divided into layer 2 and layer 3 tunneling protocols according to the division of the open systems interconnection reference model (OSI). The layer 2 tunneling protocol uses frames as data exchange units.
The Tor network is composed of three parts, namely an Onion Proxy (OP), a Directory Server (DS) and an Onion Router (OR). The OP mainly completes agent work of routing node selection, Circuit (Circuit) establishment, data packet sending and receiving and the like for the Tor user; the DS is mainly responsible for summarizing the running state of the Tor network and issuing a latest Tor routing node list to the OP; the OR is mainly composed of Tor network volunteers and is used for rerouting data packets, so that an anonymous effect is achieved. In the process of establishing connection of the Tor network, the OP randomly selects 3 available ORs as an entry Node (Guard Node, GN), a Relay Node (Relay Node, RN) and an Exit Node (Exit Node, EN) of the Tor network, obtains 3 session keys after key negotiation is sequentially performed by using Diffie-Hellman, encrypts messages in sequence by using the session keys, and finally sends a data packet encrypted for 3 times to the GN. GN, RN and EN use the shared session key to decrypt the data packet in turn, and send to the next hop, so that the data packet is finally sent to the destination station by EN in the form of plaintext. And returning the data packet, when passing the EX, the RN and the GN in sequence, encrypting the data packet by each node by using the shared session key, finally sending the data packet encrypted for 3 times to the OP by the GN, sequentially decrypting the data packet by the OP by using the 3 shared session keys, and finally transmitting the obtained plaintext to the Tor user.
The operation principle of Shadow socks is basically the same as that of other proxy tools, and data transmission is completed by using a specific transit server. After deployment is completed at the server end, the user needs to use client software to connect with the server according to the specified password, encryption mode and port. After successfully connecting to the server, the client builds a local Socks5 proxy (or VPN, transparent proxy) locally. When browsing the network, the network traffic will be distributed to local Socks5 proxies, the client encrypts it and sends it to the server, and the server returns the traffic to the client in the same encryption mode, so as to realize proxy internet surfing. The data packet encrypted by the Shadowclocks client is not obviously different from the normal TCP in the transmission process and is difficult to identify.
Currently, the research on the encryption agent detection technology is not extensive, most of the traditional rule detection is used, for example, VPN encryption agent traffic aiming at L2 TP and PPTP is detected by using a fixed destination port, and the rule detection engine is easily bypassed by the anti-detection technology.
Disclosure of Invention
The embodiment of the invention provides a method and a device for detecting encryption agent flow, which can be deployed on various network security big data analysis platforms on the premise of not changing the existing hardware equipment architecture, effectively perform encryption agent detection and avoid being bypassed by an anti-detection technology.
To achieve the object of the embodiment of the present invention, an embodiment of the present invention provides a method for detecting an encrypted proxy traffic, where the method may include:
extracting metadata from the real-time encryption proxy traffic through a traffic analysis engine;
filtering the metadata according to a preset black data rule and/or a white data rule to acquire network traffic needing to be detected;
extracting gait fingerprint characteristics of the network traffic needing to be detected;
detecting the network traffic by using the gait fingerprint characteristics and the created machine learning model and deep learning model, and judging the encryption agent type of the network traffic by adopting a preset judgment method;
the machine learning model is an optimized supervised machine learning submodel set obtained by carrying out multiple iterative training on supervised machine learning submodels of various encryption agent types through a gait fingerprint feature training set of network flow in a modeling stage of the machine learning model and carrying out multiple fusion on the optimized supervised machine learning submodels and various supervised machine learning models;
the deep learning model is an optimized deep learning submodel set obtained by carrying out multiple iterative training on the gait fingerprint feature training set of network flow in the modeling stage of the deep learning model about deep learning submodels of various encryption agent types.
In an exemplary embodiment of the present invention, the extracting gait fingerprint characteristics of the network traffic needing to be detected may include:
dividing the session of the encrypted proxy flow into different windows, respectively extracting the statistical characteristics of different session data packets in the windows, and presetting the statistical characteristics to be used as the gait fingerprint characteristics; wherein the statistical characteristics embody state information of the session and different agent behavior actions in the encrypted agent traffic.
In an exemplary embodiment of the present invention, the method for session segmentation may include: segmenting according to time and/or segmenting according to the number of packets;
the statistical features may include: statistics and ratios of time characteristics and/or uplink and downlink loads of the session data packets;
the preset treatment may include one or more of the following: and standardizing the obtained difference statistical characteristics, unifying the data precision of the statistical characteristics and carrying out normalized distribution processing.
In an exemplary embodiment of the present invention, before detecting the network traffic using the gait fingerprint features and the created machine learning model and deep learning model, the method may further include: establishing different types of encryption agent environments, extracting corresponding types of gait fingerprint characteristics based on the multidimensional metadata characteristics of different types of encryption agent flow under the encryption agent environments, and establishing the machine learning model and the deep learning model according to the gait fingerprint characteristics.
In an exemplary embodiment of the present invention, the building different types of encryption agent environments, extracting corresponding types of gait fingerprint features based on multidimensional metadata features of different types of encryption agent traffic in the encryption agent environments, and creating the machine learning model and the deep learning model according to the gait fingerprint features may include:
the method comprises the steps that a network collects different types of encryption agent flow, builds corresponding different types of encryption agent environments, captures different types of communication flow under the encryption agent environments, captures the encryption agent flow from the communication flow, adds type labels to the encryption agent flow, and takes the encryption agent flow with the type labels as a training data set;
extracting multi-dimensional metadata characteristics of different types of encryption agent flows in the training data set by using a flow analysis engine so as to respectively detect the different types of encryption agent flows according to the multi-dimensional metadata characteristics;
and according to a gait fingerprint feature engineering method, extracting gait fingerprint features from the multi-dimensional metadata features of each type of encrypted proxy flow, and respectively creating the machine learning model and the deep learning model through the gait fingerprint features.
In an exemplary embodiment of the present invention, the method may further include: extracting in units of sessions when extracting the multi-dimensional metadata features using the traffic analytics engine; wherein each session comprises the same tuple.
In an exemplary embodiment of the invention, the multi-dimensional metadata characteristics may include one or more of a packet capture time in a session, a packet load size, a packet direction, a packet time-to-live TT L flag, and an internet protocol/tag IP/flag of a packet;
the multi-element group may be a five-element group; the quintuple may include: source IP, source port, destination IP, destination port, and protocol.
In an exemplary embodiment of the present invention, creating the machine learning model by the gait fingerprint feature may include:
constructing a corresponding number of supervised machine learning submodels according to the types and numbers of different types of encrypted proxy flows in the training data set;
respectively training each corresponding supervised machine learning submodel by using a training set consisting of the gait fingerprint characteristics of various types of encrypted proxy flow in a cross validation mode, and acquiring an optimized model structure of each supervised machine learning submodel and a hyper-parameter of the optimized model structure;
selecting various types of supervised machine learning models to perform model fusion on various types of supervised machine learning submodels;
and adjusting the fusion model through multiple rounds of iteration to obtain an optimized supervised machine learning submodeset as the machine learning model.
In an exemplary embodiment of the present invention, creating the deep learning model by the gait fingerprint feature may include:
constructing a corresponding number of deep learning submodels according to the types of the encrypted proxy flows of different types in the training data set;
respectively adopting a stacked self-coding neural network to perform feature dimensionality reduction on gait fingerprint features corresponding to each type of deep learning submodel;
and respectively training a fully-connected neural network model and a long-term and short-term memory neural network model by using each type of gait fingerprint feature training set after dimension reduction to obtain an optimized deep learning sub-model set as the deep learning model.
In an exemplary embodiment of the present invention, the preset determination method may include one or more of the following: voting, averaging, and pyramid stacking.
An encryption proxy traffic detection device may include a processor and a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the instructions are executed by the processor, the encryption proxy traffic detection device implements the encryption proxy traffic detection method according to any one of the above items.
The embodiment of the invention comprises the following steps: extracting metadata from the real-time encryption proxy traffic through a traffic analysis engine; filtering the metadata according to a preset black data rule and/or a white data rule to acquire network traffic needing to be detected; extracting gait fingerprint characteristics of the network traffic needing to be detected; and detecting the network traffic by using the gait fingerprint characteristics and the created machine learning model and deep learning model, and judging the encryption agent type of the network traffic by adopting a preset judgment method. By the embodiment scheme, the method and the device can be deployed on various network security big data analysis platforms on the premise of not changing the existing hardware device architecture, effectively carry out encryption agent detection, and avoid being bypassed by a reverse detection technology.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the example serve to explain the principles of the invention and not to limit the invention.
FIG. 1 is a flow chart of a method for detecting encryption agent traffic according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method of creating a machine learning model and a deep learning model according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for creating a machine learning model from gait fingerprint characteristics according to an embodiment of the invention;
FIG. 4 is a flowchart of a method for creating the deep learning model from the gait fingerprint features according to an embodiment of the invention;
FIG. 5 is a schematic diagram of an encryption proxy traffic detection device according to an embodiment of the present invention;
fig. 6 is a schematic diagram of an encryption proxy traffic detection apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict.
The steps illustrated in the flow charts of the figures may be performed in a computer system such as a set of computer-executable instructions. Also, while a logical order is shown in the flow diagrams, in some cases, the steps shown or described may be performed in an order different than here.
To achieve the object of the embodiment of the present invention, an embodiment of the present invention provides a method for detecting an encryption proxy traffic, where as shown in fig. 1, the method may include S101 to S104:
s101, extracting metadata from the real-time encryption proxy flow through a flow analysis engine;
s102, filtering the metadata according to a preset black data rule and/or a preset white data rule to acquire network traffic needing to be detected;
s103, extracting gait fingerprint characteristics of the network traffic needing to be detected;
s104, detecting the network traffic by using the gait fingerprint characteristics and the created machine learning model and deep learning model, and judging the encryption agent type of the network traffic by adopting a preset judgment method;
the machine learning model is an optimized supervised machine learning submodel set obtained by carrying out multiple iterative training on supervised machine learning submodels of various encryption agent types through a gait fingerprint feature training set of network flow in a modeling stage of the machine learning model and carrying out multiple fusion on the optimized supervised machine learning submodels and various supervised machine learning models;
the deep learning model is an optimized deep learning submodel set obtained by carrying out multiple iterative training on the gait fingerprint feature training set of network flow in the modeling stage of the deep learning model about deep learning submodels of various encryption agent types.
In an exemplary embodiment of the invention, aiming at the difficulty of the proposed encryption proxy flow detection, the encryption proxy flow detection method based on the gait fingerprint is provided, suspicious encryption proxy flows can be filtered through a white data rule and classified into different encryption types, known encryption proxy flows are filtered through a black data rule, a multi-dimensional gait fingerprint feature matrix is generated by using a gait fingerprint feature engineering, and then the encryption proxy flow detection is carried out by using a supervised machine learning fusion model and a deep learning model.
In the exemplary embodiment of the invention, the whole network traffic captured by the network analysis engine is taken as a detection object, and accurate identification of the encryption proxy traffic is intensively studied on the premise of high similarity and large real-time traffic so as to construct a real-time network encryption proxy detection model.
In an exemplary embodiment of the present invention, the gait fingerprint refers to a setting of a session and a session window that are segmented by adopting a mode such as a maximum time period, a packet number, and the like in a feature engineering of encryption proxy traffic detection, and statistical information (or called statistical features) of a related behavior of the session for a period of time, including features such as a load, a duration, a ratio of uplink traffic to downlink traffic, and the like, is extracted, and the statistical features can accurately depict an abnormal behavior action of the encryption proxy traffic.
In an exemplary embodiment of the present invention, the extracting gait fingerprint characteristics of the network traffic needing to be detected may include:
dividing the session of the encrypted proxy flow into different windows, respectively extracting the statistical characteristics of different session data packets in the windows, and presetting the statistical characteristics to be used as the gait fingerprint characteristics; wherein the statistical characteristics embody state information of the session and different agent behavior actions in the encrypted agent traffic.
In an exemplary embodiment of the present invention, the method for session segmentation may include: segmenting according to time and/or segmenting according to the number of packets;
the statistical features may include: statistics and ratios of time characteristics and/or uplink and downlink loads of the session data packets;
the preset process may include any one or more of: and standardizing the obtained difference statistical characteristics, unifying the data precision of the statistical characteristics and carrying out normalized distribution processing.
In an exemplary embodiment of the present invention, the machine learning model and the deep learning model may be created before performing the encrypted proxy traffic detection on the real-time full-network traffic, and the creating process of the machine learning model and the deep learning model will be described in detail below.
In an exemplary embodiment of the present invention, before detecting the network traffic using the gait fingerprint features and the created machine learning model and deep learning model, the method may further include: establishing different types of encryption agent environments, extracting corresponding types of gait fingerprint characteristics based on the multidimensional metadata characteristics of different types of encryption agent flow under the encryption agent environments, and establishing the machine learning model and the deep learning model according to the gait fingerprint characteristics.
In an exemplary embodiment of the present invention, as shown in fig. 2, the building of different types of encryption agent environments, extracting gait fingerprint features of corresponding types based on multidimensional metadata features of different types of encryption agent traffic in the encryption agent environments, and creating the machine learning model and the deep learning model according to the gait fingerprint features may include S201 to S203:
s201, collecting different types of encryption agent flow by a network, building corresponding different types of encryption agent environments, capturing different types of communication flow under the encryption agent environments, capturing the encryption agent flow from the communication flow, adding type labels to the encryption agent flow, and taking the encryption agent flow added with the type labels as a training data set.
In an exemplary embodiment of the invention, a proxy environment may be built based on network collection, using common proxy tools, capturing proxy encrypted traffic and non-proxy traffic, and adding type labels as training data sets.
In an exemplary embodiment of the present invention, the method may specifically include: A1. the network collects and builds common different types of encryption agent environments (for example, building the encryption agent environment can comprise building a client and a server of an agent tunnel, and performing different types of communication on the server through the client), and capturing encryption agent flow; A2. different types of communication traffic (which may include encryption proxy traffic and normal traffic) are grabbed, which may include, but are not limited to, instant messaging, video, music, mail, P2P, Web, and the like.
S202, extracting multi-dimensional metadata characteristics of different types of encryption agent flows in the training data set by using a flow analysis engine, and respectively detecting the different types of encryption agent flows according to the multi-dimensional metadata characteristics.
In an exemplary embodiment of the present invention, a traffic parsing engine may be used to extract multidimensional metadata features of encryption proxy traffic, and during detection, traffic of different encryption types is classified and input into encryption proxy models of corresponding types according to the multidimensional metadata features for detection.
In an exemplary embodiment of the invention, the multi-dimensional metadata features may include any one or more of packet capture time, packet payload size, packet direction, packet time to live TT L flag, and internet protocol/tag IP/flag bits of the packet in the session.
In the exemplary embodiment of the present invention, the method specifically includes the following steps of b1, parsing the whole network traffic, obtaining the multidimensional metadata characteristics (including but not limited to the metadata information such as packet capture time, packet load size, packet direction, TT L flag bit of the packet, IP/flag bit of the packet, etc.) and b2, detecting that the traffic of different encryption types is classified and input to different encryption agent models according to the metadata characteristics including but not limited to application layer protocol and destination port.
In an exemplary embodiment of the present invention, the method may further include: using the flow solution
When extracting the multi-dimensional metadata features, the analysis engine can extract the multi-dimensional metadata features by taking a conversation as a unit; wherein each session may comprise the same tuple.
In an exemplary embodiment of the present invention, the tuple may be a five tuple; the quintuple may include: source IP, source port, destination IP, destination port, and protocol.
S203, extracting gait fingerprint characteristics from the multi-dimensional metadata characteristics of each type of encrypted proxy flow according to a gait fingerprint characteristic engineering method, and respectively creating the machine learning model and the deep learning model according to the gait fingerprint characteristics.
In an exemplary embodiment of the present invention, multidimensional gait fingerprint feature vectors may be extracted for various types of encrypted traffic metadata (corresponding to various types of multidimensional metadata features) according to a gait fingerprint feature engineering method.
In an exemplary embodiment of the present invention, the method for extracting the gait fingerprint feature from the multidimensional metadata feature of each type of encrypted proxy traffic according to the gait fingerprint feature engineering method may include, as the aforementioned method for extracting the gait fingerprint feature from the network traffic to be detected:
dividing the session of the encrypted proxy flow into different windows, respectively extracting the statistical characteristics of different session data packets in the windows, and presetting the statistical characteristics to be used as the gait fingerprint characteristics; wherein the statistical characteristics embody state information of the session and different agent behavior actions in the encrypted agent traffic.
In an exemplary embodiment of the present invention, extracting the gait fingerprint feature may specifically refer to: the encrypted flow session is divided into different windows, statistical characteristics of different data packets in the windows are extracted (for example, gait fingerprint characteristics of metadata are extracted by a gait fingerprint characteristic engineering method, relevant characteristics of packet time characteristics, uplink and downlink loads and the like in the session are extracted, and statistics and ratio characteristics are extracted) so as to reflect state information of the session and different agent behavior actions in the encrypted flow. Wherein the session segmentation method may include but is not limited to: the statistical characteristics of the data packets in the window may include, but are not limited to, calculating statistics and ratios of the time characteristics, uplink and downlink loads, etc. of the data packets, time slicing or packet number slicing.
In the exemplary embodiment of the present invention, the session is segmented, and the session length can be unified by using the conditions of maximum session time, packet number, etc.; and standardizing the obtained difference characteristics, unifying the accuracy of the characteristic data and carrying out normalized distribution processing.
In an exemplary embodiment of the present invention, the session of the encrypted proxy traffic is divided into different windows, statistical characteristics of different session data packets in the windows are respectively extracted, and the statistical characteristics after being subjected to the preset processing may include C1-C4:
c1: carrying out data preprocessing on various encrypted proxy traffic training metadata and real-time traffic metadata; the data preprocessing may include default value filling, outlier deletion, slicing long sessions; the session can be segmented according to the measurement modes such as the number of packets and the session time period, and the overfitting performance of the session size reduction model is unified.
C2, extracting gait fingerprint features of preprocessed metadata (i.e. the metadata with multidimensional metadata features) includes 1) extracting features such as maximum value, minimum value, mean value and variance from the duration of each stream and the interval time, load and number of up and down packets in each sub-session after segmenting the session, 2) calculating the byte number feature per second transmitted in each sub-session and calculating the number of data packets per second transmitted, 3) extracting features such as entropy value from TT L and IP/flag of each sub-session, 4) extracting gait fingerprint features including determining to be Active if the difference between the arrival time of the current packet and the arrival time of the previous packet is less than the set number of seconds when constructing session state features, and recording the time difference as an Active value.
C3: after the gait fingerprint features obtained through the steps are subjected to normal distribution processing, the processing method can comprise one or more of the following steps: logarithmic, square root, reciprocal and square root sine transformation.
C4: after the gait fingerprint features obtained through the above steps are processed by standardization, the standardization may include: normalization and normalization.
In an exemplary embodiment of the present invention, after extracting the less-than-sufficient fingerprint features of the metadata, a supervised machine learning model (various sub models can be trained respectively by using various encryption agent traffic training sets, the structure and parameters of the model are tuned, the model is used to fuse the generalization detection effect, and the optimal sub model set is saved) and a detection model based on a deep neural network, that is, a deep learning model (various sub models can be trained respectively by using various encryption agent traffic training sets, the parameters and the structural parameters of the tuning training are optimized, and the optimal deep learning sub model set is saved) can be constructed according to the gait fingerprint features.
In the exemplary embodiment of the invention, when the supervised machine learning model is constructed, the corresponding number of supervised machine learning models can be constructed according to the number of various encryption agent types, various types of supervised machine learning models are selected for model fusion, a fusion scheme and parameter configuration of each model are set, each sub-model is respectively trained by using the gait fingerprint characteristic training data set obtained in the previous step, and the optimal sub-model set is stored.
In an exemplary embodiment of the present invention, specifically, as shown in fig. 3, creating the machine learning model by the gait fingerprint features may include S301-S304:
s301, constructing a corresponding number of supervised machine learning submodels according to the types and numbers of different types of encrypted proxy flows in the training data set.
In an exemplary embodiment of the present invention, a corresponding number of supervised machine learning submodels may be constructed according to the number of classes of the training data set, and the model structure parameters may be set.
S302, respectively training each corresponding supervised machine learning submodel by using a training set consisting of the gait fingerprint features of various types of encrypted proxy flow in a cross validation mode, and acquiring an optimized model structure of each supervised machine learning submodel and a hyper-parameter of the optimized model structure.
And S303, selecting various types of supervised machine learning models to perform model fusion on various types of supervised machine learning submodels.
S304, adjusting the fusion model through multiple rounds of iteration to obtain an optimized supervised machine learning sub-model set as the machine learning model.
In an exemplary embodiment of the present invention, multiple types of supervised machine learning cryptographic agent detection models may be selected for model fusion, where the set of models (i.e., the multiple types of supervised machine learning cryptographic agent detection models) may include, but are not limited to: the SVM-based support vector machine comprises an SVM support vector machine and a derivative model, a rule tree set model, a linear classification model and a probability statistics model thereof. Fusion methods may include, but are not limited to: the voting method, the average value method, the Stacking method and the like can train each sub-fusion model by utilizing the gait fingerprint characteristics of each type of encryption agent flow obtained in the previous steps, and iterate to obtain and store the optimal sub-fusion model set through a cross validation mode.
In an exemplary embodiment of the present invention, the voting method model fusion and the Stacking method model fusion described above can adjust the sensitivity and the false alarm rate of the encrypted proxy traffic detection through threshold setting. The detection results output by different types of encryption agent detection models (namely various supervised machine learning submodels) are corresponding model probabilities, wherein the abnormal threshold of the application layer encryption agent flow detection model is higher, the abnormal threshold of the Socket encryption agent flow detection model is higher, the prediction output exceeds the threshold and is predicted to be encryption agent flow, and the average value method means that the average value of all the fusion submodels is taken for judgment.
In the exemplary embodiment of the invention, a corresponding number of deep learning submodels can be constructed according to the number of various encryption agent types, the network structure and the training parameters of each deep learning submodel are set, the gait fingerprint feature training data set obtained in the previous step is used for carrying out iterative training on each deep learning submodel, the optimal submodel structure is stored, and the final deep learning model is obtained.
In an exemplary embodiment of the present invention, specifically, as shown in fig. 4, creating the deep learning model by the gait fingerprint features may include S401 to S403:
s401, constructing a corresponding number of deep learning submodels according to the types and the numbers of different types of encrypted proxy flows in the training data set.
S402, respectively adopting a stacked self-coding neural network to perform feature dimension reduction on gait fingerprint features corresponding to each type of deep learning submodel.
In an exemplary embodiment of the present invention, each type of deep learning sub-model may adopt a self-coding neural network to reduce dimensions (the gait fingerprint features obtained in the above steps may be reduced dimensions and improved generalization performance through the self-coding neural network), and adopt a fully-connected neural network and a long-short term memory neural network model as a detection main model, and set network structure parameters and training weights.
In an exemplary embodiment of the present invention, the deep learning sub-model may use a stacked self-coding neural network and other derived neural networks to perform dimension reduction processing on the gait fingerprint feature data, and specifically may include: 1) compressing the gait fingerprint characteristics by using a self-coding neural network, then recovering the data with minimum loss, and optimizing the model structure with minimum loss through iterative training; 2) the stacked self-coding neural network can select a stacking mode of a plurality of self-encoders, and the encoders can select nonlinear activation functions to improve the characteristic extraction effect; 3) and finally, the deep learning submodel outputs the probability of each dimension characteristic by using a Softmax function through a plurality of hidden layer characteristic extractions.
And S403, respectively training the fully-connected neural network model and the long-term and short-term memory neural network model by using each type of gait fingerprint feature training set after dimensionality reduction to obtain an optimized deep learning sub-model set as the deep learning model.
In an exemplary embodiment of the invention, the deep learning model may use a fully-connected neural network, other derived neural network models and a long-short term memory neural network model as a main detection model, respectively perform fingerprint training detection on the gait fingerprint feature data, adjust training weights and model structures, iteratively optimize neural network parameters, and output and store an optimal deep learning sub-model.
In an exemplary embodiment of the invention, in the master detection model of a fully-connected neural network, the hidden layer activation function may select a nonlinear Relu to better fit the data. To prevent overfitting, the layers can be adjusted by using Dropout parameters, the output layer can output probability results of the flow of each type of encryption agent by using linear Softmax, the detection probabilities of each type of encryption agent are output, the sum of the probabilities is 1, and random values are extracted from a truncated normal distribution to serve as initialization values of all parameters.
In an exemplary embodiment of the invention, in a main detection model of a long-short term memory neural network, a hidden layer activation function can select nonlinear Relu to better fit data, a cross entropy loss function can be selected to be used for updating model parameters through a random gradient descent method, an output layer can output probability results of various types of encryption agent flows by using linear Softmax, the detection probabilities of various types of encryption agents are output, the sum of the probabilities is 1, a random value is extracted from a truncated normal distribution and used as an initialization value of all parameters, and an optimal deep learning sub-model set is saved through iterative training.
In an exemplary embodiment of the invention, after the mechanical learning model and the deep learning model are obtained through the scheme, the whole network real-time flow filtered by the black and white rule can be detected according to the machine learning model and the deep learning model obtained through training.
In an exemplary embodiment of the present invention, the specific detecting step may include: 1) inputting the real-time flow after metadata extraction and black and white rule filtration into each machine learning sub-model set for detection, wherein the machine learning sub-model set is the optimal machine learning sub-model set obtained by the training in the previous step, and outputting a detection result; 2) inputting the real-time flow after metadata extraction and black and white rule filtration into each full-connection neural network submodel and a long-short term memory neural network submodel set (namely a deep learning submodel set) for detection, wherein the deep learning submodel set is from the optimal deep learning submodel set obtained by the training in the previous step, and outputting a detection result; 3) and comprehensively utilizing the machine learning model, the full-connection neural network model and the long-short term memory neural network model to judge the encryption agent type. There are voting, averaging and pyramid stacking.
In an exemplary embodiment of the present invention, the preset determination method (i.e., the method of comprehensive determination) may include one or more of the following: voting, averaging, and pyramid stacking.
In an exemplary embodiment of the present invention, 1) the voting method refers to: respectively taking threshold values of the machine learning fusion model, the full-connection neural network model and the long-term and short-term memory neural network model, and judging the encryption agent type through threshold value judgment voting; 2) the averaging method refers to: the probabilities output by the machine learning fusion model, the full-connection neural network model and the long-short term memory neural network model are averaged firstly and then compared with the threshold value to judge the encryption agent type; 3) the pyramid stacking method refers to: and randomly selecting two models from the three models for detection, and entering the third model for detection if the two models are judged to be consistent, and finally judging the encryption agent type.
In an exemplary embodiment of the present invention, a specific implementation embodiment from modeling for creating a machine learning model and a deep learning model to full-network real-time encryption proxy traffic detection may be given as follows, and the specific implementation embodiment may include the following steps:
selecting a virtual private network VPN, an onion router Tor and a shadow shuttle ShadowSocks encryption proxy flow test, and extracting five-dimensional metadata of packet _ ts (packet capture time), packet _ size, packet _ dir (packet direction), packet _ ip _ ttl (TT L Flag bit of a packet) and packet _ ip _ flags (ip Flag bit of the packet) in each session by a network flow analysis engine;
the network flow analysis engine extracts features by taking a session as a unit, namely a packet with the same five-tuple of a source IP, a source port, a destination IP, a destination port and a protocol is taken as a session.
The network flow analysis engine distributes the extracted five-dimensional metadata to each flow encryption agent detection model in a pub/sub mode through redis during offline training for detection, and distributes data through a storm distributed message distribution system during online real-time detection.
In the case of a real-time traffic deployment model:
and (3) distributing different types of flow to each module (a corresponding quantity of models established according to the quantity of the flow types) by using a storm distributed flow distribution tool, and replacing a redis tool in a real-time environment.
And secondly, filtering network flow needing to be detected through metadata, wherein the black data rule can comprise filtering related black data through a network node IP disclosed by Tor and marking the black data as Tor flow, filtering and marking the existing specific VPN proxy through a specific target port number and an application layer protocol type of a VPN encryption proxy, classifying white data rules and flow, inputting session metadata of which the application layer is a secure socket layer protocol SS L and the port is not very used and the load is not zero into VPN and Tor encryption proxy flow detection flows, and inputting session metadata of which the transmission layer is a transmission control protocol TCP and the port is not very used and the load is not zero into a Shadowclocks encryption proxy detection flow.
The third step: and performing gait fingerprint feature engineering method processing on the metadata of the two streams in the second step to obtain the multidimensional detection features with discrimination on the target, which specifically comprises the following steps:
and (3) segmenting the long conversation, wherein the next conversation is obtained after more than fifteen seconds, and the features (gait fingerprint features) enter the next sample, so that the features with discrimination are favorably formed.
After the streams are segmented, maximum value, minimum value, mean value and variance four-dimensional characteristics are respectively extracted from the duration of each stream;
in each flow, extracting four-dimensional characteristics of a maximum value, a minimum value, a mean value and a variance from the interval time of two forward packets;
in each flow, extracting four-dimensional characteristics of a maximum value, a minimum value, a mean value and a variance respectively from the interval time of two reverse packets;
in each flow, extracting four-dimensional characteristics of a maximum value, a minimum value, a mean value and a variance from the time interval of two packets;
in each flow, calculating the byte number per second characteristic of transmission; in each stream, the number of packets per second transmitted is characterized.
When the session state characteristics are constructed, if the difference between the arrival time of the current packet and the arrival time of the previous packet is less than 5 seconds, the packet is judged to be Active, and the Active value is recorded as the time difference. And if the difference between the arrival time of the current packet and the arrival time of the previous packet is more than 5 seconds, determining that the current packet is idle, and recording the time difference as an idle value. The maximum, minimum, mean, variance (four features in total) of the state duration are calculated separately.
The fourth step: three encryption agent flow detection models can be trained, an encryption agent environment is built, and training data are constructed.
The method can respectively use a point-to-point tunneling protocol PPTP of a data link layer, a second layer tunneling protocol L2 TP, a network layer protocol IPsec (Internet connection protocol), an application layer protocol SSTP (secure socket tunneling protocol) and an openVPN protocol agent tool which is most widely applied to construct a related agent environment, 7 types of traffic (including but not limited to the 7 types of traffic) such as browse, Voip, Email, Chat, Streaming, File Transfer, P2P (peer loan platform) can be captured by a virtual machine connection agent, 24-dimensional training data is generated by a network traffic analysis engine and a feature extraction module, the target is 7 types of traffic, and the traffic types can respectively correspond to 0-6.
A Tor encryption agent environment is set up, seven types of traffic such as Browsing, Voip, Email, Chat, Streaming, File Transfer, P2P and the like are captured through a virtual machine connection agent, 24-dimensional training data are generated through a network traffic analysis engine and a feature extraction module, targets are 7 traffic types, and the traffic types can correspond to 0-6 respectively.
Establishing a ShadowSocks encryption agent environment, capturing seven types of traffic such as Browsing, Voip, Email, Chat, Streaming, File Transfer, P2P and the like through a virtual machine connection agent, generating 24-dimensional training data through a network traffic analysis engine and a feature extraction module, and setting the target as 7 traffic types which can respectively correspond to 0-6.
The fifth step: constructing an anomaly detection model of machine learning (namely the machine learning model) which can be divided into three models, namely VPN, Tor and ShadowSocks;
the VPN detection model parameters may be:
the number of n _ estimators single-tree models can be 200, oob _ score selects to adopt out-of-bag samples to evaluate the model, max _ depth is selected to be 5, and the rest parameters are default values.
The SVM (support vector machine) support vector machine model may select a default parameter configuration for skearn.
The bayesian probabilistic model may select a default parameter configuration for the model in skearn.
The model fusion can select the stacking technology, the cross validation can select 3 folds, 3 models are trained respectively, and finally the generalization model is detected and the logistic regression model is selected.
Wherein the Tor detection model parameters may be:
the number of n _ estimators single-tree models is 350, oob _ score selects to adopt out-of-bag samples to evaluate the model, max _ depth is selected to be 4, the minimum number of leaf nodes samples min _ samples _ leaf is 5, and the rest parameters are default values.
Wherein the parameters of the ShadowSocks detection model can be as follows:
the number of n _ estimators single-tree models is 100, oob _ score selects to adopt out-of-bag samples to evaluate the model, the minimum number of leaf nodes, min _ samples _ leaf, is 10, and the rest parameters are default values.
The model fusion uses a stacking model fusion mode, model SVM support vector machine default configuration and xgboost of different principles are selected as model fusion types, and the number of the model types is 3.
And a sixth step: the deep learning model is constructed, the feature dimension reduction can be realized by using a stacked self-coding neural network, the training data is optimized, the training weight is adjusted, and the model structure is optimized.
The learning rate of the self-coding neural network training can be set to 0.01, the training period can be set to 10, each batch of training samples can be 256, and Relu (linear rectification function) can be selected as the activation function of each layer in order to enable the detection speed to meet the real-time requirement.
The self-coding neural network can select a mode of stacking two self-encoders, the hidden layer of the first self-encoder is 16 neurons, and the matrix parameters (24 × 16) needing to be trained are 384; the hidden layer of the second self-encoder comprises 8 neurons, the matrix parameters (16 × 8) needing to be trained are 128, and random values are extracted from the truncated normal distribution to serve as initialization values of all parameters, so that the training is prevented from being falsely dead.
And (3) constructing a deep learning model, realizing encrypted flow characteristic detection by using a full-connection neural network, adjusting training weight and batch, and iteratively optimizing a neural network structure and neural network parameters.
The learning rate of the fully-connected neural network can be set to 0.01, the training period can be 5 times, each batch of training samples can be 256, and the activation function of each layer can select Relu.
The input data of the fully-connected neural network is from an autocoder, the output of the autocoder is 8-dimensional, the first layer of hidden layer of the fully-connected neural network is 32 neurons, and the matrix parameters to be trained are (8 × 32) ═ 256; to prevent overfitting, Dropout was used after the first layer with the parameter set to 0.5; the second hidden layer has 64 neurons, 1024 matrix parameters (32 × 0.5 × 64) which need to be trained, finally Softmax is used for outputting probability results of various types of encrypted proxy flows, and random values are extracted from the truncated normal distribution to serve as initialization values of all parameters.
The input data of the long-short term memory neural network is from an encoder, the input characteristic is 8-dimensional, one row of input samples is a time sequence flow, therefore, the set time sequence length is 1, the first hidden layer is 28 neurons, the long-short term memory neural network selects a single-layer 256-dimensional characteristic parameter, the second hidden layer is 64 neurons, the activation function uses Sigmoid, the probability result of each type of encryption proxy flow is finally output by using Softmax, and a random value is extracted in normal truncation distribution to serve as an initialization value of all parameters.
The seventh step: setting reasonable threshold values for the supervised machine learning model of the three encrypted proxy flow detection, and adjusting the sensitivity and the false alarm rate of the encrypted proxy flow detection, specifically comprising:
the three detection models output the probability value of the detection session corresponding to the encrypted proxy flow, and through the actual data flow test, the threshold value of the VPN model is set to be 70%, the threshold value of the Tor model is set to be 56%, the threshold value of the ShadowSocks model is set to be 63%, and the encrypted proxy flow is predicted when the prediction probability exceeds the threshold value.
Eighth step: the method comprises the steps of comprehensively judging the encryption agent type by using a voting method, outputting the detected encryption agent type through threshold judgment of respective models after the machine learning model and a neural network model (deep learning model) detect the encryption agent flow simultaneously, selecting the encryption agent type with more votes by using the voting method, generating a protocol type, a source IP, a destination IP, a source port, a destination port and load information of an encryption flow session, issuing a message queue and sending the message queue to a unified management platform log record through a pub/sub mode of redis in offline training, and issuing the message queue and an alarm log through a storm distributed data distribution system when online deployment is performed.
An embodiment of the present invention further provides an encryption proxy traffic detection device 1 based on gait fingerprints, as shown in fig. 5, which may include: the system comprises an encryption agent session acquisition module 11, a gait fingerprint feature engineering module 12, a training module 13, a white rule and black rule filtering module 14 and a detection module 15.
The encryption agent session acquisition module 11: the method can be used for obtaining various common encryption agent type flows and various flow types based on various encryption agents. 1) Collecting encrypted agent flow through a network and marking respective labels; 2) various encryption proxy communication environments are simulated by building various encryption proxy servers and clients, and various encryption proxy flows are obtained through various communication flows (such as instant messaging type, video type, webpage Web access type, P2P type, mail type and the like).
Gait fingerprint feature engineering module 12: can be used to generate training data needed for machine learning and deep learning: 1) uniformly segmenting the session, wherein the segmentation method comprises the packet number and the time period, and constructing gait fingerprint characteristics through state information such as packet load, packet time, packet number and the like; 2) carrying out normal distribution processing on the obtained gait fingerprint characteristics; 3) and carrying out standardization or normalization processing on the obtained gait fingerprint characteristics, and unifying the magnitude of characteristic data.
The training module 13 may be used to train machine learning models and deep learning models. Namely, a machine learning fusion learning model, a deep learning long-short term memory neural network model and a full-connection neural network model are constructed, and network structure parameters and training weights are set. And training the machine learning model and the deep learning model by using the marked training data, and performing cross validation to store the optimal model.
White and black rule filtering module 14: can be used to reduce the false alarm rate of the model. 1) The method comprises the steps of preferentially detecting known encryption agent flow by utilizing an existing encryption agent rule; 2) and filtering the non-encrypted flow by utilizing a white data rule, marking different encryption types and providing data for later different types of model detection.
The detection module 15 may be configured to detect an unknown encrypted traffic session by using a trained machine learning model and a trained deep learning model, where detection data is simply filtered according to black and white data rules, and then sent to the detection module to output a detection result.
An embodiment of the present invention further provides an encryption proxy traffic detection apparatus 2, as shown in fig. 6, which may include a processor 21 and a computer-readable storage medium 22, where the computer-readable storage medium 22 stores instructions, and when the instructions are executed by the processor 21, the encryption proxy traffic detection method described in any one of the above is implemented.
The embodiment of the invention can comprise the following steps: extracting metadata from the real-time encryption proxy traffic through a traffic analysis engine; filtering the metadata according to a preset black data rule and/or a white data rule to acquire network traffic needing to be detected; extracting gait fingerprint characteristics of the network traffic needing to be detected; and detecting the network traffic by using the gait fingerprint characteristics and the created machine learning model and deep learning model, and judging the encryption agent type of the network traffic by adopting a preset judgment method. By the embodiment scheme, the method and the device can be deployed on various network security big data analysis platforms on the premise of not changing the existing hardware device architecture, effectively carry out encryption agent detection, and avoid being bypassed by a reverse detection technology.
The embodiment of the invention at least comprises the following advantages:
1. the machine learning model (supervised machine learning fusion model) and the deep learning model are used for comprehensive detection, a reliable encryption agent flow detection model is trained through a large amount of training data, and the accuracy rate is higher than that of the traditional detection model.
2. The model output adopts a mode that the prediction probability output is controlled by a threshold value, so that the detection flexibility is enhanced, and the detection of the encrypted proxy flow detection is conveniently controlled.
3. By extracting the multi-dimensional gait fingerprint characteristics, the characteristic difference between the encrypted proxy flow session and the normal flow session can be fully extracted, and the model effect is improved.
4. Because only a small amount of characteristic extraction is performed preliminarily by using the flow analysis engine, the performance of the engine is superior to that of a common flow detection engine, the packet loss rate is reduced, and the detection effect is improved.
5. The metadata is used for filtering and classifying the encrypted traffic, namely, the interference removing effect is achieved, and the targeted model training can be performed according to different types of encrypted traffic.
6. The method is suitable for being deployed on various network security big data analysis platforms on the premise of not changing the existing hardware equipment architecture.
It will be understood by those of ordinary skill in the art that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.
Claims (10)
1. A method for detecting encryption proxy traffic, the method comprising:
extracting metadata from the real-time encryption proxy traffic through a traffic analysis engine;
filtering the metadata according to a preset black data rule and/or a white data rule to acquire network traffic needing to be detected;
extracting gait fingerprint characteristics of the network traffic needing to be detected;
and detecting the network traffic by using the gait fingerprint characteristics and the created machine learning model and deep learning model, and judging the encryption agent type of the network traffic by adopting a preset judgment method.
2. The encryption proxy traffic detection method according to claim 1, wherein the extracting gait fingerprint features of the network traffic to be detected comprises:
dividing the session of the encrypted proxy flow into different windows, respectively extracting the statistical characteristics of different session data packets in the windows, and presetting the statistical characteristics to be used as the gait fingerprint characteristics; wherein the statistical characteristics embody state information of the session and different agent behavior actions in the encrypted agent traffic.
3. The encryption agent traffic detection method according to claim 2, wherein the session slicing method comprises: segmenting according to time and/or segmenting according to the number of packets;
the statistical features include: statistics and ratios of time characteristics and/or uplink and downlink loads of the session data packets;
the preset treatment comprises any one or more of the following steps: and standardizing the obtained difference statistical characteristics, unifying the data precision of the statistical characteristics and carrying out normalized distribution processing.
4. The encryption proxy traffic detection method according to claim 1, wherein before detecting the network traffic using the gait fingerprint features and the created machine learning model and deep learning model, the method further comprises:
establishing different types of encryption agent environments, extracting corresponding types of gait fingerprint characteristics based on the multidimensional metadata characteristics of different types of encryption agent flow under the encryption agent environments, and establishing the machine learning model and the deep learning model according to the gait fingerprint characteristics.
5. The encrypted proxy traffic detection method according to claim 4, wherein the building of different types of encrypted proxy environments, extracting gait fingerprint features of corresponding types based on multidimensional metadata features of different types of encrypted proxy traffic in the encrypted proxy environments, and creating the machine learning model and the deep learning model according to the gait fingerprint features comprises:
the method comprises the steps that a network collects different types of encryption agent flow, builds corresponding different types of encryption agent environments, captures different types of communication flow under the encryption agent environments, captures the encryption agent flow from the communication flow, adds type labels to the encryption agent flow, and takes the encryption agent flow with the type labels as a training data set;
extracting multi-dimensional metadata characteristics of different types of encryption agent flows in the training data set by using a flow analysis engine so as to respectively detect the different types of encryption agent flows according to the multi-dimensional metadata characteristics;
and according to a gait fingerprint feature engineering method, extracting gait fingerprint features from the multi-dimensional metadata features of each type of encrypted proxy flow, and respectively creating the machine learning model and the deep learning model through the gait fingerprint features.
6. The encryption proxy traffic detection method of claim 5, further comprising: extracting in units of sessions when extracting the multi-dimensional metadata features using the traffic analytics engine; wherein each session comprises the same tuple.
7. The encryption proxy traffic detection method according to claim 6, wherein the multi-dimensional metadata features include any one or more of packet capture time, packet load size, packet direction, packet lifetime TT L flag bit and packet IP/flag bit in a session;
the multi-tuple is a five-tuple; the quintuple comprises: source IP, source port, destination IP, destination port, and protocol.
8. The encryption proxy traffic detection method of claim 5, wherein creating the machine learning model from the gait fingerprint features comprises:
constructing a corresponding number of supervised machine learning submodels according to the types and numbers of different types of encrypted proxy flows in the training data set;
respectively training each corresponding supervised machine learning submodel by using a training set consisting of the gait fingerprint characteristics of various types of encrypted proxy flow in a cross validation mode, and acquiring an optimized model structure of each supervised machine learning submodel and a hyper-parameter of the optimized model structure;
selecting various types of supervised machine learning models to perform model fusion on various types of supervised machine learning submodels;
and adjusting the fusion model through multiple rounds of iteration to obtain an optimized supervised machine learning submodeset as the machine learning model.
9. The encryption proxy traffic detection method according to claim 5, wherein creating the deep learning model from the gait fingerprint features comprises:
constructing a corresponding number of deep learning submodels according to the types of the encrypted proxy flows of different types in the training data set;
respectively adopting a stacked self-coding neural network to perform feature dimensionality reduction on gait fingerprint features corresponding to each type of deep learning submodel;
and respectively training a fully-connected neural network model and a long-term and short-term memory neural network model by using each type of gait fingerprint feature training set after dimension reduction to obtain an optimized deep learning sub-model set as the deep learning model.
10. A cryptographic proxy traffic detection apparatus comprising a processor and a computer-readable storage medium having instructions stored therein, wherein the instructions, when executed by the processor, implement a cryptographic proxy traffic detection method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910059354.6A CN111464485A (en) | 2019-01-22 | 2019-01-22 | Encrypted proxy flow detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910059354.6A CN111464485A (en) | 2019-01-22 | 2019-01-22 | Encrypted proxy flow detection method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111464485A true CN111464485A (en) | 2020-07-28 |
Family
ID=71679903
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910059354.6A Pending CN111464485A (en) | 2019-01-22 | 2019-01-22 | Encrypted proxy flow detection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111464485A (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111881105A (en) * | 2020-07-30 | 2020-11-03 | 北京智能工场科技有限公司 | Service data labeling model and model training method thereof |
| CN112134898A (en) * | 2020-09-28 | 2020-12-25 | 北京嘀嘀无限科技发展有限公司 | Network flow judgment method and system |
| CN112671757A (en) * | 2020-12-22 | 2021-04-16 | 无锡江南计算技术研究所 | Encrypted flow protocol identification method and device based on automatic machine learning |
| CN112769633A (en) * | 2020-12-07 | 2021-05-07 | 深信服科技股份有限公司 | Proxy traffic detection method and device, electronic equipment and readable storage medium |
| CN113014454A (en) * | 2021-03-05 | 2021-06-22 | 中电积至(海南)信息技术有限公司 | SSLTLS protocol-based user agent identification and quantity detection method |
| CN113079069A (en) * | 2021-06-04 | 2021-07-06 | 南京邮电大学 | Mixed granularity training and classifying method for large-scale encrypted network traffic |
| CN113381973A (en) * | 2021-04-26 | 2021-09-10 | 深圳市任子行科技开发有限公司 | Method, system and computer readable storage medium for identifying SSR flow |
| CN113794601A (en) * | 2021-08-17 | 2021-12-14 | 中移（杭州）信息技术有限公司 | Network traffic processing method, device and computer readable storage medium |
| CN114205095A (en) * | 2020-08-27 | 2022-03-18 | 极客信安(北京)科技有限公司 | Encrypted malicious traffic detection method and device |
| CN114422174A (en) * | 2021-12-09 | 2022-04-29 | 绿盟科技集团股份有限公司 | Network flow filtering method, device, medium, product and equipment |
| CN114422242A (en) * | 2022-01-19 | 2022-04-29 | 闪捷信息科技有限公司 | Abnormal traffic identification method, client and server |
| CN114584371A (en) * | 2022-03-04 | 2022-06-03 | 桀安信息安全技术（上海）有限公司 | Method, system and device for detecting encrypted flow behavior |
| CN115174170A (en) * | 2022-06-23 | 2022-10-11 | 东北电力大学 | VPN encrypted flow identification method based on ensemble learning |
| CN116346452A (en) * | 2023-03-17 | 2023-06-27 | 中国电子产业工程有限公司 | Multi-feature fusion malicious encryption traffic identification method and device based on stacking |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104009836A (en) * | 2014-05-26 | 2014-08-27 | 南京泰锐斯通信科技有限公司 | Encrypted data detection method and system |
| US20180018590A1 (en) * | 2016-07-18 | 2018-01-18 | NantOmics, Inc. | Distributed Machine Learning Systems, Apparatus, and Methods |
| CN108363714A (en) * | 2017-12-21 | 2018-08-03 | 北京至信普林科技有限公司 | A kind of method and system for the ensemble machine learning for facilitating data analyst to use |
| CN108768986A (en) * | 2018-05-17 | 2018-11-06 | 中国科学院信息工程研究所 | A kind of encryption traffic classification method and server, computer readable storage medium |
| CN108833360A (en) * | 2018-05-23 | 2018-11-16 | 四川大学 | A kind of malice encryption flow identification technology based on machine learning |
| CN108985361A (en) * | 2018-07-02 | 2018-12-11 | 北京金睛云华科技有限公司 | A kind of malicious traffic stream detection implementation method and device based on deep learning |
-
2019
- 2019-01-22 CN CN201910059354.6A patent/CN111464485A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104009836A (en) * | 2014-05-26 | 2014-08-27 | 南京泰锐斯通信科技有限公司 | Encrypted data detection method and system |
| US20180018590A1 (en) * | 2016-07-18 | 2018-01-18 | NantOmics, Inc. | Distributed Machine Learning Systems, Apparatus, and Methods |
| CN108363714A (en) * | 2017-12-21 | 2018-08-03 | 北京至信普林科技有限公司 | A kind of method and system for the ensemble machine learning for facilitating data analyst to use |
| CN108768986A (en) * | 2018-05-17 | 2018-11-06 | 中国科学院信息工程研究所 | A kind of encryption traffic classification method and server, computer readable storage medium |
| CN108833360A (en) * | 2018-05-23 | 2018-11-16 | 四川大学 | A kind of malice encryption flow identification technology based on machine learning |
| CN108985361A (en) * | 2018-07-02 | 2018-12-11 | 北京金睛云华科技有限公司 | A kind of malicious traffic stream detection implementation method and device based on deep learning |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111881105B (en) * | 2020-07-30 | 2024-02-09 | 北京智能工场科技有限公司 | Labeling model of business data and model training method thereof |
| CN111881105A (en) * | 2020-07-30 | 2020-11-03 | 北京智能工场科技有限公司 | Service data labeling model and model training method thereof |
| CN114205095A (en) * | 2020-08-27 | 2022-03-18 | 极客信安(北京)科技有限公司 | Encrypted malicious traffic detection method and device |
| CN114205095B (en) * | 2020-08-27 | 2023-08-18 | 极客信安(北京)科技有限公司 | Method and device for detecting encrypted malicious traffic |
| CN112134898A (en) * | 2020-09-28 | 2020-12-25 | 北京嘀嘀无限科技发展有限公司 | Network flow judgment method and system |
| CN112769633A (en) * | 2020-12-07 | 2021-05-07 | 深信服科技股份有限公司 | Proxy traffic detection method and device, electronic equipment and readable storage medium |
| CN112671757A (en) * | 2020-12-22 | 2021-04-16 | 无锡江南计算技术研究所 | Encrypted flow protocol identification method and device based on automatic machine learning |
| CN112671757B (en) * | 2020-12-22 | 2023-10-31 | 无锡江南计算技术研究所 | Encryption flow protocol identification method and device based on automatic machine learning |
| CN113014454A (en) * | 2021-03-05 | 2021-06-22 | 中电积至(海南)信息技术有限公司 | SSLTLS protocol-based user agent identification and quantity detection method |
| CN113014454B (en) * | 2021-03-05 | 2022-06-14 | 中电积至(海南)信息技术有限公司 | SSL and TLS protocol-based user agent identification and quantity detection method |
| CN113381973A (en) * | 2021-04-26 | 2021-09-10 | 深圳市任子行科技开发有限公司 | Method, system and computer readable storage medium for identifying SSR flow |
| CN113079069B (en) * | 2021-06-04 | 2021-09-17 | 南京邮电大学 | Mixed granularity training and classifying method for large-scale encrypted network traffic |
| CN113079069A (en) * | 2021-06-04 | 2021-07-06 | 南京邮电大学 | Mixed granularity training and classifying method for large-scale encrypted network traffic |
| CN113794601A (en) * | 2021-08-17 | 2021-12-14 | 中移（杭州）信息技术有限公司 | Network traffic processing method, device and computer readable storage medium |
| CN113794601B (en) * | 2021-08-17 | 2024-03-22 | 中移（杭州）信息技术有限公司 | Network traffic processing method, device and computer readable storage medium |
| CN114422174A (en) * | 2021-12-09 | 2022-04-29 | 绿盟科技集团股份有限公司 | Network flow filtering method, device, medium, product and equipment |
| CN114422174B (en) * | 2021-12-09 | 2023-07-25 | 绿盟科技集团股份有限公司 | Network traffic filtering method, device, medium and equipment |
| CN114422242A (en) * | 2022-01-19 | 2022-04-29 | 闪捷信息科技有限公司 | Abnormal traffic identification method, client and server |
| CN114584371A (en) * | 2022-03-04 | 2022-06-03 | 桀安信息安全技术（上海）有限公司 | Method, system and device for detecting encrypted flow behavior |
| CN115174170B (en) * | 2022-06-23 | 2023-05-09 | 东北电力大学 | VPN encryption flow identification method based on ensemble learning |
| CN115174170A (en) * | 2022-06-23 | 2022-10-11 | 东北电力大学 | VPN encrypted flow identification method based on ensemble learning |
| CN116346452A (en) * | 2023-03-17 | 2023-06-27 | 中国电子产业工程有限公司 | Multi-feature fusion malicious encryption traffic identification method and device based on stacking |
| CN116346452B (en) * | 2023-03-17 | 2023-12-01 | 中国电子产业工程有限公司 | Multi-feature fusion malicious encryption traffic identification method and device based on stacking |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111464485A (en) | Encrypted proxy flow detection method and device | |
| Alshammari et al. | Machine learning based encrypted traffic classification: Identifying ssh and skype | |
| Dong et al. | CETAnalytics: Comprehensive effective traffic information analytics for encrypted traffic classification | |
| Zhang et al. | Network traffic classification using correlation information | |
| WO2022083353A1 (en) | Abnormal network data detection method and apparatus, computer device, and storage medium | |
| WO2018054342A1 (en) | Method and system for classifying network data stream | |
| Labayen et al. | Online classification of user activities using machine learning on network traffic | |
| Samarakoon et al. | 5g-nidd: A comprehensive network intrusion detection dataset generated over 5g wireless network | |
| CN111147394B (en) | Multi-stage classification detection method for remote desktop protocol traffic behavior | |
| CN113206860B (en) | DRDoS attack detection method based on machine learning and feature selection | |
| US8903749B2 (en) | Method of identifying a protocol giving rise to a data flow | |
| CN112822189A (en) | Traffic identification method and device | |
| CN113452676B (en) | Detector distribution method and Internet of things detection system | |
| Soleimani et al. | Real-time identification of three Tor pluggable transports using machine learning techniques | |
| Janabi et al. | Convolutional neural network based algorithm for early warning proactive system security in software defined networks | |
| Lu et al. | A heuristic-based co-clustering algorithm for the internet traffic classification | |
| Muliukha et al. | Analysis and classification of encrypted network traffic using machine learning | |
| Liu et al. | Semi-supervised encrypted traffic classification using composite features set | |
| Alkhatib et al. | Unsupervised network intrusion detection system for AVTP in automotive Ethernet networks | |
| Tong et al. | BFSN: a novel method of encrypted traffic classification based on bidirectional flow sequence network | |
| Haghighat et al. | SAWANT: smart window based anomaly detection using netflow traffic | |
| Alizadeh et al. | Traffic classification for managing applications’ networking profiles | |
| Deebalakshmi et al. | A survey of classification algorithms for network traffic | |
| Khosroshahi et al. | Detection of sources being used in ddos attacks | |
| Dipon et al. | Detecting network intrusion through anomalous packet identification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |