CN114611619A - Abnormal flow detection method, system and storage medium - Google Patents

Abnormal flow detection method, system and storage medium Download PDF

Info

Publication number
CN114611619A
CN114611619A CN202210265222.0A CN202210265222A CN114611619A CN 114611619 A CN114611619 A CN 114611619A CN 202210265222 A CN202210265222 A CN 202210265222A CN 114611619 A CN114611619 A CN 114611619A
Authority
CN
China
Prior art keywords
data set
sample data
flow
detection
flow sample
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210265222.0A
Other languages
Chinese (zh)
Inventor
赵经宇
温嘉骏
王金梅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Guotengchuangxing Technology Co ltd
Beijing Guoteng Innovation Technology Co ltd
Original Assignee
Nanjing Guotengchuangxing Technology Co ltd
Beijing Guoteng Innovation Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Guotengchuangxing Technology Co ltd, Beijing Guoteng Innovation Technology Co ltd filed Critical Nanjing Guotengchuangxing Technology Co ltd
Priority to CN202210265222.0A priority Critical patent/CN114611619A/en
Publication of CN114611619A publication Critical patent/CN114611619A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06F18/2155—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H04L63/1425—Traffic logging, e.g. anomaly detection

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Hardware Design (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Traffic Control Systems (AREA)

Abstract

The invention relates to an abnormal flow detection method, a system and a storage medium, wherein the method comprises the following steps: preprocessing an original flow sample data set to obtain and perform data dimension reduction on a first flow sample data set to obtain a second flow sample data set; performing iterative training on the second flow sample data set by adopting a preset detection model to obtain a target detection model; and generating a detection result of the flow data to be detected according to the flow data to be detected and the target abnormal flow detection model. According to the method, the loss function of the detection model is minimized in an iterative training mode, and constraint training is performed through combination of data dimension reduction and density estimation, so that the problem of inconsistent training targets is solved, and the accuracy of abnormal flow detection is improved.

Description

Abnormal flow detection method, system and storage medium
Technical Field
The present application relates to the field of network security technologies, and in particular, to a method, a system, and a storage medium for detecting abnormal traffic.
Background
The rapid development of the internet not only brings great convenience to network users, but also brings many security threats. Abnormal traffic generated by network malicious attacks such as port scanning, SQL injection attack, Distributed Denial of Service (DDoS) attack, APT, and the like causes the network to be abnormal or even collapsed, and finally normal Service cannot be provided to the outside. The network flow is a carrier for network information interaction and transmission, and accurate detection of the network flow becomes an important link of network security. The abnormal network flow detection can detect whether the network has damage or attack behaviors or not as early as possible by analyzing the network flow, provides decision basis for network security management, and plays an important guarantee role in ensuring the overall normal operation of the network and maintaining the security of the whole network space.
With the development of artificial intelligence and big data technology, machine learning plays an increasingly important role in the field of network abnormal traffic detection. From the concept that data has labels or not, a machine learning algorithm can be divided into unsupervised learning and supervised learning, and in consideration of the fact that most network traffic in actual life is label-free data, the unsupervised learning has a better application prospect. The traditional unsupervised anomaly detection methods can be divided into three types: based on a reconstruction method, One-Class classification and cluster analysis, the method still has some problems when being applied to the field of network abnormal flow detection, such as incomplete network flow high-dimensional data feature learning, inconsistent two-stage training optimization targets and the like.
Disclosure of Invention
In order to solve the technical problem, the invention provides an abnormal flow detection method, an abnormal flow detection system and a storage medium.
The technical scheme of the abnormal flow detection method of the invention is as follows:
s1, preprocessing the original flow sample data set to obtain and perform data dimension reduction on the first flow sample data set to obtain a second flow sample data set;
s2, performing iterative training on the second flow sample data set by adopting a preset detection model to obtain a target detection model;
and S3, generating a detection result of the flow data to be detected according to the flow data to be detected and the target abnormal flow detection model.
The abnormal flow detection method has the following beneficial effects:
according to the method, data dimensionality reduction is carried out on an original flow sample data set, iterative training is carried out by adopting a preset detection model, and the flow data to be detected is detected according to a target detection model to obtain a detection result. The method minimizes the loss function of the detection model according to an iterative training mode, performs constraint training by combining data dimension reduction and density estimation, eliminates the problem of inconsistent training targets, and improves the accuracy of abnormal flow detection.
On the basis of the scheme, the abnormal flow detection method can be further improved as follows.
Further, the original traffic sample data set is: the first traffic sample data set comprises an original digital feature data set and an original symbol feature data set, and the first traffic sample data set comprises: a first set of digital signature data and a second set of digital signature data;
the preprocessing the original flow sample data set to obtain a first flow sample data set specifically includes:
converting each original symbol characteristic data in the original symbol characteristic data set by adopting one-hot coding to obtain the first digital characteristic data set;
and processing each original digital characteristic data in the original digital characteristic data set by adopting a maximum and minimum normalization method to obtain the second digital characteristic data set.
Further, performing data dimension reduction on the first traffic sample data set to obtain a second traffic sample data set, which specifically includes:
performing low-dimensional feature extraction on each flow sample data in the first flow sample data set by adopting a convolution self-encoder to obtain a compressed encoding data set;
performing data reconstruction on each flow sample data in the first flow sample data set to obtain a reconstructed flow sample data set, and obtaining a reconstruction error data set according to the reconstructed flow sample data set and the first flow sample data set;
and connecting the compressed encoding data set and the reconstruction error data set through an open to obtain the second flow sample data set.
Further, the reconstructing the error data set includes: relative Euclidean distance error and cosine similarity error;
obtaining a reconstruction error data set according to the reconstruction flow sample data set and the first flow sample data set, specifically including:
obtaining the relative Euclidean distance error according to the reconstructed flow sample data set, the first flow sample data set and a first preset calculation formula;
obtaining the cosine similarity error according to the reconstructed flow sample data set, the first flow sample data set and a second preset calculation formula;
connecting the relative Euclidean distance error with the cosine similarity error to obtain the reconstruction error data set;
wherein the first preset calculation formula is as follows:
Figure BDA0003551456640000031
the second preset calculation formula is as follows:
Figure BDA0003551456640000032
for the purpose of the relative euclidean distance error,
Figure BDA0003551456640000033
x is the first traffic sample data set, x is the cosine similarity error,
Figure BDA0003551456640000034
for said reconstructed traffic sample data set, xiFor the ith traffic sample data in the first traffic sample data set,
Figure BDA0003551456640000035
for the ith reconstructed flow sample data in the reconstructed flow sample data set, and
Figure BDA0003551456640000036
and xiAnd N is the total amount of the flow sample data in the first flow sample data set.
Further, the preset detection model comprises: presetting a multilayer neural network model and a Gaussian mixture model;
the S2 specifically includes:
s21, calculating a plurality of sub-distribution probabilities of the second flow sample data set in the preset Gaussian mixture model by adopting the preset multilayer neural network;
s22, sequentially utilizing the second flow sample data set and each sub-distribution probability to obtain and obtain a mixed probability according to each sub-distribution
Figure BDA0003551456640000041
Mean value
Figure BDA0003551456640000042
Sum covariance
Figure BDA0003551456640000043
Generating a first Gaussian mixture model;
and S23, adopting the first Gaussian mixture model as the preset Gaussian mixture model, and returning to execute S21 until the parameters in the first Gaussian mixture model converge, and obtaining the target detection model according to the first Gaussian mixture model.
Further, the preset detection model further comprises: presetting a loss function; when the parameters in the first gaussian mixture model reach convergence, obtaining the target detection model according to the first gaussian mixture model, specifically including: when the preset loss function in the first Gaussian mixture model reaches convergence, obtaining the target detection model according to the first Gaussian mixture model;
wherein the predetermined loss function is:
Figure BDA0003551456640000044
Figure BDA0003551456640000045
J(θe，θd，θm) For said predetermined loss function, θeFor the encoder parameters of said convolutional auto-encoder, θdFor the decoder parameters of said convolutional autocoder, θmFor said predetermined Gaussian mixture model parameter, λ1Is a first parameter, λ, of said predetermined detection model2Is a second element parameter of the preset detection model,
Figure BDA0003551456640000046
wherein, E (z)i) The energy value function corresponding to the preset Gaussian mixture model,
Figure BDA0003551456640000047
Figure BDA0003551456640000048
E(zi) The energy value of the ith flow sample data in the second flow sample data set is obtained;
Figure BDA0003551456640000049
Figure BDA00035514566400000410
p＝MLN(z；θm)，
Figure BDA00035514566400000411
z is the second traffic sample data set.
Further, the S3 specifically includes:
s31, obtaining a detection energy value according to the flow data to be detected and an energy value function in the target detection model;
s32, judging whether the detection energy value is larger than a discrimination threshold value;
s33, if yes, judging that the detection result of the flow data to be detected is abnormal; if not, judging that the detection result of the flow data to be detected is normal.
Further, the value of the discrimination threshold is as follows: detecting an Mth sample energy value in the target quantity detection model;
wherein M is (1-c) N, N is the total number of traffic sample data in the first traffic sample data set, and c is the abnormal proportion of the traffic sample data in the first traffic sample data set.
The technical scheme of the abnormal flow detection system is as follows:
the method comprises the following steps: the device comprises a first processing module, a second processing module and a detection module;
the first processing module is configured to: preprocessing an original flow sample data set to obtain and perform data dimension reduction on a first flow sample data set to obtain a second flow sample data set;
the second processing module is configured to: performing iterative training on the second flow sample data set by adopting a preset detection model to obtain a target detection model;
the detection module is used for: and generating a detection result of the flow data to be detected according to the flow data to be detected and the target abnormal flow detection model.
The abnormal flow detection system has the following beneficial effects:
the system performs data dimension reduction on an original flow sample data set and adopts a preset detection model to perform iterative training to obtain and detect flow data to be detected according to a target detection model to obtain a detection result. The system minimizes the loss function of the detection model according to an iterative training mode, performs constraint training by combining data dimension reduction and density estimation, eliminates the problem of inconsistent training targets, and improves the accuracy of abnormal flow detection.
The technical scheme of the storage medium of the invention is as follows:
the storage medium has instructions stored therein, which when read by a computer, cause the computer to perform the steps of an abnormal flow detection method according to the present invention.
The technical scheme of the electronic equipment is as follows:
comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor, when executing the computer program, causes the computer to perform the steps of a method for abnormal flow detection according to the invention.
Drawings
Fig. 1 is a schematic flow chart of an abnormal traffic detection method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a target detection model training process in an abnormal traffic detection method according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an abnormal flow rate detection system according to an embodiment of the present invention.
Detailed Description
As shown in fig. 1, an abnormal traffic detection method according to an embodiment of the present invention includes the following steps:
and S1, preprocessing the original flow sample data set to obtain and perform data dimension reduction on the first flow sample data set to obtain a second flow sample data set.
The original traffic sample data set in this embodiment is: KDD CUP 99 data set, the KDD CUP 99 data set source is a simulated US air force local area network, and the KDD CUP 99 data set comprises network connection data in 9 weeks, and the network connection data comprises: different attack means, different network flow and different user types can better simulate the real network environment. Each network connection is classified into normal or abnormal two types.
It should be noted that the original traffic sample data may also be network traffic data in any other field, and is not limited herein.
The preprocessing is to convert all the flow sample data of different data types in the original flow sample data set into the flow sample data of digital characteristics, and the flow sample data in the first flow sample data set are all the flow sample data of digital characteristics.
The data dimension reduction adopts a convolution self-encoder to complete dimension reduction operation, and is a conventional technical means.
Specifically, the dimension reduction operation is performed on the flow sample data of any digital feature, and the flow sample data is converted into one-dimensional data until a plurality of one-dimensional data are obtained, so that a second flow sample data set is formed.
The flow sample data of each digital feature of the first flow sample data set is multidimensional data, and the second flow sample data set is one-dimensional data.
And S2, performing iterative training on the second flow sample data set by adopting a preset detection model to obtain a target detection model.
The process of performing iterative training on the second flow sample data set by using the preset detection model is a density estimation process, and parameter values in the preset detection model can be obtained when parameters of the preset detection model tend to converge through the iterative training, so that the target detection model is obtained.
And S3, generating a detection result of the flow data to be detected according to the flow data to be detected and the target abnormal flow detection model.
The flow data to be detected may be flow data in the original flow sample data set, or flow data in other flow data sets, and no limitation is set here.
Preferably, the original traffic sample data set is: the first traffic sample data set comprises an original digital feature data set and an original symbol feature data set, and the first traffic sample data set comprises: a first digital signature data set and a second digital signature data set.
The original flow sample data set mainly comprises: a digital signature dataset and a symbolic signature dataset; the types of the flow data in the digital characteristic data set are digital characteristic types, and the types of the flow data in the symbolic characteristic data set are symbolic characteristic types.
Specifically, the first digital feature data set is a data set (symbolic feature → digital feature) obtained by quantizing the original symbolic feature data set, and the second digital feature data set is a data set (digital feature → digital feature) obtained by quantizing the original digital feature data set.
The preprocessing the original flow sample data set to obtain a first flow sample data set specifically includes:
and converting each original symbol characteristic data in the original symbol characteristic data set by adopting one-hot coding to obtain the first digital characteristic data set.
Wherein, the one-hot coding is a conventional technical means and is not described too much.
Specifically, each original symbol feature data in the original symbol feature data set is converted by one-hot coding to obtain first digital feature data corresponding to each original symbol feature data, and a first digital feature data set is obtained according to each first digital feature data.
And processing each original digital characteristic data in the original digital characteristic data set by adopting a maximum and minimum normalization method to obtain the second digital characteristic data set.
The maximum and minimum normalization method is a conventional technical means and is used for eliminating dimension influence among all data indexes, solving comparability among the data indexes, carrying out standardization processing on data in a data preprocessing stage, and keeping all characteristic indexes in the same order of magnitude after the processing.
Specifically, each original digital feature data in the original digital feature data set is normalized by a maximum and minimum normalization method to obtain second digital feature data corresponding to each original digital feature data, and a second digital feature data set is obtained according to each second digital feature data.
Wherein, the maximum and minimum normalization method is used, and the formula is as follows:
Figure BDA0003551456640000081
in the formula xnormSecond digital characteristic data corresponding to any original digital characteristic data, wherein x is any original digital characteristic data, xmaxIs the maximum value, x, of the original digital feature data setminIs the minimum value of the original digital feature data set.
Preferably, the performing data dimension reduction on the first traffic sample data set to obtain a second traffic sample data set specifically includes:
and performing low-dimensional feature extraction on each flow sample data in the first flow sample data set by adopting a convolution self-encoder to obtain a compressed encoding data set.
The convolutional self-encoder is a one-dimensional convolutional self-encoder, and the one-dimensional convolutional self-encoder is adopted for dimension reduction considering that the first stream sample data comprises multi-dimensional data (two-dimensional, three-dimensional and multi-dimensional data). The convolutional autoencoder consists of an encoder, a decoder and a hidden layer, wherein the number of hidden layer neurons is much smaller than the number of input layer neurons. The core idea of the one-dimensional convolution self-encoder is to change a fully-connected network in the self-encoder into a one-dimensional convolution neural network.
Specifically, 16 convolution kernels with kernel _ size of 1 × 3 are adopted in the first layer of convolution layer of the encoder, Same is selected as a padding item, and tanh is selected as an activation function; the second layer of Pooling layer adopts a maximum Pooling (Max Pooling) mode, and the step length is 11; the third layer of convolution layer adopts 8 convolution kernels with kernel _ size of 1 × 3, and other parameters are in the same layer 1; the fourth layer pooling layer parameter is the same as layer 2. By performing low-dimensional feature extraction on each flow sample data in the first flow sample data set by the encoder, a low-dimensional representation zc (compressed encoded data set) of the first flow sample data set can be obtained, and the decoder operates as the inverse operation of the encoder.
And performing data reconstruction on each flow sample data in the first flow sample data set to obtain a reconstructed flow sample data set, and obtaining a reconstructed error data set according to the reconstructed flow sample data set and the first flow sample data set.
The process of data reconstruction is the prior art, and is not described herein in detail.
Specifically, data reconstruction is performed on each flow sample data in the first flow sample data set to obtain a reconstructed flow sample data set after reconstruction, and error calculation is performed on the reconstructed flow sample data set and the first flow sample data set before reconstruction to obtain zr (reconstruction error data set).
Wherein the reconstruction error data set comprises two dimensions, respectively: relative euclidean distance error and cosine similarity error.
And connecting the compressed encoding data set and the reconstruction error data set through an open to obtain the second flow sample data set.
Wherein, the second sample data set z output from the encoder by convolution is zc and zr is connected by an apend, specifically, z ═ z [ -z ═ z [c，zr]。
Preferably, the reconstruction error data set includes: relative Euclidean distance error and cosine similarity error;
obtaining a reconstruction error data set according to the reconstruction flow sample data set and the first flow sample data set, specifically including:
and obtaining the relative Euclidean distance error according to the reconstructed flow sample data set, the first flow sample data set and a first preset calculation formula.
And obtaining the cosine similarity error according to the reconstructed flow sample data set, the first flow sample data set and a second preset calculation formula.
And connecting the relative Euclidean distance error with the cosine similarity error to obtain the reconstruction error data set.
Wherein the first preset calculation formula is as follows:
Figure BDA0003551456640000101
the second preset calculation formula is as follows:
Figure BDA0003551456640000102
for the purpose of the relative euclidean distance error,
Figure BDA0003551456640000103
x is the first traffic sample data set, x is the cosine similarity error,
Figure BDA0003551456640000104
for the reconstructed traffic sample dataset, xiFor the ith traffic sample data in the first traffic sample data set,
Figure BDA0003551456640000105
for the ith reconstructed flow sample data in the reconstructed flow sample data set, and
Figure BDA0003551456640000106
and xiAnd N is the total amount of the flow sample data in the first flow sample data set.
Preferably, the preset detection model comprises: presetting a multilayer neural network model and a Gaussian mixture model;
the S2 specifically includes:
and S21, calculating a plurality of sub-distribution probabilities of the second flow sample data set in the preset Gaussian mixture model by adopting the preset multilayer neural network.
And calculating the probability that each second flow sample data in the input second flow sample data set belongs to each sub-distribution in the Gaussian mixture model by using a preset multilayer neural network. The preset Gaussian mixture model contains k sub-distributions, a second flow sample data set is input and a k-dimensional vector is output after passing through the multilayer neural network, and the k-dimensional vector represents the probability of z in k sub-distributions of the Gaussian mixture model. Multi-layer neural network usedThe formula is shown below: p ═ MLN (z; θ)m)；
Figure BDA0003551456640000107
S22, sequentially utilizing the second flow sample data set and each sub-distribution probability to obtain and obtain a mixed probability according to each sub-distribution
Figure BDA0003551456640000108
Mean value
Figure BDA0003551456640000109
Sum covariance
Figure BDA00035514566400001010
A first gaussian mixture model is generated.
And further calculating model parameters by using each second flow sample data in the second flow sample data set and the probability of each second flow sample data in the k sub-distributions, wherein the model parameters comprise the mixed probability of the k sub-distributions
Figure BDA00035514566400001011
Mean value
Figure BDA0003551456640000111
Sum covariance
Figure BDA0003551456640000112
The parameter formula is as follows:
Figure BDA0003551456640000113
Figure BDA0003551456640000114
representing the probability of the ith second traffic sample data zi under the kth sub-distribution.
And S23, adopting the first Gaussian mixture model as the preset Gaussian mixture model, and returning to execute S21 until the parameters in the first Gaussian mixture model converge, and obtaining the target detection model according to the first Gaussian mixture model.
Preferably, the preset detection model further comprises: presetting a loss function; when the parameters in the first gaussian mixture model reach convergence, obtaining the target detection model according to the first gaussian mixture model, specifically including: and when the preset loss function in the first Gaussian mixture model reaches convergence, obtaining the target detection model according to the first Gaussian mixture model.
Wherein the predetermined loss function is:
Figure BDA0003551456640000115
Figure BDA0003551456640000116
J(θe，θd，θm) For said predetermined loss function, θeFor the encoder parameters of said convolutional auto-encoder, θdFor decoder parameters of said convolutional autocoder, θmFor said predetermined Gaussian mixture model parameter, λ1Is a first parameter, λ, of said predetermined detection model2Is a second element parameter of the preset detection model,
Figure BDA0003551456640000117
wherein, E (z)i) An energy value function corresponding to the preset Gaussian mixture model,
Figure BDA0003551456640000118
Figure BDA0003551456640000119
E(zi) The energy value of the ith flow sample data in the second flow sample data set is obtained;
Figure BDA00035514566400001110
Figure BDA00035514566400001111
p＝MLN(z；θm)，
Figure BDA00035514566400001112
z is the second traffic sample data set.
Specifically, the training process of the target detection model is as shown in fig. 2, and all parameters of the probability density function of the gaussian mixture model are preset
Figure BDA00035514566400001113
All are obtained, and the log-likelihood function formula is also clear. The larger the sample log-likelihood function value is, the more the distribution of the Gaussian mixture model can be met, namely, the more normal the sample is. However, the model cannot directly use the log-likelihood function as part of the loss function, because the training of the model aims to minimize the loss function, which is contrary to the fact. Therefore, an energy value function E (z) is introduced, wherein the energy value is the inverse number of the log-likelihood function value of the preset Gaussian mixture model, and the smaller the energy value, the more normal the sample is.
Specifically, the preset loss function includes three parts:
the first part
Figure BDA0003551456640000121
Representing the reconstruction error caused by the one-dimensional convolution self-encoder in the feature extraction process. The lower-dimensional features better preserve the key information of the input samples if the reconstruction error of the one-dimensional convolutional auto-encoder is smaller. Therefore, the influence caused by the reconstruction Error needs to be considered in the model training process, and the Mean Square Error (MSE) is used as the loss function of the one-dimensional convolution self-encoder in the model.
The second part e (zi) represents the energy value of the traffic sample data. The lower the sample energy value, the closer the sample approaches normal. Therefore, the model training process needs to minimize the sample energy to find the best combination of feature extraction and density estimation processes.
Third part
Figure BDA0003551456640000122
Represents a small value on the diagonal of the punished covariance matrix and plays a role of preventing the matrix from being irreversibleI.e. the covariance matrix is prevented from having 0 diagonal elements. We add a smaller value to the diagonal of the covariance matrix
Figure BDA0003551456640000123
Wherein
Figure BDA0003551456640000124
Preferably, the S3 specifically includes:
and S31, obtaining a detection energy value according to the flow data to be detected and the energy value function in the target detection model.
And substituting the flow data to be detected into an energy value function in the target detection model to obtain the detection energy value of the flow data to be detected.
And S32, judging whether the detection energy value is larger than a discrimination threshold value.
S33, if yes, judging that the detection result of the flow data to be detected is abnormal; and if not, judging that the detection result of the flow data to be detected is normal.
Preferably, the value of the discrimination threshold is: detecting an Mth sample energy value in the target quantity detection model;
wherein M is (1-c) N, N is the total number of traffic sample data in the first traffic sample data set, and c is the abnormal proportion of the traffic sample data in the first traffic sample data set.
Specifically, carrying out abnormity judgment on a detection energy value and a judgment threshold value of flow data to be detected, and judging that the flow data to be detected is abnormal when the detection energy value is greater than the judgment threshold value; and when the detection energy value is smaller than the discrimination threshold value, judging that the flow data to be detected is normal.
And the value of the discrimination threshold is to arrange the energy values corresponding to each second flow sample data in the second flow sample data set used for training in an ascending order, and take the energy value at the (1-c) N position in all the energy values as the discrimination threshold.
According to the technical scheme of the embodiment, the data dimension reduction is carried out on the original flow sample data set, the iteration training is carried out by adopting the preset detection model, the flow data to be detected is obtained and detected according to the target detection model, and the detection result is obtained. According to the technical scheme of the embodiment, the loss function of the detection model is minimized according to an iterative training mode, constraint training is performed through combination of data dimension reduction and density estimation, the problem that training targets are inconsistent is solved, and the accuracy of abnormal flow detection is improved.
As shown in fig. 2, an abnormal flow rate detecting system 200 according to an embodiment of the present invention includes: a first processing module 210, a second processing module 220, and a detection module 230;
the first processing module 210 is configured to: preprocessing an original flow sample data set to obtain and perform data dimension reduction on a first flow sample data set to obtain a second flow sample data set;
the second processing module 220 is configured to: performing iterative training on the second flow sample data set by adopting a preset detection model to obtain a target detection model;
the detection module 230 is configured to: and generating a detection result of the flow data to be detected according to the flow data to be detected and the target abnormal flow detection model.
According to the technical scheme of the embodiment, the data dimension reduction is carried out on the original flow sample data set, the iteration training is carried out by adopting the preset detection model, the flow data to be detected is obtained and detected according to the target detection model, and the detection result is obtained. According to the technical scheme of the embodiment, the loss function of the detection model is minimized according to an iterative training mode, constraint training is performed through combination of data dimension reduction and density estimation, the problem that training targets are inconsistent is solved, and the accuracy of abnormal flow detection is improved.
The above steps for implementing corresponding functions for each parameter and each module in the abnormal traffic detection system 200 of this embodiment may refer to each parameter and step in the above embodiment for an abnormal traffic detection method, which are not described herein again.
An embodiment of the present invention provides a storage medium, including: the storage medium stores instructions, and when the computer reads the instructions, the computer is caused to execute the steps of the abnormal traffic detection method, which may specifically refer to the parameters and the steps in the above embodiment of the abnormal traffic detection method, and details are not described herein.
Computer storage media such as: flash disks, portable hard disks, and the like.
An electronic device provided in an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and is characterized in that when the processor executes the computer program, the computer executes steps of an abnormal traffic detection method, which may specifically refer to each parameter and step in the above embodiment of an abnormal traffic detection method, and are not described herein again.
Those skilled in the art will appreciate that the present invention may be embodied as methods, apparatus, storage media and electronic devices.
Thus, the present invention may be embodied in the form of: the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," module "or" system. Furthermore, in some embodiments, the invention may also be embodied in the form of a computer program product in one or more computer-readable media having computer-readable program code embodied in the medium. Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.

Claims (10)

1. An abnormal traffic detection method, comprising:
s1, preprocessing the original flow sample data set to obtain and perform data dimension reduction on the first flow sample data set to obtain a second flow sample data set;
s2, performing iterative training on the second flow sample data set by adopting a preset detection model to obtain a target detection model;
and S3, generating a detection result of the flow data to be detected according to the flow data to be detected and the target abnormal flow detection model.
2. The abnormal traffic detection method according to claim 1, wherein the original traffic sample data set is: the first traffic sample data set comprises an original digital feature data set and an original symbol feature data set, and the first traffic sample data set comprises: a first set of digital signature data and a second set of digital signature data;
the preprocessing the original flow sample data set to obtain a first flow sample data set specifically includes:
converting each original symbol characteristic data in the original symbol characteristic data set by adopting one-hot coding to obtain the first digital characteristic data set;
and processing each original digital characteristic data in the original digital characteristic data set by adopting a maximum and minimum normalization method to obtain the second digital characteristic data set.
3. The abnormal traffic detection method according to claim 1, wherein the performing data dimension reduction on the first traffic sample data set to obtain a second traffic sample data set specifically comprises:
performing low-dimensional feature extraction on each flow sample data in the first flow sample data set by adopting a convolution self-encoder to obtain a compressed encoding data set;
performing data reconstruction on each flow sample data in the first flow sample data set to obtain a reconstructed flow sample data set, and obtaining a reconstruction error data set according to the reconstructed flow sample data set and the first flow sample data set;
and connecting the compressed encoding data set and the reconstruction error data set through an open to obtain the second flow sample data set.
4. The abnormal flow detection method according to claim 3, wherein reconstructing the error data set includes: relative Euclidean distance error and cosine similarity error;
obtaining a reconstruction error data set according to the reconstruction flow sample data set and the first flow sample data set, specifically including:
obtaining the relative Euclidean distance error according to the reconstructed flow sample data set, the first flow sample data set and a first preset calculation formula;
obtaining the cosine similarity error according to the reconstructed flow sample data set, the first flow sample data set and a second preset calculation formula;
connecting the relative Euclidean distance error with the cosine similarity error to obtain the reconstruction error data set;
wherein the first preset calculation formula is as follows:
Figure FDA0003551456630000021
the second preset calculation formula is as follows:
Figure FDA0003551456630000022
Figure FDA0003551456630000023
for the purpose of the relative euclidean distance error,
Figure FDA0003551456630000024
x is the first traffic sample data set, x is the cosine similarity error,
Figure FDA0003551456630000025
for said reconstructed traffic sample data set, xiFor the ith traffic sample data in the first traffic sample data set,
Figure FDA0003551456630000026
for the ith reconstructed flow sample data in the reconstructed flow sample data set, and
Figure FDA0003551456630000027
and xiAnd N is the total amount of the flow sample data in the first flow sample data set.
5. The abnormal flow detection method according to claim 1, wherein the preset detection model comprises: presetting a multilayer neural network model and a Gaussian mixture model;
the S2 specifically includes:
s21, calculating a plurality of sub-distribution probabilities of the second flow sample data set in the preset Gaussian mixture model by adopting the preset multilayer neural network;
s22, sequentially utilizing the second flow sample data set and each sub-distribution probability to obtain and obtain the mixed probability according to each sub-distribution
Figure FDA0003551456630000028
Mean value
Figure FDA0003551456630000029
Sum covariance
Figure FDA00035514566300000210
Generating a first Gaussian mixture model;
and S23, adopting the first Gaussian mixture model as the preset Gaussian mixture model, and returning to execute S21 until the parameters in the first Gaussian mixture model converge, and obtaining the target detection model according to the first Gaussian mixture model.
6. The abnormal flow detection method according to claim 5, wherein the preset detection model further comprises: presetting a loss function; when the parameters in the first gaussian mixture model reach convergence, obtaining the target detection model according to the first gaussian mixture model, specifically including: when the preset loss function in the first Gaussian mixture model reaches convergence, obtaining the target detection model according to the first Gaussian mixture model;
wherein the predetermined loss function is:
Figure FDA0003551456630000031
Figure FDA0003551456630000032
J(θe，θd，θm) For said predetermined loss function, θeFor the encoder parameters of said convolutional auto-encoder, θdFor decoder parameters of said convolutional autocoder, θmFor said predetermined Gaussian mixture model parameter, λ1Is the first parameter, lambda, of the predetermined detection model2Is a second element parameter of the preset detection model,
Figure FDA0003551456630000033
wherein, E (z)i) The energy value function corresponding to the preset Gaussian mixture model,
Figure FDA0003551456630000034
Figure FDA0003551456630000035
E(zi) The energy value of the ith flow sample data in the second flow sample data set is obtained;
Figure FDA0003551456630000036
Figure FDA0003551456630000037
p＝MLN(z；θm)，
Figure FDA0003551456630000038
z is the second traffic sample data set.
7. The abnormal flow detection method according to any one of claims 1 to 6, wherein the S3 specifically includes:
s31, obtaining a detection energy value according to the flow data to be detected and an energy value function in the target detection model;
s32, judging whether the detection energy value is larger than a discrimination threshold value;
s33, if yes, judging that the detection result of the flow data to be detected is abnormal; and if not, judging that the detection result of the flow data to be detected is normal.
8. The abnormal flow detection method according to claim 7, wherein the discrimination threshold takes the following values: detecting an Mth sample energy value in the target quantity detection model;
wherein M is (1-c) N, N is the total number of traffic sample data in the first traffic sample data set, and c is the abnormal proportion of the traffic sample data in the first traffic sample data set.
9. An abnormal flow detection system, comprising: the device comprises a first processing module, a second processing module and a detection module;
the first processing module is configured to: preprocessing an original flow sample data set to obtain and perform data dimension reduction on a first flow sample data set to obtain a second flow sample data set;
the second processing module is configured to: performing iterative training on the second flow sample data set by adopting a preset detection model to obtain a target detection model;
the detection module is used for: and generating a detection result of the flow data to be detected according to the flow data to be detected and the target abnormal flow detection model.
10. A storage medium having stored therein instructions that, when read by a computer, cause the computer to execute the abnormal flow detection method according to any one of claims 1 to 8.
CN202210265222.0A 2022-03-17 2022-03-17 Abnormal flow detection method, system and storage medium Pending CN114611619A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210265222.0A CN114611619A (en) 2022-03-17 2022-03-17 Abnormal flow detection method, system and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210265222.0A CN114611619A (en) 2022-03-17 2022-03-17 Abnormal flow detection method, system and storage medium

Publications (1)

Publication Number Publication Date
CN114611619A true CN114611619A (en) 2022-06-10

Family

ID=81864697

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210265222.0A Pending CN114611619A (en) 2022-03-17 2022-03-17 Abnormal flow detection method, system and storage medium

Country Status (1)

Country Link
CN (1) CN114611619A (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112702329A (en) * 2020-12-21 2021-04-23 四川虹微技术有限公司 Traffic data anomaly detection method and device and storage medium
CN112953924A (en) * 2021-02-04 2021-06-11 西安电子科技大学 Network abnormal flow detection method, system, storage medium, terminal and application
CN113516228A (en) * 2021-07-08 2021-10-19 哈尔滨理工大学 Network anomaly detection method based on deep neural network

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112702329A (en) * 2020-12-21 2021-04-23 四川虹微技术有限公司 Traffic data anomaly detection method and device and storage medium
CN112953924A (en) * 2021-02-04 2021-06-11 西安电子科技大学 Network abnormal flow detection method, system, storage medium, terminal and application
CN113516228A (en) * 2021-07-08 2021-10-19 哈尔滨理工大学 Network anomaly detection method based on deep neural network

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
BO ZONG 等: "DEEP AUTOENCODING GAUSSIAN MIXTURE MODEL FOR UNSUPERVISED ANOMALY DETECTION", 《ICLR》 *

Similar Documents

Publication Publication Date Title
Jiang et al. Network intrusion detection combined hybrid sampling with deep hierarchical network
Li et al. A hybrid malicious code detection method based on deep learning
CN110287983B (en) Single-classifier anomaly detection method based on maximum correlation entropy deep neural network
Kim et al. Fusions of GA and SVM for anomaly detection in intrusion detection system
CN111291860A (en) Anomaly detection method based on convolutional neural network feature compression
CN116957049B (en) Unsupervised internal threat detection method based on countermeasure self-encoder
Sikkandar et al. Soft biometrics‐based face image retrieval using improved grey wolf optimisation
CN111556016A (en) Network flow abnormal behavior identification method based on automatic encoder
CN111400713B (en) Malicious software population classification method based on operation code adjacency graph characteristics
Liu et al. An intrusion detection system integrating network-level intrusion detection and host-level intrusion detection
Zheng et al. Fighting fire with fire: A spatial–frequency ensemble relation network with generative adversarial learning for adversarial image classification
CN117081831A (en) Network intrusion detection method and system based on data generation and attention mechanism
CN114912109B (en) Abnormal behavior sequence identification method and system based on graph embedding
CN114611619A (en) Abnormal flow detection method, system and storage medium
CN114826718B (en) Multi-dimensional information-based internal network anomaly detection method and system
CN116563690A (en) Unmanned aerial vehicle sensor type unbalanced data anomaly detection method and detection system
CN116170187A (en) Industrial Internet intrusion monitoring method based on CNN and LSTM fusion network
CN113868650B (en) Vulnerability detection method and device based on code heterogeneous middle graph representation
Wang et al. Malware detection using cnn via word embedding in cloud computing infrastructure
CN113259369B (en) Data set authentication method and system based on machine learning member inference attack
CN113159181A (en) Industrial control network anomaly detection method and system based on improved deep forest
Xu et al. IoT-Oriented Distributed Intrusion Detection Methods Using Intelligent Classification Algorithms in Spark
CN117375970A (en) Network intrusion detection method, system and equipment for power system
CN118070107A (en) Deep learning-oriented network anomaly detection method, device, storage medium and equipment
Wan et al. Host Intrusion Detection Method Based on Short Sequence of System Call

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20220610