CN117830728A - Abnormal flow detection method, device, electronic equipment and medium - Google Patents
Abnormal flow detection method, device, electronic equipment and medium Download PDFInfo
- Publication number
- CN117830728A CN117830728A CN202410003474.5A CN202410003474A CN117830728A CN 117830728 A CN117830728 A CN 117830728A CN 202410003474 A CN202410003474 A CN 202410003474A CN 117830728 A CN117830728 A CN 117830728A
- Authority
- CN
- China
- Prior art keywords
- flow
- training
- identification
- model
- abnormal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000002159 abnormal effect Effects 0.000 title claims abstract description 133
- 238000001514 detection method Methods 0.000 title description 13
- 238000012549 training Methods 0.000 claims abstract description 150
- 238000000034 method Methods 0.000 claims abstract description 46
- 238000013526 transfer learning Methods 0.000 claims description 25
- 238000009792 diffusion process Methods 0.000 claims description 20
- 238000012545 processing Methods 0.000 claims description 20
- 230000006870 function Effects 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 13
- 238000010606 normalization Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 11
- 239000011159 matrix material Substances 0.000 claims description 10
- 230000008014 freezing Effects 0.000 claims description 8
- 238000007710 freezing Methods 0.000 claims description 8
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 230000000295 complement effect Effects 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 2
- 230000002829 reductive effect Effects 0.000 description 5
- 230000000153 supplemental effect Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000007689 inspection Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 210000002569 neuron Anatomy 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000002441 reversible effect Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000000137 annealing Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The application relates to the field of computer network security, in particular to a method, a device, electronic equipment and a medium for detecting abnormal flow, wherein the method comprises the steps of obtaining target flow to be detected; extracting features of the target flow to obtain flow features, generating gray images by the flow features, and taking the gray images as data to be identified; the method comprises the steps of inputting data to be identified into an abnormal flow identification model to be identified, obtaining an identification result, wherein the abnormal flow identification model is obtained based on a plurality of training samples, and the training samples are gray images generated based on flow characteristics by extracting characteristics of network flow to obtain flow characteristics. The method and the device have the effect of improving the accuracy of abnormal flow identification.
Description
Technical Field
The present disclosure relates to the field of computer network security, and in particular, to a method and apparatus for detecting abnormal traffic, an electronic device, and a medium.
Background
As the scale of video monitoring networks is continuously expanded, more and more video monitoring related applications are used, WEB access of the related applications is transited from HTTP to safer HTTPS, and encryption traffic in the networks is continuously increased; at the same time, as the use of non-standard ports increases, so too does the multi-protocol hybrid traffic, which makes traditional network anomaly traffic identification methods such as port-based, deep Packet Inspection (DPI) based and statistical based methods no longer applicable.
Currently, the method for identifying the abnormal flow of the main stream in the related technology includes directly extracting the characteristics of the network flow or converting the original flow into a gray scale map as the characteristics of the network flow, and then identifying the abnormal flow through a corresponding identification model. However, after converting the flow into the gray scale map, the interpretability is low, and there may be a case where the gray scale map does not intercept useful information, which results in a low accuracy of identifying abnormal flow in the related art.
Disclosure of Invention
In order to improve the accuracy of abnormal flow identification, the application provides an abnormal flow detection method, an abnormal flow detection device, electronic equipment and a medium.
In a first aspect, the present application provides a method for detecting abnormal flow in flow detection, which adopts the following technical scheme:
a flow detection abnormal flow detection method includes:
obtaining a target flow to be detected;
extracting features of the target flow to obtain flow features, generating gray images by the flow features, and taking the gray images as data to be identified;
and inputting the data to be identified into an abnormal flow identification model for identification to obtain an identification result, wherein the abnormal flow identification model is obtained based on a plurality of training samples, and the training samples are used for extracting characteristics of network flow to obtain flow characteristics and generating gray images based on the flow characteristics.
By adopting the technical scheme, because the training sample for training to obtain the abnormal flow identification model is the gray image generated based on the flow characteristics extracted from the network flow, when a certain section of target flow is identified, the flow characteristics in the target flow are extracted first, then the gray image is generated based on the flow characteristics, and the abnormal flow identification model is used for identification. Compared with the related art, in the scheme of the application, the useful information with more target flow can be fully contained in the gray level image, and meanwhile, the interpretability of the gray level image is improved, so that the accuracy of identifying abnormal flow can be improved.
In one possible implementation, the training step of the abnormal traffic identification model includes:
acquiring a plurality of groups of network traffic for training, extracting characteristics of the plurality of groups of network traffic to obtain flow characteristics corresponding to the plurality of groups of network traffic, and generating gray images corresponding to the plurality of groups of network traffic respectively by the flow characteristics corresponding to the plurality of groups of network traffic, wherein the gray images corresponding to the network traffic containing abnormal traffic are initial negative samples, and the gray images corresponding to the network traffic not containing abnormal traffic are positive samples;
Generating a plurality of complementary negative samples based on the initial negative samples and the trained diffusion model so that the ratio of the positive samples to the negative samples reaches a preset ratio, wherein the negative samples comprise complementary negative samples and initial negative samples;
constructing an image recognition model based on transfer learning;
and training the image recognition model by taking the positive sample and the negative sample as training sets to obtain an abnormal flow recognition model.
In one possible implementation manner, the building the image recognition model based on the transfer learning includes:
determining an initial network model of transfer learning, and determining training weights;
acquiring a target data set, wherein the target data set is a public image classification data set;
and training the initial network model by taking the target data set as a training sample set to obtain an image recognition model based on transfer learning.
In one possible implementation manner, the training the image recognition model by using the positive sample and the negative sample as training sets to obtain an abnormal flow recognition model includes:
determining a frozen junction layer, wherein the frozen junction layer is a layer structure which does not need to participate in training in the image recognition model;
Freezing the freezing layer in the image recognition model to obtain a pre-training model;
determining a training rate in a training process of the pre-training model based on an attenuation algorithm;
and taking the positive sample and the negative sample as training sets, and training the pre-training model based on the training speed to obtain an abnormal flow identification model.
In one possible implementation manner, the inputting the data to be identified into the abnormal traffic identification model for identification, to obtain an identification result, includes:
inputting the identification data into an abnormal flow identification model for identification to obtain an output result, converting the output result into probability distribution, and carrying out normalization processing to obtain an output value;
and obtaining an identification result based on the output value and the prediction model, wherein the identification result is constructed based on historical identification data of the abnormal flow identification model, and the historical identification data comprises an output value obtained by identifying the data to be identified which is input each time by the abnormal flow identification model.
In one possible implementation, constructing the predictive model includes:
constructing a prediction model based on the historical identification data of the abnormal flow identification model and a function F (x);
Wherein:
α、α 1 、α 2 and x is an output value in the history identification data, and n is a cis-position of the history identification data.
In one possible implementation, the generating the gray scale image from the flow characteristic includes:
carrying out normalization processing and single-heat encoding processing on the flow characteristics to obtain preprocessing data;
and constructing a feature matrix from the preprocessed data, and converting the feature matrix into gray maps with the same size.
In a second aspect, the present application provides an abnormal flow identification apparatus, which adopts the following technical scheme:
an abnormal flow identification device, comprising:
the target flow acquisition module is used for acquiring target flow to be detected;
the feature extraction module is used for carrying out feature extraction on the target flow to obtain flow features, generating gray images by the flow features, and taking the gray images as data to be identified;
the identification module is used for inputting the data to be identified into an abnormal flow identification model to identify and obtain an identification result, the abnormal flow identification model is obtained based on a plurality of training samples, and the training samples are gray images generated based on the flow characteristics by extracting the characteristics of the network flow to obtain the flow characteristics.
By adopting the technical scheme, because the training sample for training to obtain the abnormal flow identification model is the gray image generated based on the flow characteristics extracted from the network flow, when a certain section of target flow is identified, the flow characteristics in the target flow are extracted first, then the gray image is generated based on the flow characteristics, and the abnormal flow identification model is used for identification. Compared with the related art, in the scheme of the application, the useful information with more target flow can be fully contained in the gray level image, and meanwhile, the interpretability of the gray level image is improved, so that the accuracy of identifying abnormal flow can be improved.
In a possible implementation manner, the device further comprises a training device for the abnormal flow identification model, wherein the training device is specifically used for:
acquiring a plurality of groups of network traffic for training, extracting characteristics of the plurality of groups of network traffic to obtain flow characteristics corresponding to the plurality of groups of network traffic, and generating gray images corresponding to the plurality of groups of network traffic respectively by the flow characteristics corresponding to the plurality of groups of network traffic, wherein the gray images corresponding to the network traffic containing abnormal traffic are initial negative samples, and the gray images corresponding to the network traffic not containing abnormal traffic are positive samples;
Generating a plurality of complementary negative samples based on the initial negative samples and the trained diffusion model so that the ratio of the positive samples to the negative samples reaches a preset ratio, wherein the negative samples comprise complementary negative samples and initial negative samples;
constructing an image recognition model based on transfer learning;
and training the image recognition model by taking the positive sample and the negative sample as training sets to obtain an abnormal flow recognition model.
In one possible implementation manner, the training device is specifically used for constructing an image recognition model based on transfer learning:
determining an initial network model of transfer learning, and determining training weights;
acquiring a target data set, wherein the target data set is a public image classification data set;
training the initial network model by taking the target data set as a training sample set to obtain an image recognition model based on transfer learning, wherein in one possible implementation manner, the training device is specifically used for training the image recognition model by taking the positive sample and the negative sample as training sets to obtain an abnormal flow recognition model:
determining a frozen junction layer, wherein the frozen junction layer is a layer structure which does not need to participate in training in the image recognition model;
Freezing the freezing layer in the image recognition model to obtain a pre-training model;
determining a training rate in a training process of the pre-training model based on an attenuation algorithm;
and taking the positive sample and the negative sample as training sets, and training the pre-training model based on the training speed to obtain an abnormal flow identification model.
In one possible implementation manner, the identification module inputs the data to be identified into an abnormal flow identification model to identify, and is specifically configured to:
inputting the identification data into an abnormal flow identification model for identification to obtain an output result, converting the output result into probability distribution, and carrying out normalization processing to obtain an output value;
and obtaining an identification result based on the output value and the prediction model, wherein the identification result is constructed based on historical identification data of the abnormal flow identification model, and the historical identification data comprises an output value obtained by identifying the data to be identified which is input each time by the abnormal flow identification model.
In one possible implementation manner, the method further comprises a prediction model building module, wherein the prediction model building module is specifically used for:
Constructing a prediction model based on the historical identification data of the abnormal flow identification model and a function F (x);
wherein:α、α 1 、α 2 and x is an output value in the history identification data, and n is a cis-position of the history identification data.
In one possible implementation manner, the feature extraction module is specifically configured to, when generating a gray scale image from the flow feature:
carrying out normalization processing and single-heat encoding processing on the flow characteristics to obtain preprocessing data;
and constructing a feature matrix from the preprocessed data, and converting the feature matrix into gray maps with the same size.
In a third aspect, the present application provides an electronic device, which adopts the following technical scheme:
an electronic device, the electronic device comprising:
at least one processor;
a memory;
at least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, the at least one application configured to: the abnormal flow identification device method described above is executed.
In a fourth aspect, the present application provides a computer readable storage medium, which adopts the following technical scheme:
a computer-readable storage medium, comprising: a computer program capable of being loaded by a processor and executing the abnormal flow recognition apparatus method described above is stored.
In summary, the present application includes at least one of the following beneficial technical effects:
1. because the training sample for training to obtain the abnormal flow identification model is a gray image generated based on the flow characteristics extracted from the network flow, when a certain section of target flow is identified, the flow characteristics in the target flow are extracted first, then the gray image is generated based on the flow characteristics, and the abnormal flow identification model is used for identification. Compared with the related art, in the scheme of the application, the useful information with more target flow can be fully contained in the gray level image, and meanwhile, the interpretability of the gray level image is improved, so that the accuracy of identifying abnormal flow can be improved.
Drawings
FIG. 1 is a schematic flow chart of an abnormal flow detection method in an embodiment of the present application;
FIG. 2 is a schematic flow chart of training to obtain an abnormal flow identification model in an embodiment of the present application;
FIG. 3 is a schematic diagram of an abnormal flow detection device according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
The present application is described in further detail below in conjunction with fig. 1-4.
Modifications of the embodiments which do not creatively contribute to the invention may be made by those skilled in the art after reading the present specification, but are protected by patent laws only within the scope of claims of the present application.
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
In addition, the term "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In this context, unless otherwise specified, the term "/" generally indicates that the associated object is an "or" relationship.
The method for identifying the abnormal flow of the main stream in the related technology comprises the steps of directly extracting the characteristics of the network flow or converting the original flow into a gray level graph to serve as the characteristics of the network flow, and then carrying out abnormal flow identification through a corresponding identification model. However, after converting the flow into the gray scale map, the interpretability is low, and there may be a case where the gray scale map does not intercept useful information, which results in a low accuracy of identifying abnormal flow in the related art.
In order to improve accuracy of abnormal traffic identification, the embodiment of the present application provides an abnormal traffic identification method, which is executed by an electronic device, referring to fig. 1, and includes steps S11 to S13, where:
step S11, obtaining the target flow to be detected.
For the embodiment of the application, the target flow to be detected may be automatically acquired by the electronic device based on some screening or triggering conditions, or may be specified or input by a user.
And step S12, extracting the characteristics of the target flow to obtain flow characteristics, generating a gray image by using the flow characteristics, and taking the gray image as data to be identified.
For the embodiment of the application, the DPI (Deep Packet Inspection ) technology can be implemented on the basis of the software Ntopng for real-time monitoring and displaying of the network traffic to extract the characteristics of the target traffic, so as to obtain various traffic characteristics for representing the target traffic. The types of the traffic characteristics include, but are not limited to, five-tuple, traffic type, packet length, packet number, URG flag bit number, ACK flag bit number, flow rate, number of successfully transmitted bytes, active time, idle time, packet header length, and ratio of downstream traffic to upstream traffic. And then generating a gray image based on the extracted flow characteristics.
And S13, inputting data to be identified into an abnormal flow identification model for identification to obtain an identification result, wherein the abnormal flow identification model is obtained based on a plurality of training samples, the training samples are used for extracting characteristics of network flow to obtain flow characteristics, and gray images are generated based on the flow characteristics.
For the embodiment of the application, the network traffic is subjected to feature extraction to obtain traffic features, gray images generated based on the traffic features are used as training samples to train a plurality of gray images with labels to obtain an abnormal traffic recognition model. And then, inputting the gray level image of the target flow obtained in the step S12 into an abnormal flow identification model as data to be identified, and obtaining an identification result, wherein the identification result is that the target flow carries abnormal flow or the target flow does not carry abnormal flow.
In the solution of the embodiment of the present application, since the training sample for training to obtain the abnormal traffic recognition model is a gray image generated based on the traffic characteristics extracted from the network traffic, when a certain target traffic is recognized, the traffic characteristics in the target traffic are extracted first, then the gray image is generated based on the traffic characteristics, and the abnormal traffic recognition model is used for recognition. Compared with the related art, in the scheme of the application, the useful information with more target flow can be fully contained in the gray level image, and meanwhile, the interpretability of the gray level image is improved, so that the accuracy of identifying abnormal flow can be improved.
Further, in the embodiment of the present application, the training step of the abnormal flow identification model includes steps S01 to S04, wherein: step S01 is an initial sample acquisition step, which is mainly used for acquiring an initial sample for training; step S02 is a training sample generation step based on a diffusion model, and because the network traffic carrying abnormal traffic is not easy to obtain in reality, the number of negative samples extracted based on the network traffic carrying abnormal traffic is small, which results in easy occurrence of over-fitting during model training. In order to reduce the occurrence of the model overfitting condition, step S02 generates a part of negative samples based on the diffusion model, so that the proportion of the positive samples to the negative samples reaches the preset proportion requirement, and the probability of overfitting in model training is reduced.
Step S03 is an image recognition model construction step based on transfer learning, and step S04 is a step of training the image recognition model to obtain an abnormal flow recognition model. The transfer learning is a method for solving the problem of relatively lack of data resources in the target field, and the knowledge transfer and generalization of the target field are realized mainly by acquiring knowledge from source fields and target fields with rich resources and training related models. The method can effectively utilize the existing knowledge and model, help the target field to obtain more accurate and reliable prediction results, and optimize the model training process. In the scheme of the application, under the condition that the number of positive samples and negative samples is small, the efficiency and accuracy of model training can be improved through transfer learning.
The following describes step S01 to step S04 in detail.
Step S01, an initial sample acquisition step: obtaining a plurality of groups of network traffic for training, extracting characteristics of the plurality of groups of network traffic to obtain flow characteristics corresponding to the plurality of groups of network traffic, generating gray images corresponding to the plurality of groups of network traffic by the flow characteristics corresponding to the plurality of groups of network traffic, wherein the gray images corresponding to the network traffic containing abnormal traffic are initial negative samples, and the gray images corresponding to the network traffic not containing abnormal traffic are positive samples.
The acquired network traffic is provided with a label, and the label is provided with abnormal traffic or not. The DPI (Deep Packet Inspection ) technique may then be implemented based on the software that monitors the network traffic in real time, which extracts features for each set of network traffic. The feature extraction of the DPI technology on the basis of the Ntopng is only one possible implementation manner disclosed in the embodiments of the present application, and specific feature extraction means and techniques are not specifically limited in the embodiments of the present application.
Extracting each group of network traffic to obtain traffic characteristics, wherein the traffic characteristics comprise, but are not limited to, five-tuple, traffic type, data packet length, data packet number, URG (universal joint group) flag bit number, ACK (acknowledgement) flag bit number, flow rate, successful transmission byte number, active time, idle time, data packet head length, and ratio of downlink traffic to uplink traffic. And then converting the extracted flow characteristics of each group of network flows into json format.
Further, preprocessing is performed on each flow characteristic converted into json format, including data normalization processing and single-heat encoding processing. Specifically, normalized by Min-Max:mapping data of flow characteristics to specified [0-1 ]]In the range, thereby realizing unified processing of data. The single-hot coding adopts an N-bit state register to code N states, each state has an independent register bit, and only one bit is effective at any time, so that a classified variable can be converted into a numerical variable, and the algorithm can better utilize the characteristics; wherein 0 represents white and 1 represents black. And (3) after the preprocessed features are subjected to the single-heat coding, the number of the features is increased to a degree sufficient for constructing the number of gray images, a feature matrix is reconstructed, and the feature matrix is converted into gray images. The converted gray level image and the corresponding network traffic carry the same label, the gray level image which does not carry abnormal traffic is taken as an initial positive sample, and the gray level image which carries abnormal traffic is taken as an initial negative sample.
Step S02, training sample generation based on a diffusion model: generating a plurality of supplemental negative samples based on the initial negative samples and the trained diffusion model such that the ratio of the positive samples to the negative samples reaches a preset ratio, the negative samples including the supplemental negative samples and the initial negative samples.
The logic of the diffusion model is to apply noise to the image step by step in the forward phase until the image is corrupted to become entirely gaussian noise, and then learn to recover from gaussian noise to the original image in the reverse phase. The initial negative samples are input into a trained diffusion model to generate corresponding supplemental negative samples.
Specifically, the data processing process of the initial diffusion model includes a forward stage and a reverse stage, wherein: in the forward phase, the current image x t X being equal to the last time only t-1 In relation to each other,z is noise conforming to a standard Gaussian distribution N (0-l), is alpha t Is an empirical constant, alpha t Will decrease with increasing t, so that x t-1 Is of smaller and smaller weight, z 1 Is increasingly weighted. So as t increases, the duty cycle of the noise increases and the duty cycle at the previous time becomes smaller. />
Reverse phase, i.e. picture generation processMu and sigma 2 Representative is according to x t Guessed x t-1 Mean and variance of the distribution. />(z～N(0,I))。
The process of training the initial diffusion model to obtain a trained diffusion model comprises the following steps: randomly extracting an initial negative sample x from an existing abnormal traffic sample library 0 Initial inputThe diffusion model is started, a time t is randomly selected, a noise (E-N (0,I)) is randomly extracted, and then the diffusion model is based on And calculating a loss value, calculating a gradient, updating a weight and updating a model. And then repeating the process until the diffusion model converges to obtain a trained diffusion model. Wherein the loss function of the diffusion model is an L2 loss function, i.e. a Least Squares Error (LSE) function, the L2 loss function being the minimization of the sum of squares of the differences between the target value and the estimated value. The abnormal flow sample library comprises a plurality of gray images which are extracted based on the flow carrying the abnormal flow and are generated based on the extracted characteristics.
Specifically, each initial negative sample may be input into the trained diffusion model, or a part of the initial negative samples may be selected to be input into the trained diffusion model, specifically, the initial negative samples and the initial positive samples are determined according to the number of the initial negative samples and the initial positive samples.
Further, the initial positive sample is taken as a positive sample, the initial negative sample and the supplementary negative sample are taken as negative samples, and the negative sample and the positive sample are taken as training sets.
Step S03, an image recognition model construction step based on transfer learning: and constructing an image recognition model based on transfer learning.
Specifically, the step of constructing an image recognition model based on transfer learning includes: determining an initial network model of transfer learning, and determining training weights; acquiring a target data set, wherein the target data set is a public image classification data set; and training the initial network model by taking the target data set as a training sample set to obtain an image recognition model based on transfer learning.
Specifically, the transition learning model may be any one of VGG19, res net50, xception, mobileNet, denseNet, and conceptionv 3, which is not limited in the embodiment of the present application. In a subsequent embodiment, the instance is performed with the transfer learning model as the initial Xception model.
Further, selecting a pre-training weight, wherein the training weight can be any one of COCO, BERT, imageNet, and the like, selecting ImageNet as the training weight, wherein the ImageNet is a large-scale image classification data set, which comprises more than 100 ten thousand pictures and 1000 categories, and the ImageNet training weight is obtained by training on larger-scale data.
The training set obtained in step S03 has a larger difference from the disclosed classified data set, and the training set obtained in step S03 belongs to a small data set (classification categories are fewer), so that if the training amount of the model is less, the learning of the model is limited; if the training amount of the model is large, the model is over-fitted.
Therefore, in the solution of the embodiment of the present application, the training solution adopted is to freeze a part of the convolution layers of the bottom layer, that is, keep the weights of the convolution layers without training, and adjust the convolution of a part of the higher layers and the full connection layer, so that the model can learn more contents. Among them, the frozen layer structure is called frozen layer, and the convolutional layer and the fully-connected layer which are adjusted are called adjusted layer.
Step S04, training the image recognition model to obtain an abnormal flow recognition model: and training the image recognition model by taking the positive sample and the negative sample as training sets to obtain an abnormal flow recognition model.
Further, the process of training the image recognition model by using the positive sample and the negative sample as training sets comprises the following steps: determining a frozen layer, wherein the frozen layer is a layer structure which does not need to participate in training in the image recognition model; freezing the frozen layer in the image recognition model to obtain a pre-training model; determining a training rate in a training process of the pre-training model based on an attenuation algorithm; and taking the positive sample and the negative sample as training sets, and training the pre-training model based on the training speed to obtain the abnormal flow identification model.
Specifically, the structure of the Xreception model is set forth herein: the Xreception model consists of 36 convolution layers, is constructed into 14 modules, and introduces Entry, middle, exit three flows, each of which is internally different. The Entry flow is mainly used for continuously downsampling, so that the dimension of the space is reduced; middle flow is to constantly learn association relations and optimize features; exit flow is a summary and collation feature used to route full junction layer expressions.
In the embodiment of the application, 4 modules of the Entry flow are frozen so that the model can keep the advantages of the original model; the 8 modules of Middle flow and the 4 modules of Exit flow are adjusted so that the model can learn more of the target task.
The Entry flow module firstly puts the input picture into a 3x3 convolution layer, the channel number is 32, the step length is 2, and the activation function is a ReLU function; the separable convolution function is utilized again, wherein the separable convolution function comprises Depthwise Convolution of 3x3 and Pointwise Convolution of 1x1, and is used for carrying out the addition of residual structures by sending the obtained result into a max pulling layer of 3x3 of stride=2x2 after the dimension of the feature is increased and decreased, and the obtained feature is convolved with 1x1, and x is the sum of residual structures l+1 ＝x l +F(x l ,W l )，F(x l ,W l ) The method is characterized in that the network mapping is performed before summation, so that low-level features in the input are reserved, and high-level features are learned through subsequent layers of the network, so that when the number of layers of the network is deep, the accuracy is not greatly reduced.
Middle flow is a core part of an Xreception model, a Depthwise Separable Convolution structure is adopted, the depth and the expression capacity of a network are improved by stacking a plurality of Depthwise Separable Convolution layers, each Depthwise Separable Convolution layer is followed by batch normalization, so that output data is more stable, and a ReLU activation function is applied.
Specifically, setting the learning_rate of the frozen layer to 0.001, and gradually reducing the learning rate by using a cosine annealing attenuation mode in the training process so as to update the frozen layer more slowly; the learning_rate for the tuned layer is set to 0.01, so that the features of the target data set can be learned faster.
Specifically, for Middle flow, the residual structure constructed by adding input and output of the original Depthwise Separable Convolution layers is changed to a convolution layer with 1x1 added with one stride=2x2Then adding the residual structure and the output to construct an adjusted residual structure, x l+1 ＝x l +F(x l ,W l ). The adjusted model improves the degree of retaining the original model, so that the model can learn the advantages of more original models, and the probability of overfitting is reduced.
Because the original model has high detail requirements and the classification of the data set used for training is very various, the whole structure and global characteristics are more emphasized in the content of the application. The original model is a 3x3 convolution with a small perceived field of view (representing the size of the range of input information that a neuron can receive). Thus, in the embodiment of the present application, the original 3x3 convolution kernel is changed to a 5x5 convolution kernel, and the original 2 3x3 convolution kernels are changed to 1 5x5 convolution kernels. Although 2 convolution kernels of 3x3 would be less computationally efficient than 1 convolution kernel of 5x5 (parameters of convolution kernels of 1x 5: parameters of convolution kernels of 5x5 = 25,2: parameters of convolution kernels of 3x 3: 3x3x2 = 18), the modified neurons are identical to the original neural receptive fields of the features, and thus modifying the model after the neurons can be done to improve the processing efficiency for the overall and global features.
Further, training is carried out on the target data set, a cross entropy loss function is set as a loss odd function of the model, an Adam optimizer is selected to carry out automatic adjustment on the learning rate, the trained epoch is set as 20, and the target data set is used for training the Xattention model, so that an image recognition model is obtained. The image recognition model is then trained based on a training set comprising positive and negative samples to obtain a flow recognition model.
Further, in step S103, the data to be identified is input into an abnormal flow identification model to be identified, so as to obtain an identification result, which may specifically include:
inputting the identification data into an abnormal flow identification model for identification to obtain an output result, converting the output result into probability distribution, and carrying out normalization processing to obtain an output value;
and obtaining a recognition result based on the output value and the prediction model, wherein the recognition result is constructed based on the historical recognition data of the abnormal flow recognition model, and the historical recognition data comprises the output value obtained by recognizing the data to be recognized input each time by the abnormal flow recognition model.
Further, the method further comprises step S05 after step S04, wherein the step of constructing the prediction model by S05 includes: and (3) pre-constructing test sets for testing the flow identification model, wherein each test set is a gray level image obtained through the processing of the step (S01). The output of the flow identification model is converted into probability distribution through a softmax function and output. At the same time, the output is normalized so that the output range is 0-1 ]And the sum of all outputs is 1. The output result is retained, and based on,and constructing a prediction model. Wherein alpha, alpha 1 、α 2 And x is an output value in the history identification data, and n is a cis-position of the history identification data. According to the method and the device, the historical prediction probability is also integrated into the influence on the current prediction, the weighted analysis is carried out, the judgment is carried out according to the fusion calculation of the historical data and the current prediction probability, the final prediction result is obtained, the result is more stable and accurate, and the probability of misjudgment caused by the accidental of the data is reduced.
The above embodiment describes an abnormal flow detection method from the viewpoint of a method flow, and the following embodiment describes an abnormal flow detection device from the viewpoint of a virtual module or a virtual unit, specifically the following embodiment.
The embodiment of the application provides an abnormal flow detection device, as shown in fig. 3, the abnormal flow detection device may specifically include a target flow acquisition module 301, a feature extraction module 302, and an identification module 303, where:
a target flow obtaining module 301, configured to obtain a target flow to be detected;
the feature extraction module 302 is configured to perform feature extraction on the target flow to obtain a flow feature, generate a gray image from the flow feature, and use the gray image as data to be identified;
The identifying module 303 is configured to input data to be identified into an abnormal traffic identifying model to identify, so as to obtain an identifying result, where the abnormal traffic identifying model is obtained based on a plurality of training samples, and the training samples are used to perform feature extraction on network traffic to obtain traffic features, and generate a gray scale image based on the traffic features.
By adopting the technical scheme, because the training sample for training to obtain the abnormal flow identification model is the gray image generated based on the flow characteristics extracted from the network flow, when a certain section of target flow is identified, the flow characteristics in the target flow are extracted first, then the gray image is generated based on the flow characteristics, and the abnormal flow identification model is used for identification. Compared with the related art, in the scheme of the application, the useful information with more target flow can be fully contained in the gray level image, and meanwhile, the interpretability of the gray level image is improved, so that the accuracy of identifying abnormal flow can be improved.
In one possible implementation manner, the device further comprises a training device for the abnormal flow identification model, wherein the training device is specifically used for:
acquiring a plurality of groups of network traffic for training, extracting characteristics of the plurality of groups of network traffic to obtain flow characteristics corresponding to the plurality of groups of network traffic, generating gray images corresponding to the plurality of groups of network traffic by the flow characteristics corresponding to the plurality of groups of network traffic, wherein the gray images corresponding to the network traffic containing abnormal traffic are initial negative samples, and the gray images corresponding to the network traffic not containing abnormal traffic are positive samples;
Generating a plurality of supplemental negative samples based on the initial negative samples and the trained diffusion model so that the ratio of the positive samples to the negative samples reaches a preset ratio, the negative samples including the supplemental negative samples and the initial negative samples;
constructing an image recognition model based on transfer learning;
and training the image recognition model by taking the positive sample and the negative sample as training sets to obtain an abnormal flow recognition model.
In one possible implementation manner, the training device is specifically used for constructing an image recognition model based on transfer learning:
determining an initial network model of transfer learning, and determining training weights;
acquiring a target data set, wherein the target data set is a public image classification data set;
training an initial network model by taking a target data set as a training sample set to obtain an image recognition model based on transfer learning, wherein in a possible implementation mode, the training device is used for training the image recognition model by taking a positive sample and a negative sample as training sets, and is particularly used for obtaining an abnormal flow recognition model:
determining a frozen layer, wherein the frozen layer is a layer structure which does not need to participate in training in the image recognition model;
freezing the frozen layer in the image recognition model to obtain a pre-training model;
Determining a training rate in a training process of the pre-training model based on an attenuation algorithm;
and taking the positive sample and the negative sample as training sets, and training the pre-training model based on the training speed to obtain the abnormal flow identification model.
In one possible implementation manner, the identifying module 303 inputs the data to be identified into the abnormal traffic identifying model to identify, and is specifically configured to:
inputting the identification data into an abnormal flow identification model for identification to obtain an output result, converting the output result into probability distribution, and carrying out normalization processing to obtain an output value;
and obtaining a recognition result based on the output value and the prediction model, wherein the recognition result is constructed based on the historical recognition data of the abnormal flow recognition model, and the historical recognition data comprises the output value obtained by recognizing the data to be recognized input each time by the abnormal flow recognition model.
In one possible implementation manner, the method further comprises a prediction model building module, wherein the prediction model building module is specifically used for:
constructing a prediction model based on the history identification data of the abnormal flow identification model and the function F (x);
wherein:α、α 1 、α 2 and x is an output value in the history identification data, and n is a cis-position of the history identification data.
In one possible implementation, the feature extraction module 302 is specifically configured to, when generating a gray scale image from the flow feature:
carrying out normalization processing and single-heat encoding processing on the flow characteristics to obtain preprocessing data;
and constructing a feature matrix from the preprocessed data, and converting the feature matrix into gray maps with the same size.
In an embodiment of the present application, as shown in fig. 4, an electronic device 400 shown in fig. 4 includes: a processor 401 and a memory 403. Processor 401 is connected to memory 403, such as via bus 402. Optionally, the electronic device 400 may also include a transceiver 404. It should be noted that, in practical applications, the transceiver 404 is not limited to one, and the structure of the electronic device 400 is not limited to the embodiment of the present application.
The processor 401 may be a CPU (Central Processing Unit ), general purpose processor, DSP (Digital Signal Processor, data signal processor), ASIC (Application Specific Integrated Circuit ), FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic device, transistor logic device, hardware components, or any combination thereof. Which may implement or perform the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. Processor 401 may also be a combination that implements computing functionality, such as a combination comprising one or more microprocessors, a combination of a DSP and a microprocessor, or the like.
Bus 402 may include a path to transfer information between the components. Bus 402 may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or EISA (Extended Industry Standard Architecture ) bus, among others. Bus 402 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in fig. 4, but not only one bus or one type of bus.
The Memory 403 may be, but is not limited to, a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory ) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory ), a CD-ROM (Compact Disc Read Only Memory, compact disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
The memory 403 is used for storing application program codes for executing the present application and is controlled to be executed by the processor 401. The processor 401 is arranged to execute application code stored in the memory 403 for implementing what is shown in the foregoing method embodiments.
Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. But may also be a server or the like. The electronic device shown in fig. 4 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.
The present application provides a computer readable storage medium having a computer program stored thereon, which when run on a computer, causes the computer to perform the corresponding method embodiments described above.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.
The foregoing is only a partial embodiment of the present application, and it should be noted that, for a person skilled in the art, several improvements and modifications can be made without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.
Claims (10)
1. An abnormal traffic recognition method, comprising:
obtaining a target flow to be detected;
extracting features of the target flow to obtain flow features, generating gray images by the flow features, and taking the gray images as data to be identified;
and inputting the data to be identified into an abnormal flow identification model for identification to obtain an identification result, wherein the abnormal flow identification model is obtained based on a plurality of training samples, and the training samples are used for extracting characteristics of network flow to obtain flow characteristics and generating gray images based on the flow characteristics.
2. The abnormal traffic identification method according to claim 1, wherein the training step of the abnormal traffic identification model comprises:
acquiring a plurality of groups of network traffic for training, extracting characteristics of the plurality of groups of network traffic to obtain flow characteristics corresponding to the plurality of groups of network traffic, and generating gray images corresponding to the plurality of groups of network traffic respectively by the flow characteristics corresponding to the plurality of groups of network traffic, wherein the gray images corresponding to the network traffic containing abnormal traffic are initial negative samples, and the gray images corresponding to the network traffic not containing abnormal traffic are positive samples;
Generating a plurality of complementary negative samples based on the initial negative samples and the trained diffusion model so that the ratio of the positive samples to the negative samples reaches a preset ratio, wherein the negative samples comprise complementary negative samples and initial negative samples;
constructing an image recognition model based on transfer learning;
and training the image recognition model by taking the positive sample and the negative sample as training sets to obtain an abnormal flow recognition model.
3. The abnormal traffic identification method according to claim 2, wherein the constructing the image identification model based on the transfer learning comprises:
determining an initial network model of transfer learning, and determining training weights;
acquiring a target data set, wherein the target data set is a public image classification data set;
and training the initial network model by taking the target data set as a training sample set to obtain an image recognition model based on transfer learning.
4. The abnormal traffic identification method according to claim 2, wherein training the image identification model using the positive sample and the negative sample as training sets to obtain an abnormal traffic identification model comprises:
Determining a frozen junction layer, wherein the frozen junction layer is a layer structure which does not need to participate in training in the image recognition model;
freezing the freezing layer in the image recognition model to obtain a pre-training model;
determining a training rate in a training process of the pre-training model based on an attenuation algorithm;
and taking the positive sample and the negative sample as training sets, and training the pre-training model based on the training speed to obtain an abnormal flow identification model.
5. The method for identifying abnormal traffic according to any one of claims 1 to 4, wherein the step of inputting the data to be identified into the abnormal traffic identification model to identify the data to be identified to obtain an identification result includes:
inputting the identification data into an abnormal flow identification model for identification to obtain an output result, converting the output result into probability distribution, and carrying out normalization processing to obtain an output value;
and obtaining an identification result based on the output value and the prediction model, wherein the identification result is constructed based on historical identification data of the abnormal flow identification model, and the historical identification data comprises an output value obtained by identifying the data to be identified which is input each time by the abnormal flow identification model.
6. The method of claim 5, wherein constructing the predictive model comprises:
constructing a prediction model based on the historical identification data of the abnormal flow identification model and a function F (x);
wherein:
α、α 1 、α 2 and x is an output value in the history identification data, and n is a cis-position of the history identification data.
7. The method of claim 1, wherein generating a gray scale image from the flow characteristics comprises:
carrying out normalization processing and single-heat encoding processing on the flow characteristics to obtain preprocessing data;
and constructing a feature matrix from the preprocessed data, and converting the feature matrix into gray maps with the same size.
8. An abnormal flow rate identification device, comprising:
the target flow acquisition module is used for acquiring target flow to be detected;
the feature extraction module is used for carrying out feature extraction on the target flow to obtain flow features, generating gray images by the flow features, and taking the gray images as data to be identified;
the identification module is used for inputting the data to be identified into an abnormal flow identification model to identify and obtain an identification result, the abnormal flow identification model is obtained based on a plurality of training samples, and the training samples are gray images generated based on the flow characteristics by extracting the characteristics of the network flow to obtain the flow characteristics.
9. An electronic device, comprising:
at least one processor;
a memory;
at least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, the at least one application configured to: performing the abnormal traffic identification method of any of claims 1-7.
10. A computer-readable storage medium, comprising: a computer program that can be loaded by a processor and that performs the abnormal traffic identification method according to any one of claims 1-7 is stored.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410003474.5A CN117830728A (en) | 2024-01-02 | 2024-01-02 | Abnormal flow detection method, device, electronic equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410003474.5A CN117830728A (en) | 2024-01-02 | 2024-01-02 | Abnormal flow detection method, device, electronic equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117830728A true CN117830728A (en) | 2024-04-05 |
Family
ID=90520579
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410003474.5A Pending CN117830728A (en) | 2024-01-02 | 2024-01-02 | Abnormal flow detection method, device, electronic equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117830728A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200065616A1 (en) * | 2017-10-30 | 2020-02-27 | Tsinghua University | Unsupervised exception access detection method and apparatus based on one-hot encoding mechanism |
| CN111585997A (en) * | 2020-04-27 | 2020-08-25 | 国家计算机网络与信息安全管理中心 | Network flow abnormity detection method based on small amount of labeled data |
| CN112953924A (en) * | 2021-02-04 | 2021-06-11 | 西安电子科技大学 | Network abnormal flow detection method, system, storage medium, terminal and application |
| WO2021114231A1 (en) * | 2019-12-11 | 2021-06-17 | 中国科学院深圳先进技术研究院 | Training method and detection method for network traffic anomaly detection model |
| CN116304804A (en) * | 2023-02-20 | 2023-06-23 | 中国电子科技集团公司第三十研究所 | Network abnormal flow detection method and device |
| CN116347492A (en) * | 2023-04-27 | 2023-06-27 | 国网智能电网研究院有限公司 | 5G slice flow abnormality detection method, device, computer equipment and storage medium |
| CN116743633A (en) * | 2023-05-08 | 2023-09-12 | 天津大学 | Method for detecting abnormal flow of Internet of things based on attention mechanism |
-
2024
- 2024-01-02 CN CN202410003474.5A patent/CN117830728A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200065616A1 (en) * | 2017-10-30 | 2020-02-27 | Tsinghua University | Unsupervised exception access detection method and apparatus based on one-hot encoding mechanism |
| WO2021114231A1 (en) * | 2019-12-11 | 2021-06-17 | 中国科学院深圳先进技术研究院 | Training method and detection method for network traffic anomaly detection model |
| CN111585997A (en) * | 2020-04-27 | 2020-08-25 | 国家计算机网络与信息安全管理中心 | Network flow abnormity detection method based on small amount of labeled data |
| CN112953924A (en) * | 2021-02-04 | 2021-06-11 | 西安电子科技大学 | Network abnormal flow detection method, system, storage medium, terminal and application |
| CN116304804A (en) * | 2023-02-20 | 2023-06-23 | 中国电子科技集团公司第三十研究所 | Network abnormal flow detection method and device |
| CN116347492A (en) * | 2023-04-27 | 2023-06-27 | 国网智能电网研究院有限公司 | 5G slice flow abnormality detection method, device, computer equipment and storage medium |
| CN116743633A (en) * | 2023-05-08 | 2023-09-12 | 天津大学 | Method for detecting abnormal flow of Internet of things based on attention mechanism |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109583501B (en) | Method, device, equipment and medium for generating image classification and classification recognition model | |
| CN111027576B (en) | Cooperative significance detection method based on cooperative significance generation type countermeasure network | |
| CN114549913B (en) | Semantic segmentation method and device, computer equipment and storage medium | |
| JP7139749B2 (en) | Image recognition learning device, image recognition device, method, and program | |
| TWI803243B (en) | Method for expanding images, computer device and storage medium | |
| CN115761366A (en) | Zero sample picture classification method, system, device and medium for supplementing missing features | |
| CN111160526A (en) | Online testing method and device for deep learning system based on MAPE-D annular structure | |
| CN111753729B (en) | False face detection method and device, electronic equipment and storage medium | |
| CN115049852B (en) | Bearing fault diagnosis method and device, storage medium and electronic equipment | |
| CN114155388B (en) | Image recognition method and device, computer equipment and storage medium | |
| CN117830728A (en) | Abnormal flow detection method, device, electronic equipment and medium | |
| CN117011219A (en) | Method, apparatus, device, storage medium and program product for detecting quality of article | |
| Shah et al. | Knowledge-based analogical reasoning in neuro-symbolic latent spaces | |
| CN112634869B (en) | Command word recognition method, device and computer storage medium | |
| CN117523218A (en) | Label generation, training of image classification model and image classification method and device | |
| CN114510592A (en) | Image classification method and device, electronic equipment and storage medium | |
| CN116912920B (en) | Expression recognition method and device | |
| CN114708467B (en) | Bad scene identification method, system and equipment based on knowledge distillation | |
| CN116704244A (en) | Course domain schematic diagram object detection method, system, equipment and storage medium | |
| CN113987488A (en) | Malicious software detection method and system based on memristive neural network | |
| CN116824194A (en) | Training method of image classification model, image processing method and device | |
| CN117456219A (en) | Training method of image classification model, image classification method and related equipment | |
| CN117576504A (en) | Training method of social media false news detection model | |
| Richards et al. | Deep Fake Face Detection using Convolutional Neural Networks | |
| Gerdprasert et al. | Disease Area Detection for Chest X‐Ray Image Diagnosis Using Deep Learning with Pseudo Labeling and Ensemble Learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |