CN109728939B - Network flow detection method and device - Google Patents
Network flow detection method and device Download PDFInfo
- Publication number
- CN109728939B CN109728939B CN201811526712.1A CN201811526712A CN109728939B CN 109728939 B CN109728939 B CN 109728939B CN 201811526712 A CN201811526712 A CN 201811526712A CN 109728939 B CN109728939 B CN 109728939B
- Authority
- CN
- China
- Prior art keywords
- flow
- network
- network flow
- model
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
A network traffic detection method and device are disclosed. A network flow detection method is applied to a model training side of a network flow detection system, and comprises the following steps: obtaining a plurality of network flow samples; extracting a plurality of preset optional features from a network flow sample, inputting the extracted optional features into a preset feature selection engine, and determining a plurality of key features according to the output of the engine; sending a key characteristic determining instruction to a flow detection side; determining the label of each network flow sample, and training a network flow detection model through a supervised learning algorithm according to the labels and key characteristics of the plurality of network flow samples; and updating the model at the traffic detection side according to the trained network traffic detection model.
Description
Technical Field
The embodiment of the specification relates to the technical field of network communication, in particular to a network traffic detection method and device.
Background
The network intrusion detection system can detect whether the network traffic entering the system is an attack or an abnormal traffic invading the system according to the characteristics of the network traffic.
In the prior art, one detection scheme is based on a deep learning technology, adopts technologies such as a convolutional neural network and a cyclic neural network, and distinguishes normal network traffic from abnormal network traffic, and has high detection precision due to high dimensionality of features extracted by a deep model and large depth of the neural network, but has high consumption of calculation and storage resources and high requirements on equipment in practical application; the other detection scheme is based on a shallow neural network or a data mining model, and although the requirement on equipment is reduced, the scheme has high relative false alarm rate and low detection precision.
Disclosure of Invention
In view of this, embodiments of the present specification provide a method and an apparatus for detecting network traffic, where the technical scheme is as follows:
a network flow detection method is applied to a model training side of a network flow detection system, and comprises the following steps:
obtaining a plurality of network flow samples;
extracting a plurality of preset optional features from a network flow sample, inputting the extracted optional features into a preset feature selection engine, and determining a plurality of key features according to the output of the engine;
sending a key feature determination instruction to a traffic detection side so that the traffic detection side determines features for extracting network traffic to be detected according to the instruction;
determining a label of each network flow sample, wherein the label is used for indicating whether the network flow of the corresponding sample is abnormal or not, and training a network flow detection model through a supervised learning algorithm according to the labels and key characteristics of the plurality of network flow samples;
and updating the model on the traffic detection side according to the trained network traffic detection model so that the traffic detection side can predict whether the network traffic to be detected is abnormal or not based on the updated model.
A network flow detection device is applied to a model training side of a network flow detection system, and the device comprises:
the system comprises a sample obtaining module, a data processing module and a data processing module, wherein the sample obtaining module is used for obtaining a plurality of network flow samples;
the system comprises a characteristic determining module, a characteristic selecting engine and a characteristic selecting module, wherein the characteristic determining module is used for extracting a plurality of preset alternative characteristics from a network flow sample, inputting the extracted alternative characteristics into the preset characteristic selecting engine, and determining a plurality of key characteristics according to the output of the engine;
the characteristic updating module is used for sending a key characteristic determining instruction to the flow detection side so that the flow detection side can determine the characteristic extracted from the network flow to be detected according to the instruction;
the model training module is used for determining a label of each network flow sample, wherein the label is used for indicating whether the network flow of the corresponding sample is abnormal or not, and training a network flow detection model through a supervised learning algorithm according to the labels and key characteristics of the plurality of network flow samples;
and the model updating module is used for updating the model on the traffic detection side according to the trained network traffic detection model so as to enable the traffic detection side to predict whether the network traffic to be detected is abnormal or not based on the updated model.
According to the technical scheme provided by the embodiment of the specification, model training and flow detection in a network flow detection system are separated and divided into 2 asynchronously-executed subsystems, a model training side screens a plurality of important features in advance, other features with small functions are eliminated, and a flow detection side performs flow detection by using the trained models and the screened features, so that the requirement on equipment is reduced, and meanwhile, high detection precision is achieved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of embodiments of the invention.
In addition, any one of the embodiments in the present specification is not required to achieve all of the effects described above.
Drawings
In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present specification, and other drawings can be obtained by those skilled in the art according to the drawings.
Fig. 1 is a schematic structural diagram of a network traffic detection system according to an embodiment of the present disclosure;
fig. 2 is a schematic flow chart of a network traffic detection method according to an embodiment of the present disclosure;
FIG. 3 is a schematic diagram of the operation of a feature selection engine according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram illustrating a training process of a network traffic detection model according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of a network traffic detection system according to an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of a network traffic detection apparatus according to an embodiment of the present disclosure;
fig. 7 is another schematic structural diagram of a network traffic detection device according to an embodiment of the present disclosure.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the embodiments of the present specification, the technical solutions in the embodiments of the present specification will be described in detail below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only a part of the embodiments of the present specification, and not all the embodiments. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of protection.
To solve the problems in the prior art, the present specification provides a network traffic detection scheme, which is applied to a network traffic detection system.
The network monitoring process in the embodiment of the present specification relates to a model training side and a traffic detection side, and a corresponding system architecture schematic diagram is shown in fig. 1 and includes a model training device 10 and a traffic detection device 20.
Fig. 2 is an interaction flowchart of a network traffic detection scheme provided in an embodiment of the present specification, which may specifically include the following steps:
s101, a model training side obtains a plurality of network flow samples; extracting a plurality of preset optional features from a network flow sample, inputting the extracted optional features into a preset feature selection engine, and determining a plurality of key features according to the output of the engine;
in the embodiments of the present description, the key features may be specifically screened out from the alternative features in various ways.
In a specific embodiment of the present specification, the extracted several candidate features may be first input into a preset feature selection engine, to obtain the weight of each candidate feature calculated by the engine, and then, according to the calculated weight, several features with the highest weight may be determined as the key features.
When the weight of each candidate feature is obtained by the feature selection engine, specifically, the engine may include a self-encoder, the extracted several candidate features are input to a preset feature selection engine, a reconstructed flow feature value of the network flow sample is calculated by a self-encoder in the engine, then an original flow feature value of the network flow sample is determined, and the weight corresponding to each candidate feature is obtained when a mean square error between the reconstructed flow feature value and the original flow feature value is minimum.
As an example, a schematic diagram of the operation of the feature selection engine may be as shown in FIG. 3. The main part of the feature selection engine is a weighting self-encoder, and the self-encoder calculates the weight of each feature aiming at the flow detection scene that the normal flow quantity applied by the scheme is larger than the abnormal flow quantity.
As shown in fig. 3, the weighted self-encoder performs weighting processing on the loss function based on the denoising self-encoder, and this processing makes the weighted self-encoder focus more on the abnormal traffic sample, so that a feature that can more characterize the abnormal traffic can be selected during feature selection.
The training goal of the self-encoder is to reconstruct a value close to the original sample for the corrupted sample through the encoding-decoding process of the self-encoder. The process of corruption is often gaussian noise added to the original sample, and can be defined as equation (1) as follows:
in training, the mean square error of the reconstructed flow characteristic value and the original flow characteristic value of the flow sample needs to be minimized, which can be defined as the following formula (2):
wherein, WEIs the weight of the part encoded from the encoder, and bEFor the bias values from the encoder encoding part, the sig (x) function is a Sigmoid activation function. And, WERow vector w of ith row in matrixiL2 norm value | | w ofi||2The importance degree of the ith feature is represented, and the feature selection task needs to select and output the largest plurality of | | wi||2。
In order to make the difference between the row vectors larger to achieve better feature selection effect, it is often necessary to constrain WETo make it more sparse. Here, the l2,1 norm constraint is adopted, i.e. the metric is added to the loss functionEquation (3) is a regularization term:
wherein, WijIs WEThe value of the ith row and the jth column in the middle row and alpha are regular coefficients and can be adjusted according to actual needs.
In order to improve the effect of feature selection, labels of training data are also used in the weighting self-encoder, and the mean square error calculated by the formula (2) is weighted, so that the feature selection engine is more concerned with the reconstruction process of the abnormal flow sample, and therefore features capable of accurately describing the characteristics of the abnormal flow are more prone to be selected in the selection result. The training goal of the final weighted self-encoder is to minimize the loss function as shown in equation (4) below:
wherein, WLThe weight matrix of the loss function is calculated by the following formula (5):
WL＝(β-1)·Y+1 (5)
where Y is the label matrix of the samples and β is the value of the weight coefficient applied to the abnormal traffic samples. 1 is a matrix value of all 1's. In training to LWDAEUntil convergence, W can be comparedEL2 norm | | w of each row vector ini||2The most important features are selected.
S102, a model training side sends a key feature determination instruction to a traffic detection side so that the traffic detection side determines features for extracting network traffic to be detected according to the instruction;
s103, receiving the network flow to be detected by a flow detection side; extracting a plurality of key features of the received network traffic to be detected, wherein the key features are determined according to a key feature determination instruction at a model training side;
for convenience of description, S102 and S103 are explained in combination.
After the model training side selects the key features from the alternative features, the traffic detection side can be informed that the key features of the network traffic to be detected are extracted only when the traffic detection side detects the network traffic. Correspondingly, after receiving the instruction of the model training side, the traffic detection side extracts a plurality of key features specified in the instruction under the condition of receiving the network traffic to be detected.
The characteristics of the network traffic are not always unchanged, so the model training side can update the key characteristics along with the time, specifically, the key characteristics can be repeatedly screened by the characteristic selection engine, and the traffic detection side is informed of new key characteristics needing to be extracted. The update may set specific trigger conditions, for example, to perform the update periodically, may be triggered by maintenance personnel, and/or may perform critical feature updates in the event that a certain amount of network traffic samples accumulate.
S104, determining a label of each network flow sample by a model training side, wherein the label is used for indicating whether the network flow of the corresponding sample is abnormal or not, and training a network flow detection model by a supervised learning algorithm according to the labels and key characteristics of the plurality of network flow samples;
the model training side screens key features through a feature selection engine, so that the flow detection side only extracts the key features of the network flow to be detected, consumed resources are reduced, and detection accuracy is guaranteed. In addition, the model training side needs to train the network traffic detection model.
In the solution provided in the embodiment of the present specification, the trained and used network traffic detection model may be specifically implemented by various models, which are not limited in the embodiment of the present specification, and a person skilled in the art may select a suitable model according to actual requirements.
In a specific implementation manner of the embodiments of the present specification, the network traffic detection model may be a compact multi-layer perceptron C-MLP, where a multi-layer perceptron (MLP) is a deep learning algorithm model that includes one or more hidden layers and is densely connected between units of adjacent layers, and may fully utilize all features of input samples to mine potential associations between features, thereby implementing high-performance malicious behavior discrimination. The C-MLP used in the scheme is a compact model realized by reducing the number of hidden layers and the number of hidden units on the basis of MLP, and the C-MLP reduces the parameters of the model and the consumption of model resources while keeping the high performance advantage of MLP, thereby reducing the resources required by the detection process and ensuring higher accuracy.
The process of training the compact multi-layered perceptron C-MLP can be as shown in FIG. 4.
Firstly, a deep MLP model with a large number of hidden units needs to be established, training is carried out by using training data, and performance test is carried out to obtain the standard performance of the model.
Then, based on the MLP model obtained by the training, the number of hidden layers can be reduced, and the number of hidden units in each layer can be reduced. After each reduction, model training and performance testing can be carried out, and the performance of the reduced model is still above the reference performance.
When the number of hidden layers and the number of units in the model are reduced to the minimum and the performance of the model still can reach the benchmark performance, the current model can be determined to be the C-MLP model obtained by training
In the training process of the C-MLP model, in particular, it is necessary to minimize the cross entropy between the output value of the training sample and the sample label value, which can be represented by a loss function as shown in the following equation (6):
wherein, XSFor the training samples after feature selection, Y is the sample label, fC-MLP(x) For the transfer function of C-MLP, H (p, q) is the cross-entropy calculation function, λ · ∑i‖Wi‖2Is a regular term of L2, where λ is a regular coefficient, WiA weight matrix for each hidden layer.
And S105, the model training side updates the model on the traffic detection side according to the trained network traffic detection model, so that the traffic detection side predicts whether the network traffic to be detected is abnormal or not based on the updated model.
S106, inputting the extracted key features into a pre-trained network traffic detection model by a traffic detection side, and determining whether the network traffic to be detected is abnormal or not according to the output of the model; the network flow detection model is trained by a model training side.
For convenience of description, S105 and S106 are explained in combination.
After the model training side trains and obtains the network traffic detection model, the traffic detection side may use the model to perform traffic detection, as described above, since the characteristics of the network traffic are not always unchanged, the model training side may update the model over time, specifically, the update may set a specific trigger condition, for example, to periodically perform the update, may perform the update trigger operation by a maintenance person, and/or perform the key characteristic update when the network traffic sample is accumulated to a certain amount.
In a specific implementation manner of the embodiment of the present specification, an architecture of the network traffic detection system provided in the embodiment of the present specification may be as shown in fig. 5, and the system may specifically include two sets of asynchronous working systems, namely, a model training subsystem deployed at a central office point and a traffic detection subsystem deployed at a branch office point. The traffic detection subsystem can be an intrusion detector which is actually deployed on line at each node in the network, and the number of the traffic detection subsystems can be multiple; the model training subsystem is deployed at a central office point, and all working systems are controlled by the signaling of the training system.
The model training subsystem mainly comprises a feature data collector T, a feature selection engine and a discriminator T. The work flow can be as follows: firstly, a feature data collector T collects a group of preset alternative features from a network flow and forms a training sample set, and the collector can collect as many alternative features as possible. The training data may then be fed into the feature selection engine by manual decision and annotation.
Then, a plurality of key features which can represent the network traffic features are selected by the feature selection engine and transmitted to the data collector W in the traffic detection subsystem, namely, the key features of the traffic detection subsystem are updated. Meanwhile, the data after feature selection is also sent to the discriminator T for training. After the discriminator T receives the sample data, firstly, a relatively complex C-MLP model which can realize the highest abnormal detection precision is trained, and then the model is reduced.
Each flow detection subsystem can be composed of 1 characteristic data collector W, 1 discriminator W and 1 intrusion alarm, and the working flow can be as follows: after receiving the network traffic to be detected, the feature data collector W extracts key features of the network traffic to be detected according to the key features specified by the model training subsystem, and transmits the key features to the discriminator W. And the discriminator W is a copy version of the discriminator T in the training system and is used for discriminating whether the flow in the network is malicious flow or not and transmitting the result to the alarm, and the alarm selects to trigger or not to trigger the intrusion alarm according to the discrimination result sent by the discriminator W.
Corresponding to the above method embodiment, an embodiment of the present specification further provides a network traffic detection apparatus, which is applied to a model training side of a network traffic detection system, and as shown in fig. 6, the apparatus may include:
a sample obtaining module 110, configured to obtain a plurality of network traffic samples;
the feature determination module 120 is configured to extract a plurality of preset candidate features from the network traffic sample, input the extracted plurality of candidate features into a preset feature selection engine, and determine a plurality of key features according to an output of the preset feature selection engine;
the feature updating module 130 is configured to send a key feature determination instruction to the traffic detection side, so that the traffic detection side determines, according to the instruction, a feature extracted from the network traffic to be detected;
the model training module 140 is configured to determine a label of each network traffic sample, where the label is used to indicate whether network traffic of a corresponding sample is abnormal, and train a network traffic detection model according to the labels and key features of the network traffic samples through a supervised learning algorithm;
and the model updating module 150 is configured to update the model on the traffic detection side according to the trained network traffic detection model, so that the traffic detection side predicts whether the network traffic to be detected is abnormal based on the updated model.
An embodiment of the present disclosure further provides a network traffic detection apparatus, which is applied to a traffic detection side of a network traffic detection system, and as shown in fig. 7, the apparatus includes:
a traffic receiving module 210, configured to receive network traffic to be detected;
the feature extraction module 220 is configured to extract a plurality of key features of the received network traffic to be detected, where the key features are determined according to a key feature determination instruction at a model training side;
the traffic detection module 230 is configured to input the extracted key features into a pre-trained network traffic detection model, and determine whether the network traffic to be detected is abnormal according to the output of the model; the network flow detection model is trained by a model training side.
The implementation process of the functions and actions of each unit in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the embodiments of the present specification. One of ordinary skill in the art can understand and implement it without inventive effort.
While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. In other instances, features described in connection with one embodiment may be implemented as discrete components or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Further, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.
The above description is only a preferred embodiment of the present disclosure, and should not be taken as limiting the present disclosure, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (10)
1. A network flow detection method is applied to a model training side of a network flow detection system, and comprises the following steps:
obtaining a plurality of network flow samples;
extracting a plurality of preset optional features from a network flow sample, inputting the extracted optional features into a preset feature selection engine, and determining a plurality of key features according to the output of the engine; the engine comprises a weighted self-encoder, wherein the weighted self-encoder is used for converting an original flow characteristic value of an input sample into a reconstructed flow characteristic value during training, and the loss function comprises a weighted calculation value of a mean square error of the reconstructed flow characteristic value and the original flow characteristic value according to a sample label, so that the weighted self-encoder is more concerned with the reconstruction process of an abnormal flow sample;
sending a key feature determination instruction to a traffic detection side so that the traffic detection side determines features for extracting network traffic to be detected according to the instruction;
determining a label of each network flow sample, wherein the label is used for indicating whether the network flow of the corresponding sample is abnormal or not, and training a network flow detection model through a supervised learning algorithm according to the labels and key characteristics of the plurality of network flow samples;
and updating the model on the traffic detection side according to the trained network traffic detection model so that the traffic detection side can predict whether the network traffic to be detected is abnormal or not based on the updated model.
2. The method of claim 1, wherein the inputting the extracted candidate features into a preset feature selection engine, and determining key features according to the output of the engine comprises:
inputting the extracted multiple candidate features into a preset feature selection engine to obtain the weight of each candidate feature calculated by the engine;
and determining a plurality of characteristics with the highest weight as key characteristics according to the calculated weight.
3. The method according to claim 2, wherein the inputting the extracted several candidate features into a preset feature selection engine to obtain the weight of each candidate feature calculated by the engine comprises:
inputting the extracted multiple candidate features into a preset feature selection engine, and calculating a reconstructed flow feature value of the network flow sample through a self-encoder in the engine;
and determining an original flow characteristic value of the network flow sample, and obtaining the weight corresponding to each alternative characteristic under the condition that the mean square error of the reconstructed flow characteristic value and the original flow characteristic value is minimum.
4. A network flow detection method is applied to a flow detection side of a network flow detection system, and comprises the following steps:
receiving network traffic to be detected;
extracting a plurality of key features of the received network traffic to be detected, wherein the key features are determined according to a key feature determination instruction at a model training side;
inputting the extracted key features into a pre-trained network flow detection model, and determining whether the network flow to be detected is abnormal or not according to the output of the model; the network flow detection model is trained by a model training side;
the network flow detection model is trained by a model training side through the following steps:
obtaining a plurality of network flow samples;
extracting a plurality of preset optional features from a network flow sample, inputting the extracted optional features into a preset feature selection engine, and determining a plurality of key features according to the output of the engine; the engine comprises a weighted self-encoder, wherein the weighted self-encoder is used for converting an original flow characteristic value of an input sample into a reconstructed flow characteristic value during training, and the loss function comprises a weighted calculation value of a mean square error of the reconstructed flow characteristic value and the original flow characteristic value according to a sample label, so that the weighted self-encoder is more concerned with the reconstruction process of an abnormal flow sample;
sending a key feature determination instruction to a traffic detection side so that the traffic detection side determines features for extracting network traffic to be detected according to the instruction;
determining a label of each network flow sample, wherein the label is used for indicating whether the network flow of the corresponding sample is abnormal or not, and training a network flow detection model through a supervised learning algorithm according to the labels and key characteristics of the plurality of network flow samples;
and updating the model on the traffic detection side according to the trained network traffic detection model so that the traffic detection side can predict whether the network traffic to be detected is abnormal or not based on the updated model.
5. The method of claim 4, wherein the network traffic detection model is a compact multi-tier perceptron C-MLP.
6. A network flow detection device is applied to a model training side of a network flow detection system, and the device comprises:
the system comprises a sample obtaining module, a data processing module and a data processing module, wherein the sample obtaining module is used for obtaining a plurality of network flow samples;
the system comprises a characteristic determining module, a characteristic selecting engine and a characteristic selecting module, wherein the characteristic determining module is used for extracting a plurality of preset alternative characteristics from a network flow sample, inputting the extracted alternative characteristics into the preset characteristic selecting engine, and determining a plurality of key characteristics according to the output of the engine; the engine comprises a weighted self-encoder, wherein the weighted self-encoder is used for converting an original flow characteristic value of an input sample into a reconstructed flow characteristic value during training, and the loss function comprises a weighted calculation value of a mean square error of the reconstructed flow characteristic value and the original flow characteristic value according to a sample label, so that the weighted self-encoder is more concerned with the reconstruction process of an abnormal flow sample;
the characteristic updating module is used for sending a key characteristic determining instruction to the flow detection side so that the flow detection side can determine the characteristic extracted from the network flow to be detected according to the instruction;
the model training module is used for determining a label of each network flow sample, wherein the label is used for indicating whether the network flow of the corresponding sample is abnormal or not, and training a network flow detection model through a supervised learning algorithm according to the labels and key characteristics of the plurality of network flow samples;
and the model updating module is used for updating the model on the traffic detection side according to the trained network traffic detection model so as to enable the traffic detection side to predict whether the network traffic to be detected is abnormal or not based on the updated model.
7. The apparatus of claim 6, wherein the feature determination module comprises:
the weight calculation unit is used for inputting the extracted multiple alternative features into a preset feature selection engine to obtain the weight of each alternative feature calculated by the engine;
and the feature screening unit is used for determining a plurality of features with the highest weights as key features according to the calculated weights.
8. The apparatus according to claim 7, wherein the weight calculation unit is specifically configured to:
inputting the extracted multiple candidate features into a preset feature selection engine, and calculating a reconstructed flow feature value of the network flow sample through a self-encoder in the engine;
and determining an original flow characteristic value of the network flow sample, and obtaining the weight corresponding to each alternative characteristic under the condition that the mean square error of the reconstructed flow characteristic value and the original flow characteristic value is minimum.
9. A network flow detection device is applied to a flow detection side of a network flow detection system, and the device comprises:
the flow receiving module is used for receiving the network flow to be detected;
the characteristic extraction module is used for extracting a plurality of received key characteristics of the network flow to be detected, and the key characteristics are determined according to a key characteristic determination instruction at a model training side;
the flow detection module is used for inputting the extracted key features into a pre-trained network flow detection model and determining whether the network flow to be detected is abnormal or not according to the output of the model; the network flow detection model is trained by a model training side;
the network flow detection model is trained by a model training side through the following steps:
obtaining a plurality of network flow samples;
extracting a plurality of preset optional features from a network flow sample, inputting the extracted optional features into a preset feature selection engine, and determining a plurality of key features according to the output of the engine; the engine comprises a weighted self-encoder, wherein the weighted self-encoder is used for converting an original flow characteristic value of an input sample into a reconstructed flow characteristic value during training, and the loss function comprises a weighted calculation value of a mean square error of the reconstructed flow characteristic value and the original flow characteristic value according to a sample label, so that the weighted self-encoder is more concerned with the reconstruction process of an abnormal flow sample;
sending a key feature determination instruction to a traffic detection side so that the traffic detection side determines features for extracting network traffic to be detected according to the instruction;
determining a label of each network flow sample, wherein the label is used for indicating whether the network flow of the corresponding sample is abnormal or not, and training a network flow detection model through a supervised learning algorithm according to the labels and key characteristics of the plurality of network flow samples;
and updating the model on the traffic detection side according to the trained network traffic detection model so that the traffic detection side can predict whether the network traffic to be detected is abnormal or not based on the updated model.
10. The apparatus of claim 9, wherein the network traffic detection model is a compact multi-tier perceptron C-MLP.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811526712.1A CN109728939B (en) | 2018-12-13 | 2018-12-13 | Network flow detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811526712.1A CN109728939B (en) | 2018-12-13 | 2018-12-13 | Network flow detection method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109728939A CN109728939A (en) | 2019-05-07 |
| CN109728939B true CN109728939B (en) | 2022-04-26 |
Family
ID=66294896
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811526712.1A Active CN109728939B (en) | 2018-12-13 | 2018-12-13 | Network flow detection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109728939B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110474815B (en) * | 2019-09-23 | 2021-08-13 | 北京达佳互联信息技术有限公司 | Bandwidth prediction method and device, electronic equipment and storage medium |
| CN111064721A (en) * | 2019-12-11 | 2020-04-24 | 中国科学院深圳先进技术研究院 | Training method and detection method of network flow abnormity detection model |
| CN111294341B (en) * | 2020-01-17 | 2021-12-28 | 成都信息工程大学 | Vehicle-mounted system intrusion detection method based on self-encoder and recurrent neural network |
| CN111401447B (en) * | 2020-03-16 | 2023-04-07 | 腾讯云计算（北京）有限责任公司 | Artificial intelligence-based flow cheating identification method and device and electronic equipment |
| CN111582016A (en) * | 2020-03-18 | 2020-08-25 | 宁波送变电建设有限公司永耀科技分公司 | Intelligent maintenance-free power grid monitoring method and system based on cloud edge collaborative deep learning |
| CN112668699B (en) * | 2020-12-30 | 2022-06-17 | 燕山大学 | Rolling force prediction method and device |
| CN113343244B (en) * | 2021-05-18 | 2024-03-19 | 广东电网有限责任公司 | Method and device for predicting health degree of electric power information system based on probe injection |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102685016B (en) * | 2012-06-06 | 2015-01-07 | 济南大学 | Internet flow distinguishing method |
| US10187413B2 (en) * | 2016-03-25 | 2019-01-22 | Cisco Technology, Inc. | Network-based approach for training supervised learning classifiers |
| CN106817248B (en) * | 2016-12-19 | 2020-10-16 | 西安电子科技大学 | APT attack detection method |
| US10205634B2 (en) * | 2017-05-12 | 2019-02-12 | Salesforce.Com, Inc. | Adaptive multi-phase network policy optimization |
| CN108881192B (en) * | 2018-06-04 | 2021-10-22 | 上海交通大学 | Encryption type botnet detection system and method based on deep learning |
| CN108900556B (en) * | 2018-08-24 | 2021-02-02 | 海南大学 | DDoS attack detection method based on HMM and chaotic model |
-
2018
- 2018-12-13 CN CN201811526712.1A patent/CN109728939B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109728939A (en) | 2019-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109728939B (en) | Network flow detection method and device | |
| CN111914873A (en) | Two-stage cloud server unsupervised anomaly prediction method | |
| US8510234B2 (en) | Embedded health monitoring system based upon Optimized Neuro Genetic Fast Estimator (ONGFE) | |
| US11120127B2 (en) | Reconstruction-based anomaly detection | |
| US20210116076A1 (en) | Anomaly detection in pipelines and flowlines | |
| EP3087445A1 (en) | Systems and methods for event detection and diagnosis | |
| US20190163552A1 (en) | System and method for contextual event sequence analysis | |
| CN112416643A (en) | Unsupervised anomaly detection method and unsupervised anomaly detection device | |
| CN115013298B (en) | Real-time performance online monitoring system and monitoring method of sewage pump | |
| CN111858526A (en) | Failure time space prediction method and system based on information system log | |
| CN117421684B (en) | Abnormal data monitoring and analyzing method based on data mining and neural network | |
| CN112416662A (en) | Multi-time series data anomaly detection method and device | |
| CN114842371B (en) | Unsupervised video anomaly detection method | |
| CN114528190B (en) | Single index abnormality detection method and device, electronic equipment and readable storage medium | |
| CN116684878A (en) | 5G information transmission data safety monitoring system | |
| CN114004331A (en) | Fault analysis method based on key indexes and deep learning | |
| CN117892921A (en) | Intelligent water affair comprehensive management system and method based on big data | |
| CN114048546A (en) | Graph convolution network and unsupervised domain self-adaptive prediction method for residual service life of aircraft engine | |
| CN110033105B (en) | Suspension system fault detection method for unbalanced data set under complex working condition | |
| JP6858798B2 (en) | Feature generator, feature generator and program | |
| CN115392381A (en) | Unscented Kalman filtering-based time series anomaly detection method | |
| CN115473748A (en) | DDoS attack classification detection method, device and equipment based on BiLSTM-ELM | |
| CN114819350A (en) | RUL prediction method under sensor fault condition based on multiple tasks | |
| CN113986673A (en) | Fault correlation method and device and computer readable storage medium | |
| EP4193259A1 (en) | Self-diagnosis for in-vehicle networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |