CN112671768A - Abnormal flow detection method and device, electronic equipment and storage medium - Google Patents
Abnormal flow detection method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112671768A CN112671768A CN202011545306.7A CN202011545306A CN112671768A CN 112671768 A CN112671768 A CN 112671768A CN 202011545306 A CN202011545306 A CN 202011545306A CN 112671768 A CN112671768 A CN 112671768A
- Authority
- CN
- China
- Prior art keywords
- data
- vector
- flow
- word segmentation
- abnormal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application provides an abnormal flow detection method, an abnormal flow detection device, an electronic device and a storage medium, wherein the method comprises the following steps: acquiring flow data, and performing field extraction on the flow data according to the protocol type of the flow data to obtain field data; segmenting word data to obtain segmented words, and vectorizing the segmented words to obtain segmented word vectors; reconstructing the word segmentation vector by using a pre-trained unsupervised neural network model to obtain a reconstructed vector; judging whether the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold value or not; and if so, determining the flow data as the abnormal flow. In the implementation process, the unsupervised neural network model is used for detecting the flow data, so that the probability of false alarm of the flow data can be effectively reduced, the condition of false alarm abnormal detection results is also reduced, and the accuracy of the flow data detection is improved.
Description
Technical Field
The application relates to the technical field of machine learning, supervised learning and network security, in particular to an abnormal traffic detection method and device, electronic equipment and a storage medium.
Background
Supervised learning (also called Supervised learning, Supervised training, or Supervised training) is a learning paradigm of machine learning, and a learning mode (learning model) or learning function can be learned or established from training data, and a new instance can be inferred according to the mode; training data is composed of input data (usually vectors) and data labels of the input data corresponding to expected outputs, and a supervised learner's task, after observing some previously labeled training paradigms (data labels of input data and expected outputs), predicts the output of this function to any input that may occur.
At present, most of abnormal detection aiming at flow data is based on supervised learning, namely a supervised neural network model is used for predicting the flow data so as to determine whether the flow data is abnormal flow; specific examples thereof include: analyzing data records formed by a source Internet Protocol (IP) address, a destination IP address, a source port, a destination port and a Protocol group from flow data passing through a network firewall, extracting data characteristics from the data records by using a characteristic extraction network in a supervised neural network model, and classifying the data characteristics by using a classifier in the supervised neural network model so as to obtain a classification result, wherein the classification result comprises: abnormal traffic and normal traffic. However, in the actual process, it is found that the traffic data is easy to change along with the environment, and the situation that false alarm occurs frequently when the traffic data is detected by using the supervised neural network model. Therefore, the prior art has the problem that the accuracy rate of detecting the flow data by using the supervised neural network model is not high.
Disclosure of Invention
An object of the embodiments of the present application is to provide an abnormal traffic detection method, an abnormal traffic detection device, an electronic device, and a storage medium, which are used to solve the problem of low accuracy in detecting traffic data.
The embodiment of the application provides an abnormal flow detection method, which comprises the following steps: acquiring flow data, and performing field extraction on the flow data according to the protocol type of the flow data to obtain field data; segmenting word data to obtain segmented words, and vectorizing the segmented words to obtain segmented word vectors; reconstructing the word segmentation vector by using a pre-trained unsupervised neural network model to obtain a reconstructed vector; judging whether the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold value or not; and if so, determining the flow data as the abnormal flow. In the implementation process, the supervised neural network model is not sensitive enough to the data characteristic change, and the unsupervised neural network model is more sensitive to the data characteristic change, so that the flow data is detected by using the unsupervised neural network model, the probability of false alarm of the flow data can be effectively reduced, the condition of abnormal detection results of the false alarm is also reduced, and the accuracy of the flow data detection is improved.
Optionally, in an embodiment of the present application, the unsupervised neural network model includes: an encoder, a hidden layer, and a decoder; reconstructing the word segmentation vectors by using a pre-trained unsupervised neural network model, wherein the method comprises the following steps: performing encoding operation on the word segmentation vectors by using an encoder to obtain encoding vectors; randomly sampling the coding vector by using a hidden layer to obtain a sampling vector; and performing decoding operation on the sampling vector by using a decoder to obtain a reconstructed vector. In the implementation process, the word segmentation vector is reconstructed by using the unsupervised neural network model, and the unsupervised model is more sensitive to the data characteristic change compared with the supervised model, so that the unsupervised neural network model can more easily capture the change of the flow data between the reconstructed vector after reconstruction and the word segmentation vector before reconstruction, the probability of false alarm of flow data detection is reduced, and the accuracy of flow data detection is effectively improved.
Optionally, in this embodiment of the present application, determining the flow data as an abnormal flow includes: judging whether data information corresponding to the flow data is in a white list or not, wherein the flow data is data which is detected and confirmed by a network firewall and has no abnormity; if not, the flow data is sent to the terminal equipment, and the abnormal matching rule in the network firewall is updated according to the data information. In the implementation process, the anomaly detection is carried out on the data which is detected and confirmed by the network firewall to be free from anomaly, so that the possibility that attack data exists in the flow data is reduced; further, by updating the abnormal matching rule in the network firewall according to the data information, the abnormal matching rule in the firewall can be updated in time according to the abnormal detection result, so that the possibility that attack data exists in the flow data is further reduced.
Optionally, in this embodiment of the present application, after sending the traffic data to the terminal device, the method further includes: and after the safety personnel corresponding to the terminal equipment confirm that the flow data is not abnormal flow, adding the data information corresponding to the flow data to a white list. In the implementation process, after the safety personnel corresponding to the terminal equipment confirm that the traffic data is not abnormal traffic, the data information corresponding to the traffic data is added to the white list, so that the white list can be updated in time according to the abnormal detection result, the condition of mistakenly reporting the abnormal detection result is reduced, and the accuracy of detecting the traffic data is effectively improved.
Optionally, in this embodiment of the present application, updating an exception matching rule in a network firewall according to data information includes: and after the security personnel corresponding to the terminal equipment confirm that the flow data is abnormal flow, sending data information to the network firewall so that the network firewall updates the abnormal matching rule. In the implementation process, after the security personnel corresponding to the terminal device confirms that the flow data is abnormal flow, the data information is sent to the network firewall, so that the network firewall updates the abnormal matching rule, the abnormal matching rule in the firewall can be updated in time according to the abnormal detection result, and the possibility that attack data exists in the flow data is reduced.
Optionally, in this embodiment of the application, after determining the flow data as the abnormal flow, the method further includes: calculating the accuracy and the false alarm rate of the historical reference data according to the historical reference data of the flow data and the abnormal result corresponding to the historical reference data; and if the accuracy rate is smaller than the first preset proportion or the false alarm rate is larger than the second preset proportion, training the unsupervised neural network model again. In the implementation process, if the accuracy rate is smaller than the first preset proportion, or the false alarm rate is larger than the second preset proportion, the unsupervised neural network model is trained again, so that the condition that the flow data is detected abnormally by the unsupervised neural network model and the concept drift occurs is reduced, and the possibility that attack data exists in the flow data is reduced.
Optionally, in an embodiment of the present application, the training of the unsupervised neural network model includes: performing field extraction and word segmentation on the historical reference data to obtain a field word segmentation result; encoding the field word segmentation result by using an encoder in an unsupervised neural network model to obtain an encoding result; randomly sampling the coding result by using a hidden layer in an unsupervised neural network model to obtain a sampling result; decoding the sampling result by using a decoder in the unsupervised neural network model to obtain a decoding result; and calculating a loss value between the encoding result and the decoding result, and updating the parameters in the encoder and the parameters in the decoder according to the loss value.
The embodiment of the present application further provides an abnormal flow detection device, including: the field data acquisition module is used for acquiring the flow data and extracting the field of the flow data according to the protocol type of the flow data to acquire field data; the word segmentation vector obtaining module is used for performing word segmentation on the field data to obtain words after word segmentation, and performing vectorization on the words after word segmentation to obtain word segmentation vectors; the device comprises a reconstruction vector obtaining module, a word segmentation module and a word segmentation module, wherein the reconstruction vector obtaining module is used for reconstructing a word segmentation vector by using a pre-trained unsupervised neural network model to obtain a reconstruction vector; the vector similarity judging module is used for judging whether the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold value or not; and the abnormal flow determining module is used for determining the flow data as the abnormal flow if the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold.
Optionally, in an embodiment of the present application, the unsupervised neural network model includes: an encoder, a hidden layer, and a decoder; a reconstruction vector obtaining module comprising: the vector coding operation module is used for carrying out coding operation on the word segmentation vectors by using an encoder to obtain coding vectors; the coding vector sampling module is used for randomly sampling the coding vector by using the hidden layer to obtain a sampling vector; and the vector decoding operation module is used for performing decoding operation on the sampling vector by using a decoder to obtain a reconstructed vector.
Optionally, in an embodiment of the present application, the abnormal traffic determining module includes: the data information judging module is used for judging whether the data information corresponding to the flow data is in a white list or not, wherein the flow data is data which is detected by a network firewall and is confirmed to be free from abnormity; and the matching rule updating module is used for sending the flow data to the terminal equipment if the data information corresponding to the flow data is not in the white list, and updating the abnormal matching rule in the network firewall according to the data information.
Optionally, in this embodiment of the present application, the abnormal flow rate detecting device further includes: and the data information adding module is used for adding the data information corresponding to the flow data to the white list after the safety personnel corresponding to the terminal equipment confirm that the flow data is not abnormal flow.
Optionally, in an embodiment of the present application, the matching rule updating module includes: and the data information sending module is used for sending data information to the network firewall after security personnel corresponding to the terminal equipment confirm that the flow data is abnormal flow, so that the network firewall updates the abnormal matching rule.
Optionally, in this embodiment of the present application, the abnormal flow rate detecting device further includes: the historical data calculation module is used for calculating the accuracy and the false alarm rate of the historical reference data according to the historical reference data of the flow data and the abnormal result corresponding to the historical reference data; and the model retraining module is used for retraining the unsupervised neural network model again if the accuracy rate is smaller than the first preset proportion or the false alarm rate is larger than the second preset proportion.
Optionally, in an embodiment of the present application, the model retraining module includes: the word segmentation result obtaining module is used for carrying out field extraction and word segmentation on the historical reference data to obtain a field word segmentation result; the encoding result obtaining module is used for encoding the field word segmentation result by using an encoder in the unsupervised neural network model to obtain an encoding result; the sampling result obtaining module is used for randomly sampling the coding result by using a hidden layer in the unsupervised neural network model to obtain a sampling result; a decoding result obtaining module, configured to decode the sampling result by using a decoder in the unsupervised neural network model to obtain a decoding result; and the network parameter updating module is used for calculating a loss value between the encoding result and the decoding result and updating the parameters in the encoder and the parameters in the decoder according to the loss value.
An embodiment of the present application further provides an electronic device, including: a processor and a memory, the memory storing processor-executable machine-readable instructions, the machine-readable instructions when executed by the processor performing the method as described above.
Embodiments of the present application also provide a storage medium having a computer program stored thereon, where the computer program is executed by a processor to perform the method as described above.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a schematic flow chart of an abnormal traffic detection method provided in an embodiment of the present application;
FIG. 2 is a schematic diagram illustrating a network structure of a convolutional variational self-encoder according to an embodiment of the present application;
FIG. 3 is a flow chart illustrating the handling of concept drift provided by an embodiment of the present application;
fig. 4 is a schematic structural diagram of an abnormal flow rate detection device provided in the embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.
Detailed Description
The technical solution in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.
Before introducing the abnormal traffic detection method provided in the embodiment of the present application, some concepts related in the embodiment of the present application are introduced:
it should be noted that the abnormal traffic detection method provided in the embodiments of the present application may be executed by an electronic device, where the electronic device refers to a device terminal or a server having a function of executing a computer program, and the server refers to a device providing a computing service through a network, and the server includes, for example: x86 server and non-x 86 server, non-x 86 server includes: mainframe, minicomputer, and UNIX server.
Before introducing the abnormal traffic detection method provided in the embodiment of the present application, an application scenario applicable to the abnormal traffic detection method is introduced, where the application scenario includes but is not limited to: the abnormal flow detection method is used for carrying out abnormal detection on network flow data passing through equipment such as a router, a switch and a hardware firewall, or the abnormal flow detection method is used for enhancing the abnormal detection function of a software firewall or a hardware firewall and improving the accuracy and speed of the abnormal detection on the network flow data, or the abnormal flow detection method is used for increasing the safety protection of an enterprise intranet and the like.
Please refer to fig. 1, which illustrates a flow diagram of an abnormal traffic detection method provided in the embodiment of the present application; the abnormal flow detection method has the main ideas that the supervised neural network model is not sensitive to the characteristic change of the data enough, and the unsupervised neural network model is more sensitive to the characteristic change of the data, so that the flow data is detected by using the unsupervised neural network model, the probability of false alarm of the flow data can be effectively reduced, the condition of false alarm abnormal detection results is also reduced, and the accuracy rate of the flow data detection is improved; the abnormal flow monitoring method may include:
step S110: and acquiring flow data, and performing field extraction on the flow data according to the protocol type of the flow data to acquire field data.
The traffic data refers to network traffic data that needs to be subjected to anomaly detection, and may be traffic data of a hypertext Transfer Protocol (HTTP).
There are many embodiments of acquiring the traffic data in step S110, including:
a first embodiment, which acquires traffic data that has been determined to have no anomaly from a firewall and performs anomaly detection on the traffic data again, includes: carrying out anomaly detection on the flow data by using a hardware firewall or a software firewall to obtain a first detection result; the electronic equipment receives and stores the flow data and a first detection result corresponding to the flow data to the storage equipment, and judges whether the first detection result is abnormal flow; if the first detection result is not abnormal flow, acquiring the flow data from the storage device; if the first detection result is abnormal flow, an alarm signal can be sent to a hardware firewall or a software firewall, or the alarm signal can be directly generated and output.
In a second embodiment, the method for acquiring traffic data from a switch or a router includes: determining a source port needing to be detected in the two-layer switch or the three-layer switch, performing port mirroring on the source port to obtain a target port, and then acquiring flow data from the target port of the two-layer switch or the three-layer switch. Alternatively, the embodiment of acquiring traffic data from the router includes: and screening the flow data passing through the router to obtain screened flow, and carrying out flow mirroring on the screened flow to obtain the flow data needing abnormal detection.
There are many embodiments of extracting the field of the traffic data according to the protocol type of the traffic data in step S110, including but not limited to the following:
the first implementation mode is used for extracting fields of traffic data of an HTTP protocol, and comprises the following steps: extracting fields of the flow data of the HTTP protocol, and extracting the contents of the fields comprises the following steps: request mode, complete request Uniform Resource Locator (URL), request version, host, user agent, client acceptance coding (Accept-encoding), client receiving language (Accept-language), content type, content length, and Cookie value.
In a second embodiment, the field extraction is performed on traffic data of other protocols besides the HTTP protocol, where the other protocols include but are not limited to: file Transfer Protocol (FTP), Simple Mail Transfer Protocol (SMTP), Internet Message Access Protocol (IMAP), Simple Object Access Protocol (SOAP), and the like, and fields to be extracted in these protocols may be different according to different types of specific protocols and may be set according to specific situations.
After step S110, step S120 is performed: and segmenting the field data to obtain segmented words, and vectorizing the segmented words to obtain segmented word vectors.
There are many embodiments of the above step S120, including but not limited to the following:
in a first embodiment, segmenting word data by using a segmentation table constructed by a mechanical segmentation method (i.e. based on a dictionary), the embodiment may include: firstly, adding the < PAD >, < UNK >, < ENCRYPT >, < lowercase letters a to Z, uppercase letters A to Z and all symbols in the field data into a newly-built word bag according to the sequence; each piece of extracted data is split according to symbols (such as #,%,/and the like) in the field data, stop words are removed from the split data, and words with the same root or affix are converted into the same word for replacement. And removing words with too low and too high frequencies according to all the segmented words and the occurrence frequency after the statistical splitting, and adding the remaining high-frequency words into the word bag. Finally, the bag of words with the vocabulary of M is obtained. And (5) respectively assigning numbers to the words in the word bag from 0 to M according to the sequence to obtain a word segmentation table. After the word segmentation table is obtained, splitting each piece of data according to the sequence of the symbols, the encrypted data, the high-frequency words and the letters in the word segmentation table; the method comprises the following steps of performing matching search on encrypted data by using a regular expression, searching for an encrypted sequence with a multiple of 16 bits in the data, and replacing the encrypted sequence with a digital number corresponding to' ENCRYPT >; for the non-encrypted data, a regular expression can be used for searching a number corresponding to the non-encrypted data in the word segmentation table, and if the number cannot be searched, the number is replaced by a corresponding number of "< UNK >"; finally, a word segmentation vector is obtained.
In a second embodiment, the method for segmenting words in field data using a statistical-based segmentation method may include: splitting field data in a character and word form to obtain a participled word (tokenizer), and vectorizing the participled word by using a pre-constructed participle table to obtain a participled vector (tokenized vector); the word segmentation table herein is a word (token) obtained by segmenting the previously collected traffic data, then performing statistics and de-duplication on the word, and assigning a vector (which can also be understood as a number) to each word, so as to obtain a word segmentation table, where the constructed word segmentation table is, for example: { "A":1, "B":2, …, "http":103, … }; the word segmentation process described above can also be understood as a process of converting corresponding characters or words into corresponding numerical numbers through a word segmentation table to form numerical vectors (numeric vectors).
After step S120, step S130 is performed: and reconstructing the word segmentation vector by using a pre-trained unsupervised neural network model to obtain a reconstructed vector.
An Unsupervised Neural Network (Unsupervised Neural Network) refers to an Unsupervised Neural Network for reconstructing Network traffic data, and the Unsupervised Neural Network is also referred to as Unsupervised training or Unsupervised Learning (Unsupervised Learning), and refers to that a data label needing to be trained is not manually labeled, but is a data label automatically constructed in large-scale Unsupervised data, and the automatically constructed data label is used for supervised Learning or training; the unsupervised neural network can specifically adopt a deep neural network such as a convolution variational self-encoder.
Please refer to fig. 2, which illustrates a schematic network structure diagram of a convolutional variational self-encoder according to an embodiment of the present application; the convolution variational self-encoder may include: an encoder, a hidden layer, and a decoder; the specific functions of the encoder, the hidden layer and the decoder are described in detail below, and the implementation of step S130 may include:
step S131: and performing encoding operation on the word segmentation vectors by using an encoder to obtain encoding vectors.
The Encoder (Encoder) may be composed of two residual convolutional network blocks, where the residual convolutional network blocks may be formed by stacking and adding three one-dimensional convolutional layers and an activation function (e.g., ReLu function), and an output of the residual block is formed by adding an output matrix of a first one-dimensional convolutional layer and an output matrix of a third one-dimensional convolutional layer, and inputting the one-dimensional maximum pooling layer, the batch normalization layer, and the ReLu activation layer. The size and parameters of each neural network can be set according to specific situations, for example: the number of filters of the three convolutional layers of the first residual convolutional block is 256; nuclear size specification 5; the number of filters in the three convolutional layers of the second residual convolutional block is 128; the kernel size specification is 5.
The embodiment of step S131 described above is, for example: firstly, converting an input word segmentation vector from a high-dimensional vector to a low-dimensional vector retaining the same information to obtain a converted word segmentation vector; performing coding operation on the converted word segmentation vectors by using a coder to obtain coding vectors; if the data input requirement of the convolution variational self-encoder is fixed, the participle vector can be filled with a preset number at the tail part, so that the participle vector can meet the data input requirement of the convolution variational self-encoder.
Step S132: and randomly sampling the coding vector by using the hidden layer to obtain a sampling vector.
The embodiment of step S132 described above is, for example: since the output of the encoder satisfies the gaussian probability density, the coding vector can be randomly sampled using the hidden layer to obtain a sampling vector.
Step S133: and performing decoding operation on the sampling vector by using a decoder to obtain a reconstructed vector.
The decoder refers to a neural network for performing decoding operation on the sampling vector, and it is understood that the size and parameters of each neural network can be set according to specific situations, and the network structure of the decoder is similar to that of the encoder, except that a convolutional network is used in the encoder, and a deconvolution network is used in the decoder.
The embodiment of step S133 described above includes, for example: after obtaining the sample vector using the hidden layer, a decoder may be used to perform a decoding operation on the sample vector in an attempt to restore the sample vector to an input participle vector, where the decoder may be formed by two residual deconvolution network blocks, where the residual deconvolution network block may be formed by stacking three one-dimensional deconvolution layers plus an activation function (e.g., ReLu function), and the output of the residual block is formed by adding an output matrix of a first one-dimensional deconvolution layer and an output matrix of a third one-dimensional deconvolution layer, and inputting a one-dimensional max pooling layer, a batch normalization layer, and a ReLu activation layer.
After step S130, step S140 is performed: and judging whether the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold value.
The embodiment of step S140 described above is, for example: calculating the similarity between the reconstructed vector and the participle vector, and judging whether the similarity between the reconstructed vector and the participle vector is smaller than a preset threshold value or not; the similarity here can be Cosine (Cosine) Distance, Euclidean Distance (Euclidean Distance), Hamming Distance (Hamming Distance), Information Entropy (Information Entropy), or the like; the preset threshold value here may also be set as the case may be, for example, set to 70%, 80%, or 90%. Wherein, assuming cosine distance as an example, the following formula can be used:
calculating the cosine distance between the reconstructed vector and the participle vector; wherein S isiRepresenting the cosine distance similarity, x, of the ith reconstructed vector and the ith word segmentation vectoriRepresents the ith word segmentation vector and the ith word segmentation vector,
denotes the ith reconstruction vector, and n denotes the number of participle vectors or the number of reconstruction vectors.
The preset threshold in step S140 may be set according to specific situations, for example: the method using the mean standard deviation and the 68-95-99.7 rule (empirical rule) are calculated as follows:
wherein threshold represents a preset threshold,
the average value of all the similarity degrees can be expressed by formula
And calculating, wherein Si represents the cosine distance similarity between the ith reconstruction vector and the ith word segmentation vector, mu represents the mathematical expectation of all similarities, and sigma represents the variance of all similarities.
After step S140, step S150 is performed: and if the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold value, determining the flow data as abnormal flow.
The flow data is detected by a network firewall and is confirmed to be free of abnormal data, and if the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold value, the flow data is determined to be abnormal flow; the preset threshold value here may also be set as the case may be, for example, set to 70%, 80%, or 90%.
The embodiment of the step S150 includes: if the similarity between the reconstructed vector and the word segmentation vector is smaller than or equal to a preset threshold value, determining the flow data as abnormal flow; if the similarity between the reconstructed vector and the word segmentation vector is larger than a preset threshold value, determining the flow data as normal flow; in a specific practical process, the formula can be used as:
wherein threshold represents a preset threshold value, SiRepresenting the cosine distance similarity of the ith reconstruction vector and the ith word segmentation vector,
indicating that the flow data is determined to be an abnormal flow,
indicating that the flow data is determined to be a normal flow.
Optionally, after the traffic data is determined to be abnormal traffic, the white list and the abnormal matching rule in the firewall may also be updated according to the judgment of security personnel, and this embodiment may include: judging whether data information corresponding to the traffic data is in a white list or not; if the data information corresponding to the flow data is not in the white list, sending the flow data to the terminal equipment, and updating an abnormal matching rule in a network firewall according to the data information; and after the security personnel corresponding to the terminal equipment confirm that the flow data is abnormal flow, sending data information to the network firewall so that the network firewall updates the abnormal matching rule.
Optionally, in this embodiment of the present application, after sending the traffic data to the terminal device, the security personnel corresponding to the terminal device may further confirm, and update the white list after the confirmation, where the implementation may include: after the security personnel corresponding to the terminal device confirm that the traffic data is not abnormal traffic, adding data information corresponding to the traffic data to a white list, specifically for example: an Internet Protocol (IP) address or a domain name address in the HTTP traffic data is added to the white list.
In the implementation process, firstly, field extraction is carried out on the flow data according to the protocol type of the flow data to obtain field data, then word segmentation is carried out on the field data to obtain words after word segmentation, vectorization is carried out on the words after word segmentation to obtain word segmentation vectors, and then the word segmentation vectors are reconstructed by using a pre-trained unsupervised neural network model to obtain reconstructed vectors; finally, judging whether the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold value or not; and if the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold value, determining the flow data as abnormal flow. That is to say, since the supervised neural network model is not sensitive enough to the data characteristic change and the unsupervised neural network model is more sensitive to the data characteristic change, the probability of false alarm occurring when the flow data is detected can be effectively reduced by using the unsupervised neural network model to detect the flow data, the condition of abnormal detection result of the false alarm is also reduced, and thus the accuracy of detecting the flow data is improved.
Please refer to fig. 3, which is a schematic flow chart illustrating the processing of concept drift according to the embodiment of the present application; optionally, in the embodiment of the present application, after the flow data is determined as the abnormal flow, a concept drift may also occur, where the concept drift refers to a case where the accuracy of the abnormal detection of the flow data decreases or the false alarm rate decreases with time; the manner of handling the concept drift may include:
step S210: and calculating the accuracy and the false alarm rate of the historical reference data according to the historical reference data of the flow data and the abnormal result corresponding to the historical reference data.
The embodiment of step S210 described above is, for example: acquiring historical reference data of the flow data, manually marking the historical reference data to obtain abnormal results corresponding to the historical reference data, and then calculating the accuracy and the false alarm rate of the historical reference data according to the historical reference data of the flow data and the abnormal results corresponding to the historical reference data; the specific process of manually marking the historical reference data includes: firstly, carrying out anomaly detection on historical reference data by using a traditional network firewall, marking the abnormal data as 1, and marking the normal data as 0; then, the manual work is used for screening and checking whether the mark of the abnormal result corresponding to the historical reference data is correct or not.
Step S220: and if the accuracy rate is smaller than the first preset proportion or the false alarm rate is larger than the second preset proportion, training the unsupervised neural network model again.
In a specific implementation process, since the unsupervised neural network model may have a concept drift periodically, the unsupervised neural network model may also be trained periodically, so as to reduce the probability of the concept drift of the unsupervised neural network model.
The above-mentioned embodiment of training the unsupervised neural network model in step S220 includes:
step S221: and performing field extraction and word segmentation on the historical reference data to obtain a field word segmentation result.
The implementation principle and implementation manner of step S221 are similar to those of steps S110 to S120, and therefore, the implementation principle and implementation manner of step are not described herein, and if not clear, reference may be made to the description of steps S110 to S120.
Step S222: and (5) encoding the field word segmentation result by using an encoder in the unsupervised neural network model to obtain an encoding result.
Step S223: and randomly sampling the coding result by using a hidden layer in the unsupervised neural network model to obtain a sampling result.
Step S224: and decoding the sampling result by using a decoder in the unsupervised neural network model to obtain a decoding result.
The implementation principle and implementation manner of the above steps S222 to S224 are similar to those of the steps S131 to S133, and therefore, the implementation manner and implementation principle of the steps are not described herein, and if not clear, reference may be made to the description of the steps S131 to S133.
Step S225: and calculating a loss value between the encoding result and the decoding result, and updating the parameters in the encoder and the parameters in the decoder according to the loss value.
The embodiment of step S225 described above is, for example: calculating a loss value between the encoding result and the decoding result according to the loss function, and updating parameters in the encoder and parameters in the decoder according to the loss value; the loss function here can be designed using KL divergence (KLD) or JS divergence (JSD). For ease of understanding and explanation, the KL divergence is used herein to calculate the loss value between the encoded result and the decoded result, for example, as:
wherein li(θ, φ) represents the loss function, and the overall loss resulting from the combination of the loss functions for each piece of data can be expressed as
Denotes the encoder, pφ(xz) denotes the decoder described above, i ═ 1,2, …, k denotes the ith encoding result or the ith decoding result, k denotes the number of ith encoding results or the ith decoding result,
representing the loss value, KL (q), for reconstructing the vector of wordsθ(z|xi) | p (z)) represents KL divergenceThe loss value of (a). In a specific practical process, the corresponding parameters of the encoder and the decoder can be optimized and updated by using the gradient descent method of Adam.
Please refer to fig. 4, which illustrates a schematic structural diagram of an abnormal flow rate detection apparatus provided in the embodiment of the present application; the embodiment of the present application provides an abnormal flow detection device 300, including:
the field data obtaining module 310 is configured to obtain the traffic data, and perform field extraction on the traffic data according to the protocol type of the traffic data to obtain field data.
The word segmentation vector obtaining module 320 is configured to perform word segmentation on the field data to obtain words after word segmentation, and perform vectorization on the words after word segmentation to obtain word segmentation vectors.
And a reconstructed vector obtaining module 330, configured to use a pre-trained unsupervised neural network model to reconstruct the word segmentation vector to obtain a reconstructed vector.
The vector similarity determining module 340 is configured to determine whether a similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold.
And an abnormal traffic determination module 350, configured to determine the traffic data as the abnormal traffic if the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold.
Optionally, in an embodiment of the present application, the unsupervised neural network model includes: an encoder, a hidden layer, and a decoder; a reconstruction vector obtaining module comprising:
and the vector coding operation module is used for carrying out coding operation on the word segmentation vectors by using the coder to obtain the coding vectors.
And the sampling vector obtaining module is used for randomly sampling the vector points in the coding vector by using the hidden layer to obtain a sampling vector.
And the vector decoding operation module is used for performing decoding operation on the sampling vector by using a decoder to obtain a reconstructed vector.
Optionally, in an embodiment of the present application, the abnormal traffic determining module includes:
and the data information judging module is used for judging whether the data information corresponding to the flow data is in the white list, and the flow data is data which is detected by the network firewall and is confirmed to be not abnormal.
And the matching rule updating module is used for sending the flow data to the terminal equipment if the data information corresponding to the flow data is not in the white list, and updating the abnormal matching rule in the network firewall according to the data information.
Optionally, in this embodiment of the present application, the abnormal flow rate detecting device further includes:
and the data information adding module is used for adding the data information corresponding to the flow data to the white list after the safety personnel corresponding to the terminal equipment confirm that the flow data is not abnormal flow.
Optionally, in an embodiment of the present application, the matching rule updating module includes:
and the data information sending module is used for sending data information to the network firewall after security personnel corresponding to the terminal equipment confirm that the flow data is abnormal flow, so that the network firewall updates the abnormal matching rule.
Optionally, in this embodiment of the present application, the abnormal flow rate detecting device may further include:
and the historical data calculation module is used for calculating the accuracy and the false alarm rate of the historical reference data according to the historical reference data of the flow data and the abnormal result corresponding to the historical reference data.
And the model retraining module is used for retraining the unsupervised neural network model again if the accuracy rate is smaller than the first preset proportion or the false alarm rate is larger than the second preset proportion.
Optionally, in an embodiment of the present application, the model retraining module includes:
and the word segmentation result obtaining module is used for carrying out field extraction and word segmentation on the historical reference data to obtain a field word segmentation result.
And the coding result obtaining module is used for coding the field word segmentation result by using a coder in the unsupervised neural network model to obtain a coding result.
And the sampling result obtaining module is used for randomly sampling the coding result by using a hidden layer in the unsupervised neural network model to obtain a sampling result.
And the decoding result obtaining module is used for decoding the sampling result by using a decoder in the unsupervised neural network model to obtain a decoding result.
And the network parameter updating module is used for calculating a loss value between the encoding result and the decoding result and updating the parameters in the encoder and the parameters in the decoder according to the loss value.
It should be understood that the apparatus corresponds to the above-mentioned embodiment of the abnormal flow rate detection method, and can perform the steps related to the above-mentioned embodiment of the method, and the specific functions of the apparatus can be referred to the above description, and the detailed description is appropriately omitted here to avoid redundancy. The device includes at least one software function that can be stored in memory in the form of software or firmware (firmware) or solidified in the Operating System (OS) of the device.
Please refer to fig. 5, which illustrates a schematic structural diagram of an electronic device according to an embodiment of the present application. An electronic device 900 provided in an embodiment of the present application includes: a processor 910 and a memory 920, the memory 920 storing machine readable instructions executable by the processor 910, the machine readable instructions when executed by the processor 910 performing the method as above.
The present embodiment also provides a storage medium 930, where the storage medium 930 stores a computer program, and the computer program is executed by the processor 910 to perform the method as above.
The storage medium 930 may be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as a Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic Memory, a flash Memory, a magnetic disk, or an optical disk.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules of the embodiments in the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
The above description is only an alternative embodiment of the embodiments of the present application, but the scope of the embodiments of the present application is not limited thereto, and any person skilled in the art can easily conceive of changes or substitutions within the technical scope of the embodiments of the present application, and all the changes or substitutions should be covered by the scope of the embodiments of the present application.
Claims (10)
1. An abnormal traffic detection method, comprising:
acquiring flow data, and performing field extraction on the flow data according to the protocol type of the flow data to acquire field data;
segmenting the field data to obtain segmented words, and vectorizing the segmented words to obtain segmented word vectors;
reconstructing the word segmentation vector by using a pre-trained unsupervised neural network model to obtain a reconstructed vector;
judging whether the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold value or not;
and if so, determining the flow data as abnormal flow.
2. The method of claim 1, wherein the unsupervised neural network model comprises: an encoder, a hidden layer, and a decoder; the reconstructing the word segmentation vector by using a pre-trained unsupervised neural network model comprises the following steps:
performing encoding operation on the word segmentation vectors by using the encoder to obtain encoding vectors;
randomly sampling the coding vector by using the hidden layer to obtain a sampling vector;
and performing decoding operation on the sampling vector by using the decoder to obtain the reconstruction vector.
3. The method of claim 1, wherein the determining the traffic data as abnormal traffic comprises:
judging whether data information corresponding to the flow data is in a white list or not, wherein the flow data is data which is detected and confirmed by a network firewall and has no abnormity;
if not, the flow data is sent to the terminal equipment, and the abnormal matching rule in the network firewall is updated according to the data information.
4. The method of claim 3, wherein after the sending the traffic data to a terminal device, further comprising:
and after the safety personnel corresponding to the terminal equipment confirm that the flow data is not abnormal flow, adding data information corresponding to the flow data to the white list.
5. The method of claim 3, wherein updating the exception matching rule in the network firewall according to the data information comprises:
and after the security personnel corresponding to the terminal equipment confirm that the flow data is abnormal flow, sending the data information to the network firewall so that the network firewall updates the abnormal matching rule.
6. The method according to any of claims 1-5, further comprising, after said determining said traffic data as anomalous traffic:
calculating the accuracy and the false alarm rate of the historical reference data according to the historical reference data of the flow data and the abnormal result corresponding to the historical reference data;
and if the accuracy rate is smaller than a first preset proportion, or the false alarm rate is larger than a second preset proportion, the unsupervised neural network model is trained again.
7. The method of claim 6, wherein the training the unsupervised neural network model comprises:
performing field extraction and word segmentation on the historical reference data to obtain a field word segmentation result;
encoding the field word segmentation result by using an encoder in the unsupervised neural network model to obtain an encoding result;
randomly sampling the coding result by using a hidden layer in the unsupervised neural network model to obtain a sampling result;
decoding the sampling result by using a decoder in the unsupervised neural network model to obtain a decoding result;
and calculating a loss value between the encoding result and the decoding result, and updating the parameters in the encoder and the parameters in the decoder according to the loss value.
8. An abnormal flow rate detecting device, comprising:
the field data acquisition module is used for acquiring flow data and extracting fields of the flow data according to the protocol type of the flow data to acquire field data;
the word segmentation vector obtaining module is used for performing word segmentation on the field data to obtain words after word segmentation, and performing vectorization on the words after word segmentation to obtain word segmentation vectors;
the reconstructed vector obtaining module is used for reconstructing the word segmentation vector by using a pre-trained unsupervised neural network model to obtain a reconstructed vector;
the vector similarity judging module is used for judging whether the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold value or not;
and the abnormal flow determining module is used for determining the flow data as the abnormal flow if the similarity between the reconstructed vector and the word segmentation vector is smaller than a preset threshold.
9. An electronic device, comprising: a processor and a memory, the memory storing machine-readable instructions executable by the processor, the machine-readable instructions, when executed by the processor, performing the method of any of claims 1 to 7.
10. A storage medium, having stored thereon a computer program which, when executed by a processor, performs the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011545306.7A CN112671768A (en) | 2020-12-24 | 2020-12-24 | Abnormal flow detection method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011545306.7A CN112671768A (en) | 2020-12-24 | 2020-12-24 | Abnormal flow detection method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112671768A true CN112671768A (en) | 2021-04-16 |
Family
ID=75409733
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011545306.7A Pending CN112671768A (en) | 2020-12-24 | 2020-12-24 | Abnormal flow detection method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112671768A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113141373A (en) * | 2021-04-30 | 2021-07-20 | 平安普惠企业管理有限公司 | Method, device, equipment and storage medium for detecting abnormal intrusion |
| CN113472654A (en) * | 2021-05-31 | 2021-10-01 | 济南浪潮数据技术有限公司 | Network traffic data forwarding method, device, equipment and medium |
| CN114301629A (en) * | 2021-11-26 | 2022-04-08 | 北京六方云信息技术有限公司 | IP detection method, device, terminal equipment and storage medium |
| CN114500018A (en) * | 2022-01-17 | 2022-05-13 | 武汉大学 | Web application firewall security detection and reinforcement system and method based on neural network |
| CN115277477A (en) * | 2022-07-24 | 2022-11-01 | 杭州迪普科技股份有限公司 | Flow detection method and device based on simple object access protocol |
| CN115277098A (en) * | 2022-06-27 | 2022-11-01 | 深圳铸泰科技有限公司 | Intelligent learning-based network flow anomaly detection device and method |
| CN117640252A (en) * | 2024-01-24 | 2024-03-01 | 北京邮电大学 | Encryption stream threat detection method and system based on context analysis |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103532949A (en) * | 2013-10-14 | 2014-01-22 | 刘胜利 | Self-adaptive trojan communication behavior detection method on basis of dynamic feedback |
| CN110266675A (en) * | 2019-06-12 | 2019-09-20 | 成都积微物联集团股份有限公司 | A kind of xss attack automated detection method based on deep learning |
| CN110691100A (en) * | 2019-10-28 | 2020-01-14 | 中国科学技术大学 | Hierarchical network attack identification and unknown attack detection method based on deep learning |
| CN111914873A (en) * | 2020-06-05 | 2020-11-10 | 华南理工大学 | Two-stage cloud server unsupervised anomaly prediction method |
| CN111967571A (en) * | 2020-07-07 | 2020-11-20 | 华东交通大学 | MHMA-based anomaly detection method and equipment |
-
2020
- 2020-12-24 CN CN202011545306.7A patent/CN112671768A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103532949A (en) * | 2013-10-14 | 2014-01-22 | 刘胜利 | Self-adaptive trojan communication behavior detection method on basis of dynamic feedback |
| CN110266675A (en) * | 2019-06-12 | 2019-09-20 | 成都积微物联集团股份有限公司 | A kind of xss attack automated detection method based on deep learning |
| CN110691100A (en) * | 2019-10-28 | 2020-01-14 | 中国科学技术大学 | Hierarchical network attack identification and unknown attack detection method based on deep learning |
| CN111914873A (en) * | 2020-06-05 | 2020-11-10 | 华南理工大学 | Two-stage cloud server unsupervised anomaly prediction method |
| CN111967571A (en) * | 2020-07-07 | 2020-11-20 | 华东交通大学 | MHMA-based anomaly detection method and equipment |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113141373A (en) * | 2021-04-30 | 2021-07-20 | 平安普惠企业管理有限公司 | Method, device, equipment and storage medium for detecting abnormal intrusion |
| CN113472654A (en) * | 2021-05-31 | 2021-10-01 | 济南浪潮数据技术有限公司 | Network traffic data forwarding method, device, equipment and medium |
| CN114301629A (en) * | 2021-11-26 | 2022-04-08 | 北京六方云信息技术有限公司 | IP detection method, device, terminal equipment and storage medium |
| CN114500018A (en) * | 2022-01-17 | 2022-05-13 | 武汉大学 | Web application firewall security detection and reinforcement system and method based on neural network |
| CN114500018B (en) * | 2022-01-17 | 2022-10-14 | 武汉大学 | Web application firewall security detection and reinforcement system and method based on neural network |
| CN115277098A (en) * | 2022-06-27 | 2022-11-01 | 深圳铸泰科技有限公司 | Intelligent learning-based network flow anomaly detection device and method |
| CN115277477A (en) * | 2022-07-24 | 2022-11-01 | 杭州迪普科技股份有限公司 | Flow detection method and device based on simple object access protocol |
| CN115277477B (en) * | 2022-07-24 | 2024-03-01 | 杭州迪普科技股份有限公司 | Flow detection method and device based on simple object access protocol |
| CN117640252A (en) * | 2024-01-24 | 2024-03-01 | 北京邮电大学 | Encryption stream threat detection method and system based on context analysis |
| CN117640252B (en) * | 2024-01-24 | 2024-03-26 | 北京邮电大学 | Encryption stream threat detection method and system based on context analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112671768A (en) | Abnormal flow detection method and device, electronic equipment and storage medium | |
| CN108737406B (en) | Method and system for detecting abnormal flow data | |
| US11783046B2 (en) | Anomaly and causation detection in computing environments | |
| Min et al. | Network anomaly detection using memory-augmented deep autoencoder | |
| CN111585955B (en) | HTTP request abnormity detection method and system | |
| US8112484B1 (en) | Apparatus and method for auxiliary classification for generating features for a spam filtering model | |
| Santos et al. | Twitter content-based spam filtering | |
| CN111818198B (en) | Domain name detection method, domain name detection device, equipment and medium | |
| US20210295183A1 (en) | Systems and methods for automated alert processing | |
| CN114079579B (en) | Malicious encryption traffic detection method and device | |
| JP6868416B2 (en) | Failure response support system | |
| US20220253526A1 (en) | Incremental updates to malware detection models | |
| CN112052451A (en) | Webshell detection method and device | |
| CN112131249A (en) | Attack intention identification method and device | |
| EP4241163A1 (en) | Computer-based systems configured for automated computer script analysis and malware detection and methods thereof | |
| Anton et al. | Security in process: Detecting attacks in industrial process data | |
| CN116107834A (en) | Log abnormality detection method, device, equipment and storage medium | |
| Rücker et al. | FlexParser—The adaptive log file parser for continuous results in a changing world | |
| CN110414229B (en) | Operation command detection method, device, computer equipment and storage medium | |
| CN116842520A (en) | Anomaly perception method, device, equipment and medium based on detection model | |
| CN113918936A (en) | SQL injection attack detection method and device | |
| CN115622793A (en) | Attack type identification method and device, electronic equipment and storage medium | |
| Plaisted et al. | DIP: a log parser based on" disagreement index token" conditions | |
| CN115934484A (en) | Diffusion model data enhancement-based anomaly detection method, storage medium and equipment | |
| CN113535458B (en) | Abnormal false alarm processing method and device, storage medium and terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210416 |
|
| RJ01 | Rejection of invention patent application after publication |