CN108540338B - Application layer communication protocol identification method based on deep cycle neural network - Google Patents

Abstract

The invention belongs to the technical field of protocols as characteristics, and discloses a method for identifying an application layer communication protocol based on a deep cycle neural network, which comprises the steps of obtaining original data of the application layer communication protocol, and carrying out format and system processing on the original data to obtain bit stream data of 0-255; evaluating and preprocessing according to frame lengths of different protocols to obtain an experimental data set required by neural network training; constructing an LSTM-DNN neural network, wherein the LSTM-DNN neural network comprises two LSTM layers, two DNN layers and a classification layer, and setting a cross entropy loss function and an Adam weight updating function to obtain a network architecture; and training and verifying the network by using the data set to obtain a deep circulation neural network with application layer communication protocol recognition capability, and preprocessing and recognizing the bit stream data to be detected. The invention does not need protocol prior knowledge, has higher identification accuracy and can identify protocol data in real time after network training is finished.

Description

Application layer communication protocol identification method based on deep cycle neural network

Technical Field

The invention belongs to the technical field of protocols as characteristics, and particularly relates to a method for identifying an application layer communication protocol based on a deep cycle neural network.

Background

Currently, the current state of the art commonly used in the industry is such that:in the information explosion age, the propagation medium thereof must be rapidly developed, and the network for carrying information is one of them. Due to the rapid development of the network, various protocols emerge endlessly, and especially a large number of private protocols emerge before people like bamboo shoots in the spring after rain, so that the protocols can be accurately identified, and the method has great significance for network security, network planning and management, network traffic optimization and the like. The problem of recognition is a focus of the attention of scholars at home and abroad, and a great deal of research work is carried out. Current application layer protocol identificationThere are three other methods commonly used: one is the conventional port-based identification technology, i.e. a protocol identification technology for matching port numbers. The technology generally aims at identifying known ports, and the identification method has the greatest advantages of simplicity and feasibility and can draw conclusions without carrying out complex grouping processing. Commonly used ports are: known ports (port numbers 0-1024), registered ports (port number 1025 plus 49151), dynamic, private ports (port number 49152 plus 65535). However, with the widespread use of various P2P protocols and the need to escape firewall detection, problems have arisen with conventional approaches based on commonly used port identification protocols. The port identification technology is only applied to identification of the known ports, the identification range is narrow, and the prospect is not optimistic. And secondly, based on the protocol identification of the characteristic string, respectively finding out fixed fields which must appear in the actual interaction process and have the highest appearance frequency for various protocols, and identifying the various protocols by taking the fixed fields as the characteristic string of the protocols. Compared with the former method, the method improves the quantity and the accuracy of the recognizable protocols and is easy to update. And thirdly, protocol identification is carried out based on static load characteristics, and the method mainly uses a deep packet detection technology and a deep stream detection technology. The method matches the application layer message characteristics of various services, needs to analyze the content of the application layer message, and relates to invasion of user privacy. In addition, with the continuous appearance of new services, the protocol feature library needs to be updated continuously, and the workload is large. In addition, it is not policy-oriented for anonymous encryption traffic as well as private protocol traffic. Compared with the three, the port-based protocol identification has the advantages of small quantity of identifiable protocols, low algorithm complexity, low accuracy and easy updating. The protocol identification based on pattern matching and the protocol identification based on feature code extraction have the advantages of large quantity of identifiable protocols, high algorithm complexity and high accuracy, but the latter is more difficult to update than the former.

In summary, the problems of the prior art are as follows:

(1) the port-based identification technology is only applied to identification of known ports, the identification range is narrow, a large number of dynamic ports and private ports are used along with wide application of various P2P protocols and the need of avoiding firewall detection, protocol identification cannot be carried out by only utilizing port numbers, and the traditional port-based identification technology cannot adapt to protocol identification at the present stage.

(2) The protocol identification is carried out based on the static load characteristics, when the load is not encrypted, the strategy is very accurate, but the safety and privacy problems of user data are involved, and under the condition of high-speed network connection, the classifier also needs more resource allocation; when the payload is encrypted, protocol identification cannot be performed. Meanwhile, the algorithm is high in complexity, a huge feature knowledge base needs to be established, and updating is not easy.

The difficulty and significance for solving the technical problems are as follows:

the method solves the problems by using a deep learning method, and realizes the protocol identification method which has strong universality, high accuracy and high identification speed and does not relate to privacy. The method starts from the whole data, does not consider a specific port number, and solves the problem caused by a dynamic port number or a private port; the neural network algorithm is used for extracting automatic features as a classification basis, the characteristics are not readable to a user, and the problems of safety and privacy brought by plaintext analysis are solved; meanwhile, the problem that the encrypted information cannot be identified is solved; finally, because the method completes the training of the model in advance, the speed in the identification process is very high, and the problem of real-time performance is solved.

Disclosure of Invention

Aiming at the problems in the prior art, the invention provides an application layer communication protocol identification method based on a deep cyclic neural network.

The invention is realized in this way, a method for identifying the application layer communication protocol based on the deep circulation neural network, the method for identifying the application layer communication protocol based on the deep circulation neural network obtains the original data of the application layer communication protocol, and carries out format and system processing on the original data to obtain the bit stream data of 0 to 255; evaluating and preprocessing according to frame lengths of different protocols to obtain an experimental data set required by neural network training; constructing an LSTM-DNN neural network, wherein the LSTM-DNN neural network comprises two LSTM layers, two DNN layers and a classification layer, and setting a cross entropy loss function and an Adam weight updating function to obtain a network architecture; and training and verifying the network by using the data set to obtain a deep circulation neural network with application layer communication protocol recognition capability, and preprocessing and recognizing the bit stream data to be detected.

Further, the method for identifying the application layer communication protocol based on the deep circular neural network comprises the following steps:

step one, acquiring the protocol data of the application layer of the computer network, carrying out format processing on the protocol data, and removing redundant information except bit streams. Converting the data into decimal data which can be identified by an algorithm to obtain a data set D1, wherein the data range is 0-255, and turning to the second step;

step two, respectively calculating and respectively calculating the reference length of each data segment in the data set D1 to obtain a reference length set L, wherein L is { L1, L2.. LK }, calculating the difference between the maximum value max (L) in L and the minimum value min (L) in L to be P, if P is less than min (L)/2, turning to the step three, otherwise, turning to the step four;

step three, when P is less than min (L)/2, selecting the average mean (L) of the L set as the reference length, if the reference length is an even number, not processing, otherwise, adding 1 to the average value for processing; cutting and zero padding operations on the data set D1, specifically: for the data segment with the data length larger than the reference length, retaining the data with the front reference length; for data with data length smaller than the reference length, zero padding is carried out to the reference length after the data; obtaining a data set D2, and turning to the fifth step;

step four, when P > min (L)/2, taking max (L) as a reference length, if the reference length is an even number, not processing, and if not, adding 1 to the reference length, and performing zero padding operation on each data segment in the data set D1, specifically: for data with the data length smaller than the reference length, zero padding is carried out on the data to the reference length to obtain a data set D3, and the step five is turned to;

step five, constructing a circulating neural network, selecting LSTM neurons and common neurons, constructing an LSTM-DNN neural network, wherein the LSTM-DNN neural network comprises two LSTM layers, two DNN layers and a classification layer, obtaining an initial network architecture S1, and turning to step six;

step six, when the data source is a data set D2, training a network framework S1 by using the first 80% of data, verifying by using the remaining last 20% of data, adjusting the hyper-parameters according to the obtained training accuracy and the verification accuracy, and finally obtaining a model S2 with the highest verification accuracy, when the data source is a data set D3, training a network framework S1 by using the first 80% of data, verifying by using the remaining last 20% of data, adjusting the hyper-parameters according to the obtained training accuracy and the verification accuracy, and finally obtaining a model S3 with the highest verification accuracy, and turning to step seven;

and step seven, processing the data to be detected according to the step one to the step four to obtain a data set D5 to be detected, and inputting the data set D5 to the model S2 or S3 according to the type D5 to obtain a final recognition result.

Further, the step two of calculating the lengths of the protocol data to be identified respectively includes:

(1) respectively calculating the length of each protocol data segment in the data set D1, and sorting according to the length; selecting the length of the first 90% of data in each protocol as a reference length of each protocol to obtain a reference length set L, { L1, L2,. LK }, and turning to (2);

(2) calculating the difference between the maximum value max (L) in L and the minimum value min (L) in L as P, if P < min (L)/2, turning to the third step, if P > min (L)/2, turning to the fourth step.

Further, the step five of constructing a recurrent neural network, setting the number of network layers, the network connection mode, the number of neurons in each layer, and setting a loss function and a weight update function includes:

(1) selecting LSTM as a basic neuron, 256 neurons in each layer, two layers in total, selecting common neurons in the second part, 256 neurons in the first layer, 64 neurons in the second layer, and the number of neurons in the third layer as the number of protocol types to be identified, and turning to (2);

(2) and (5) updating the function by using the cross entropy loss function and the Adam weight to obtain a network architecture S1, and turning to the step six.

Further, in the step (1), LSTM is selected as a basic neuron, each layer includes 256 neurons, and a total of two layers specifically include:

1) judging the input data type, if the source is the data set 1, turning to 2), otherwise, turning to 3);

2) operating according to 1), without change, turning to (2);

3) and adding a layer consisting of common neurons before the LSTM layer, wherein the number of the neurons is 1.5 times of the shortest protocol reference length, and when the number of the neurons is an even number, the neurons are not processed, otherwise, the number of the neurons is processed by adding 1, and the process is turned to (2).

Another object of the present invention is to provide an information data processing terminal applying the method for application layer communication protocol identification based on a deep recurrent neural network.

In summary, the advantages and positive effects of the invention are:the method uses a neural network to automatically extract data characteristics, and updates the weight through a gradient descent function; the training of the model and the recognition of the protocol can be completed only by making a corresponding data set, so that the method has better universality and improves the recognition accuracy.

Drawings

Fig. 1 is a flowchart of a method for identifying an application layer communication protocol based on a deep recurrent neural network according to an embodiment of the present invention.

Fig. 2 is a flowchart of network training according to an embodiment of the present invention.

Fig. 3 is a diagram of a circular network architecture of a data set D2 provided by an embodiment of the present invention.

Fig. 4 is a diagram of a circular network architecture of a data set D3 provided by an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

The invention carries out protocol identification under the condition of only acquiring the original bit stream, does not need protocol prior knowledge, has higher identification accuracy and can identify protocol data in real time after network training is finished.

As shown in fig. 1, the method for identifying an application layer communication protocol based on a deep recurrent neural network according to an embodiment of the present invention includes the following steps:

s101: acquiring original data of an application layer communication protocol, and carrying out format and system processing on the original data to obtain bit stream data of 0-255;

s102: evaluating and preprocessing according to frame lengths of different protocols to obtain an experimental data set required by neural network training; constructing an LSTM-DNN Neural Network, wherein the LSTM-DNN Neural Network comprises two LSTM layers (Long Short-Term Memory), two DNN layers (sense Neural Network) and a classification layer, and setting a cross entropy loss function and an Adam weight updating function to obtain a Network architecture;

s103: and training and verifying the network by using the data set to obtain a deep circulation neural network with application layer communication protocol recognition capability, and preprocessing and recognizing the bit stream data to be detected.

The application of the principles of the present invention will now be described in further detail with reference to the accompanying drawings.

The embodiment of the invention provides a method for identifying an application layer communication protocol based on a deep cycle neural network, which is applied to identifying the application layer protocol of a computer network. Different application layer protocol data are adopted, data processing is carried out according to different data characteristics in different modes, different network models are constructed according to different data sets, the models are trained by using the data sets, a classifier with the optimal classification effect is obtained by continuously adjusting parameters, and classification and identification are carried out on the data to be detected.

The method for identifying the application layer communication protocol based on the deep cycle neural network comprises the following steps:

step 1, obtaining protocol data of a computer network application layer by using a wireshark, carrying out format processing on the protocol data, removing other fixed output formats such as time, separators and the like, converting the data into decimal data which can be identified by an algorithm, dividing and storing the decimal data by using a blank space to obtain a data set D1, wherein the data range is 0-255, and turning to step 2;

and 2, calculating reference lengths of the data segments in the data set D1 respectively to obtain a reference length set L, wherein L is { L1, L2,. LK }, and calculating the difference between the maximum value max (L) in L and the minimum value min (L) in L as P. If P < min (L)/2, the calculation method is as follows:

and respectively calculating the length of each protocol data segment in the data set D1, and sorting according to the length. Selecting the length of the first 90% of data in each protocol as the reference length of each protocol to obtain a reference length set L, wherein L is { L1, L2.. LK }, calculating the difference between the maximum value max (L) in L and the minimum value min (L) in L as P, if P is less than min (L)/2, turning to the step 3, otherwise, turning to the step 4;

step 3, when P < min (L)/2, selecting the average mean (L) of the L set as the reference length, if the reference length is an even number, not processing, otherwise, adding 1 to the average value for processing; cutting and zero padding operations on the data set D1, specifically: for the data segment with the data length larger than the reference length, retaining the data with the front reference length; and for data with the data length smaller than the reference length, zero padding is carried out after the data to the reference length. Obtaining a data set D2, and turning to the step 5;

and 4, when P > min (L)/2, taking max (L) as a reference length, if the reference length is an even number, not processing, and if not, adding 1 to the reference length, and performing zero padding operation on each data segment in the data set D1, specifically: and for data with the data length smaller than the reference length, zero padding is carried out after the data to the reference length. Obtaining a data set D3, and turning to the step 5;

and 5, constructing a recurrent neural network, wherein the method comprises the following steps: setting the number of network layers, the network connection mode, the number of neurons in each layer, a loss function and a weight updating function, and setting an accuracy calculation method, wherein the specific method comprises the following steps:

judging the data source is from that data set, and for data set 1, setting the network structure as follows:

the first part selects LSTM as basic neuron, each layer of 256 neurons, two layers are total, the second part selects common neurons, the first layer of 256 neurons, the second layer of 64 neurons, and the number of the third layer of neurons is the number of protocol types to be identified;

for data set 2, the network structure is set as follows:

the first part is a layer composed of a layer of ordinary neurons, the number of the neurons is 1.5 times of the shortest protocol reference length, when the number of the neurons is an even number, the processing is not carried out, otherwise, the number plus 1 is processed, the second part selects LSTM as basic neurons, each layer of 256 neurons, two layers in total, the third part selects ordinary neurons, the first layer of 256 neurons, the second layer of 64 neurons, and the number of the third layer of neurons is the number of the protocol types to be identified, and the step 6 is turned to;

step 6, training a network architecture S1 by using data of the data set D2 or the first 80% of data of the data set D3, verifying by using the remaining last 20% of data, adjusting hyper-parameters according to the obtained training accuracy and verification accuracy, finally obtaining a model S2 with the highest verification accuracy, and turning to step 7;

and 7, processing the data to be detected according to the steps 1-4 to obtain a data set D5 to be detected, and inputting D5 into the model S2 to obtain a final recognition result.

The identification results of the selected part of the data are shown in the following table

Type (B)	icmp	ftp	dns	ssh	oicq	http
							Rate of accuracy	99.9686	1.0000	99.9573	99.8925	1.0000	98.9532

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.

Claims (4)

1. The method for identifying the application layer communication protocol based on the deep circulation neural network is characterized in that the method for identifying the application layer communication protocol based on the deep circulation neural network is used for obtaining the original data of the application layer communication protocol and carrying out format and system processing on the original data to obtain bit stream data of 0-255; evaluating and preprocessing the frame length according to the frame lengths of different protocols to obtain an experimental data set required by neural network training; constructing an LSTM-DNN neural network, wherein the LSTM-DNN neural network comprises two LSTM layers, two DNN layers and a classification layer, and setting a cross entropy loss function and an Adam weight updating function to obtain a network architecture; training and verifying the network by using a data set to obtain a deep circulation neural network with application layer communication protocol recognition capability, and preprocessing and recognizing bit stream data to be detected;

the method for identifying the application layer communication protocol based on the deep circular neural network comprises the following steps:

acquiring protocol data of an application layer of a computer network, carrying out format processing on the protocol data, and removing redundant information except bit streams; converting the data into decimal data which can be identified by an algorithm to obtain a data set D1, wherein the data range is 0-255, and turning to the second step;

step two, respectively calculating the reference length of each data segment in the data set D1 to obtain a reference length set L, wherein L is { L1, L2.. LK }, recording the difference between the maximum value max (L) in L and the minimum value min (L) in L as P, if P is less than min (L)/2, turning to the step three, otherwise, turning to the step four;

step three, when P is less than min (L)/2, selecting the average mean (L) of the L set as the reference length, if the reference length is an even number, not processing, otherwise, adding 1 to the average value for processing; cutting and zero padding operations on the data set D1, specifically: for the data segment with the data length larger than the reference length, retaining the data with the front reference length; for data with data length smaller than the reference length, zero padding is carried out to the reference length after the data; obtaining a data set D2, and turning to the fifth step;

step four, when P > min (L)/2, taking max (L) as a reference length, if the reference length is an even number, not processing, and if not, adding 1 to the reference length, and performing zero padding operation on each data segment in the data set D1, specifically: for data with the data length smaller than the reference length, zero padding is carried out on the data to the reference length to obtain a data set D3, and the step five is turned to;

step five, constructing a circulating neural network, selecting LSTM neurons and common neurons, constructing an LSTM-DNN neural network, wherein the LSTM-DNN neural network comprises two LSTM layers, two DNN layers and a classification layer, obtaining an initial network architecture S1, and turning to step six;

step six, when the data source is a data set D2, training a network framework S1 by using the first 80% of data, verifying by using the remaining 20% of data, adjusting the hyper-parameters according to the obtained training accuracy and the verification accuracy, and finally obtaining a model S2 with the highest verification accuracy, when the data source is a data set D3, training a network framework S1 by using the first 80% of data, verifying by using the remaining 20% of data, adjusting the hyper-parameters according to the obtained training accuracy and the verification accuracy, and finally obtaining a model S3 with the highest verification accuracy, and turning to step seven;

step seven, processing the data to be tested according to the step one to the step four to obtain a data set D5 to be tested, inputting the data set D5 to the model according to the type D5, and inputting the data set D2 to the model when P is less than min (L)/2; when P is greater than min (L)/2, inputting into the model S3, obtaining the final recognition result.

2. The method for identifying an application layer communication protocol based on a deep recurrent neural network as claimed in claim 1, wherein the step two of calculating the reference length of each data segment in the data set Dl respectively comprises:

(1) respectively calculating the length of each protocol data segment in the data set D1, and sorting according to the length; selecting the length of the first 90% of data in each protocol as a reference length of each protocol to obtain a reference length set L, { L1, L2,. LK }, and turning to (2);

(2) and (3) recording the difference between the maximum value max (L) in the L and the minimum value min (L) in the L as P, if P is less than min (L)/2, turning to the third step, and if P is more than min (L)/2, turning to the fourth step.

3. The method for application layer communication protocol identification based on the deep recurrent neural network as claimed in claim 1, wherein the step five of constructing the recurrent neural network, selecting LSTM neurons and normal neurons, and constructing the LSTM-DNN neural network comprises:

(1) selecting LSTM as a basic neuron, 256 neurons in each layer, two layers in total, selecting common neurons in the second part, 256 neurons in the first layer, 64 neurons in the second layer, and the number of neurons in the third layer as the number of protocol types to be identified, and turning to (2);

(2) and (5) updating the function by using the cross entropy loss function and the Adam weight to obtain a network architecture S1, and turning to the step six.

4. The method for application layer communication protocol identification based on the deep recurrent neural network of claim 3, wherein in the step (1), LSTM is selected as a basic neuron, each layer has 256 neurons, and a total of two layers specifically include:

1) judging the input data type, if the source is the data set 1, turning to 2), otherwise, turning to 3);

2) operating according to 1), without change, turning to (2);

3) and adding a layer consisting of common neurons before the LSTM layer, wherein the number of the neurons is 1.5 times of the shortest protocol reference length, and when the number of the neurons is an even number, the neurons are not processed, otherwise, the number of the neurons is processed by adding 1, and the process is turned to (2).

Images