CN101282251A - A Method of Mining Feature of Application Layer Protocol Identification - Google Patents

Abstract

The invention discloses a mining method for application layer protocol identification features. The method comprises the following steps: Step A, filtering and encoding the training data packet set for the first time, and extracting quasi-protocol identification feature data information; Step B, performing first mining from the extracted quasi-protocol identification feature data information , to obtain a multi-level frequent itemset; step C, performing the first filtering on the multi-level frequent itemset, and correcting the frequency of the remaining multi-level frequent itemsets after the first filtering and mining for the second time, Filter it for the second time to obtain the final protocol identification features; step D, if the byte recognition rate of all final protocol identification features meets the requirements, or the sum of data packet recognition rates meets the requirements, then no longer dig the second and subsequent The data of the data packet; otherwise, the second and subsequent data packets are cyclically mined until the total recognition rate reaches the requirement. It can analyze and mine the data packet collection, and can extract all the identification features of the corresponding application layer protocol, which greatly improves the feature extraction efficiency and the overall identification rate.

Description

A Method of Mining Feature of Application Layer Protocol Identification

technical field

The invention relates to the technical field of computer network flow monitoring and analysis, in particular to a mining method for application layer protocol identification features.

Background technique

Network application layer traffic identification is crucial to network planning, network management, traffic engineering, security detection, etc. The traditional application layer identification method is mainly based on the Internet Assigned Numbers Authority (IANA) to define the protocol port corresponding to the application, but for the purpose of hiding traffic or other aspects, the existing technology uses a large number of dynamic protocol ports or encrypted data Packet load, which brings great challenges to traditional application layer identification methods.

In order to solve this problem, a deep packet inspection (Deep Packet Inspection, DPI) identification method has been proposed. The method finds out the characteristic string of the data packet to compose the application layer protocol identification characteristic database, and adopts the characteristic matching method to carry out traffic identification.

However, the premise of using this method is that the application layer protocol characteristics of the protocol must be correctly found out. The accuracy of the identification characteristics of the application layer protocol has a great impact on the recognition rate, accuracy rate and false recognition rate.

Currently, there are two main methods for extracting application layer protocol features:

First, find the definition document of the application layer protocol, and obtain the application layer characteristics of the protocol according to the provisions of the document. However, many new application layer protocols are currently implemented using private protocols, and protocol definition documents, such as PPlive, cannot be obtained. In addition, even if the protocol is public, the frequency of protocol updates brings great difficulties to the update of protocol identification features.

Second, use wireshark, tcpdump and other packet capture tools to capture the data packets in the communication process of the protocol, and manually check and compare the data packets corresponding to each flow to find the application layer characteristics of the protocol. However, this method has low efficiency and reliability.

With the continuous emergence of new applications, to achieve accurate application-layer traffic identification, it is necessary to constantly find and update the application-layer characteristics of each protocol. However, since there is no better mining method for application layer protocol features at present, manual analysis is still the most commonly used mining method for application layer protocol identification features. At present, many network monitoring and analysis product companies have dedicated a strong team to track various applications on a regular basis, including data packet capture and manual protocol analysis, in order to improve the accuracy of the identification signature database.

However, relying on manual analysis to realize the mining method of application layer protocol identification features, the feature extraction efficiency is very low, and it cannot meet the needs of new application layer protocols emerging and frequent upgrades of existing protocols. completely, resulting in low overall recognition efficiency.

Contents of the invention

The purpose of the present invention is to provide a method for mining application layer protocol identification features, which can fully automatically analyze and mine data packet collections, extract all identification features of corresponding application layer protocols, and greatly improve feature extraction efficiency and overall recognition rate.

A kind of application layer protocol identification feature mining method provided for realizing the purpose of the present invention comprises the following steps:

Step A, the training data packet set is filtered for the first time, and encoded, and the quasi-protocol identification feature data information is extracted;

Step B, performing the first mining from the extracted quasi-protocol identification feature data information to obtain multi-level frequent itemsets;

Step C, performing the first filtering on the multi-level frequent itemsets, correcting the frequency of the remaining multi-level frequent itemsets after the first filtering and mining for the second time, performing a second filtering on them, Get the final protocol identification feature.

The described application layer protocol identification feature mining method also includes the following steps:

Step D, if the byte recognition rate of all final protocol identification features meets the requirements, or the sum of the data packet recognition rates meets the requirements, then no longer mine the data of the second and subsequent data packets; otherwise, cyclically mine the second and subsequent data packages until the total recognition rate meets the requirement.

Described step A comprises the following steps:

A1. Capture the training data packet set and store the training data packet set in the flow structure after dividing the training data packet set by flow;

A2. Utilize the mixed traffic filtering method to filter the mixed traffic in the training data packet set;

A3. Encoding the application layer payload by encoding the bytes in the extracted data packet based on the location;

A4. Extract the encoded data information, and extract the quasi-protocol identification feature data information.

In the step A2, the hybrid flow filtering method includes the following steps:

A21. Filter out the content of the flow that meets the HTTP protocol and the FTP protocol;

A22. Filter out the flows that do not have a complete three-way handshake in the TCP flow.

In step A21, the content of the flow that satisfies the FTP protocol is filtered out, including the following steps:

Filter out the data packets of the flow structure that communicates in PASV mode;

Filter out the packets using the structure of ports 20 and 21.

The method for judging the flow structure of the PASV mode communication using FTP comprises the following steps:

Find the flow structure using port 21, and judge whether the data packets belonging to the structure have data packets starting with 227;

If there is, it is further judged whether it is a response packet of the PASV mode, and if so, the data packet includes the IP address and the port number that the server prepares to carry out the PASV mode data connection with the client, and the destination IP address of the data packet is also the client The IP address of the terminal, the FTP data connection adopts the TCP protocol, and records these four data;

After traversing all flow structures using port 21, all flow information using FTP in PASV mode communication is obtained.

Said filtering out the packets of the flow structure that adopts PASV mode communication includes the following steps:

Compare each flow with the recorded data connection flow information in PASV mode using FTP. If four of the five-tuple information in the flow structure are the same as the recorded flow information in PASV mode data connection, then this is considered A flow is a flow that communicates in PASV mode of FTP, and all packets in this flow are discarded.

In said step A3, the method for encoding the bytes in the extracted data packet based on the position comprises the following steps:

Encode the value of a byte represented by two hexadecimal numbers in the data packet into a single item represented by 5 characters. The first character from the left is I, which means Item, and the second and third characters Indicates which bit the byte is in the first N bytes extracted above, expressed in hexadecimal, counting from zero, if less than sixteen, the second bit is zero; the fourth and fifth characters are Two hexadecimal characters of the original byte.

In the step A4, the quasi-protocol identification feature data information includes encoded information with the same offset data packet, and related statistical auxiliary data information;

The extraction of quasi-protocol identification feature data information includes the following steps:

A41. Extract the quasi-protocol identification feature data information for the TCP flow, and import it into the transaction database;

A42. Extract the quasi-protocol identification feature data information from the UDP stream, and import it into the transaction database.

Said step B comprises the following steps:

Step B1, first set an initial frequency, multiply the initial frequency by the total number of transactions in the database, and then round up to get the initial frequency; calculate the frequency of each single item in the database, filter out the frequency less than the initial frequency The single item of the degree, each of the remaining single items is called a first-level frequent item, and the set of all first-level frequent items is called a first-level frequent item set;

Step B2, select two K-1 level frequent items A and B from the K-1 level frequent item set, these two K-1 level frequent items must satisfy the first K-2 single items of K-1 level frequent item A and The K-2 individual items of the K-1 frequent item B are the same, and the K-1th individual item of the K-1 frequent item A is different from the K-1th individual item of the K-1 frequent item B. If B The position of the K-1th single item of A is greater than the position of the K-1th single item of A, then the K-1th single item of B is appended to the back of A to obtain a quasi-K-level frequent item, and the quasi-K-level frequent item is calculated If the frequency of the item is greater than or equal to the initial frequency, it is indeed a K-level frequent item. According to this method, all K-level frequent items are obtained to form a K-level frequent itemset; where K≥2.

The transaction includes four attribute fields of coded bytes, packet length, byte percentage and packet percentage to store the extracted data.

Described step C comprises the following steps:

C1. Using frequency correction and frequent item filtering methods, performing the first frequent item filtering, frequency correction and the second frequent item filtering on the multi-level frequent item set;

C2. Convert the frequent items in the corrected and filtered frequent item set into feature strings, mine the absolute packet length feature and the feature of the relationship between packet length and content difference, and mark the corresponding transaction, and then identify the quasi-protocol corresponding to the transaction The feature data information is filtered to obtain the final protocol identification feature.

Said step C1 comprises the following steps:

Step C11, performing the first frequent item filtering on the multi-level frequent item set;

Step C12, perform frequency correction and second frequent item filtering on the multi-level frequent item set after the first frequent item filtering, and eliminate the inclusion relationship between frequent items.

Described step C12 comprises the following steps:

Step C121, using the following formula to correct the frequency of the first-level frequent items;

${\mathrm{<span\; class="notranslate">\; freq</span>}}_{\mathrm{<span\; class="notranslate">\; new</span>}}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; freq</span>}}_{\mathrm{<span\; class="notranslate">\; old</span>}}<span\; class="notranslate">\; ;</span>& {\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0,0.9</span>}<span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; freq</span>}}_{\mathrm{<span\; class="notranslate">\; old</span>}}<span\; class="notranslate">\; ;</span>& {\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; 0,0</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\end{array}\right.$

Among them, freq _new represents the new frequency after correction, freq _old represents the frequency of frequent items before correction, pos _i represents the position of the i-th single item in frequent items, i starts from zero, and k represents the frequency of single items in frequent items number; the position of a single item is numbered from zero, k ₁ is a constant, and f(pos ₀ ) is a continuous monotonically decreasing function;

Step C122, using the following formula to correct the frequency of the 2-level frequent item;

Among them, k ₂ is a constant, f((pos ₁ -pos ₀ )) is a continuous monotone decreasing function;

Step C123, at first, use the following formula to find the average distance between items in frequent items, which is recorded as ave _dist ;

${\mathrm{<span\; class="notranslate">\; ave</span>}}_{\mathrm{<span\; class="notranslate">\; dist</span>}}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}}{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}$

The distance refers to the absolute value of the difference between the positions of two adjacent single items in the item;

If ave _dist ≠1, use the following formula to correct ave _dist ;

Among them, k ₃ and k ₄ are constants;

Then, the following formula is used to modify the frequency of frequent items of the 3rd and 4th grades;

Among them, f ₁ (k) is a continuous monotonically increasing function about k; f ₂ (ave _dist ) and f ₃ (ave _dist ) are continuous monotonically decreasing functions about ave _dist , and satisfy f ₂ (ave _dist )＞f ₃ (ave _dist );

Step C124, filtering out the frequent items with small frequency among the frequent items having inclusion relationship.

Said step C2 comprises the following steps:

Step C21, convert the frequent items in the corrected and filtered frequent item set into a feature string, retrieve the transactions satisfying the feature string from the transaction database, and mine the absolute packet length feature and the packet length and content difference relationship feature, And mark the transactions that satisfy the characteristics of the absolute packet length and the relationship between the packet length and the content difference;

Step C22 , filtering the quasi-protocol identification feature data information corresponding to the transaction to remove repeated mining features, weak features, and suspected mixed features to obtain the final protocol identification feature.

The beneficial effects of the present invention are:

1. It is highly efficient to analyze the identification characteristics of each application layer protocol by the method of the present invention. Using the technology of the present invention makes it possible to realize the periodic update of the identification characteristics of each protocol in the feature library. In order to realize complete and accurate identification All traffic lays the foundation;

2. The completeness and reliability of the identification features of the application layer protocol obtained by the method of the present invention are greatly improved compared with the current technology. Because this method mines the identification features of the application layer protocol on the basis of extracting and analyzing a large amount of data, it can obtain a more complete and complete identification feature of the application layer protocol, and has analyzed multiple A possible feature has been filtered and verified at multiple levels, and its reliability is greatly improved compared to the current technology that only analyzes the limited data packets;

3. The method of the present invention proposes the index of measuring whether the identification feature of the application layer protocol is correct or not: recognition rate, accuracy rate, positive and false recognition rate, negative false recognition rate, and measures the automatically drawn application layer protocol identification feature from all aspects , so that the automatically obtained protocol identification features can achieve a high recognition rate, while ensuring a high accuracy rate and a low misidentification rate, so that the correctness of the automatically identified application layer protocol identification features can be judged in accordance with;

4. The method of the present invention is not only to transform the manual analysis process of current identification protocol features into a process of automatic computer mining and processing of application layer protocol identification features, but also lies in the accuracy, completeness and reliability of the protocol identification features obtained by it In other aspects, this has greatly changed the phenomenon that the misrecognition rate between protocols increased with the increase of identification protocols. This method reduces the misrecognition rate between protocols to a small range, making reliable Sexuality is greatly improved.

Description of drawings

Fig. 1 is the working flow chart of application layer protocol identification feature mining method of the present invention;

Fig. 2 is the work flowchart of training data extraction step among the present invention;

Fig. 3 is the work flowchart of quasi-feature primary filtering step in the present invention;

Fig. 4 is a working flow chart of the quasi-feature secondary mining and filtering step in the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, a method for mining application layer protocol identification features of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The present invention proposes a method for mining application layer protocol identification features. The method only needs to capture the data packet set in the communication process of the application layer protocol, and the application layer can be extracted by analyzing and mining the data packet set. All the recognition features of the protocol, the feature extraction efficiency and the overall recognition rate are greatly improved compared with the prior art.

In order to better illustrate a kind of application layer protocol identification feature mining method of the present invention, below at first the terms used in the present invention and several features that can be mined are described:

Flow: Refers to a five-tuple including the source IP, destination IP, source port, destination port and protocol of both communicating parties.

Training data packet set: a set of captured data packets in the communication process of a specific protocol, and use this captured training data packet set to mine application layer protocol identification features.

Transaction: a record in the transaction database. In the present invention, a transaction includes four attribute fields of encoded byte, packet length, byte percentage and packet percentage, and stores the extracted data.

Single item: the encoded representation of one byte B ₁ in the application layer payload. The first bit is the character I, which means Item. The hexadecimal value represented by the second and third bits is called the position of the item. The fourth and fifth bits Bits represent the hexadecimal value of byte B ₁ , eg: I0039.

Item: A sequence consisting of one or more single items. The single items in the sequence are separated by punctuation marks (such as commas), and the position of the single items in the sequence satisfies the increasing relationship from left to right.

Frequency: The number of times an item occurs in a transactional database.

Frequent item: an item whose frequency is greater than or equal to the initial frequency (a preset positive integer).

K-level frequent item: a frequent item containing K individual items.

Frequent itemset: A set composed of frequent items at all levels.

Feature string: A combination of characters that appear at a fixed position in the application layer payload and can identify the protocol.

Absolute packet length feature: all data packets satisfying a feature string have the same length of application layer payload, so they are said to satisfy the absolute packet length feature. The absolute packet length feature is a combined feature with the feature string, so when the absolute packet length feature is satisfied, a string feature must be satisfied, which effectively reduces misidentification and improves recognition accuracy.

The difference relationship between packet length and content: All data packets that satisfy a characteristic string, if the length of the application layer payload and the value at a fixed position in the application layer payload differ by a fixed value, it is said to meet Difference relationship between packet length and content. The feature of the difference between packet length and content is a combination feature of the feature string, so when the feature of the difference between packet length and content is satisfied, a string feature must be satisfied, which effectively reduces misidentification and improves recognition accuracy.

The invention proposes a mining method for application layer protocol identification features, which can mine and extract the following three features: feature character strings, absolute packet length features, and difference relationship features between packet length and content. Introduce a kind of application layer protocol identification feature automatic mining method of the present invention below in conjunction with above-mentioned object, as shown in Figure 1, comprise the following steps:

Step S100, using the hybrid traffic filtering method and the application layer payload encoding method to filter and encode the training data packet set, and extract quasi-protocol identification feature data information;

As shown in Figure 2, the step S100 includes the following steps;

Step S110, capturing the training data packet set and storing the training data packet set into the flow structure after dividing the training data packet set according to flow;

Extract its source IP, destination IP and protocol number from the IP datagram header of each data packet. If the protocol number is 6, it means that the transport layer adopts the TCP protocol, and the source port number and the destination port number are extracted from the TCP header; if the protocol number is 17, it means that the transport layer uses the UDP protocol, and the source port number and the destination port number are extracted from the UDP header. number; if the protocol number is other values, the data packet will be discarded.

Packets with the same source IP, destination IP, source port, destination port and protocol are divided into one category, and stored in a flow structure according to their arrival sequence, and the five-tuple of the flow is used as its label.

Step S120, using a hybrid traffic filtering method to filter the hybrid traffic in the training data packet set;

Step S121, filtering out the content of the flow determined to meet the HTTP protocol and the FTP protocol;

Because many softwares currently embed some advertisements on the page, these advertisements are transmitted by HTTP protocol, and some software also supports file download or data transmission based on HTTP protocol or FTP protocol at the same time, which leads to the fact that although only one software is used for communication, However, traffic of different protocols is generated, and if any software adopts any of these two methods, the HTTP or FTP protocol traffic generated in the communication will generally account for a large proportion of the total traffic. If it is not filtered out, The features of the HTTP or FTP protocol will be mined while mining the features of this protocol, which will cause the HTTP or FTP protocol to be identified as the protocol during real-time identification.

Therefore, when capturing a set of training data packets, in order to make the captured data packets basically communicate with the same application layer protocol, and only open one application software, it is necessary to filter out the content of the flow that meets the HTTP protocol and the FTP protocol .

Experiments show that the mixed traffic generated by other application layer protocols accounts for a small proportion of the total traffic, and will not have a great impact on the mining results. They can be filtered out by weak feature filtering.

Step S121A, first filter out the content of the HTTP protocol stream;

Judge each TCP flow structure in turn, and discard the data packets contained in the structure if the following two conditions are met: 1) the flow structure contains port 80; 2) the beginning of the fourth data packet in the structure is Any of GET, HEAD, POST, PUT, PATCH, COPY, MOVE, DELETE, LINK, UNLINK, OPTION.

Step S121B, then filter out the content of the flow of the FTP protocol;

(1) Filter out the data packets of the flow structure that adopts PASV mode communication;

(2) filter out the data packet adopting the structure of port 20 and port 21;

The method of judging the stream structure of PASV mode communication using FTP:

First look for the flow structure using port 21, and determine whether the data packets belonging to the structure have data packets starting with 227;

If there is, it is further judged whether it is the response packet of the PASV mode (i.e. according to the format of the FTP protocol response packet), if so, the IP address and the port number that the server side prepares to carry out the PASV mode data connection with the client are included in the packet, and at the same time The destination IP address of the data packet is also the IP address of the client. The FTP data connection adopts the TCP protocol. So far, 4 quintuple information of the flow using the FTP PASV mode communication have been obtained, and these four data are recorded; After using the flow structure of port 21, all the flow information of PASV mode communication using FTP can be obtained;

For the structure data packet of port 20, filter it out directly.

Compare each flow with the recorded data connection flow information using FTP in PASV mode, if four of the five-tuple information in the flow structure are the same as the recorded flow information of the PASV mode data connection , the flow is determined to be a flow using FTP PASV mode communication, and all data packets in the flow are discarded.

Step S122, filtering out flows without a complete three-way handshake in the TCP flow;

According to the flag bit of the three-way handshake data packet of the TCP protocol, it is judged whether each TCP flow is a flow with a complete three-way handshake, and the TCP flow without a complete three-way handshake data packet is filtered out;

Step S130, encoding the application layer payload by using the method of encoding the bytes in the extracted data packet based on the location;

In the header of the protocol, the same data in different positions represent different meanings, and many contents analyzed by the present invention relate to the header of the protocol. In addition, the processing result obtained by the method of finding frequent itemsets under the condition of no encoding is not enough for the problem of the present invention. Words are also meaningless. In order to solve this problem, the present invention proposes a method for encoding the bytes in the extracted data packets based on the position, and uses this method to encode all the contents of the data packets, which not only distinguishes the same bytes in different positions , and has decisive significance for the filtering of the identification characteristics of the quasi-application layer protocol and the automatic generation of the identification characteristics of the application layer protocol.

The encoding method is: the value of a byte in the original data packet is represented by two hexadecimal numbers, now this byte is encoded as a single item with 5 characters, and the first character from the left is I , which means Item, the second and third characters indicate the number of the byte in the first N bytes extracted above, expressed in hexadecimal, counting from zero, if less than sixteen, the second is zero; the fourth and fifth characters are two hexadecimal characters of the original byte.

Step S140, extracting the encoded data information, and extracting quasi-protocol identification feature data information.

The quasi-protocol identification feature data information includes encoded information with the same offset data packet, and related statistical auxiliary data information.

To the filtered flow, according to the upper layer protocol type in the IP datagram header, be divided into TCP flow and UDP flow, and carry out quasi-protocol identification characteristic data information extraction respectively, comprise the following steps:

Step S141, extracting quasi-protocol identification feature data information from the TCP flow, and importing it into the transaction database;

Extract the encoded information of the packets with the same offset for the TCP stream, that is, the first N encoded bytes of the i-th data packet in the stream, the packet length of the i-th data packet, and the total number of bytes in the stream. The percentage of the total number of bytes of all TCP flows with complete three-way handshake in the training data packet set, and the percentage of the total number of data packets of this flow to the total number of data packets of all TCP flows with complete three-way handshake in the training data packet set.

Then, import it into different tables of the transaction database according to the value of i, preferably, where 4≤i≤(n+4), the extraction from the fourth package is to avoid extracting the three-way handshake data for establishing a connection For the data of the packet, the maximum value of i is (n+4), which means that a total of n data packets are extracted. The purpose of extracting n data packets is to use the second iteration when the information obtained by mining the first data packet is not enough to effectively identify the protocol The contents of the 3 to nth data packets continue to be mined;

Step S142, extract quasi-protocol identification feature data information from the UDP stream, and import it into the transaction database.

The extraction of UDP flow is similar to the extraction of TCP flow, but 1≤i≤n, because UDP does not have the process of establishing a connection;

Step S200, mining multi-level frequent itemsets from the extracted quasi-protocol identification feature data information;

Feature strings (such as fixed version number, status, etc.) are one of the most common identification features, which generally appear in the header of the protocol, and each piece of data extracted in the previous step is likely to be part of the protocol header. If the combination strings that appear frequently can be found in these data, then these strings are likely to be the features that identify the protocol, so the problem of extracting feature strings can be transformed into the problem of finding frequent itemsets for coded data mining analysis question.

The mining method of the present invention obtains a first-level frequent itemset processing method: first set an initial frequency rate, multiply the total number of transactions in the transaction database by the initial frequency rate, and then round to obtain the initial frequency; calculate each The frequency of a single item, filter out the single item whose frequency is less than the initial frequency, each remaining single item is called a first-level frequent item, and the set of all first-level frequent items is called a first-level frequent item set;

The mining method of K (K≥2) level frequent items is: select two (K-1) level frequent items A and B from the (K-1) level frequent item set, these two (K-1) level frequent items It must be satisfied that the first (K-2) individual items of the (K-1) level frequent item A are the same as the (K-2) individual items of the (K-1) level frequent item B, and the (K-1) level frequent item A's The position of the (K-1)th single item and the (K-1)th single item of the (K-1) level frequent item B is different, if the position of the (K-1)th single item of B is greater than that of A's (K -1) single item position, then the (K-1)th single item of B is appended to A to obtain a quasi-K-level frequent item (composed of K single items), and the frequency of the quasi-K-level frequent item is calculated , if it is greater than or equal to the initial frequency, it is indeed a K-level frequent item. According to this method, all K-level frequent items are obtained to form a K-level frequent item set.

For example, the mining method of the second-level frequent items is: select two first-level frequent items from the first-level frequent item set, combine them into quasi-second-level frequent items according to the positional relationship, and then scan the transaction database to calculate the value of the quasi-second-level frequent items Frequency, if it is greater than or equal to the initial frequency, it is determined that the quasi-secondary frequent item is indeed a second-level frequent item, and all second-level frequent items can be obtained by this method to form a second-level frequent item set.

Because if there is a second-level frequent item, it must be a combination of two first-level frequent items, and due to the characteristics of the encoding in the present invention, the position of the item in the second-level frequent item must satisfy the arrangement from left to right from small to large , rather than random combinations of first-level frequent items, which greatly reduces the number of combinations and improves processing efficiency.

The mining method of the third-level frequent item is: select two second-level frequent items A and B from the second-level frequent item set, and these two second-level frequent items satisfy the first single item of the second-level frequent item A and the second-level frequent item B The first single item of A is the same, the second single item of the second-level frequent item A is different from the second single item of the second-level frequent item B, if the position of the second single item of B is greater than the position of the second single item of A , then add the second single item of B to the back of A to obtain a quasi-third-level frequent item (composed of three single items), calculate the frequency of the quasi-third-level frequent item, if it is greater than or equal to the initial frequency, then it is indeed is a three-level frequent item, and all three-level frequent items are obtained by this method to form a three-level frequent item set.

The frequent item mining method of the fourth-level frequent item and subsequent levels is the same as this one.

Iterative processing until no longer frequent items are found, the processing ends. Combining frequent items at all levels can obtain a frequent item set whose frequency is greater than or equal to a set value, and the frequent items containing a relatively large number of single items are called high-level frequent items, and the frequent items containing a small number of single items are called low-level frequent items frequent items.

Step S300, using the method of frequency correction and frequent item filtering to perform the first filtering on the multi-level frequent itemset, and perform correction, secondary mining and second filtering on the frequency of the remaining multi-level frequent items after the first filtering Secondary frequent item filtering to obtain the final protocol identification features;

Step S310, using frequency correction and frequent item filtering methods to perform the first frequent item filtering, frequency correction and second frequent item filtering on the multi-level frequent item set;

This step includes the first frequent item filtering process, the frequency correction and the second frequent item filtering process.

The first frequent item filtering process filters redundant quasi-features and all-zero quasi-features;

Frequency correction and the second frequent item filtering process: Correct the frequency of frequent items at levels 4 and below, and then filter out features with relatively low reliability through the second frequent item filtering method.

The frequency correction method effectively reduces the frequency of unreliable features, ensures that unreliable features are filtered out in the second frequent item filtering process, and greatly reduces the number of quasi-features. The recognition accuracy of the features is greatly improved, and at the same time, the probability of feature crossover between different protocols is reduced.

As shown in FIG. 3, the step S310 includes the following steps.

Step S311, performing the first frequent item filtering on the multi-level frequent item set;

The multi-level frequent item set processed in step S200 has great redundancy, that is, includes many redundant features, and needs to be subjected to the first frequent item filtering.

The redundant feature refers to: if the frequency of a frequent item is equal to the frequency of the high-level frequent item containing it, then the frequent item is called a redundant feature.

Because the existence of high-level frequent items means the existence of any sub-frequent items whose frequency is not less than that of the current high-level frequent items.

The first frequent item filtering starts from the first-level frequent item to check and filter out all redundant features. Because this is equivalent filtering, no loss of features will occur.

In addition, the first frequent item filtering also filters out frequent items with all zero content.

The frequent item with all zeros means that the last two digits of each single item in the frequent item are all zeros.

Firstly, the all-zero feature is less likely to be used in practice, because it can only identify a small amount of information; second, the all-zero feature is unreliable in practice, because many protocols use zero padding. A large number of frequent items of all zeros appear in the process of feature mining and analysis. If they are not filtered, it will lead to cross misidentification between protocols.

Step S312, performing frequency correction on the multi-level frequent item set after the first frequent item filtering, and then performing the second frequent item filtering to eliminate the inclusion relationship between frequent items;

After the first frequent item filtering, there may still be frequent items that contain each other, which is never allowed in the final result, otherwise the high-level frequent items will be automatically blocked, and the low-level frequent items appear more frequently, but High-level frequent items are less prone to misidentification.

Therefore, the present invention proposes to modify the frequency of frequent items of level 4 and below based on the starting position and the average distance between individual items in the frequent items.

Because the probability of misrecognition of low-level frequent items is greater than that of high-level frequent items, and there are generally more low-level frequent items, some low-level frequent items have a larger starting position and a larger distance between individual items, and such frequent items generally If the identification characteristics of the protocol are not met, their existence will cause high misidentification, so their frequency can be reduced according to these values to achieve the purpose of filtering them out. The reason for using length 4 as the boundary for correcting frequency is that there are generally fewer frequent items with a length greater than 4, and the frequency is relatively low, and the maximum probability of causing errors is 1/2. ⁴⁰ will not cause major errors even without filtering. identify.

Further, the frequency correction and the second frequent item filtering for the frequent items of level 4 and frequent items below level 4 include the following steps:

Step S3121, using the formula (1) to correct the frequency of the first-level frequent items;

${\mathrm{<span\; class="notranslate">\; freq</span>}}_{\mathrm{<span\; class="notranslate">\; new</span>}}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; freq</span>}}_{\mathrm{<span\; class="notranslate">\; old</span>}}<span\; class="notranslate">\; ;</span>& {\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0,0.9</span>}<span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; freq</span>}}_{\mathrm{<span\; class="notranslate">\; old</span>}}<span\; class="notranslate">\; ;</span>& {\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; 0,0</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Among them, freq _new represents the new frequency after correction, freq _old represents the frequency of frequent items before correction, pos _i represents the position of the i-th single item in frequent items, i starts from zero, and k represents the frequency of single items in frequent items number. The positions of the individual terms are numbered starting from zero, k ₁ is a constant, and f(pos ₀ ) is a continuous monotonically decreasing function.

Step S3122, using the formula (2) to correct the frequency of the 2-level frequent item;

Among them, k ₂ is a constant, and f((pos ₁ -pos ₀ )) is a continuous monotone decreasing function.

For frequent items whose starting position is in the first four positions and two single items are adjacent to each other, the frequency is not reduced, but the frequency is increased. The purpose is to eliminate the first-level frequent items whose frequency is slightly larger than it. The frequency of a high item has no effect on the frequency of longer items containing it, because its frequency is greater than that of the long frequent items it contains, if not increased.

Likewise, the purpose of increasing the frequency in the following steps is the same.

Step S3123, using formula (5) to correct the frequency of frequent items of level 3 and level 4;

First, use formula (3) to find the average distance between items in frequent items, which is recorded as ave _dist .

${\mathrm{<span\; class="notranslate">\; ave</span>}}_{\mathrm{<span\; class="notranslate">\; dist</span>}}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}}{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

The distance refers to the absolute value of the difference between the positions of two adjacent single items in an item.

If ave _dist ≠ 1, use the formula (4) to correct the ave _dist to ensure that the frequency is reduced appropriately and not too large at one time;

Among them, k ₃ and k ₄ are constants.

The corrected formula for the frequency of frequent items of level 3 and 4 is formula (5)

Among them, f ₁ (k) is a continuous monotonically increasing function about k; f ₂ (ave _dist ) and f ₃ (ave _dist ) are continuous monotonically decreasing functions about ave _dist , and satisfy f ₂ (ave _dist )＞f ₃ (ave _dist ).

Step S3124, filtering out frequent items with small frequency among the frequent items having inclusion relationship.

After frequency correction, compare the frequency of any two frequent items with inclusion relationship, and use the second frequent item filtering method to filter out the frequent items with smaller frequency.

Step S320, convert the frequent items in the corrected and filtered frequent item set into feature strings, mine the absolute packet length feature and the feature of the relationship between packet length and content difference, mark the corresponding transaction, and then convert the quasi-protocol corresponding to the transaction The identification feature data information is filtered to obtain the final protocol identification feature.

This step includes re-mining the transactions that satisfy the characteristic string, mining the absolute packet length feature and the packet length and content difference relationship feature, and marking the transactions that satisfy the absolute packet length feature and the packet length and content difference relationship feature ; and filter out weak features, repeated mining features, and suspected mixed features from the quasi-protocol identification feature data information corresponding to the transaction, reduce the false identification rate, and obtain the final protocol identification feature.

The function of this step is to increase the accuracy and reliability of protocol identification features, which greatly improves the accuracy of identification and reduces the false identification rate. The mining absolute packet length feature and the difference between packet length and content proposed in this step The method of relational features is the guarantee for accurate identification of protocols. Without this method, the misidentification rate between protocols will rise sharply; in addition, the weak features, repeated mining features and suspected mixed feature filtering methods proposed in this step are also reduced to a certain extent. Misidentification rate between protocols.

Step S321, converting the frequent items in the corrected and filtered frequent item set into a feature string, retrieving transactions satisfying the feature string from the transaction database, mining the absolute packet length feature and the packet length and content difference relationship feature, And mark the transactions that satisfy the characteristics of the absolute packet length and the relationship between the packet length and the content difference;

Convert each frequent item after the first frequent item filtering and the second frequent item correction and filtering into a characteristic string, scan each transaction in the transaction database in turn, and for the transaction that satisfies the characteristic string, First record the actual packet length of the data packet corresponding to the transaction, and secondly, the actual content of the extracted first N bytes (that is, the value of the last two digits of each single item) starts from the 0th byte, and the adjacent Two bytes are combined to form (N-1) big-endian (Big-Endian) double-byte unsigned integers and (N-1) little-endian (Little-Endian) double-byte integers. Byte unsigned integer, and set two storage structures capable of storing 2n×(N-1) values for storage. Subtract the previous (N-1) big-endian unsigned integers from the data packet length corresponding to the transaction, and if its value is between [-n, n-1], it will be stored in the above storage structure The corresponding position count is incremented by one, and its position is recorded at the corresponding position, and the same process is done for (N-1) little-endian unsigned integers.

If there is an absolute packet length feature, the actual packet lengths of the data packets corresponding to all transactions satisfying the characteristic string are the same value or the actual packet lengths of the data packets corresponding to most transactions are the same value. If there is a characteristic of the difference between the packet length and the content, if it satisfies the characteristic that the difference between the big-endian double-byte unsigned integer whose starting position is 2 is 5, then all or absolutely The difference between the actual packet length of most transactions and the big-endian double-byte unsigned integer whose starting position is 2 is 5.

After scanning all the transactions in the transaction database, it can be judged whether most (a certain percentage) of all transactions satisfying the characteristic string have the same package length. If so, it means that the absolute Packet length feature, append the absolute packet length feature after the feature string; if not, then judge whether there are most transactions whose packet length differs from a fixed start double-byte value by a fixed size value, if so Then the difference between the package length and content is satisfied. At this time, the feature string must be corrected first. The correction method is to remove the single item in the feature string whose position is the position of the difference between the package length and the content, and then add The difference relationship between packet length and content.

The frequency of scanning the transaction database to recalculate the corrected characteristics, the percentage of the total number of data packets of all flows satisfying this characteristic to the total number of data packets corresponding to the protocol (TCP or UDP), all the flows satisfying this characteristic The percentage of the total bytes of the protocol corresponding to the total number of bytes, marking transactions that meet the new characteristics.

In step S322, the quasi-protocol identification feature data information corresponding to the transaction is filtered out of repeated mining features, weak features and suspected mixed features to obtain the final protocol identification feature.

The remaining quasi-protocol identification feature data information features after filtering in the previous step are called second-level quasi-features. For any two second-level quasi-features, record the number of times they appear simultaneously in the transaction database. If the number of simultaneous appearances accounts for p ₁ % of the frequency of the quasi-feature with the smaller frequency, where p ₁ is a preset value, 80≤p ₁ ≤100, it is considered that these two quasi-features are actually repeated, they are The second-level quasi-features with low frequency found in basically the same transactions are called repeated mining features, and the repeated mining features are filtered out.

After filtering, it is judged in turn whether each second-level quasi-feature is only a level 1 frequent item. If so, it is called a weak feature, because its occurrence probability is 1/256, which will cause a high misidentification, so To filter it out; at the same time, for all the remaining second-level quasi-characteristics, it is judged that the total data packets and total bytes of all flows that meet the second-level quasi-characteristics account for the total data packets and total bytes corresponding to the protocol Whether the percentage of the number is less than p ₂ % (wherein, p ₂ is a preset value), if any second-level quasi-feature is said to be less than p ₂ %, then the feature is a suspected confounding feature, and the reason for the suspected confounding feature It may be that the traffic of other protocols is mixed in the training data packet set, which should be filtered out (experimental results show that there are many quasi-features with less than 1% of the traffic that cause a lot of misidentification. After filtering out these quasi-features, the recognition rate Basically no reduction, but the false recognition rate becomes very low), and the remaining quasi-features are used as the final protocol recognition features of the protocol.

Step S400, if the byte recognition rates of all final protocol identification features meet the requirements, or the sum of data packet recognition rates meets the requirements, then no longer mine the data of the second and subsequent data packets; otherwise, the second and subsequent data packets are mined in a loop packets until the total recognition rate reaches the requirement.

After correction, mining and filtering, if the sum of the byte recognition rates or packet recognition rates of all final protocol identification features reaches p ₃ % or more (wherein, p ₃ is a preset value, 80≤p ₃ ≤ 100), then the data of the second and subsequent data packets will not be mined, otherwise, the second and subsequent data packets will be mined cyclically until the total recognition rate reaches more than p ₃ %, and then the final protocol identification feature generation feature will be obtained by mining The corresponding feature file in the library. At this point, the flow identification program can be run for real-time online identification.

The use of final protocol identification features to generate relevant feature files in the feature library, and the use of traffic identification programs for real-time online identification are a prior art, and therefore, will not be described in detail in the embodiments of the present invention.

Taking the automatic analysis and generation of the application layer protocol that Thunder adopts the TCP protocol for communication and extracting the identification features as an example, the method for mining the identification features of the application layer protocol of the present invention will be further explained.

Step S100', extracting the data that Xunlei satisfies the conditional TCP flow;

Table 1 shows part of the data of the first packet of each flow in the 804 TCP flows that Xunlei meets.

Table 1:

affairs The first N encoded bytes package length Byte Percentage package percentage 1 I0038, I0100, I0200, I0300, I040d, I0500, I0600, I0700, I0884, I09ab, I0a0c, I0b00 twenty one 0.007039% 0.00597% 2 I0038, I0100, I0200, I0300, I0472, I0500, I0600, I0700, I0864, I09e5, I0a5c, I0b00 122 0.002133% 0.000246% 3 I0038, I0100, I0200, I0300, I040d, I0500, I0600, I0700, I0884, I09f4, I0a1e, I0b00 twenty one 0.011092% 0.00957%

4 I0038, I0100, I0200, I0300, I040d, I0500, I0600, I0700, I0884, I0933, I0a00, I0b00 twenty one 0.014291% 0.012682% 5 I0038, I0100, I0200, I0300, I040d, I0500, I0600, I0700, I0884, I095d, I0a13, I0b00 twenty one 0.011305% 0.011376% 6 I0038, I0100, I0200, I0300, I0472, I0500, I0600, I0700, I0864, I099d, I0a5f, I0b00 122 0.001493% 0.000159% 7 I0032, I0132, I0230, I032d, I0453, I0565, I0672, I0776, I082d, I0955, I1020, I1146 44 0.001067% 0.000103% 8 I0038, I0100, I0200, I0300, I0472, I0500, I0600, I0700, I0864, I09dc, I0a5c, I0b00 122 0.002133% 0.000246% ... ... ... ... ... 803 I0038, I0100, I0200, I0300, I0472, I0500, I0600, I0700, I0864, I09e5, I0a5c, I0b00 122 0.001493% 0.000269% 804 I0038, I0100, I0200, I0300, I0472, I0500, I0600, I0700, I0864, I09e5, I0a5c, I0b00 122 0.001493% 0.000269%

Step S200', finding all frequent items greater than a certain frequency.

Initial frequency is set to 0.02; Initial frequency = (initial frequency × number of transactions) rounding = (0.02 × 804) rounding = 16, mining all frequent items with a frequency greater than 16, this embodiment excavates a total of There are 31680 frequent items, some of which are listed in Table 2.

Table 2

serial number Frequency Frequent items at all levels 1 27 I0032 2 723 I0038 3 248 I040d 4 29 I0955 5 29 I0a20 ... ... ...

51 27 I0032, I0132 52 27 I0032, I0230 53 640 I0000, I0200 ... ... ... 397 27 I0032, I0132, I0230 398 27 I0032, I0132, I032d ... ... ... 5423 720 I0038, I0100, I0200, I0500, I0600 5424 720 I0038, I0100, I0200, I0500, I0700 5425 720 I0038, I0100, I0200, I0300, I0500, I0600, I0700 5426 228 I0100, I0200, I0300, I040d, I0500, I0600, I0700 ... ... ... 10000 718 I0038, I0100, I0200, I0300, I0500, I0600, I0700, I0b00 ... ... ... 31678 84 I0038, I0100, I0200, I0300, I0472, I0500, I0600, I0700, I0864, I09dc, I0a5c, I0b00 31679 63 I0038, I0100, I0200, I0300, I0472, I0500, I0600, I0700, I0864, I09e5, I0a5c, I0b00 31680 20 I0067, I0165, I0274, I032f, I040d, I050a, I066c, I0765, I086e, I0967, I0a74, I0b68

Step S300', quasi-feature filtering;

Step S310', (filtering for the first time) carries out the first filtering to the frequent items obtained in the previous step:

It is obvious that the frequent items obtained in step S200' are highly redundant and need to be filtered. The first filtering includes filtering redundant features and frequent items whose content is all zeros (see the embodiment for the filtering method).

In this example, a total of 31649 frequent items are filtered for the first time, which is very effective in reducing the number of candidate features, and redundant feature filtering is equivalent filtering, ensuring that there is no information loss during the filtering process, and 31 frequent items remain after filtering items, some of which are listed in Table 3.

table 3:

serial number Frequency Frequent items at all levels 2 723 I0038 3 248 I040d 4 29 I0955 5 29 I0a20 5425 720 I0038, I0100, I0200, I0300, I0500, I0600, I0700 5426 228 I0100, I0200, I0300, I040d, I0500, I0600, I0700 10000 718 I0038, I0100, I0200, I0300, I0500, I0600, I0700, I0b00 ... ... ... 31678 84 I0038, I0100, I0200, I0300, I0472, I0500, I0600, I0700, I0864, I09dc, I0a5c, I0b00 31679 63 I0038, I0100, I0200, I0300, I0472, I0500, I0600, I0700, I0864, I09e5, I0a5c, I0b00 31680 20 I0067, I0165, I0274, I032f, I040d, I050a, I066c, I0765, I086e, I0967, I0a74, I0b68

Step S320', (frequency correction and second filtering of frequent items) For frequent items of level 4 and above, the frequency is corrected based on the starting position and the average distance between individual items, and the second filtering is carried out. filter.

For the method of correcting the frequent items, refer to the implementation manner, and the corrected frequent items are shown in Table 4.

Table 4:

serial number Frequency Frequent items at all levels 2 686 I0038 3 137 I040d 4 10 I0955 5 9 I0a20 5425 720 I0038, I0100, I0200, I0300, I0500, I0600, I0700 5426 228 I0100, I0200, I0300, I040d, I0500, I0600, I0700 10000 718 I0038, I0100, I0200, I0300, I0500, I0600, I0700, I0b00 ... ... ... 31678 84 I0038, I0100, I0200, I0300, I0472, I0500, I0600, I0700, I0864, I09dc, I0a5c, I0b00 31679 63 I0038, I0100, I0200, I0300, I0472, I0500, I0600, I0700, I0864, I09e5, I0a5c, I0b00 31680 20 I0067, I0165, I0274, I032f, I040d, I050a, I066c, I0765, I086e, I0967, I0a74, I0b68

The serial numbers 2, 3, 4, and 5 in Table 4 are the corrected frequencies. Since other frequent items contain four or more individual items, their frequencies are not corrected.

After correcting the frequency, filter out the frequent items with small frequency among any two frequent items that have an inclusion relationship, and the filtered results are listed in Table 5.

table 5:

serial number Frequency Frequent items at all levels 5425 720 I0038, I0100, I0200, I0300, I0500, I0600, I0700 5426 228 I0100, I0200, I0300, I040d, I0500, I0600, I0700 31680 20 I0067, I0165, I0274, I032f, I040d, I050a, I066c, I0765, I086e, I0967, I0a74, I0b68

Step S400', secondary mining and filtering of quasi-features;

Step S410', mining whether each frequent item after the second filtering satisfies both the absolute packet length relationship and the packet length-content difference relationship. If the above relationship exists, the newly discovered features are added, and the old features (ie, frequent items) are corrected.

Convert the frequent item represented by the serial number 5425 in Table 5 into a characteristic string, the meaning of the characteristic string: the 0th byte (numbered from zero) of the application layer load is 0x38, the 1st, 2nd, 3rd, Bits 5, 6, and 7 are zero, and their frequency is recorded as Support=720.

Taking this feature string as an example, the method of mining absolute packet length features is as follows:

For example, a structure can be defined to record the distribution information of the packet length satisfying the characteristic string, as shown in Table 6.

Table 6:

package length 0 1 2 3 ... 1497 1498 1499 1500 Count 0 0 0 0 ... 0 0 0 0

Scan the transaction database, and record the corresponding packet length for the transaction that satisfies the feature string (for example: if the packet length is 21, add 1 to the count value unit with the packet length of 21). After scanning the transaction database, the value Count _max of the counting unit with the largest count value and the corresponding packet length P _len can be obtained, if (Count _max /Support)≥P ₅ %, (a certain percentage such as 95%) , then it is considered that the transaction that satisfies the characteristic string also satisfies the characteristic that the absolute packet length is P _len , and this characteristic is appended to the characteristic string to form a combined characteristic.

Taking the feature string converted from the frequent item with serial number 5425 as an example, the method of mining the difference relationship between packet length and content is as follows:

The maximum number of single items that can be included in a transaction in the transaction database is recorded as N, then for a transaction that contains N single items and satisfies the characteristic string, it can be extracted and combined into (N-1) big-endian bytes Big-Endian unsigned integers and (N-1) Little-Endian unsigned integers. (Refer to the embodiment for the detailed combination method) It can be represented in the following manner: Table 7 represents (N-1) big-endian unsigned integers, and the serial number also represents the position. For example: if the transaction contains the following three single items I0001, I0102, and I0203, it can be composed of two big-endian double-byte unsigned integers and two little-endian double-byte unsigned integers, composed of I0001 and The big-endian double-byte unsigned integer extracted by I0102 is 18, and the serial number is 0 (equal to the position of the single item with the smaller position among the two combined single items), and the combined little-endian word is extracted by I0001 and I0102 The endian double-byte unsigned integer is 33, and the sequence number is 0. Table 8 represents (N-1) little-endian two-byte unsigned integers.

Table 7 (N-1) big-endian unsigned integer table

serial number (position) value (big-endian) 0 BigVal ₀ 1 BigVal ₁ ... ... N-2 BigVal _N-2 N-1 BigVal _N-1

Table 8 (N-1) little-endian unsigned integer table

serial number (position) value (little endian) 0 LittVal ₀ 1 LittVal ₁ ... ... N-2 LittVal _N-2 N-1 LittVal _N-1

The packet length corresponding to each transaction in the database is recorded as P _len . In addition, two global structures are required to record the difference relationship between the packet length and the above 2×(N-1) value. For example, the following structure can be used to record the difference relationship between P _len of each transaction and (N 1) big-endian double-byte unsigned integers, as shown in Table 9.

Table 9 difference relationship table

serial number (position) -n -(n-1) ... 0 1 ... (n-1) 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 N-2 0 0 0 0 0 0 0 N-1 0 0 0 0 0 0 0

In Table 9, n represents a positive integer, record (P _len -BigVal _i )=MinVal _i , if -n≤MinVal _i <n, then add 1 to the count of MinVal _i column in row i, and scan the database once to get The largest count value, Count _Max , is located in row Pos and column Val in the table 9 structure. If (Count _Max /Support)≥P ₆ %, (a certain percentage such as 95%), it is considered that the transaction that satisfies the feature string also satisfies the packet length and the application layer load Pos and (Pos+1) word The big-endian two-byte unsigned integers that make up the section are characterized by Val. Determine whether the feature string contains a value whose position is equal to Pos or (Pos+1). If so, delete the values at these two positions from the feature string, and then add the difference between the packet length and content to the feature character After stringing, a combined feature is formed.

The frequency of scanning the transaction database to recalculate the corrected feature, the percentage of the total number of data packets of all streams satisfying the feature to the total number of data packets corresponding to the protocol (that is, the packet recognition rate of the quasi-feature), The total number of bytes of all streams that meet this characteristic is the percentage of the total number of bytes corresponding to the protocol (that is, the byte recognition rate of this standard characteristic), and the transactions that meet this characteristic are marked. The quasi-characteristics corrected by secondary excavation are shown in Table 10.

Table 10:

serial number Frequency Packet recognition rate Byte recognition rate quasi-features 5425 720 99.5% 99.94% I0038, I0100, I0200, I0600, I0700*BE*BG: 3*MV: 8*LE*BG: 4*MV: 8 5426 228 18.79% 20.58% I0100, I0200, I0300, I040d, I0500, I0600, I0700*APL: 21 31680 20 0.04% 0% I0067, I0165, I0274, I032f, I040d, I050a, I066c, I0765, I086e, I0967, I1074, I1168*APL: 75

Step S420', filtering weak features and repeated mining features.

Filter out the quasi-features that only contain one level of frequent items, the probability of this quasi-feature is 1/256, which will cause a high misrecognition rate.

Filter out repeated mining features (refer to the implementation method for the concept of repeated mining features), the quasi-features with serial numbers 5425 and 5426 obtained in the previous step are mined from basically the same transaction, because when the quasi-feature with serial number 5426 appears, the serial number is All of the quasi-features (99%) of 5425 have appeared, so the quasi-features represented by serial number 5426 are considered to be repeated mining features and should be filtered.

Filter out the quasi-features whose packet recognition rate or byte recognition rate is less than p ₂ %, and call this feature a suspected mixed feature. The reason for this feature may be that a small amount of traffic from other protocols is mixed into the training data packet set. Experiments prove that many features with a recognition rate of less than 1% cause serious misrecognition between protocols. After filtering out these features, the recognition rate is basically not affected, but the misrecognition rate is greatly reduced. The quasi-features with ordinal numbers 3 and 4 were filtered out according to this rule.

Currently only the following feature remains, as shown in Table 11.

Table 11:

serial number Frequency package percentage Byte Percentage Application Layer Features 5425 720 99.502% 99.939% I0038, I0100, I0200, I0600, I0700*BE*BG: 3*MV: 8*LE*BG: 4*MV: 8

I0038, I0100, I0200, I0600, and I0700 are characteristic strings, indicating that the 0-2 bits are 0x38, 0x00, 0x00, and the 6th and 7th bits are 0x00, 0x00; *BE*BG: 3*MV: 8 means package The difference between length and content, *BE means big-endian byte order, *BG: 3 means the start bit is at the third bit, MV: 8 means big-endian byte order composed of the packet length and the 3rd and 4th bits The difference is 8 for double-byte unsigned integers. LE*B6: 4*MV: 8 means the difference between packet length and content, *LE means little-endian byte order, *BG: 4 means the start bit is at the 4th bit, MV: 8 means the difference between the packet length and the 4. The difference between 5-bit little-endian double-byte unsigned integers is 8.

Step S500', use the mined protocol to identify features, and generate a feature file of the protocol. In this example, an XML file is used to represent the signature file. After the signature file is generated, the traffic identification program can be run for online real-time identification.

The following is an example of a computer-implemented algorithm for a signature file generated with the features mined above:

<tcp>

<type value="1">

<content_length>

<offset>4</offset>

<byte_number>10</byte_number>

<differ_value>8</differ_value>

</content_length>

<byte>

<content>0x38</content>

<offset>0</offset>

</byte>

<byte>

<content>0x00</content>

<offset>1</offset>

</byte>

<byte>

<content>0x00</content>

<offset>2</offset>

</byte>

<byte>

<content>0x00</content>

<offset>6</offset>

</byte>

<byte>

<content>0x00</content>

<offset>7</offset>

</byte></type><type value="2">

<content_length>

<offset>3</offset>

<byte_number>2</byte_number>

<differ_value>8</differ_value>

</content_length>

<byte>

<content>0x38</content>

<offset>0</offset>

</byte>

<byte>

<content>0x00</content>

<offset>1</offset>

</byte>

<byte>

<content>0x00</content>

<offset>2</offset>

</byte>

<byte>

<content>0x00</content>

<offset>6</offset>

</byte>

<byte>

<content>0x00</content>

<offset>7</offset>

</byte>

</type>

</tcp>

The beneficial effects of the present invention are:

1. It is highly efficient to analyze the identification characteristics of each application layer protocol by the method of the present invention. Using the technology of the present invention makes it possible to realize the periodic update of the identification characteristics of each protocol in the feature library. In order to realize complete and accurate identification All traffic lays the foundation;

2. The completeness and reliability of the identification features of the application layer protocol obtained by the method of the present invention are greatly improved compared with the current technology. Because this method mines the identification features of the application layer protocol on the basis of extracting and analyzing a large amount of data, it can obtain a more complete and complete identification feature of the application layer protocol, and has analyzed multiple A possible feature has been filtered and verified at multiple levels, and its reliability is greatly improved compared to the current technology that only analyzes the limited data packets;

3. The method of the present invention proposes the index of measuring whether the identification feature of the application layer protocol is correct or not: recognition rate, accuracy rate, positive and false recognition rate, negative false recognition rate, and measures the automatically drawn application layer protocol identification feature from all aspects , so that the automatically obtained protocol identification features can achieve a high recognition rate, while ensuring a high accuracy rate and a low misidentification rate, so that the correctness of the automatically identified application layer protocol identification features can be judged in accordance with;

4. The method of the present invention is not only to transform the manual analysis process of current identification protocol features into a process of automatic computer mining and processing application layer protocol identification features, but also lies in the accuracy, completeness and reliability of the protocol identification features obtained by it In other aspects, this has greatly changed the phenomenon that the misidentification rate between protocols increased with the increase of identification protocols. This method reduces the misidentification rate between protocols to a small range, making reliable Sexuality is greatly improved.

Other aspects and features of the present invention will be apparent to those skilled in the art from the above description of specific embodiments of the present invention in conjunction with the accompanying drawings.

The specific embodiments of the present invention have been described and illustrated above, and these embodiments should be considered as exemplary only, and are not used to limit the present invention, and the present invention should be interpreted according to the appended claims.

Claims (15)

1. An application layer protocol identification feature mining method is characterized in that, comprising the following steps: Step A, the training data packet set is filtered for the first time, and encoded, and the quasi-protocol identification feature data information is extracted; Step B, performing the first mining from the extracted quasi-protocol identification feature data information to obtain multi-level frequent itemsets; Step C, performing the first filtering on the multi-level frequent itemsets, correcting the frequency of the remaining multi-level frequent itemsets after the first filtering and mining for the second time, performing a second filtering on them, Get the final protocol identification feature. 2. application layer protocol identification feature mining method according to claim 1, is characterized in that, also comprises the following steps: Step D, if the byte recognition rate of all final protocol identification features meets the requirements, or the sum of the data packet recognition rates meets the requirements, then no longer mine the data of the second and subsequent data packets; otherwise, cyclically mine the second and subsequent data packages until the total recognition rate meets the requirement. 3. the application layer protocol identification feature mining method according to claim 1 or 2, is characterized in that, described step A comprises the following steps: A1. Capture the training data packet set and store the training data packet set in the flow structure after dividing the training data packet set by flow; A2. Utilize the mixed traffic filtering method to filter the mixed traffic in the training data packet set; A3. Encoding the application layer payload by encoding the bytes in the extracted data packet based on the location; A4. Extract the encoded data information, and extract the quasi-protocol identification feature data information. 4. a kind of application layer protocol identification feature mining method according to claim 3, is characterized in that, in described step A2, the mixed traffic filtering method comprises the following steps: A21. Filter out the content of the flow that meets the HTTP protocol and the FTP protocol; A22. Filter out the flows that do not have a complete three-way handshake in the TCP flow. 5. a kind of application layer protocol identification feature mining method according to claim 4, is characterized in that, in step A21, described filter out the content that satisfies the flow of FTP agreement, comprises the following steps: Filter out the data packets of the flow structure that communicates in PASV mode; Filter out the packets using the structure of ports 20 and 21. 6. a kind of application layer protocol identification feature mining method according to claim 5, is characterized in that, the judging method of the flow structure of the PASV pattern communication that adopts FTP, comprises the following steps: Find the flow structure using port 21, and judge whether the data packets belonging to the structure have data packets starting with 227; If there is, it is further judged whether it is a response packet of the PASV mode, and if so, the data packet includes the IP address and the port number that the server prepares to carry out the PASV mode data connection with the client, and the destination IP address of the data packet is also the client The IP address of the terminal, the FTP data connection adopts the TCP protocol, and records these four data; After traversing all flow structures using port 21, all flow information using FTP in PASV mode communication is obtained. 7. a kind of application layer protocol identification feature mining method according to claim 6, is characterized in that, described filtering out adopts the data packet of the flow structure body of PASV pattern communication, comprises the following steps: Compare each flow with the recorded data connection flow information in PASV mode using FTP. If four of the five-tuple information in the flow structure are the same as the recorded flow information in PASV mode data connection, then this is considered A flow is a flow that communicates in PASV mode of FTP, and all packets in this flow are discarded. 8. a kind of application layer protocol identification feature mining method according to claim 3, is characterized in that, in described step A3, the method for encoding the byte in the data packet that extracts based on position, comprises the following steps : Encode a byte value represented by two hexadecimal numbers in the data packet into a single item represented by 5 characters. The first character from the left is I, which means Item, and the second and third characters Indicates which bit the byte is in the first N bytes extracted above, expressed in hexadecimal, counting from zero, if less than sixteen, the second bit is zero; the fourth and fifth characters are Two hexadecimal characters of the original byte. 9. A kind of application layer protocol identification characteristic mining method according to claim 3, is characterized in that, in described step A4, described quasi-protocol identification characteristic data information comprises the encoded information with the same offset data packet , and related statistical auxiliary data information; The extraction of quasi-protocol identification feature data information includes the following steps: A41. Extract the quasi-protocol identification feature data information for the TCP flow, and import it into the transaction database; A42. Extract the quasi-protocol identification feature data information from the UDP stream, and import it into the transaction database. 10. A kind of application layer protocol identification feature mining method according to claim 3, is characterized in that, described step B comprises the following steps: Step B1, first set an initial frequency, multiply the initial frequency by the total number of transactions in the database, and then round up to get the initial frequency; calculate the frequency of each single item in the database, filter out the frequency less than the initial frequency The single item of the degree, each of the remaining single items is called a first-level frequent item, and the set of all first-level frequent items is called a first-level frequent item set; Step B2, select two K-1 level frequent items A and B from the K-1 level frequent item set, these two K-1 level frequent items must satisfy the first K-2 single items of K-1 level frequent item A and The K-2 individual items of the K-1 frequent item B are the same, and the K-1th individual item of the K-1 frequent item A is different from the K-1th individual item of the K-1 frequent item B. If B The position of the K-1th single item of A is greater than the position of the K-1th single item of A, then the K-1th single item of B is appended to the back of A to obtain a quasi-K-level frequent item, and the quasi-K-level frequent item is calculated If the frequency of the item is greater than or equal to the initial frequency, it is indeed a K-level frequent item. According to this method, all K-level frequent items are obtained to form a K-level frequent itemset; where K≥2. 11. A kind of application layer protocol identification characteristic mining method according to claim 10, it is characterized in that, described transaction comprises four attribute fields of encoded byte, packet length, byte percentage and packet percentage, store extracted The data. 12. A kind of application layer protocol identification characteristic mining method according to claim 1, is characterized in that, described step C comprises the following steps: C1. Using frequency correction and frequent item filtering methods, performing the first frequent item filtering, frequency correction and the second frequent item filtering on the multi-level frequent item set; C2. Convert the frequent items in the corrected and filtered frequent item set into feature strings, mine the absolute packet length feature and the feature of the relationship between packet length and content difference, and mark the corresponding transaction, and then identify the quasi-protocol corresponding to the transaction The feature data information is filtered to obtain the final protocol identification feature. 13. A kind of application layer protocol identification characteristic mining method according to claim 12, is characterized in that, described step C1 comprises the following steps: Step C11, performing the first frequent item filtering on the multi-level frequent item set; Step C12, perform frequency correction and second frequent item filtering on the multi-level frequent item set after the first frequent item filtering, and eliminate the inclusion relationship between frequent items. 14. A kind of application layer protocol identification characteristic mining method according to claim 13, is characterized in that, described step C12 comprises the following steps: Step C121, using the following formula to correct the frequency of the first-level frequent items; ${\mathrm{<span\; class="notranslate">\; freq</span>}}_{\mathrm{<span\; class="notranslate">\; new</span>}}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; freq</span>}}_{\mathrm{<span\; class="notranslate">\; old</span>}}<span\; class="notranslate">\; ;</span>& {\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0,0.9</span>}<span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; freq</span>}}_{\mathrm{<span\; class="notranslate">\; old</span>}}<span\; class="notranslate">\; ;</span>& {\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; 0,0</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\end{array}\right.$ Among them, freq _new represents the new frequency after correction, freq _old represents the frequency of frequent items before correction, pos _i represents the position of the i-th single item in frequent items, i starts from zero, and k represents the frequency of single items in frequent items number; the position of a single item is numbered from zero, k ₁ is a constant, and f(pos ₀ ) is a continuous monotonically decreasing function; Step C122, using the following formula to correct the frequency of the 2-level frequent item;

Among them, k ₂ is a constant, f((pos ₁ -pos ₀ )) is a continuous monotone decreasing function; Step C123, at first, use the following formula to find the average distance between items in frequent items, which is recorded as ave _dist ; ${\mathrm{<span\; class="notranslate">\; ave</span>}}_{\mathrm{<span\; class="notranslate">\; dist</span>}}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; pos</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}}{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}$ The distance refers to the absolute value of the difference between the positions of two adjacent single items in the item; if ave _dist ≠1, use the following formula to correct ave _dist ;

Among them, k ₃ and k ₄ are constants; Then, the following formula is used to modify the frequency of frequent items of the 3rd and 4th grades;

Among them, f ₁ (k) is a continuous monotonically increasing function about k; f ₂ (ave _dist ) and f ₃ (ave _dist ) are continuous monotonically decreasing functions about ave _dist , and satisfy f ₂ (ave _dist )＞f ₃ (ave _dist ); Step C124, filtering out the frequent items with small frequency among the frequent items having inclusion relationship. 15. A kind of application layer protocol identification feature mining method according to claim 12, is characterized in that, described step C2 comprises the following steps: Step C21, converting the frequent items in the corrected and filtered frequent item set into a feature string, retrieving transactions satisfying the feature string from the transaction database, mining the absolute packet length feature and the packet length and content difference relationship feature, And mark the transactions that satisfy the characteristics of the absolute packet length and the relationship between the packet length and the content difference; Step C22, filtering the quasi-protocol identification feature data information corresponding to the transaction to remove repeated mining features, weak features and suspected mixed features, to obtain the final protocol identification feature.

Images