CN114389792B - WEB log NAT (network Address translation) front-back association method and system - Google Patents

Abstract

The invention provides a method and a system for correlating WEB log NAT (network address translation) front and back, which comprise the following steps: acquiring network traffic before and after NAT; carrying out IP/TCP protocol analysis on the data packets before and after the NAT to acquire network quintuple information; searching the TCP stream existing state in a TCP stream database by taking the quintuple as a key; and checking a TCP flag bit, calculating a final hash value H when the TCP stream is ended, generating a final log together with the quintuple, and storing the final log in a log database. The method can realize the accurate combination of the logs segmented by the NAT under any scene without depending on NAT equipment or developing the logs customized for each application, is a simple method, realizes the accurate combination of the segmented logs, provides an accurate data source for big data analysis, can realize the purpose of not consuming a large amount of computing resources and does not need to be integrated with other systems under the actual environment.

Description

WEB log NAT (network Address translation) front-back association method and system

Technical Field

The invention relates to the technical field of computers, in particular to a method and a system for correlating WEB log NAT (network Address translation) front and back.

Background

Whether cloud application services or applications developed by enterprises, WEB services provided by HTTP or HTTPs protocols are basically the main form. In order to monitor the security and performance problems of these applications, enterprises often need to deploy a large number of probes (collectors) at different network nodes to collect traffic, generate logs related to these applications, and then track the traceable security events or investigate the performance bottleneck of business applications through big data analysis. Because of the complexity of the Network, the traffic accessing the service application often passes through a firewall or a load balancing device, and these devices may be configured with a Network Address Translation (NAT-Network Address Translation) to convert the original IP/port of the Network five-tuple (source IP, source port, destination IP, destination port, protocol) into a new IP Address or port. Therefore, before and after NAT, five groups of WEB logs acquired by a traffic collector cannot be matched, so that the logs belonging to the same traffic are divided into two unrelated groups, and the purpose of completely analyzing the whole data stream cannot be realized. Because the NAT is completed by a firewall or a third-party device, the association relationship between the IP/ports before and after the translation is logic inside the NAT device itself, and the probe deployed in the network cannot acquire the association relationship.

Disclosure of Invention

The invention aims to provide a method and a system for correlating WEB log NAT (network address translation) front and back so as to solve the problems in the background technology.

In order to achieve the purpose, the invention provides the following technical scheme:

a method for correlating WEB log NAT (network Address translation) front and back comprises the following steps:

step S101, network traffic before and after NAT is obtained;

step S102, IP/TCP protocol analysis is carried out on the data packets before and after the NAT, and network quintuple information is obtained;

step S103, taking the quintuple as key, and searching the existing state of the TCP stream in a TCP stream database;

step S104, checking a TCP flag bit, calculating a final hash value H when the TCP stream is finished, generating a final log together with the quintuple, and storing the final log into a log database;

s105, searching the collected and stored logs related to the NAT source IP or the target IP by taking the hash value H of the log X as a search in a given time range, and searching the logs matched with the same hash value;

and S106, combining the two logs X and Y after the log Y with the same hash value is matched.

Preferably, the network five-tuple information includes two groups, which are group a and group B, respectively, and both the two groups of information include: source IP, source port, destination IP, destination port, and protocol name.

Preferably, if the TCP flow does not exist, checking whether the data packet has a TCP handshake message, wherein if the TCP handshake message has the TCP handshake message, a data structure entry of the TCP flow is newly established in the database with the quintuple as a key, the expected sequence number is recorded, and a hash algorithm is selected to generate the initial value H, and if the TCP handshake message does not exist, the message is ignored.

Preferably, the hashing algorithm includes, but is not limited to, MD5, SHA-1 and SHA-2; the initial value H = HashInit (), and the TCP stream includes a five-tuple and an expected sequence number S.

Preferably, the TCP flag bit is checked, and if the TCP flag bit is RST or FIN, the TCP stream is indicated to be ended; the final hash value H = hashfinal (H), and the log contents include a timestamp, a quintuple, and the final hash value H.

Preferably, whether the data packet contains HTTP or HTTPs layer data is checked, and if not, the message is ignored; the sequence number of the check packet is the expected sequence number S: a) if the value is smaller than the expected value, the message is a TCP retransmission message, and the message is ignored; b) if the value is larger than the expected value, temporarily storing the value and stopping the subsequent steps; c) if the data length is equal to the expected value, calculating the data length of the application layer and adjusting the expected value; d) checking whether the stored packet sequence number and the new expected value are equal, if so, repeating step c) and merging the data in the message.

Preferably, the hash value is calculated by updating the newly received application layer data, and then the IP/TCP protocol analysis is performed on the data packet to obtain the network five-tuple information: if the received data packet is HTTP, the collector analyzes the HTTP protocol header, only calculates the hash value H for the HTTP header and updates the TCP stream database table entry; if the received data packet is HTTPs, the hash value H needs to be calculated for the entire TCP protocol content, and the TCP flow database entry is updated, H = HashUpdate.

Preferably, starting a timing working program, searching logs matched with the same hash value by taking the hash value H of the log X as a search in a given time range for collected and stored logs related to the NAT source IP or the target IP; and after matching the log Y with the same hash value, merging the two logs X and Y, selecting to keep the log before NAT or after NAT by configuration, deleting the log after NAT if selecting to keep the log before NAT, and deleting the log before NAT if selecting to keep the log after NAT.

In order to achieve the above purpose, the invention also provides the following technical scheme:

a WEB log NAT front-back association system comprises:

the acquisition module is used for acquiring network traffic before and after the NAT;

the analysis module is used for carrying out IP/TCP protocol analysis on the data packets before and after the NAT to acquire network quintuple information;

the retrieval module is used for retrieving the existing state of the TCP stream in the TCP stream database by taking the quintuple as a key;

the generating module is used for checking the TCP flag bit, calculating a final hash value H when the TCP stream is ended, generating a final log together with the quintuple and storing the final log into a log database;

the search matching module is used for searching the collected and stored logs related to the NAT source IP or the target IP by taking the hash value H of the log X as a search in a given time range and matching the logs with the same hash value;

and the merging module is used for merging the two logs X and Y after the log Y with the same hash value is matched.

Wherein: the acquisition module may be a switch or other network equipment, and the analysis module may be a traffic collector, etc.

In order to achieve the above purpose, the invention also provides the following technical scheme:

a computer device comprising a memory storing a computer program and a processor implementing the steps of the method as claimed in any one of the above when the computer program is executed.

In order to achieve the above purpose, the invention also provides the following technical scheme:

a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of the preceding claims.

Compared with the prior art, the invention has the beneficial effects that:

the method can realize the accurate combination of the logs segmented by the NAT under any scene without depending on NAT equipment or developing the logs in a customized manner aiming at each application.

The invention is a simple method to realize the correct combination of the divided logs, thereby providing an accurate data source for big data analysis.

The realization method of the invention is simple, does not consume a large amount of computing resources, and does not need to be integrated with other systems in the actual environment.

Drawings

FIG. 1 is a block diagram of the method of the present invention;

FIG. 2 is a block diagram of the system of the present invention;

FIG. 3 is a diagram of a conventional NAT topology that the present invention is required to address;

FIG. 4 is a diagram of a TCP header field used for parsing by the collector of the present invention;

fig. 5 is an internal structural view of the computer device of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example (b):

referring to fig. 1 to 5, the present invention provides a technical solution:

when network traffic passes through nat (network Address translation) devices, both the IP Address and the port of the network five-tuple may change. Sessions that belong to the same business application are split into two unrelated sessions, and when such logs are collected, they are a catastrophic error to the log-dependent analysis system. The invention aims to introduce a simple method to realize correct combination of the segmented logs, thereby providing an accurate data source for big data analysis.

The problem to be solved by the present invention is how to correctly merge the partitioned weblogs under the condition that the partitioned weblogs cannot be integrated with the NAT device, so as to realize the complete analysis of the whole traffic.

To solve the above problem, there may be different methods according to different environments and scenarios, as follows: 1) and configuring a firewall or load balancing to export NAT logs (wherein the five-tuple before the NAT and the five-tuple after the NAT are required to be included) to a big data platform, and associating different WEB together by the relationship. However, this method has the following disadvantages: 1) firewalls or load balancing devices may not support exporting such logs; 2) even if the export can be carried out, the working performance of the equipment is obviously influenced; 3) the storage and calculation pressure of a large data platform is increased.

2) And analyzing each specific WEB application by using a probe, extracting the session ID of the application from the HTTP protocol of the traffic before and after the NAT, adding the session ID into a WEB log, and sending the WEB log to a big data platform. And the data analysis system uses the Session ID as the correlation evidence of the application to correlate the WEB logs before and after the NAT, and combines the correlation logs. However, the method needs to develop each different application individually, not only is the workload large, but also once a new application appears in the network, complete analysis cannot be realized, and no method is available for all.

The method can realize the accurate combination of the logs segmented by the NAT under any scene without depending on NAT equipment or developing the logs in a customized way aiming at each application.

A simple method for correlating WEB log NAT (network Address translation) front and back comprises the following steps:

1) and mirroring the network traffic before and after the NAT to a network traffic collector through a switch or other network equipment.

2) The flow collector needs to have network protocol analysis capability, and respectively carries out IP/TCP protocol analysis on the received data packets before and after the NAT to obtain network quintuple information: group a (source IP, source port, destination IP, destination port, protocol name) and group B (source IP, source port, destination IP, destination port, protocol name).

3) And searching whether the TCP stream exists in a TCP stream database by taking the quintuple as a key, if so, jumping to a step 4), and if not, checking whether the data packet has a TCP handshake message.

a) If yes, a data structure table entry of the TCP flow is newly established in the database by taking the quintuple as a key, an expected sequence number (sequence number) is recorded, and a hash algorithm is selected and an initial value is generated. MD5, SHA-1 and SHA-2 are commonly used.

H = HashInit()；

TCP flow table: key (quintuple), value (expected sequence number S and hash value H);

b) if not, ignoring the message;

HashInit is an initialization function corresponding to the selected hash algorithm, where H is an initial value

The TCP flow table is a data structure table entry corresponding to the TCP flow, and includes a key corresponding to the TCP flow and a value to be stored.

4) Checking the TCP flag bit, jumping to step 5) if there is neither RST nor FIN, otherwise, if there is RST or FIN, indicating that the TCP stream is finished, calculating the final hash value H, and generating a final log together with the quintuple and storing the final log in a log database. Then jump to step 8)

H= HashFinal(H)；

The log content is as follows: timestamp + quintuple + H + others;

RST and FIN are flag bits defined in the TCP protocol, the former represents resetting, the latter represents ending, the flag is seen to indicate that no data exists later, and the final hash value can be calculated;

the HashFinal is a finishing function corresponding to the selected hashing algorithm to generate the final hash value.

5) Checking whether the data packet contains HTTP or HTTPs layer data, if yes, jumping to step 6), and if not, ignoring the message

6) It is checked that the sequence number of the data packet is the expected sequence number S,

a) if the value is smaller than the expected value, the message is a TCP retransmission message, and the message is ignored;

b) if the value is larger than the expected value, temporarily storing the value and stopping the subsequent steps;

c) if equal to the expected value, the application layer data length is calculated and the expected value is adjusted. For example, if the original expected value is 5 and the newly received data length is 6, the new expected value is: expected serial number S =5+6= 11;

d) checking whether the stored data packet sequence number is equal to the new expectation value, if so, repeating the step c) and merging the data in the message, and if not, jumping to the step 7).

7) Updating and calculating the hash value of the newly received application layer data, and then returning to the step 2);

a) if the received data packet is HTTP, the collector analyzes the HTTP protocol header, only calculates the hash value H for the HTTP header and updates the TCP stream database table entry.

H=HashUpdate(H, HTTP_header, header_len)；

b) If the received data packet is HTTPs, because the data is encrypted, the hash H needs to be calculated for the entire TCP protocol content and the TCP stream database entry needs to be updated.

H=HashUpdate (H, tcp_data, data_len)；

8) Starting a timing working program, searching logs matched with the same hash value by taking the hash value H of the log X as retrieval in a given time range for the collected and stored logs related to the NAT source IP or the target IP.

9) And when the log Y with the same hash value is matched, merging the two logs X and Y. The choice of keeping the log before or after NAT can be made by configuration.

a) And if the log before the NAT is selected to be reserved, deleting the log after the NAT.

b) And if the log after the NAT is selected to be reserved, deleting the log before the NAT.

As shown in fig. 3, is a typical topology with NAT generation. An intranet user uses a PC terminal to access a Web application oa.abc.com of an enterprise, the IP address of the intranet user is assumed to be 100.100.100.100, traffic can pass through a switch firstly, the switch can mirror the traffic to a traffic analysis device HF, the HF sees the original traffic at this time, then the switch transfers the traffic to a firewall above, the firewall finds that the 100.100.100.100 corresponding to the oa.abc.com is 192.168.100.100 of an intranet actually, the switch can perform NAT conversion and then re-gives the traffic to the switch, and the switch can mirror the traffic after NAT to the traffic analysis device HF again and simultaneously forwards the traffic to a real destination device oa.abc.com server.

The invention solves the problem of log association segmented by NAT by a simple method. The key point is what is used to uniquely identify two seemingly unrelated logs.

The realization method of the invention is simple, does not consume a large amount of computing resources, and does not need to be integrated with other systems in the actual environment.

In the present invention, a computer device may include a memory, a storage controller, one or more processors (only one shown in the figure), and the like, which are electrically connected directly or indirectly between the elements to realize the transmission or interaction of data. For example, electrical connections between these components may be made through one or more communication or signal buses. The method for pre-and-post association of the WEB log NAT includes at least one software functional module that can be stored in a memory in the form of software or firmware (firmware), for example, the software functional module or the computer program included in the system for pre-and-post association of the WEB log NAT. The memory may store various software programs and modules, such as program instructions/modules corresponding to the method and system for associating the WEB log NAT and the system provided in the embodiment of the present application. The processor executes various functional applications and data processing by running software programs and modules stored in the memory, that is, implements the parsing method in the embodiments of the present application.

The invention, the remaining parts not described, are the same as, or known or realizable by the prior art and will not be described in detail here.

Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims (7)

1. A method for correlating a WEB log NAT (network Address translation) front and back is characterized by comprising the following steps:

acquiring network traffic before and after NAT;

carrying out IP/TCP protocol analysis on the data packets before and after the NAT to acquire network quintuple information;

searching the TCP stream existing state in a TCP stream database by taking the quintuple as a key;

if the TCP stream exists, checking a TCP flag bit, calculating a final hash value H after the TCP stream is ended, generating a final log together with the quintuple, and storing the final log into a log database;

searching the collected and stored logs related to the NAT source IP or the target IP by taking the Hash value H of the log X as a search within a given time range, and searching the logs matched with the same Hash value;

when the log Y with the same hash value is matched, combining the two logs X and Y;

if the TCP flow does not exist, checking whether the data packet has a TCP handshake message, wherein if the TCP handshake message exists, a quintuple is taken as a key to newly establish a data structure table entry of the TCP flow in a database, an expected sequence number is recorded, a hash algorithm is selected to generate an initial value H, and if the TCP handshake message does not exist, the message is ignored;

checking whether the data packet contains HTTP or HTTPs layer data, and if not, ignoring the message; the sequence number of the check packet is the expected sequence number S: a) if the value is smaller than the expected value, the message is a TCP retransmission message, and the message is ignored; b) if the value is larger than the expected value, temporarily storing the value and stopping the subsequent steps; c) if the data length is equal to the expected value, calculating the data length of the application layer and adjusting the expected value; d) checking whether the stored packet sequence number and the new expected value are equal, if so, repeating step c) and merging the data in the message.

2. The method of claim 1, wherein the five-tuple information of the network includes two groups, namely group a and group B, and the two groups of information both include: source IP, source port, destination IP, destination port, and protocol name.

3. The method of claim 1, wherein the hash algorithm includes MD5, SHA-1, and SHA-2; the initial value H = HashInit (), and the TCP stream includes a five-tuple and an expected sequence number S.

4. The method of claim 1, wherein the flag bit of TCP is checked, and if the flag bit is RST or FIN, the TCP stream is terminated; the final hash value H = hashfinal (H), and the log contents include a timestamp, a quintuple, and the final hash value H.

5. The method of claim 1, wherein the hash value of the newly received application layer data is updated and calculated, and then the IP/TCP protocol analysis is performed on the data packet to obtain the network quintuple information: if the received data packet is HTTP, the collector analyzes the HTTP protocol header, only calculates the hash value H for the HTTP header and updates the TCP stream database table entry; if the received data packet is HTTPs, the hash value H needs to be calculated for the entire TCP protocol content, and the TCP stream database entry is updated, H = HashUpdate.

6. The method of claim 1, wherein a timed working procedure is started, and logs related to a source IP or a target IP of the NAT are collected and stored, and a hash value H of a log X is used as a search within a given time range to search for logs matching the same hash value; and after the log Y with the same hash value is matched, combining the two logs X and Y, selecting to keep the log before NAT or after NAT by configuration, deleting the log after NAT if selecting to keep the log before NAT, and deleting the log before NAT if selecting to keep the log after NAT.

7. A WEB log NAT front-back association system is characterized by comprising:

the acquisition module is used for acquiring network traffic before and after the NAT;

the analysis module is used for carrying out IP/TCP protocol analysis on the data packets before and after the NAT so as to obtain network quintuple information;

the retrieval module is used for retrieving the existing state of the TCP stream in the TCP stream database by taking the quintuple as a key;

the generating module is used for checking a TCP flag bit if the TCP stream exists, calculating a final hash value H after the TCP stream is ended, generating a final log together with a quintuple and storing the final log into a log database;

the searching and matching module is used for searching the collected and stored logs related to the NAT source IP or the target IP by taking the Hash value H of the log X as a search in a given time range and matching the logs with the same Hash value;

the merging module is used for merging the two logs X and Y after the log Y with the same hash value is matched;

if the TCP flow does not exist, checking whether the data packet has a TCP handshake message, wherein if the TCP handshake message exists, a quintuple is taken as a key to newly establish a data structure table entry of the TCP flow in a database, an expected sequence number is recorded, a hash algorithm is selected to generate an initial value H, and if the TCP handshake message does not exist, the message is ignored;

checking whether the data packet contains HTTP or HTTPs layer data, and if not, ignoring the message; the sequence number of the check packet is the expected sequence number S: a) if the value is smaller than the expected value, the message is a TCP retransmission message, and the message is ignored; b) if the value is larger than the expected value, temporarily storing the value and stopping the subsequent steps; c) if the data length is equal to the expected value, calculating the data length of the application layer and adjusting the expected value; d) checking whether the stored packet sequence number and the new expected value are equal, if so, repeating step c) and merging the data in the message.

Images