WO2001074028A1

WO2001074028A1 - System for processing log data

Info

Publication number: WO2001074028A1
Application number: PCT/GB2001/001167
Authority: WO
Inventors: Christopher James Davies
Original assignee: British Telecommunications Public Limited Company; Drickey, John, Victor
Priority date: 2000-03-29
Filing date: 2001-03-16
Publication date: 2001-10-04
Also published as: AU4087001A

Abstract

A method is described for processing log data from a set of firewalls between the public Internet (12) and a private intranet (14). The log data comprises a series of records, each of which relates to a data traffic session originating at a source address and specifying a destination address. The set of firewalls are arranged to accept or reject each session on a selected basis. In the method, log data is transferred from the firewalls (10) to a computer system (16). In the computer system (16), the source address in each record is reduced to the corresponding network address, which then regarded as the source address. Next, duplicated sessions are counted and a record is generated from each set of duplicated sessions. Two sessions are regarded as duplicates if they originate at the same source address and specify the same destination address and port number at the destination address. Records relating to accepted sessions are discarded and records which relate to sessions originating at specified known networks are also discarded.

Description

SYSTEM FOR PROCESSING LOG DATA

This invention relates to a method of processing log data from a data network. More particularly, but not exclusively, the invention is concerned with a method of processing log data from a firewall separating at least two data networks.

As is well known, a firewall, or more usually a set of firewalls, is used to control the passage of data traffic sessions between two or more data networks.

Each data traffic session originates at a source address and specifies a destination address. A firewall, or each firewall in a set of firewalls, will be configured to accept or reject data traffic sessions on a selected basis. For example, a firewall may be arranged to prevent a user in one network from accessing a data server in another access unless that user has appropriate authorisation.

A firewall may also be arranged to generate log data, such log data comprising a series of records, each of which relates to a data traffic session. Such log data is typically produced in very large quantities and is, consequently, therefore difficult to analyse. For example, in the case of a set of firewalls which separate a private intranet belonging to a large organisation from the public Internet, the logged data may comprise many millions of records each week.

In accordance with a first aspect of this invention, there is provided a method of processing log data from a firewall separating at least two data networks, said data comprising a series of records, each record containing log data relating to a data traffic session originating at a source address and specifying a destination address, said firewall being arranged to accept or reject each session on a selected basis, said method comprising the steps of: transferring log data from the firewall to a computer system for processing; counting duplicated sessions, two sessions being taken as duplicated if, at least, they originate at the same source address and specify the same destination address; generating a record for each set of duplicated sessions; and generating a report from said records relating to duplicated sessions. Preferably, prior to counting the duplicated sessions, the source address in each record relating to a session is reduced to a network address, the network address being taken as a source address for the subsequent steps.

Preferably, records which relate to accepted sessions are discarded. Conveniently, records which relate to sessions originating at specified networks are discarded.

According to another aspect of this invention, there is provided a method of processing log data from a data network, said data comprising a series of records, each record containing log data relating to a data traffic session originating at a source address and specifying a destination address, said method comprising the steps of: transferring log data to a computer system for processing; counting duplicated sessions, two sessions being taken as duplicated if, at least, they originate at the same source address and specify the same destination address; generating a record for each set of duplicated sessions; generating a report from said records relating to duplicated sessions. This invention will now be described in more detail, by way of example, with reference to the accompanying drawings in which: Figure 1 is a block diagram of a set of firewalls which are arranged to control data traffic between the public Internet and a private intranet together with a computer system for analysing log data produced by this set of firewalls.

Figure 2 is a block diagram of one of the computers forming part of the computer system of Figure 1 ;

Figure 3 is a flow chart illustrating a method of processing log data which embodies this invention; and

Figures 4 and 5 are each a flow chart illustrating part of the process of analysing log data, shown in Figure 3, in greater detail. Referring now to Figure 1 , there is shown a set of firewalls 1 0, which control the passage of data traffic sessions between a public Internet 1 2 and a private intranet 14. Each data traffic session originates at a source address in either the public Internet 1 2 or the private intranet 1 4 and specifies a destination address. The firewalls 10 either accept or reject each data traffic session. When a session is accepted, it is permitted to pass through the firewalls 1 0. In contrast, when a session is rejected it is prevented from passing through the firewalls 1 0. The firewalls 10 generate log data. The log data comprises a series of records, each of which relates to a data traffic session. The log data is transmitted to a computer system 1 6. Usually, the rate at which log data is generated by a set of firewalls is very high. For example, in the case of a set of firewalls which control the passage of data between the public Internet and a private intranet belonging to a large organisation, many millions of records of log data are generated each week. Within the computer system 1 6, the log data is processed and a report is generated at periodic intervals. In this example, as a result of processing the log data in the computer system 1 6, in generating each report the quantity of data records is reduced by a factor of 1 00,000. Also, each report is presented in a way in which it can be analysed easily.

Within the set of firewalls 1 0, each individual firewall takes the form of a computer which is arranged to analyse traffic and to accept or reject each session in accordance with a set of rules. A rejected session is also called a dropped session. The firewalls are arranged as logical groups. In this example, each logical group comprises a resilient pair of identical physical firewalls. Each resilient pair of firewalls is arranged to perform a particular function. For example, one resilient pair may be arranged to accept or reject sessions relating to a TCP (Transmission Control Protocol) service. For such a resilient pair, the rules may specify that only sessions which originate from specified addresses will be accepted. A second resilient pair may be arranged to reject a session in which analysis of the .incoming data shows that it contains a hacking algorithm. A third resilient pair may be arranged to reject a session received from the public Internet 1 2, if, on analysing the incoming data, it is found that the purpose of the session is to scan the ports of an item of equipment in the intranet 1 4. A fourth resilient pair may be arranged to reject a session originating in the intranet 1 4 but which is destined for an unauthorised address in the public Internet 1 2. In general each firewall will have a complete security policy which depends on the requirements of the networks in question. Each individual physical firewall has an address and also a name. Each resilient pair has a name. Also, each individual physical firewall has an input interface and an output interface and each interface has a name.

The computer system 1 6 comprises a store and forward computer 1 8 and a data processing computer 20. In this example, each day a set of twelve log files is transmitted from the firewall 1 0 to the store and forward computer 1 8. Each file contains approximately two hours worth of log data. However, the length of time represented by each file is configured in the firewall and a longer or shorter time period may be used. The data processing computer 20 requests each log file in turn from the store and forward computer 1 8, processes the records and produces a report at periodic intervals. The report is transmitted to a server within the private intranet 14.

The computers 1 8 and 20 are constructed physically in an identical manner and the construction of one of these computers is shown in Figure 2. The construction of the computer shown in Figure 2 is conventional and comprises a central processing unit (CPU) 30, keyboard 32, display unit 34, input and output ports 36, a mouse 38 and a store in the form of a hard drive 40, random access memory (RAM) 42 and read only memory (ROM) 44 connected to a central bus 46. The operating system, which in this example is the well-known UNIX operating system, and the individual application programs are held in this store.

In this example, the log files generated by the firewalls 10 during a week are stored in a computer 1 8. At the end of the week, the files are processed in the computer 20.

In each file, the individual records are prepared in a proprietary format. An example of a record in the proprietary format is shown below:

23:59:00 drop 1 93.1 1 3.1 39.1 69 > nfO proto udβ src 1 47.1 50.204.220 dst 1 93.0.14.1 29 service domain-udp s_port domain-udp len 62 rule 29

The record shown above comprises a series of fields separated by spaces. In the record shown above the individual fields are as follows: 1 . GMT time hh:mm:ss (log file date is written as the first record, e.g. "Date: Apr 1 5, 98", and written again later in a file if the recorded date changes)

2. Action, accept (accept) or drop 3. Registered firewall interface address (ignored)

4. " > " (inbound) or possibly " < " (outbound) followed immediately by firewall interface name

5. "proto" followed by protocol name or number, e.g. TCP

6. "src" followed by originating machine (source) IP (Internet Protocol) address

7. "dst" followed by destination machine IP address

8. "service" followed by destination machine port name or number

9. "s_port" followed by originating machine (source) port name or number

10. "len" followed by data length 1 1 . "rule" followed by the firewall rule number which triggered this action

In the data processing computer 1 6, in an initial operation, the individual records are converted from the proprietary format to a standardised format. More specifically, as shown in Figure 3, for each physical firewall in turn, the individual records are converted in a step 50 to the standardised format.

An example of a record in the standardised format is shown below:

1 998-04-1 5 23:59:00 a hmeO btdsfmy udp 1 47.1 50.204.23 domain-udp 1 93.0.1 4.1 29 domain-udp pe 29

In the record shown above, the individual fields are as follows:

1 . GMT date (yyyy-mm-dd)

2. GMT time (hh:mm:ss) 3. Flag indicating whether the session was accepted ("a") or dropped

("d") 4. Name of the firewall from which the session originated 5. Name of the firewall interface from which the session originated

6. Protocol name or number (e.g. tcp,)

7. Originating (source) machine IP address

8. Originating (source) port name or number 9. Destination machine IP address

10. Service requested on destination machine (i.e. destination port name or number)

1 1 . Composite flag indicating whether the source port is privileged ("p") or not ("n"), followed by an indication ("e") if the source port and destination port numbers are equal

1 2. Firewall rule number which triggered this action

After standardising the records for the files from all of the individual physical firewalls, the computer 1 6 moves to the next operation. In this operation, the records are summarised, in a step 51 , for each individual physical firewall in turn. Step 51 is described in more detail below.

In the next operation, in a step 52, the individual records are summarised for each logical firewall group in turn. As mentioned above, in this example, each logical firewall group comprises a resilient pair of identical firewalls.

Then, in a step 53, a report is generated and, in a step 54, the report is transferred to a server in the private Intranet 14.

Referring now to Figure 4, there are shown the individual sub-steps which are performed during each pass through the step 51 . Initially, in a sub-step 60, each source and each destination address is reduced to the corresponding network address. Each source or destination address is reduced to its network address by performing a look-up operation in a known network database and then replacing the full IP address with a corresponding network address. The reason for reducing the source and destination addresses to network addresses is as follows. If a person is attempting to gain access to the private intranet 14 from a network connected to the public Internet, it should not be expected that this person will mount the entire attack from a single IP address within the network which the person is using. Instead, it is more likely that the person will mount the attack from several different IP addresses within the network which they are using. By reducing the addresses to network addresses, this will merge attacks made from a particular network, thereby highlighting a potential problem.

Next, in a sub-step 61 , duplicate sessions are counted. Two sessions are considered to be duplicates if the source address, destination address and destination port number are equal . In this step, an individual record is produced for each set of duplicated sessions. By way of example, four such records are shown below:

• 21 97 d hmeO btdsfmy udp 1 47.1 50.0.0 domain-udp 1 93.0.1 4.0 domain-udp pe 29 • 48 d hmeO udp 1 28.33.0.0 domain-udp 147.148.1 4.0 domain- udp pe 29

• 2 d hmeO tcp 1 93.1 1 7.220.0 1 0293 1 92.1 68.76.0 telnet n 29

• 1 3 a qfel tcp 1 0.0.0.0 8203 1 2.0.0.0 s tp n 25.

Next, in a sub-step 62, in the records generated in step 61 , the records which relate to accepted sessions are discarded.

Next, in a sub-step 63, the records generated in sub-step 61 and which remain after performing sub-step 62 are grouped by source network. After performing sub-step 63, in a sub-step 64, from the records generated in sub-step 61 and which remain after performing sub-steps 62 and 63, the records which relate to specified networks are discarded. The object of performing sub-step 64 is to discard records from known networks. For each known network, inbound sessions will be expected at a particular firewall. The computer 1 6 has access to a known networks file. For each known network, this file contains the address of the network and the name of the firewall at which inbound sessions are expected from this network. In sub-step 54, if a record relates to a known network and the firewall name corresponds to the firewall name at which sessions are expected, the record is discarded. If a record relates to sessions from a known network but specifies that the sessions have been received at a firewall which does not correspond to the specified firewall name in the known networks file, then the sessions are recorded as unexpected sessions.

The individual sub-steps which form step 52 will now be described with reference to Figure 5.

As indicated in box 70, there is a list of services on which it is required to obtain session data. In this sub-step 71 , the records are extracted which relate to the services contained in the list of services 70. For each service, the number of records originating from each network is counted and the record is produced for each service for each originating network address. These records are then grouped by the originating network address.

The computer 1 6 also has information, as indicated by box 71 , on the organisation which owns each originating network. In a sub-step 72, the address of the source network is extracted and used to obtain the name of the organisation which owns the network having that address. The name and address of the owning organisation is then added to the record.

In a sub-step 73, the records produced in sub-step 72 are grouped by organisation. An example of the records produced by sub-step 73 is shown below.

64860 1 61 udp 147.2.0.0 147.149.0.0 Novell, Inc...

64804 1 61 udp 1 47.2.0.0 147.147.0.0 Novell, Inc...

58093 1 61 udp 1 50.1 52.0.0 147.1 51 .0.0 Director of...

53451 1 61 udp 1 50.1 52.0.0 1 47.1 40.0.0 Director of...

78952 8 + 0 icmp 1 67.221 .0.0 1 93.1 1 3.21 3.0 Grand Metro.

30239 53 udp 1 34.225.0.0 147.1 49.0.0 Reading Uni .

1 7339 53 udp 1 34.225.0.0 147.1 50.0.0 Reading Uni .

4 1 352 tcp 204.1 62.80.0 147.1 51 .0.0 Cnet, 1 50 C. The field descriptions are:

1 Number of sessions between source and destination networks for the specified service and protocol. 2 Service Number.

3 Protocol name or number.

4 Originating network IP address.

5 Destination network IP address.

6 Description of the owner of the originating network. It should be noted that each service has a number as well as a name.

Although not shown above, preceding the organisation name is a severity indication. This is a number calculated by a proprietary algorithm, with higher numbers indicating higher severity.

As mentioned above, in one form of attack, a session contains instructions to perform port scanning. In sub-step 74, the check is made for records which relate to port scanning and summary records are produced in respect of any such records which are found.

Referring back to Figure 3, after performing steps 50, 51 and 52, and as mentioned above, a report is generated in step 53. The report is generated from the records produced in step 50 and in sub-steps 73 and 74. Then, as also mentioned above, the report is transferred to a server in the private

Intranet 14.

Although the invention has been described with reference to processing log data from a firewall, steps 50 and 51 shown in Figure 3 may also be used to process log data generated at any point within a network.

Claims

1 . A method of processing log data from a firewall separating at least two data networks, said data comprising a series of records, each record containing log data relating to a data traffic session originating at a source address and specifying a destination address, said firewall being arranged to accept or reject each session on a selected basis, said method comprising the steps of: transferring log data from the firewall to a computer system for processing; counting duplicated sessions, two sessions being taken as duplicated if, at least, they originate at the same source address and specify the same destination address; generating a record for each set of duplicated sessions; and generating a report from said records relating to duplicated sessions.

2. A method as claimed in claim 1 , in which two sessions are taken as duplicated if, at least, they originate at the same source address and specify the same destination address and same port number at the specified destination address.

3. A method as claimed in claim 1 , comprising the further step of: prior to counting the duplicated sessions, reducing the source address in each record relating to a session to a network address, the network address being taken as a source address during the subsequent steps.

4. A method as claimed in any one of the preceding claims, comprising the additional steps of: discarding records which relate to accepted sessions.

5. A method as claimed in any one of the preceding claims, comprising the steps of: discarding records relating to sessions which originate at specific networks.

6. A method of processing a log data from a data network, said data comprising a series of records, each record containing log data relating to a data traffic session originating at a source address and specifying a destination address, said method comprising the steps of: transferring log data to a computer system for processing; counting duplicated sessions, two sessions being taken as duplicated if, at least, they originate at the same source address and specify the same destination address; generating a record for each set of duplicated sessions; generating a report from said records relating to duplicated sessions.

7. A method as claimed in claim 6, in which two sessions are taken as duplicated if, at least, they originate at the same source address and specify the same destination address and same port number at the destination address.