GB2317792A - Virtual Private Network for encrypted firewall - Google Patents

Abstract

A system (10) for regulating the flow of messages through a firewall (18) having a network protocol stack, wherein the network protocol stack includes an Internet Protocol (IP) layer where if the message is not encrypted, it passes the unencrypted message up the network protocol stack to an application level proxy (50), and if the message is encrypted, it decrypts the message and passes the decrypted message up the network protocol stack to the application level proxy. The step of decrypting the message includes the step of executing a process at the IP layer to decrypt the message.

Description

2317792 VIRTUAL PRIVATE NETWORK ON APPLICATION GATEWAY RackgMund of thú

Inventi

Field of the Invention

The present invention pertains generally to network communications, and in particular to a system and method for securely transferring information between firewalls over an unprotected network.

BackgroWd Information Firewalls have become an increasingly important part of network design.

Firewalls provide protection of valuable resources on a private network while allowing communication and access with systems located on an unprotected network such as the Internet. In addition, they operate to block attacks on a private network arriving, from the unprotected network by providing a single connection with limited services. A well designed firewall limits the security problems of an Internet connection to a single firewall computer system. This allows an organization to focus their network security efforts on the definition of the security policy enforced by the firewall. An example of a firewall is given in "SYSTEM AND METHOD FOR PROVIDING SECURE INTERNETWORK SERVICES" by Boebert et al. (PCT Published Application No. WO 96/13113, published on May 2, 1996), the description of which is hereby incorporated by reference. Another description of a firewall is provided by Dan Thomsen in "Type Enforcement.. tile new security model% Proceedings: Multimedia: Full-

Service Impact on Business, Education and the Home, SPIE Vol. 2617, p. 143, August 1996. Yet another such system is described in "SYSTEM AND METHOD FOR ACHIEVING NETWORK SEPARATIOW'by Gooderum et al. (PCT Published Application No. WO 97129413, published on August 14,1997), the description of which is hereby incorporated by reference. All the above 30 systems are examples of application level gateways. Application level gateways use proxies or other such mechanisms operating at the application layer to process traffic through the firewall. As such, they can review not only the

2 message traffic but also message content In addition, they provide authentication and identification services, access control and auditing.

Data to be transferred on unprotected networks like the Internet is susceptible to electronic eavesdropping and accidental (or deliberate) corruption.

Although a firewall can protect data within a private network from attacks launched from the unprotected network, even that data is vulnerable to both eavesdropping and corruption when erred from the private network to an exterrial machine- To address this danger, the Internet Engineering Task Force (IETF) developed a standard for protecting data erred between firewalls over an unprotected network. The Internet Protocol Security (IPSE Q standard calls for encrypting data before it leaves the first firewall, and then decrypting the data when it is received by the second firewall. The decrypted data is then delivered to its destination, usually a user workstation connected to the second firewall. For this reason IPSEC encryption is sometimes called_fircwall-to- firewall encryption (FFE) and the connection between a workstation connected to the first firewall and a client or saver connected to the second firewall is termed a virtual private network, or VPN.

The two main components of IPSEC security are data encryption and sender authentication. Data encryption increases the cost and time required for the eavesdropping party to read the transmitted dat& Sender authentication ensures that the destination system can verify whether or not the encrypted data was actually sent from the workstation that it was supposed to be sent from. The IPSEC standard defines an encapsulated payload (ESP) as the mechanism used to transfer encrypted data. The standard defines an authentication header (AH) as the mechanism for establishing the sending workstation's identity.

Through the proper use of encryption, the problems of eavesdropping and corruption can he avoided; in effect, a protected connection is established from the internal network connected to one firewall through to an internal network connected to the second firewall. In addition, IPSEC can be used to provide a protected connection to an external computing system such as a portable personal computer.

3 IPSEC encryption and decryption work within the IP layer of the network protocol stack. This means that all communication between two IP addresses will be protected because all interfirewall communication must go through the IP layer. Such an approach is preferable over encryption and decryption at higher levels in the network protocol stack since when encryption is performed at layers higher than the IP layer more work is required to ensure that all supported communication is properly protected. In addition, since IPSEC encryption is handled below the Transport layer, IPSEC can encrypt data sent by any application, IPSEC therefore becomes a transparent add- on to such protocols as 10 TCP and UDP.

Since, however, IPSEC decryption occurs at the IP layer, it can be difficult to port IPSEC to an application level gateway while still maintaining control at the proxy over authentication, message content, access control and auditing. Although the IPSEC specification in RK 1925 suggests the use of a mandatory access control mechanism in a multi-level secure (NES) network to compare a security level associated with the message with the security level of the receiving process, such an approach provides only limited utility in an application level gateway environment. In fact implementations on application level gateways to date have simply relied on the fact that the message was IPSEC-encrypted as assurance that the message is legitimate and have simply decoded and forwarded the message to its destination. This creates, however, a potential chink in the firewall by assuming that the encrypted communication has access to all services.

What is needed is a method of handling IPSEC messages within an application level gateway which overcomes the above deficiencies. The method should allow control over access by an IPSEC connection to individual services within the internal network.

S M of gal_vention The present invention is a system and method for regulating the flow of messages through a firewall having a network protocol stack, wherein the network protocol stack includes an Internet Protocol (IP) layer, the method 4 comprising the steps of determining, at the 1P layer, if a message is encrypted, if the message is not encrypted, passing the unencrypted message up the network protocol stack to an application level proxy, and if the message is encrypted, decrypting the message and passing the decrypted message up the network protocol stack to the application level proxy, wherein the step of decrypting the message includes the step of executing a procedure at the 1P layer to decrypt the message.

According to another aspect of the present invention, a system and method is described for authenticating the sender of a message within a computer system having a network protocol stack, wherein the network protocol stack includes an Internet Protocol (T) layer, the method comprising the steps of deter, at the IP layer, if the message is encrypted, if the message is encrypted, decrypting the message, wherein the step of decrypting the message includes the step of executing a procedure at the IP layer to decrypt the message, passing the decrypted message up the network protocol stack to an application level proxy, determining an authentication protocol appropriate for the message, and executing the authentication protocol to authenticate the sender of the message.

Brief Desúription of the Drawings In the following detailed description of example embodiments of the invention, reference is made to the accompanying drawings which form a pan hereof, and which is shown by way of illustration only, specific embodiments in which the invention may be practiccd. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.

In the drawings, where like numerals refer to like components throughout the several views:

Figure 1 is a functional block dia of an application level gateway implemented firewall-to-firewall encryption scheme according to the present invention; Figure 2 is a block diagram showing access control checking of both encrypted and unencrypted messages in network protocol stack according to the present invention; Figure 3 is a block diagram of a representative application level gateway- implemented firewall-to-firewall on scheme; Figure 4 is a block diagram of one embodiment of a network-separated protocol stack implementing IPSEC according to the present invention; and Figure 5 is a functional block diagram of a firewall-to-workstation encryption schemc according to the present invention. 10 DescriptiM of the P Emb2dirn In the following detailed description of the preferred embodiment, references made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific prefi=ed embodiments in which the invention may be practiced. These embodiments am described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that structural, logical, physical, architectural, and electrical changes may be made without departing from the spirit and scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims and their equivalents,

A system 10 which can be used for firewall-to-firewall encryption (FFE) is shown in Figure 1. In Figure 1, system 10 includes a workstation 12 communicating through a fir 14 to an unprotectcd network 16 such as the Internet. System 10 also includes a workstation 20 communicating through a firewall 18 to unprotected network 16. In one embodiment, firewall 18 is an application level gateway.

As noted above, IPSEC encryption and decryption work within the IP laycr of the network protocol stack. This means that all communications between two IP addresses will be protected because all interfirewall communication must pass through the IP layer. IPSEC takes the standard 6 Internet packet and converts it into a carrier packet. The carrier packet is designed to do two things: to conceal the contents of the original packet (encryption) and to provide a mechanism by which the receiving firewall can verify the source of the packet (authentication). In one embodiment of the present invention, each IPSEC carrier packet includes both an authentication header used to authenticate the sending machine and an encapsulated payload con encrypted data. The authentication header and the encapsulated payload features of IPSEC ban, however, be used independently. As required in RK 1825, DES-CBC is provided for use in encrypting the encapsulated payload while the authentication header uses keyed MD5.

To use IPSEC, you must create a security association (SA) for each destination IP address. In one embodiment, each SA contains the IbIlowing information., Security Parameters Index (SPI) - The index used to find a SA on receipt of an IP SEC datagram.

Destination IP address - The address used to find the SA and trigger use of IPSEC processing on output, The peer SPI - The SPI value to put on a IPSEC datagram on output, The peer IP address - The destination IP address to be put into the packet header if IPSEC Tunnel mode is used.

The Encryption Security Payload (ESP) algorithm to be used.

The ESP key to used for decryption of input datagrams.

The ESP key to used flar encryption of output datagrams.

The authentication (AH) algorithm to be used.

The AH key to he used for validation of input packcls.

The AH key to be used for generation of the authentication data for output grams.

The combination of a given Security Parameter Index and Destination IP address uniquely identifies a particular "Security Association." In one 7 embodimen the sending firewall uses the sending userid and Deslination Address to select an appropriate Security Association (and hence SPI value).

The receiving firewall uses the combination of SPI value and Source address to obtain the appropriate Security Association.

A security association is normally one-way. An authenticated communications session between two firewalls will normally have two Security Parameter Indexes in use (one in each don). The combination of a particular Security Parameter Index and a particular Destination Address uniquely identifies the Security Association.

More information on the specifics of an IPSEC FFE implementation can be obtained from the standards developed by the IPSEC work group and documented in Security Architecturefor IP (RFC 1825) and in RFC's 1826 1829.

When a datagrarn is received from unprotected network 16 or is to be transmitted to a destination across unprotected network 16, the firewall must be able to determine the algorithms, keys, etc. that must be used to process the datagram correctly. In one embodiment, this information is obtained via a security association lookup. In one such embodiment, the lookup, routine is passed several arguments: the source IP address if the datagrarn is being received from network 16 or the destination IP address if the datagram is to be transmitted across network 16, the SPI, and a flag that is used to indicate whether the lookup is being done to receive or trarismit a datagram.

When an IPSEC datagram is received by firewall 18 from unprotected network 16. the SPI and source IP address are determined by looking in the datagram. In one embodiment a Security Association Database (SADB) stored within firewall 18 is searched for the entry with a matching SPI, In one such embodiment, security associations can be set up based on network address as well as a more granular host address. This allows the network administrator to create a security association between two firewalls with only a couple of lines in a configuration file on each machine. For such embodiments, the entry in the Security Association Database that has both the matching SPI and ihe longest address match is selected as the SA cntry. In another such embodiment, each SA has a prefix length value associated with the address. An address match on a SA entry means that the addresses match for the number of bits specified by the prefix length value.

There are two exceptions to this search process- First, when an SA entry is set marked as being dynamic it implies that the user of this SA may not have a fixed IP address. In this case the match is My determined by the SPI value.

Thus it is necessary that the SPI values for such SA entries be unique in the SADB. The second exception is for SA entries marked as tunnel mode entries.

10' In this case it is normally the case that the sending entity will hide its source address so that all that is visible on the public wire is the destination address. In this case. like in the case where the SA entries are for dynamic 1P addresses, the search is done exclusively on the basis of the SPI.

When transmitting a datagram across unprotected network 16 the SADB is searched using only the destination address as an input. In this case the entry which has the longest address match is selected and returned to the calling routine.

In one embodiment if firewall 18 receives datagrams which are identified as either an IP-PROTO-IPSEC-ESP or IP-PROTO-PSEC-AH protocol datagram, there must be a corresponding SA in the SADB or else firewall 18 will drop the packet and an audit message will be generated. Such an occurrence might indicate a possible attack or it might simply be a symptom of an erroneous key entry in the Security Association Database.

In a system such as system 10, application level gateway fu-ewall 18 acts as a buffer betw een unprotected network 16 and workstations such as workstation 20. Messages coming from unprotected network 16 are reviewed and a determination is made as to whether execution of an authentication and identification protocol is warranted. In contrast to previous systern s, system 10 also performs this same determination on IPSEC-encrypted messages. If desired, the same authentication and identification can be made on messages to be ferred from workstation 20 to unprotected network 16. Figure 2 9 illustrates one way of authenticating both encrypted and unencrypted messages in a system such as system 10.

In the system of Figure 2 a network protocol stack 40 includes a physical layer 42, an Internet protocol (IP) layer 44, a Transport layer 46 and an application layer 48. Such a protocol stark exists, for ins onapplication level gateway firewall 18 of Figure 1 - An application executing in application layer 48 can communicate to an application executing on another system by preparing a message and transmitting it through one of the existing transport services executing on transport layer 46. Transport layer 46 in tuni uses a process executing in 1P layer 44 to continue the transfer. Physical layer 42 provides the software needed to transfer data through the communication hardware (e,g., a network interface card or a modem). As noted above, rPSEC executes within 1P layer 44. Encryption and authentication is transparent to the host as long as the network administrator has the Security Associalion Database correctly configured and a key management mechanism is in place on the firewall.

In application level gateway firewall 18, a proxy 50 operating within application layer 48 processes messages transferred betw= internal and external networks. All network-to-network traffic must pass through one of the proxies within application layer 49 before being the transfer across networks is allowed. A message arriving from external network 16 is examined at 1P layer 44 and an SADB is queried to determine if the source address and SPI are associated with an SA. In the embodiment shown in Figure 2, an SADB Master copy 52 is maintained in persistent memory at application layer 48 while a copy 54 of SADB is maintained in volatile memory within the kernel. If the message is supposed to be encrypted, the message is decrypted based on the algorithm and key associated with the particular SA and the message is transferred up through port layer 46 to proxy 50. Proxy 50 examines the source and destination addresses and the type of service desired and decides whether authentication of the sender is warranted. If so, proxy 50 initiates an authentication protocol. The protocol may be as simple as requesting a user name and password or it may include a challenge/response authentication process. Proxy 50 also looks to see whether the message coming in was encrypted or not and may factor that into whether a particular type of authentication is needed. In Telnet, for instance, user namelpassword authentication may be sufficient for an FFE link while the security policy may dictate that a more stringent challengelresponse protocol is needed for unencrypted links, In that case, proxy 50 will be a Telnet proxy and it will base its authentication protocol on whether the link was encrypted or not.

Since IPSEC executes within IP layer 44 there is no need for host firewalls to update their applications. Users that already have IPSE'C available on their own host machine will, however, have to request that the firewall administrator set up SA's in the SADB for their traffic.

In the embodiment shown in Figure 2, a working copy 54 of the Security Association Database consisting of all currently active SA's is kept resident in memory for ready access by IP layer processing as datagrarns are received and transmitted. In addition, a working msa ter copy 52 of the SADB is maintained in a file in nonvolatile memory. During system startup and initialization processing the content of all of the required SA's in master SADB 52 is added to the working copy 54 stored in kernel memory.

In one embodiment firewall 18 maintains different levels of security on internal and external network interfaces. It is desirable for a firewall to have different levels of security on both the internal and external interfaces. In one embodiment, fircwall 18 supports three different levels, numbered 0 through 2.

These levels provide a simple policy mechanism that controls pernlission for both in-bound and out-bound packets.

Level 0 - do not allow any in-bound or out-bound traffic unless there is a security association between the source and destination.

11 Level 1 - Allow both in-bound and out-bound nonAPSEC traffic but force the use of IPSEC if a SA exists for the address. (To support this firewall 18 must look for a SA for each in-bound datagram.) - Level 2 - allow NULL security associations to exist, NULL associations are just like normal security associations, except no encryption or authentication transform is performed on in-bound or out-bound packets that correspond to this NULL association. With Level 2 led, the machine will still receive unprotected traffic, but it will not umwnit unless Level 1 is enabled.

The default protection level established when the Security Association 10 Database (SADB) is initialized at boot time is 1 for in-bound traffic and 2 for out-'bound traffic.

An Access Control List, or ACL, is a list of rules that regulate the flow of Internet connections through a firewall. These rules control how a firewall's servers and proxies will react to connection attempts. When a server or proxy receives an incoming connection, it performs an ACL check on that connection.

An ACL check compares a set of parameters associated with the connection against a list of ACL rules. The rules detwnine whether the connection is allowed or denied. A rule can also have one or more side effects.

A side effect causes the proxy to change its behavior in some fashion. For example, a common side effect is to redirect the destination JP address to an alternate machine. In addition to IP connection attempts, ACL cheeks can also made on the console logins and on logins made from serial ports. Finally, ACL checks can also be made on behalf of JP access devices, such as a Cisco box, through the use of the industry standard TACACS+ protocol.

In one embodiment the ACL is managed by an aeld daemon running in the kernel of firewalls 10 and 30. The acId daemon receives two types of requests, one to query the ACL and one to administer it. In one such ernbodiment, the ACL is stored in a relational database such as the Oracle database for fast access. By using such a database, query execution is asynchronous and many queries can be executing concurrently. In addition, these types of databases am designed to manipulate long lists of rules quickly 12 and efficiently. These qualities ensure that a given query cannot hang up the process that issued the query for any appreciable time (> 1-2 secoDds). In one such embodiment, the database can hold up to 100,000 users and up to 10,000 hosts but can be scaled up to the capacity of the underlying database en&e. The results of an ACL check is cached allowing repeated checks to be turned around very quicklyApplications on firewalls 10 and 30 can query acId to determine if a given connection attempt should be allowed to succeed. in one embodiment, the types of applications (i.e. "agents") that can make ACL queries can be divided into four classes:

1) Proxies. These allow connections to pass through firewall 10 or 3 0 in order to provide access to a remote service. They include tnauthp (authenticated telnet proxy), pftp (FTP proxy), hrtpp (HTTP proxy), and tcpgsp (TCP generic service proxy).

15.2) Servers. llese provide a service on the firewall itself They include ftpd and httpd.

3) Login agents. Login agent is a program on the fin-wall that can create a Unbx shell. It is not considered a server because it cannot receive IP connections. One example is lusr/binllogin when used to create a dialup session or a console session on firewall 10 or 30. Another example is the command srole.

4) Network Access Servers (NAS). NAS is a remote IP access device, typically a dialup box manufactured by such companies as Cisco or Bridge. The NAS usually provides dialup telnet service and may also provide SLIP or PPP service.

Proxies, servers, login agents, and NASes make queries to acid to determine if a given connection attempt should be allowed to succeed. All of the agents except NAS make their queries directly. NAS, because it is remote, must communicate via an auxiliary daemon that typically uses an industry standard protocol such as RADIUS or TACACS+. The auxiliary daemon (e.g., tacradd) in turn forwards the query to local acId.

13 As a side effect of the query, acld tells the agent if authentication is needed. If no authentication is needed, the connection proceeds immediately. Otherwise acId provides (as another side effect) a list of allowed authentication methods that the user can choose from. The agent can present a menu of choices or simply pick the first authentication method by default, Typical authentication methods include plain password, SNK DSS, SDI SecurID, LOCKout DES, and LOCKout FORTEZZA. In one embodiment, the list of allowed authentication methods varies depending on the host name, user name, time of day, or any combination thereof In the case of a Level 0 policy, it would be safe to assume that all incoming traffic is encrypted or authenticated. In the case of Levels 1 through 2. a determination must he made whether or not a security association exists for a given peer. Otherwise an application may believe that in-bound traffic has been authenticated when it really has not. (That is why it is necessary to look for an SA on input of each non-IPSEC datagram.) In one embodiment, a flag which accompanies the message as it is sent from IP layer 44 to proxy 50 indicates whether the incoming message was or was not encrypted. In another embodiment proxy 50 accesses Security Association Database 54 (the table in the kemel can be queried via an SADB routing socket (PF-SADB)) to determine whether or not a security association exists for a given peer.. The SADB socket is much like a routing socket found in the stock BSD 4.4 kemel (protocol family PF-ROUTE) except that PF-SADB sockets are used to maintain the Security Association Database (SADB) instead of the routing table. Because the private keys used for encryption, decryption, and keyed authentication are stored in this table, access must be strictly prohibited and allowed to only administrators and key management daemons. Care must be taken when allowing user-level daemons access to /dev/mem or /dev/kmern as well, since the keys are stored in kernel memory and could be exposed with some creative hacking.

In one embodiment, a command-line tool called a adb is itsed to support the generation and maintenance of in-kernel version 54 of SADB. The primarf 14 interface between this tool and the SADB is the PF-SADB socket. The kernel provides socket processing to receive client requests to add, update, or change entries in in-kernel SADB 54. As noted above, the default protection level established when the Security Association Database (SADB) is initialized at boot time is 1 for in-bound traffic and 2 for out-bound traffic. This may be changed by the use of the sadb command. The existing sadb command was derived from the NIST implementation of IPSEC. As noted above, this tool is much like route in that it uses a special socket to pass data structures in and out of the kernel. There are three commands recognized by the sadb command: get, set, delete. The following simple shell script supports adding and removing a single SA entry to SADB 54. It shows one embodiment of a parameter order for adding a SA to the SADB.

# /bin/eh if $# -ne 1 1 then echo "usage. so <on>l<cff>" >&2 exit 1 fi ONOFF-$1 addea IPADDRESS=$2 PEER.ADDRESS.O.O.O.0 PREFIXLEN=0 # Num of bits, 0 full 32 bit match LOCALADDRESS=0.0.0.0 REALADDRESS=0.0.0.0 PORTmO PROTOCOL=0 UID=0 DESALG=1 # I = DES-C5C 1VLEN-4 # bytesDESKEY=ObObObObObObObOb DESKEYLENwS # bytes ABALG=1 # 1 = MD5 AHKEY=30313233343536373031323334353637 AHKEYLEN=16 bytes LOCAL-SPI=$1 is PEER-SPI=$1 TUNNEL MODE=0 AHRESULTLEN=4 COMBINED - MODEwl # On output, 1 = ESP, then AH; 0 " AR, then ESP DYWAMIC-FLAG=0 if [ II$0NOFF11 - don@' then 10./sadb add dst $1PADDRESS $PREFIXLEN $LOCAL-SPI $UID $PEERADDRESS $PEER - SPI $TUNNEL-MODE $LOCALADDRESS $REALADDRESS $PROTOCOL $PORT $DESALG $1VLEN $DESKEYLEN $DESKEY $DESKEYLEN $DESKEY $AHALG $ARKEYLEN $AHKEY $AHKEYLEN $AHKEY $AHRESULTLEN $COMBINED-MODE $DYNAMIC-FLAG else /sadb delete dst $IPADDRESS $LOCAL-SPI fi # Get down to work:

addsa 500 172.17.128.115 # numberG.sctc.com The current status of in-kernel SADB 54 can be obtained with the sadb command. The get option allows dumping the entire SADB or a single entry. In one embodiment the complete dump approach uses /dev/kmem to find the information. The infbrmation may be presented as follows # sadb get dst Local-SPI Address-Family Destination-Addr Preflx-length UID Peer-Address Peer-SPI Transport-Type Local-Address Real-Address Protocol Port ESP-Alg_ID ESP IVEC - Length ESP-Enc-Key-length ESP-Enc-ESP_Key ESP-Dec_Key_length ESP-Dec-ESP-Key Al_Alg_ID AR-Data-Length AH-Gen_Key-Length AH - Gen - Key AH-Check-Key_Length AH-dheck-Key Combined-Mode Dynamic-Flag 16 ------------------- ---------------------------------- 500 INET: number6.sctc.com 0 0 0.0.0.0 500 Transport(O) 0 0.0.010 0.0.0.0 None None DES/CBC-RFC1829 (1) 4 8 ObObobObobobobob 8 ObObObObObObobob MD5-RPC182&W 4 16 30313233343536373031323334353637 16 30313233343536373031323334353637 ESP+AH(I) 0 501 INET: spokes.sctc.com 0 0 is 0.0.0.0 501 Transport(O) 0 0.0.0.0.0.0.0.0 None None DES/CBC-RFC1829 (1) 4 8 ObObObobobobObob 8 ObobObObObObobob MD5-RM8280)4 16 30313233343536373031323334353637 1630313233343536373031323334353637 ESP+AH (1) 0 End of list.

When a new entry is added to in-kemel SADB 54, the add process first checks to see that no existing entry will match the values provided in the new entry. If no match is found then the entry is added to the end of the existing SADB list.

To illustrate the use and administration of an FFE, we'll go through an example using FFE 70 in Figure 3. Firewalls 14 and 18 are both application level gateway firls implemented according to the present invention.

Workstations 112 and H3 both want to communicate with F11. For the administrator of firewalls 14 and 18, thb is easy to accomplish. The administrator sets up a line something like this (we'll only show the IP address part and SPI parts of the SA, since they're the tricIdest values to configure. Also, assume that we are using tmel mode):

# Hypothetical SW1 Config File 17 # Fields are laid out in the following manner: # srcaddrornet- localSPI= peeraddr= peerSPI:= realsrcaddr= localaddr= key= 5 # The following entry sets up a tunnel between hosts behind SWI # and hosts behind SW2. src=172. 16.0.0 localSPI-666 peer=192.166.100.5 10 peerSP1=777 \ realsrcaddr=192. 168.100.5 localaclclrs=0.0.0.0 key=oxdeadbeeffadebabe # Hypothetical SW2 Config File is # # Fields are laid out in the following manner: # srcaddrornet= localSPI= peeraddr= peerSPI= realsrcaddr= localaddr" key.

# The following entry sets up a tunnel between hosts behind SWI and # hosts behind SW2. src=172.17.0.0 localSPI=777 peer=192. 168.20.1 peerSPIx.666 \ 25 realsrcaddr=192.168.20.1 localaddr=0.0.0-0 key=Oxdeaelbeeffadebabe With this setup, all traffic is encrypted using one key, no matter who is tiUdng to whom. For example, traffic from H2 to F11 as well as traffic from H3 to Ifl will be encrypted with one key. Although this setup is and simple, it may not be enough.

What happens if H2 cannot trust H3? In this case, the administrator can set up security associations at the host level. In this case, we have to rely on the SPI field of the SA- since the receiving firewall cannot tell from the datagram header which host behind the sending firewall sent the packet- Since the SPI is stored in IPSEC datagrams, we can do a lookup to obtain its value, Below are the sample configuration files for both firewalls again, but this tinie, each host combination communicates with a different key. Moreover, H2 excludes H3 from communications with Ill, and H3 excludes H2 in the same way.

is # Hypothetical SW1 Config File # Fields are laid out in the following manner: # srcaddrornet= localSPI= peeraddr- peerSPI= 5 realsrcaddrlocaladdr= key= # The following entry sets up a secure link between H2 and H1 src=172.16. 0.2 localSPI=666 peer=192.168.100.5 peerSPI=777 \ realsrcaddr=192.168.100.5 localaddrswl?8.17.128.71 \ keyeOxOaOaOaOaOaOaOaOa # The following entry sets up a secure link between H3 and HI src=172.16. 0.1 localSPI=555 peer-192.168.100.5 peerSP1=888 \ realsrcaddr=192.16B.100.5 localaddra=178.17.129.71 \ key=OxObObObObObObObOb # Hypothetical SW2 Config File # Fields are laid out in the following manner: # arcaddrornet= localSPI= peeraddr= peerSPI= realsrcaddr= localaddr= key= # The following entry sets up a secure link between H2 30 and HI arc=172.17.128.71 localSPI=777 peer=192.168.20.1 peerSPI=666 \ realsrcaddr=192.168.20.1 localaddrs=172.16.0.2 key=OxOaOaOaOaOaOaOaOa 35 # The following entry sets up a secure link between H3 and H1 src=172.17. 128.71 localSPI=888 peer=192.168.20.1 peerSPI=555 \ 40 realsrcaddr=192. 168.20.1 localaddrB=172.16.0.1 key=OxObObObObObObObOb Figure 4 is a block diagram showing in more detail one embodiment of an IPSEC-enabled application level gateway firewall 18. Application level gateway firewall 18 provides access control checking of both encrypted and 19 unencrypted messages in a more secure environment due to its networkseparated architecture. Network separation divides a system into a set of independent regions or burbs, with a domain and a protocol stack assigned to each burb. Each protocol stack 40x has its own independent set of data structures, including routing information and protocol information. A givcn socket will be bound to a single protocol stack at creation time and no data can pass between protocol stacks 40 without going through proxy space. A proxy 50 therefore acts as the go-between for transfers between domains. Because of this, a malicious attacker who gains control of one of the regions is prevented from being able to compromise processes executing in other regions. Network separation and its application to an application level gateway is describcd in "SYSTEM AND IvETHOD FOR ACHIEVING NETWORK SEPARATION, U.S. Application No. 081599,232, filed Feb 9, 1996 by Gooderum et al. In the system shown in Figure 4, the in-bound and out-bound datagram processing of a security association continues to follow the conventions defined by the network separation model. Thus all datagrams received on or sent to a given burb remain in that burb once decrypted. In one such embodiment SADB socket 78 has been defined to have the type 'sadb'. Each proxy 50 that requires access to SADB socket 78 to execute its query as to whether the received message was encrypted must have create permission to the sadb type. The following is list of specific requirements that a systeni such as is shown in Figure 4 must provide. Many of the requirements were discussed in the information provided earlier in this document.

1 Firewall applications may query the IPSEC subsystem to determine if 25 traffic with a given address is guaranteed to be encrypted. 2. Receipt of an unencrypted datagram from an address that has a SA results in the datagram being dropped and an audit message being generated. 3. On receipt of eneryptcd protocol datagrams the SADB searches will be done using the SPI as the primary key. The source address will a 30 secondary key. The SA returned by the search will be the SA which matches the SPI exactly and has the longest match with the address.

4. A search of the SADB for a SPI that finds an entry that is marked as SA for a dynamic IP will not consider the address in the search process.

5. A search of the SADB for a SPI that finds an entry that is marked as a SA for a tunnel mode connection will to consider the address if it is (0.0.0. 0) Le INADDR.

6. On receipt of a non-IPSEC datagram the SADB will be searched for an entry that matches the sre address. If a SA is found the datagrarn will be dropped and an audit message sent.

7, SADB searches on output will be done using the DST address as key. If more than one SA entry in the SADB has that address the first one with the maximum address match will be returned.

8. The SADB must be structured so that searches are fast regardless if the search is done by SPI or by address.

9. The SADB must provide support for connections to a site with a fixed is SPI but changing IP address. SA entries for such connections will be referred to as Dynamic Address Sites. or just Dynamic entries.

10. When a dynamic entry is found by a SPI search, the current datagram's SRC address, which is required to ensure that the return datagrams are properly encrypted, will be recorded in the SA only after the AH checking has passed successfully. (This is because if the address is recorded before AH passes then an attacker can cause retilm packets of an outgoing connection to be transmitted in the clear.) 11. A failure of an All check on a dynamic entry results in art audit message.

12. In an embodiment where the firewall requires that, all connections use both AH and ESP, on receipt the order should be AH first ESP second.

13. The processing structure on both input and output should try to minimize the number of SADB required lookups.

Returning to Figure 4, in one embodiment firewall 18 includes a crypto engine interface 80 used to encrypt an IPSEC payload. Crypto engme interface may he connected to a software encryption engine 82 or to a hardware 21 encryption engine 84. Engines 82 and 84 perform the actual encryption function using, for example, DES-CBC. In addition, software encryption engine 82 may include the keyed MD5 algorithm used for AH.

In one embodiment, crypto engine interface 80 is a utility which provides a consistent interface between the software and hardware encryption engines. As shown in Fig= 4, in one such embodiment interface 80 only supports the use of the use of hardware cryptographic engine 84 for IPSEC ESP processing. The significant design issue that interface 80 must deal with is that use of a hardware encryption engine requires that the processing be down in disjoint steps operating in different interrupt contexts as engine 84 completes the various processing steps.

The required infbrmation is stored in a request structure that is bound to the IP datagram being processed. The request is of type crypto. _request - t. This structure is quite large and definitely does not contain a minimum state set.

is In addition to the definition of the request data structure, this software implementing interface 80 provides two functions which isolate the decision of which cryptographic engine to use. The crypt-des-encxwi-function is for use by the 1P output processing to encrypt a datagram. The crypt-des-decrypt function is for use by the IP input processing to decrypt a datagram. If hardware encryption engine 84 is present and other hardware usage criteria are met the request is enqueued on a hardware processing queue and a return code indicating that the cryptographic processing is in progress is returned. If software engine 82 is used, the return code indicates thai the cryptographic processing is complete. In the former case, the'continuation of the IP processing is delayed until after hardware encryption is done. Otherwise it is completed as immediately in the same processing stream.

There are two software cryptographic engines 82 provided in the IPSEC software. One provides the MD5 algorithm used by the IPSEC AH processing, and the other provides the DES algorithm used by the IPSEC ESP processing.

This software can be obtained from the US Government IPSEC implementation.

22 In one embodiment hardware cryptographic engine 84 is provided by a Cylink SafeNode processing board. The interface to this hardware card is provided by the Cylink device driver. A significant aspect of the Cylink card that plays a major part in the design of the IPSEC Cylink driver is that the card functions much like a low level subroutine interface and requires software support to initiate each processing step. Thus to encrypt or decrypt an individual data there are a minimum of two steps, one to set the DES initialization vector and one to do the encryption. Since the IP processing can not suspend itself and wait while the hardware completes and then be rescheduled by the hardware interrupt handler, in one embodiment a finite state machine is used to tie sequences of hardware processing elements together. In one such embodiment the interrupt handler looks at the current state, executes a defined after state function, transitions to the state and then executes that state's start function.

One function, cyl_enqueue_request, is used to initiate either an encrypt or a decrypt action, This function is designed to be called by cryptographic engine interface 80. All of the information required to initiate the processing as well as the function to he performed after the encryption operation is completed is provided in the request structure. This function will enqueue the request on the hardware request queue and start the hardware processing if necessary.

A system 30 which can be used for firewall-to-workstation encryption is shown in Figure 5. In Figure 5, system 30 includes a workstation 12 communicating tbrough a firewall 14 to an unprotected network 16 such as the Internet. System 30 also includes a workstation 32 communicaling directly with firewall 14 through unprotected network 16. Firewall 14 is an application level gateway incorporating IPSEC handling as described above. (It should be noted that IPSEC security cannot be used to authenticate the personal identity of the scnder for a firewall to eewall transfer. When IPSEC is used, however, on a single user machine such as a portable personal computer, IPSE C usage should 23 be protected with a personal identification number (PN. In these cases IPSEC can be used to help with user identification to the firewall.) According to the, IPSEC RMs, you can use either tunnel or transport mode with this embodiment based on your security needs. In certain situations, the communications must be sent in tunnel mode to hide unregistered addresses.

Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ord skill in the art that any arrangement which is calculated to achieve the same purpose may be substituted for the specific embodiment shown. This application is intended to cover any adaptations or variations of the prescnt invention. Therefore, it is intended that this invention be limited only by the claims and the equivalents thereof.

24

Claims (11)

What is claimed is:

1 - A method of regulating the flow of messages through a firewall having a network protocol stack, wherein the network protocol stack includets an Internet Protocol (2) layer, the method comprising the steps ofi.

determining, at the 1P layer, if a message is encrypted; if the message is not encrypted, passing the unencrypted message up the network protocol stack to an application level proxy; and if the message is encrypted, decrypting the message and passing the decrypted message up the network protocol stack to the application level proxy, wherein the step of decrypting the message includes the step of executing a procedure at the 1P layer to decrypt the message.

2- A method of authenticating the sender of a message within a computer system having a network protocol stack, wherein the network protocol stack includes an Internet Protocol (T) layer, the method comprising the steps of determining, at the IP layer, if the message is encrypted; if the message is encrypted, decrypting the message, wherein the step of decrypting the message includes the step of executing a process at the IP layer to decrypt the message; passing the decrypted message up the network protocol stack to an application level proxy; determining an authentication protocol appropriate for die message, and executing the authentication protocol to authenticate the sender of the message.

3. The method according to claim 2 wherein the step of determining an authentication protocol appropriate for the message includes the steps of.

determining a source IP address associated with the message; and detennining the authentication protocol associated with the source IP address.

4. The method according to claim 2 wherein the message includes security parameters index and wherein the step of determining an authentication protocol appropriate for the message includes the steps of. deteg the authentication protocol associated with a dynamic IP address, wherein the step of determining the authentication protocol includes the step of looking up a security association based on the sectirity parameters index; determining a current address associated with the dynamic source IP address; and binding the current address to the security parameters index.

5. A firewall, comprising:

a first communications interface; a second communications interface; a network protocol stack connected to the first and the second communications interfaces, wherein the network protocol stack includes an Internet Protocol (IP) layer and a transport layer, a decryption procedure. operating at the IP layer, wherein the decryption procedure decrypts encrypted messages received at one of said first and second communications interfaces and outputs decrypted messages; and a proxy, connected to the transport layer of said network protocol stack wherein the proxy receives decrypted messages from the decryption procedure and executes an authentication protocol based on the content of the decrypted message.

6. A firewall, comprising a first communications interface; a second communications interface; a first network protocol stack connected to the first communications interface, whereirb the first network protocol stack includes an Internet Protocol (T) layer and a transport layer; 26 a second network protocol stack connected to the second communications interface, wherein the second network protocol stick includes an Internet Protocol (2) layer and a transport layer; a decryption procedure, operating at the IP layer of the first network protocol stack, the decryption procedure receiving encrypted messages received by said first communications interface and outputting decrypted messages; and a proxy, connected to the port layers of said first and second network protocol stacks, the proxy receiving decrypted messages from the decryption procedure and executing an authentication protocol based on the content of the decrypted message.

7. The firewall according to claim 6 wherein the firewall finiher includes: a third communications interface; and a third network protocol stark connected to the third conitnunications, interface and to the proxy, wherein the third network protocol stack includes an Internet Protocol (T) layer and a transport layer and wherein the second and third network protocol stacks are restricted to first and second burbs, respectively.

8. A method of establishing a virtual private network between a first and a second network, wherein each network includes an application level gateway firewall which uses a proxy operating at the application layer to process traffic through the firewall, wherein each firewall includes a network protocol stack and wherein each network protocol stack includes an Internet Protocol (2) layer, the j method comprising the steps of. transferring a comection request from the first network to the second network., determining, at the IP layer of the network protocol stack of the second network's firewall, if the comection request is encrypted; 27 if the connection request is encrypted decrypting the request, wherein the step of decrypting the request includes the step of executing a procedure at the IP layer of the second network's firewall to decrypt the message; passing the connection request up the network protocol stack to an application level proxy; determining an authentication protocol appropriate for the cormection request; executing the authentication protocol to authenticate the connection request; and if the connection request is authentic, establishing an active connection between the first and second networks.

9. The method according to claim 8 wherein the step of executing the authentication protocol includes the step of executing program code within the firewall of the second network to mimic a challengelresponse protocol executing on a server intemal to the second network.

10. The method according to clairn 8 wherein the step of executing the authentication protocol includes the step of executing program code to execute the authentication protocol in line to the session.

11. The method according to claim 8 wherein the step of determining an authentication protocol includes the step of determining if the connection request arrived encrypted and selecting the authentication protocol based on wbether the connection request was encrypted or not encrypted.