WO2002005494A1

WO2002005494A1 - High speed packet processing architecture

Info

Publication number: WO2002005494A1
Application number: PCT/US2001/021496
Authority: WO
Inventors: Sriram Krishnan; Alarabi Omar Hassen; Emil Yu-Ming Chao; Nirav Pravinkumar Dagli; Kwei-Yao Peng
Original assignee: Entridia Corporation
Priority date: 2000-07-07
Filing date: 2001-07-06
Publication date: 2002-01-17
Also published as: AU2001271901A1

Abstract

A method and apparatus for high speed routing in a packet switched network utilizing a router (104) having dedicated memory (180) and dedicated processor. The dedicated memory (180) operates in conjunction with hybrid buffering having features of input and output buffering in input and output queues. In an alternative embodiment, the input and output queue are further divided into sub-queues to provide more precise queue monitoring and output queue prioritization. Internal to each router resides dedicated route table, dedicated flow table, and dedicated ARP table that may optionally be embodied in a high-speed content addressable memory (CAM) structure. The CAM structures described herein provide numerous advantages over the prior art. The present invention may also be embodied in a multi-router system wherein a plurality of routers communicate via a shared bus structure.

Description

S P E C I F I C A T I O N

HIGH SPEED PACKET PROCESSING ARCHITECTURE

BACKGROUND OF THE INVENTION

1. Field of the Invention The present invention relates generally to routers for switching and routing data packets in a packet switched network, and, more particularly, to a router capable of switching and routing at high speed, such as at or near the rate supported by the physical interface to the router.

2. Background In the recent years, substantial attention has been directed to computer networks. One common type of a computer network is a local area network (LAN). A LAN is a communication network that serves users within a confined geographical area. A LAN is made up of servers, workstations, a network operating system and a plurality of communications links that transfer information between devices in the LAN. In one embodiment the communication link within the network operates under a protocol such as TCP/IP and IPX. Physical transmission of data may be achieved using the Ethernet standard.

Currently Ethernet, defined by the IEEE (Institute of Electrical and Electronics Engineers) 802.3 standard, is the most widely used LAN access method. The workstations on the segment share the total bandwidth, which ranges from 10 Mbps

(Mega Bits-Per-Second) to 1000 Mbps. Ethernet transmits variable length data packets, each containing a header with the addresses of the source and destination stations. Each data packet is broadcast onto a communications path. All stations attached to the Ethernet monitor the communication path and the station with the matching destination address accepts the data packet.

More recently, the principles of packet switch networks have been applied on a large scale to create a worldwide network known as the Internet. The Internet, and the components that comprise the Internet, are known by those of ordinary skill in the art. For example, the Internet or a LAN may be configured to operate in accordance with the Ethernet and OSI seven-layer model. The plurality of devices communicate over some form of communication channel or medium. This channel or medium connects the network devices to a plurality of workstations and other devices.

A device commonly referred to as a router interconnects networks to thereby allow different networks to communicate with each other. In large networks, a router serves to interconnect sub-sections of a single large network. To facilitate such interconnectivity the router possesses the capability to receive data packets in a packet switched network, analyze the packet and determine the final destination or next-hop of incoming packets, and forward the packet in the proper direction, i.e. to the proper output port on the router with the proper next hop address. In addition, packet repackaging occurs as dictated by the various layers in a packet switched network. Communication protocols dictate the manner in which the router packages and transmits data from one node to another and also includes network address look-up that facilitates routing from one network to another. As discussed above, the router forwards data packets either between networks or within larger networks. Typically, the router reads the network destination address in each received data packet and determines, based on look-up tables and routing protocols, the most expedient route for transmitting the packet. In contrast to bridges and switches, which work at layer 2 in the network protocol stack, routers work at layer 3.

Layer 2 is a communications protocol level that is sometimes called the data link layer or MAC (Medial Access Control) layer. The layer 2 protocol is a data communication protocol that controls transmission of data in packet form within the local network based on unique physical address of networking hardware. Because the data packets in the layer 2 contain the physical address, less processing of the packets is required, and thus the packets are transmitted at the wire speed. The term wire speed as used herein means data transfer at a rate the Layer 2 link level in a packet switched network. Stated another way, the data transfer at a wire speed rate is fastest rate possible for a given signaling protocol.

The layer 3, in contrast, is the communications protocol that contains the logical or IP address assigned to devices attached to network. The router inspects the IP address of the address header of the packet before forwarding the packet through the network. Because the layer 3 protocols include logical addresses, which must be translated into physical addresses before being forwarded, switching in a router requires more processing that simply forwarding the packet, such as in a switch or repeater. As a result, prior art devices operating at layer 3 are substantially slower than devices operating at layer 2. Additional analysis of additional routing data also increases routing overhead and slows the routing process.

A further disadvantage of router of the prior art arises due to the operation of the router's internal or external data transfer bus. In routers of the prior art having internal or external buses, the buses operated on an interrupt basis. Thus, for a device connected to a prior art router bus to utilize the bus, it must initiate interrupts. However, the device must undesirably wait until the bus is available and continue to generate interrupts to gain bus access. This method of bus utilization in routers of the prior art is undesirably slow.

Yet another disadvantage of routers of the prior art lies in the data packets buffering and queuing methods. Routers of the prior art utilize queuing and buffering schemes that suffer from head of line blocking and output buffer delays. The present invention overcomes these disadvantages in the prior art.

Still another disadvantage of routers of the prior art rests in their antiquated apparatus and methods used for look-up table realization. Routers of the prior art rely on software-based look-up table operations. This manner of operation is undesirably slow as data transmission rates increase.

There is, therefore, a need within the industry to provide a layer 3 routing device that runs substantially at the layer 2 speed, i.e., the wire speed.

As described below, the present invention overcomes the disadvantages of the prior art by providing a new method and apparatus for data routing in a packet switched network.

SUMMARY OF THE INVENTION

In accordance with the purpose of the invention as broadly described herein there is provided a routing system for use in a packet switched communication environment wherein high data rate packet routing is desired.

A first aspect of the subject invention provides improved systems and methods for routing packets in a packet switched network. In one configuration the present invention is embodied in a computerized network router. In such an embodiment the present invention comprises a network router capable of high speed packet processing. To achieve data packet processing at rates higher than previously possible in the systems of the prior art, the present invention includes a number of improved systems configured to achieve high-speed data processing. One example embodiment of the present invention includes a routing system configured with a single router system and contained within a protective outer housing. It is contemplated that each router device service several ports. In another embodiments the present invention utilizes two or more router devices in a single enclosure to provide scalability. When embodied as a router in a packet switched network, the present invention comprises a router that may optionally be embodied on a single chip or on a plurality of chips. In one configuration, the router may comprise one or more input ports, one or more physical layer interface modules, one or more address resolution (ARP) tables, one or more flow tables, one or more route tables, and one or more memory managers that oversee one or more memory units. One or more data buses interconnect the above-mentioned apparatus to facilitate inner-router communication. The router may optionally include a CPU port and interface to provided access and communication with a processor. Similarly, the router may optionally include one or more external bus ports and interfaces to facilitate communication between routers or other devices in a multiple router/device environment.

When the various features described herein are embodied in a router, either alone or in combination, for use in a computer over system of the prior art. These advantages are discussed in greater detail below and mentioned here for overview purposes. In various configurations the router may be utilized in a single router system or a multi-router system with various numbers of input/output ports per router. Thus, the present invention is scalable to meet the needs of users or the demands of the system.

One advantage of the present invention is a distributed ARP table configuration for data-link layer address resolution and for data link layer to network link layer address translation. In one embodiment the router associates an ARP table with each port on the router. Thus, for an eight-port router, each router includes eight ARP tables. This provides the advantage of faster ARP table lookup. In addition, the ARP table may optionally be realized in a content addressable memory (CAM) structure to speed operation. In preferred embodiment, the CAM structure within the present invention includes features and capabilities beyond those of a conventional CAM.

Another feature of the present invention is a distributed buffering arrangement wherein each routing device is allocated dedicated memory resources. Distributed buffer allocation speeds operation over systems of the prior art that rely on a single memory resource for multiple router devices.

Another aspect of the present invention comprises improved buffering methods and apparatus which, in one embodiment, is realized in the form of hybrid input and output buffering. One form of hybrid input and output buffering comprises selective use of both input buffering and output buffering depending on the dynamic response and behavior of the individual ports of the router system. Another aspect of the improved buffering comprises a form of sub-queue buffering. Sub-queue buffering comprises use of a plurality of sub-queues for each port wherein each sub- queue is dedicated to a particular manner or class of data. Such improved buffering monitors and reacts to undesirable port situations and improves data throughput. Sub- queues also provide for selective control over data transmission priority based on data types and data priority.

Another advantageous feature of the present invention comprises time division multiplexing (TDM) on the internal and external buses of the routing device. TDM on the data buses provides for more efficient use of the bus resources thereby increasing routing speed.

Another aspect of the present invention is the use of dedicated route and flow tables per router, both of which may be embodied in content addressable memory (CAM) structure. In various embodiments, the CAM structure includes advantages and improvements over the prior art. Use of an improved CAM structure for a route table and/or flow table increases processing rate of the routing system. Further advantages are gained from use of a non-conventional CAM structure. A distributed route table and/or flow table comprises allocation and use of a dedicated route table and flow table at each router device in a multiple router system. Distributed and dedicated route tables and flow tables for each router increase router speed by eliminating bottle-necks that often occurred in prior art systems which utilize a single route table and/or flow table for a plurality of router devices. In general, each route table in a multi-router device contains generally identical information, although it is contemplated that at times the information within each route table may be different.

Another aspect of the present invention comprises configuration of the route table, flow table, and ARP table in content addressable memory structures. The content addressable memory structures are used to implement longest prefix match searches of portions of a mask or other information field. Another aspect of the present invention comprises operation of the present invention at wire speed where wire speed comprises the data transfer rate of the Layer 2 link in a packet switched network. To achieve wire speed operation, the present invention may utilize hardware driven packet profile identification. Packet profile identification comprises analysis of either or both of the OSI layer 3 information and

OSI layer 4 information of the packet address header and the packet payload.

High-speed operation is also achieved by use of packet forwarding and filtering decision engine operating at or near wire speed. Likewise, high-speed operation is also achieved through the packet prioritization and discard decision engine for bandwidth management operating at Layer 2 speed.

Other objects, features, and advantages of the present mvention will become apparent to one of ordinary skill in the art upon examination of the following drawings and detailed descriptions. It is intended that all such additional objects, feature, and advantages be included herein within the scope of the present invention, as defined by the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 illustrates a block diagram of an exemplary embodiment of a router system having a single router. Figure 2 illustrates a block diagram of an exemplary embodiment of a multi- router system.

Figure 3 illustrates a block diagram of an exemplary network configuration.

Figure 4 illustrates a first implementation example of a router of the present invention. Figure 5 illustrates a block diagram of an exemplary embodiment of an input/output port of the present invention.

Figure 6A illustrates exemplary interconnections between devices in a network. Figure 6B illustrates basic entries of a route table.

Figure 7 illustrates a block diagram of an exemplary embodiment of a route table of the present invention.

Figure 8 illustrates a block diagram of an exemplary embodiment of a flow table of the present invention. Figure 9 illustrates an exemplary block diagram of a content addressable memory structure as contemplated for use in a route table or flow table.

Figure 10 illustrates a block diagram of a TAG system of an exemplary content addressable memory.

Figure 11 illustrates an exemplary embodiment of a memory cell. Figure 12 illustrates a block diagram of an exemplary embodiment of a TAG system with ranging functionality.

Figure 13 illustrates a block diagram of an exemplary embodiment of the compare logic of Figure 12.

Figure 14 illustrates an exemplary embodiment of a TAG system of a content addressable memory structure having aging capability.

Figure 15 illustrates an exemplary embodiment of a content addressable memory structure having aging capability with a validity cell.

Figure 16 illustrates an exemplary embodiment of a content addressable memory structure having aging capability with a validity cell and a static cell. Figure 17 illustrates a block diagram of a packet queue configuration.

Figure 18 illustrates an exemplary block diagram of sub-queues.

Figure 19 illustrates an operational flow diagram of an exemplary method of buffering data packets in a hybrid queue system.

Figure 20 illustrate an operational flow diagram of an exemplary method of receiving and responding to ARP packet requests.

Figure 21A and 2 IB illustrate an operational flow diagram of an exemplary method of receiving and routing data packets.

Figure 22 illustrates an exemplary IP packet header.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 1. Definitions

The following is a list of these terms and their definitions as used within the specification. These definitions are provided for purposes of understanding and should not be considered to limit the scope of the invention.

"ARP" stands for Address Resolution Protocol and is the protocol used to dynamically bind a high-level IP address (layer 3) to a low-level physical (layer 2) hardware address.

"Bridge" is a computer that connects two or more networks and forwards packets among them. Bridges operate at physical network level.

"Byte" is an 8-bit tuple information unit that is also referred to as Octet.

"Hop" is a link between two network nodes.

"Host" is any end-user computer system that connects to a network.

"LAN" stands for Local Area Network and is a communications network that serves users within a confined geographical area. LAN is made up of servers, workstations, a network operating system and a communications link.

"LAN Switch" is a network device that cross connects stations or LAN segments. LAN switches are available for Ethernet, Fast Ethernet and Token Ring. A LAN switch is also known as a frame switch.

"Layer 2" refers to link level communication (e.g., frame formats) or link level connections derived from the ISO 7-layer model. Layer 2 refers to frame format and addressing.

"Layer 3" refers to network layer communication derived from the ISO 7-layer model. Layer 3 also refers to IP and the IP datagram format.

"MAC" stands for Media Access Control and refers to the low-level hardware protocols used to access a particular network. The term 'MAC Address' is often used to refer to Internet protocol physical address.

"TCP/IP" stands for Transmission Control Protocol/Internet Protocol and is a communications protocol developed to inter-network dissimilar systems. TCP/IP is the protocol of the Internet and has become the global standard for communications. IP provides the routing mechanism and TCP provides the transport functions that ensure that the total amount of bytes sent is received correctly at the other end. TCP/IP is a routable protocol. "TOS" stands for Type of Service and is a field in each IP datagram header that allows the sender to specify the type of service desired.

"WAN" stands for Wide Area Network and is a communications network that covers a wide geographic area, such as state or country.

"Word" is a 32-bit tuple information unit consisting of four bytes. 2. Overview of the Invention

The present invention provides improved systems and methods for routing packets in a packet switched network. In one configuration the present invention is embodied in a computerized network router. In such an embodiment the present invention comprises a network router capable of high speed packet processing. To achieve data packet processing at speeds higher than previously possible in the systems of the prior art, the present invention includes a number of improved systems configured to achieve high-speed data processing. One example embodiment of the present invention includes a routing system configured with a single router system 100 as shown in Figure 1. A protective outer cover 102 protects the router 104, communication lines 106, physical layer processor 108, CPU 110, and memory 112. A number of ports 116A-116D or connectors are external to the outer cover 102. The connectors 116 connect to communication lines 106, each of which feed into a physical layer processor 108. The physical layer processor 108 is responsible for conversion from layer 2 to layer 3, for data reception, and conversion from layer 3 to layer 2, for data transmission, in the OSI seven layer standard. In turn, the router 104 connects to the physical layer processor 108. The router 104 performs address conversion, router look-up and other processes as described herein in greater detail.

The router 104 utilizes a processor or CPU 110 to update route tables and flow tables and to provide special treatment for data packets requiring CPU assistance. The router 104 utilizes memory 112 to store or buffer data packets during the route table look-up or when one of the input output ports 116 are momentarily occupied. The present invention includes several advantages over systems of the prior art. Each of these advantages are discussed in greater detail below and mentioned here for overview purposes. In various configurations the router may be utilized in a single router system or a multi-router system. One advantage of the present invention is a distributed ARP table configuration for data link layer address resolution and for data link layer to network link layer address translation. Another feature of the present invention is a distributed buffering arrangement wherein each routing device is allocated dedicated memory resources. Distributed buffer allocation speeds operation. In addition to distributed buffering, the present invention may optionally include improved buffering in the form of hybrid input and output buffering or a sub- queue buffering system. Such improved buffering advantageously monitors and reacts to undesirable port activity and improves data throughput.

Another advantageous feature of the present invention comprises time division multiplexing (TDM) on any of the internal and external buses of the routing device. TDM on the data buses provides for more efficient use of the buses, and more deterministic transfer times thereby increasing routing speed.

Yet another feature of the present invention is the use of distributed route table(s) and flow table(s), which may be optionally embodied in content addressable memory (CAM) structure. Use of a CAM structure for route and flow tables increases speed of the routing system. Distributed dedicated route tables and flow tables for each router increase router speed by eliminating bottlenecks that often occurred in prior art systems.

In alternative embodiment, the router 104 described above is connected to and in communication with one or more additional routers to form a multi-router system 120. Figure 2 illustrates a multi-router system 120 having a plurality of routers 104A- 104C therein. As shown, one embodiment of multi-router system comprises an outer housing 102 with a plurality of input / output ports 116 A- 116L therethrough. Inside the outer housing 102 is a plurality of routers 104A-104C interconnected by a data bus 220 through a data port (not shown) of each respective router 104A-104C.

Each router 104 contained within the multi-routing system also connects to associated memory 112A-112C and may optionally share a connection to an associated processor or CPU 110A via a processor bus.

In this manner the routers 104A-104C may combine in various ways to expand the capability of the routing system. Although shown with three routers 104A-104C, it is contemplated that the multiple router routing system 120 may be configured with any number of routers. Likewise, those of ordinary skill in the art could scale each router 104A-104C to have any number of ports 116. In various embodiments the routers 104 are connected in varying manner including but not limited to parallel configuration, serial configuration, circular configuration, or a star configuration.

3. Example Environment

One example environment where the present invention is well suited for use is in a computerized packet switched network. The exemplary components of a computerized packet switched network operating under the OSI seven layer model are shown in Figure 3. In the network of Figure 3, the router R0 of the present invention is utilized to accurately move packets of information across a network from source to a destination. Figure 3 illustrates a typical network configuration that might operate in accordance with the Ethernet and OSI seven layer model. As shown in Figure 3 the Internet 130 comprises a plurality of communication channels and computers. Connected to the Internet 130 or part of the Internet are one or more routers R0-R4. Routers use address headers and one or more forwarding tables to determine the routing of packets in a packet switched network. The routers R0-R4 connect individual networks, such as network 132 (enclosed by dashed line). The individual networks may vary in size and also include a router Rl within the network 132. One or more devices (D1-D4) connect to the router Rl . These devices may comprise computers, printers or servers. Some form of communication medium 134 governed by some form of communication protocol, such as Ethernet, connects the devices in the network 132. In large networks the network or LAN may be divided into sections 142, 144. These sections are connected by apparatus known as a bridge 146. A bridge 146 interconnects heterogeneous networks. Other embodiments utilized a switch (not shown) in replacement of the bridge to both interconnect networks and perform switching functions.

To facilitate such interconnectivity the routers R0-R4 possess the capability to receive data packets in a packet switched network, analyze the final destination or next-hop of incoming packets, and forward the packet in the proper direction, i.e. to the proper output port on the router.

A discussed above, the router R0-R4 forwards data packets either between networks or within larger networks. Typically, the router R0-R4 reads the network address in each received data packet and determines, based on look-up tables and routing protocols, the most expedient route for transmitting the packet to its destination, identified by a destination address. In contrast to bridges and switches, which work at layer 2 in the network protocol stack, routers work at layer 3 and layer 4.

Layer 2 is a communications protocol level that is sometimes called the data link layer or MAC (Medial Access Control) layer. The layer 2 protocol is a data communication protocol that controls the physical transmission of data on the network, including transmission within switches and bridges. Because the data packets in the layer 2 contain the physical address, less processing of the packets is required, and thus the packets are transmitted and processed at wire speed. The term wire speed as used herein means the maximum rate of data transfer within communication links of a network. In one embodiment, wire speed comprises the data transfer rate of the Layer 2 link in a packet switched network.

The layer 3, in contrast, is the communications protocol that contains the logical address of a server. The router inspects the address header of the packet before forwarding the packet through the network. Because the layer 3 protocols include logical addresses, which must be translated into physical addresses before being forwarded, switching in a router requires more processing than simply forwarding the packet, such as in a switch or repeater. As a result, prior art devices operating at layer 3 are substantially slower than devices operating at layer 2. Layer 3 protocols of the present invention utilize a type of packet (layer 4) and type of service field so that traffic in the network can be prioritized and forwarded based on message type as well as network destination. Layer 4 comprises the transport layer and it controls error recovery and flow control. While the example environment discussed herein comprises a computer network operating under the principle of the seven layer OSI model, it is fully contemplated that the principles of the present invention will find application in any packet switched network configured to transfer packets of data between two or more locations.

3. Example Embodiment

The example embodiment of the present invention described herein for purposes of understanding, and shown in Figure 4, is a router 104 in a computer network. As such, the example embodiment router 104 includes input-output ports

140-147. Each input-output port 140-147 is in communication with a data bus 160. The data bus 160 connects to memory manager 166, a processor port 170, and an external data bus port 174. The data bus 160 may also carry control information, or as shown, in other embodiments a control bus 162 is configured to transport control signals. The memory manager 166 is in communication with memory 180, which in this embodiment, is partitioned into status memory 184 and data memory 182 as shown. The processor port 170 may optionally connects to a computer processor (not shown). The external data bus port 174 may optionally connect the router 104 to other routers in a multi-router system. A more detailed discussion of each aspect of the router is now provided. It should be noted that the particulars of each aspect of the router is provided for purposes of understanding. The scope of the present invention is not limited by the enabling disclosure provided herein. Input / Output Ports

In the embodiment described herein the router is configured with 8 ports 140- 147 per router 104 although in various other configurations the number of ports per router may be expanded or reduced. Each port includes a RJ45 connector in the case of copper wire and an FX type connector in the case of fiber, although in various embodiments other types of connectors may be utilized.

In communication with the external connector is internal hardware of the port, including an ARP table. Figure 5 illustrates a block diagram of the components of a port 140-147 in greater detail. As shown, the port includes input line 200 and output line 201, both of which connect to a port connector (not shown) and transmit and receive logic. The input/output lines 200, 201 connect to a receive module 202 and transmit module 204. The receive module 202 and the transmit module 204 of the port communicate with a look-up table configured in this embodiment as an Address Resolution Protocol (ARP) table 206. The transmit module 204 utilizes the ARP table 206 to match IP addresses to corresponding MAC addresses (physical address) of next hop devices connected to the router and to determine if the address of the outgoing data packet corresponds to a device in the network served by the router. If the ARP table 206 contains the MAC address of the outgoing packet, the input/output port supplements the packet using the corresponding MAC address and transmits the packet out on the network.

Alternatively, if the ARP table 206 does not contain the IP address of the outgoing address, the ARP table uses the output port to send an ARP request packet onto the network in search of a device corresponding to the IP address. Devices receiving the ARP packet respond to the request. In this manner the ARP table 206 is updated. In one embodiment the ARP table 206 includes an aging mechanism wherein ARP table entries are deleted if not utilized within a certain time period.

Establishment and maintenance of the ARP table 206 provides means to store recently acquired IP to MAC address mapping thereby avoiding having to continually broadcast ARP requests when transmitting data. The entries in the ARP table 206 are routinely updated during data packet transmission, receipt and ARP packet transmission and receipt.

In one embodiment the ARP table 206 is embodied using content addressable memories (CAM) structures. Use of a CAM structure for accelerated data look-ups greatly improves the speed of the routing device. In preferred embodiments the CAM structure is modified to provide enhanced features and capability.

In contrast to systems of the prior art, the present invention advantageously includes and maintains an ARP table 206 for each port 140-147 of the router 104.

The dedicated ARP structure desirably increases speed by eliminating the slowing bottleneck that could occur when a single ARP table 206 is shared between a number of ports 140-147. The speed of the dedicated ARP table 206 is further increased by the use of one or more CAM structures within the hardware portion of each port. Use of an ARP table 206 embodied in a CAM structure in the hardware of each port 140-

147 achieves data processing and data transfer at previously unobtainable rates. In addition, the ARP table entries of the present invention can be established as either static or dynamic in nature. Static ARP table entries are set by the processor.

Dynamic ARP table entries are learned and/or may be aged out of the ARP table.

The port apparatus also includes logic 210 appropriately interspersed within the ARP table 206, the receive module 202, and the transmit module 204 to achieve desired operation. The logic 210 can be configured to perform error checking and MAC address verification and other associated tasks as known by those of ordinary skill in the art.

In reference to the ARP table of the present invention, the CAM structures that may be utilized to realize the ARP tables are discussed below in greater detail. The

ARP table utilized herein, or any CAM structures as may be embodied based on the teachings contained herein, may advantageously include the following features or operations: read operation, write operation, look-up operation, invalidate operation, learn mode or operation, check operation, age operation, and refresh operation. Some of these operations are discussed in more detail herein. A brief explanation is now provided. The read operation reads one or more values from the ARP table. The write operation writes one or more values to the ARP table. The look-up operation, upon receipt of a tag value, searches the ARP table for a matching entry and returns the associated value from memory. The look-up operation is commonly performed during the routing operation to obtain data for packet routing. The invalidate operation invalidates, removes, or overwrites one or more entries in the ARP table based on commands from ARP table logic, ARP table controller logic, CPU or other source. The learn mode operation or process comprises a process of writing new entries into the ARP table. For any reason it is desired to write additional data to the ARP table. The learn process, when active, allows entries looked-up in the ARP table but now found, to be written to the ARP table automatically. In one configuration, the learn process automatically overwrites entries that have been invalidated or aged out by the invalidate operation or the age process. The check operation interrogates the ARP table to determine if an entry is present. In one configuration the check operation outputs a yes/no output which may be in the form of a binary one or zero. The age operation is described below in greater detail. The refresh operation is similar to a reset function, but instead performs a reset of all age cell values to in effect prevent the aging out of entries until the counter cycles through it values.

Data Bus

Each of the network communication ports 140-147 connect to the data bus 160. The data bus 160 interconnects each of the ports 140-147 with the other devices or portions of the router 104. To distinguish the data bus 160 operating within the router 104 and those bus apparatus that connect separate routers, the bus apparatus that connects routers in a multi-router configuration is referred to herein as an external data bus 220, described below in greater detail.

In a preferred embodiment, the internal data bus 160 and external bus 220 support three types of transactions that result in transfer of control information and data. The first type of transaction involves the transfer of control information and data between the routers 104 using the external bus 220. The second type of transaction involves the transfer of control information and data between the processor port 170, network communication ports 140-147, the external data bus port 174 and the buffer manager 166. The third type of transaction results in transfer of control information and data across the internal data bus 160 to the processor port 170.

In one embodiment the internal and external buses 160, 220 utilizes a time- division multiplexed (TDM) protocol to communicate information between ports 140- 147 in the router 104. The TDM protocol of the router 104 significantly increases the overall speed of data transfer on the data bus 160 and reduces the overhead in establishing bus access that occurs in non-TDM data transfer mechanism, and also makes bus access more deterministic in nature.

In a preferred embodiment, the high speed operation is achieved by combining interleaved access control cycles and data cycles, eliminating non-deterministic read cycles and limiting each bus access to a single cycle. As a result, the efficiency of the bus structures are significantly increased since data transfers may occur on every clock cycle. This significantly reduces the bus cycle overhead and creates a deterministic behavior on either of the internal and/or external buses of the present invention. In order to support high-speed data transfer, in a preferred embodiment, the external data bus 220 runs on a low- voltage high-speed interconnect system capable of operating at GTL levels. In one embodiment the data bus 220 runs transactions at a single clock rate of 100MHz and for up to 800MB/sec bandwidth and the data bus 220 comprises a 64-bit data bus using a 6-bit transaction, specifier, a 27-bit packet ID, a bus request signal and a bus grant signal. In a preferred embodiment, the data bus

220 is an arbitrated bus, and thus, at least one of the routers 104 in the router system act as a master bus controller to arbitrate bus access based on a TDM basis.

Memory Manager and Memory A memory manager 166 is in communication with data bus 160. The memory manager 166 facilitates efficient utilization of the memory 180, serves to allocate memory space to incoming data packets on a dynamic basis, and maintains a record of data packets that are stored in memory. Moreover, the memory manager 166 also serves to retrieve and update packets stored in memory. In one embodiment the queue, as discussed below in greater detail, utilizes of the memory for storage and the organization of the data packets in memory is oversaw by the memory manager 166. High speed memory managers 166 are known by those of ordinary skill in the art and accordingly not described in detail herein. In communication with the memory manager 166 and the data bus 160 is memory 180. In a preferred embodiment, the memory 180 comprises external SRAM operating at 100MHz, such as a ZBT synchronous SRAM or a Sync-Burst SRAM, and status SRAM. The external SRAM 182 is used as a packet buffer. The status SRAM 184 serves as a buffer for storing the status of network data packets in queued data structures. The packet data SRAM 182 store the data. Thus, data packets are received via the network communications port 140-147, the external bus port 174 or the processor port 170 and are thereafter buffered in the packet buffer 180 for additional processing before transmission.

In the embodiment shown in Figure 3, the memory 180 is external to the router 104, which may be embodied on a single integrated circuit or chip. In the embodiment described herein, the router 104 logically addresses up to eight rows of the external SRAM(s) 180 and up to a 20-bit address range per each external device. In various other embodiments, the router 104 drives up to eight external SRAMs 180 in various configurations.

In the embodiment described herein each row of the packet buffer 182 is preferably 64-bits wide. Accordingly, depending on whether the external SRAM 180 is 16-bit or 32-bit wide, four or two external SRAMs are contemplated per row, respectively. Each row of the status buffer 184 is 16-bit wide, and thus in a preferred embodiment, 16-bit wide external SRAMs 180 are used. However, in other embodiments, the router 104 may communicate with additional external SRAMs 180 as desired.

As an advantage over the prior art, each router 104 accesses dedicated memory resources 180 and an associated memory manager 166. Such dedication of memory resources and memory management resources increases data processing speed as compared to system of the prior art that utilize a shared memory. Shared memory systems suffered from memory access bottlenecks and as a result, undesirably slowed operation of the router 104. In embodiments configured as multi- router systems (Figure 2), each router 104 has dedicated memory 180 associated therewith to facilitate high speed operation.

Look-up Tables

Preferred embodiments of the router include two additional look-up tables 172, 175 to assist in analysis and forwarding of incoming and outgoing data packets. Other embodiment may however, include additional tables. The look-up tables discussed herein include a route table 172 and a flow table 175. Both are embodied in content addressable memory (CAM) structures as described above with regard to the ARP tables 206. The route table 172 and the flow table 175 are both discussed in greater detail below. For purposes of understanding the route table 172 and the flow table 175 are shown in conjunction with the processor port 170 as both the route table and flow table may be accessed via the processor. Route Table

As known by those of ordinary skill in the art, route tables 172 determine the proper output port, i.e. next hop, when forwarding data packets. Operation of the route table 172 is understood by those of ordinary skill in the art and accordingly, description of the route table 172 is limited to topics concerning the present invention and its advantages over the prior art.

In a preferred embodiment the present invention utilizes classless inter-domain routing to facilitate use of fewer route table entries. In one embodiment the router of the present invention preferably utilizes one or more CAM structures to facilitate a pipeline three-stage look-up. In another embodiment the look-up procedure is achieved in a single cycle.

In a single cycle operation of the route table, a data packet that is to be forwarded is first analyzed to determine the destination address. Once the destination address is revealed, the route table 172 is utilized to determine the output port 140- 147 on which to send the data packet, and the destination address of the data packet so that the data packet is sent to the proper next hop.

In a preferred embodiment, the route table 172 utilizes custom memory structures to implement longest prefix match searches for the best routing match. This feature may optionally be implemented in a CAM structure. Figure 6 illustrates an exemplary route table entry and associated exemplary network connections. As shown, a first router Rl has a first port R1P1 and a second port R1P2. A first host HI connects to the first router Rl . The second port R1P2 connects to a second router R2. The second router R2 has a first port R2P1 and a second port R2P2. A second host H2 connects to the port R2P1 while a third host H3 connects to port R2P2.

In reference to the sample route table 239 of Figure 6B the process of analyzing the data packet destination address involves accessing the subnet address 240 of the data packet. The subnet address 240 is provided to the route table so that the closest next hop address match may be found based on hierarchical address matching. The route table 239 retrieves the next hop address 242 and the interface port 244 on which to output or forward the data packet. The route table 239 inserts the next hop address 242 into the data packet address and forwards the packet to the proper port 244. Thus, in the case of a packet being sent to a host H2 the packet is provided to the next hop R2 via port R1P2. This provides a general understanding of basic route table operation.

Figure 7 illustrates the basic entries of a route table 172 and exemplary interconnections between devices of an example network. In the embodiment discussed herein, the route table 172 and associated hardware are co-located with the processor port 170 to facilitate processor access to the route table.

As shown the route table 172 comprises a route table interface 250, route table hardware 172, and a processor interface 252. In one preferred embodiment the route table hardware 172 comprises one or more CAM structures operating in conjunction with SRAM. The route table hardware is in communication with the data bus 160 via a route table interface 250. The route table interface 250 comprises a compilation of logic and memory structures configured to arbitrate access to the route table 172. The route table interface 250 separates the data packet heading into subparts that are used to identify routing information for the packet. In communication with the route table 172 is a processor interface 252. The processor interface 252 arbitrates and facilitates access by the processor 110 (Fig. 1) to the route table 172. The processor interface 252 also writes route table updates into the routers 104 internal route table data structures. In one embodiment the processor interface 252 comprises a compilation of logic and memory structures configured to arbitrate access to the route table.

In the embodiment described herein, the route table 172 comprises a compilation of dynamically changing data relationships. The route table data entries may be categorized as four types of data fields. A first data field 260 comprises the type of service (TOS) data field. This data field comprises an 8 bit data field that identifies the data packet handling instructions such as the type of service to which a packet is entitled. Examples of type of service may include flags that indicate a destination receives any of high priority, high reliability, or high capacity type service. The terms high priority, high reliability, and high capacity type of services are known by those of ordinary skill in the art and are defined by various IP standards.

Thus, nodes served by the router may be allocated varying types of service based on the needs and/or requests of the node. It is contemplated that this varying type of service may be utilized to further tailor router operation to the particular needs of users or network managers. A second data field 262 comprises the IP source address (IPSA). The IP source address identifies the source of the data packet, i.e., the device that sent the data packet. A third data field 264 comprises the next hop IP address (NHIP) field. Data stored in this field identifies the next hop for the data packet based on the destination address of the received data packet. A fourth data field 266 comprises the physical interface number (PIN). The physical interface number identifies the proper port in the router to send the packet so the packet will reach the next hop identified in the next hop field. In addition, the packet may include or be associated with a tag field to speed routing. A tag field comprises an additional level of routing information and generally indicates a particular handling procedure, tag specific, for the packet.

In use, the route table interface 250 analyzes the incoming data packets on the data bus 160 and identifies the address portions of the data packets used for routing. The route table interface 250 then provides information to the route table 172, which in turn provides the next hop IP 264 and the physical interface number 266.

Likewise, the type of service 260 identified with the data packet is discovered during in the routing process. After obtaining this information from the route table 172, the route table interface 250 reassembles the data packet address portion using the next hop address and provides the packet on the data bus. The physical interface number 266 is provided to the bus 160 to ensure the data packet is routed to the proper network port. The TOS information is used in special handling of the packet.

Advantageously, each router 104 maintains it own route table. Thus, in a multi-router system there is a route table 172 for each of the routers 104. A further advantage of the present invention is that each route table 172 is embodied using a CAM structure and is established in an IPSA format to reduce the time required for each route table look-up procedure.

In preferred embodiment, multiple redundant route tables are utilized to insure reliable operation and to reduce down time. In such an embodiment, a first route table is updated while a second route table is in use by the router. Flow Table

Figure 8 illustrates an exemplary embodiment of the flow table 175 of the present invention. As shown in this exemplary embodiment the flow table 175 is associated with the hardware that embodies the processor port 170 and the route table

172. This provides access by the processor 110 to the flow table entries via the processor interface 252.

The flow table 175 shown in Figure 8 communicates with the flow table interface 300 and the processor interface 252. The flow table 175 is also embodied using one or more CAM structures and associated SRAM to obtain look-up speed previously unobtainable in the prior art. Likewise, a flow table 175 is associated with each router 104. Hence, in a multi-router system (Figure 2) a flow table 175 is maintained within each router 104. This increases the speed of the routing system in that bottlenecks that occur in a shared flow table environment are eliminated. The flow table comprises several fields including a source IP address field

(SIP) field 310, a source port range (SP) field 312, a destination IP address (DIP) field 314, a destination port range (DP) field 316, a protocol field 318, and an action field 320. The source IP address field (SIP) field 310, source port range (SP) field 312, destination IP address (DIP) field 314, and destination port range (DP) field 316 are generally self-explanatory to those of ordinary skill in the art. The fields with range comprise a minimum and maximum value. The protocol field 318 contains information regarding the type of layer 4 payload in the packet. The action field 320 stores information regarding the action to be taken on a particular packet. For example, if information, derived from a packet, is presented to the table and the information provided matches the data stored in one or more of the source IP address field (SIP) field 310, a source port range (SP) field 312, a destination IP address (DIP) field 314, a destination port range (DP) field 316, and protocol field 318, the router is instructed to take action on the packet based on the entry stored in the action field 320. Moreover, the route process may incorporate a type of service look-up, an interface ID look-up, and/or a protocol look-up to determine how to handle a particular packet. Type of service look-up is particularly useful when queuing packets and determining transmit priorities in both transmit queues and drop queues. The flow table 175 operates in a manner generally described above. However, the flow table 175, sometimes referred to as a filter table, supports as a list of rules that are utilized by the router to identify packets, classify packets and filter packets. In operation, the flow table 175 isolates identifying information from within the data packets such as source and destination addresses. Using this information the router 104 executes a search within the flow table 175 for a matching entry. Upon discovering a matching entry, the filtering identified by flow table 175 is executed. In one embodiment the identified filtering may comprise discarding the packet, prioritizing the packet and directing the packet to a particular queue, or directing the packet to the processor via the processor interface 252. The use of multiple queue system is discussed below in greater detail. In this manner the flow table 175 further directs, filters, and prioritizes packets passing through the router 104. By way of example, and not limitation, data packets may carry data comprising portion of a voice conversation or a portion of an e-mail message, or unwanted data packet traffic from a known source of such unwanted traffic. Accordingly the flow table of the present invention desirably includes means to appropriately deal with these varying types of data packets.

In one embodiment the various types of actions include drop packet, forward packet (standard), forward packet with priority, and notify CPU of packet and address. For example, based on the source IP address or the destination IP address, the packet is handled in a manner corresponding to the action field for the corresponding flow table entry. Hence, a data packet carrying voice data information can not be delayed and therefore is designated an action entry of forward with priority. If however, the data packet is determined to arrive from a source address that generates unwanted data packet traffic, the data packet can be discarded, i.e. dropped. Other appropriate action can be taken based on the entries in the flow table. Analysis can be made on either of the source of the data packet or the destination of the data packet. Entries in the flow table may be added or changed dynamically or on a static basis via user intervention through the CPU interface 252.

ARP Table

As discussed above, various embodiments of the present include an ARP table. One exemplary configuration of an ARP table is shown in Figure 9 embodied in a content addressable memory (CAM) 600. The CAM 600 shown and described herein may be utilized for apparatus other than an ARP table.

The CAM 600 comprises an address decoder 602, a TAG system 604, a priority decoder 606 and memory 608. A CAM 600 serves to store data and data associations. First data A is associated with a second data B. Instead of accessing second data B by an address, the CAM allows access to second data B based on the associated data A. Thus, the second memory structure, in this embodiment, the memory 608, is addressable or accessible based on the content of the first memory structure, in this embodiment, the TAG system 604. The TAG system 604 and the memory 608 comprise memory structures capable of storing data in a retrievable and addressable manner. In one embodiment the memory 608 comprises SRAM.

The address decoder 602 includes an address input 610 configured to communicate an address to the address decoder. The address decoder 602 facilitates entry and association of data into the memory 608 and TAG system 604 in a manner known in the art. The TAG system 604 comprises a memory structure configured to store data.

One example embodiment of a TAG structure is shown in Figure 10. As shown, the TAG system 604 comprises a plurality of memory cell 620 configured in a manner to store data presented on input lines 622 shown at the bottom of the TAG system. The cells are arranged in a row and column configuration. It is contemplated that some CAM structures include about 1000 rows. Using logic structures, the memory cells

620 store inputs presented at input lines 622.

An example implementation of a memory cell 620 is shown in Figure 11. As shown two gates 629 are connected in feed back mode so that as long as power is provided to the gates, the logic level presented at the input 630 is also presented at the output 632, but in inverted form.

Returning now to Figure 10 the output of each memory cell 620 connects to a two input exclusive NOR gate 640. The other input to the two input gate connects to one of input lines 605. The exclusive NOR gate 640 operates in a manner known in the art. The output of each exclusive NOR gate 640 connects to a multi-input AND gate 642 as shown. The AND gate 642 operates in a manner known in the art. The output of the AND gate 642 connect to the priority decoder 606. In addition, there exists, additional complex logic and dynamic logic to increase density and/or increase efficiency. In operation, a TAG value is presented at the TAG inputs 605. Using logic, the input to the TAG system 604 and the value on each memory cell 620 is presented as inputs to the exclusive NOR gate 640. If the input line value and the memory cell value are identical then the exclusive NOR gate 640 outputs high value. The plurality of outputs from the plurality of exclusive NOR gates 640 connect to inputs of the AND gate 642. If each input line value matches each memory cell value then all the inputs to the AND gate 642 are high and the AND gate outputs a high value. The AND gate 642 forwards this high value to the priority decoder 606. This high output signal to the priority decoder 606 indicates which particular row of the TAG system 604 contains matching data. The priority decoder 606 may perform further processing on the output from the TAG system 604 if more than one received input comprises a high logic level signal to evaluate which of the plurality of high inputs is the desired matching row in the TAG system 604. In reference to Figure 9, the priority decoder 606 outputs a single high logic level signal on outputs 607 to thereby select a particular row in the memory 608 to be provided on memory output 609. It should be understood that this is but one possible configuration and method of operation of a CAM. Further, the size and/or dimension of the memory 608, TAG system 604 and other apparatus is not limited to the size shown in the Figures.

As an advantage over the prior art, one embodiment of the modified CAM structure that may desirably be used with the present invention includes a CAM structure having address range look-up and aging capability. These features are desirable attributes of a CAM structure in general and these features are particularly desirable when utilized in conjunction with any of an ARP table, route table, or flow table.

The address range look-up feature of one embodiment of a CAM structure comprises storage of a single memory entry for a plurality or range of TAG entries. Advantages gained by address range look-up comprises reduced space requirements, increased storage density, faster operation, reduced power consumption, and greater ease of management and manual updating. For example, the following table assists in the understanding of address range look-up feature.

In such an exemplary CAM content, the memory entry for TAG entry 100 through TAG entry 105 is an identical memory entry, that is, Port B. Undesirably, this manner of association, a one-to-one association, requires a TAG system row and memory row for each entry. In systems of larger dimensions or greater repetition, advantages gained by overcoming the use of a one-to-one association become evident. The table shown below provides an example of address range look-up. As shown, a range of TAG entries 100-105 correspond to a single memory entry. Thus, all TAG entries having values 100 to 105 inclusive result in a match and output to memory entry "Port B".

To achieve the advantages of address range look-up, also referred to as range association, one embodiment of the CAM structure described herein comprises a configuration of circuitry and logic within the TAG system and other systems of the CAM to achieve the address range look-up. In one embodiment the operation occurs at 16 bit complexity. It should be noted that the use of the term CAM structure herein should be interpreted to mean a CAM structure that may or may not utilize some or all of the advanced features associated described herein. A further desirable aspect of a range look-up configuration is high packing density, device regularity and speed.

Figure 12 illustrates block diagram of an example embodiment of a system configured to achieve range look-up. Shown in Figure 12 for purposes of discussion is a simplified version of a structure with ranging capability as described above. Shown is a priority decoder 606 having generally similar construction and operation as that shown in Figure 9. The output of the priority decoder 606 connects to memory (not shown) over a plurality of lines 607. Input to the priority decoder arrives from a compare logic module 752. The compare logic module 752 receives input from a high limit memory module 754, a low limit memory module 756, and a bit line 750. The high limit memory 754 stores a value representing the high limit in the range of values used in the ranging function. The low limit memory 756 stores a value representing the low limit in the range of values used in the ranging function. The bit line 750 carries the value to be compared to the high limit value and low limit value. The bit line 750 is generally equivalent to the Match Word Inputs 605 of Figure 9. It is contemplated that the bit line 750 comprises a parallel conductor. In one embodiment the bit line 750 is a 16 bit wide conductor.

The compare logic 752 executes one or more comparisons on the input from the bit line 750, low limit 756 and high limit 754 to determine if the value on the bit line is intermediate the low limit value 756 and the high limit value 754. If the value on the bit line 750 is between the value stored in the low limit location 756 and the high limit location 754, then the compare logic 752 outputs a high value on lines 758 to the priority decoder 606. The priority decoder operates in the manner described above.

Figure 13 illustrates one method and apparatus for achieving the ranging function with a CAM structure as described herein. As shown, Figure 13 illustrates a block diagram of the compare logic 752 of Figure 15. For ease of discussion and understanding, the method and apparatus used to achieve the compare function between the high range value memory 754 and the value on the bit line 750 is provided. A similar method and apparatus, as would be understood by one of ordinary skill in the art, is implemented to achieve a compare process between the low range value memory 756 and the value on the bit line 750.

In reference to Figure 13, a bit line750, in one embodiment a 16 bit wide conductor, connects to a first input of a plurality of two bit comparators 801 - 816. In this embodiment, each bit on the 16 bit wide bit line 750 feeds into one of the two bit comparators 801-816.

The contents of a memory cell 754ι - 754ι₆ connects to a second input of comparators 801-816 wherein each memory cell connects to one of the comparators as shown. The memory cells 754] - 754i₆ store the value of the high value in the range.

The comparators 801-816 are known by those of ordinary skill in the art and accordingly are not described in great detail herein. In this example embodiment the comparators 801-816 provide output signals based on the following equations:

Badx = 1 (high) if BtLx > MCx and

Nextx = 1 (high) if and only if BtLx and MC are the same,

where the variable x defines the position in the binary value carried on the bit line or in the memory cells 754ι - 754ι₆. Based on these relationships, comparator 802 will output a high value on Bad2 if the corresponding value on the bit line 750 (BtL) is greater than the value in memory cell (MC) 754₂. The Next2 output of the comparator 802 will be high if the corresponding value on the bit line 750 (BtL) input to the comparator and the value in memory cell 754₂ are identical. The operation of comparators 801-816 continues in this manner.

The outputs of two bit comparators 801-816 connect to a 4:1 compare unit 821-824 (only two of a total four are shown). The inputs to the four 4:1 comparators 801-804 are in the manner shown in Figure 13. The exact routing of each line is omitted for purposes of discussion. The 4:1 compare units 821-823 include eight inputs as shown, the last 4:1 compare unit 824 utilizes only 7 inputs.

The compare processes apparatus is sub-divided into four subgroups in this particular implementation example as a means to simplify the overall logic of the system. Those of ordinary skill in the art may contemplate other method and configurations to effectuate the compare process described herein.

Each 4: 1 compare unit includes two outputs, in this configuration a CxBad and a CxNext, where x is the number (1-4) of the 4:1 compare unit. Operation of each 4:1 compare unit 821-824 is defined in part for this particular example by the following table. This table does not provide a complete listing of every possible combination and hence it is provide for purposes of understanding.

In general, the compare units 821-824 evaluate the output of each of the two bit comparators 801-816. As can be understood, to determine the greater of two binary digits, comparison is first made of the most significant bits of the two numbers. If the two most significant bits are different, then a determination can be made as to the larger number. If the two most significant bits are identical, then a determination cannot be immediately made and analysis of the next most significant bits is initiated. In this manner determination if a first binary number is larger than a second binary number. This is the general manner of operation of the ranging function of the present invention.

The two bit comparators provide an indication of the two compared bits are identical, or if the corresponding value on the bit line is greater than the stored value. Using this input the 4:1 compare unit 821 systematically performs analysis using logic structures to provide an output ClBad 830 and ClNext 832.

A final compare unit 840 receives the output of the four 4:1 compare units 821-824 on its input lines 842. Additional logic structures within the final comparator 840 analyze each input 842 to determine if the input indicates that the value on the bit line is outside the high value range of the ranging compare process. It should be noted that the 4: 1 compare unit 824 does not include a C4Next output and final comparator 840 does not include a C4Next input because there is not a next bit to determine, and hence a signaling of whether to evaluate the next bit is not needed. The final comparator 840 includes an output OutRange 850 to signal when the value on the bit line is out of range, i.e. not within the range of values. Thus, in this embodiment, when OutRange 850 is high, the value on the bit line 750 is not within the range of the low value and the high value. It should be understood that this is but one possible method and apparatus to achieve the ranging features within a CAM structure as defined herein. The present invention is not limited to the particular logic configurations described herein.

In addition to the advantages gained by CAM structures having range association capability, an alternative embodiment of a CAM structure includes a feature referred to as aging. For purpose of discussion aging of CAM structure entries comprises monitoring, disabling, or removing TAG entries and/or associated memory entries after a period of time since being entered into the CAM structure or since last being used or accessed by the CAM structure. Aging of CAM structure entries is particularly desirable when the CAM structure serves as an ARP table. In such an embodiment ARP table entries not utilized within a certain time period are disabled or deleted from the ARP table. In this manner, entries in the ARP table are the most recently used entries. If ARP tables are not updated or policed, the table entries become outdated. In one embodiment a number of flip-flops or a counter structure is associated with each row of the CAM structure. As the clock of the device triggers, the counter counts up. At a determined counter value, the entry is aged out of the table. When the system uses an ARP table entry the counter or flip-flop structure is reset to thereby restart the aging process. In one configuration, a flip-flop is associated with each bit or memory cell and a clock line is provided to each flip-flop array. It is contemplated that in one embodiment, the number of flip-flops is related to the time-out period.

In an alternative embodiment, flip-flops are provided for the counting function and utilized as a source of data to compare to the content of memory cells of the aging system. In such a configuration, a time stamp is utilized from the counter value and thereafter a matching function occurs. Figure 14 illustrates a block diagram of an example embodiment of a system configured in this manner. In reference to Figure

10 and Figure 14, like elements are referenced with like reference numerals. As shown in Figure, 14 an aging system 680 operates in conjunction with the TAG system 604 described above. In this configuration, there is a reduction of apparatus by about the number of flip-flops times the number of rows.

This embodiment includes a plurality of memory cells including for the purposes of discussion a first memory cell 682, a second memory cell 684 and a third memory cell 686. Each of these cells 682, 684, 686 contains a value, such as a one bit value. The memory cell has an input node used to write data to the cell and an output node, used to read data from the cell. The output node of each cell connects to an input of a two input exclusive NOR gate 688, 690, 692 as shown. Connecting to the other input of the exclusive NOR gate 688 is an output line 698 from a three bit counter 696. The three bit counter 696 is known by those of ordinary skill in the art and accordingly is not described herein beyond that the three bit counter receives a triggering input from a clock or other timing device which thereby increments the counter. The output of the counter is provided on output lines 698, each of which connect as an input to the exclusive NOR gates 688, 690, 692.

The output of each exclusive NOR gate 688, 690, 692 connects to a multi- input NAND gate 700. In one embodiment this gate comprises a three input NAND. In other embodiments described below, the number of inputs and type of gate 700 may vary to suit the particular needs of the particular application.

By providing this input to the AND gate 642 of the TAG system, the aging system is able to control whether the memory output of the row, in this example the top row is considered to have valid data. In this embodiment a three bit counter is used for purposes of discussion. Other embodiments are not limited to three bit counters.

For example, if the counter 696 value matches the value in the memory cells 682, 684, 686, then all inputs to NAND gate 700 are high. Presenting all high inputs to a NAND gate 700 forces the output of NAND gate 700 to output a low logic level value. This low logic level is then provided to AND gate 642 thereby forcing output of AND gate 642 low. Thus, the values in this row are never asserted.

In other embodiments, the system includes additional circuitry to supplement operation of the CAM structure. In one configuration, shown in Figure 15, a row validity memory cell 710 is provided to store a bit indicating the validity of the cell. In one embodiment the row validity memory cell 710 is an extension of the TAG system CAM. In one configuration, the input to the row validity cell 710 arrives from the output of a gate 711. Thus, the output of NAND gate 700 connects as an input of gate 711. The second input to gate 711 comprises an age enable signal. The age enable signal provides means to enable or disable the output of gate 700 thus providing means to disable and enable the aging of entries from the CAM structure. This prevents the incrementing of the counter and the compare function from aging out a recently written entry. Those of ordinary skill in the art understand the operation of gate 711. Aging is described below. The output node of the row validity cell 710 connects to the AND gate 642 to control whether the row values in memory 608 are provided on output 609 (shown in Figure 9). Thus, the row validity cell 710 stores a validity bit or flag for its associated row. At the time the TAG and memory values for a particular row are written, the value of the counter is stored in the aging cells for that particular row, for example, cells 682, 684, 686. Then, as the counter 696 increments, the value of the counter only matches the values in memory cells 682, 684, 686 when the counter has stepped through its entire value set. If the values of the memory cells 682, 684, 686 match the counter value, the output forces the value in the row validity cell 710 to permanently designate the row values as invalid or aged out. The row validity cell 710 is reset when new data is written to the row, or if a TAG entry in the CAM structure is used, the age cells 682, 684, and 686 are reset to the current counter value.

Cell 710 is monitored, and when set, indicates that the row may be re- written with new data. In one configuration, the first row with aged data, i.e. a set bit or flag in the valid cell 710, that is encountered during the re-write process is re- written with new data. If the system uses the row, the counters value is re- written to the memory cells 682, 684, 686. In such an embodiment, the increment process of the counter occurs prior to the step of comparing the counter values to the aging cells values 682, 684, 686. Additional logic may be utilized as would be known in the art to maintain the value in the row validity cell 710 until reset, a new write operation, or other event that would validate the values in the row.

Turning to Figure 16, yet another embodiment of the present invention is provided. Figure 16 is generally similar to Figure 15, but includes an additional cell referred to herein as static cell 720. In one embodiment static cell 720 comprises an extension of the TAG system 604, that is to say, it is also embodied in a CAM structure. In one configuration, the static memory cell 720 controls, overrides, or disables the aging function of the row with which it is associated. In one method of operation the static cell 720 can be thought of as disabling NAND gate 724. As shown, the input of static cell 720 may be written to by any means known in the art. The output of the static memory cell 720 connects as an inverted input to a NAND gate 724. Thus, as long as the static memory cell 720 is high, at least one input to the NAND gate 724 will remain low. Therefore, the output of the NAND gate 724 will remain high and the row will never age out.

In yet another embodiment of the CAM structure embodied as any of an ARP table, route table, or flow table, the feature of learning is advantageously included to facilitate greater efficiency of operation. In such an embodiment, the packet header information may be analyzed to obtain the desired information and such information compared to the data stored in the appropriate CAM structure. If the information is not contained within the CAM structure, the information from the packet header is copied into the CAM structure.

In the case of an ARP table, a received ARP request is received and analyzed. The information contained within the received ARP request is analyzed and compared against data stored in the ARP table. If the data is not found in the ARP table, the data is stored in the ARP table for future use at a row location having a valid row cell 710 indicating that the row has been aged out and may be written to. In this manner the ARP table automatically learn new entries.

Processor Port

The embodiment shown in Figure 4 includes a processor port 170 in communication with the data bus. In one embodiment the processor port comprises an interface that emulates a standard SRAM memory interface. As a result, the processor port 170 allows the router 104 to be connected to various types of processors. The processor port 170 is compatible with most popular processors, such as MIPS, StrongARM, Motorola 683xx and other general purpose processors that may or may not include synchronous bus interface.

In a preferred embodiment, the processor port 170 supports a maximum clock of 66 MHz, yielding up to 2 Gbit/s peak bandwidth. The processor port 170 performs clock synchronization and contains several FIFOs to increase burst throughput. In order to maximize the bandwidth utilization for high-speed data transfers, a flexible flow control interrupt scheme is devised and four FIFOs (Control Read, Control Write, Data Read and Data Write FIFOs) are implemented to allow burst read/write. The processor port 170 is further configured for memory interface via a processor to allow system original equipment manufacturers (OEMs) to manage the router 104 as a memory-mapped device. The processor port 170 supports the use of an external controller or a CPU (shown in Fig. 1) to allow existing systems to integrate the router 104 while maintaining use of existing network operating system software. The processor port 170 communicates with a processor (CPU) 110 via a processor port bus 125.

In at least one embodiment the external processor 110 (not shown in Figure 4) runs a multi-tasking RTOS (real time operating system). In a preferred embodiment, the external CPU 110 may perform functions such as maintaining a route table, encryption, VPN Support, handling non-IP and unknown protocols, setting up multicast sessions and system maintenance and management. A single CPU 110 may support up to eight routers 104 in the router system such that each the CPU 110 individually addresses each router 104 via the chip select lines (not shown) for each router. Thus, the processor can select multiple routers concurrently to update the route table information across the router system in a single operation. s

In various embodiments when operating with the external CPU 110, the router 104 is able to implement various layer 3 routing protocols, such as RIP (Routing Information Protocol) and OSPF (Open Shortest Path First). The external CPU 110 can also handle exception non-IP packets to insure the router system is compatible with the existing and older networks without concern for the nature of the layer 3 traffic on the network. This feature of the router 104 is a significant improvement for operation in heterogeneous networks wherein a variety of networking protocols coexist while the network behaves in a homogeneous fashion. In one embodiment the processor port 170 achieves three major functions. First, the processor port 170 provides access to an external processor or CPU 110 (see Fig. 1) to internal resources for set up and configuration of the router 104. Second, the processor port 170 provides the external processor 110 with high-speed communication access to the packet buffer 180 (memory) and the network communication ports 140-147. Such access may be necessary to allow the CPU 110 to assist in forwarding a packet should the router 104 need assistance in accurately directing the packet if the packet requires modification. Third, the processor port 170 provides a route table look-up function to all network communication modules to facilitate automated routing.

Although most IP packets are received and subsequently retransmitted automatically without any firmware intervention, the processor 210 must handle some packets, such as by way of example, broadcast packets and multicast packets, ARP and other routing protocol packets, non-IP packets, packets with unknown IP headers. These types of packets may be optionally transferred into and out of the router 104 via the processor port 170.

In some embodiments, the processor port 170 includes a packet buffer queue for interception of packets based on their IP addresses. These packets are buffered to allow for packet modifications, moves, and insertion into queues for re-transmission. The processor port 170 facilitates transparent communication between the router 104 and the CPU 110 to implement protocols in addition to IP and the routing protocols required to manage a router. Furthermore, the control and data transactions may be separated to reduce the processing time. In operation, when a packet arrives at the network communication port 140- 147, a determination is made as to whether the processor 210 intervention is required and, if so, the packet is transferred to the appropriate queue in the packet buffer 132. The network communication port 140-147 then sends a UPRR (Unicast Packet Resolution Request) or MPRR (Multicast Packet Resolution Request) transaction to the CPU 110 via port 170. The UPRR and MPRR transactions contain information that helps the CPU 110 determine the course of action. The CPU 110 can then request a complete packet or portions of a packet that may be stored in a CPU memory in communication with the CPU 110 and port 170. The CPU 110 can also discard the packet or redirect it to the proper queue for re-transmission.

To further save cycles in operation of the router 104, the port 170 communicates with the CPU 110 in response to interrupts. Interrupts are the mechanism used by the router 104 to inform the CPU 110 about exception handling requirements. When an interrupt condition occurs, an interrupt services routine is called which causes a jump to an interrupt handler. In the interrupt handler, the CPU 110 saves its then existing status and determines the source of the interrupt. After determining which peripheral and condition have caused the mterrupt, the CPU 110 responds by reading/writing a corresponding register, calls appropriate routines to handle the condition, restores the saved status and continues execution.

Data Port and External Bus

The external data bus 220 operates very similar to the data bus 160 although, the external data bus carries data between routers 104A-104C within a multi-router device (Fig. 2). In one embodiment the external data bus 220 connects to each router 104A-104C in parallel manner as shown in Figure 2.

In one embodiment, the external data bus 220 is electrically different from other parts of the router 104 in that it operates at a lower GTL level voltage to cut down on signal rise/fall delay. Although the external bus 220 operates as a synchronous bus, in a preferred embodiment, the bus clock is not explicitly distributed between the routers 104. For this reason, in such a configuration a common reference clock is used for all of the connected routers 104. It should be noted that although referred to herein as an external bus 220 of a multi-router system, the external bus is contained within the outer housing 102 of a multi-router system and is configured to interconnect routers 104 within the multi-router system to thereby achieve expandability.

The external bus 220 or system expansion bus utilizes a similar time division multiplexed (TDM) architecture as is found in the internal data bus of the router. Use of the TDM architecture provides a guaranteed portion of the bus bandwidth to all the devices connected to the external bus. As a result, the inherent overhead cycle of the prior art is overcome as data is carried at every clock cycle and the use of TDM overcomes deterministic latency. These attributes are particularly desirable when transmitting time sensitive data packets, such as those carrying voice data. Yet another feature of the TDM bus architecture of the present invention comprises dynamic TDM slot assignment. Dynamic TDM slot assignment of the present invention may occur on either or both of the internal data 160 and/or the external data bus 220. Dynamic TDM slot assignment detects when one of the routers in a multi- router system does not require use of the data bus 160, 220 during that particular router's TDM time slot assignment. Upon detection that one of the routers 104A- 104C does not have data to transmit in its bus access time slot, the present invention assigns the now open transmit slot to the next router 104A-104C. In this manner the TDM operating internal and external buses increase routing speed and more fully utilizes bus capacity.

For the external bus 220 to perform its function, each interconnected router 104 in the router system must be assigned a unique identification. One of the routers 104 may also be designated as the external bus master to function as an arbitrator. In one embodiment, this is done via writing to a Chip ID register in the processor port 170 after power-on-reset. A router 104 is assigned as bus master and arbitrates the right to access the external bus 220 for all routers 104 connected to the external bus 220.

In one embodiment the external bus port 174 includes six built-in FIFOs configured to handle incoming and outgoing transactions via the data bus 160 and external bus 220.

Data Buffering

In various embodiments of the present invention, buffering arrangements are implemented which increase router speed and efficiency over systems of the prior art. In one embodiment the present invention advantageously includes a hybrid buffering arrangement that combines aspects of input buffering with aspects of output buffering. Packet transfers to the packet buffer 166 are initiated for three possible cases. The first case is when there is a packet that needs to be transmitted through one of the network communication ports 140-147. The CPU 110 will transfer this packet using a writing packet process.

The second case is when a received packet must be substantially modified by the CPU 110 before retransmission. In this case the CPU 110 first transfers the complete packet to the CPU memory (not shown) using the read packet process.

Once the packet is appropriately modified, the CPU 110 can send the completely new packet using the write packet process, as discussed in the first case.

The third case is when the original packet needs to be sent out as received or needs to be slightly modified. If the packet needs a slight modification, the CPU 110 can perform an in-place update of the packet. Once the packet has been updated, the

CPU 110 can place a pointer for this packet in one or more transmit queue for retransmission.

Figure 17 illustrates an exemplary buffering arrangement for an exemplary input / output port arrangement. As shown, a first input port 350, and second input port 352 connect to a first input buffer 354 and a second input buffer 356. Also included are a third output port 358 and associated third output buffer 360 and a fourth output port 362 and a fourth output buffer 364. In general, input buffering comprises storing data packets arriving from an input, such as the first input port 350, in the first input buffer 354 before being accepted into the router for processing. Such buffering may occur when the router receives data packets faster than the receiver can analyze and transmit the packets from the router. In contrast output buffering comprises storing data packets in the output buffer 360 or 364 before transmitting the data packets from the router. In one embodiment, the present invention utilizes hybrid buffering. Hybrid buffering comprises a combination of input and output buffering wherein the packets are initially buffered, if such buffering is necessary, at the output side 360, 364. This manner of buffering continues and is monitored for excessive buffering at the output. If excessive buffering occurs such that one of the output buffers is full, additional buffering is implemented in the form of input buffering 354, 356.

In various embodiments of hybrid buffering, different implementations of input buffering may occur. In one embodiment a memory manager monitors the input port(s) that is/are receiving data packets to determining the cause of the output port becoming excessively buffered. Usually, the cause is excessive data from an input port as shown by arrows 370. Once the input port is identified, input buffering is established on that input port. Once input buffering is established, that particular input may be optionally shut down or incoming packets discarded for a period of time. This allows data 372 to be transmitted out port 4 362. Another manner of input buffering is implemented wherein all of the inputs are forced into a mandatory buffer state.

Hybrid buffering therefore overcomes the disadvantage associated with input only buffering. For example, a plurality of data packets 370 arrives on the first port 350 and is destined for the fourth output port 362, and data packet 372 arrives on the second port 352 and is destined for the fourth output port 362. However the fourth port 362 is busy and the fourth buffer 364 is full of packets from Port 1. Hence, the second input port data 372 must undesirably wait. The present invention overcomes this disadvantage of input buffering IP. In yet another embodiment, the present invention utilizes a multi-queue arrangement for port buffering. Figure 18 illustrates a multi-queue arrangement wherein one or more input queues or buffer and output buffer is segmented into sub- queues. By way of example and not limitation, the output sub-queues 390-398 and/or input sub-queues 380-388 in one embodiment are categorized by Type of Service.

Thus, the Type of Service associated with the packet determines the queue into which the packet is placed. Queue types may comprise high priority, medium priority, and low priority, voice packets, and discard or unwanted packets. In other embodiments, different types of service may also be established based on a service level established by a service provider. Thus packets from and/or to users who obtain the highest service level are given priority service (i.e. the highest priority queue) while packet from and/or to low service level customers use the low priority queue. In various embodiments any number of output queues per port may be established, although in preferred embodiment eight output queues per port are established. Likewise, on the input side, multiple input queues 380-388 may be established depending on the particular needs of the designer. These multiple input queues 380- 388 are categorized based on the different types of web traffic. Thus, similar types of data packet traffic are directed certain queues in the multi-queue input queue system. Thus the multiple input queues 380-388 each contain categorized data packet traffic grouped by type of data packet traffic. Such categorization allows the memory manager to monitor the individual sub-queues 380-388, and hence, the type of traffic. In various other embodiments, a combined form of multiple queue input output buffering is utilized. Data in any one or more sub-queues may be discarded as needed. Thus in operation and as shown by Figure 19, at a step 400 the router executes a typical receive packet process. Next, at step 402 the operation analyzes the packet to determine the packet type. Various types of packets include but are not limited to packets which fall into categories based on the 1) Type of Service assigned to a particular packet as might be contained in the 8 bit field in the address header, 2) the

IP destination route, or 3) application types as based on the layer 4 data contained within the packet.

Thereafter, at a step 404, the router executes route processing on the packet to determine next hop addressing. Thereafter, the packet is stored at a step 406. It is contemplated that the packet may be stored in an inputs sub-queue corresponding to the packet's category, or the packet is stored directly to the output queue if the desired output port is on the same chip. Thus, the packets are stored in a unique sub-queue. Each port on the router has a queue segregated in this manner as shown in Fig. 18 elements 380-398. Subsequently at step 408, the processed packet is transferred to the corresponding output queue based on the priority of service assigned to the packet. High priority packets are assigned to output sub-queues with high priority, low priority packets are assigned to output sub-queues with low priority. The priority of a sub-queue determines that sub-queue's access to the output port. For example in one embodiment, a high priority output sub-queue is given priority for 60 of 100 (60%>) of the output opportunities for a port while a low priority sub-queue may only be given 10 of 100 (10%>) of the output opportunities for a port. Thus, high priority sub-queues are given priority to gain access to the output port. Next, at a step 410, the operation transmits the data in the sub-queue based on the sub-queue priority. Thereafter, at a step 412, an analysis is made regarding whether there is output queue overloading. If there is not output queue overloading, i.e. the output queue has not exceeded a predetermined capacity, the operation returns to a step 400 wherein the queuing process is repeated. Alternatively, if one or more of the output queues are overloaded, the operation progresses to a step 414 wherein the system determines which input sub-queue is misbehaving. Such determination can be achieved by analyzing the status and history of the several input sub-queues for each input port. Upon determining which input sub-queue(s) is/are misbehaving, the treatment of that particular sub-queues(s) is adjusted accordingly at a step 416. One form of adjustment comprises dropping all inputs arriving at the misbehaving input port. Another form of adjustment comprises dropping all packets receive through one or more ports that are channeled to one of the particular sub-queues that are associated with the overload. Other arrangements are contemplated for handling excessive data from a particular port sub-queue.

Operation

Figure 20 illustrates an operational flow diagram of one exemplary method of operation of the present invention. For purposes of understanding, a router operating under the principles of the present invention includes capability to perform as a router in a packet switched network. Accordingly, functions of the present invention known by those of ordinary skill in the art are not described.

As known by those of ordinary skill in the art, various types of packets are transmitted over current packet switched networks. One example packet type is an ARP packet, another is a data packet. Figure 20 illustrates an exemplary method of operation of the router for determining the packet type and a method for processing ARP packets. Figure 21 A and 21B illustrates an exemplary method of operation for processing packets determined to be data packets. In reference to Figure 20, at a step 450 the operation receives a packet at an input port of the router device. Upon receipt of the packet the router is not aware of the type of packet received. Next at a step 452, the operation analyzes the MAC address of the received packet and updates the ARP table accordingly.

Next, at decision step 454, the operation determines if the received packet is an ARP packet or a standard data packet. If the received packet is an ARP packet, then the packet is a request from another device for a response regarding the devices connected thereto. In this manner the ARP tables of the routers in the network are updated and maintained. In response to an ARP request packet, the router sends a response. Alternatively, if the packet is a standard data packet that is to be forwarded to a port on the router or another router, the operation progresses to a step 456. Step 456 references Figures 21A and 21B, both of which regard transmission and reception of a data packet. If however, at decision step 454 the operation determines that the packet is an ARP packet the operation progresses to a step 458 wherein the port isolates the senders address within the received packet.

Thereafter, at a step 460, the operation updates the router's dedicated ARP table with the sender's address. This occurs to ensure that the entry is not aged out of ARP table and to keep the table current. Next, at a step 462 the operation isolates the destination address contained within the ARP packet about which the sending device is inquiring.

In response, at decision step 464, the operation utilizes the ARP table to determine if the router is the next hop router for data having the destination address identified in the ARP packet. If the router is not the destination identified in the ARP packet or the next hop for the destination identified in the ARP packet, the operation progresses to a step 466 wherein the operation drops the packet. Alternatively, some other action may be taken to account for a packet for which a destination is not known. Alternatively, if the router is the destination identified in the ARP packet or the next hop for the destination identified in the ARP packet, the operation progresses to a step 468 and the router transmits a response to the sender of the ARP packet regarding the path to the packet destination. It should be understood that this is but one possible method of operation for receiving and responding to ARP packets. It is fully contemplated that other methods of operation are available which do not depart from the scope of the present invention.

Figures 21 A and 2 IB illustrate an operational flow diagram of one exemplary method of operation of data packet transmission, reception and routing in accordance with the present invention. This process occurs when the received data packets are data packets intended for reception by a device connected to the router or when the router is on the path to the packet destination.

In reference to Figure 21 A, at a step 500 a user's computer generates a data packet in the form of a data request. At a step 502, the user's computer packages the data request into packet format. One example of such a data request is generated by a user at a computer requesting data from a remote server, such as a communication request over the Internet for information from a web site. Generation and packaging of the data request is performed by software and hardware on the user's computer and accordingly is not discussed herein.

At a step 504, the user requesting data transmits the data request onto the network. As known by those or ordinary skill in the art, within the packet is information identifying the intended destination of the packet and the address of the sender of the packet. For purposes of the present discussion it is assumed that the computer network over which the data packet is sent is a packet switched network operating under the standard as laid out in RFC 1812, which is fully incorporated by reference herein and known by those of ordinary skill in the art. It fully anticipated that the principles of the present invention may operate and be applied under other standards than that provided in RFC 1812. At one or more of the routers intermediate the destination device as identified by the destination address, a router receives the data packet, step 506. Thereafter, at a step 508, the router performs physical layer IP processing on the packet. Physical layer IP processing, which is generally known by those of ordinary skill in the art, comprises, but is not limited to, verifying MAC layer address matches, error checking and monitoring of broadcast rules. In one embodiment (shown in Fig. 1), this occurs in a physical layer chip 108 residing intermediate the router 104 and the ports 116A- 116D.

After physical layer processing, the operation at step 520 receives the packet header at the router circuitry. The packet heading contains the address information, type of service information and packet type information for the data packet. Utilizing the information from the packet header the operation performs layer two processing on the data packet step 522. Thereafter, at step 524, the operation passes the packet to a layer 3 engine for layer 3 processing. As known by those of ordinary skill in the art, layer 3 processing comprises, but is not limited to, route table look-up process, header check-sum process and TTL decrementing.

In accordance with layer 3 processing the operation, at step 526, performs flow table look-up. The router performs flow table look-up to identify the Type of Service assigned to the received packet. This information is located in the address header portion of the packet header. An exemplary address header is shown in Figure

22. In one embodiment various types of service may comprise drop packet, forward packet, forward packet with priority, or inform CPU regarding packet. For example, the router, and in particular the flow table may optionally be instructed to take action on certain types of packets. For example if the packet is identified to be a voice packet or streaming video packet then the flow table would instruct the router to forward this packet with priority because voice data and streaming video data is time sensitive. Similarly, packets arriving from a source banned from use of the router will be dropped.

After flow table look-up and assignment of packet handling instructions the operation performs route table look-up, step 528. In this example method of operation, it is assumed that the route table, upon analyzing the destination address of the packet determines that the packet should be routed to port 7 of router 1. The route table matches the destination address of the data packet to a router port to provide the data packet to the proper next hop or to the packet destination. Next, at a step 530 of Figure 21B, the operation determines the proper output sub-queue in which to place the packet. The router determines the proper output sub- queue to utilize based on the Type of Service designation assigned to the packet by either or both of the route table and the flow table. After this process, at a step 540 the operation awaits the arrival of the remainder of the packet. As can be understood, the above-described process occurs at high speed as it most often is complete before arrival of the payload of the packet.

Next, at a step 542, the system allocates buffer space in the proper output sub- queue for storage of the packet until the proper route is determined and the output port is available for data transmission. In one embodiment the allocation of buffer space, i.e. memory space, occurs on a dynamic basis based on known memory allocations for each of the plurality of sub-queues. Next, at a step 544, an address to the allocated buffer space in the memory is assigned for the data packet. This address is utilized by the several systems of the router when accessing the data packet. At a step 546 the data packet is transferred to the packet buffer at the assigned address. The router stores the packet at this location until the output port is ready to transmit the packet to the device connected thereto.

Next, at step 548, the router memory manager monitors the desired port and the data packet's position in the sub-queue so that when the port, in this example, port 7 of the first router, is able to send the packet it can be provided for packaging and transmission. When the packet is ready to be sent, the operation, at a step 550 provides the packet to the ARP table module for repackaging. The ARP table is used to obtain the MAC address coπesponding to the destination IP address. At a step 551, the operation may optionally attach a tag to the packet. A tag comprises an additional packet information item for sorting, filtering, or identifying packets in an efficient manner. Thereafter, at a step 552, the router system transmits the data packet via port

7 on router 1.

This is but one exemplary method of operation of router configured in accord with the present invention. It is fully contemplated that other methods of operation are possible which do not depart in scope from the teachings of the invention as described herein.

While particular embodiments and examples of the present invention have been described above, it should be understood that they have been presented by way of example only and not as limitations. Those of ordinary skill in the art will readily appreciate that other various embodiments or configurations adopting the principles of the subject invention are possible. The breadth and scope of the present invention is defined by the following claims and their equivalents, and is not limited by the particular embodiments described herein.

Claims

CLAIMS What is claimed is:

1. A router having two or more input/output ports for use in a packet switched network to route data packets comprising: an input/output port configured to receive data via a packet switched network; a router device comprising an input queue in communication with the input/output port; memory configured to buffer data; a route table configured to store and provide route information for data packets stored in memory, wherein the route table in embodied in content addressable memory; an output queue in communication with the input/output port; and a bus configured to facilitate communication between the input queue, the memory, the route table, and the output queue wherein said bus operates using time division multiplexing.

2. The router of claim 1 wherein said input queue and said output queue are divided into sub-queues.

3. The router of claim 1 further including a bus controller to oversee time division multiplexed bus operation.

4. The router of claim 1 wherein said bus control further implements dynamic time division multiplexed time slot allocation.

5. The router of claim 1 further including an address resolution protocol table embodied in content addressable memory.

6. The router of claim 1 wherein an address resolution protocol table is associated with each input output port.

The router of claim 1 further including a flow table.

8. The router of claim 1 further including a processor interface in communication with the bus.

9. The router of claim 1 further including an external communication port and external communication interface in communication with said bus to facilitate communication with another router.

10. The router of claim 1 wherein the memory is located external from the router.

11. A router having a two or more input ports, the router for use in a packet switched network, the router comprising: an input port module associated with each input port, the input port module configured to receive data packets; an address resolution table associated with each input port module, the address resolution table configured to convert physical address to IP addresses; a memory structure configured to store data packets received from the input port module during route table look-up; and a route table configured to provide route table look-up for data packets.

12. The router of claim 11 wherein each address resolution protocol table is embodied using content addressable memory.

13. The router of claim 11 wherein the memory structure comprises SRAM.

14. The router of claim 11 wherein each address resolution table is independently updated.

15. A method for routing data packets having an address header containing address information in a packet switched network comprising: receiving a data packet storing said data packet in memory; performing route table look-up packet information to obtain a next-hop address, the route table embodied in a content addressable memory; performing flow table look-up based on information from the packet; assigning a type of service to the packet based on the performing flow table lookup; prioritizing the packet in relation to other packets based on the type of service; assembling and attaching a data packet address header to the data packet using the next hop IP address obtained during route table look-up; converting the IP address of the packet to a MAC address; and transmitting the packet.

16. The method of claim 15 wherein converting comprises performing a internet protocol address to network layer address conversion using a look-up table embodied in content addressable memory.

17. The method of claim 15 further including placing the data packet into a queue after prioritizing the data packet.

18. The method of claim 17 wherein the queue comprises a plurality of sub- queues.

19. A multi-router system defined by two or more routers contained within a protective housing, each router having one or more ports, the system comprising; two or more routers wherein each router is associated with a dedicated memory structure; a plurality of ports, each port having an ARP table associated therewith; a bus configured to facilitate communication between the two or more routers; and a protective outer housing generally surrounding the multi-router system.

20. The system of claim 19 wherein one router of the two or more routers is configured to serve as an arbitrator for the bus.

21. The system of claim 19 wherein the bus is configured to transmit data using time division multiplexing.

22. The system of claim 19 wherein the dedicated memory structure comprises RAM.

23. The system of claim 19 further including a processor in communication with each of the two or more routers.

24. A method for determining routing information for a packet in a router operating in a packet switched network: receiving a packet, the packet having a data portion and an address portion; reading aspects of the address portion of the packet; performing longest prefix match searches regarding packet information using a modified content addressable memory structure to obtain a next hop IP address and a router port assignment; modifying the address portion of the packet to reflect the next hop IP address; and transmitting the modified data packet to the router port assignment.

25. The method of claim 24 wherein reading portions of the address portion comprises reading at least a destination IP address portion of the address portion.

26. The method of claim 24 wherein performing longest prefix match searches comprises inputting the IP address into a content addressable memory with associated logic configured to find the longest prefix match of a destination IP address.

27. The method of claim 24 further including the step of storing the packet in memory while determining routing information.

28. A method of routing data packets in a packet switched network wherein different packet types are assigned different levels of service, the method comprising: receiving a packet, the packet having an address portion and a data portion; analyzing the packet to determine which of several types of service to provide to the packet; and routing the packet based on the type of service associated with the packet as determined by analyzing the packet.

29. The method of claim 28 wherein analyzing the packet comprises analyzing any of a type of service field, a destination address, or application information.

30. The method of claim 28 wherein analyzing the packet comprises providing the packet to a flow table configured in a content addressable memory.

31. The router of claim 30 wherein analyzing further comprises executing a longest prefix match search of routes.

32. A method for transmitting data in a routing device having at least one input port module, a memory structure, a route table and at least one output port module comprising: receiving a data packet at an input port module; transmitting the data packet during an assigned time slot on a bus operating under time division multiplexed standard from the input port module to a memory structure; performing a route table look-up for the next hop address for the data packet; updating an address header associated with the data packet to create an updated data packet; transmitting the updated data packet during an assigned time slot on a bus operating under time division multiplexed standard from the memory structure to the output port module.

33. The method of claim 32 wherein performing a route table lookup occurs in hardware using a longest prefix match search.

34. The method of claim 32 further comprising reassigning one or more time slot if a time slot is not being utilized.

35. The method of claim 32 wherein one of the input port module or output port modules serves as a bus controller to oversee operation of the time division multiplexed bus.

36. The method of claim 32 further including; transmitting the updated data packet to an external bus interface; and transmitting the updated data packet to a second routing device in an assigned time slot on the external bus operating under time division multiplexed standard from the external bus interface to the second routing device.

37. A method for queuing data packets in a packet switched device comprising: evaluating the data packet to determine a data packet type; storing said data packet into one of two or more output sub-queues that corresponds to the data packet type; and outputting data from the two or more sub-queues based on a priority assigned to each of the two or more sub-queues.

38. The method of claim 37 further including assigning priorities to each of the two or more sub-queues wherein the priority of the sub-queue dictates how often each sub- queue may utilize the output port.

39. The method of claim 37 wherein evaluating the data packet comprises analyzing a portion of the address header.

40. The method of claim 37 wherein storing includes allocating a memory address and space in memory.

41. The method of claim 37 wherein packets containing voice data are assigned a higher priority than packets containing text data.

42. The method of claim 37 further including the step of providing notice to an output port controller to request access to the output port.

43. A method for reducing transmission delay associated with data packet queuing comprising: storing data packets in an output queue associated with an output port until the output port is able to output the data packets; monitoring the output queue to detect when the output queue is at capacity; and storing data packets in an input queue associated with an input port when monitoring reveals that the output port is at capacity to reduce delay associated with the output queue.

44. The method of claim 43 further including monitoring one or more input ports to determine which input port is causing said output queue to be at capacity.

45. The method of claim 44 further comprising deleting data packets arriving at the input port that is causing said output queue to be at capacity.

46. The method of claim 43 wherein the input queue and the output queue comprise memory structures.

47. The method of claim 43 wherein the input queue comprises a plurality of sub- queues.

48. The method of claim 43 wherein the output queue comprises a plurality of sub-queues.

49. The method of claim 43 further comprising: analyzing the data types; assigning priority to the data packets based on the analyzing; and storing the data packets into one or more input sub-queues and output sub- queues based on the priority assigned to the data packets.

50. The method of Claim 28, wherein the method of routing data packets occurs at Layer 2 speed in an IP based network.

51. The method of Claim 50, wherein analyzing the packet to determine which of several types of service to provide is achieved in a content addressable memory having a ranging feature.

52. The router of Claim 11 , wherein the address resolution table is configured to operate at Layer 2 speed of the packet switched network.

53. The method of Claim 24, wherein determining routing information occurs at Layer 2 speed in an IP protocol based network.

54. A method for performing a data look-up to determine if a match value is within a range of memory entries to obtain an output from a content addressable memory structure, said content addressable memory structure having a match value input and a memory output, the method comprising: storing in one or more memory cells, a low range value and a high range value; entering a match value on said match value input; executing a compare process to determine if said match value is intermediate said low range value and said high range value; and providing on said memory output a value associated with said low range value and said high range value if said executing determines said match value is intermediate said low range value and said high range value.

55. The method of Claim 54, wherein said one or more memory cells comprise a TAG system.

56. The method of Claim 54, wherein entering a match value comprises entering a value to be looked up in said content addressable memory structure.

57. The method of Claim 54, wherein executing a compare process comprises performing a first compare to determine if said match value is greater than said low range value and less than said high range value.

58. The method of Claim 54, further including resetting an aging module upon said providing said memory output a value.

59. A method for performing a range look-up in a content addressable memory structure, said content addressable memory structure having a range entry and an associated output, said range entry having an upper limit and a lower limit, the method comprising: providing an input on an input line to said content addressable memory structure; comparing said input to determine if said input is greater than said upper limit; comparing said input to determine if said input is less than said lower limit; and outputting said associated output if said comparing reveals that said input is intermediate said upper limit and said lower limit.

60. The method of Claim 59, wherein said input comprises a 16 bit digital value.

61. The method of Claim 59, wherein said associated output is stored in SRAM.

62. The method of Claim 59, wherein said comparing said input to determine if said input is greater than said upper limit comprises; providing said input to a logic structure; obtaining said upper limit from a memory cell; comparing in sequential order the most significant bits of said input to said upper limit to determine if said input is greater than said upper limit.

63. A memory structure configured to output an associated value upon entry of an input having value within a range, said range having first value and a second value, the structure comprising: a first memory space storing said first value; a second memory space storing said second value; an input line configured to receive said input to said memory structure; and one or more comparators connected to said first memory space, said second memory space, and said input and further configured compare said first value to said input and said second value to said input and provide an associated value if said input between said first value and said second value.

64. The memory structure of Claim 63, wherein said memory structure comprises a content addressable memory structure.

65. The memory structure of Claim 63, wherein said first memory space and said second memory space comprise a plurality of memory cells.

66. The memory structure of Claim 63, further including a third memory space configured to store said associated value, said third memory space including logic configured to output said associated value if said one or more comparators determine that said input is intermediate said first value and said second value.

67. The memory structure of Claim 63, wherein said memory structure is configured as a content addressable memory having a plurality of rows, each row having at least one said associated value and at least one corresponding input.

68. A content addressable memory structure configured to include range look-up for two or more TAG values and output an associated memory value associated with each TAG value, said content addressable memory structure further comprising; a plurality of first memory cells configured to store a first range value; a plurality of second memory cells configured to store a second range value; a match value input configured to receive an input to said content addressable memory; and a logic structure configured to compare in order of significance the bits of the match value to both of the first range value and the second range value to determine if said match value is intermediate the first range value and the second range value.

69. The memory structure of Claim 68, wherein said logic structure comprises: a first comparator set configured to compare each bit of the match value input to each bit of the first data range; a second comparator set configured to compare the output of the first comparator set to determine the magnitude relationship between the match value input and the first data range.

70. The memory structure of Claim 68, wherein said memory structure is within a router.

71. The memory structure of Claim 68, further including an output memory structure configured to store said associated memory value and output said associated memory value upon receiving a signal from said logic structure.

72. The memory structure of Claim 68, wherein said first range value and said second range value are TAG values.

73. A method for aging entries in a content addressable memory wherein each entry is associated with an age memory indicative of the last use of said entry in relation to a counter, said counter repeating in a generally consistent manner, the method comprising: continually incrementing said counter device in accord with operation of said content addressable memory; writing an entry in said content addressable memory; writing said counter output as a value to said age memory cell associated with said entry; and ceasing further use of said entry if said counter output matches said value in said age memory cell.

74. The method of Claim 73, wherein said counter comprises a three bit counter.

75. The method of Claim 73, wherein said age memory cell comprises a three bit memory cell.

76. The method of Claim 73, wherein said entry comprises a TAG system entry and associated memory entry.

77. The method of Claim 73, wherein ceasing further use comprises setting a flag bit to indicate when said entry is aged.

78. A method for tracking the validity of an entry in a content addressable memory, the content addressable memory having a plurality of entries and associated data values in the memory of the content addressable memory, the method comprising; in conjunction with the storage of or use of a valid entry in the content addressable memory, writing a counter value as a time stamp to age memory associated with said entry; comparing said counter value to said time stamp; and designating said valid entry as invalid if said counter value matches said time stamp.

79. The method of Claim 78, wherein designating comprises setting an invalid entry flag in memory.

80. The method of Claim 78, wherein said time stamp comprises a writing of said counter value to age memory to record the counter value at the time of initial storage or use of an entry in the content addressable memory.

81. The method of Claim 78, wherein the counter comprises a three bit counter and age memory is configured to store three bits.

82. The method of Claim 78, further including checking a static memory cell to determine if said entry can be aged out.

83. The method of Claim 78, wherein storage of a valid entry in the content addressable memory further comprises resetting said designation of said entry as invalid to valid.

84. A system to age entries in a content addressable memory comprising a TAG memory space configured to store TAG values; a match memory space configured to store data associated with said TAG values; a counter configured to provide a cyclic output; an age memory associated with a TAG memory space configured to store a data value representing said counter output at the time said TAG memory space is written to or utilized; and compare logic configured compare said counter output to said data value in said age memory and disable further use of the TAG value associated with said age memory if said counter value matches said data value in said age memory.

85. The system of Claim 84, wherein said counter comprises a sequential three bit counter.

86. The system of Claim 84, wherein said age memory is an extension of said TAG memory space.

87. The system of Claim 84, wherein said compare logic comprises at least one comparator.

88. The system of Claim 84, further including an invalid memory cell connected to said compare logic, said compare logic configured to write to said invalid memory cell if said counter value matches said data value in said age memory.

89. The system of Claim 84, further including a static memory space, said system to age entries being generally disabled if so designated by said static memory space.

90. An apparatus to monitor the age of an entry in a content addressable memory structure, said content addressable memory having a plurality of memory addresses, said apparatus comprising: a counter to generate a cyclic signal, said signal comprising a counter output; and a first memory structure associated with one of said memory addresses, said first memory structure configured to store said counter output at the time data is written to said associated memory addresses.

91. The apparatus of Claim 90, further including compare logic to monitor contents of said first memory structure and said counter output, such that if said contents of said first memory structure and said counter output are identical, the said contents of said first memory structure are deleted.

92. The apparatus of Claim 90, further including a static memory cell in communication with said compare logic, said static memory cell configured to override said compare logic thereby preventing the aging out of said memory address content.

93. The apparatus of Claim 90, further including a memory flag, said memory flag configured to be set if after operation of said counter said counter value and first memory structure contents match.

94. The apparatus of Claim 90, further mcluding secondary logic to monitor for set memory flag and direct a next write to a memory address to be directed to a said memory address having associated memory flag set.