US9509622B2

US9509622B2 - Exporting real time network traffic latency and buffer occupancy

Info

Publication number: US9509622B2
Application number: US14/707,139
Authority: US
Inventors: Thomas J. Edsall; Yue J. Yang; Wei-Jen Huang; Chih-Tsung Huang
Original assignee: Cisco Technology Inc
Current assignee: Cisco Technology Inc
Priority date: 2012-09-18
Filing date: 2015-05-08
Publication date: 2016-11-29
Anticipated expiration: 2032-12-07
Also published as: CN104641602A; EP2898637B1; EP2898637A1; IN2015MN00444A; US9641407B2; US20170026256A1; CN104641602B; US20140078915A1; US9077619B2; US20150244637A1; WO2014046932A1; USRE48645E1

Abstract

Techniques are presented herein to facilitate the monitoring of occupancy of a buffer in a network device. Packets are received at a network device. Information is captured describing occupancy of the buffer caused by packet flow through the buffer in the network device. Analytics packets are generated containing the information. The analytics packets from the network device for retrieval of the information contained therein for analysis, replay of buffer occupancy, etc.

Description

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No. 13/708,265, filed Dec. 7, 2012, which in turn claims priority to U.S. Provisional Application No. 61/702,320, filed Sep. 18, 2012, entitled “Exporting Real Time Network Traffic Latency and Buffer Occupancy.” The entirety of both of these applications is incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates generally to analysis of occupancy of a buffer in a network device.

BACKGROUND

In a computer network, data is transmitted from a source to a destination in the form of packets that generally pass through one or more network devices (e.g., switches, routers, firewalls, etc.). During the transmission, certain errors may arise that result in, for example, redundant data being added to the original data, dropped packets, etc. Massively Scalable Data Center and Cloud Computing systems are putting more traffic load on network equipment such that over-provisioned networks are no longer possible. Monitoring of a buffer in a network device is useful to gain knowledge for network administration, analysis, and performance.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a network device configured to generate buffer analytics packets based on occupancy of a buffer in the network device.

FIG. 2 is a block diagram illustrating one example implementation of the buffer analytics logic.

FIG. 3 is a diagram that generally illustrates a format of a buffer analytics packet.

FIG. 4 is a flow chart depicting operations in a network device to generate and output buffer analytics packets.

FIG. 5 is a flow chart depicting operations in a device that receives and retrieves information from the buffer analytics packets.

FIG. 6 is a diagram illustrating an example of playback of buffer occupancy from buffer analytics packets.

DESCRIPTION OF EXAMPLE EMBODIMENTS Overview

Techniques are presented herein to facilitate the monitoring of occupancy of a buffer in a network device. Packets are received at a network device. Information is captured describing occupancy of the buffer caused by packet flow through the buffer in the network device. Analytics packets are generated containing the information. The analytics packets from the network device are used for retrieval of the information contained therein for analysis, replay of buffer occupancy, etc.

Example Embodiments

Complete network visibility into buffer occupancy and the ability to replay occupancy via export and post processing is important since network disruptions (e.g., microbursts) can occur at any time. Furthermore, the ability to replay buffer occupancy allows for effective diagnosis of network issues to provide corrective actions. Existing solutions such as port mirroring (i.e., Switched Port Analyzer (SPAN)) do not provide visibility of buffer occupancy. As such, presented herein are techniques for monitoring and replaying buffer occupancy.

Referring now to FIG. 1, a diagram is shown of a network environment 5 in which a network device 10 is provided that is configured to generate buffer analytics packets based on occupancy of a buffer the network device 10. The network device 10 comprises a plurality of ports 12(1)-12(N), any of which can serve as an ingress port or egress port at any time. The network device includes a buffer 14, buffer analytics logic 16, a central processing unit (CPU) 18 and memory 19. It should be understood that there are other components of the network device 10, such as a switch fabric or application specific integrated circuit (ASIC), and the buffer 14 may reside the switch fabric. There are typically numerous buffers in the network device 10, but for simplicity only one is shown in FIG. 1. It should be understood that the techniques presented herein are useful for each of a plurality of buffers in a network device. The buffer analytics logic 14 may be implemented in hardware by digital logic gates (and embedded in the switch fabric) or by software stored in memory 19 and executed by CPU 18.

Packets

20 arrive at the network device 10 via any of the ports 12(1)-12(N). FIG. 1 shows an example where packets are arriving at ports 12(1), 12(2) and 12(3). The network device 10 is coupled to a network 40, e.g., a local area network or wide area network (the Internet), via ports 12(5)-12(N) to ultimately communicate with any one or more of the network devices 50(1)-50(M).

Generally, the buffer analytics logic 16 captures information describing occupancy of the buffer 14 caused by packet flow through the buffer in the network device 10, and generates buffer analytics packets 30 containing the information. As will become apparent from the description below in connection with FIG. 2, there are two types of buffer analytics packets: enqueue buffer analytics packets and dequeue buffer analytics packets. The buffer analytics packets 30 are then output from the network device 10 at a programmable time schedule (or based of packet size) in any one of several ways to allow for replay of the occupancy of the buffer.

First, the network device 10 may insert into buffer analytics packets 30 an address for a destination of the buffer analytics packet, e.g., address for any device connected to the network 40, such as collector device 60 having a CPU 62 and memory 64. The network device 10 sends the analytics packet 30 via network 40 to the destination collector device 60, which may be at any location, local or remote from network device 10.

Second, the network device 10 may output the analytics packet 30 to a dedicated port, e.g., port 12(4) of the network device 10 to which a collector device 70 is connected. The dedicated analytics port 12(4) can participate in port channel or fixed port distribution to expand bandwidth to a single or multiple monitor ports. The collector device 70, since it is connected directly to port 12(4), is usually local to the network device 10. The collector device 70 includes a CPU 72 and memory 74.

Third, the analytics packets 30 may be output to the onboard CPU 18 and memory 19 in the network device 10, such that CPU 18 and memory 19 also serve as a collector device. In any of these scenarios, the

CPUs

18, 62 and 72 may replay and analyze the occupancy of the buffer 14 based on software instructions stored in its

associated memory

19, 64 and 74, respectively. Moreover, the analytics packets are stored in the

memory

19, 64 and 74 for the associated

CPU

18, 62 and 72, respectively.

The network device 10 can be any network device now known or hereinafter developed, including a switch, router, gateway, a software stack on a host device, virtual network interface cards (VNICs) virtual switches, physical network interface cards (including those that support virtualization).

Memory

19, 64 and 74 may comprise read only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. Thus, in general, the

memory

19, 64 and 74 may comprise one or more tangible (non-transitory) computer readable storage media (e.g., a memory device) encoded with software comprising computer executable instructions and when the software is executed (by the associated CPU) it is operable to perform the operations described herein.

Reference is now made to FIG. 2 for a more detailed description of the buffer analytics logic 16. FIG. 2 shows that the buffer analytics logic 16 comprises an enqueue analytics packet generator 80 and a dequeue analytics packet generator 82. In addition, there are an admission control block 84, a departure control block 86, a packet assembler 88 and a multiplexer 90. The admission control block 84 and departure control block 86 are commonly found in a network device and are hardware (or software) blocks used to make processing decisions, such as a drop, scheduling, rate limiting, policing, shaping, etc.

The enqueue analytics packet generator 80 is configured to generate an analytics packet, called an enqueue buffer analytics packet shown at reference numeral 32, that describes/summarizes a packet being enqueued into buffer 14. Similarly, the dequeue analytics packet generator 82 is configured to generate an analytics packet, called a dequeue buffer analytics packet shown at reference numeral 34, that describes/summarizes a packet being dequeued from buffer 14. The packet assembler 88 assembles a packet 20 ready out from the buffer 14 for output from the network device.

The enqueue analytics packet generator 80 captures, for a packet enqueued to buffer 14, information describing one or more of identification of ingress port of arrival of the packet at the network device, Layer 2 source address and destination address, Layer 3 source address and destination address, Layer 4 source address and destination address, class of service, and timestamp of arrival at the ingress port. Similarly, the dequeue analytics packet generator 82 captures, for a packet dequeued from the buffer 14, information describing one or more of identification of egress port for departure of the packet from the network device, Layer 2 source address and destination address, Layer 3 source address and destination address, and timestamp of departure from the egress port.

The enqueue buffer analytics packet 32 generated by the enqueue analytics packet generator 80, dequeue buffer analytics packet 34 generated by the dequeue analytics packet generator 82, and packet 20 output by the packet assembler 88, are all supplied to a corresponding input of the multiplexer 90. The multiplexer 90 selectively outputs, at any given time, either a packet 20, an enqueue buffer analytics packet 32 or a dequeue buffer analytics packet 34. Priority is given to output of a packet 20 in order to maintain proper flow of network traffic through the network device 10. Trigger for output of an analytics packet may be based on time (according to a schedule) or size of a packet enqueued to the buffer or dequeued from the buffer.

Reference is now made to FIG. 3. FIG. 3 shows an example format of an enqueue buffer analytics packet 32 or dequeue buffer analytics packet 34. As explained above, an enqueue buffer analytics packet 32 summarizes a packet that is being enqueued to a buffer and a dequeue buffer analytics packet 34 summarizes a packet that is being dequeued from the buffer. These analytics packets, when accumulated over time for packets that pass through the buffer, allow for playback of occupancy characteristics of the buffer and traffic flow of packets through the buffer. As shown in FIG. 3, an enqueue buffer analytics packet 32 and a dequeue buffer analytics packet 34 includes an Ethernet Header field 100, a Common Header field 110, one or more Records fields 120(1)-120(N) and a cyclic redundancy check (CRC) field 130.

The Ethernet header field 110 is field that is used to encapsulate the destination address of the analytics packet, e.g., to direct the analytics packet to a destination, i.e., a local or remote collector device (as indicated in FIG. 1), including to the CPU of the network device itself. To this end, the Ethernet header field 110 includes information, such as media access control (MAC) destination address/source address (DA/SA), optional IEEE 802.1q virtual local area network (VLAN) routing information, an optional Internet Protocol (IP) header including an IP SA and IP DA. Again, the Ethernet header field 110 contains information used to route the buffer analytics packet to its desired destination.

The common header field 110 contains information captured from the header of a packet that has been enqueued to or dequeued (as the case may be) from the buffer. Thus, the common header field summarizes the header of a packet that is enqueued to and dequeued from the buffer in the network device. For example, the common header field includes information for a common header version (to allow for backward/future compatibility), timescale information representing the timescale of the enqueued or dequeued packet, a timestamp of the packet arrival and/or departure to/from the buffer to allow for replay, a record number to allow a collector to determine how many, if any records, have been lost in between the current analytics packet and the last received analytics packet, and one or more user defined fields such as class of service, type of service, etc.

The record field 120 contains data for an enqueued or dequeued packet that a user configures the buffer analytics logic to capture. Examples of data that may be include in a record field includes:

Format version to indicate a format version of the record field for backward/future compatibility.

L2 Header Fields (MAC SA/DA) or compressed versions (i.e. last 24 bits) and priority

L3 Header (IP SA/DA) or compressed versions (i.e. last 16 bits) and priority and protocol type

L4 Header (TCP/UDP SA/DA)

User defined fields, including one or more of:

- Input/output port
- Drop—an indication of whether the packet was dropped.
- Queue id—identifier of the queue (unicast or multicast) to which the packet is associated.
- Queue length—length of the queue to which the packet is associated.
- Packet length—overall length of size of the packet.
- Timestamp (absolute or relative to common header from protocols such as Precision Time Protocol (PTP) or Network Time Protocol (NTP))
- Programmable bytes—any user configurable one or more bytes of the payload of a packet

Internally specific fields such as logical interface mapped from table with keys such as {ingress/egress port, vlan}

Last record—to indicate that this is last record field in the analytics packet.

Thus, to summarize, the record field 120 for an analytics packet contains information about an enqueued packet or dequeued packet to describe buffer occupancy characteristics such as overall buffer occupancy, buffer occupancy based on packet priority, unicast queue length, multicast queue length; packet properties such as drop, port mirrored, load balanced, bridged or routed, and packet length; and packet error properties such as Cyclic Redundancy Check (CRC), and various error protocols such as Runt, Giant, and Jabber. More specifically, for a packet enqueued to the buffer, information is included in the record field describing one or more of identification of ingress port of arrival of the packet at the network device, Layer 2 source address and destination address, Layer 3 source address and destination address, Layer 4 source address and destination address, class of service, and timestamp of arrival at the ingress port. Similarly, for a packet dequeued from the buffer, information is included in the record field describing one or more of identification of egress port for departure of the packet from the network device, Layer 2 source address and destination address, Layer 3 source address and destination address, and timestamp of departure from the egress port. Other examples of data captured into user defined fields include an indication of a packet being rate limited, shaped, policed as well as any programmable bytes of the packet including payload.

The size of the analytics packet (Ethernet header field, common header field and records) may be the Maximum Transmit Unit (MTU), a switch specific analytics MTU, determined using a time-based method (e.g., analytics packet generated and transmitted at predetermined times), determined based on a selected number of packets, or by other techniques.

Reference is now made to FIG. 4. FIG. 4 provides a flow chart that depicts the high level operations performed in a network device in generating and outputting analytics packets. At 200, a network device receives a packet. At 210, the network device captures information describing occupancy of a buffer caused by packet flow through the buffer in the network device. At 220, an analytics packet is generated for each packet that is enqueued to and/or dequeued from the buffer. At 230, a destination address is inserted into the analytics packet. At 240, the network device processes the packet in the normal course, and outputs an analytics packet to its destination (local or remote network destination) or to a local CPU of the network device. The capturing, generating, and outputting operations are triggered to be performed based on at least one of time and size of enqueued packet or dequeued packet.

FIG. 5 illustrates a high level flow chart depicting the operations performed at a destination of the analytics packets. At 300, a collector device receives the analytics packets over time. At 310, the collector device parses the analytics packets to retrieve information in the individual records as well as the common header, and uses this information to replay buffer occupancy, perform traffic latency and perform other analysis.

FIG. 6 shows an example of how a replay of buffer occupancy, subject to certain filtering criteria, may be made. In FIG. 6, a “*” represents data that has been stored into buffer and lack of “*” represents absence or removal of data from the buffer.

By generating and exporting analytics packets that summarize properties of packets enqueued to and dequeued from a buffer in a network device, a replay of the buffer may be achieved using specific pieces of information that are of interest to network administrators and application developers. Recording each of these categories would require enormous bandwidth if a complete enqueued or dequeued packet is captured.

In summary, presented herein are techniques that enable a time-based complete replay of the buffer occupancy with resolution determined by a sampling period. These techniques provide visibility of traffic flows received by network devices. The information provided can be used by network administrators to gain insight into their specific network traffic, such as per-packet latency, buffer occupancy, and possible congestion sources. This information can lead to better allocation and provisioning of network resources, reduced congestion, and higher overall throughput. By parsing and aggregating relevant characteristics from each packet according to the techniques presented herein, bandwidth requirements associated with network monitoring are greatly reduced. As such, these techniques assist in reducing the amount of data exported for analysis.

The above description is intended by way of example only.

Claims

What is claimed is:

1. A method comprising:

receiving packets at a network device;

capturing information describing occupancy of a buffer caused by packet flow through the buffer in the network device;

generating analytics packets containing the information, each analytics packet comprising a header containing information to be used for directing the analytics packet to a destination, and a record summarizing characteristics of a packet enqueued in the buffer or a packet dequeued from the buffer;

outputting the analytics packets from the network device to a collector device; and

replaying at the collector device the information pertaining to the occupancy of the buffer over time based on the analytics packets subject to one or more filtering criteria, wherein replaying comprises visually presenting to a user the information pertaining to the occupancy of the buffer over time.

2. The method of claim 1, wherein outputting comprises:

inserting into the analytics packets an address for the collector device; and

sending the analytics packets into a network to the collector device.

3. The method of claim 1, wherein outputting comprises outputting the analytics packets to a dedicated port of the network device to which the collector device is connected.

4. The method of claim 1, wherein capturing comprises capturing, for a packet enqueued to the buffer, information describing one or more of: identification of ingress port of arrival of the packet at the network device, Layer 2 source address and destination address, Layer 3 source address and destination address, Layer 4 source address and destination address, class of service, and timestamp of arrival at the ingress port.

5. The method of claim 1, wherein capturing comprises capturing, for a packet dequeued from the buffer, information describing one or more of: identification of egress port for departure of the packet from the network device, Layer 2 source address and destination address, Layer 3 source address and destination address, and timestamp of departure from the egress port.

6. The method of claim 1, wherein generating comprises generating enqueue analytics packets and dequeue analytics packets, the enqueue analytics packets comprising information describing properties associated with a packet being enqueued to the buffer in the network device and the dequeue analytics packets comprising information describing properties associated with a packet being dequeued from the buffer in the network device, and further comprising inserting an address into the enqueue analytics packets and into the dequeue analytics packets, and sending the enqueue analytics packets and dequeue analytics packets into the network to a destination.

7. The method of claim 1, wherein capturing comprises capturing information describing buffer occupancy characteristics of the buffer including at least one of: overall buffer occupancy, buffer occupancy based on packet priority, unicast queue length or multicast queue length.

8. The method of claim 1, wherein capturing comprises capturing information describing packet processing properties for packets processed by the network device including at least one of: drop, port mirrored, load balanced, bridged or routed, or packet length.

9. The method of claim 1, wherein capturing comprises capturing information describing packet processing properties for packets corresponding to user defined parameters for one or more of: rate limited, shaped, policed or any programmable bytes of the packet including payload.

10. The method of claim 1, wherein capturing, generating, and outputting are triggered to be performed based on at least one of time and size of packet.

11. A system comprising:

a collector device; and

a network device comprising:

a plurality of ports at which packets are received at a network device and packets are sent out from the network device;

a buffer configured to buffer packets that are received at the network device and are to be processed for routing in a network by the network device; and

a processor coupled to the buffer and configured to:

capture information describing occupancy of a buffer caused by packet flow through the buffer in the network device;

generate analytics packets carrying the information, each analytics packet comprising a header containing information to be used for directing the analytics packet to a destination, and a record summarizing characteristics of a packet enqueued in the buffer or a packet dequeued from the buffer; and

output the analytics packets to the collector device;

wherein the collector device is configured to replay the information pertaining to the occupancy of the buffer over time based on the analytics packets subject to one or more filtering criteria, by visually presenting to a user the information pertaining to the occupancy of the buffer over time.

12. The system of claim 11, wherein the processor is further configured to insert into the analytics packets an address for the collector device, and to send the analytics packets into the network to the collector device.

13. The system of claim 11, wherein the processor is configured to generate enqueue analytics packets and dequeue analytics packets, the enqueue analytics packets comprising information describing properties associated with a packet being enqueued to the buffer and the dequeue analytics packets comprising information describing properties associated with a packet being dequeued from the buffer, and the processor is configured to insert an address into the enqueue analytics packets and into the dequeue analytics packets, and to send the enqueue analytics packets and the dequeue analytics packets into the network to a destination.

14. The system of claim 13, wherein the information describing properties associated with a packet being enqueued to the buffer includes one or more of: identification of ingress port of arrival of the packet at the network device, Layer 2 source address and destination address, Layer 3 source address and destination address, Layer 4 source address and destination address, class of service, and timestamp of arrival at the ingress port; and the information describing properties associated with a packet being dequeued from the buffer includes one or more of: identification of egress port for departure of the packet from the network device, Layer 2 source address and destination address, Layer 3 source address and destination address, and timestamp of departure from the egress port.

15. The system of claim 11, wherein the processor is configured to capture information describing buffer occupancy characteristics of the buffer including at least one of: overall buffer occupancy, buffer occupancy based on packet priority, unicast queue length or multicast queue length.

16. The system of claim 11, wherein the processor is configured to capture information describing packet processing properties for packets corresponding to user defined parameters for one or more of: rate limited, shaped, policed or any programmable bytes of the packet including payload.

17. A non-transitory computer readable tangible storage media encoded with instructions that, when executed by a processor, cause the processor to:

capture information describing occupancy of a buffer caused by packet flow through the buffer in a network device;

generate analytics packets containing the information, each analytics packet comprising a header containing information to be used for directing the analytics packet to a destination, and a record summarizing characteristics of a packet enqueued in the buffer or a packet dequeued from the buffer; and

output the analytics packets from the network device to a collector device at which the information pertaining to the occupancy of the buffer is replayed over time based on the analytics packets subject to one or more filtering criteria, by visually presenting to a user the information pertaining to the occupancy of the buffer over time.

18. The non-transitory computer readable tangible storage media of claim 17, wherein the instructions that cause the processor to generate comprise instructions that cause the processor generate at least one of an enqueue analytics packet and a dequeue analytics packet, the enqueue analytics packet comprising information describing properties associated with a packet being enqueued to the buffer in the network device and the dequeue analytics packet comprising information describing properties associated with a packet being dequeued from the buffer in the network device.

19. The non-transitory computer readable tangible storage media of claim 17, wherein the information describing buffer occupancy characteristics of the buffer includes at least one of: overall buffer occupancy, buffer occupancy based on packet priority, unicast queue length or multicast queue length.

20. The non-transitory computer readable tangible storage media of claim 17, wherein the instructions operable to capture include instructions operable to capture: for a packet dequeued from the buffer, information describing one or more of: identification of egress port for departure of the packet from the network device, Layer 2 source address and destination address, Layer 3 source address and destination address, and timestamp of departure from the egress port.