GB2507991A - Network latency determination using compressed timestamps - Google Patents

Abstract

As an IP packet traverses across a network each node adds timestamp information to the header of the packet as well as information identifying the node e.g. IP address. The destination node, or an inline device, e.g. firewall, near the destination node, can determine from the timestamp information and identifying information contained within a received packet the progress of the packet across the network and determine overall and individual link latency calculations or general application latency. The timestamp information is added so that it is in a compressed form and recorded in the header of the packet. As a packet traverses the network multiple timestamps may be present in the packet header. The main embodiment described is to record the time stamp in a modulo 600 (modular arithmetic) format but the format (and size of the timestamp information) can be adaptable based on the accuracy required for latency calculations e.g. nanosecond, microsecond, millisecond resolution. The arrangement also requires that the clocks of all the nodes that place timestamps in the headers are synchronised to a common clock e.g. in accordance with Precision Time Protocol, (PTP, IEEE 1588-2002). The timestamps may be added to an IP header in the Option field as defined in RFC 791.

Description

Title: Method and apparatus to include compressed tirnestamp in an IP packet.

Description:

Field of the Invention

The present invention generally relates to compress timestamp information and place them into an IF packet, these information will be used to calculate network latency and application latency if possible.

-Section 1 -Background of the invention

Latency measurement on packet networks represents new demand rising from several industries like financial service, soienrifio, media and telecommunication comnanies.

Real time data of any kind are affected by network latency and the awareness of its value can provide a significant advantage to the guality of data distribution.

Monitoring can be performed actively or passively, the first reguire generating some packets to be sent on the network, the latter relies on packet observation.

Active moniroring must be carefully implemented and ir may not Provide accurate results due the nature of network latency which is influenced by packet type, packet size, Q0S settings and by how congested the network is at any given moment.

Passive monitoring traditionally implements different mechanisms based on packets/events correlation.

The correlarion between two different observations of the same packet on differenu network points has several drawbacks, the biggest is the additional management traffic introduced.

For example on one gigabit interface can travel more Than 1.5 million packets per second and some type of metadata have to be generated to represent them and sent over the network to be correlated by another device.

Correlation doesn't also scale with observation points, the complexity of correlating two samples of the same packet is few magnitude orders smaller then several sample taken in a wide network.

The idea of marking a packet containing real application data, which they already travel on a network, presents only two drawbacks: intrusivity and overhead.

Intrusivity means a new apparatus has to manipulate real data therefore ins robustness is critical, it will also add an intrinsic latency which it is generically irrelevant.

The overhead, which limits the amount of real data that can be sent on the network, can be limited compressing the time data before including in the data packet, further considerations are required if the packet size is equal to the MTU (Maximum Transmission Unit) Placing the timestemp inside the IP packet ensures this information will travel with the application data without any need of packet correlation.

This functionality could be implemented by the originator of the data stream to be measured or by a specially designed device which is manipulating the real live traffic (usually called!Tinline device") Embodiments of the invention relate to use a device which is operating on same network packets which the latency want to be calcuiated.

Form a logical point of view e device interposed between network elements can be defined as "Inline device", a network firewall represents a valid example.

Its deploy in a generic network is illustrated in fig 1 Figi Source -> Inline -> * Packet * -> Inline -> Destination Device * network * Device The apparatus described in this invention can be implemented using an hardware device (i.e. FPGA) or a software implementation (i.e. Linux(RTM) kernel) , its purpose will be applying the timestamp according to the algorithm described in section 2 Both implementations will be interchangeabie and they can be utilized on the same network segment since they modify the same data with the same algorithm.

Applying a cimestamp to a packet which carries a plain text protocol can effectively bind application level information and latency. Any network sniffer will be able to read the content of the packet (e.g. FIX protocol) and the several timestamps included in it.

For binary protocols (i.e. information streams like V?N with lzo compression) the network latency can still be caiculaced, but, since the application data may be not interpretable, the application latency cold be not easily attainable.

One requirement to ensure accuracy of the network latency measurement is precise clock synchronization across all inline devices which implement the functionality described in this invention.

PIP (defined in the IEEE 1588-2002) is the most suitable protocol for this scope because it has a nanosecond resolution and it's designed to The clock source may be synchronized via PIP on the same network use to receive packets, the device recognizes the PIP packet and updates its clock and it will forward any other packet with added timestamp.

Using this technique the device will require only two network interface cards.

The universal timestamp format is the POSIX time (also called Epoch) defined as the number of seconds that have elapsed since midnight Coordinated Universal Time WIg) , 1 January 1970, not counting leap seconds.

This value is represented by a 32-bit number, 4 byte, and it will wrap around to zero in the year 2036, but for latency purposes, only differences between timestamps are used.

Today the sub second resolution is fundamental, especially considering that network infrastructures use optical fibres and in one second a packet can cross several times any ocean/continent Microseconds and nanoseconds are the most common resolutions, where millisecond is inadequate, and the highest of them, nanoseconds requires 32-bit, 4 bytes.

A meaningful timestamp can be composed by 8 bytes, where 4 are used for Epoch time and 4 for nanoseconds, this number has to be multiplied per every device the flow will traverse.

This basic implementation adds a consistent amount of overhead, which it makes unsuitable for more than two measuring points (where the timestamp is added) The new approach discussed in this invention reduces the additional time information in a format which may fit in 32 bits, 4bytes, the length of an IP option header which will greatly simplify its embedding in the IP packet and its parsing.

-Section 2 -Detailed description

The IP header contains the Option field which, according to REC 791, must be implemented by any device, therefore every network component have to handle correctly the information present in this field.

This will ensure the timestamp will travel on any IP network (e.g. LAN, NAN, and VPN) without any impact on the infrastructure, often referred as transparent deploy' From RFC 791 The Option Length is the number of octets in the option counting the type, length, pointer, and overflow/flag octets (maximum length 40) The size of the option does not change due to adding addresses. The initial contents of the route data area must be zero.

If the timestamp data area is already full (the pointer exceeds the length) the datagram is forwarded without inserting the timestamp, but the overflow count is incremented by one.

If there is some room but not enough room for a full cimestamp to be inserted, or the overflow count itself overflows, the original datagram is considered to be in error and is discarded.

In either case an IO4P parameter problem message may be sent to the source host, this is situation must be avoided since it will add a degradation of service.

The following diagram illustrates the structure of the Option field: Fig 2 bytes 1 2 3 4 + + + + + 01O0O100 length pointeroflwfig + + + + + internet address + + + + + timestamp + + + + + Please note every line is composed by 4 bytes and the maximum number of lines allowed is 10 (40 bytes) The key of The strategy is the dictated by the Flag (flg) [4 bits] values implemented. These are o --time stamps only, stored in consecutive 32-bit words, 1 --each timestamp is preceded with internet address of the registering entity, 3 --the internet address fields are prespecified. An IP module only registers its timestamp if it matches its own address with the next specified internet address.

In order to be completely transparent and fully compliant to any network device, the invention described here will use the option 3 because it will force any device to not add its timesramp if it is not listed.

This approach will leave the TP option values to be used only by other similar inline devices and it will avoid the overflow which will cause a packet drop.

Artificially selecting an unused IF will avoid any interference with the ip Options this invention has specially crafted.

The last line contains an IF not in use in the target network; if it will be used on Internet, it is strongly recommended to choose a reserved unicast IF range (REC 3927, 1918, 5736, 5737, 2544) The preferred chosen time resolution will be microseconds, which is definitely a high resolution for every latency measurement related network performance, streaming, scientific and environmental analysis.

For specific ultra-low latency environments where nanoseconds are really needed (i.e. high frequency trading) this invention is not ideal because any inline device adds a small amount of latency, but it removes the need of tap devices and packet correlation.

The first (the closest device to the source of the traffic) inline device will assemble the IP option, which cannot change size to be compliant to the REC 791, and inject in the IP packet (the header checksum has to be recalculated) The device that will create the header should be aware of how many similar devices are deployed in the same network, this will ensjre the right length of the IP option minimizing the additional data.

This device can also allocate the maximum allowed space (40 bytes, padded with zero) without knowing how many similar devices are present in the network.

Every following device will simply overwrite the first zero padded IP option packet (4 bytes) with the compressed format described here (the size of the header cannot change) Fiq3 shows the I? option composition of the first inline device assuming it is only deployed another one similar device in the same network.

In order to provide the absolute Epoch time, the second line has an optional timestamp ruodulo as explained in section 3 Fiq3 + + + + + 0lO0Cl00 length pointeroflwfig + + + + + timestamp modulo + + + + + timestamp microseconds + + + + + C 0 0 0 + + + + + Reserved IP + + + + + ig4 shows the IF option composition of the second inline device (with optional timestamp modulo) Fig4 + + + H-+ OlOCClOO length pointeroflwflg + + + + + timestamp modulo + + + + + timestamp microseconds + + + + + timestamp microseconds -4--4-H-+ -4-Reserved IP + + + + + This algorithm can scale to include multiple devices and provide a fine granularity of the latency for every segment of the network delimited by two devices. This method can provide a complete picture of the network latency which will be more accurate than the end-to-end latency possible with just two devices.

This invention states that a modified Linux or BSD kernel car implement this algorithm, and if this kernel is the originator of the packet, data packets could leave the source with their generation time embedded in the IP protocol.

An example of an implementation with multiple devices is the following illustrated in the next page.

Fig5 shows the IF option composition of the first inline device which will reserve space for other two devices using the same algorithm (three inline devices in total in this example) Fig5 + -I--I-H--I-OlOOClOO length pointerIoflwflg + + + + + timestamp microseconds + + + + + 0 0 0 0 + + + + + o o 0 0 + + + + + Reserved IP I + + + + + ig6 shows IP option composition of the second inline device Eig6 + + + H-+ 0l00Cl00 length pointerotlwtig + + + + + timestamp microseconds + + + + + timestamp microseconds + + + + + 0 0 0 0 + + + + + Reserved IP + + + + + ig7 shows IP option composition of the third inline device Fig7 + -I--I--I--I- 0100010O length pointeroflwfig + + + + + timestamp microseconds + + + + + timestamp microseconds + + + + + timestamp microseconds + + + + + Reserved IP + + + + + The maximum number of inline devices is dictated by the maximum allowed bytes for the option field which is 40. This means 10 lines and, excluding the first and the fast, it derives it is possible include up no eight timestamps. This amount is sufficient to calculate the latency for 10 links which is more than adequate for a big network.

This invention can also be extended to cope with network which implements dynamic routing.

The previous examples assumed that the packet was going from point A to point B uraversing two or three inline devices in a specific order dictated by the network topology.

In a more complex network a packet could reach point B using different paths based on routing decisions. Due to the dynamic routing it is not possible to assume that different packets will travel on the same path and receive timestaraps by the same devices.

One line of the IF option can be dedicated to include the identifier of every device will apply its timestamp.

Eig8 shows IP option composition of the first inline device which its identifier is XX Fig S + + + + + O1OCC1OO length pointeroflwflg + + + + + XX I 0 0 0 0 I 0 0 0 + + + + + timestamn microseconds + + + + + C 0 0 0 + Reserved IP + + + + + FigO shows IF option composition of the second inline device (id YY) Fig 9 + + + + + 0lO0Cl00 length pointerofiwflg + + + + + XX IYY 0 0 0 IC 0 0 + + + + + timestamp microseconds + + + + + timestamp microseconds + + + + + Reserved IP + + + + + Piter the second device the packet will contain enough information to describe where the packet was at a specific point in rime.

The maximum number of device identifiers which this specific line can include is eight which is the maximum number of possible timestaraps can fit in the 12 option. Every ID is represented by 4 bits therefore they can span between 1 and 16. If it is necessary install more inline devices it is possible to extend to S bits which will make increase the device count to 256 but it may require two lines of options.

The IPv6 implementation is described as follow: From RFC 2460 -IPv6 Extension Headers In IFv6, optional internet-layer information is encoded in separate headers that may be placed between the IFv6 header and the upper-layer header in a packet. There are a small number of such extension headers, each identified by a distinct Next Header value.

hn IPv6 packet may carry zero, one, or more extension headers, each identified by the Next Header field of the preceding header, as illustrated in fig 10 Fig 10 + + I IPv6 header I TOP header -I-data I Next Header = I

I TOP I + + + + +

I IPv6 header I Routing header TOP header H-data I Next Header = I Next Header = I Routing I TOP + + + From RFC 2460, section 4.2 -Options Two of the currently-defined extension headers --the Hop-by-Hop Options header and the Destination Options header --carry a variable number of type-length-value (TLV) encoded "options", of the following format: Fig 11 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ I Option Type I Opt Data Len Option Data +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Option Type 8-bit identifier of the type of option.

Opt Data Len 8-bit unsigned integer. Length of the Option

Data field of this option, in octets.

Option Data Variable-length field. Option-Type-specific data.

The Option Type identifiers are internally encoded such that their highest-order two bits specify the action that must be taken if the processing IPv6 node does not recognize the Option Type: 00 -skip over this option and continue processing the header.

01 -discard the packet.

-discard the packet and, if Destination Address was a multicast address, send an IOMP Parameter Problem, Oode 2.

11 -discard the packet and send an ICMP Parameter Problem, Code 2.

The option 00 ensures the timestamp will be Ignored by any device not timestamp aware therefore avoiding any artificial mechanism designed to guarantee the integrity of the packet. the compressed time data will be placed according to the algorithm described in section 3.

The composition of this payload is designed to satisfy the "Appendix B. Formatting Guidelines for Options" E'igl2 shows the proposed TPv6 option composition: Fig 12 --+ Next hdrhdr ten option I lenght + + + + + timestamp microseconds + + + + + The option field could be 00011111 (the two highest-order bits are 00 as reguired, the third highest indicates if it may change en-route) -Section 3 -compression algorithm The algorithm is based on modular arithmetic, every epoch time can be expressed a sum of a division by 600 and the modulo 600 of the same value.

600 seconds ecuals lOminutes which is more than the upper limit for any latency in modern telecommunications.

For latency purposes it is safe to assume a packet will never take more than a 10 minutes to reach its destination due to the IP loop detection (the time to live -ITt-expires for every hop traversed and prevents routing loops) and due to the robustness of connection oriented protocols like TOP which it will detect a missed packet in few seconds and it will retransmit.

Even if the data could reach its destination in 10 minutes, it will be very likely discarded by the application since it is not more relevant.

Every epoch time is unique and it can be represented as a couple of values: Epoch divided by 600 and Epoch modulo 00.

The mathematical representation is the following: (Epoch time) /600 + (Epoch time) modulo 600 Latency is calculated as a difference between two observation tines therefore only the rnodulo may change value assuming its journey will not last more than 10 minutes.

Modulo may be the same if the packet is seen by inline devices during the same Epoch time, but the sub second value, expressed in microsecond, should change value.

If both values are the same on two different inline devices there must be a clock discrepancy or the packet took less than a microsecond to travel that network segment.

The first element (the division component) could be included in the timestamp information to be carried in the IP packet if the absolute time reference is desired (e.g. packet analysis may need it to reference events) The algorithm is exemolified by the following: Given a random date like Wed Oct 17 11:43:47 UTO 2012 which expressed in epoch is 1350474227 This epoch is 2251088 lOminutes slots 1350474227/600 = 2250790 1350474227 modulo 00 is 227 In order to validate the formula, multiplying 2250790 by 600 and adding 227 gives 1350474227.

The maximum value for any number modulo 600 is 599, which is in binary 1001010111, 11 bit.

The microsecond resolution requires 21 bit since the maximum value 999999 (11110100001000111111) If the binary representation is smaller than respectively 11 and 21 bit, a zero padding will be required. For example 423 is in binary 110100111 which are 9 digit so it will be padded with two zeros to occupy 11 bits: 00110100111.

The concatenation of these two binary data makes a string long exactly 32 bit, which it will fit in one line of the IP option which is illustrated in fig 13 Fig 13 bytes 1 2 3 4 + + + + + 01001010 I 11111110 10000100 I 00111111 + + + + + In some embodiments off the invention, the absolute time reference (in this example the number 2250790 which represents the lOminutes slots could be included in the timestamp once as described in figure 3-4 kt the destination, it will possible calculate the neuwork latency simply make differences between timestarnps without using any complex algorithm or processing extra management traffic.

This invention can contribute to determine the cause of anomalies in IF Packet Delay Variation (ipdv, described RE'S 3393) by simply observing the timestamp added in the ip Option.

ipdv represents a key factor for voice/video communications on network packets and all codecs which compress data are sensitive to delay variations.

Claims (3)

1. Claims: 1) A method of placing compressed timestamp information in the IP packet to allow network latency calculation.
2. 2) The method of claim 1, wherein uses a specific timestamp format to compress the information to greatly reduce the overhead introduced.
3. 3) An apparatus comprising: means for receiving an IP data packet, means for keeping apparatus clock synchronized with ocher similar devices present on the same network, means for edit the TP header to add a compressed timestamp, means for transmitting the received packet 4) An apparatus as set forth in claim 3 wherein said the 12 packet receives timestamps at multiple nodes, with the timesramps reflecting arrival times at those nodes.6) The article of manufacture of claim 3, wherein the apparatus can be a hardware implementation!or a software implementation.7) The method of claim 1, implemented by the apparatus described on claim3, wherein can manipulate 12v4 and IPv6 packets 8) The time resolution and format of claim 1 is adjustable to suit different latency measurement reguirements 9) A method of placing location information in the IP packet related to where the compressed timestamp was applied