AU2014101296B4 - Apparatus and Method for Improved Physical Layer Switching - Google Patents

Abstract

A method for physical layer data switching. Data received at one input port of a switch is circuit switched to at least one output port of the switch, resulting in low latency and flexibility of protocols. The data is also copied to a higher layer process, providing higher layer functionality without adding any latency overhead to data throughput.

Description

Technical Field [0001] The present invention relates to network data switching, and in particular to a physical layer switch having improved functionality.

Background of the Invention [0002] It is often desirable to take data packets from one source and to send the data packets to one or more of a number of possible destinations. This is the case in the downstream direction from a stock exchange, for example, where it is desirable for a number of machines such as stock traders’ computers to potentially receive packets from the exchange.

[0003] Addressable switching is a layer 2 or above function which can be implemented in an integrated circuit or combination of multiple integrated circuits by processing downstream packets using some mechanism (e g. such as a content addressable memory in a traditional packet switch). Conventionally this involves receiving data on a single transceiver, checking the packet address, referring to a lookup table to identify a port or interface associated with the address, and then directing the packet to only that port. This approach is based on the principle of optimising bandwidth usage, security and privacy, but requires relatively complex circuitry within the switch.

[0004] Such network switches require that the layer 1 (physical layer) data be converted to layer 2 (the data link layer) for address checking, before being returned to layer 1. Encapsulation in an Ethernet frame involves grouping a payload of data bits together with addressing information and error checking information, and encoding the packet using a scheme such as 8B/10B, and this process must be reversed by a layer 2 switch in order to check the addressing and error checking information. Converting data from layer 1 to layer 2 requires data deserialization into a parallel stream because it is difficult to implement layer 2 features in a serial stream at the frequencies involved. This means that any layer 2 processing is much slower than layer 1, and layer 2 address checking introduces a significant delay of approximately 200-400 ns for even the lowest latency layer 2 network switches.

[0005] In contrast, physical layer switches provide low latency connectivity because they work at the lowest level of the OSI network stack. The physical layer is concerned with bit-level communication and employs circuit switching (for example, via a crossbar or matrix switch that can be configured to create circuits between an input and one or more outputs) as opposed to packet switching, and therefore does not need to interpret the high level encapsulation, including the addressing information, that is specific to higher layer protocols such as Ethernet. Because physical layer circuit switching does not need to assess higher level encapsulation, physical layer switches can generally pass data of any protocol because the circuit switching approach does not require specific protocol interpretation logic. Physical layer switches can also reproduce or regenerate bits (including techniques such as clock and data recovery, equalisation and preemphasis) with very little overhead and without any buffering.

[0006] Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is solely for the purpose of providing a context for the present invention. It is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present invention as it existed before the priority date of each claim of this application.

[0007] Throughout this specification the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.

[0008] In this specification, a statement that an element may be “at least one of’ a list of options is to be understood that the element may be any one of the listed options, or may be any combination of two or more of the listed options.

Summary of the Invention [0009] According to a first aspect the present invention provides a method of physical layer data switching, the method comprising; receiving data at at least one input port of a switch; configurably circuit switching the data within the switch to at least one output port of the switch; and copying the data to a device within the switch for performance of a higher layer process on the data.

[0010] According to a second aspect the present invention provides a switch comprising: a plurality of data inputs which can each be configurably circuit switched via a cross point switch to one or more of a plurality of data outputs, a first data output of the plurality of data outputs receiving data which has been circuit switched from a first input of the plurality of data inputs; and higher layer function circuitry in communication with the cross point switch, the cross point switch being configurable to copy the data to the higher layer function circuitry.

[0011] According to a third aspect the present invention provides a non-transitory computer readable medium for configuration of a data switch, comprising instructions which, when executed by one or more processors, causes performance of the following: circuit switching of data received at at least one input port of a switch to at least one output port of the switch; and copying of the data to a higher layer process.

[0012] According to a fourth aspect the present invention provides a method of physical layer data switching, the method comprising; receiving data at at least one input port of a switch; circuit switching the data to at least one output port of the switch; and a higher layer process modifying or supplementing the data and producing a second data output at a second output port of the switch.

[0013] The present invention thus recognises that physical layer processing results in low latency and flexibility of protocols, but that physical layer switches do not provide useful counters and statistics which can only be extracted at higher layers, and that physical layer switches thus present a disadvantage to network administrators because for example it is not possible to use such a switch to monitor a data link and diagnose problems.

[0014] In contrast to prior physical layer switches, embodiments of the present invention instead provide a physical layer switch that retains the benefits of low latency and protocol flexibility, while also providing higher layer functionality, all without adding any latency overhead to data throughput. In particular, by circuit switching data direct from an input to an output the physical layer switch operates at a low latency level. Moreover, by copying the same data to a higher layer process, higher level monitoring functions can be implemented without affecting the latency of the physical layer switching because such functions are performed in parallel with the switching. For example in some embodiments the higher layer process may comprise identifying IP packets and Ethernet frames, which are common protocols used on networks. Other protocols may include Fibre Channel, InfiniBand™ and SDI. This parallelism means that even though the higher level monitoring counters and statistics take additional time to process, this does not delay the data as it passes through the switch. In some embodiments, the outputs of the higher layer process(es) such as counters and statistics are accessed via a management interface of the switch and do not impede the low latency of the data path.

[0015] Reference herein to a higher layer process is to be understood as referring to a process which is not carried out at the physical layer. In some embodiments the higher layer process may comprise conversion of physical layer data for higher level processing, including detecting physical layer idle patterns and physical layer errors, for example 8B/10B code violations.

[0016] In some embodiments of the invention, data received at a plurality of data inputs, and preferably all data inputs, is copied to one or more respective higher layer process. Such embodiments may thus effect layer 1 switching of each or all of the data inputs while simultaneously providing a higher layer process or processes such as packet identification for each interface, without impacting latency.

[0017] In some embodiments of the invention, the higher layer function may comprise a counter of data packet statistics for the respective interface.

[0018] In some embodiments of the invention, the higher layer function may additionally or alternatively comprise a counter of a number of octets received and transmitted.

[0019] In some embodiments of the invention, the higher layer function may additionally or alternatively comprise a counter of a number of packets, by type (unicast, multicast and broadcast).

[0020] In some embodiments of the invention, the higher layer function may additionally or alternatively comprise a counter of a number of packets, by protocol (Ethernet, IP, UDP, TCP, HTTP, SSH, SNMP, ARP, etc).

[0021] In some embodiments of the invention, the higher layer function may additionally or alternatively comprise a counter of the number of packets, by size. For example a count of the number of 64 byte, 65-127 byte, 256-511 byte, etc, data packets; [0022] In some embodiments of the invention, the higher layer function may additionally or alternatively comprise a counter of the number of packet errors due to invalid frame check sequence (FCS), packet ‘runts’, packet ‘giants’ and/or alignment errors or the like; [0023] In some embodiments of the invention, the higher layer function may additionally or alternatively comprise a counter of the number of dropped frames.

[0024] In some embodiments of the invention, the higher layer function may additionally or alternatively comprise VLAN and/or MACsec counters.

[0025] In some embodiments of the invention, the higher layer function may additionally or alternatively comprise detecting a link idle state.

[0026] In some embodiments of the invention, the higher layer function may additionally or alternatively comprise the ability to inject an idle pattern, or cause an idle pattern to be transmitted onto the wire.

[0027] In some embodiments of the invention, the higher layer function may additionally or alternatively comprise a counter for physical layer errors, for example 8B/10B code errors from which it is possible to calculate an estimate for the physical layer bit error rate.

[0028] In some embodiments of the invention, the higher layer function may additionally or alternatively comprise packet inspection to provide heuristics on the switching device, for example identifying the hosts and/or devices present at the other end of a or each connection by inspecting packet MAC addresses.

[0029] In some embodiments of the invention, the higher layer function may be applied in respect of data captured on either or both of the receive and transmit interfaces.

[0030] The higher layer process output, such as a statistic counter value, in some embodiments can be reported by a management module of the switch. In some such embodiments the management module functionality can include a command line, a web interface and/or other mechanisms such as SNMP that allow a separate computer to query or receive published counter values.

[0031] In some embodiments, the higher layer function may additionally or alternatively comprise the ability to timestamp a packet. Such embodiments recognise that timestamping in a switch is useful for measuring network latency, and for accurate records that can be used for post examination, testing/modelling/replaying historical network traffic. In embodiments handling Ethernet packets, and noting that such packets do not have a field for accurate timestamping information, the timestamp information may be added to the packet by increasing the size of the packet (prepend or append the timestamp data).

[0032] Alternatively, timestamp data may be added to each packet by replacing a part of the packet that is used for other information. Such embodiments have the benefit that it is possible to timestamp the packets on a fully saturated link, i.e a link which has no more space on the channel for larger packets. One approach that does not impact the size of a packet is to replace the frame check sequence (FCS) field in the Ethernet frame with a 32-bit timestamp value. This might be done only after the FCS is checked and found to be error-free. If the FCS is found to be in error then a special timestamp value can be used in the affected packet. The special timestamp will be one that never occurs for a valid packet. The 32-bit timestamp might represent the number of nanoseconds from a known reference instant in time that can be interpreted as an absolute unit of time by a separate process that also knows the reference instant. For example, the 32-bit timestamp may represent the least significant 32 bits of the absolute time in nanoseconds. In another embodiment the Ethernet frame preamble may be partially replaced with a timestamp value.

[0033] Embodiments providing time stamping of the copy of the data in the data stream thus provide a layer 2 function in the same device as layer 1 switching. To effect timestamping the device can detect the start of packets on its input data interfaces and generate a timestamp which can be added to the packet as metadata, stored for later processing, sent to another device for analysis and/or added to the packet in place of the frame check sequence (FCS) or the interframe gap (IFG).

[0034] In embodiments in which the higher layer function comprises timestamping packets, the higher layer device may in some embodiments output a second data stream comprising a delayed timestamped copy of the original data stream. Such embodiments thus deliver the data to the output at low latency, while also producing a delayed copy of the same data with added timestamp information which might for example be connected to a logging device/computer for separate analysis by an end user.

[0035] In embodiments in which the higher layer function comprises timestamping packets, the timestamp value may be obtained by sampling a counter register at the moment the start of packet is detected on the receive interface of the switch. In other embodiments timestamping may be performed on the output interface of the switch. The counter register is preferably updated regularly (e g. every nanosecond) so that it is a precise representation of the current time in nanoseconds. The counter may be synchronised to an external time reference external to the switch via a pulse per second (PPS) signal, precision time protocol (PTP) or the like.

Alternatively, the timestamp value may be taken at some fixed time offset after the start of packet, and the actual time that the start of packet was received may then be obtained by subtracting the fixed time offset from the timestamp value. In such embodiments this subtraction can be performed either before the timestamp is written to the FCS field or alternatively it can be performed when post-processing the timestamped data stream.

[0036] In some embodiments, the higher layer function may additionally or alternatively comprise data packet capture, whereby some or all of the duplicate data stream is processed via a media access controller (MAC) and then delivered to a data packet capture RAM, disk and/or PCIe link. In such embodiments, the management platform may be requested to capture a packet that is passing through an interface, such that the packet will be transmitting at line rate without delay, while it is concurrently copied and reported to the end user via a software / management port. In some embodiments, the higher layer function may capture data without MAC processing. In an embodiment, the higher layer function may aggregate the captured data from more than one port.

[0037] The data may be downstream data originating from one or more servers and intended for one or more client devices. Additionally or alternatively, the data may be upstream data originating from one or more client devices and intended for one or more servers. Additionally or alternatively, the data may be peer to peer data such as data being switched between nodes of a high performance computing cluster, or between nodes of a data centre network.

[0038] In some embodiments, the higher layer function may additionally or alternatively comprise deep packet inspection in the case of a firewall, intrusion detection, or extrusion detection.

[0039] In some embodiments, the higher layer function may additionally or alternatively comprise the ability to inject a packet into an interface MAC (media access controller), with the output of the MAC being passed to the physical layer. That is, in such embodiments user software may communicate a packet to the switch via a management interface, and the higher layer function may involve injecting the packet into either the receive or transmit MAC for a specified interface. The packet will then be transmitted on the wire from an interface. The packet may alternatively be created by software or hardware (e g. FPGA) on the device, i.e. originating on the device instead of being communicated over the management interface. Some such embodiments may inject a packet or packets in order to subscribe to multicast on behalf of downstream hosts, for example, in accordance with the Internet Group Management Protocol

Version 2 (IGMP) the subject of the Internet Society RFC2236, and/or in accordance with a protocol-independent multicast (PIM) routing protocol, and/or in accordance with a border gateway protocol (BGP), or the like. In the case of IGMP, subscribing or unsubscribing to a multicast group may comprise injecting IGMP packets for transmission to a neighbouring multicast router to register in a multicast group to receive multicast packets on behalf of downstream clients connected to the ports of the switch of the present invention, thereby obviating the need to have a downstream host perform this function.

[0040] Data ports of the switch may receive data from any suitable cable including for example copper cables using 10GBASE-T interface type or 1000BASE-T interface type, 100BASE-TX copper, optic fibres using 10GBASE-R/1000BASE-X, direct-attach copper. The switch may receive data from any suitable pluggable module, such as SFP, SFP+, QSFP, QSFP+, CFP. Those modules might allow a connection via direct-attach copper, lGBase-SX, lOGBase-SR, lOGBase-LR, DWDM, or other connectivity standards. Alternatively the switch might be connected directly to another device using a connector or printed circuit board.

[0041] In some embodiments of the invention, the higher layer function to which the data is copied may be adaptively selected by switching the copied data stream to one of a plurality of outputs of the switch, each of the plurality of outputs being connected to a respective higher layer function. Such embodiments permit dynamic selection of which higher layer function or functions is or are applied to the copied data stream. In such embodiments, the output of the higher layer function may be returned to the crosspoint switch to permit dynamic and configurable switching of the output of the higher layer function to any chosen output of the crosspoint.

[0042] In some embodiments of the invention, the higher layer function may be effected by a signal conditioning block positioned at an input and/or output of the crosspoint switch, to which the data is copied.

Brief Description of the Drawings [0043] An example of the invention will now be described with reference to the accompanying drawings, in which:

Figure 1 is a system schematic of the hardware used to implement one embodiment of the present invention;

Figure 2 is a system schematic of the hardware used to implement another embodiment of the present invention;

Figure 3 is a system schematic of the hardware used to implement yet another embodiment of the present invention;

Figure 4 is a system schematic of the hardware used to implement still another embodiment of the present invention;

Figure 5 is a system schematic of the hardware used to implement a further embodiment of the present invention; and

Figure 6 is a schematic of a signal conditioning block in accordance with the embodiment of Figure 5.

Description of the Preferred Embodiments [0044] Figure lisa system schematic of the hardware used to implement one embodiment of the present invention. The switch 100 comprises a number of front panel ports indicated generally at 102 which can be connected to other devices, and a crosspoint switch 110 which can remap, as well as multicast data from one port to multiple ports.

[0045] In the configuration shown in Figure 1, which can be reconfigured at other times, input data is received from a data source 120 and is circuit switched by crosspoint 110 to a respective data output 104 associated with the intended destination 130. In this sense, switch 100 performs as a physical layer switch. It is noted that outputs 104, while being shown separate to inputs 102, in a typical rack mounted device may be co-located on a front panel of the device with inputs 102. The inputs 102 and outputs 104 may be effected by a plurality of transceivers, each transceiver effecting one data input 102 and one data output 104. Currently active switching paths within crosspoint 110 are indicated in Figure 1 by firmer lines simply for illustrative purposes.

[0046] In accordance with the present invention, the data received from data source 120 is not only delivered to destination 130, but is also copied by the crosspoint switch 110 to a higher layer process which is performed by device 106 within switch 100.

[0047] In this embodiment, device 106 comprises a timestamping function so that a second stream of data is delivered from device 106 to destination 130. Destination 130 thus receives the original data via an extremely low latency path, while also receiving a timestamped copy of the data albeit with greater latency, due to the additional latency introduced by device 106.

[0048] Figure 2 is a system schematic of the hardware used to implement one embodiment of the present invention. The switch 200 comprises a number of front panel ports indicated generally at 202 which can be connected to other devices, and a crosspoint switch 210 which can remap, as well as multicast data from one port to multiple ports.

[0049] In the configuration shown in Figure 2, which can be reconfigured at other times, input data is received from a data source 220 and is circuit switched by crosspoint 210 to a respective data output 204 associated with the intended destination 230. In this sense switch 200 performs as a physical layer switch. It is noted that outputs 204, while being shown separate to inputs 202, in a typical rack mounted device may be co-located on a front panel of the device with inputs 202. The inputs 202 and outputs 204 may be effected by a plurality of transceivers, each transceiver effecting one data input 202 and one data output 204. Currently active switching paths within crosspoint 210 are indicated in Figure 2 by firmer lines simply for illustrative purposes.

[0050] In accordance with the present invention, the data received from data source 220 is not only delivered to destination 230, but a higher layer process which is performed by device 206 within switch 100 and produces packets which are also switched to output 230.

[0051] In this embodiment, device 206 comprises a multicasting subscribing function so that IGMP packets are injected into the data stream sent to a neighbouring router 230 in order for devices 220 and/or 222 to subscribe or unsubscribe to a multicast group. Destination 230 thus receives data from device 220 via an extremely low latency path, while also receiving higher layer function data such as IGMP data packets injected into either the transmit MAC for the specified interface, albeit with greater latency, due to the additional latency introduced by device 206.

[0052] Figure 3 is a system schematic of the hardware used to implement yet another embodiment of the present invention. The switch 300 comprises a number of front panel ports indicated generally at 302 which can be connected to other devices, and a crosspoint switch 310 which can remap, as well as multicast data from one port to multiple ports.

[0053] In the configuration shown in Figure 3, which can be reconfigured at other times, input data is received from a data source 320 and is circuit switched by crosspoint 310 to a respective data output 304 associated with the intended destination 330. In this sense switch 300 performs as a physical layer switch. It is noted that outputs 304, while being shown separate to inputs 302, in a typical rack mounted device may be co-located on a front panel of the device with inputs 302. The inputs 302 and outputs 304 may be effected by a plurality of transceivers, each transceiver effecting one data input 302 and one data output 304. Currently active switching paths within crosspoint 310 are indicated in Figure 3 by firmer lines simply for illustrative purposes.

[0054] In accordance with the present invention, the data received from data source 320 is not only delivered to destination 330, but is also copied by the crosspoint switch 310 to a higher layer process which is performed by device 306 within switch 300.

[0055] In this embodiment, device 306 comprises a link controller configured to provide data link statistics to an administrator 332. In this embodiment device 306 provides a plurality of higher layer functions including detecting physical layer idle patterns and physical layer errors, counting data packets by type, protocol and size, counting data packet errors by error type including invalid FCS, packet size or alignment, and packet MAC address inspection to identify connected devices.

[0056] Destination 330 thus receives the original data via an extremely low latency path, while device 332 receives data link statistics albeit with greater latency, due to the additional latency introduced by device 306.

[0057] Figure 4 is a system schematic of the hardware used to implement yet another embodiment of the present invention. The switch 400 comprises a number of front panel ports indicated generally at 402 which can be connected to other devices, and a crosspoint switch 410 which can remap, as well as multicast data from one port to multiple ports.

[0058] In the configuration shown in Figure 4, which can be reconfigured at other times, input data is received from a data source 420 and is circuit switched by crosspoint 410 to a respective data output 404 associated with the intended destination 430. In this sense switch 400 performs as a physical layer switch. It is noted that outputs 404, while being shown separate to inputs 402, in a typical rack mounted device may be co-located on a front panel of the device with inputs 402. The inputs 402 and outputs 404 may be effected by a plurality of transceivers, each transceiver effecting one data input 402 and one data output 404. Currently active switching paths within crosspoint 410 are indicated in Figure 4 by firmer lines simply for illustrative purposes.

[0059] In accordance with the present invention, the data received from data source 420 is not only delivered to destination 430, but is also copied by the crosspoint switch 410 to a higher layer process which is performed by device 406 within switch 400. Device 406 in this embodiment has a plurality of inputs each associated with a selected high layer function, so that a desired function can be selected simply by switching the copied data to the respective input of device 406. Moreover, the output of device 406 is returned to crosspoint 410 and thereby may be selectively switched to any output port 404, and at the time shown in Figure 4 the device 400 is configured to switch the output of higher layer device 406 to destination 430, although at other times the output may be switched to other destinations. A management system processor 440 receives a pulse per second (PPS) input to enable timestamping and also communicates via a management interface to permit external management. Management system 440 further controls and communicates with crosspoint 410 and with higher layer device 406.

[0060] Figure 5 is a system schematic of the hardware used to implement yet another embodiment of the present invention. The switch 500 comprises a number of front panel ports indicated generally at 502 which can be connected to other devices, and a crosspoint switch 510 which can remap, as well as multicast data from one port to multiple ports.

[0061] In the configuration shown in Figure 5, which can be reconfigured at other times, input data is received from a data source 520 and is circuit switched by crosspoint 510 to a respective data output 504 associated with the intended destination 530. In this sense switch 500 performs as a physical layer switch. It is noted that outputs 504, while being shown separate to inputs 502, in a typical rack mounted device may be co-located on a front panel of the device with inputs 502. The inputs 502 and outputs 504 may be effected by a plurality of transceivers, each transceiver effecting one data input 502 and one data output 504. Currently active switching paths within crosspoint 510 are indicated in Figure 5 by firmer lines simply for illustrative purposes.

[0062] In accordance with the present invention, the data received from data source 520 is not only delivered to destination 530, but is also copied by the crosspoint switch 510 to a higher layer process which is performed by device 564 within switch 500. Device 564 in this embodiment is a signal conditioning block of the type shown in Figure 6. The output of higher layer device 564 is passed to destination 430, although at other times the output may be switched to other destinations. A management system processor 540 receives a pulse per second (PPS) input to enable timestamping and also communicates via a management interface to permit external management. Management system 540 further controls and communicates with crosspoint 510 and controls all of the signal conditioning blocks 550...554 and 560...566.

[0063] In the signal conditioning block 564 shown in Figure 6, input data is passed through a EQ block 602 which is configurable via the management interface 540 as indicated by double-ended arrows throughout Figure 6. EQ block 602 is an equaliser that performs filtering on the received signal to compensate for the effects of passing through the communications channel (cable), thus reducing the likelihood of errors. EQ block 602 can adjust the frequency components of the signal and may also remove multi-path distortion, the output of which is much closer to the ideal representation of the data than the unequalised received signal. From EQ 602 data is passed to clock data recovery (CDR) block 604 which is configurable via the management interface 540. From CDR 604 data is passed both to a higher layer function 606 and to a multiplexer 608, both controlled by management system processor 540. The output of the higher layer function block 606 is also passed to multiplexer 608. From multiplexer 608, multiplexed low latency data and higher layer function output data are passed to a pre-emphasis block 610, which modifies the frequency components of the transmitter signal to compensate, in advance, for the expected effects of the communication channel (cable), resulting in the destination receiving a signal which is closer to an ideal representation of the data thus reducing the likelihood of errors. From pre-emphasis block 610 multiplexed low latency data and higher layer function output data are output by signal conditioning block 564. Any or all of signal conditioning blocks 550... 554 and 560... 566 in the embodiment of Figure 5 may be configured as shown in Figure 6.

[0064] It will be appreciated by persons skilled in the art that numerous variations and/or modifications may be made to the invention as shown in the specific embodiments without departing from the spirit or scope of the invention as broadly described. The present embodiments are, therefore, to be considered in all respects as illustrative and not limiting or restrictive.

Claims (5)

1. A method of physical layer data switching, the method comprising; receiving data at at least one input port of a switch; configurably circuit switching the data within the switch to at least one output port of the switch; and copying the data to a device within the switch for performance of a higher layer process on the data.

2. A switch comprising: a plurality of data inputs which can each be configurably circuit switched via a cross point switch to one or more of a plurality of data outputs, a first data output of the plurality of data outputs receiving data which has been circuit switched from a first input of the plurality of data inputs; and higher layer function circuitry in communication with the cross point switch, the cross point switch being configurable to copy the data to the higher layer function circuitry.

3. The switch of claim 2, wherein the higher layer function circuitry is configured to perform packet timestamping.

4. The switch of claim 2 or claim 3, wherein the higher layer function circuitry is configured to perform packet capture.

5. The switch of claim 2 or claim 3, wherein the higher layer function circuitry is configured to perform a count of data packet statistics without affecting latency at the first data output.