AU2021203290B2 - Lane synchronisation - Google Patents
Lane synchronisation Download PDFInfo
- Publication number
- AU2021203290B2 AU2021203290B2 AU2021203290A AU2021203290A AU2021203290B2 AU 2021203290 B2 AU2021203290 B2 AU 2021203290B2 AU 2021203290 A AU2021203290 A AU 2021203290A AU 2021203290 A AU2021203290 A AU 2021203290A AU 2021203290 B2 AU2021203290 B2 AU 2021203290B2
- Authority
- AU
- Australia
- Prior art keywords
- lane
- packet
- lanes
- link
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 claims abstract description 48
- 238000012549 training Methods 0.000 claims abstract description 25
- 238000012544 monitoring process Methods 0.000 claims abstract description 15
- 230000008859 change Effects 0.000 claims description 11
- 238000004891 communication Methods 0.000 claims description 10
- 238000010079 rubber tapping Methods 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 2
- AWNBSWDIOCXWJW-WTOYTKOKSA-N (2r)-n-[(2s)-1-[[(2s)-1-(2-aminoethylamino)-1-oxopropan-2-yl]amino]-3-naphthalen-2-yl-1-oxopropan-2-yl]-n'-hydroxy-2-(2-methylpropyl)butanediamide Chemical compound C1=CC=CC2=CC(C[C@H](NC(=O)[C@@H](CC(=O)NO)CC(C)C)C(=O)N[C@@H](C)C(=O)NCCN)=CC=C21 AWNBSWDIOCXWJW-WTOYTKOKSA-N 0.000 claims 1
- 102000011202 Member 2 Subfamily B ATP Binding Cassette Transporter Human genes 0.000 claims 1
- 230000005540 biological transmission Effects 0.000 abstract description 67
- 230000008569 process Effects 0.000 description 8
- 230000008901 benefit Effects 0.000 description 4
- 239000004744 fabric Substances 0.000 description 4
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000010200 validation analysis Methods 0.000 description 2
- GNFTZDOKVXKIBK-UHFFFAOYSA-N 3-(2-methoxyethoxy)benzohydrazide Chemical compound COCCOC1=CC=CC(C(=O)NN)=C1 GNFTZDOKVXKIBK-UHFFFAOYSA-N 0.000 description 1
- FGUUSXIOTUKUDN-IBGZPJMESA-N C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 Chemical compound C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 FGUUSXIOTUKUDN-IBGZPJMESA-N 0.000 description 1
- YTAHJIFKAKIKAV-XNMGPUDCSA-N [(1R)-3-morpholin-4-yl-1-phenylpropyl] N-[(3S)-2-oxo-5-phenyl-1,3-dihydro-1,4-benzodiazepin-3-yl]carbamate Chemical compound O=C1[C@H](N=C(C2=C(N1)C=CC=C2)C1=CC=CC=C1)NC(O[C@H](CCN1CCOCC1)C1=CC=CC=C1)=O YTAHJIFKAKIKAV-XNMGPUDCSA-N 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/06—Generation of reports
- H04L43/062—Generation of reports related to network traffic
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/38—Information transfer, e.g. on bus
- G06F13/382—Information transfer, e.g. on bus using universal interface adapter
- G06F13/385—Information transfer, e.g. on bus using universal interface adapter for adaptation of a particular data processing system to different peripheral devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2213/00—Indexing scheme relating to interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F2213/38—Universal adapter
- G06F2213/3808—Network interface controller
Abstract
The invention relates to a method of monitoring a plurality of transmission
lanes in a data transmission system, characterised by the step of extracting
individual lane parameters in the data transmission system independently of a
5 transmitted training sequence. The invention also relates to a computer
network device configured to monitor a plurality of transmission lanes in a
data transmission system, the computer network device configured in use to
receive a packet transmitted over the plurality of transmission lanes and to
use the data symbols contained in the packet to extract individual lane
10 parameters in the data transmission system independently of a transmitted
training sequence.
Description
The present invention relates to a method of monitoring transmissions in a data transmission system, and in particular to monitoring multilane data transmission between nodes in a computing network.
System Area Networks (SANs) have been developed in which one or more processor nodes are connected to one or more I/O nodes through a fabric of switches and routers.
InfiniBandTM (1B) is an example of a system architecture providing a common I/O specification in a channel based, switched-fabric technology.
High density data links between the various processor nodes, switches, routers and I/O nodes in an 1B SAN typically involve parallel transmission of data over a plurality of serial connections ("parallel - serial" links). Each link can include a number of physical lanes (e.g., copper cable or fibre optic cable). Typical multilane 1B links can have 4, 8 or 12 physical lanes per link (denoted 4x, 8x and 12x respectively), although in principle a link may have any number of lanes.
The standard serial signalling rate is 2.5 Gbits/s (Single Data Rate, SDR) in each direction per connection. IB currently also supports double (DDR) and quad (QDR) data rates. Data in an IB system uses the industry standard 8B/10B encoding (i.e., every 10 bits carries 8 bits of data). Therefore, a 4x QDR link can carry 32 Gbits/s of useful data.
Each 10 bit entity is known as a symbol or character. Data symbols are commonly denoted Dx.y, where x ranges from 0 to 31 and y from 0 to 7. The 8B/1 OB standard also defines a number of control symbols, denoted by Kx.y, that can be sent in place of data symbols.
A data or link packet sent over a link is an ordered sequence of control symbols and data symbols. For example, a data packet always begins with a Start of Data Packet symbol (SDP (K27.7)) and ends with an End of Data Packet symbol (EGP (K29.7) or EBP (K30.7) depending on whether or not the packet loses integrity during transmission through the fabric). A similar system is used with link packets using SLP (K28.2) and EGP/EBP control symbols. Other control and data symbols are inserted between the SDP and EGP (or EBP) symbols, the data being byte striped across the lanes, usually sequentially.
In order to establish (or re-establish) a link between nodes over a multi-lane link it is necessary to establish the corresponding lane order and polarity of each of the lanes of the multi-lane ports at both the transmitting and receiving nodes. This is typically achieved by a process of link training. During link training an ordered set of symbols, known as a Training Sequence (TS1 or TS2) is transmitted over the link.
TS1 and TS2 contain sufficient information to enable a port receiving it to establish both the correct lane order and polarity to match the lane order and polarity of the transmitting port for each of the lanes in the link. A two way link is established (link "up") when a port's receiver has been correctly configured (using TS1 and TS2) and the port is both receiving and transmitting Idle symbols.
A method of lane alignment in a multi-lane transmission system utilising training sequences, together with a detailed description of the InfiniBandTM sytem, is disclosed in U. S. Pat. No. 6,985,502 (to Bunton), the disclosure of which is incorporated here by way of reference.
Through the exchange of training sequences, such as TS1 and TS2, a link can be established for a node within an IB SAN. However, there can be occasions when it is important to monitor the flow of information through the SAN without establishing a new link. For example, an operator of a SAN may wish to monitor the performance of the SAN without disturbing the network by adding new nodes and links.
Network monitoring systems, such as EndaceTM's DAGTM-based system, use a passive tap to read the packets as they are transmitted over a link. While this is relatively straightforward in a single lane link, the information in a packet transmitted over a multi-lane IB link is meaningless without knowledge of the lane alignment and polarity of each lane.
One method of establishing the correct lane alignment and polarities is to wait until a training sequence is transmitted over the link. The training sequence can be readily recognised and used to set up the appropriate lane order and polarities.
However, this is not practical in general as transmission of training sequences typically only occurs when a link is being established or re-established, which can be an infrequent occurance. Monitoring of high performance computing networks, such as those using multi-lane 1B technology, requires that access to the information flowing along the link be as rapid as possible. In such cases it would be an advantage to have a method of lane alignment which could be applied at any time.
It is an object of the present invention to address the foregoing problems or at least to provide the public with a useful choice.
All references, including any patents or patent applications cited in this specification are hereby incorporated by reference. No admission is made that any reference constitutes prior art. The discussion of the references states what their authors assert, and the applicants reserve the right to challenge the accuracy and pertinency of the cited documents. It will be clearly understood that, although a number of prior art publications are referred to herein, this reference does not constitute an admission that any of these documents form part of the common general knowledge in the art, in New Zealand or in any other country.
It is acknowledged that the term 'comprise' may, under varying jurisdictions, be attributed with either an exclusive or an inclusive meaning. For the purpose of this specification, and unless otherwise noted, the term 'comprise' shall have an inclusive meaning - i.e. that it will be taken to mean an inclusion of not only the listed components it directly references, but also other non-specified components or elements. This rationale will also be used when the term 'comprised' or
'comprising' is used in relation to one or more steps in a method or process.
Further aspects and advantages of the present invention will become apparent from the ensuing description which is given by way of example only.
According to one aspect of the present invention there is provided a method for a computing network, the method comprising:
tapping a high density data link between a first node and a second node of the computing network, wherein the high density data link has a plurality of parallel-serial data communication lanes over which data packets are communicated;
determining a lane order and a polarity of each of the plurality of lanes of a transmitting port between the first node and the second node without a training sequence; and
monitoring a flow of data packets transmitted between the first node and the second node of the computing network in response to the lane order and the polarity of each of the plurality of lanes of the transmitting port.
Preferably, the flow of data packets between the first node and the second node are monitored to determine a performance of the computing network.
Preferably, the method further comprising:
establishing the high density data link between the first node and the second node with a training sequence; and
wherein the determining of the lane order and polarity is performed after the high density data link is established and without the training sequence.
According to another aspect of the present invention there is provided a computer network device for a high density data link between a pair of nodes, the computer network device comprising: a tapping device coupled to the high density data link, wherein the high density data link has a plurality of parallel-serial data communication lanes over which data packets are communicated, the tapping device reading the data communicated over the high density data link; a switch coupled to the tapping device, the switch to selectively change polarity of one or more lanes of the plurality of parallel-serial data communication lanes in the high density data link; a logic circuit coupled to the switch, the logic circuit to calculate an error code for each packet and compare the calculated error code with a check code in each packet, and control the switch to change the polarity of one or more lanes until the calculated error code for each packet matches a Variant Cycle Redundancy Check (VCRC) for that packet.
Preferably, the switch further selectively realigns the order of the plurality of the parallel-serial data communication lanes; and
the logic circuit further controls the switch to realign the order of the plurality of the parallel-serial data communication lanes to reduce a difference between the error code and the check code.
Also described is a method of monitoring a plurality of transmission lanes in a data transmission system,
characterised by the step of
extracting individual lane parameters in the data transmission system independently of a transmitted training sequence.
In a preferred embodiment the data transmission system includes a plurality of transmission lanes in a parallel -serial architecture.
Reference throughout this specification to a parallel - serial architecture should be understood to refer to a link in which a plurality of physical lanes form serial connections, with information transmitted over each of the serial connections at substantially the same time.
In a preferred embodiment there are four transmission lanes.
The method may be used with any number of transmission lanes and reference to four transmission lanes only throughout this specification should not be seen as limiting.
A typical link in a multi-lane transmission system of the type consider here, such as the InflniBandTM system, uses four reception and four transmission lanes, denoted 4x, or a multiple thereof (8x, 12x etc). The lanes of a multi-lane link are conventionally labelled sequentially beginning with 0. Hence, a 4x link has physical lanes labelled 0, 1, 2, and 3.
Reference to lane parameters throughout this specification should be understood to mean the physical lane identifier and the polarity of a lane.
Reference to a training sequence throughout this specification should be understood to refer to a specific sequence of control and data symbols that is transmitted over a link when the link is established or re-established in order to establish the lane parameters for the link (ie, correlation of the lane parameters of the transmitting and receiving nodes of a link).
Also described is a method of determining the lane parameters of a plurality of transmission lanes in a data transmission system,
the method including the steps of:
a) receiving a packet transmitted over the plurality of transmission lanes, the packet including a start of packet delimiter, an end of packet delimiter and an error detecting code; b) assigning a first lane identifier to the reception lane that received the start of packet delimiter according to a preset protocol for the data transmission system; c) assigning a second lane identifier to the reception lane that received the end of packet delimiter according to a preset protocol for the data transmission system; d) assigning a unique lane identifier to each of the remaining reception lanes; e) assigning a lane polarity to each of the reception lanes; f) calculating an error code utilising the assigned lane identifier and lane polarity for each lane; g) comparing the calculated enor code with the received error detecting code; and h) repeating steps d) to g) until the calculated error code matches the received error detecting code wherein on each iteration the assignment of at least one lane identifier or lane polarity is changed.
Also described is a method of determining the lane parameters of a plurality of transmission lanes in a data transmission system substantially as outlined above, wherein step h) includes repeating steps a) to c).
Optionally the data transmission system includes a plurality of lanes in a parallel -serial architecture.
Optionally there are four transmission lanes.
Optionally the preset protocol requires the start of packet delimiter is transmitted in a prescribed transmission lane.
Optionally the preset protocol requires the packet to have a length restricted to a multiple of four bytes.
Optionally the error detecting code is a cycle redundancy check (CRC).
Optionally the cycle redundancy check is a variant cycle redundancy check (VCRC).
Also described herein is a computer network device configured to monitor a plurality of transmission lanes in a data transmission system, the computer network device configured in use to:
a) receive a packet transmitted over the plurality of transmission lanes, the packet including a Start of Packet delimiter, an End of Packet delimiter and an error detecting code;
b) assign a first lane identifier to the reception lane that received the Start of Packet delimiter according to a preset protocol for the data transmission system;
c) assign a second lane identifier to the reception lane that received the End of Packet delimiter according to a preset protocol for the data transmission system;
d) assign a unique lane identifier to the remaining transmission lanes;
e) assign a lane polarity to each of the transmission lanes;
f) calculate an error code utilising the assigned lane identifier and lane polarity for each lane; and
g) compare the calculated error code with the received error detecting code; and change at least one lane identifier of the said remaining reception lanes or one lane polarity if the calculated error code does not match the received error detecting code.
Information may be transmitted over a 4x link in the form of a packet. A packet may be a data packet or a link packet. In each case data is byte striped onto the four lanes with each packet beginning with a Start of Packet delimiter (SDP or SLP) and ending with an End of Packet delimiter (EGP or EBP).
In a preferred embodiment the preset protocol requires the Start of Packet delimiter is transmitted in a transmission lane having a prescribed physical lane identifier.
In a preferred embodiment the preset protocol requires the packet to have a length restricted to a multiple of four bytes.
Preferable the preset protocol corresponds to the InfiniBand protocol. Under the InfiniBand protocol the Start of Packet delimiter (SDP or SLP) is always transmitted in physical lane 0. Furthermore, packet lengths are restricted to multiples of four bytes, so that for a 4x IB system the End of Packet delimiter (EGP or EBP) is always transmitted in physical lane 3.
In a preferred embodiment the error detecting code is a cycle redundancy check (CRC).
A Cycle Redundancy Check (CRC) is an error detecting code that is transmitted with each packet. When the packet is received a calculation is performed based on the information contained in the packet to produce an error code as is well known by those skilled in the art. The error code is compared to the transmitted CRC and, if they do not match, an error is detected.
There are generally two CRCs in each packet, an Invariant CRC (1CRC) that covers all fields that should not change as the packet (including headers etc) traverses the fabric or network, and a Variant CRC (VCRC) that covers all of the fields of the packet.
The combination of the two CRCs allow switches and routers to modify appropriate fields and still maintain an end to end data integrity for the transport control and data portion of the packet.
In a preferred embodiment the error detecting code is a Variant Cycle Redundancy Check (VCRC)
Since ICRC is not present in all IB packets, only VCRC is used to perform CRC validation as described herein.
In a preferred embodiment the computer network device includes firmware.
An advantage of using firmware is that the VCRC can be checked and the lane parameters set as each packet is received, so that the correct lane parameters for all lanes can be established very quickly.
In a preferred embodiment the firmware is a Field Programmable Gate Array (FPGA).
A FPGA may be programmed to carry out the computation of the error code and to compare it with the transmitted error code, and to change the lane identifier and polarity if required, quickly and efficiently.
In alternative embodiments the computer network device may be a processor and a set of computer executable instructions. However, the applicant has found that a software-based solution may take many seconds to establish the correct lane alignment. With data transmission rates at several Gbits/s this delay means that a considerable amount of data is lost before the correct lane alignment is established.
In practice a high impedance tap may be used to read the symbols in each lane of a packet as it is transmitted over a multi-lane link. The order in which symbols are placed on the lanes when transmitted is determined by the protocol for the particular transmission system.
Also described is a method of determining when determination of lane parameters is required and re-establishing determination of the lane parameters of a plurality of transmission lanes in a data transmission system in accordance with the previous method, the method including: the further steps of a) receiving a packet for which the calculated error code matches the received error code and setting a realign parameter to zero; b) incrementing the realign parameter by one for each consecutive packet received for which the calculated error code does not match the received error detecting code; and c) determining the lane parameters for each lane independently of a transmitted training sequence when the value of the realign parameter exceeds a preset threshold value; or d) resetting the realign parameter to zero if a packet is received for which the calculated error code matches the received error detecting code.
In a preferred embodiment the preset threshold value is 5.
Although any value greater than 1 may be used for the threshold value before initiation of the method of lane alignment, the applicants have found that a value of 5 provides a reasonable indication that re-establishment of the lane alignment is required.
Also described herein is a data transmission system having a plurality of transmission lanes including a computer network device configured in use to: a) receive a packet transmitted over the plurality of transmission lanes, the packet including a start of packet delimiter, an end of packet delimiter and an error detecting code;
b) assign a first lane identifier to the reception lane that received the start of packet delimiter according to a preset protocol for the data transmission system;
c) assign a second lane identifier to the reception lane that received the end of packet delimiter according to a preset protocol for the data transmission system; d) assign a unique lane identifier to each of the remaining transmission lanes; e) assign a lane polarity to each of the transmission lanes; f) calculate an error code utilising the assigned lane identifier and lane polarity for each lane; and g) compare the calculated error code with the received error detecting code; and change at least one lane identifier of the said remaining transmission lanes or one lane polarity if the calculated error code does not match the received error detecting code.
Further aspects of the present invention will become apparent from the following description which is given by way of example only and with reference to the accompanying drawings in which:
Figure 1 shows a schematic representation of a sequence of packets; and Figure 2 shows a flow chart of the steps of a multi-lane alignment method according to one embodiment of the present invention; and. Figure 3 shows a flow chart of the steps of another multi-lane alignment method according to an embodiment of the present invention; and Figure 4 shows a schematic view of a network device according to one embodiment of the present invention.
In order to monitor transmissions on a multi-lane transmission system it is first necessary to establish the lane parameters (physical lane identifier and lane polarity) for each of the transmission lanes. In the method of the present invention this is achieved independently of a transmitted training sequence.
The invention will be illustrated with reference to an InfiniBandTM (IB) 4x link. However, it will be appreciated by those of ordinary skill in the art that the method of the present invention is not limited to InfiniBand T M, or to a 4x link, but can be applied to any transmission system using a multi-lane parallel - serial link.
The 4x link between two nodes of an IB network consists of four physical lanes (e.g, copper wire or fibre optic cable) which are conventionally numbered 0 to 3. The combined data and link packet symbol streams (control and data) are byte striped onto the physical lanes 0 to 3.
Data and link control information is transmitted over the IB links in packets that include at least: one or more headers; a payload packet; one or two Cyclic Redundancy Checks (CRCs) and an End of Packet delimiter.
The first symbol of a data packet is a Start of Data Packet Delimiter symbol, SDP (K27.7), which is transmitted to identify the start of a data packet. The first symbol of a link packet is a Start of Link Packet Delimiter symbol, SLP (K28.2), which is transmitted to identify the start of a link control packet.
Packet formatting rules specify which physical lanes may be used by the SDP and SLP control symbols. In the 1B protocol SDP and SLP are always transmitted in physical lane 0.
The End of Good Packet Delimiter symbol, EGP (K29.7), is used to mark the end of each packet (when it is transmitted by the originating port). Packet length rules restrict which physical lanes may be used to transmit the EGP control symbol. In an IB system packet lengths are restricted to multiples of 4 bytes. Therefore the EGP delimiter is always transmitted in physical lane 3 of a 4x 1B system, but may, for example, be in physical lane 3, 7 or 11 of a 12x IB system.
The End of Bad Packet Delimiter symbol, EBP (K30.7), is used to mark the end of a bad packet forwarded by a switch or router node. When an error (e.g., a CRC error) is detected in a data packet received by a switch or router node, it is marked bad by replacing the original EGP symbol with an EBP packet symbol. An EBP delimiter is always transmitted in the same lane as an EGP delimiter.
Receiving nodes are required to recognize either EGP or EBP as the end of packet delimiter. Any data (or link) packet terminated with an EBP symbol must be treated as if it had a CRC error. In the context of the present invention any received packet terminated with an EBP symbol should be ignored when determining lane alignment or polarity.
A number of other control symbols are commonly used in transmissions in an IB system.
The COMMA control symbol, COM (K28.5), is used by the physical lane receiver logic to identify symbol boundaries and for channel bonding purposes. Comma symbols are required to synchronise the receive logic when links are being trained. The comma symbol is also used as the start of ordered-set delimiter.
The SKIP symbol, SKP (K28.0), is included as part of the SKIP ordered-set used for clock tolerance compensation. A SKIP ordered-set begins with a COM followed by a one or more SKP symbols in each lane. The SKIP ordered-set is not byte striped across the lanes; rather it is transmitted simultaneously on alllanes.
The Padding symbol, PAD (K23.7), is added to the end of a packet (for IB systems having more than 4 lanes) to align the total transmitted packet length with the number of physical lanes. For example, in the case of a 12x IB system the End of Packet delimiter (EGB/EBP) may be transmitted in physical lane 3, 7 or 11, depending on the length of the packet. In this case 8, 4 or 0 PAD symbols respectively are added to the packet so that the total number of symbols transmitted is a multiple of 12.
An IB training sequence (TS1 or TS2) consists of an ordered sequence of symbols beginning with a COM followed by a physical lane identifier data symbol and a number of data symbols unique to the training sequence. The symbols are transmitted simultaneous across the lanes (ie not byte striped), starting with a COM in each lane, followed by a lane identifier for each lane and so on. The information contained in the training sequence is sufficient to determine the lane parameters of the receiving lanes so that they correspond to the lane parameters of the transmission lanes.
Figure 1 shows a schematic representation of a typical sequence of transmitted packets in a 4x IB system. In the example shown in Figure 1, two data packets are followed by a link packet, a SKIP ordered-set, a link packet and a data packet.
Each packet begins with a Start of Packet delimiter (SDP/SLP) transmitted in physical lane 0, and ends with an End of Packet delimiter (EGB/EBP) transmitted in physical lane 3. Only the Start of Packet delimiters and End of Packet delimiters are shown for each packet in Figure 1 - details of the symbols making up the header, CRCs etc are not shown for simplicity.
A flow chart of the steps of the method of determining the lane parameters according to one embodiment of the present invention for a 4x IB system is shown in Figure 2.
In step a) a packet containing a Start of Packet delimiter, End of Packet delimiter and a VCRC, is received. In step b) the lane containing the Start of Packet delimiter (SDP/SLP) is assigned to physical lane 0. In step c) the lane containing the End of Packet delimiter (EGP/EBP) is assigned to physical lane 3. If the End of Packet delimiter is EBP the packet is ignored and the method reverts to step a) for a new packet.
In step d) the remaining two lanes are arbitrarily assigned to physical lanes 1 and 2.
In step e) a polarity (either normal or inverted is assigned to each lane. In step f) an error code is calculated (using methods well know in the art) for the packet using the assigned lane order and polarities
The calculated error code is compared with the received VCRC in step g). If the calculated error code agrees with the VCRC then the physical lane identifier and polarity for each lane corresponds to the transmitted lanes and monitoring can commence.
If the calculated error code does not agree with the VCRC then, in step h), steps d) to g) are repeated with either the lane assignment for lanes 1 and 2 reversed, or one of the lane polarities reversed. This process is repeated for each of the 32 possible combinations (16 possible combinations of polarity for the four lanes and 2 possible lane assignments for physical lanes 1 and 2) until a combination is found for which the calculated error code matches the VCRC.
A similar method may be applied to determine the lane parameters for 4x and 12x 1B systems by counting the number of PAD symbols in the packet and using this to determine the physical lane identifier for the lane containing the End of Packet symbol. For example, in an 8x IB system the End of Packet symbol may be in physical lane 3 or physical lane 7. The End of Packet symbol will be in physical lane 7 unless the packet contains 4 PAD symbols (or indeed any PAD symbol) in which case will be in physicallane 3. Similarly, for a 12x IB system the End of Packet symbol will be in physical lanes 3, 7 or 11 as determined by the packet having 8, 4 or 0 PAD symbols respectively.
In principle the method of determining lane parameters may be carried out for a single (data or link) packet. For example, the information from a single packet may be processed in a computing device using software to step through each of the various combinations until agreement is found.
However, the applicant has found that a software-based system may take a significant time (seconds) on average to find a solution. In addition, it is possible that no solution can be found for an individual packet, for example if it has lost integrity during transmission (from the previous transmitting node) or during reception. In such cases a new packet is received and steps a) to h) repeated.
In a preferred embodiment of the present invention the alignment process is carried out by a FPGA. In this embodiment the error code is calculated and compared with the VCRC as normal when receiving an 1B packet. If the calculated error code does not agree with the VCRC the FPGA is programmed to either reverse the lane assignment of physical lanes I and 2, or change the assigned lane polarity (for at least one lane). This new lane alignment is used to calculate an error code for the next packet received and to compare that with the VCRC for that packet and so on (repeating steps a) to h) until the correct lane alignment is established.
This process is relatively fast and in principle should determine the correct lane order and polarity at least during receipt of around 32 packets. Typically, using an FPGA the correct lane alignment may be established, and monitoring started, within a few microseconds of receipt of the first packet.
Once agreement is found between the calculated error code and the VCRC for a packet the method of alignment is stopped and the lane assignments and polarities fixed. At this stage monitoring of the transmissions can begin.
Under normal operation of an IB system a validation check is carried out on each received packet to check the integrity of the contents of the packet. As each new packet is received an error code is calculated and compared with the VCRC for the packet. At some point a packet may be received for which the calculated error code does not agree with the VCRC. This is most likely to be due to a loss of integrity during transmission or reception, but could be caused by a change in lane alignment.
Although the lane alignment method of the present invention may be fast, especially when implemented on firmware, there is still a delay and a significant amount of information may be lost while lane alignment is re-established.
The present invention includes a method for re-establishing alignment of a plurality of transmission lanes when the number of consecutive packets received for which the calculated error code does not match the received error code exceeds a preset threshold value (typically 5 consecutive invalid packets).
A flow chart of the method is shown in Figure 3. In step i) a packet is received which includes an SOP, EOP and VCRC. The error code is calculated and compared with the VCRC. If the calculated error code agrees with the VCRC then the realign parameter is set to zero, shown in Figure 3 as step I).
If the calculated error code does not agree with the VCRC then the realign parameter is incremented by 1 in step j) and the realign parameter is compared with the preset threshold value, typically 5. If the value of the realign parameter is greater than the preset threshold value then the lane parameters are reset using steps a) to h) of the method of determining the lane parameters as described above and shown in Figure 2. Otherwise the method returns to step i) and the same process repeated for the following packet.
In this manner re-determination of the lane parameters only occurs following a sequence (5 in the present case) of invalid packets (ie., calculated and received error codes do not match) are received. This method avoids resetting of the lane parameters unless the link is down. Random invalid packets can occur from time to time, but the probability of 5 or more sequential invalid packets is small unless the lane parameters have changed.
Figure 4 shows a schematic view of a 4x IB link between an IB node A and an IB node B and connected to a computer network device (20). Data transmitted along each of the four lanes (indicated (arbitrarily) by lane identifiers 0-3) is read by a tap linked to the computer network device (20).
The FPGA in the computer network device (20) includes a switch configured to switch the lane identifier or polarity of data for each transmission lane. The switch settings for each transmission lane are controlled by firmware in the FPGA.
The network device is set up initially to one of the 384 (24x16) possible combinations of the lane identifiers and polarities for the four lanes. On receipt of the first package, transmitted SOP and EOP delimiters are detected by the FGPA and the lanes containing them identified as lanes 0 and 3 respectively in accordance with the 4x IB protocol. The FGPA is programmed to set the lane identifiers 0 and 3 to the physical lanes that contained SOP and EOP delimiters respectively.
The FGPA is further programmed to calculate the error code and to compare it with the received VCRC. If the calculated error code matches the VCRC for the packet then the lane parameters are correct (link up) and monitoring of further packets can commence.
If the calculated error code does not match the VCRC then the FPGA is programmed to change the polarity setting for one of the lanes or the lane identifier for one of the remaining lanes (1 and 2) and this setting is used to receive the following packet.
The above process is repeated for each further packet received until a calculated error code for a packet matches the VCRC for that packet, after which the settings are fixed and normal monitoring can occur.
The above process for determination of the line parameters is repeated by the FPGA when a sequence of consecutive packets is received for which the calculated error code does not match the VCRC for each packet. In practice a sequence of 5 (or so) consecutive such packets is received before commencing the realignment of the lane parameters.
The present invention provides a number of advantages over the prior art, including the ability to establish the correct lane alignment and parity for each lane in a multi-lane parallel - serial link without requiring transmission of a training sequence. The corresponding lane alignment and polarity for each lane can be established relatively quickly, especially when implemented by firmware. Furthermore, the method is applicable to any number of lanes provided the protocol for transmitting the Start of Packet and End of Packet delimiters is known, and an error detecting code is transmitted with the packet. These provisos are likely to be met by any commercially available network architecture.
The method and apparatus of the present invention enable monitoring of a SAN to be established quickly without disturbance of the network.
Aspects of the present invention have been described by way of example only and it should be appreciated that modifications and additions may be made thereto without departing from the scope thereof of the appended claims.
Claims (5)
1. A method for a computing network, the method comprising:
tapping a high density data link between a first node and a second node of the computing network, wherein the high density data link has a plurality of parallel-serial data communication lanes over which data packets are communicated;
determining a lane order and a polarity of each of the plurality of lanes of a transmitting port between the first node and the second node without a training sequence; and
monitoring a flow of data packets transmitted between the first node and the second node of the computing network in response to the lane order and the polarity of each of the plurality of lanes of the transmitting port.
2. The method of claim 1, wherein
the flow of data packets between the first node and the second node are monitored to determine a performance of the computing network.
3. The method of claim 1, further comprising:
establishing the high density data link between the first node and the second node with a training sequence; and
wherein the determining of the lane order and polarity is performed after the high density data link is established and without the training sequence.
4. A computer network device for a high density data link between a pair of nodes, the computer network device comprising:
a tapping device coupled to the high density data link, wherein the high density data link has a plurality of parallel-serial data communication lanes over which data packets are communicated, the tapping device reading the data communicated over the high density data link;
a switch coupled to the tapping device, the switch to selectively change polarity of one or more lanes of the plurality of parallel-serial data communication lanes in the high density data link; a logic circuit coupled to the switch, the logic circuit to calculate an error code for each packet and compare the calculated error code with a check code in each packet, and control the switch to change the polarity of one or more lanes until the calculated error code for each packet matches a Variant Cycle Redundancy Check (VCRC) for that packet.
5. The computer network device of claim 4, wherein
the switch further selectively realigns the order of the plurality of the parallel-serial data communication lanes; and
the logic circuit further controls the switch to realign the order of the plurality of the parallel-serial data communication lanes to reduce a difference between the error code and the check code.
WO 2010/039050 PCT/NZ2009/000210 1/4
FIGURE 1 2021203290
Physical Lanes I #0 #1 #2 #3 5DP | 1 Data ' ■ ■■
Jv V".-" r : Packet U: ■. •
Payload.' mil 3 SDP Data ss.-- oi Packet.. PayloacT I f IOF SLP I Link Racket I Pavioad EGP MSI mmm ftjr 8^
SRI SKP §H SKF SKP SKP SKR SKP SRI SKP SKP SKP SLR LinkPacket...^_____ Pavioad I / EGP SDP | Data X’-XT&wsr §
! -ivyS-stA :i: W’i:;" i': •k ;• g 8's 8' 8 8 :i: ?*■!• S.i Packet I ...i Paylsad ■E^pai
4x Packet Formats
Figure 2 Receive packet including SOP, EOF and error detecting code * O hj
Y Link monitoring O Link Up ? Ui See figure 3 N o
Assign SOP to a first lane ID according to protocol b) 1 f
C) Assign EOF to a second lane ID according to protocol
'r
Assign unique lane ID to each remaining lane d)
Assign polarity to each lane e) U
Calculate error code ffl Calculated error code Y Correct lane ID = and lane polarity found -c g) Received error code ? = Link Up n H 2 hj N 3 ^ f o o h) Reverse lane ID of the remaining lanes or reverse lane polarity of one of the lanes O
Figure 3 Receive packet including SOP, EOF and error detecting code * O bJ
i) O Calculate error code <2 % O
ir
i) Calculated error code Y Clear realign parameter Received error code ?
N
Increment realign parameter j)
Realign parameter N > Threshold ?
Y -c n Invalid lane ID and/or lane polarity H k) = Link Down 2 hj
'r 3 o s Link Training See figure 2 O
Figure 4 2 o hJ — >■
IB «- IB o u * + \o node 4 «- node o A 4 Zzz * B
ir v w ir
0 TAP 1 2 ir ir ir ir 3 4^ Transceivers
Polarity and lane FPGA identifier 20 switches -c n H 2 bJ
3 O o
o
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| AU2021203290A AU2021203290B2 (en) | 2008-10-02 | 2021-05-21 | Lane synchronisation |
Applications Claiming Priority (6)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US12/244,076 US8369233B2 (en) | 2008-10-02 | 2008-10-02 | Lane synchronisation |
| US12/244,076 | 2008-10-02 | ||
| AU2009300444A AU2009300444B2 (en) | 2008-10-02 | 2009-10-01 | Lane synchronisation |
| AU2016253617A AU2016253617A1 (en) | 2008-10-02 | 2016-11-03 | Lane synchronisation |
| AU2018274871A AU2018274871B2 (en) | 2008-10-02 | 2018-12-04 | Lane synchronisation |
| AU2021203290A AU2021203290B2 (en) | 2008-10-02 | 2021-05-21 | Lane synchronisation |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| AU2018274871A Division AU2018274871B2 (en) | 2008-10-02 | 2018-12-04 | Lane synchronisation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| AU2021203290A1 AU2021203290A1 (en) | 2021-06-17 |
| AU2021203290B2 true AU2021203290B2 (en) | 2023-07-27 |
Family
ID=42074048
Family Applications (4)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| AU2009300444A Active AU2009300444B2 (en) | 2008-10-02 | 2009-10-01 | Lane synchronisation |
| AU2016253617A Abandoned AU2016253617A1 (en) | 2008-10-02 | 2016-11-03 | Lane synchronisation |
| AU2018274871A Active AU2018274871B2 (en) | 2008-10-02 | 2018-12-04 | Lane synchronisation |
| AU2021203290A Active AU2021203290B2 (en) | 2008-10-02 | 2021-05-21 | Lane synchronisation |
Family Applications Before (3)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| AU2009300444A Active AU2009300444B2 (en) | 2008-10-02 | 2009-10-01 | Lane synchronisation |
| AU2016253617A Abandoned AU2016253617A1 (en) | 2008-10-02 | 2016-11-03 | Lane synchronisation |
| AU2018274871A Active AU2018274871B2 (en) | 2008-10-02 | 2018-12-04 | Lane synchronisation |
Country Status (5)
| Country | Link |
|---|---|
| US (1) | US8369233B2 (en) |
| EP (2) | EP2963858B1 (en) |
| AU (4) | AU2009300444B2 (en) |
| NZ (2) | NZ619407A (en) |
| WO (1) | WO2010039050A2 (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI423007B (en) * | 2009-12-31 | 2014-01-11 | Via Tech Inc | Serial bus device and clock difference compensation method thereof |
| US9380005B2 (en) * | 2011-11-03 | 2016-06-28 | Cisco Technology, Inc. | Reliable transportation of a stream of packets using packet replication |
| US9054974B2 (en) | 2012-07-30 | 2015-06-09 | Cisco Technology, Inc. | Reliably transporting packet streams using packet replication |
| US8891357B2 (en) | 2012-08-31 | 2014-11-18 | Cisco Technology, Inc. | Switching to a protection path without causing packet reordering |
| CN105793828B (en) | 2013-12-26 | 2019-06-21 | 英特尔公司 | The system and method quickly enhanced with PCI |
| WO2015099733A1 (en) * | 2013-12-26 | 2015-07-02 | Intel Corporation | Interconnect retimer enhancements |
| US9485334B2 (en) * | 2014-03-12 | 2016-11-01 | Qualcomm Incorporated | Response time relaxation for high efficiency WLAN |
| US10491611B2 (en) * | 2016-01-08 | 2019-11-26 | Belden, Inc. | Method and protection apparatus to prevent malicious information communication in IP networks by exploiting benign networking protocols |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030103253A1 (en) * | 2001-11-19 | 2003-06-05 | Bunton William P. | Time-division and wave-division multiplexed link for use in a service area network |
| US7305047B1 (en) * | 2003-03-12 | 2007-12-04 | Lattice Semiconductor Corporation | Automatic lane assignment for a receiver |
| US7426597B1 (en) * | 2003-05-07 | 2008-09-16 | Nvidia Corporation | Apparatus, system, and method for bus link width optimization of a graphics system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7010607B1 (en) * | 1999-09-15 | 2006-03-07 | Hewlett-Packard Development Company, L.P. | Method for training a communication link between ports to correct for errors |
| US6985502B2 (en) * | 2001-11-19 | 2006-01-10 | Hewlett-Packard Development Company, L.P. | Time-division multiplexed link for use in a service area network |
| US7653842B2 (en) * | 2004-12-15 | 2010-01-26 | Fluke Corporation | CRC format automatic detection and setting |
-
2008
- 2008-10-02 US US12/244,076 patent/US8369233B2/en active Active
-
2009
- 2009-10-01 EP EP15161252.0A patent/EP2963858B1/en active Active
- 2009-10-01 EP EP09818040.9A patent/EP2347544B1/en active Active
- 2009-10-01 AU AU2009300444A patent/AU2009300444B2/en active Active
- 2009-10-01 NZ NZ619407A patent/NZ619407A/en unknown
- 2009-10-01 NZ NZ592547A patent/NZ592547A/en unknown
- 2009-10-01 WO PCT/NZ2009/000210 patent/WO2010039050A2/en active Application Filing
-
2016
- 2016-11-03 AU AU2016253617A patent/AU2016253617A1/en not_active Abandoned
-
2018
- 2018-12-04 AU AU2018274871A patent/AU2018274871B2/en active Active
-
2021
- 2021-05-21 AU AU2021203290A patent/AU2021203290B2/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030103253A1 (en) * | 2001-11-19 | 2003-06-05 | Bunton William P. | Time-division and wave-division multiplexed link for use in a service area network |
| US7305047B1 (en) * | 2003-03-12 | 2007-12-04 | Lattice Semiconductor Corporation | Automatic lane assignment for a receiver |
| US7426597B1 (en) * | 2003-05-07 | 2008-09-16 | Nvidia Corporation | Apparatus, system, and method for bus link width optimization of a graphics system |
Also Published As
| Publication number | Publication date |
|---|---|
| AU2021203290A1 (en) | 2021-06-17 |
| AU2018274871A1 (en) | 2019-01-03 |
| EP2963858B1 (en) | 2020-03-18 |
| WO2010039050A3 (en) | 2010-06-03 |
| EP2347544A2 (en) | 2011-07-27 |
| NZ619407A (en) | 2015-05-29 |
| NZ592547A (en) | 2014-01-31 |
| AU2009300444A1 (en) | 2010-04-08 |
| US20100085873A1 (en) | 2010-04-08 |
| AU2018274871B2 (en) | 2021-02-25 |
| EP2963858A1 (en) | 2016-01-06 |
| EP2347544A4 (en) | 2013-10-30 |
| EP2347544B1 (en) | 2015-04-01 |
| AU2016253617A1 (en) | 2016-11-24 |
| WO2010039050A2 (en) | 2010-04-08 |
| US8369233B2 (en) | 2013-02-05 |
| AU2009300444B2 (en) | 2016-08-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2021203290B2 (en) | Lane synchronisation | |
| US7613183B1 (en) | System and method for router data aggregation and delivery | |
| US20050083934A1 (en) | Header compression/decompression device and header compression/decompression method | |
| US20090190595A1 (en) | Method, system and device for transmitting overhead information | |
| US11108675B2 (en) | Methods, systems, and computer readable media for testing effects of simulated frame preemption and deterministic fragmentation of preemptable frames in a frame-preemption-capable network | |
| CN108631873B (en) | Method and device for transmitting and receiving network management information, transmitting equipment and receiving equipment | |
| US9479277B2 (en) | Mechanism for channel synchronization | |
| EP3672167B1 (en) | Packet forwarding method and device | |
| US20160285776A1 (en) | Physical-layer signaling of flow control updates | |
| WO2021135625A1 (en) | Pon multi-channel binding transmission method, pon node and storage medium | |
| US9485053B2 (en) | Long-distance RapidIO packet delivery | |
| EP2974109B1 (en) | Systems and methods to explicitly realign packets | |
| CN107783922A (en) | A kind of data decoding method and device | |
| CN108574530B (en) | Data transmitting and receiving method and device and multichannel EPON system | |
| US10148454B2 (en) | 128 gigabit fibre channel speed negotiation | |
| US8050294B2 (en) | Method and system for transmitting in TDM mode | |
| US11362958B2 (en) | Device and method for transferring identification and/or data flow control information between devices | |
| US7242685B1 (en) | System and method for initiating multilink fragment reassembly | |
| CN103199934B (en) | Transmitting/receiving system and transmission/method of reseptance | |
| JP4858852B2 (en) | Network testing system | |
| WO2022160260A1 (en) | Data transmission method and apparatus | |
| KR100638015B1 (en) | Data packet process method | |
| CN101170541A (en) | Packet receiving and transmission system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| FGA | Letters patent sealed or granted (standard patent) |