JP2005522147A - Method and apparatus for fiber channel frame delivery - Google Patents

Abstract

Methods and apparatus are provided for improving fibre channel frame delivery. Techniques are provided for the in order delivery of frames by intelligently delaying or dropping selected fibre channel frames. Other techniques are provided for in order delivery by using label switching and frame labels. The various techniques can be applied during circumstances such as a link state or channel change.

Description

The present invention relates to a fiber channel network. More particularly, the present invention relates to a method and apparatus for providing in-order delivery of fiber channel frames in a fiber channel network during an environment where link conditions change or channel conditions change.
This application is based on Scott S., incorporated herein by reference in its entirety for all purposes. Lee and Danish G. US patent application Ser. No. 10 / 114,394 (Attorney Docket No. ANDIP009) filed at the same time entitled “Label Switching In Fiber Channel Networks”

  Many conventional network protocols allow out-of-order delivery of packet sequences. A network node in a TCP / IP based network receives an out-of-order set of packets and reorders the packets upon receipt. Packets often arrive out of order as they travel along different paths and reach their destination.

  However, some Fiber Channel devices, such as disks, disk arrays, and other storage mechanisms cannot handle out-of-order frames. Link and channel state changes are some of the environments that can cause out-of-order delivery of frames in a fiber channel fabric. Multiple links that appear as a single link between two Fiber Channel entities are referred to herein as channels. Some mechanisms in existing networks require flushing all frames in the network when link conditions change. Flushing all frames can prevent out-of-order delivery when routes and routes change in the network. All frames are flushed even if the frame path to the associated destination does not change. However, clearing or implicitly flushing all frames will significantly hinder network operations because more frames are dropped than necessary and network operations are at least temporarily suspended. .

  Therefore, it would be desirable to provide a method and apparatus that improves fiber channel frame delivery, and in particular provides delivery in the correct order during link conditions and channel changes.

  Methods and apparatus are provided for improving fiber channel frame delivery. Techniques are provided for delivering frames in order by intelligently delaying or dropping selected Fiber Channel frames. Other techniques for in-order delivery by using label switching and frame labels are also provided. Various techniques can be applied in environments such as link or channel state changes.

  In accordance with various embodiments, a method is provided for selectively delivering fiber channel frames in a fiber channel fabric. The next set of hops is identified in the Fiber Channel entity. The next hop set is used to forward the frame based on the destination identifier. A change in the fiber channel fabric link is detected. An updated set of next hops is identified. The updated set of next hops is used to forward frames received at the Fiber Channel entity based on the destination identifier, taking into account changes in the Fiber Channel fabric link. The next hop set is compared to the updated set of next hops. Forwarding frames towards the updated set of next hops is prevented for a predetermined time if it is determined that the next set of hops is different from the updated set of next hops.

  In accordance with another embodiment, a method is provided for selectively delivering frames in a fiber channel fabric. A channel for transferring frames from the first fiber channel entity to the second fiber channel entity is identified. The channel includes a plurality of links connecting a first fiber channel entity to a second fiber channel entity. Channel changes are detected at the first fiber channel entity. An updated channel for transferring a frame from the first fiber channel entity to the second fiber channel entity is identified. The updated channel is different from the channel. Frames received for transfer over an updated channel that have not yet been placed in the output queue associated with the updated channel are blocked.

  In yet another embodiment, a method is provided for selectively delivering frames in a fiber channel fabric. A change in the fiber channel fabric link is detected in the fiber channel switch. An updated routing table associated with the topology version number is generated. Generating an updated routing table includes determining the next hop, received label, and destination corresponding to each entry in the updated routing table. A frame is received at the fiber channel switch. The frame includes a first destination and a first label corresponding to the first entry in the updated routing table. It is determined whether the Fiber Channel switch has received a first transmission label corresponding to the first entry in the updated routing table, and the received first transmission label is the same topology version number as the updated routing table. Have If it is determined that the Fiber Channel switch has not received the first transmission label, the frame is dropped.

These and other features and advantages of the present invention will be described in detail below with reference to the accompanying drawings which illustrate, by way of example, the principles of the invention.
The invention will be best understood from the following description taken in conjunction with the accompanying drawings, which illustrate certain embodiments thereof.

  Several specific embodiments of the invention are described below, including the best mode contemplated by the inventors for carrying out the invention. These particular embodiments are illustrated in the accompanying drawings. While the invention will be described with reference to these specific embodiments, it will be understood that it is not intended to limit the invention to the embodiments shown. On the other hand, it is intended to cover alternatives, modifications and equivalents that fall within the spirit and scope of the invention as defined by the claims.

  The method and apparatus of the present invention provides in-order delivery of fiber channel frames. Based on various embodiments, a number of network conditions can lead to out-of-order delivery of frames to Fiber Channel devices. The technique of the present invention delays, blocks, drops and / or labels certain Fiber Channel frames in order to deliver the frames to the Fiber Channel device in the correct order. In one embodiment, frames that traverse the new path are blocked to allow frames traveling along the old path to reach the destination first or drop from the network.

  FIG. 1 is a diagram illustrating an example of a network in which the technology of the present invention can be used. FIG. 1 shows a storage area network implemented using Fiber Channel. The switch 101 is connected to the switches 103 and 105 and to the host 111 and the storage device 121. In one embodiment, the host 111 is a server or client system, while the storage device 121 is a storage subsystem such as a single disk or a redundant array of independent disks (RAID). The switch 105 is connected to the switch 107. The switch 107 is connected to the host 113, and the switch 103 is connected to the storage device 123. The switch 109 is connected to the host 115, the switch 107, the host 153, and the external network 151 that may or may not use the fiber channel. A path through the switch 105 can be used for the host 111 to access the network 151. Note that the device that includes the processor, memory, and connections to the Fiber Channel fabric is a Fiber Channel switch.

  The ports used to connect switches to each other in a Fiber Channel network are referred to herein as non-F ports, while the ports used to connect the switches to hosts are referred to herein as F ports. In one example, the non-F port is used to connect switch 105 to switch 107, while the F port is used to connect switch 107 to host 113. Similarly, the FL port is used to connect the switch 103 to the storage device 123. Ports such as F port and FL port are referred to herein as edge ports. Other ports are referred to as non-edge ports.

  According to various embodiments, a frame transmitted from the host 111 to the network 151 or the storage device 153 includes parameters such as an exchange identifier, a sequence, and a sequence number. The exchange identifier can give information on what exchange the frame belongs to. The sequence can give information on which part of the exchange the frame belongs to, while the sequence number can give information on how to order the frames. The sequence number can be used to allow in-order delivery of Fiber Channel frames.

  Certain Fiber Channel devices, such as certain storage disks and disk arrays, require that frames be received in the order in which they were transmitted. Conventional networks, such as TCP / IP networks, do not have such a requirement because TCP / IP networks generally have a mechanism to reorder packets as they are received. If frames with sequence numbers 191, 192 and 193 are transmitted in order in the Fiber Channel network, the Fiber Channel device receiving those frames expects the frames to be in the same order as transmitted. The Fiber Channel device may not be able to handle the reception of ordered frames.

  In static Fiber Channel networks, frames are usually received in the order in which they were transmitted. However, numerous events can result in out-of-order delivery of fiber channel frames. In particular, when the link state changes, out-of-order delivery may occur.

  FIG. 2 is a diagram illustrating a fiber channel fabric subjected to a change in the link of the fiber channel fabric. FIG. 2 shows an example of link changes that can result in out-of-order delivery of Fiber Channel frames. A new link using a non-edge port is introduced between the switch 103 and the switch 107. When a new link is introduced between the switch 103 and the switch 107, a new version of the routing table can be generated. Various routing table generation algorithms such as Fiber Channel Shortest Path First (FSPF) can be used. Traffic from the host 111 to the switch 109 via the switches 101, 105 and 107 can now proceed to the switch 109 via the switches 101, 103 and 107. The next set of hops that can be used to transmit the storage device 153 frame from the switch 111 is the switch 105 before the introduction of a new link.

  A set of adjacent Fiber Channel entities that can be used to transmit a frame from one Fiber Channel entity to another is referred to herein as the next set of hops. After the link changes, the next set of hops can be updated at the switch. In one example, the next set of hops for sending frames from switch 101 to network 151 will simply change from switch 105 to both switches 103 and 105 after the link state changes. The set of neighboring entities that can be used to send frames from the source to the updated destination after the link changes or the updated routing table is generated, here the updated set of next hops Called. Note that the next hop set may include one or more neighboring nodes. In one example, the next hop set is a single neighboring entity. In another example, the next set of hops includes a number of neighboring entities.

  The updated set of next hops can result in out-of-order delivery of Fiber Channel frames. In one example, the first frame transmitted in switch 101 in a sequence proceeds through switch 105, while subsequent frames in the same sequence proceed through switch 103. Due to various network conditions, the first frame traveling through switch 105 may arrive at switch 109 after a subsequent frame traveling through switch 103 arrives at switch 109. In one example, the first frame is slowed down at switch 105 due to congestion at switch 105, while subsequent frames are at switch 103 because there is a new high bandwidth link between switch 103 and switch 107. To proceed quickly. Before receiving the first frame transmitted from the host 111, the storage device 153 that receives a frame transmitted thereafter may not be able to handle out-of-order frames.

  FIGS. 3A and 3B show a routing table that shows information related to the next set of hops and the updated set of next hops in the switch 101. FIG. 3A shows the next set of hops for the frame received at switch 101 and the network where the link between switch 103 and switch 107 has not yet been established. When a frame is received by the switch 101, an entry indicating the destination of the frame can be used to reference an entry in the routing table. In one example, the destination of the frame is switch 107 and the entry 309 can be referenced to determine that the next set of hops is switch 105. If it is determined that the destination of the received frame is the switch 101, the frame can be passed to the processor associated with the switch 101 using the routing table. It is also possible to drop frames using a routing table. In one example, a value, such as a null value, can be placed in the next set of hops and a frame with a destination associated with the null value can be dropped when referencing the routing table. An entry that instructs to drop a frame with a specific destination is referred to as an adjacency drop.

  In one embodiment, a routing table is provided for each virtual storage area network (VSAN) of which the switch is a part. Note that a Fiber Channel switch can be part of many different VSANs and a routing table is provided for each VSAN associated with the switch.

  After the link is added and the switch 103 is connected to the switch 107, the routing table is updated. FIG. 3B is a routing table showing the updated set of next hops. According to various embodiments, a frame having a destination set as switch 107 can be forwarded along either switch 103 or switch 105 based on entry 329. In a stable topology, all frames in a particular flow or exchange follow the same path.

  Note that the routing table may allow both routes or select the best route. If the selected best path has the next hop of switch 105, the updated set of next hops is the same as the next hop set prior to the link change, as shown in FIG. 3A. is there. If the selected route is switch 103 for a frame with the destination of switch 107, the updated set of next hops is different from the next hop set in the routing table prior to the link change. Determining whether the updated set of next hops is different from the first set of next hops is useful in determining whether a particular frame should be blocked or dropped. In one example, if the path for a particular sequence of frames remains unchanged after a link change in the Fiber Channel fablock, the frame is not blocked or dropped. If the path for the sequence of frames is kept the same, the frames are delivered in order to the destination. If the path to the sequence of frames changes, the frames may be delivered out of order.

  One mechanism that can affect the order in which Fiber Channel frames are delivered is a queue within the Fiber Channel switch. Frames initially transmitted from the host can be held in a queue associated with the switch 105, while frames transmitted later from the host can be quickly delivered via the switch 103. FIG. 4 is a diagram illustrating a queue that can be associated with a Fiber Channel switch in accordance with various embodiments. Although one particular type of queue is described, it should be noted that the techniques of the present invention can be implemented using a variety of different input and output queues associated with different input and output ports.

  The switch 401 is connected to the external nodes 451, 453, 455, and 457. The switch 401 includes a shared memory buffer 403 associated with each switch port. An external node 451 is associated with the buffer 403. The buffers associated with external nodes 453, 455, and 457 are not shown for clarity. The buffer 403 can hold traffic destined for the external nodes 453, 455, and 457 and loopback traffic to the external node 451.

  In the exemplary embodiment, all frames destined for various foreign nodes are placed in the same buffer 403. As a result, when the switch 401 receives a large number of frames destined for a specific node such as the external node 453, the frame related to the external node 453 can use the entire buffer 403. According to various embodiments, frames stored in buffer 403 are referenced by pointers in frame descriptor queues 411-447. Each frame descriptor may include a pointer or reference that identifies where in the buffer 403 the frame is stored. A pointer or reference to a shared buffer is referred to herein as a descriptor. The descriptor can also identify other information such as frame priority.

  In one example, the arbitrator 405 selects a frame using a round robin method. In the first round, a frame destined for the external node 453 is selected. In the second round, frames destined for external node 455 are selected, and so on. More specifically, the arbitrator 405 first selects the high priority frame associated with the descriptor 411 destined for the external node 453 and then the high priority frame associated with the descriptor 421 destined for the external node 455. And then the high priority frame associated with descriptor 431 destined for external node 457, and so on. It should be noted that various techniques for selecting frames can be used, as will be apparent to those skilled in the art.

  A queuing system having buffers that are allocated based on destination is referred to herein as a virtual output queue (VOQ). VOQ is a Proc. Of June 1988, incorporated herein by reference in its entirety for all purposes. Of 15th Ann. Symp. on Comp. It is described in detail in “High Performance multi-queue buffers for VLSI communications switches” by Tamil Y and Fragia G, published on pages 343-354 of Arch. An abstraction that identifies traffic with certain characteristics between two nodes is referred to herein as a flow. In one example, a flow is referenced by a source identifier, a destination identifier, a priority, a class, and an exchange identifier. Other characteristics are possible. Note, however, that flows may be referenced only with source and destination identifiers.

  According to various embodiments, frames of a particular flow may be blocked because buffer 403 is full. If another route is provided for the same sequence of frames, later frames may be able to traverse the Fiber Channel fabric more quickly than frames that remain in a congested switch. In one embodiment, a later frame along another route can be blocked to make the earlier frame reach the destination first. The frame transmitted first is referred to herein as the previous frame, while the frame transmitted later by the source is referred to herein as the subsequent frame. A number of mechanisms can be used to block later frames. In one embodiment, the arbitrator 405 may not simply select a sequence of frames for transmission to an external node. In another embodiment, later frames may not be queued at all for transmission scheduling until a certain period of time has elapsed.

  The time period can be determined in a number of ways. According to various embodiments, the Fiber Channel switch is configured to buffer the frame at or below the Fiber Channel switch latency until the frame is dropped. Note, however, that some switches may not drop frames after the fiber channel switch latency has elapsed. In one embodiment, a frame is removed from the virtual output queue and dropped if the frame is held in a buffer in the switch for longer than the latency period. Fiber Channel switch latency depends on switching speed and network congestion. The amount of time that can be spent in the Fiber Channel switch before the frame is dropped is referred to herein as Fiber Channel switch latency.

  The amount of time that can be spent in the Fiber Channel network before a frame is dropped is referred to herein as the Fiber Channel fabric drain latency or network drain latency. According to various embodiments, the fiber channel fabric drain latency or network drain latency is calculated by multiplying the fiber channel switch latency by the maximum number of hops required for the frame to traverse the fiber channel fabric. A wide variety of techniques for determining Fiber Channel switch latency and Fiber Channel fabric drain latency are contemplated.

  FIG. 5 is a process flowchart illustrating the transfer of frames arriving from a connected host or disk. At 501, a change in the fiber channel fabric link is detected. As described above, a change in the link of the fiber channel fablock results in an out-of-order delivery of the frame sequence to the fiber channel device. According to various embodiments, a change in the fiber channel fabric link may be detected based on receipt of a link update message or transmission of a link update message. At 503, an updated set of next hops is generated for each destination based on the new link state information. The updated set of next hops can be generated using an algorithm such as Fiber Channel Shortest Path First (FSPF). With new information about the network topology, the switch should be able to determine what is the best route to send a frame to a particular destination. At 505, it is determined for each destination whether the next hop set is equal to the updated set of next hops. If it is determined at 507 that all sets of next hops are equal to the updated set of corresponding next hops, no action is taken.

  For example, if the original route for sending a frame to the destination passes through the switches 103 and 107, but no new route for sending the frame to the destination also passes through the switches 103 and 107, no action is required. . However, if the updated paths for sending the frame to the destination are 105 and 107, the updated set of next hops is not equal to the original set of next hops. Blocking and dropping mechanisms can be applied. At 509, for each destination whose next hop set is different from its corresponding updated set of next hops, the queue or virtual output queue for sending frames with that destination is blocked. According to various embodiments, blocking the virtual output queue does not involve sending the frame to the next hop. At 511, the routing table is updated using the updated set of next hops. Queues with changes to the next set of hops can be blocked at 513 during a fabric drain latency or fabric drain period.

  Fabric drain latency allows time for earlier packets that still remain in a congested network switch to be delivered or dropped from the network. A different fabric drain period may be associated with each virtual output queue, or a single fabric drain period may be applied to all virtual output queues. After the fabric drain period has elapsed, the blocked queue is unblocked at 515. Later frames in the blocked queue can now be sent because the previous frame has been dropped or delivered to the destination. Later frames can be transmitted to the destination without the risk of waiting for the previously transmitted frame to arrive at the destination in an out-of-order state in the network.

  The technique shown in FIG. 5 can be applied to any port of the Fiber Channel fabric on a destination basis, per VSAN. However, based on various embodiments, the technique shown in FIG. 5 is applied to an edge port that connects the switch to a host or disk. Techniques for edge ports focus on blocking traffic to avoid injecting traffic that would potentially be dropped.

  FIG. 6 is a process flowchart illustrating the transfer of frames between fiber channel switches. At 601, a change in the fiber channel fabric link is detected. At 603, an updated set of next hops is calculated for each destination. At 605, it is determined for each destination whether the next set of hops is equal to the updated set of next hops. At 607, if all sets of next hops are equal to the updated set of corresponding next hops, no action is taken. If the next set of hops is not equal to the corresponding updated set of next hops, the frame with the destination associated with the changed set of next hops is dropped at 609.

  The routing table can be updated at 611 and the Fiber Channel switch waits at 613 for the corresponding fabric drain period associated with the virtual output queue. After the fabric drain period has elapsed for a particular queue, the frames that are placed in the queue associated with that fabric drain period are now forwarded rather than dropped.

  As described above, the technique of the present invention can be applied to orderly deliver Fiber Channel frames when link changes are detected in the Fiber Channel fabric. Typically, when new nodes or new links are added or old nodes or old links are removed from the Fiber Channel topology, there is a risk that Fiber Channel frames will be delivered out of order.

  However, a change in network topology is not the only event that causes out-of-order delivery. Channel changes between the two switches can also cause out-of-order frame delivery.

  FIG. 7 illustrates channel changes that can cause out-of-order delivery. Switches 107 and 109 may initially be interconnected via redundant links 701, 703 and 705 that form channel 709. Traffic traveling between switches 107 and 109 can be distributed across different links in channel 709 based on factors such as fairness and load balance. In another example, the same hash function can be used at the transmit and receive switches to determine what link to access next. Using the same hash function can provide in-order delivery.

  As new links 707 are added to form the updated channel 711, more links can be used for transmission between the switch 107 and the switch 109. Note, however, that no change in network topology occurs. Switches 107 and 109 are still connected to the fiber channel fabric. According to various embodiments, no changes or updates to the routing table are required when the channel changes. The routing table simply commands the transfer of frames from switch 107 to switch 109. Transfer from switch 107 to switch 109 is not affected by adding a new channel 707 to the original channel 709 to form an updated channel 711.

  However, adding a link to a channel may cause out-of-order delivery of fiber channel frames. Using the example in which frames with sequence numbers 1-6 are transmitted over links 701, 703 or 705, frames with sequence numbers 7 and 8 are transmitted along link 707. Link 707 may be a high-bandwidth link that is not congested or that allows frames with sequence numbers 7 and 8 to leave switch 109 before the arrival of frames 1-6. The technique of the present invention performs fiber channel frame blocking and dropping to allow in-order delivery to switches and nodes downstream of switch 109. Blocking is also referred to herein as delay.

  As described above, even if the channel changes, the link still exists between the switch 107 and the switch 109, so the routing table does not necessarily change. The logic and mechanism for determining what links are available for the channel for transmitting frames is referred to herein as the forwarding channel table. Note that the forwarding channel table can vary. FIG. 8A shows a transfer channel table for the switch 107. For the next hop 109, the entry 803 indicates that the link 701, 703, and 705 can be used to transmit a frame from the switch 107 to the next hop 109.

  FIG. 8B shows the transfer channel table for switch 107 after the channel has changed. After a channel change occurs, entry 813 provides information that links 701, 703, 705, and 707 can be used to send frames from switch 107 to the next hop 109.

  FIG. 9 is a process flowchart illustrating the transfer of frames during channel changes in accordance with various embodiments. At 901, the switch detects a channel change. Adding or removing one or more links causes a change in the channel. At 903, traffic destined for the channel based on the next hop in the routing table is dropped. In one embodiment, traffic that has been directed to transmit over a channel but has not yet been queued is dropped, allowing the forwarding channel table to be modified to accept the channel change. However, traffic that has already been queued is retained. As described above, traffic that has already been queued may reside in a virtual output queue associated with a particular flow. At 905, the queue for transmission along the link in the channel is marked and blocked. In one example, marking and blocking the queue includes allowing transmission of all frames already queued while blocking new frames during the switch drain latency time period.

  The forwarding channel table can then be updated at 907 to add and / or remove links to the channel. At 909, traffic destined for the channel that was already dropped at 903 is now allowed to queue. The switch can then wait at 911 during the changed Fiber Channel switch drain latency. According to various embodiments, the modified Fiber Channel switch drain latency is longer than the standard Fiber Channel switch drain latency, so that all frames already queued can be forwarded rather than dropped. To do. At 913, the queue associated with the link in the channel is released and all frames in the queue are transmitted, or dropped if the frame is too old, and the new frame sequence is now delivered in order. can do.

  While mechanisms such as delaying frames can be used in accordance with various embodiments, the techniques of the present invention are also intended to provide in-order delivery of fiber channel frames using labels. One of the many reasons for using labels is that they provide a fast mechanism for accessing routing table entries. Rather than looking at the address of the destination, the destination identifier is an “input label”, ie, a received label that can be used to quickly reference entries in the routing table. Labels can also be used for various other reasons. The use of labels in a Fiber Channel network is based on the inventor Scott S., who is hereby incorporated by reference in its entirety for all purposes. No. 10 / 114,394 (Attorney Docket No. ANDIP009), filed concurrently with Lee and Dinesh G. Dat, entitled “Label Switching in Fiber Channel Networks”. An example of an architecture for implementing label switching is Multiple Protocol Label Switching (MPLS) described in RFC 3031, which is hereby incorporated by reference in its entirety for all purposes.

  FIG. 10 is a diagram illustrating a fiber channel network including a label switching router that can forward a packet based on a label associated with a frame. In addition to including the destination address, the frame includes an input label, also called a received label, that allows the switch to quickly access the routing table entries as a destination identifier. For example, the label switching router 1004 may receive a frame with 2 destinations and 420 input labels. Note that in FIGS. 10-12, 1-5 destination or next hop refers to switches 1001-1005. The label switching router 1004 accesses its routing table 1014 and confirms that the next hop must be the label switching router 1002 and the “output label” should be 220. The output label is also referred to as a transmission label. According to various embodiments, the label switching router 1004 replaces the 420 frame label value corresponding to the input label of the routing table with the 220 frame label corresponding to the output label of the routing table 1014.

  By replacing the label value, the label switching router 1004 provides label information to the next hop router 1002 to allow the label switching router 1002 to quickly access the routing table entry as well. It should be noted that although label switching can be provided for quick access to routing table entries, label switching can be used for a variety of reasons. The technique of the present invention allows frames to be delivered in order by using labels.

  When the label switching router 1002 receives a frame from the label switching router 1004, the label switching router uses the label 220 to access an entry in the routing table 1012. Using the input label 220, the label switching router 1002 confirms that the frame has arrived at the final hop switch and no longer needs to be forwarded to another switch. The frame can then be transferred to the final destination, which is the host or disk.

  Various techniques can be used to generate a routing table with a label. In one embodiment, the routing table is generated when a link state update packet is received under the FSPF protocol. The routing table can be generated periodically or when a link state change is identified. According to various embodiments, the newly generated routing table is associated with an incarnation number. All incarnation number combinations in the Fiber Channel fabric are referred to herein as topology version numbers. In one embodiment, the incarnation number is incremented by 1 each time a new routing table is generated at the switch. In accordance with various embodiments, each label switching router in the Fiber Channel network not only generates a new forwarding router towards each destination, but each label switching router also has a new input that is different from the previous set of input labels. Labels are also generated.

  FIG. 11 is a diagram illustrating the label switching router during the change of the link state. Here, the link between the label switching router 1004 and the label switching router 1001 can no longer be used. Link state update packets or link state records are propagated throughout the fiber channel fabric. The label switching router 1004 can no longer directly transfer to the label switching router 1001 for transmission from the label switching router 1004 to the label switching router 1001. Rather, the label switching router 1004 transmits the frame toward either the label switching router 1002 or the label switching router 1003. The new router is reflected in the routing table 1114. The label switching router 1004 generates a new routing table 1114 with a new set of next hops, and generates a new input label 411 that replaces the old input label 410.

  According to various embodiments, the label switching router 1004 sends a label switching control message to other label switching routers in the fiber channel fabric to remove the old 410 input label. The label switching control message sent to other label switching routers in the fiber channel fabric includes the topology version number. In one embodiment, the other label switching router verifies that the topology version number of the label switching control message matches the topology version number of the routing table. If the topology version number of the label switching control message does not match the topology version number of the routing table at a particular label switching router, or if the control message contains an old version number, the label switching control message is discarded.

  If the topology version number of the label switching control message is newer than the topology version number of the routing table at a particular label switching router, store the label switching control message and use it later when a new version of the routing table is available can do.

  If the topology version numbers match, the label switching router that receives the label switching control message can remove the old label. For example, if the label switching router 1004 generates a new input label 411 that replaces the old input label 410 and sends a label switching control message to the label switching router 1002 to retrieve the old label, the label switching router 1002 Removes the output label 410 associated with 1 destination ID and 4 next hops. According to various embodiments, a label switching control message for removing old labels is referred to herein as a label retrieval message.

  Output labels are not analyzed, but Fiber Channel frames are dropped to prevent out-of-order delivery of Fiber Channel frames. In FIGS. 11 and 12, “?” Is used to indicate that the label is not analyzed. For example, assume that the label switching router 1005 sends frames 1 and 2 to the label switching router 101 and finally reaches the label switching router 1004. After the link between the label switching router 1001 and the label switching router 1004 fails, the label switching router 1005 transmits frames 3 and 4 to the label switching router 1002 and finally reaches the label switching router 1004. If frames 1 and 2 do not drop, they can reach label switching router 1004 after frames 3 and 4. However, since the output label of the label switching router 1001 for the destination label switching router 1004 is not analyzed after the link fails, frames 1 and 2 are dropped, and frames 3 and 4 arrive at the label switching router 1004 in order. be able to. In other words, the output label 440 associated with destination 4 in the routing table 1111 is no longer accurate, so frames 1 and 2 can be dropped.

  To analyze the output label, the label switching router 1004 advertises the new input label including the input label 411 to another label switching router such as the label switching router 1002. A label switching control message for adding a new label is referred to as a label mapping message. In one example, the label switching router 1004 sends a label mapping message with a label to the label switching router 1002. The label mapping message received by the label switching router 1002 can instruct the label switching router 1002 to use the new input label 411 as the new output label associated with the 4 next hops and 1 destination. .

  FIG. 12 shows a label switching router in a fiber channel fabric where the output labels have not yet been fully analyzed. The label switching router 1004 generates a routing table 1214 that replaces the old set of input labels 410, 420, 430, 440 and 450 with updated stacks of input labels 411, 421, 431, 441 and 451, respectively. . Also, the label switching router 1004 is analyzing its output labels and replacing the output labels 110, 110, 220, 330, 250 and 350 with new output labels 211, 311, 221, 331, 251 and 351. ing. In accordance with various embodiments, the output label is analyzed at the label switching router 1004 after the label switching router 1004 receives a label mapping message with a topology version number corresponding to the routing table 1214 from all other routers. .

  However, the label switching router 1002 does not completely generate the output label in the routing table 1212. The routing table 1212 includes a new set of input labels, but the set of output labels is only partially analyzed. In particular, the label switching router 1002 has a new output label corresponding to the next hop 4, but does not have a new output label corresponding to the next hop 5. This is caused by receiving a label mapping message from the label switching router 1004 but not receiving a label mapping message from the label switching router 1005. When the label switching router 1002 receives a frame for transmission to the next hop 5, the frame is dropped because the output label associated with the next hop 5 is not analyzed.

  Since link state update messages and label switch control messages are passed through the entire fiber channel fabric, each switch eventually becomes the same as label switching router 1004 and label switching router 1001 analyze the output labels in FIG. The output label can be analyzed.

  FIG. 13 is a flow process diagram illustrating frame forwarding using labels. At 1301, a change in the fiber channel fabric link is detected. At 1303, an updated set of next hops for each destination is calculated based on the link state information. Based on various embodiments, a topology version number is identified at 1305. The topology version number is a combination of the incarnation numbers of all switches in the fiber channel fabric. In one embodiment, the topology version number includes all incarnation numbers attached together. In another embodiment, the topology version number is a checksum of various incarnation numbers. In yet another embodiment, the topology version number is a unique number that is meaningful to the switches in the network. At 1307, the switch can send a label switching control message to retrieve the previously advertised label. At 1309, each label switching router can generate a new input or received label and advertise that new input or received label to other switches in the fabric.

  Note that the techniques of the present invention need not be performed in any particular order. For example, in one embodiment, a label switching router can generate and advertise input labels at the same time that output labels are collected at other label switching routers.

  At 1311, the frame with the old label is dropped. Also, the frame is dropped, but the output label is not analyzed. For example, if the output label associated with the next hop 6 is not analyzed, the frame configured to transmit to the next hop 6 is dropped. At 1313, the output label is analyzed by using information from the label mapping message or from advertisements from other label switching routers. Since the output label is analyzed at 1315, the frame is no longer dropped.

  While the invention has been illustrated and described with specific embodiments, those skilled in the art will recognize that changes may be made in the form and details of the embodiments disclosed herein without departing from the spirit and scope of the invention. For example, embodiments of the present invention can be used with various network protocols and architectures. Instructions such as a reject message can be sent at various different times. Accordingly, this invention is to be construed as including all modifications and equivalents that fall within the true spirit and scope of the invention.

FIG. 1 shows a network that can use the techniques of the present invention. It is a figure which shows the fiber channel fabric which receives the change of the link of a fiber channel fabric. It is a figure which shows the routing table which shows the set of the next hop. FIG. 6 shows a routing table showing an updated set of next hops. FIG. 4 is a diagram illustrating a virtual output queue. 2 is a process flowchart illustrating blocking of a Fiber Channel frame. 6 is a process flowchart illustrating fiber channel frame dropping. FIG. 6 illustrates possible reordering in a channel. It is a figure which shows the transfer channel table. It is a figure which shows the transfer channel table updated. FIG. 6 is a process flow diagram illustrating fiber channel frame transfer when a channel changes. It is a figure which shows a label switching router. It is a figure which shows the label switching router during the change of a link. FIG. 6 illustrates a label switching router while output labels are partially analyzed. FIG. 6 is a process flow diagram illustrating a technique for orderly delivery using input and output labels.

Claims (34)

A method for selectively delivering Fiber Channel frames in a Fiber Channel fabric, comprising:
Identifying a next set of hops at the Fiber Channel entity, wherein the next set of hops is used to forward a frame based on a destination identifier;
Detecting a change in a link of the Fiber Channel fabric;
Identifying an updated set of next hops, the updated set of next hops being received at the Fiber Channel entity based on a destination identifier, taking into account changes in the link of the Fiber Channel fabric. Steps such as those used to forward the frame
Comparing the set of next hops with the updated set of next hops;
Prevent forwarding of frames towards the updated set of next hops during a predetermined period of time if it is determined that the set of next hops is different from the updated set of next hops Steps,
With a method.   The method of claim 1, wherein the predetermined time period is a fiber channel fabric drain period. Said step of preventing forwarding of frames towards said updated set of next hops;
3. A method according to claim 1 or 2, comprising blocking frames scheduled to be forwarded to the updated set of next hops.   4. The method of claim 3, wherein the frame is blocked in a virtual output queue associated with the updated set of next hops.   The method of claim 3, further comprising updating a routing table to forward frames to the updated set of next hops.   The method of claim 5, wherein the Fiber Channel entity is a Fiber Channel switch including a plurality of routing tables, each routing table being associated with a particular virtual storage area network.   After the predetermined time period has elapsed and the routing table is updated to schedule a frame for forwarding to the updated set of next hops, a frame to the updated set of next hops The method of claim 5, further comprising the step of allowing transfer of:   The method of claim 7, further comprising dropping a frame held in the switch for longer than a fiber channel switch drain period.   9. The method of claim 8, further comprising determining that the Fiber Channel switch is associated with a non-edge port before preventing forwarding of a frame to the updated set of next hops. Said step of preventing forwarding of frames towards said updated set of next hops;
10. A method as claimed in any preceding claim, comprising dropping a frame scheduled to be forwarded to the updated set of next hops.   The method of claim 10, further comprising updating a routing table to forward frames to the updated set of next hops.   12. The method of claim 11, wherein the Fiber Channel entity is a Fiber Channel host that includes a plurality of routing tables, each routing table associated with a particular virtual storage area network.   After the predetermined time period has elapsed and the routing table has been updated to schedule a frame for forwarding to the updated set of next hops, the frame to the updated set of next hops The method of claim 11, further comprising the step of allowing transfer of:   The method of claim 13, further comprising determining that a Fiber Channel host is associated with an edge port before preventing forwarding of frames to the updated set of next hops. A method for selectively delivering frames in a fiber channel fabric, comprising:
Identifying a channel for transferring a frame from a first fiber channel entity to a second fiber channel entity, the channel connecting the first fiber channel entity to the second fiber channel entity A step that includes a plurality of links to:
Detecting a channel change at the first fiber channel entity;
Identifying an updated channel for transferring a frame from the first fiber channel entity to the second fiber channel entity, wherein the updated channel is different from the channel; And the steps
Dropping frames received for transfer over the updated channel that have not yet been placed in the output queue associated with the updated channel;
With a method.   16. The method of claim 15, further comprising marking the output queue associated with the updated channel link to block all frames after frames already in the output queue have been delivered. .   The method of claim 16, wherein the step of dropping a frame comprises updating a forwarding channel table to forward the frame to a drop neighbor.   The method of claim 16, wherein the output queue is a virtual output queue.   The method of claim 16, further comprising updating a transfer channel table to transfer a frame over the updated channel.   20. The method of claim 19, further comprising waiting for a time period derived using fiber channel switch latency.   The method of claim 19, further comprising waiting for a period of time longer than the Fiber Channel switch latency.   The output queue to allow forwarding frames over the updated channel after the time period has elapsed and the forwarding channel table has been updated to forward frames over the updated channel. 21. The method of claim 20, further comprising the step of releasing.   The method of claim 22, wherein the transport channel table is updated using a hash modulo. A method for selectively delivering frames in a fiber channel fabric, comprising:
Detecting a change in the fiber channel fabric link at the fiber channel switch;
Generating an updated routing table associated with the topology version number, wherein the generation of the updated routing table determines a next hop, receiving label and destination corresponding to each entry in the updated routing table A step that includes:
Receiving a frame at the Fiber Channel switch, wherein the frame includes a first destination and a first label corresponding to a first entry of the updated routing table;
Determining whether the Fiber Channel switch has received a first transmission label corresponding to a first entry of the updated routing table, wherein the received first transmission label is updated; A step having the same topology version number as the routing table
Dropping a frame when it is determined that the Fiber Channel switch has not received the first transmission label;
With a method.   25. The method of claim 24, wherein the first label is equivalent to a first received label associated with the first entry.   26. The method according to claim 24 or 25, further comprising the step of forwarding a frame to the next hop if it is determined that the Fiber Channel switch has received the first transmission label.   27. The method of claim 26, wherein the routing table is associated with a specific virtual storage area network.   28. The method of claim 27, wherein the topology version number is derived using an incarnation number with each switch in the virtual storage area network.   29. The method of claim 28, wherein the reception and transmission labels are MPLS labels.   30. The method of claim 29, wherein the first label is included in an EISL header associated with the fiber channel frame.   The method of claim 30, wherein the first label of the frame is included in the Fiber Channel frame header.   32. The method of claim 25, further comprising receiving a first link state control message having the first transmission label and a topology version number corresponding to the topology version number of the routing table. Method.   36. The method of claim 32, further comprising advertising the first received label to other switches in the Fiber Channel fabric. A Fiber Channel entity for selectively delivering Fiber Channel frames in a Fiber Channel fabric,
Means for identifying a next set of hops at the Fiber Channel entity, wherein the next hop set is used to forward a frame based on a destination identifier;
Means for detecting a change in the fiber channel fabric link;
Means for identifying an updated set of next hops, the updated set of next hops being received at the Fiber Channel entity based on a destination identifier, taking into account changes in the Fiber Channel fabric link Means such as those used to forward the generated frames;
Means for comparing the set of next hops with the updated set of next hops;
Prevent forwarding of frames towards the updated set of next hops for a predetermined period of time if it is determined that the set of next hops is different from the updated set of next hops. Means to
Fiber Channel entity with

Images