WO2021209131A1

WO2021209131A1 - A system and method for handling out of order delivery transactions

Info

Publication number: WO2021209131A1
Application number: PCT/EP2020/060659
Authority: WO
Inventors: Ben-Shahar BELKAR; Reuven Cohen; Lior Khermosh; Guy Shattah; Huichun QU; Shengwen Lu; Tal Mizrahi
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2020-04-16
Filing date: 2020-04-16
Publication date: 2021-10-21
Also published as: CN114531907A

Abstract

A device and method for transmitting a plurality of transactions are disclosed. In the disclosure, the device is configured to: in each of a plurality of packet-based transactions of an operation encoded according to a network transport protocol, identify an index indication of delivery contained in a host data (601), incorporate the index indication of delivery in each of the plurality of packets of the respective packet-based transaction (602) and instruct transmitting the plurality of packets incorporated with the index indication of delivery (603).

Description

A SYSTEM AND METHOD FOR HANDLING OUT OF ORDER DELIVERY

TRANSACTIONS

TECHNICAL FIELD

The present disclosure, in some embodiments thereof, relates to communication systems and, more specifically, but not exclusively, to a system and method for handling out of order delivery transactions.

BACKGROUND

Remote Direct Memory Access (RDMA) is a direct memory access from a memory of one computer into that of another computer without involving either one of the computers operating system. The RDMA approach permits high-throughput and low- latency networking, which is especially useful in massively parallel computer clusters.

RDMA over Converged Ethernet (RoCE) is a standard protocol, which enables efficient data transfer of RDMA over Ethernet networks allowing transport offload with hardware RDMA engine implementation, and superior performance. RoCE is a standard protocol defined in the InfiniBand Trade Association (IBTA) standard. RoCE makes use of User Datagram Protocol (UDP) encapsulation allowing it to transcend Layer 3 networks. RDMA is a key capability natively used by the InfiniBand interconnect technology. Both InfiniBand and Ethernet RoCE share a common user Application Programming Interface (API) but have different physical and link layers.

SUMMARY

It is an object of the present disclosure to provide a device for transmitting a plurality of transactions, a device for receiving a plurality of out of order transactions, a method for transmitting a plurality of transactions and a method for receiving a plurality of out of order transactions.

The foregoing and other objects are achieved by the features of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures.

According to a first aspect of the present disclosure, a device for transmitting a plurality of transactions is disclosed. The device is configured to: in each of a plurality of packet-based transactions of an operation encoded according to a network transport protocol: identify an index indication of delivery contained in a host data; incorporate the index indication of delivery in each of a plurality of packets of the respective packet-based transaction; and instruct transmitting the plurality of packets incorporated with the index indication of delivery.

According to a second aspect of the present disclosure, a device for receiving a plurality of transactions is disclosed. The device is configured to receive a plurality of packet-based transactions of an operation encoded according to a network transport protocol, wherein an index indication of delivery is incorporated into each of a plurality of packets of the respective packet-based transaction; and manage processing of out of order packets from the plurality of packet-based transactions according to the index indication.

According to a third aspect of the present disclosure, a method for transmitting transactions is disclosed. The method comprises: in each of a plurality of packet-based transactions of an operation encoded according to a network transport protocol: identifying an index indication of delivery contained in a host data; incorporating the index indication into a plurality of packets of the respective packet- based transaction; and instructing transmitting the plurality of packets incorporated with the index indication.

According to a fourth aspect of the present disclosure, a method for receiving transmitted transactions is disclosed. The method comprises receiving a plurality of packet- based transactions of an operation encoded according to a network transport protocol, wherein an index indication of delivery is incorporated into each of a plurality of packets of the respective packet-based transaction; and managing processing of out of order packets from the plurality of packet-based transactions according to the index indication.

In a further implementation of the first aspect, the index indication is based on at least one of the following: a host internal queue index, an added specific transaction type counter, a specific packet-based transaction indication, an absolute number, and an infinite running index indication.

In a further implementation of the first aspect, the index indication is incorporated into one of the following: a plurality of payloads of the plurality of packets of the respective packet- based transaction; one packet of the respective packet-based transaction; and at least one header of the plurality of packets of the respective packet-based transaction. This way the index indication may be incorporated in any available place in the packet.

In a further implementation of the first aspect, in each of the plurality of packet-based transactions, the device is configured to incorporate the index indication by overwriting unused fields in the at least one header of the plurality of packets of the respective packet-based transaction.

In a further implementation of the first aspect, in each of the plurality of packet-based transactions, the device is configured to incorporate the index indication in dedicated fields intended for the purpose of transaction information in the at least one header of each of the plurality of packets of the respective packet-based transaction.

In a further implementation of the first aspect, in each of the plurality of packet-based transactions the device is further configured to receive an acknowledgment, ACK, for a successful transmission of a first packet of the respective packet-based transaction of the operation and to stop incorporate the index indication in to the plurality of packets of the respective packet-based transaction of the operation, in response to the received ACK. Thereby, saving the computational power of incorporating the index indication into each packet.

In a further implementation of the first aspect, in each of the plurality of packet-based transactions the device is further configured to receive an acknowledgment, ACK, for a successful transmission of a first packet of the respective packet-based transaction of the operation, and to continue incorporate the index indication in to the plurality of packets of the respective packet-based transaction of the operation, in response to the received ACK.

In a further implementation of the first aspect, the index indication is relative to a last time an acknowledgment, ACK, was received for the index indication. In this implementation, the index indication starts over the count of the transactions after the last transaction that was acknowledged. This way, the lowest possible number for representing the index indication is used and therefore when sending a new transaction, less bits are used on the header or payload in which the index indication is incorporated.

In a further implementation of the second aspect, the device id further configured to: process at least one out of order packet from the plurality of packets of the packet-based transactions according to the index indication; or create a queue, which corresponds to an order of delivery of the packet-based transactions of the operation by generating an element for each of the packet-based transactions of the operation, at a first time a packet from the respective packet-based transaction of the operation is received; and remove the element of each of the packet-based transactions of the operation from the queue, when the respective packet-based transaction of the operation is completed. Thereby, each transaction id identified by the index indication of the transaction and the device knows to what transaction each received packet belongs. In this way, an out of order processing can be carried out at the device receiving the packets. In a further implementation of the fourth aspect, the step of managing comprises: processing at least one out of order packet from the plurality of packets of the packet- based transactions according to the index indication; or creating a queue, which corresponds to an order of delivery of the packet-based transactions of the operation by generating an element for each of the packet-based transactions of the operation, at a first time a packet from the respective packet-based transaction of the operation is received; and removing the element of each of the packet-based transactions of the operation from the queue, when the respective packet-based transaction of the operation is completed.

In a fifth aspect, the present disclosure relates to a computer program product comprising computer readable code instructions which, when run in a computer will cause the computer to perform the method according to any one of the third and fourth aspect of the disclosure and their implementations.

Other systems, methods, features, and advantages of the present disclosure will be or become apparent to one with skill in the art upon examination of the following drawings and detailed description. It is intended that all such additional systems, methods, features, and advantages be included within this description, be within the scope of the present disclosure, and be protected by the accompanying claims.

Unless otherwise defined, all technical and/or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which embodiments. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments, exemplary methods and/or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGfS)

Some embodiments are herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of embodiments. In this regard, the description taken with the drawings makes apparent to those skilled in the art how embodiments may be practiced.

In the drawings: FIG. 1 schematically shows an example of the problem of out of order arrival of transactions to a receiver;

FIG. 2 schematically shows an apparatus for transmitting and receiving a plurality of packet based transaction, where some of the packet-based transaction are received out of the order of the transactions, according to some embodiments of the present disclosure;

FIG. 3 schematically shows an example of an out of order reception of packet-based transactions, according to some embodiments of the present disclosure;

FIG. 4 schematically shows another example of an out of order reception of packet- based transactions, according to some embodiments of the present disclosure;

FIG. 5 schematically shows an example of an out of order reception of packet-based transactions of three RDMA send operations, according to some embodiments of the present disclosure;

FIG. 6 schematically shows a flowchart of a method for handling transmission of a plurality of packet-based transactions from a sender to a receiver, which are received out of order at the receiver, according to some embodiments of the present disclosure; and

FIG. 7 schematically shows a flowchart of a method for handling reception of a plurality of packet-based transactions, which are received out of order at the receiver, according to some embodiments of the present disclosure.

DETAILED DESCRIPTION

Some embodiments described in the present disclosure relate to a communication system and, more specifically, but not exclusively, to systems and methods for handling out of order delivery transactions.

Remote Direct Memory Access (RDMA) requires an underlying reliable transport. Data packets are transmitted from a sender (also referred to as a transmitter) to a receiver, and are retransmitted when necessary, for example, in case a packet was lost and did not get to the destination.

RDMA over converged Ethernet (RoCE) is a network protocol that allows remote direct memory access (RDMA) over an Ethernet network. There are two versions of RoCE protocol, the first, RoCEvlalso referred to as non-routable RoCE, allows communication between two hosts in the same Ethernet broadcast domain. The second, RoCEv2 is a protocol, in which packets are transported over User Datagram Protocol (UDP), which mean packets can be routed. RoCEv2 is also referred to as routable RoCE. Reliability (i.e. retransmission of non- received packets or incorrectly received packets) is provided at the RDMA layer. The data traffic in RDMA consists of memory transactions. Correct behavior of the RoCE protocol requires the receiver to be able to know the correct order of the transactions. The receiver may be able to process received packets even if they are received out-of-order within a transaction. However, in case packets are received out of the order of the transactions, the receiver cannot process the packets, as the receiver does not know to what transaction the packets belong. For example, when a transaction of an inbound RDMA send operation is received in the receiver, the receiver places the received transaction in a Work Queue Element (WQE) index in a receive queue (RQ). When the transaction of an inbound RDMA send operation is received out of order, the receiver cannot know in which WQE index in the receive queue should this transaction be placed.

FIG. 1 schematically shows an example of the problem of out of order arrival of transactions to the receiver. In this example, a sender 101 sends three transactions to a receiver 102 in the following order: transaction A, transaction B, transaction C. Transaction A includes packets 1, 2, 3. Transaction B includes packets 4, 5, 6, and transaction C includes packets 7 and 8. The transactions are not received at the same order they were delivered. The packets within the transactions are out of order. Transaction A is received first with the following order of packets: 1, 3, 2. Since all the packets 1, 2 and 3 belongs to transaction A, the receiver is able to process all the packets. Transaction B is received second with the following order of packets: 4, 5 and 6, where before packet 6, the receiver receives transaction C, which includes packet 7 and packet 8. Although transaction C, which includes packets 7 and 8, is received before transaction B is completed, packets 7 and 8 cannot be processed since the transaction is out of order of the transactions. Only after transaction B, which is the second transaction, is completed and packet 6 is received, the receiver can process the third transaction, with packets 7 and 8. In many cases when the receiver cannot process packets the packets are discarded and then the sender has to send the discarded packets again.

According to some embodiments of the present disclosure, an apparatus and a method are provided for assuring the receiver knows the correct order of delivery of transactions. Especially, RDMA transaction over RoCE in all of RoCE possible versions. According to some embodiments of the present disclosure the apparatus and method of the disclosure detect in each transaction an index indication based on a host internal queue index of delivery (herein after index indication) contained in the host software and incorporate the index indication into all the packets of the respective transaction. Thereby, when the receiver receives a packet out of order, the receiver is aware of the correct order of delivery of the packet and of the transaction and is able to continue with an out of order processing for the received packet. Before explaining at least one embodiment in detail, it is to be understood that embodiments are not necessarily limited in its application to the details of construction and the arrangement of the components and/or methods set forth in the following description and/or illustrated in the drawings and/or the Examples. Implementations described herein are capable of other embodiments or of being practiced or carried out in various ways.

Embodiments may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the embodiments.

The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.

Computer readable program instructions for carrying out operations of embodiments may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of embodiments.

Aspects of embodiments are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the fimctions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

Reference is now made to FIG. 2, which schematically shows an apparatus for transmitting and receiving a plurality of packet based transaction, where some of the packet- based transaction are received out of the order of the transactions, according to some embodiments of the present disclosure. Apparatus 200 comprises a sender 210 and a receiver 220. The sender 210 includes a processor 211, a memory 212 and a host software 213. Receiver 220 includes a processor 221, a memory 222 and a host software 223. The processor 211 of the sender identifies in each transaction of an operation a host index indication of delivery, which is contained in the host software 213. The transaction is a packet-based transaction and includes a plurality of packets. An operation may be for example a read operation, a write operation, a send operation and the like. The processor 211 incorporates the index indication in each one of the plurality of packets of the transaction of the operation and instructs transmitting a plurality of packet-based transactions of the operation incorporated with the index indication, over a network transport protocol to the receiver 220. The processor 211 may be in the sender, on a network device as a dedicated hardware block or as a processing unit or the processor may be on the host software.

According to some embodiments of the present disclosure, the index indication, which is based on a host internal queue index, is a Work Queue Element (WQE). According to some other embodiments of the present disclosure, the index indication is a specific transaction type counter. The specific transaction type counter may be implemented in firmware, hardware or software. The specific transaction type counter may be implemented for example as an opcode counter inside a network device.

According to some embodiments of the present disclosure, the index indication is incorporated into a plurality of payloads of the plurality of packets of the respective packet- based transaction. Alternatively, the index indication is incorporated into at least one header of each of the plurality of packets of the respective packet-based transaction.

According to some embodiments of the present disclosure, the index indication may also be incorporated into each of the headers of each packet. In some embodiments of the disclosure, in each of the plurality of packet-based transactions, the processor 211 of the sender incorporates the index indication by overwriting unused fields in the header of the plurality of packets of the respective packet-based transaction. According to some embodiments of the present disclosure, in each of the plurality of packet-based transactions, the sender incorporates the index indication in dedicated fields intended for the purpose of transaction information in the at least one header of each of the plurality of packets of the respective packet-based transaction.

The receiver 220 receives the transmitted packet-based transactions of an operation encoded according to a network transport protocol with the index indication of delivery incorporated in each of a plurality of packets of the respective packet-based transaction. The receiver then manages the processing of out of order packets from the plurality of packet-based transactions according to the index indication. In some embodiments of the present disclosure the processor 221 of the receiver 220, manages of the processing of out of order packets from the plurality of packet-based transactions. The processor 221 may be in the sender, on a network device as a dedicated hardware block or as a processing unit or the processor may be on the host software. The processor 221 processes at least one out of order packet from the plurality of packets of the packet-based transactions according to the index indication, in case the packet is not out of the order of the transactions. Alternatively, in case where a packet is received out of the order of the transaction, processor 221 of receiver 220 handles a special receiving queue, which corresponds to the order of delivery of the transactions of the operation. The processor generates an element for each of the packet-based transactions of the operation, at a first time a packet from the respective packet-based transaction of the operation is received, and stores the element in the memory 222 of the receiver 220. When the respective packet-based transaction of the operation is completed, processor 221 removes the element of each of the packet-based transactions of the operation from the special receiving queue.

Optionally, the index indication of the present disclosure is based on a host internal queue index or on an added specific transaction type counter.

According to some embodiments of the present disclosure, the index indication may be an absolute number or a relative number and /or an infinite running index indication.

According to some embodiments of the present disclosure, in each of the plurality of packet-based transactions the sender receives an acknowledgment, ACK, from the receiver, for a successful transmission of a first packet of the respective packet-based transaction of the operation. In response to the ACK, the sender may stop incorporating the index indication in to the plurality of packets of the respective packet-based transaction of the operation, since the first packet of a transaction usually comprises the addresses for the placement of each packet of the transaction. According to some other embodiments of the present disclosure, the sender may continue incorporating the index indication into the plurality of packets of the respective packet-based transaction of the operation, in response to the received ACK. That is, since not always the information in the first packet of a transaction is sufficient for placing all the received packets, which are received, out of order. For example, in case the receiver receives three first packets of three transactions of an operation out of six transactions of the operation that have been transmitted. The receiver does not know how many transactions are missing accept for the three transactions of the three first packets that have been received in the receiver.

According to some embodiments of the present disclosure, the index indication may be relative to a last time an ACK was received for the index indication. This means that when the sender receives an ACK for all the packets of a specific transaction so that all the packets of the specific transaction are accepted at the receiver, the sender may start the count of the index indication from the beginning. Thereby, the sender uses the lowest possible number for the index indication, and consumes less bits on the header or payload when the packet is sent. In this case where the index indication is relative, each time or every few times the sender receives an ACK that all the packets of a specific transaction were successfully processed in the receiver side, the sender starts over the count of the index indication of the transactions. In this case, the host data holds a number, which indicates the total number of transactions and which is different from the index indication, which started the count of transaction from the beginning after the last ACK was received. In case the index indication is infinite, according to some embodiments of the present disclosure, the index indication is limited to the maximum number that can be represented by the bits available for incorporating the index indication in the packets. In this case, once the index indication reaches the maximum number that can be represented, the index indication restart the count.

FIG. 3 schematically shows an example of an out of order reception of packet-based transactions, according to some embodiments of the present disclosure. In this example, sender 310 transmits three transactions, transaction A3 transaction B3 and transaction C3 to receiver 320.

Transaction A3 is transmitted first and includes packets 301, 302 and 303. The processor of the sender identifies an index indication in the host software of the sender and incorporates the index indication in packets 301, 302, 303 of transaction A3. Since A3 is the first transaction, the index indication of transaction A3 may be for example the absolute number 1, which is incorporated into all the packets of transaction A3. Alternatively, the index indication may be a relative number for example x+1, where x is a predefined number. The receiver 320 receives packet 301 first, and the processor of the receiver generates an element denoted as “1”, which corresponds to transaction A3, as taken from packet 301. In the element “1” all the packets, which belongs to transaction A3 are stored. After packet 301 is received, the receiver receives packet 303. Since packet 303 contains the same index indication of packet 301, the processor of the receiver stores packet 303 at element “1”. Then the receiver receives packet 302. As packet 302 contains the same index indication of packets 301 and 303, the receiver stores packet 302 at element “1”. Once transaction A3 is completed and all the packets in the transaction are received, element “1” is removed from the special receiving queue of the receiver.

The second transaction, transaction B3 contains packets 304, 305 and 306. At the sender, the index indication of transaction B3 is identified at the host software and is incorporated in each of the packets of transaction B3. Since transaction B3 is the second transaction, the index indication may be the absolute number 2, or the relative number x+2, where x is a predefined number. The packets of transaction B3 are transmitted in the following order: 304, 305 and 306. Receiver 320 receives first packet 304. As packet 304 is the first time a packet from transaction B3 is received at the receiver, the processor of the receiver generates a new element denoted as “2” in the special receiving queue of the receiver as taken from packet 304, which corresponds to transaction B3. In the element “2”, all the packets which belongs to transaction B3 and which contains the same index indication of transaction B3 are stored. The receiver then receives packet 305, which contains the same index indication as packet 304. The processor of the receiver stores packet 305 at element “2”. Then the receiver receives packet 307, which does not belong to the transaction B3. Packet 307 belongs to a new transaction C3 and therefore packet 307 contains an index indication different from the index indication of transactions A3 and B3. Since transaction C3 is the third transaction, the index indication may be the absolute number 3, or the relative number x+3, where x is a predefined number. Therefore, the processor of the receiver generates a new element denoted as “3”, as taken from packet 307, which corresponds to transaction C3. Once element “3” is generated the special receiving queue contains two elements: “2” and “3”. After packet 307, the receiver receives packet 306 which is the last packet of transaction B3 and which contains the index indication of transaction B3. The processor of the receiver stores packet 306 in element “2”. At this stage, since all the packets of transaction B3 have been received, processor of receiver 320 removes element “2” from the special receiving queue. Then, the receiver receives packet 308, which belongs to transaction C3 and therefore, contains the same index indication as packet 307. The processor of the receiver 320 stores packet 308 at element “3” and since all the packets of transaction C3 have been received the receiver removes element “3” from the special receiving queue.

FIG. 4 schematically shows another example of an out of order reception of packet- based transactions, according to some embodiments of the present disclosure. In this example, sender 410 transmits three transactions to receiver 420: transaction A4 transaction B4 and transaction C4.

Transaction A4 is transmitted first and includes packets 401, 402 and 403. The processor of the sender 410 identifies an index indication in the host software of the sender and incorporates the index indication into packets 401, 402, 403 of transaction A4. The receiver 420 receives packet 401 first, and the processor of the receiver generates an element denoted as “1”, which corresponds to transaction A4. In the element “1”, all the packets, which belongs to transaction A4 are stored. After packet 401 is received, the receiver receives packet 403. Since packet 403 contains the same index indication of packet 401, the processor of the receiver 420 stores packet 403 at element “1”. Then the receiver receives packet 402. As packet 402 contains the same index indication of packets 401 and 403, the processor of the receiver stores packet 402 at element “1”. Once transaction A4 is completed and all the packets of the transaction A4 are received and stored in element “1”, element “1” is removed from the special receiving queue of the receiver.

The second transaction, transaction B4 contains packets 404, 405 and 406. At the sender, the index indication of transaction B4 is identified at the host software and is incorporated into each of the packets of transaction B4. The packets of transaction B4 are transmitted in the following order: 404, 405 and 406. Receiver 320 receives first packet 405. As packet 405 belongs to a new transaction and contains a new index indication of transaction B4, the processor of the receiver 420 generates a new element denoted “2” at the special receiving queue of the receiver, which corresponds to the index indication of transaction B4. Then, the receiver receives packet 406, which contains the same index indication of packet 405. The processor of the receiver stores packet 406 in element “2” in the special receiving queue. Then, the receiver receives packet 407, which is a packet from transaction C4. Since 407 is a packet of a new transaction, packet 407 contains a new index indication, which is different from the index indication of transactions A4 and B4. Therefore the processor of the receiver 420 generates a new element denoted “3”, in the special receiving queue. Packet 407 is stored into element “3” and packet 408, which is received at the receiver 420 after packet 407 is also stored in element “3”, as packet 408 belongs to transaction C4. Since transaction C4 only contains packet 407 and 408, once all the packets of the transaction are received and stored, element “3” is removed from the special receiving queue. After element “3” is removed packet 404, is received at the receiver. Since packet 404 belongs to transaction B4, which already has a corresponding element “2”, the packet 404 is stored in element “2”. Since packet 404 is the last packet of transaction B4, and all the packets of transaction B4 have been received, element “2” is removed from the special receiving queue of the receiver 420.

Optionally, the index indication is incorporated into one packet of the respective packet- based transaction.

According to some embodiments of the present disclosure, the plurality of transactions are Infmiband based Remote Direct Memory Address (RDMA) protocols, RDMA over converges Ethernet (RoCE) vl protocol and RoCE v2 protocol. The transactions may be for tagged operations and /or untagged operations. For example, the transaction may be for a send operation, which is an untagged operation or for a read operation, which is a tagged operation.

FIG. 5 schematically shows an example of an out of order reception of packet-based transactions of three RDMA send operations, according to some embodiments of the present disclosure. In the receiver 520, a producer generates n WQEs (indexes), where a consumer points to the next expected transaction, which the receiver expects to receive. Therefore, before the receiver receives the transactions the consumer points to WQE index 1. When the receiver 520 receives the first packet 501 of transaction A5, which is a transaction of an RDMA send operation, the receiver reads the index indication incorporated in packet 501 and sees that packet 501 contains an index indication of “1”. The receiver therefore takes the “1” WQE element from the receive queue (RQ) and process packet 501 straight to a buffer that WQE element “1” represents. Element “1” is taken from the index indication of packet 501 and corresponds to the index indication of transaction A5 of the first RDMA send operation. When the receiver receives the first transaction (which means any packet of the first transaction), the consumer in the receive queue (RQ) points to the next expected transaction. In this case, the consumer points to WQE index 2. Then, after transaction A5 is successfully completed, the second transaction of RDMA send operation is not received as expected by the consumer. Instead, the third RDMA send operation is received, with packets 507 and 508. Therefore, the processor of the receiver 520 generated an element “3” as taken from packet 507 index indication field (the payload or at least one of the headers), which corresponds to transaction C5, the receiver therefor takes the “3” WQE element from the receive queue and process packet 507 straight to the buffer that WQE element “3” represents. The consumer then jumps to index 4, which is the next expected index after index 3. Then, transaction B5, which is the second RDMA send operation, is received at the receiver, with the following order of packets: 506, 505 and 504. In this case, the receiver reads the index indication incorporated in packet 506 and sees that packet 506 contains an index indication of “2”. The receiver therefore takes the “2” WQE element from the receive queue (RQ) and process packet 506 straight to a buffer that WQE element “2” represents. Element “2” is taken from the index indication of packet 506 and corresponds to the index indication of transaction B5 of the second RDMA send operation. However, the consumer still points to index 4, in the receive queue (RQ), as 4 is still the next expected index after transaction B5 is completed.

FIG. 6 schematically shows a flowchart of a method for handling transmission of a plurality of packet-based transactions from a sender to a receiver, which are received out of order at the receiver, according to some embodiments of the present disclosure. At 601, in each of a plurality of packet-based transactions of an operation encoded according to a network transport protocol, an index indication of delivery, contained in a host data of a host software is identified by the processor of the sender. At 602, the identified index indication of delivery is incorporated into each of a plurality of packets of the respective transaction. The index indication may be based on a host internal queue index, for example, a Work Queue Element (WQE), or it may be based on a specific transaction type counter. The index indication may be an absolute number, a relative number and/or an infinite running index. In some embodiments of the present disclosure, the index indication is incorporated into at least one of the headers of each packet or into the payload of each packet. The index indication is incorporated into the at least one header by overwriting unused fields in the at least one header of the plurality of packets of the respective packet-based transaction. In some other embodiments of the present disclosure, the index indication is incorporated into the at least one header of each packet in dedicated fields intended for the purpose of transaction information. Optionally the index indication may also be incorporated into one packet of the respective transaction.

At 603, an instruction for transmitting the plurality of packets incorporated with the index indication of delivery is given by the processor of the sender.

FIG. 7 schematically shows a flowchart of a method for handling reception of a plurality of packet-based transactions, which are received out of order at the receiver, according to some embodiments of the present disclosure. At 701, a plurality of packet-based transactions of an operation encoded according to a network transport protocol, with the index indication of delivery incorporated into each of a plurality of packets of the respective packet-based transaction is received at the receiver.

At 702, the processing of out of order packets from the plurality of packet-based transactions according to the index indication of delivery is managed by the processor of the receiver. According to some embodiments of the present disclosure, in case the out of order packets are out of the order of the transaction, a special receiving queue, which corresponds to the order of delivery of the packet-based transactions of the operation, is created. In the special receiving queue, an element is generated, for each of the packet-based transactions of the operation, at a first time a packet from the respective packet-based transaction of the operation is received. When the respective packet-based transaction of the operation is completed, the element is removed from the queue. According to some embodiments of the present disclosure, in case the out of order packets are not out of the order of the transaction, the packets are processed out of order according to the index indication.

According to some embodiments of the present disclosure, an ACK is sent from the receiver to the sender for a successful transmission of a first packet of the respective packet- based transaction of the operation. In response, according to some embodiments of the present disclosure, the incorporation of the index indication into each packets may stop. According to some other embodiments of the present disclosure, the incorporation of the index indication into each packets may continue, in response to the received ACK.

According to some other embodiments of the present disclosure, the index indication is relative to a last time an ACK was received for the index indication.

The methods of transmission and reception of out of order delivery transactions, according to some embodiments of the present disclosure may indicate of packet loss or out- of-order delivery of the first packet of the transaction. In RDMA and/or RoCE only the first packet of a transaction provides context, and includes the memory address in which the transaction takes place. Therefore, the methods of transmission and reception of out of order delivery transactions of the present disclosure may be with a high importance for RDMA implementations, for enabling the storage of packets of a transaction in the element generated in the special receiving queue even before the first packet of the transaction is received.

According to some embodiments of the present disclosure, a computer program product is provided. The computer program product comprises computer readable code instructions which, when run in a computer causes the computer to perform the methods described above in FIGs 6 and 7.

The descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

It is expected that during the life of a patent maturing from this application many relevant devices and methods for handling out of order delivery transactions will be developed and the scope of the term devices and methods for handling out of order delivery transactions is intended to include all such new technologies a priori.

As used herein the term “about” refers to ± 10 %.

The terms "comprises", "comprising", "includes", "including", “having” and their conjugates mean "including but not limited to". This term encompasses the terms "consisting of' and "consisting essentially of'.

The phrase "consisting essentially of means that the composition or method may include additional ingredients and/or steps, but only if the additional ingredients and/or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

As used herein, the singular form "a", "an" and "the" include plural references unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" may include a plurality of compounds, including mixtures thereof.

The word “exemplary” is used herein to mean “serving as an example, instance or illustration”. Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and/or to exclude the incorporation of features from other embodiments. The word “optionally” is used herein to mean “is provided in some embodiments and not provided in other embodiments”. Any particular embodiment may include a plurality of “optional” features unless such features conflict.

Throughout this application, various embodiments may be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of embodiments. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

Whenever a numerical range is indicated herein, it is meant to include any cited numeral (fractional or integral) within the indicated range. The phrases “ranging/ranges between” a first indicate number and a second indicate number and “ranging/ranges from” a first indicate number “to” a second indicate number are used herein interchangeably and are meant to include the first and second indicated numbers and all the fractional and integral numerals therebetween.

It is appreciated that certain features of embodiments, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of embodiments, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination or as suitable in any other described embodiment. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperative without those elements.

Although embodiments have been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

All publications, patents and patent applications mentioned in this specification are herein incorporated in their entirety by reference into the specification, to the same extent as if each individual publication, patent or patent application was specifically and individually indicated to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to embodiments. To the extent that section headings are used, they should not be construed as necessarily limiting.

Claims

1. A device (210) for transmitting a plurality of transactions, the device being configured to: in each of a plurality of packet-based transactions of an operation encoded according to a network transport protocol: identify an index indication of delivery contained in a host data; incorporate the index indication of delivery in each of a plurality of packets of the respective packet-based transaction; and instruct transmitting the plurality of packets incorporated with the index indication of delivery.

2. The device (210) of claim 1, wherein the index indication is based on at least one of the following: a host internal queue index, an added specific transaction type counter, a specific packet-based transaction indication, an absolute number, and an infinite running index indication.

3. The device (210) of claim 1, wherein the index indication is incorporated into one of the following: a plurality of payloads of the plurality of packets of the respective packet-based transaction; one packet of the respective packet-based transaction; and at least one header of the plurality of packets of the respective packet-based transaction.

4. The device (210) of claim 3, wherein in each of the plurality of packet-based transactions, the device is configured to incorporate the index indication by overwriting unused fields in the at least one header of the plurality of packets of the respective packet-based transaction.

5. The device (210) of claim 3, wherein in each of the plurality of packet-based transactions, the device is configured to incorporate the index indication in dedicated fields intended for the purpose of transaction information in the at least one of header layers of each of the plurality of packets of the respective packet-based transaction.

6. The device (210) of claim 1, wherein in each of the plurality of packet-based transactions the device is further configured to receive an acknowledgment, ACK, for a successful transmission of a first packet of the respective packet-based transaction of the operation and to stop incorporate the index indication in to the plurality of packets of the respective packet-based transaction of the operation, in response to the received ACK.

7. The device (210) of claim 1, wherein in each of the plurality of packet-based transactions the device is further configured to receive an acknowledgment, ACK, for a successful transmission of a first packet of the respective packet-based transaction of the operation, and to continue incorporate the index indication in to the plurality of packets of the respective packet-based transaction of the operation, in response to the received ACK.

8. The device (210) of claim 1, wherein the index indication is relative to a last time an acknowledgment, ACK, was received for the index indication.

9. A device (220) for receiving a plurality of transactions, configured to: receive a plurality of packet-based transactions of an operation encoded according to a network transport protocol, wherein an index indication of delivery is incorporated into each of a plurality of packets of the respective packet-based transaction; manage processing of out of order packets from the plurality of packet-based transactions according to the index indication.

10. The device (220) of claim 9, wherein the device further configured to: process at least one out of order packet from the plurality of packets of the packet-based transactions according to the index indication; or create a queue, which corresponds to an order of delivery of the packet-based transactions of the operation by generating an element for each of the packet-based transactions of the operation, at a first time a packet from the respective packet-based transaction of the operation is received; and remove the element of each of the packet-based transactions of the operation from the queue, when the respective packet-based transaction of the operation is completed.

11. A method for transmitting transactions, comprising: in each of a plurality of packet-based transactions of an operation encoded according to a network transport protocol: identifying an index indication of delivery contained in a host data; incorporating the index indication into a plurality of packets of the respective packet-based transaction; and instructing transmitting the plurality of packets incorporated with the index indication.

12. A method for receiving transmitted transactions, comprising: receiving a plurality of packet-based transactions of an operation encoded according to a network transport protocol, wherein an index indication of delivery is incorporated into each of a plurality of packets of the respective packet-based transaction; managing processing of out of order packets from the plurality of packet-based transactions according to the index indication.

13. The method of claim 12, wherein the step of managing comprises: processing at least one out of order packet from the plurality of packets of the packet- based transactions according to the index indication; or creating a queue, which corresponds to an order of delivery of the packet-based transactions of the operation by generating an element for each of the packet-based transactions of the operation, at a first time a packet from the respective packet-based transaction of the operation is received; and removing the element of each of the packet-based transactions of the operation from the queue, when the respective packet-based transaction of the operation is completed.

14. A computer program product comprising computer readable code instructions which, when run in a computer will cause the computer to perform the method according to any one of claims 11 to 13.