GB2460217A - Storing data segments using linked lists - Google Patents

Abstract

The invention relates to a system for storing a plurality of data segments using head and tail pointers. The system includes a queuing engine which is used to progress the head pointer for a specific queue through the memory space in accordance with write requests, and a dequeuing engine which progresses the tail pointer in accordance with a size parameter that indicates a memory space that can be read. Control logic maintains a list of writes to a queue, each with a sequence number and an indication of the location of the head pointer, for the purpose of calculating the size of a permitted read. This size may be updated provided that the given write and all the preceding writes are complete.

Description

Controlling a read order of asynchronously written data segments in the same queue

Field of the invention

This invention relates to data storage and particularly to the storage of data segments from a multiplicity of sources in queues awaiting dispatch, for example from ports of a network device.

Background to the invention

It is customary in the processing of data segments to store the segments in queues.

How the data segments are allotted to the queues depends on many factors such as the number of sources or others which are at the choice of the designer or determined by the various protocols that may be encountered, factors related to quality of service (QOS) and other factors and the manner of allotment is not directly relevant to the invention.

The term "data segment" used herein is intended to embrace but is not limited to control entry' which, as explained in for example our prior co-pending European patent application No. 08250403.6, denotes at least one packet but does not include a payload.

It is customary to control the reading and writing of data segments in a memory space such as a FIFO by means of pointers, the write pointer indicating where the next entry shall be written and the read pointer indicating the location for the next read operation. The pointers are controlled to progress through the memory space, usually in a recycling sequence. The initiation of a read operation is usually a non-empty queue, i.e. effective non-zero memory space between the locations indicated by the write and read pointers.

The present invention is particularly concerned with the problem that arises if multiple write controllers such as write buffers, are writing, asynchronously, to a single queue. This problem may occur for a variety of reasons, but principally arises in the circumstance of a fat pipe' when many or all the sources of entries to the memory are destined for one queue. If several controllers service the same queue the order of completion of the write operations may not be the same as the order in which the write operations are initiated. Consequently, a read operation may be performed for a location to which data in the respective segment has not yet been S written.

A further problem may arise when multiple controllers service a single queue, a write operation is in progress from a first controller and a second write operation is initiated before the first write operation finishes. The manager may then pass the same head pointer for the second write as for the first write, because head pointers are normally updated at the end of a write.

Summary of the invention

The present invention is based on the concept of maintaining a list of writes to a queue each with a sequence number and an indication of the location to which a head pointer employed by the dequeueing engine for the purpose of calculation of the size of a permitted read may be updated provided that the given write and all the preceding writes are complete.

According to the invention a system for the storage of data segments comprises a memory organised to define at least one queue, the queue having associated therewith a head pointer indicating an address for the writing of a data segment and a tail pointer indicating an address for the reading of a data segment; a queuing engine for progressing the head pointer for a queue through the memory space in accordance with write requests; a dequeuing engine for progressing the tail pointer through the memory space accordance with a size parameter that indicates a memory space that can be read; and control logic for maintaining for the queue a list of control register entries identifying writes of data segments each with a sequence number commencing from a datum and an indication of a head pointer value; the control logic being responsive to the completion of a write for which the sequence number differs from the datum to clear the respective control register entry, to transfer the head pointer value from that entry to the entry having an immediately preceding sequence number and to decrement the sequence numbers for later entries each by unity; in which system said control logic allows updating of said size parameter for use by the dequeueing engine and in accordance with a register entry only after the write denoted by that control register entry is complete and has the datum sequence number.

Preferably each control register entry includes a size value for the data segment and the control logic on the completion of the respective write for which the sequence number differs from the datum adds the respective size value to the size value for the entry with the immediately previous sequence number and on completion of the write for which the respective control register entry has a sequence number equal to the datum allows updating of the size value for use by the dequeuing engine.

The system is applicable when the memory can maintain a large multiplicity of queues. The memory may be a random access memory which can define a multiplicity of elastic' FIFOs each for a respective queue.

The queueing engine may comprise a multiplicity of write buffers which are capable of writing to the same queue.

Brief description of the drawings

Figure 1 is a schematic diagram of one embodiment of a data storage system according to the invention Figure 2 illustrates a FIFO Figure 3 is a diagram illustrating asynchronous write operations Figures 4A to 4H illustrate consecutive states of context registers in the performance of one example of the invention.

Detailed description

Figure 1 illustrates schematically one embodiment of the invention. This particular example is preferably mainly embodied on a processing chip which stores the data segments off-chip' i.e. in an external memory. However, this generally preferable expedient is not essential to the invention.

The data segments which are employed in this example are control entries, each of which denotes a data packet. As is explained in more detail in the aforementioned European patent application typically a ordinary packet comprises a header and a payload. The header includes the destination address, the source address, the packet type and information from which, among other things, the priority can be ascertained. The packet may conform to any of known protocols. When the packet is received it is processed. This may be a complex operation but for the present purposes it is sufficient to state that the processing, denoted by packet processing stage, includes a lookup in a database to obtain forwarding data for the packet. The payload of the packet is preferably temporarily stored in data memory. The processing obtains a control entry, which is a data set derived from the header and is used to control the flow of the packet through the device. The control entry (or data set) normally includes destination information, which is derived from the network address information during the lookup, protocol identifiers, an identification of the destination port (in the case of a unicast packet) and an address pointer which indicates where the packet is stored in the data memory. The control entry will be placed on one of the queues defined in control memory.

The invention is applicable to the storage of such control entries' from multiple sources in a multiplicity of queues but is also applicable to storage of other kinds of data segment in queues.

The system shown in Figure 1 receives control entries from various sources 10 and mainly comprises a queue controller block (QCTL) 11 for the control of storage and readout of the control entries, by way of a control RAM interface block CFCI 12 and a read/write interface 13, in memory 14, which is usually but not necessarily external memory, i.e. off-chip' memory. The control entries are stored in the storage of those entries queues, each queue being held in an elastic' FIFO defined in the memory 14. Queue controller block (QCTL) 11 also controls the reading from the FlFOs to provide control entries to various destinations 15. Such destinations may include ports for the dispatch of packets but may comprises processing blocks or other local destinations' within the processing chip.

One example of the external memory comprises 32 megabytes of usable memory space organised into, in this example, a maximum of 1112 FIFOs each of which can store one queue of control entries.

A simple illustration of a FIFO 20 is shown in Figure 2A. It is convenient to define such a FIFO elastically' in memory space 30 by means of an initial address (which may be prefetched) and consecutive addresses. In this example the FIFO 31 comprises a set of consecutive locations having addresses denoted 100, 101 102 and 103. Figure 2A also illustrates a head pointer 32 (indicating the next location for a write) and a tail pointer 33 (indicating the next location for a read). In Figure 2A there is zero space between the head and tail pointers.

Figure 2B illustrates the FIFO 31 with data written in locations 100 to 103, so that the head pointer denotes location 103 whereas tail pointer denotes location 100. This (as will be familiar to those skilled in the art) allows a read of size 3 (the number of locations). It is (again) customary to organize the FIFOs so that a read can be automatically initiated when there is non-zero memory space between the pointers, i.e. (since there are variations in nomenclature) the head and tail pointers embrace at least one data segment that has been written but not read).

Figure 2B includes a virtual' head pointer 34 which will be explained later.

Reverting now to Figure 1, the queue controller (QCTL) block 13 comprises three main sub-modules, namely a control entry queuing engine (CEQE) 17, a control dequeueing engine (CDQE) 18, and a control queue buffer manager (CQBM) 19.

The control queue buffer manager (CQBM) 19 maintains head and tail pointers and size values for all the queues in the (external) memory 11. It provides to the control dequeueing engine (CDQE) 18 a tail pointer 20 for the reading of a control entry from a queue and receives from the control dequeueing engine (CDQE) a value 21 by which a queue's size is to be decremented by the CQBM after a read.

The control dequeueing engine (CDQE) 18 reads (using the tail pointer provided) from the external memory 11. For this purpose it sends read requests RRs to the CFCI 12, which executes the read operations through the read/write interface 13 and sends read acknowledgements RAs to the control dequeueing engine (CDQE) 18. The CDQE module includes control read buffers (not shown) from which the read requests go to the memory 14 via the CFCI and which receive the read acknowledgements.

The queue controller block QCTL is shown as including includes three control write buffers CWB1, CWB2 and CWB3 for holding data segments (e.g. control entries) before they are written to the external memory via the CFCI 17. They are shown within block 17 for convenience of illustration only. Each of these buffers (in this example) comprises two internal buffers and therefore can provide two write requestors WR (i.e. data segments) to the interface block. The control write buffers send write requests WR to the memory via the CFCI 12 and receive write acknowledgements WA.

The control RAM interface block (CFCI) 12 controls the reading from and writing to the memory and arbitrates between the various read and write requests. For write operations it receives from the CWBs head pointers which indicate the respective location to which the control entry is to be written. It will receive from the CDQE module read requests including a tail pointer identifying the respective the location of the read. How these pointers are coordinated according to the invention is explained below.

Owing to the random access nature of the memory writing to different locations in the same queue can be performed contemporaneously as can reading and writing (with respect to different locations). The reading and writing locations are determined by the head and tail pointers. However, it should be understood that the write and read times are inevitably variable.

When the CEQE initiates a write to the memory it obtains a head pointer 22 from the CQBM. A write request goes to the CFCI and the write operation is performed; when the write operation is complete the CFCI provides the write acknowledgement. The CEQE updates the head pointer and sends to the CQBM a size increment 23 for the respective queue.

The CDQE can initiate a read of the external memory (by way of a CRB) when the CQBM indicates that the size' (as defined by the respective head and tail pointers) for the respective queue is non-zero.

As thus far described the system shown in Figure 1 operates in accordance with known techniques. -7-.

As indicated in the introduction, there are circumstances when multiple control write buffers are writing to the same queue in the memory. One such circumstance (known colloquially as a fat pipe') occurs when the data segments from all the relevant sources are destined for the same queue.

If multiple CWBs are writing to the same queue then the order in which the writes are completed is not necessarily the same as that in which those write were initiated. If a first write (i.e. a write which is initiated first in time) is not completed before the completion of a second write to the same queue, the updating of the head pointer and size which is a consequence of the completion of the second write will automatically allow reading from the queue, so that a read may be performed for a location to which data has not yet been written.

A further problem may arise when multiple controllers service a single queue, a write operation is in progress from one CWB and a second write operation is initiated from another CWB before the first write operation finishes. The CQBM block may then pass the same head pointer for the second write as for the first write, because head pointers are normally updated at the end of a write.

The system includes within the CEQE a means, implemented by registers called herein context registers' for maintaining various parameters that define in effect for each write requestor a size value by which the size held in the CQBM for the queue is to be incremented (increment 23). These parameters include a head pointer, headptr_inmem, indicating the address in memory to which the control entry (data segment) is to be written (head pointer 32 in Figures 2A and 2B); and a head pointer, headptr_for_cqbm, to which the CQBM is to updated for that write.

There are in the example three context registers CR1, CR2 and CR3, coupled to control register control logic (CRCL) 24. As will become apparent, the system is scalable to any desired size. The registers are shown separately in Figure 1 but may be defined within a common memory space.

Each context register contains for its respective CWB entries which identify a queue, a head pointer address, a write status and a size value. Where (as in this example) the CWB can each provide two requestors to the memory (from respective ports of the buffer) the register identifies the requestor. The register entry also has a field for a sequence number.

When the CEQE initiates a write to the memory 14, it sends a write request to the CWB sub-module which contains the context registers and the control register control logic. When the control register control logic sees the write request it first determines whether there are context registers (other than the one for the respective CWB) active for the respective queue. If there is no such other active context register, the CRCL 24 loads the head pointer in the context register for the requesting ICWB with the head pointer (22) from the CQBM 19. The CRCL 24 loads the requesting CWB's context register with a datum sequence number (conveniently zero). The CRCL 24 loads the size' for the requesting CWB's context register with unity (1). The CRCL 24 loads the requesting CWB's context register location for headptr_focqbm with head_ptr_in.mem.

If there are other CWB context registers active for the queue, the CRCL determines which are active. It takes the headptr_in_mem value from the context register with the highest sequence number, increments that number by unity, and loads the incremented head_ptr_inmem for the requesting CWB. The CRCL 24 also increments the highest sequence number and loads the incremented sequence number into the requesting CWB's context register. The size value is loaded as unity and the entry for head_ptr_for_cqbm is loaded with the value for the (incremented) head_ptr.Jn_mem.

When the CFCI 12 indicates that the write has been successfully completed, the CRCL 24 performs as follows.

If the sequence number for the completed write is at the datum (zero) that fact is used as an indication that there is no write initiated before the completed write but still itself incomplete. The CRCL 24 updates the CQBM with the head_ptr_for_cqbm and the respective size increment 23.

If the sequence number for the completed context register entry is not at the datum (i.e. is non-zero) it is presumed that there is still at least one write still pending in memory locations for write earlier than the completed write. No update to the CQBM occurs. Instead, the head_ptr_forcqbm value for the completed write is popped' into the headptrjor_cqbm space for the context register entry having a sequence number one less than that of the completed entry. The size value for that context register entry is incremented by unity. All the context register entries having a sequence number greater than the completed entry have their sequence numbers decremented by unity.

This scheme means that if the write are completed in an order different to that in which they are initiated, the control logic waits until all writes in a sequence including the one for which the sequence number is the datum are complete and then updates the CQBM with one update that has the correct head pointer value and a size that embraces all the completed writes.

The foregoing is a complex operation and a practical example will be explained with reference to Figure 3, which shows an example of asynchronous write operations to a single queue, and Figures 4A to 4G which show the contents of the respective context register after the various events in the example shown in Figure 3.

It will be understood that the register entries shown in Figure 4 are not those of a single register. Figures 4A to 4H each show all the relevant entries grouped together for convenience.

Figure 3 illustrates the timing of four asynchronous writes to a singe queue, arbitrarily denoted Q7.

A first write to the queue Q7 from port CWB1a is requested at the time I in Figure 3.

The control logic generates for the context register for CWBIa an entry, shown in Figure 4A. For this first write a prefetched address may be used (according to known art) and is inserted in the head address field Head Add'. The address pointer is arbitrarily indicated as lOO'.The identification la is in the CWB Port field indicates that this register location is in the control register for buffer CWB1a. The identity of the queue (Q7) is inserted in the QID field, a datum sequence number (in this example, zero) is set in the SeqNum field, a busy' bit is set in the bsy' field and a size (1) is set in the Size upd' field. The set bsy' bit indicates that the write operation is in progress and is not complete and the size indicate the potential update of the head pointer (one location). At this stage the value for head_ptrjn_mem (i.e. 100) is inserted in the location for headptr_forcqbm.

Since the write is in progress (absent a write acknowledgement) there is no update of the head pointer or queue size to the CQBM.

Figure 4B indicates the state of the context registers at time point 2 in Figure 3. Now a second write, from buffer b of CWB1 has been initiated. The head pointer 32, i.e. head_ptr_in_mem (now 101) can be locally generated by incrementing the address pointer already in the register. The entry in the context register for buffer CWB1b is shown in the second line of Figure 4B, and is identified by the value lb in the CWB Port field. The sequence number is incremented to 1, the bsy bit is set and the size value for the entry is set to 1. As before, the value for head_ptr_inmem (i.e. 101) is inserted in the location (Head Add) for head_ptr_for_cqbm.

Since both the writes are in progress (absent a write acknowledgement) there is no update of the head pointer or queue size to the CQBM.

Figure 4C indicates the state of the context registers at time point 3 in Figure 3. Now a third write, from buffer CWB2a has been initiated. The relevant control register's entry is shown in the third line of Figure 4C. The sequence number is incremented to 2, the bsy bit is set and the size value for the entry is set to 1. As before, the value for headptr_inmem (i.e. 102) is inserted in the location (Head Add) for headptr_for_cqbm Since all the writes are in progress (absent a write acknowledgement) there is no update of the head pointer or queue size to the CQBM.

Figure 4D indicates the state of the context register at time point 4 in Figure 3. Now a fourth write, from buffer 2b of CWB2 has been initiated. The control register's entry is shown in the fourth line of Figure 4D. The sequence number is incremented to 3, the bsy bit is set and the size value for the entry is set to I. As before, the value for head_ptr_in_mem (now 103) is inserted in the location (Head Add) for headptr_for_cqbm.

Since all the writes are in progress (absent a write acknowledgement) there is no update of the head pointer or queue size to the CQBM.

Although therefore the real head pointer is at address location 103, the head pointer control CQBM maintains for the present a virtual head pointer 34 (or, in other words, a size value) which has not changed (i..e the size value is still apparently zero) and so presumes that there is zero memory space between the tail pointer and the head pointer for the queue Q7.

Figure 4E indicates the states of the context registers at time point 5 in Figure 3. This is an important stage. The third write to queue Q7 from CWB2a is complete. The first change from the states shown in Figure 4D is that the busy bit for the third write (register 2a) is cleared. Since the entry has a non-zero sequence number (in general, a sequence number which is greater than the datum number used for the first write) this criterion prevents the update of the head pointer and size value to the CQBM. However the register entry of the pointer is shifted (popped') into the position previously occupied by the entry for the preceding entry the sequence and the size value for the completed write is added to the size value for the preceding (incomplete) write. This, context register entry for buffer lb holds now 102 (instead of 101) and a size value of 2 instead of 1. Also, higher sequence numbers (in this example that for the fourth write) are each decremented by unity.

Figure 4F shows the context registers after time point 6 in Figure 3. The fourth write (to address 103) is complete. A process similar to that described with reference to Figure 4E occurs. Since requestor CWB2b has a non-zero sequence number there will be no update for CQBM. However the value for head_ptr_for_cqbm (i.e. 103) is popped into the position to that previously occupied by the entry for CWB1b and the size value (1) for the fourth write is added to the size value (2) for that entry.

There would be a decrement by unity of all higher sequence numbers if there were any, but in this example there is no higher sequence number which can be decremented.

After time point 6 in Figure 3 the first and second writes are still in progress. The real head pointer is at address 103 but the virtual head pointer maintained by the CQBM is still at address 100 and the size value maintained by the CQBM is still zero, so the tail pointer does not alter.

Figure 4G shows the context register after time point 7 in Figure 3. Now the first write from buffer CWB1a (to address 100) is complete. Since this write has a datum (zero) sequence number the head pointer supplied to the CQBM is updated with the address 100 + 1(.i.e. incremented to a virtual value 101 by the size shown in the S entry), and the busy bit is cleared. Any higher sequence number is decremented. At this stage only the register entry for the second write is in progress and its sequence number is decremented to the datum (zero).

At this time (the interval between time points 7 and 8 in Figure 3, the second write is still in progress, the actual head pointer for queue Q7 in memory is at address 103, the virtual head pointer (employed by the CQBM) is at 101. When the second' write completes, the head pointer employed by the CQBM will be updated to the value shown for the remaining write in the control register.

At the time point 8 in Figure 3, the write to address 103 from buffer CWBIb is complete. Since the context register entry has a datum (zero) sequence number the head pointer supplied to the CQBM is updated with the size (3) to the address 103 + 1 and the busy bit is cleared, All the context registers are now cleared, as shown in Figure 4H.

Claims (5)

1. Claims 1. A system for the storage of data segments comprising a memory (14) organised to define at least one queue, the queue having associated therewith a head pointer indicating an address for the writing of a data segment and a tail pointer indicating an address for the reading of a data segment; a queuing engine (17) for progressing the head pointer for the queue through the memory space in accordance with write requests; a dequeuing engine (18) for progressing the tail pointer through the memory space in accordance with a size parameter that indicates a memory space that can be read; and control logic (24) for maintaining for the queue a list of control register entries identifying writes of data segments each with a sequence number commencing from a datum and an indication of a head pointer value; the control logic being responsive to the completion of a write for which the sequence number differs from the datum to clear the respective control register entry, to transfer the head pointer value from that entry to the entry having an immediately preceding sequence number and to decrement the sequence numbers for any later entries each by unity; in which system said control logic (24) allows updating of said size parameter for use by the dequeueing engine and in accordance with a register entry only after the write denoted by that control register entry is complete and has the datum sequence number.
2. 2. A system according to claim 1 in which each control register entry includes a size value for the data segment and in which the control logic (24) on the completion of the respective write for which the sequence number differs from the datum adds the respective size value to the size value for the entry with the immediately previous sequence number, and on completion of the write for which the respective control register entry has a sequence number equal to the datum allows updating of the size value for use by the dequeuing engine.
3. 3. A system according to claim 1 or claim 2 in which the memory (14) can maintain a multiplicity of queues.
4. 4. A system according to claim 3 in which the memory (14) defines a multiplicity of elastic' FIFOs each for a respective queue.
5. 5. A system according to any forgoing claim in which the queueing engine (17) comprises a multiplicity of write buffers which are capable of writing to the same queue.