CN112486874B

CN112486874B - Order-preserving management method and device for I/O (input/output) instructions in wide-port scene

Info

Publication number: CN112486874B
Application number: CN202011368747.4A
Authority: CN
Inventors: 宁佐林; 刘亿民
Original assignee: Wuxi Zhongxing Microsystem Technology Co ltd
Current assignee: Wuxi Zhongxing Microsystem Technology Co ltd
Priority date: 2020-11-30
Filing date: 2020-11-30
Publication date: 2021-12-10
Anticipated expiration: 2040-11-30
Also published as: CN112486874A

Abstract

The invention provides an order-preserving management method and device for I/O (input/output) instructions in a wide-port scene, which comprises the following steps: the system comprises a plurality of physical I/O channels, a plurality of storage units and a plurality of DMA transmission modules, wherein each I/O channel comprises an independent receiving cache and a DMA transmission module, the receiving cache is used for caching I/O instruction data frames from a disk, and the DMA transmission module is used for transporting the cached I/O instruction data frames to a main memory; the instruction receiving manager is arranged between the DMA transmission modules of the physical I/O channels and a bus of the host, and is configured to receive I/O instruction data frames from the DMA transmission modules, set a frame counter for each I/O instruction, and determine the sequence of the data frames of the I/O instruction sent to a memory of the host through the value of the frame counter; and the DMA transmission module carries the I/O instruction data frame to the host memory according to the determined sequence.

Description

Order-preserving management method and device for I/O (input/output) instructions in wide-port scene

Technical Field

The invention belongs to the field of disk array reading and writing, and particularly relates to an order-preserving management method and device for I/O (input/output) instructions in a wide-port scene.

Background

SAS is one of the most widely used hard disk connection technologies in a storage network, and has the highest interface rate in the current hard disk channel technology, and in an SAS disk device array, a wide link established by an SAS device is usually implemented by a wide port of one or more PHYs included in the SAS device. Wide port technology doubles the transmission bandwidth. In the wide port architecture, since a data Frame (Frame) to be received may be received from any PHY in the wide port, when writing the Frame information into a host memory (DDR) and feeding back the Frame information to application layer software, it is necessary to ensure that the correct address space is written in sequence according to the sequence of the frames during writing. Referring to fig. 1, a process of receiving a data stream from an SAS disk device by a host memory in the prior art is illustrated as follows:

1. the host computer sends a read data I/O instruction;

2. the disk receives a read data I/O command and starts to prepare for reading data;

3. after the disk prepares the first 3 frames (assuming that the capacity of the Rx Buffer of each channel at the receiving end is exactly 3 frames), it applies to establish a connection, and the Expander selects PHY0 as the current transmission channel.

4. After the connection is successfully established, the disk writes 3 frames of data into the Rx buffer of the channel 0, then closes the connection, and continues to prepare the remaining data of the current I/O read instruction.

5. After receiving the 3 frames of read data, channel 0 starts DMA transfer and transfers the data to the host memory (DDR).

6. The disk prepares the second batch of 3 frame data, reappears to establish connection, and the Expander selects the PHY1 as the current transmission channel according to the busy and idle condition of the channel.

7. After the connection is successfully established, the disk writes the second batch of 3 frames of data into the Rx Buffer of the channel 1, then closes the connection and continues to prepare the remaining data of the current I/O read instruction.

And repeating the operation of the steps 3-6.

After the channel 1 receives the 3 frames of data, before starting the DMA transfer, it is necessary to ensure that the DMA transfer of the first 3 frames is completed and obtain a correct DMA breakpoint, and then the current DMA transfer can be started, otherwise, an error occurs in the data frame sequence written into the DDR, and finally a read data error of the I/O instruction is caused. It can be seen that, because the transfer speed of the DMA is ideally faster than the return data speed of the hard disk, the previous data batch is completely transferred to the DDR by the DMA before the hard disk returns the next data batch. In practice, it is not always guaranteed that the first 3 frame DMA transfer has been completed before the DMA transfer is initiated. However, since the order-preserving processing is not performed, it is difficult to avoid the problem that read data is overwritten out of order to cause read data errors, especially in the case of bus congestion.

Disclosure of Invention

The present invention provides an order-preserving management apparatus for I/O instructions in a wide-port scenario, including:

the system comprises a plurality of physical I/O channels, a plurality of storage units and a plurality of DMA transmission modules, wherein each I/O channel comprises an independent receiving cache and a DMA transmission module, the receiving cache is used for caching I/O instruction data frames from a disk, and the DMA transmission module is used for transporting the cached I/O instruction data frames to a host memory;

the command receiving manager is arranged between the DMA transmission modules of the physical I/O channels and a bus of the host, and is configured to receive I/O command data frames from the DMA transmission modules, set a frame counter for each I/O command, and determine the sequence of the data frames of the I/O command to be sent to a memory of the host according to the value of the frame counter;

the DMA transfer module is further configured to transfer the I/O command data frames to the host memory according to the determined order.

Preferably, the frame counter includes a reception frame counter indicating the number of frames in which the I/O command has been DMA-handled, and a transmission frame counter indicating the number of frames in which the I/O command has been received from the physical channel.

Preferably, the instruction receiving manager is further configured to store an entry for each I/O instruction, where the entry includes a number corresponding to each I/O instruction and a mapping relationship between a receiving frame counter and a transmitting frame counter thereof.

Preferably, each of the I/O channels includes a temporary counter, the instruction receipt manager being further configured to:

when a new received data frame is detected, the serial number of the I/O instruction reported by the receiving SAS channel is searched for whether the serial number of the current I/O instruction is already in the table entry;

if the current number exists in the table entry, assigning the value of the received frame counter corresponding to the current number to a local temporary counter of each I/O channel, and adding 1 to the value of the received frame counter corresponding to the current number; if the current number does not exist in the table entry, the number is stored into an empty table entry, meanwhile, the value of the receiving frame counter corresponding to the table entry is added with 1, the value of the transmission frame counter is set to be 0, and a local temporary counter of an I/O channel is set to be 0;

then, searching the transmission frame counter corresponding to the current number in the table item, and comparing the transmission frame counter with the value of the temporary counter in the I/O channel corresponding to the current frame;

if the value of the transmission frame counter is equal to that of the temporary counter, starting DMA transmission of an I/O channel corresponding to the current frame, namely setting a start signal as valid, and adding 1 to the value of the transmission frame counter after the DMA transmission is finished;

if not, if the values of the transmission frame counter and the temporary counter are not equal, the DMA transmission of the I/O channel corresponding to the current frame is forbidden;

and updating the transmission frame counter in a preset period, and starting the DMA transmission of the I/O channel until the transmission frame counter is equal to the value of the temporary counter of the I/O channel corresponding to the current frame.

Preferably, the instruction receiving manager is further configured to delete the number from the entry after all data frames of the I/O instruction corresponding to the current I/O instruction number are subjected to DMA transfer.

Correspondingly, according to another aspect of the present invention, there is provided an order-preserving management method for I/O instructions in a wide-port scenario, including:

in a plurality of physical I/O channels, buffering I/O instruction data frames from a disk by utilizing independent receiving buffers;

setting a frame counter for each I/O instruction, and determining the sequence of the current data frame sent to the host memory according to the value of the frame counter;

and carrying the I/O instruction data frames to the host memory through DMA transmission according to the determined sequence.

Preferably, an entry is saved for each I/O instruction, where the entry includes a number corresponding to each I/O instruction and a mapping relationship between a received frame counter and a transmitted frame counter thereof.

Preferably, each I/O channel of the plurality of physical I/O channels includes a temporary counter, the method further comprising:

then searching the transmission frame counter corresponding to the current number in the table item, and comparing the transmission frame counter with the value of the temporary counter in the I/O channel corresponding to the current frame;

Preferably, after all data frames of the I/O command corresponding to the current I/O command number are transferred by the DMA, the number is deleted from the entry.

Compared with the prior art, the invention has the following advantages:

the invention solves the problem of order preservation of multiple data frames of received data in an SAS wide port scene, realizes sequential writing of data frames of read data written back to a host memory of an I/O read instruction, avoids data reading errors when multiple ports simultaneously initiate DMA operation and a bus is congested, and improves the robustness of the system.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.

FIG. 1 shows a schematic diagram of a wide port I/O instruction host receiving a data stream according to the prior art.

Fig. 2 shows an I/O instruction uplink/downlink interaction diagram of Rx _ Manager in a wide port scenario according to an embodiment of the present invention.

FIG. 3 is a block diagram of an apparatus for order preserving management of I/O instructions in a wide port scenario according to an embodiment of the present invention.

FIG. 4 is a flowchart illustrating an order preserving management method for I/O instructions in a wide port scenario according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

As described above, if the bus congestion occurs due to frequent bus accesses in the system without performing the I/O order preservation, the DMA may not finish the transfer of the previous batch of data until the next batch of data is returned, and a read data command error may occur. The invention provides an order-preserving management method and device for an I/O instruction in a wide port scene, which are used for preserving the order of multiple data frames of received data in an SAS wide port scene.

In a wide port scenario, a data Frame (Frame) to be received may be received from any PHY in the wide port, so the receiving policy includes not only receiving itself, but also performing necessary detection on the received data Frame (Frame), and the main function is to ensure that Frame data received between channels in the wide port mode is order-preserved and written in as a continuous address of a host memory (DDR).

According to another aspect of the invention, correspondingly, an I/O instruction order-preserving management device in a wide-port scene is provided.

In a preferred embodiment of the present invention, for all channels (PHYs) under one wide port, a receive frame counter recv _ tag and a transmit frame counter tran _ tag are set for each I/O instruction, and the position and the sequence of the current data frame in the host memory (DDR) are determined by the value of the frame counters.

The order-preserving management device of the I/O instruction in the wide-port scene comprises an instruction receiving Manager module called Rx _ Manager, which is used for realizing the receiving detection and order-preserving functions. FIG. 3 is a block diagram of an apparatus for order preserving management of I/O instructions in a wide port scenario according to an embodiment of the present invention. Rx _ Manager is arranged between the DMA transmission module and the data bus and is used for directly controlling the transmission of the DMA of all channels.

In Rx _ Manager, an entry is kept for each I/O instruction currently being processed, for keeping the sequence number of the currently received data frame. The format of the table entry is as follows:

bit[21:17]	bit[16:12]	bit[11:0]
			recv_tag_0	tran_tag_0	iptt_0
recv_tag_1	tran_tag_1	iptt_1
			…	…	…
recv_tag_15	tran_tag_15	iptt_15

the entry contains 3 fields:

1, iptt: indicating that the I/O instruction with the order-preserving requirement corresponds to a number IPTT;

tran _ tag: representing the number of frames of the I/O instruction which are carried out by the DMA;

recv _ tag: indicating the number of frames that the I/O instruction has received from the physical channel.

The RX _ Manager module is specifically configured to, when detecting a new received data frame, receive a number IPTT of an I/O instruction reported by an SAS channel, and find whether the number IPTT of the current I/O instruction is already in the entry.

If the current IPTT exists in the table entry, the RX _ Manager module assigns a recv _ tag corresponding to the current IPTT to a local exp _ tag temporary counter of each channel to represent an expected tag number, and adds 1 to the recv _ tag corresponding to the current IPTT; if the current IPTT does not exist in the table entry, the Rx _ Manager module stores the IPTT into an empty table entry, and meanwhile adds 1 to recv _ tag of the table entry, sets 0 to tran _ tag and sets 0 to an exp _ tag counter of the local channel.

At this time, each IPTT currently has a corresponding entry. And the Rx _ Manager module continuously searches for the tran _ tag corresponding to the current IPT in the table item and compares the tran _ tag with the exp _ tag number in the channel.

If the tran _ tag is equal to the exp _ tag, the Rx _ Manager module starts DMA transmission of the channel, namely, a start signal is set to be effective, and 1 is added to the tran _ tag after the transmission is finished;

if the tran _ tag and exp _ tag are not equal, the Rx _ Manager module disables DMA transfers for the current channel.

If the tran _ tag and the exp _ tag are not equal, indicating that the previous data DMA has not been completely transferred, the Rx _ Manager needs to wait for the previous data DMA to be completed to start the DMA transfer of the current channel, so as to ensure the data frame sequence written into the memory of the host.

The Rx _ Manager module updates the tran _ tag at a preset period, and starts DMA transmission of the channel until the tran _ tag is equal to the exp _ tag of the current channel.

Preferably, the Rx _ Manager module deletes the IPTT from the entry after all data frames of the I/O instruction corresponding to the current IPTT number are transferred by the DMA.

See specifically fig. 2 for an uplink/downlink data interaction diagram of Rx _ Manager. When an RX DMAC (receiver/transmitter) of a certain physical I/O channel detects that a frame from a buffer is received, setting a validity signal vld valid, and simultaneously providing an instruction number IPTT corresponding to the received frame for an Rx _ manager; after detecting vld, the Rx _ manager adds 1 to a frame receiving counter, and simultaneously feeds back a ready handshake signal rdy receiving the vld to the RX DMAC; the RX DMAC zeroes the vld signal after receiving the rdy signal.

When the Rx _ manager is ready to authorize the Rx DMAC to perform the transfer operation from the received frame buffer to the host memory DDR, the Rx DMAC performs the data transfer operation when detecting that the Start is high, and after the end of the transfer operation, raises the end signal end by one clock cycle, and after the Rx _ manager receives the end signal, the Rx _ manager sets the Start signal to zero.

It should be noted that the numerical sequence numbers (e.g., iptt _0, …, iptt _15) in the entries represent the number of each I/O instruction having an order-preserving requirement, which corresponds to the number of I/O instructions in the SAS channel. It should be understood by those skilled in the art that any adjustment of the number of I/O channels and the number of I/O instructions for which an order-preserving requirement exists may be made according to actual needs on the basis of the present invention, and the present invention should not be limited to the specific values illustrated above. Similarly, bits (e.g., [21:17]) in the table entry are only examples of the number of bits, and it should be understood by those skilled in the art that the number of bits in each field can be arbitrarily adjusted according to actual needs based on the present invention, and the present invention should not be limited to the specific values illustrated above.

According to another aspect of the invention, correspondingly, an order-preserving management method for I/O instructions in a wide-port scene is provided. FIG. 4 is a flowchart illustrating an order preserving management method for I/O instructions in a wide port scenario according to an embodiment of the present invention. The method comprises the following steps:

buffering I/O instruction data frames from a disk by using a receiving buffer in a plurality of physical I/O channels;

the RX _ Manager module sets a frame counter for each I/O instruction, and determines the sequence of the current data frame sent to the host memory according to the value of the frame counter;

and carrying the cached I/O instruction data frames to the host memory through the DMA transmission module according to the determined sequence.

According to the method, the RX _ Manager module carries out necessary detection on the received data frames, is used for ensuring the order preservation of the data frames received among all channels in the wide port mode and writes the data frames into the host memory (DDR) according to continuous addresses.

In a preferred embodiment, the specific operation flow is as follows:

the method comprises the following steps that 1, an SAS channel reports the number IPTT of an I/O instruction when detecting a new received data frame;

2, the RX _ Manager module searches whether the number IPTT of the current I/O instruction is already in the table entry;

if the current IPTT exists in the table entry, assigning a recv _ tag corresponding to the current IPTT to a local exp _ tag counter of each channel to represent an expected tag number, and adding 1 to the recv _ tag corresponding to the current IPTT;

if the current IPTT does not exist in the table entry, the Rx _ Manager stores the IPTT into an empty table entry, and meanwhile adds 1 to recv _ tag of the table entry, sets 0 to tran _ tag and sets 0 to an exp _ tag counter of the local channel;

3, the Rx _ Manager module searches for a tran _ tag corresponding to the current IPT and compares the tran _ tag with an exp _ tag number in the channel;

if the tran _ tag is equal to the exp _ tag, starting DMA transmission of the channel, namely setting a start signal as valid, and adding 1 to the tran _ tag after the transmission is finished;

if the tran _ tag and the exp _ tag are not equal, then the DMA transfer of the current channel is disabled.

If the tran _ tag and the exp _ tag are not equal, it indicates that the previous data DMA has not been completely transferred, and the DMA transfer of the current channel can be started only by waiting for the completion of the previous data DMA, so as to ensure the data frame sequence written into the host memory.

The Rx _ Manager module updates the tran _ tag at a preset period, and starts the DMA transmission of the corresponding channel until the tran _ tag is equal to the exp _ tag of the current channel.

4. And after all data frames of the I/O instruction corresponding to the current IPTT number are carried by the DMA, deleting the IPTT from the table entry.

In the above method, only when the condition that the tran _ tag is equal to the exp _ tag of the current channel is satisfied, the start signal is set to be valid, and the DMA transmission of the current channel can be started after the previous data DMA is completed and the tran _ tag is updated. This ensures that the data frame sequence written to the host memory (DDR) is correct.

Optionally, in step 3, the data frame sequence written into the host memory may also be ensured by controlling the disk. Specifically, if the tran _ tag is equal to the exp _ tag, disk data reception of the channel is started, and 1 is added to the tran _ tag after transmission is completed; if the tran _ tag is not equal to the exp _ tag, the new data of the disk is forbidden to be received, because the tran _ tag is not equal to the exp _ tag, the previous data DMA is not completely transmitted, and the subsequent data can be received from the disk only by waiting for the completion of the previous data DMA.

Furthermore, those skilled in the art will appreciate that the order preserving management apparatus for I/O instructions in the above port scenario does not constitute a limitation to the SAS disk array architecture, and a SAS disk array device may include more or less components, or some combination of components, as is known in the art.

It can be seen that, according to the technical solutions of the above specific embodiments, the data frame writing of the read data of the I/O read instruction back to the host memory (DDR) in the wide port scenario in the disk array system is sequentially written, thereby avoiding the data frame address error when the read data is written into the host memory (DDR) due to the bus congestion and the simultaneous DMA operations initiated by multiple ports, and improving the robustness of the system.

Although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. An order-preserving management device for I/O instructions in a wide-port scene is characterized by comprising:

the DMA transfer module is further configured to transfer the I/O command data frames to the host memory according to the determined sequence;

the instruction receipt manager is further configured to maintain an entry for each I/O instruction;

each of the I/O channels includes a temporary counter, the instruction receipt manager being further configured to:

2. The apparatus for order preserving management of I/O commands in a wide port scenario as claimed in claim 1, wherein the frame counter comprises a receive frame counter and a transmit frame counter, the transmit frame counter represents a number of frames that the I/O commands have been DMA-handled, and the receive frame counter represents a number of frames that the I/O commands have received from a physical channel.

3. The apparatus for order-preserving management of I/O instructions in a wide-port scenario as claimed in claim 2, wherein:

the table entry includes the number corresponding to each I/O command and the mapping relationship between the received frame counter and the transmitted frame counter.

4. The apparatus for order preserving management of I/O commands in a wide port scenario as claimed in claim 1, wherein the command receiving manager is further configured to delete the number from the entry after all data frames of the I/O command corresponding to the current I/O command number are carried by the DMA.

5. An order-preserving management method for I/O instructions in a wide-port scene is characterized by comprising the following steps:

according to the determined sequence, the I/O instruction data frame is transmitted and carried to a host memory through DMA;

saving a table entry for each I/O instruction;

each I/O channel of the plurality of physical I/O channels includes a temporary counter, the method further comprising:

6. The method for order preserving management of I/O commands in wide port scene as claimed in claim 5, wherein said frame counter comprises a receiving frame counter and a transmitting frame counter, said transmitting frame counter represents the number of frames that said I/O commands have been DMA carried out, said receiving frame counter represents the number of frames that I/O commands have been received from physical channel.

7. The method according to claim 6, wherein the entry includes a number corresponding to each I/O instruction and a mapping relationship between a received frame counter and a transmitted frame counter.

8. The I/O instruction order-preserving management method under the wide-port scene according to claim 5, further comprising deleting the number from the table entry after all data frames of the I/O instruction corresponding to the current I/O instruction number are DMA-carried.