KR100723879B1 - Hardware acceleration apparatus for iscsi target system using toe and method for handling read/write command using the apparatus - Google Patents

Abstract

The present invention relates to a hardware acceleration device on an iSCSI target system using a TOE and a method of performing a read / write command using the device. The inventors of the present invention provide a hardware acceleration device on an iSCSI (Internet Small Computer System Interface) target system. TCP / IP Offload Engine (TOE) controller, disk controller, and network controller are connected through one local PCI bus, and large memory is installed through I / O processor. Combining the acceleration capabilities of each internal hardware, the hardware accelerator improves the processing performance of the iSCSI target system and reduces the CPU load in performing complex and time-consuming data processing tasks on the iSCSI target system.

Description

Hardware acceleration apparatus for iSCSI target system using TOE and method for handling read / write command using the apparatus}

1 is a hierarchical model of the iSCSI protocol on which the present invention is based;

2 is a flowchart illustrating transmission of a read / write command through an iSCSI protocol on a network according to an embodiment of the present invention;

3 is an internal configuration diagram of a hardware accelerator on an iSCSI target system according to an embodiment of the present invention;

4 is a diagram illustrating a movement path of data when a read command is performed on an iSCSI target system in FIG. 3;

FIG. 5 is a diagram illustrating a movement path of data when a write command is performed on an iSCSI target system in FIG. 3. FIG.

6 is a flowchart illustrating a method of performing a read / write command on an iSCSI target system according to an embodiment of the present invention.

<Explanation of symbols for the main parts of the drawings>

301: main CPU 302: main memory

303: main PCI bus 304: I / O processor

305: network controller 306: TOE controller

307: local memory 308: local PCI bus

309: disk controller 310: disk storage device

311: Hardware Accelerator on iSCSI Target System

The present invention relates to a hardware acceleration device on a storage system via a network and a data processing method through the device, and more particularly, to a hardware acceleration device on an iSCSI target system using a TOE and a method of performing a read / write command using the same. Include.

Since computer systems have evolved, the amount of data stored on servers has increased exponentially, and the performance demands on data storage devices have steadily increased. As a result, there is a limit to storing data only in a storage device directly connected to a conventional server. Therefore, various storage connection methods through a network have been devised.

Conventional network storage methods and server connection methods that are widely used up to now include a network file system (NFS), a network attached storage (NAS), a storage area network (SAN), and the like.

NFS is a network storage device connection method that has been used in the past, and stores data in file units through a TCP connection between a general server and a server. However, this increases the load on the file system and degrades transmission performance.

Similar to NFS, NAS stores files on a TCP connection, but reduces the load on the server by using dedicated hardware. However, it is true that the load of the file system interferes with the improvement of the storage performance on the target system due to the limitation of the file unit storage method.

Thus, unlike NAS or NFS, SAN connects storage and server through a dedicated network and reduces the load of storage devices by reducing the load on storage devices by storing the file system on the server side through block-by-block storage rather than file. It is an improved method. However, this also has a disadvantage in that it is expensive to construct a dedicated network and it is difficult to form a connection network over a broadband.

It is the approach to network protocol that turns to this. In particular, there have been various attempts to use IP networks that are familiar, relatively inexpensive, and routable, and therefore have no distance limitations, and have emerged with the iSCSI (Internet Small Computer System Interface) protocol. This is a method of placing an iSCSI protocol on top of the TCP / IP protocol on the Internet and transmitting and receiving data using the same.

However, since such iSCSI is connected through an IP network, there is a side that generates a lot of load in the TCP / IP protocol processing. However, if there is a way to appropriately control this load can be useful to use the iSCSI protocol, the following will be discussed and discussed the related inventions.

First, the invention of U.S. Patent Application Publication No. 2004-0174893, filed on February 23, 2004 by Yoshihiro Saito and filed with US Patent Office by NEC Corporation on September 23, 2004, includes an iSCSI device and a communication control method thereof. A technique is disclosed.

The present invention classifies incoming packets using a discreet network controller on an iSCSI target system and then places two computation processors for each distributed process in the case of iSCSI packets and not in case of large packets. By improving the iSCSI input / output performance, the performance is improved.

In this publication, the operation processor for processing two network protocols is provided so that iSCSI input / output processing performance is not degraded even if the amount of IP traffic increases. However, another important part of the server is a separate TCP for reducing the CPU load. There is no special method for the / IP protocol processing part.

The following is an invention of U.S. Patent Application Publication No. 2004-0062267 filed by John Shigeto Minami et al., Filed with the U.S. Patent Office on June 5, 2003, and published on April 1, 2004, which supports the iSCSI and IPSec protocols. Gigabit Ethernet adapters are disclosed. In this patent, a TCP / IP protocol processing method is added to a Gigabit Ethernet controller by adding a processor, a memory, and a program, and in particular, a protocol processing such as iSCSI or IPSec is performed on a built-in processor and memory to improve performance. Doing.

In general, the present disclosure relates to an intelligent network adapter and its processing structure that improve performance by having a protocol processing performed by a general processor embedded in a network controller.

However, when processing TCP / IP protocols using a general operation processor and a program, there is a problem that performance is reduced compared to a dedicated TOE controller. If the processing program for all protocols proposed in the present invention is an Ethernet controller, If it is included internally, there is a possibility of further deterioration. Also, when used in an iSCSI target system, the movement path optimization of data is just as important as the processing of a protocol. The use of a general processing processor, as in this disclosure, is simply a dedicated TOE controller, a network controller, a disk controller, There is a shortcoming to fall short of optimizing paths using I / O processors.

Accordingly, in order to solve the above problems, the present invention provides a hardware acceleration device configured by integrating its own memory with specialized controllers of an I / O operation processor, a disk controller, and a network controller, including a TCP / IP offload engine (TOE) controller. Use

In particular, the hardware accelerator device effectively processes read / write commands on the iSCSI protocol by TCP / IP offloading data transmitted and received through a network system instead of the main CPU.

According to an embodiment of the present invention, a hardware accelerator device on an iSCSI target system using the TCP / IP Offload Engine (TOE) may be configured to store data transmitted between a disk storage device and a network system. A TCP / IP Offload Engine (TOE) controller that performs TCP / IP stack offloading by processing TCP / IP packets of data stored in the memory and the local memory on behalf of the main CPU, and the data offloaded from the local memory. a network controller which transmits data to the network system by using an iSCSI protocol or controls to store data received from the network system in the local memory, a disk controller which stores and retrieves data between the disk storage device and the local memory; I / O for controlling data input and output to the local memory It is configured to include a processor and a local PCI bus that connects each of the devices to serve as a data transmission path.

In addition, an embodiment of a method of performing a read / write command on an iSCSI target system in such a hardware accelerator device is performed through the following steps.

Regarding a method of performing a read command on an iSCSI target system, the method may include: extracting desired data from a disk storage device and storing the read data in a local memory, and processing TCP / IP packets of the stored data instead of the main CPU to offload the TCP / IP stack. performing offloading, and transmitting the offloaded data to a network system using a checksum offloading and a scatter / gather transmission function.

As to a method of performing a write command on an iSCSI target system, storing the data received from a network system to a local memory, and processing TCP / IP packets of the stored data instead of the main CPU to perform TCP / IP stack offloading. And storing the offloaded data in a disk storage device.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the present invention.

1 is a hierarchical model of the iSCSI protocol on which the present invention is based. TCP / IP is a standard protocol for transferring data between computers on the Internet and basically consists of four layers. It is divided into data link layer, network layer, transport layer and application layer. The iSCSI protocol layer model of FIG. 1 is a model based on the TCP / IP layer, and in addition to the data link layer 105, the IP layer 104 as the network layer and the TCP layer 103 as the transport layer. It can be seen that the iSCSI protocol layer 102 is placed on top of each other. In other words, it can be seen that the iSCSI protocol uses the TCP / IP protocol as a reliable method for transmitting and receiving data on the network.

However, three components are required for a storage consolidation system using the iSCSI protocol. First, an iSCSI target (target system) refers to storage that is the final subject of disk consolidation, and second, an iSCSI network is generally referred to as TCP / S. IP network 106 is referred to. Finally, the third is the iSCSI Initiator (= server), which is essentially an iSCSI protocol that works through the server's Ethernet LAN card, so there must be an iSCSI driver that can understand the iSCSI packets like any other driver.

In particular, this is called a software initiator when only the iSCSI driver is installed and used in an existing LAN card, and a hardware initiator when using a dedicated card (iSCSI HBA) that performs iSCSI-related operations.

In addition, since iSCSI is a standard standard that defines the serialization of the SCSI command set for TCP / IP, under the model structure, SCSI commands received from the iSCSI initiator as a server are transmitted to the target system through the iSCSI protocol. They are encrypted and sent with other data.

2 is a flow chart of a read / write command via an iSCSI protocol on a network. This shows how commands on the SCSI, such as read or write, are passed from the iSCSI initiator 201 to the target system 203 and the final response after the command execution is transmitted from the target system 203 to the initiator 201.

This includes a process of transmitting data to be read / write. In the present invention, a process in which a read / write command reaches the target system 203 through an actual iSCSI protocol or the target system 203 Except for sending a final response to the initiator 201, the focus of the command execution process 205 of the target system on the read / write command 204 will be discussed.

3 is a diagram illustrating an internal configuration of a hardware accelerator device on an iSCSI target system according to an exemplary embodiment of the present invention, and illustrates a connection state of each hardware accelerator device and its internal structure when the hardware accelerator device is connected to a PCI bus of a main board. It is.

The hardware accelerator 311 is configured to be connected to the system via the main PCI bus 303. In addition, the hardware acceleration device 311 is implemented in the form of a single PCI card, each device inside is configured to be interconnected by a local PCI bus (308).

The hardware accelerator 311 is an I / O processor 304 that functions as a PCI bridge function and a memory controller that manages a PCI bus connection with a main board, and serves as a buffer and data storage space when data is input and output. Local memory 307, network controller 305 that is responsible for connecting to external network systems and supports TCP / IP / UDP checksum and scatter / gather transfers, TCP / IP stack off The controller includes a TOE controller 306 for loading and reducing a load of the main CPU, and a disk controller 309 for performing input / output to the disk storage device 310.

When there are multiple disk storage devices, they are used in redundant array of inexpensive disks (RAID), that is, in mass storage devices that consist of multiple hard disks as one virtual disk. It is essential to perform an XOR (Exclusive OR) operation to prevent redundant storage of data and duplicate withdrawal of recorded data.

In response, the I / O processor 304 can perform this by using a dedicated XOR (Exclusive OR) engine. In addition, when the XOR engine is used, parity check for error detection during data transmission and reception between the iSCSI initiator and the target system can be performed at high speed.

In FIG. 3, the local memory 307 is connected to an internal bus of the I / O processor 304, but is connected to an I / O processor 304 by an apparatus such as the network controller 305 and the TOE controller 306 and the disk controller 309. The local memory 307 is directly accessed through the / O operation processor 304 to store and retrieve data.

If the data on the storage space is not contiguous, there are various ways to store and retrieve it as one complete data. Among them, in particular, a gathering function for directly collecting and using data blocks and a scattering function for distributing and storing data in non-contiguous storage space under the premise of such gathering function.

Even in a network system, when the IP header, the TCP header, the UDP header and the payload are not contiguous in memory, the network controller 305 uses the scatter / gather function to send data. Can transmit

In addition, when the network controller 305 separately takes charge of the checksum function for accurate data transmission, the CPU load can be reduced. The checksum is used to check whether there is a change in data when there is a movement of data on the network system.The sender calculates a checksum for the data to be transmitted and stores it in a header. It calculates, compares with the checksum stored in the header, and if it matches, acknowledges that it was received normally, otherwise discards the received data.

The network controller 305 has a scatter / gather function that can transmit an IP header, a TCP header, a UDP header, and a payload even when the data is not contiguous in the local memory 307. It is assumed that the dedicated hardware capable of performing this function can perform the TCP / IP / UDP checksum offloading function which handles the checksum calculation performed in the TCP, IP or UDP layer instead of the CPU.

The TOE / TCP Offloading Engine (TOE) will first be described with reference to the TOE controller 306. Currently TCP / IP is the most widely used communication protocol between computers, but unfortunately TCP / IP is acting as the largest load on the system's target CPU. Generally, 1Hz CPU cycles are required to process 1bit / sec TCP / IP data. At the previous 10 / 100Mbps network speeds, there was no significant difficulty in handling TCP / IP by the CPU alone. But with the advent of Gigabit Ethernet, the target CPU spent most of its cycles processing TCP / IP.

The TCP / IP Offload Engine (TOE) developed in this way allows the CPU / IP packet processing load to be processed by the NIC (Network Interface Card), thereby greatly reducing the CPU load.

The process of transmitting / receiving data by the current TCP / IP protocol divides the data according to the MTU (Maximum Transmission Unit) size and adds a TCP / IP header to transmit the data, thereby removing the headers and combining the MTUs again. This TCP / IP stack processing is very expensive. In order to improve this, the TCP / IP stack processing is separated from the processing of the CPU and assigned to dedicated hardware (Engine), and in the present invention, the TOE controller 306 is configured.

That is, the TOE controller 306 hardwarely handles the load of TCP / IP packet processing of the main CPU 301 by performing TCP / IP stack offloading instead of the main CPU 301.

Media access control (MAC) is one of the network layers existing in the physical layer and the data link layer of the hardware, and is responsible for physical address, frame synchronization, and error checking. The TOE controller 306 can directly handle this MAC layer to directly handle the physical address to reduce the load on the main CPU and to process data at high speed.

In general OS (Operating System) server, the process of reading file from disk device and transferring it to network is the process of reading data from disk device to kernel buffer in kernel, data from kernel buffer to user memory area designated by user program. Copying the data from the user memory area to the socket buffer in the kernel, and transferring the contents of the socket buffer to the direct memory access (DMA) by the network controller.

In the above-described process, two inter-memory copies occur. This copy-to-memory copy not only occupies memory bandwidth, but also uses the main CPU to copy, resulting in an absolutely high CPU share, especially for high-transmission applications such as streaming services. In addition, since data is transferred from the SCSI controller to main memory and back from the main memory to the network interface, more than twice the actual data transfer occupies the bandwidth of the main PCI bus.

As a way to reduce such inter-memory copying, there is a zero-copy function for directly transferring data without using a user buffer area and using a kernel buffer without copying between other memories.

In order to quickly transmit data on the network system, the network controller 305 of the present invention has a zero copy function for directly transmitting data through the local memory 307 without going through another memory copying process as described above. It is assumed that can be implemented.

4 is a diagram illustrating a movement path of data when a read command is performed on an iSCSI target system according to an exemplary embodiment of the present invention in the internal configuration diagram of FIG. 3.

When a read command is simply sent to the iSCSI target system, the iSCSI target system reads data from the connected disk storage device and transmits the data to the network system to perform the read command. A closer look at this process follows.

Referring to FIG. 4, first, the disk controller 309 reads data requested from the disk storage device 310 and stores the data requested in the disk storage device 310 in the local memory 307 of the I / O processor 304. Next, according to the flow 402, the TOE controller 306 reads the data, offloads the entire TCP / IP stack without interference from the main CPU 301, and then returns the local memory of the I / O processor 304. 307). Finally, according to flow 403, the network controller 305 transmits the offloaded data to the network system using the TCP / IP / UDP checksum function and the scatter / gather function.

FIG. 5 illustrates a movement path of data when a write command is performed on an iSCSI target system according to an exemplary embodiment of the present invention in the internal configuration diagram of FIG. 3.

Simply, when a write command is issued to an iSCSI target system, the target system receives data from the network system and executes the write command by storing the data in a disk storage device attached to the iSCSI target system. A closer look at this process follows.

Referring to FIG. 5, first, according to the flow 501, the network controller 305 transmits data transmitted by TCP / IP / UDP checksum and scatter / gather to the I / O processor 304. Store in the local memory 307. Next, according to flow 502, the TOE controller 306 reads the data stored in the local memory 307 of the I / O processor 304, offloads the entire TCP / IP stack without interference from the main CPU, and then reloads it. It is stored in the local memory 307 of the I / O processor 304. Finally, according to flow 503, the disk controller 309 stores the offloaded data in the corresponding disk storage device 310.

6 is a flowchart illustrating a method of performing a read / write command on an iSCSI target system according to an embodiment of the present invention. That is, as a flowchart of a data processing method for an iSCSI read / write command, the data movement paths of FIGS. 4 and 5 are expressed together as a flowchart.

Referring to FIG. 6, the iSCSI target system checks what an iSCSI command is issued from the iSCSI initiator (S601).

As a result, when the command is a read command, the data desired to be read out is extracted from the disk storage device 310 and stored in the local memory 307 (S602). Next, the stored data is retracted from the local memory 307 (S603). The TCP / IP stack offloading of the retrieved data is performed by the TOE controller 306 (S604). Next, the offloaded data is stored in the local memory 307 again (S605). Finally, the stored data is extracted from the local memory 307 and transmitted to the network system (S606).

If the command is a write command, the data received from the network system is stored in the local memory 307 (S607). Next, the stored data is extracted from the local memory 307 (S608). The TCP / IP stack offloading of the retrieved data is performed by the TOE controller 306 (S609). Next, the offloaded data is stored in the local memory 307 again (S610). Finally, the data is extracted from the local memory 307 and stored in the disk storage device 310 (S611).

In this process, the components performed by each component to reduce the load of the main CPU will be described in detail.

When the read command is transmitted to the network system in step S606 of FIG. 6, a scatter / gather function and a TCP / IP / UDP checksum offloading function are supported. do.

In addition, a zero-copy function of transmitting the data in the local memory 307 to the network system without excluding an inter-memory copy is performed.

In steps S604 and S609 of FIG. 6, the TOE controller 306 reads the data from the local memory 307 and directly handles a MAC (Media Access Control) to process the data without interfering with the main CPU 301. It is assumed that the entire TCP / IP stack can be offloaded.

In a RAID system of a plurality of disk storage devices 310, in order to stably store data and improve performance, a dedicated XOR engine is used to implement an XOR operation or calculate a parity operation.

The invention can also be embodied as computer readable code on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD_ROM, magnetic tape, floppy disks, and optical data storage, and may also include those implemented in the form of carrier waves (eg, transmission over the Internet). . The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

As described above, optimal embodiments have been disclosed in the drawings and the specification. Although specific terms have been used herein, they are used only for the purpose of describing the present invention and are not intended to limit the scope of the present invention as defined in the claims or the claims. Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible from this. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

When using the hardware accelerator on the iSCSI target system using the TOE as described above, TCP / IP protocol processing, RAID function support, parity by offloading and processing the TCP / IP stack, which is the bottleneck in the implementation of the iSCSI target system Acceleration of hardware functions in operations and the like reduces the load on the main CPU and enables data input and output at high speed.

In addition, it minimizes the number of copy-to-memory copies during iSCSI protocol processing, eliminating bottlenecks that can occur in the PCI bus, thereby improving data processing speed.

Claims (13)

A local memory for storing data transferred between the disk storage device and the network system; A TCP / IP Offload Engine (TOE) controller that processes TCP / IP packets of data stored in the local memory to perform TCP / IP stack offloading; A network controller configured to transmit data offloaded from the local memory to the network system using an iSCSI protocol, or to store data received from the network system in the local memory; And And a disk controller for performing data storage and retrieval between the disk storage device and the local memory. The method of claim 1, An I / O processor for controlling data input / output to the local memory; And And a local bus connecting each of the components to serve as a data transmission path. The method of claim 2, And the local bus is a PCI bus. The method of claim 2, The I / O processor is an XOR engine, the hardware acceleration device on the iSCSI target system, characterized in that performing a XOR operation and a parity operation of the iSCSI protocol for distributed processing of the data in a plurality of disk storage devices (RAID). The method of claim 1, The network controller may have a scatter / gather function and / or TCP that may transmit IP headers, TCP headers, UDP headers and payloads in the local memory even if they are not contiguous in the local memory. And a TCP / IP / UDP checksum offloading function for processing a checksum calculation performed on an IP or UDP layer on behalf of a CPU. The method of claim 1, The TOE controller performs TCP / IP offloading by handling a physical layer (MAC) on a network without interference of the main CPU. And the network controller implements a zero-copy implementation of directly transmitting the data through the local memory without going through another memory copy process. (a) retrieving desired data from the disk storage device and storing the data in local memory; (b) processing TCP / IP packets of the stored data to perform TCP / IP stack offloading; And (c) transmitting the offloaded data to a network system. The method of claim 7, wherein step (b) (b1) extracting the stored data from the local memory; (b2) performing a TCP / IP stack offloading by processing a TCP / IP packet of the retrieved data by a TOE controller; And (b3) storing the offloaded data back into the local memory; and executing the read command on the iSCSI target system.  The method of claim 8, In the step (b2), the TCP controller processes the TCP / IP packet of the retrieved data and performs TCP / IP stack offloading while handling the physical layer (MAC) on the network without interference of the main CPU by the TOE controller. Method of performing a read command on the iSCSI target system, characterized in that. The method of claim 9, wherein in step (c) Scatter / Gather function that can transmit data header even if protocol header and payload are not continuous in the local memory,  TCP / IP / UDP Checksum Offloading, which handles checksum calculations performed on the TCP, IP or UDP layer on behalf of the CPU, and ISCSI read command on the iSCSI target system, characterized in that it has at least one or more of the zero-copy (Zero-Copy) function to directly transfer data through the local memory without going through another memory copying process when transmitting data to the network system How to do it. (a) storing the data received from the network system into a local memory; (b) processing TCP / IP packets of the stored data to perform TCP / IP stack offloading; And (c) storing the offloaded data on a disk storage device. The method of claim 11, wherein step (b) (b1) fetching the data from the local memory; (b2) performing a TCP / IP stack offloading by processing a TCP / IP packet of the retrieved data by a TOE controller; And (b3) storing the data in the local memory again. The method of claim 12, Step (c) is a write command performed on the iSCSI target system, characterized in that the XOR operation and the parity operation of the iSCSI protocol for data distribution processing in a plurality of disk storage (RAID) devices using the XOR engine during data storage. Way.

Images