Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
  • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
  • G06F3/0658—Controller construction arrangements
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
  • G06F3/0604—Improving or facilitating administration, e.g. storage management
  • G06F3/0605—Improving or facilitating administration, e.g. storage management by facilitating the interaction with a user or administrator
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
  • G06F3/061—Improving I/O performance
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
  • G06F3/0629—Configuration or reconfiguration of storage systems
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
  • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L67/00—Network arrangements or protocols for supporting network services or applications
  • H04L67/01—Protocols
  • H04L67/10—Protocols in which an application is distributed across nodes in the network
  • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Definitions

• FIG. 1 In a Storage Area Network (SAN) architecture, the division is typically set forth as described in Figure 1.
• software functionality managing block related functionality such as the block virtualization layer 134 and block cache management 136, are implemented on a separate storage subsystem.
• Higher-level functionality such as file system functionality 124 and other data management functionality 122 is implemented on a traditional computer system operating as a server 102.
• Examples of data management functionality include databases, data life cycle management software, hierarchical storage management software, and specialized software such as PACCS software used in the medical industry.
• PACCS software used in the medical industry.
• Various data management software systems may be used in combination with each other.
• Communication between the server 102 and the storage subsystem 104 involves industry standard protocols such as Fibre Channel 142 driven by layers of device drivers 126 and 128 on the server side and target drivers 130 and 132 on the storage subsystem side.
• This physical network 142 combined with layers of device drivers and target software adds considerable latency to I/O operations.
• Positioning the file system within the server makes heterogeneous operation a challenge as building a single file system that supports multiple operating systems is non-trivial.
• What is commonly referred to as Network Attached File Systems (NAS), as shown in Figure 2 moves most of the file system functionality 132 into the .storage subsystem.
• Industry standard protocols, such as NFS and CIFS allow multiple operating systems to communicate to a single file system image.
• multiple heterogeneous servers can share a single file system.
• Communication between the server 202 and the storage subsystem 204 typically uses common networks such as Ethernet.
• a server is embedded directly into a storage subsystem.
• Data management functionality written for traditional servers may be implemented within a stand-alone storage subsystem, generally without software changes to the ported subsystems.
• the hardware executing the storage subsystem and server subsystem are implemented in a way that provides reduced or negligible latency, compared to traditional architectures, when communicating between the storage subsystem and the server subsystem.
• a plurality of clustered controllers are used.
• traditional load-balancing software can be used to provide scalability of server functions.
• One end-result is a storage system that provides a wide range of data management functionality, that supports a heterogeneous collection of clients, that can be quickly customized for specific applications, that easily leverages existing third party software, and that provides optimal performance.
• a method for embedding functionality normally present in a server computer system into a storage system.
• the method typically includes providing a storage system having a first processor and a second processor coupled to the first processor by an interconnect medium, wherein processes for controlling the storage system execute on the first processor, porting an operating system normally found on a server system to the second processor, and modifying the operating system to allow for low latency communications between the first and second processors.
• a storage system typically includes a first processor configured to control storage functionality, a second processor, an interconnect medium communicably coupling the first and second processors, an operating system ported to the second processor, wherein said operating system is normally found on a server system, and wherein the operating system is modified to allow low latency communication between the first and second processors.
• a method is provided for optimizing communication performance between server and storage system functionality in a storage system.
• the method typically includes providing a storage system having a first processor and a second processor coupled to the first processor by an interconnect medium, porting an operating system normally found on a server system to the second processor, modifying the operating system to allow for low latency communications between the first and second processors, and porting one or more file system and data management applications normally resident on a server system to the second processor.
• a method for implementing clustered embedded server functionality in a storage system controlled by a plurality of storage controllers.
• the method typically includes providing a plurality of storage controllers, each storage controller having a first processor and a second processor communicably coupled to the first processor by a first interconnect medium, wherein for each storage controller, an operating system normally found on a server system is ported to the second processor, wherein said operating system is allows low latency communications between the first and second processors.
• the method also typically includes providing a second interconnect medium between each of said plurality of storage controllers.
• the second communication medium may handle all inter-processor communications.
• a third interconnect medium is provided in some aspects, wherein inter-processor communications between the first processors occur over one of the second and third mediums and inter- processor communications between the second processors occur over the other one of the second and third mediums.
• a storage system that implements clustered embedded server functionality using a plurality of storage controllers.
• the system typically includes a plurality of storage controllers, each storage controller having a first processor and a second processor communicably coupled to the first processor by a first interconnect medium, wherein for each storage controller, processes for controlling the storage system execute on the first processor, an operating system normally found on a server system is ported to the second processor, wherein said operating system is allows low latency communications between the first and second processors, and one or more file system and data management applications normally resident on a server system are ported to the second processor.
• the system also typically includes a second interconnect medium between each of said plurality of storage controllers, wherein said second interconnect medium handles inter- processor communications between the controller cards.
• a third interconnect medium is provided in some aspects, wherein inter-processor communications between the first processors occur over one of the second and third mediums and inter-processor communications between the second processors occur over the other one of the second and third mediums.
• FIG. 1 illustrates traditional storage area network (SAN) software towers.
• FIG. 2 illustrates traditional network attached storage (NAS) software towers.
• NAS network attached storage
• FIG. 3 illustrates a server tower embedded in a storage system, such as a storage controller node, according to one embodiment of the present invention.
• FIG. 4 illustrates embedded server hardware in a storage system, such as a storage controller node, according to one embodiment of the present invention.
• FIG. 5 illustrates an alternate I/O module supporting Infiniband according to one embodiment of the present invention.
• FIG. 6 illustrates an alternate I/O module supporting 8-Gigabit Ethernet ports according to one embodiment of the present invention.
• FIG. 7 illustrates embedded server software modules according to one embodiment.
• FIG. 8 illustrates a memory allocation scheme according to one embodiment.
• the data management functionality is moved within the storage subsystem, i order to maximize the utilization of existing software, including third party software, and to minimize porting effort
• the data management functionality is implemented as two separate software towers running on two separate microprocessors. While any high speed communication between the processors could be used, a preferred implementation involves implementing hardware having two (or more) microprocessors that are used to house a storage software tower and a server software tower, but allowing each microprocessor having direct access to a common memory.
• An example of a server tower embedded in a storage system according to one embodiment is shown in Figures 3 and 4.
• both processors 410 and 412 can access both banks of memory 420 and 422 via the HyperTransportTM bus 330.
• the HyperTransportTM architecture is described in http://www.hvpertransport.org/tech specifications.htmL which is hereby incorporated by reference. It will be apparent to one skilled in the art that alternate bus architectures may be used, such as Ethernet, a system bus, PCI, proprietary networks and busses, etc.
• the processors 410 and 412, bus 430 and memory 420 and 422 in Figure 4 are implemented in a single storage controller node , e.g., in a single NetStoragerTM controller card as shown in FIG. 3, according to one embodiment.
• processor virtualization software can be used to emulate two separate processors executing a single 'real' processor. It will also be apparent that the software tower can run as a task of the server tower.
• connectors 430 are used to connect the I/O portions of the hardware. This allows alternate I/O modules to be used to provide alternate host protocol connections such as hifiniBand®, e.g. as shown in Figure 5, or to increase Ethernet connectivity, e.g. as shown in Figure 6. The preferred implementation allows resulting I/O ports to be assigned to either software tower as desired.
• a second tower that normally runs on an external server 706 is placed on the second processor, e.g., processor 412 of FIG. 4.
• a traditional operating system, such as Linux 756 is ported to the second processor and used to host the overlying software layers. This allows easy adoption, usually without modification, of existing software designed for a traditional server environment.
• the common memory e.g., memory 420 and 422 of FIG. 4, is partitioned into two regions, one for each software tower.
• a small common region is reserved for managing data structures involved with inter-processor communications.
• a two-way mailbox algorithm is used in one aspect for communicating between the "shared memory device drivers" running on each of the two processors as follows. Each processor maintains a work queue for the other processor. Only the initiator links work onto the end of the queue. When one processor "A” needs to message processor "B" of a communication, the following steps occur in one aspect:
• Processor A (the initiator) allocates a control block from its own memory space. 2. Processor A sets the "completed" flag to false. 3. Processor A fills in other fields as required by the request. 4. Processor A links the request on the end of a linked list of requests destined for processor B. 5. ⁇ Processor A notifies processor B via an interrupt or event trap of the presence of work in the queue. 6. Processor B starts at the top of the queue processing uncompleted requests. When a request is completed, Processor B sets the respective "completed" flag to True and provides an interrupt to Processor A. 7. Processor A begins at the top of the queue, noting which transactions have been completed and unlinking them. The order of storing addresses is important to ensure that transactions can be unlinked without a semaphore.
• an integer field representing priority is included and the link is scanned multiple times looking for decreasing priorities of requests.
• data buffers are pre-allocated by the request target and can be used by the source processor to receive actual data
• the processor initiating the request is responsible for copying the data blocks from its memory to the pre-allocated buffer on the receiving processor.
• the actual data coping is deferred until deemed more convenient, thus minimizing latency associated with individual transactions.
• This is preferably done without modifications outside the device driver layer of the Linux operating system; e.g. during a write operation, by "nailing" the I/O page to be written and using the Linux page image for the I/O operations in the storage system.
• the page can be replicated as a background function on the Storage System processor (the processor implementing storage system control functionality).
• the Server Device Driver is notified that the page is now "clean” and can be "un-nailed.”
• the virtual memory management modules of both the Sever operating system and the storage system work cooperatively in using common I O buffers, thus advantageously avoiding unnecessary copies and minimizing the redundancy of space usage.
• MMUs memory management units
• multiple storage system controller nodes are clustered together.
• the concept of clustering controllers was introduced in US Patent No. 6, 148,414. Additional refinements of clustered controllers were introduced in US Application No. 2002/0188655.
• One advantageous result of implementing aspects of the present invention in multiple storage system controllers is that multiple Storage System Towers can export a given virtual volume of storage to multiple embedded servers. The performance scales as additional Storage System towers are added.
• Clustered file systems are now common, wherein multiple file system modules running on multiple servers can export to their host a common file system image.
• An example of a clustered file system is the Red Hat Global File System (http://www.redhat.com/software/rha/gfs/). If the file system 726 (FIG. 7) chosen is a clustered file system, then software layers above the file system, regardless what controller they are residing on, will see a common file system image.
• Data Management Applications 722 that support multiple invocations to a single file image will now scale as more storage controller modules are added. Examples of software that can benefit from this environment include web servers and parallel databases. I/O intensive applications, such as data mining applications, obtain significant performance benefits from running directly on the storage controller.
• the file system allocates its buffer space using the common buffer allocation routines described above.
• the buffers are the largest storage consumer of a file system. Allocating them from the common pool 810, rather than the Server Tower specific pool 840, optimizes the usage of controller memory and makes the overall system more flexible.
• Porting common software that balances application software execution load between multiple servers, such as LSF from Platform computing, onto the server tower 724 allows single instance applications to benefit from the scalability of the overall platform.
• the load-balancing layer 724 moves applications between controllers to balance the execution performance of controllers and allow additional controllers to be added to a live system to increase performance.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Human Computer Interaction (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Computer Networks & Wireless Communication (AREA)
• Signal Processing (AREA)
• Multi Processors (AREA)
• Stored Programmes (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Applications Claiming Priority (2)

Publications (2)

Family

ID=34135305

Family Applications (1)

Country Status (3)

Family Cites Families (6)

• 2004
  • 2004-08-06 EP EP04780250A patent/EP1668518A4/de not_active Withdrawn
  • 2004-08-06 WO PCT/US2004/025383 patent/WO2005015349A2/en not_active Ceased
  • 2004-08-06 CA CA002535097A patent/CA2535097A1/en not_active Abandoned

Also Published As

Similar Documents

Legal Events