JP2011108258A - System and method of virtual disk drive - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an improved system for dynamically allocating data. <P>SOLUTION: The system includes a RAID subsystem having a pool of storage (a page pool of storage that maintains a free list of RAIDs and the like) or a matrix of disk storage blocks that maintain a null list of RAIDs and a disk manager having a disk storage system control device. The subsystem and the manager dynamically allocate data across the pool of storage and a disk drive and determine whether additional disk drives are required, and a notification is sent if necessary. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates generally to disk drive systems and methods, and more particularly to disk drive systems having functions such as dynamic data allocation and disk drive virtualization.

  Existing disk drive systems are designed so that the virtual volume data storage space is statically associated with a physical disk having a specific size and location for storing data . In such disk drive systems, it is necessary to know and monitor / control the exact location and size of the virtual volume in the data storage space in order to store the data. In addition, this system often requires more data storage space, thus adding RAID devices. Often, however, such additional RAID devices are expensive and are not required until extra data storage space is actually required.

  FIG. 14A illustrates a conventional existing disk having virtual volume data storage space associated with a physical disk having a specific size and location for data storage, read / write, and / or recovery. Indicates drive system. In this disk drive system, data allocation is done statically based on the specific location and size of the virtual volume in the data storage space. As a result, empty data storage space is not used, and extra and sometimes expensive data storage devices, such as RAID devices, are acquired in advance to store, read / write, and / or recover data in this system. Is done. Such extra data storage space is not needed and / or may not be used until a later time.

  Accordingly, there is a need for improved disk drive systems and methods. There is a further need for efficient dynamic data allocation and disk drive space and time management systems and methods.

  The present invention provides an improved disk drive system and method with the ability to dynamically allocate data. The disk drive system can include a RAID subsystem having a matrix of disk storage blocks and a disk manager having at least one disk storage system controller. The RAID subsystem and the disk manager are based on RAID-to-disk mapping (RAID-to-disk mapping) through a matrix of disk storage blocks and multiple disk drives. Assign data dynamically. The RAID subsystem and disk manager determine whether additional disk drives are needed and send notifications if additional disk drives are needed. Dynamic data allocation allows the user to obtain a disk drive at a later time when needed. Dynamic data allocation for efficient data storage, data backup, recovery, etc. for disk storage block virtual volume matrix or pool snapshot / point-in-time (hour) copy Instant data replay and data instant fusion, remote data storage, and data progression. Data progression also allows for cheaper disk drives to be postponed because they are purchased at a later time.

  In one embodiment, a matrix or pool of virtual volumes or disk storage blocks is provided for associating with physical disks. Monitoring / control of a matrix or pool of virtual volumes or disk storage blocks is performed dynamically by a plurality of disk storage system controllers. In one embodiment, the size of each virtual volume can be default or user-predefined, and the location of each virtual volume defaults to null. The virtual volume is null until data is allocated. Data can be assigned to any grid in a matrix or pool (eg, after data is assigned to that grid, it becomes a “dot” in that grid). After the data is deleted, the virtual volume becomes available again and is shown as “null”. Thus, extra data storage space and sometimes expensive data storage devices, such as RAID devices, can be obtained at a later time as needed.

  In one embodiment, a single disk manager can manage multiple disk storage system controllers, and multiple redundant disk storage system controllers can be implemented and operated. System controller failures can be covered.

  In one embodiment, the RAID subsystem includes a combination of at least one of RAID types such as RAID-0, RAID-1, RAID-5, RAID-10. It will also be appreciated that other RAID types may be used in alternative RAID subsystems, such as RAID-3, RAID-4, RAID-6, RAID-7, etc.

  The present invention also provides a method for dynamic data allocation, which provides a default size for logical blocks or disk storage blocks so that the disk space of the RAID subsystem is disk storage storage. Based on the RAID subsystem disk space occupancy history, writing the data, assigning the data to a disk storage block matrix, and creating a RAID matrix disk space history. Determining disk space occupancy, determining whether additional disk drives are required, and sending notification to the RAID subsystem when additional disk drives are required. Including. In one embodiment, the notification is sent by email.

  One advantage of the disk drive system of the present invention is that the RAID subsystem can use RAID techniques across a virtual number of disks. The remaining storage space can be used freely. By monitoring the storage space of the RAID subsystem and determining the occupancy of the storage space, the user does not have to obtain a large number of drives that are expensive but are not used at the time of purchase. Thus, the overall cost of the disk drive is significantly reduced by adding drives when they are actually needed to meet the growing demand for storage space. On the other hand, the efficiency of use of the drive is significantly improved.

  Another advantage of the present invention is that the disk storage system controller is universal for any computer file system, not just a particular computer file system.

  The present invention also provides a method of data instant replay. In one embodiment, the data instant replay method provides a default size for logical blocks or disk storage blocks, and the RAID subsystem disk space is a page pool of storage or a matrix of disk storage blocks. Automatically generating snapshots of a storage page pool volume or disk storage block matrix at predetermined time intervals, snapshots or variations The (delta) address index is stored in the storage page pool or disk storage block matrix, and a disk storage block matrix snapshot. The object or delta, and a step to make Mitsukedaseru immediately via the address index and its storage.

  In this method of data instant replay, the automatic generation of RAID subsystem snapshots can be performed at user-defined time intervals or by user-configured dynamic time stamps, eg, minutes or hours. Every time, or at the time indicated by the server. In the event of a system failure or virus attack, virtual snapshots with such time stamps allow data instant replay and data instant recovery in about minutes or hours. This technique is also called instant replay fusion. That is, the data immediately before the crash or attack is fused without delay, and the snapshot saved before the crash or attack can be used immediately in subsequent operations.

  In one embodiment, the snapshot is stored on a local RAID subsystem or a remote RAID subsystem, and for data integrity when a serious system crash occurs, for example, due to a terrorist attack The data can be recovered immediately.

  Another advantage of this data instant replay method is that the snapshot can be used for inspection while the system continues to operate. Live data can be used for real-time inspection.

  The present invention also provides a data instant replay system including a RAID subsystem and a disk manager having at least one disk storage system controller. In one embodiment, the RAID subsystem and disk manager dynamically allocates data across the disk space of multiple drives based on RAID-to-disk mapping, where the RAID subsystem disks The space forms a matrix of disk storage blocks. The disk storage system controller automatically generates a snapshot of the disk storage block matrix at predetermined time intervals and stores the snapshot or delta address index in the disk storage block matrix. Thus, a snapshot or delta of a disk storage block matrix can be immediately found via its stored address index.

  In one embodiment, the disk storage system controller monitors the frequency of data usage from a snapshot of the disk storage block matrix, and the less frequently used or accessed data, the less expensive RAID sub- Apply aging rules, such as moving to the system. Similarly, if data in an inexpensive RAID subsystem begins to be used frequently, the controller moves that data to a more expensive RAID subsystem. Thus, the user can select a desired RAID subsystem portfolio that meets the user's own storage needs. Thus, the cost of the disk drive system can be significantly reduced and dynamically controlled by the user.

  These and other features and advantages of the present invention will be apparent to those skilled in the art from the following detailed description, which illustrates and describes exemplary embodiments of the present invention, including the best mode contemplated for carrying out the invention. It will be. It will be understood that the invention is capable of modifications in various obvious aspects, all without departing from the spirit and scope of the invention. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.

FIG. 1 illustrates one embodiment of a disk drive system in a computer environment in accordance with the principles of the present invention. FIG. 2 illustrates one embodiment of dynamic data allocation with a paged pool of storage for a disk drive RAID subsystem in accordance with the principles of the present invention. FIG. 2A illustrates conventional data allocation in a RAID subsystem of a disk drive system. FIG. 2B illustrates data allocation in a RAID subsystem of a disk drive system in accordance with the principles of the present invention. FIG. 2C illustrates a dynamic data allocation method according to the principles of the present invention. 3A and 3B are schematic diagrams of snapshots of disk storage blocks of a RAID subsystem at multiple time intervals in accordance with the principles of the present invention, and C is in accordance with the principles of the present invention. Data instant replay method is shown. FIG. 4 is a schematic diagram of data instant fusion functionality by using a snapshot of a RAID subsystem disk storage block in accordance with the principles of the present invention. FIG. 5 is a schematic diagram of local / remote data replication and instant replay functionality by using RAID subsystem disk storage block snapshots in accordance with the principles of the present invention. FIG. 6 shows a schematic diagram of a snapshot that connects multiple RAID devices into a single volume using the same RAID interface for performing I / O in accordance with the principles of the present invention. FIG. 7 illustrates one embodiment of a snapshot structure in accordance with the principles of the present invention. FIG. 8 illustrates one embodiment of a PITC life cycle in accordance with the principles of the present invention. FIG. 9 illustrates one embodiment of a PITC table structure with a multi-level index in accordance with the principles of the present invention. FIG. 10 illustrates one embodiment of PITC table recovery in accordance with the principles of the present invention. FIG. 11 illustrates one embodiment of a writing process having owned and non-owned page sequences in accordance with the principles of the present invention. FIG. 12 illustrates an exemplary snapshot operation in accordance with the principles of the present invention. FIG. 13A shows a conventional existing disk drive system having virtual volume data storage space associated with a physical disk having a specific size and location for statically allocating data. FIG. 13B illustrates volume logical block mapping in the conventional existing disk drive system of FIG. 13A. FIG. 14A illustrates one embodiment of a disk drive system having a virtual volume matrix of disk storage blocks for dynamically allocating data in the system in accordance with the principles of the present invention. FIG. 14B illustrates one embodiment of dynamic data allocation in the virtual volume matrix of disk storage blocks shown in FIG. 14A. FIG. 14C shows a schematic diagram of a volume-RAID page remapping of one embodiment of a virtual volume page pool of storage in accordance with the principles of the present invention. FIG. 15 illustrates an example of three disk drives mapped to multiple disk storage blocks of a RAID subsystem in accordance with the principles of the present invention. FIG. 16 shows an example of remapping of a disk drive storage block after adding one disk drive to the three disk drives shown in FIG. FIG. 17 illustrates one embodiment of accessible data pages in a data progression operation in accordance with the principles of the present invention. FIG. 18 shows a flow diagram of one embodiment of a data progression process in accordance with the principles of the present invention. FIG. 19 illustrates one embodiment of a compressed page layout in accordance with the principles of the present invention. FIG. 20 illustrates one embodiment of data progression in a high level disk drive system in accordance with the principles of the present invention. FIG. 21 illustrates one embodiment of external data flow in a subsystem in accordance with the principles of the present invention. FIG. 22 illustrates one embodiment of internal data flow in the subsystem. FIG. 23 illustrates one embodiment of each subsystem that maintains coherency independently. FIG. 24 illustrates one embodiment of mixed RAID waterfall data progression in accordance with the principles of the present invention. FIG. 25 illustrates one embodiment of multiple free lists of storage page pools in accordance with the principles of the present invention. FIG. 26 illustrates one embodiment of an example database in accordance with the principles of the present invention. FIG. 27 illustrates one embodiment of an example MRI image in accordance with the principles of the present invention.

  The present invention provides an improved disk drive system and method with the ability to dynamically allocate data. The disk drive system includes at least one RAID subsystem having a page pool of storage that maintains a RAID free list, or alternatively a matrix of disk storage blocks. And a disk manager having two disk storage system controllers. The RAID subsystem and disk manager dynamically allocates data across a storage page pool or matrix of disk storage blocks and multiple disk drives based on a RAID-to-disk mapping. The RAID subsystem and disk manager determine whether additional disk drives are needed and send notifications if additional disk drives are needed. Dynamic data allocation allows the user to obtain a disk drive later when it is needed. Dynamic data allocation also enables efficient data storage of disk storage block virtual volumes matrix or pool snapshot / point-in-time copy, data backup and recovery, etc. Enables instant data replay and data instant fusion, remote data storage, data progression, and more. Data progression also allows for cheaper disk drives to be purchased at a later time and postponed.

  FIG. 1 illustrates one embodiment of a disk drive system 100 in a computer environment 102 in accordance with the principles of the present invention. As shown in FIG. 1, the disk drive system 100 includes a RAID subsystem 104 and a disk manager 106 having at least one disk storage system controller (FIG. 16). The RAID subsystem 104 and the disk manager 106 dynamically allocate data across the disk space of multiple disk drives 108 based on a RAID-to-disk mapping. In addition, the RAID subsystem 104 and the disk manager 106 are capable of making a determination of whether additional disk drives are required based on data allocation across the entire disk space. If additional disk drives are needed, a notification is sent to the user, and additional disk space can be added if desired.

  A disk storage system 100 with dynamic data allocation (or referred to as “disk drive virtualization”) in accordance with the principles of the present invention is shown in the form of one embodiment in FIG. A and FIG. 14B show another embodiment. As shown in FIG. 2, the disk storage system 110 includes a storage page pool 112, ie, a pool of data storage that includes a list of data storage spaces where data can be freely stored. The page pool 112 maintains a free list of RAID devices 114 and manages read / write assignments based on user requests. The user requested data storage volume 116 is sent to the page pool 112 to obtain storage space. Each volume can request the same or different class of storage devices at the same or different RAID levels, eg, RAID 10, RAID 5, RAID 0, etc.

  Another embodiment of the dynamic data allocation of the present invention is illustrated in FIGS. 14A and 14B, in this example, a plurality of disk storage system controllers 1402 and a plurality of disk storages. A disk storage system 1400 having a matrix of disk storage blocks 1404 controlled by a system controller 1402 dynamically allocates data in that system in accordance with the principles of the present invention. A virtual volume or block matrix 1404 is provided to be associated with the physical disk. The virtual volume or block matrix 1404 is dynamically monitored / controlled by a plurality of disk storage system controllers 1402. In one embodiment, the size of each virtual volume 1404 can be predetermined, eg, 2 megabytes, and the location of each virtual volume 1404 is null by default. Each of the virtual volumes 1404 is null until data is allocated. Data can be assigned to any grid in the matrix or pool (eg, after the data is assigned to the grid, it becomes a “dot” in the grid). After the data is deleted, the virtual volume 1404 becomes available again and is shown as “null”. Thus, an extra and sometimes expensive data storage device, such as a RAID device, can be obtained later when needed.

  Thus, the RAID subsystem can use RAID techniques across a virtual number of disks. The remaining storage space can be used freely. By monitoring the storage space of the RAID subsystem and determining the occupancy of the storage space, the user does not have to obtain a large number of drives that are expensive but are not used at the time of purchase. In this way, adding the drive when it is actually needed to meet the growing demand for storage space significantly reduces the overall cost of the disk drive. On the other hand, the efficiency of use of the drive is also significantly improved.

  The dynamic data allocation of the disk drive system of the present invention also allows the snapshot / point-in-time of the virtual volume page pool of storage or the virtual volume matrix of disk storage blocks. Enables copy efficient data storage, instant data replay and data instant fusion for data recovery and remote data storage, and data progression.

  The above-described features and advantages resulting from dynamic data allocation systems and methods in disk drive system 100 and implementations thereof are discussed in detail below.

Dynamic Data Allocation FIG. 2A shows traditional data allocation in a disk drive system RAID subsystem, where the freed data storage space is captive and allocated for data storage. I can't.

  FIG. 2B illustrates data allocation in a RAID subsystem of a disk drive system in accordance with the principles of the present invention, where empty data storage that can be used for data storage is Together, they form a page pool, eg, one page pool in one embodiment of the invention.

  FIG. 2C illustrates a method 200 for dynamic data allocation in accordance with the principles of the present invention. The dynamic data allocation method 200 defines a default size of logical blocks or disk storage blocks so that the disk space of the RAID subsystem forms a matrix of disk storage blocks 202. And writing 204 the data 204 and assigning the data to a disk storage block of “null” in the matrix. The method further determines step 206 that determines the disk space occupancy of the RAID subsystem based on the disk space occupancy history of the RAID subsystem and determines whether additional disk drives are required. Sending a notification to the RAID subsystem if necessary. In one embodiment, the notification is sent by email. Furthermore, the size of the disk storage block can be set to a default and can be changed by the user.

  In one embodiment, dynamic data allocation, sometimes referred to as “virtualization” or “disk space virtualization”, efficiently handles many read and write requests in seconds. This architecture may require an interrupt handler that calls the cache subsystem directly. Dynamic data allocation does not optimize requests because they do not queue requests, but can have many pending requests at once.

  Also, dynamic data allocation can maintain data integrity and protect the data content against any controller failure. To do so, dynamic data allocation writes state information to the RAID device for reliable storage.

  Dynamic data allocation can further maintain the order of read and write requests and complete read or write requests in the order in which the requests were received. Dynamic data allocation provides maximum system availability and supports remote replication of data to different geographic locations.

  Furthermore, dynamic data allocation provides a recovery function from data corruption. The snapshot allows the user to view the state of the disc in the past.

  Dynamic data allocation manages RAID devices and provides storage abstraction for creating and expanding large devices.

  In dynamic data allocation, a virtual disk device is presented to a server, and this device is called a volume. For servers, volumes work the same way. Volumes can return different information regarding serial numbers, but essentially act like a disk drive. Volumes provide a storage abstraction for multiple RAID devices to create larger dynamic volume devices. A volume includes multiple RAID devices and allows efficient use of disk space.

  FIG. 21 shows a conventional existing volume logical block mapping. FIG. 14C illustrates a volume-RAID page remapping of one embodiment of a storage virtual volume page pool in accordance with the principles of the present invention. Each volume is divided into a set of pages, eg, 1, 2, 3, etc., and each RAID is divided into a set of pages. The volume page size and the RAID page size may be the same in one embodiment. Therefore, in an example of the volume-RAID page remapping according to the present invention, page # 1 using RAID-2 is mapped to RAID page # 1.

  Dynamic data allocation maintains volume data integrity. Data is written to the volume and confirmed with the server. Data integrity applies to various controller configurations, including stand-alone and redundant, through controller failures. Controller failures include power failures, power cycles, software exceptions, and hard resets. Dynamic data allocation generally does not handle disk drive failures covered by RAID.

  Dynamic data allocation provides the highest level of data abstraction for the controller. Dynamic data allocation accepts requests from the front end and ultimately uses a RAID device to write the data to disk.

Dynamic data allocation involves several internal subsystems.
Cache-Smooth read and write operations to the volume by providing a quick response time to the server and bundling writes to the data plug-in.
Configuration—includes methods for creating, deleting, retrieving, and modifying data allocation objects. Provides components for creating toolboxes for higher level system applications.
Data plug-in—Distributes volume read and write requests to various subsystems depending on the volume configuration.
RAID interface—Provides a RAID device abstraction for creating larger volumes to users and other dynamic data allocation subsystems.
Copy / Mirror / Swap-Replicates volume data to local and remote volumes. In one embodiment, only the written block of the server can be copied.
Snapshot-provides incremental volume recovery of data. Instantly create a view volume in the past volume state.
Proxy volume—Supports remote replication by allowing communication of requests to a remote destination volume.
Billing-the ability to charge users for the recovery of allocated storage, activity, performance, and data.

  Dynamic data allocation also logs errors and significant changes in configuration.

  FIG. 21 illustrates one embodiment of external data flow in the subsystem. External requests come from the front end. The requests include volume information acquisition (get volume information), reading (read), and writing (write). Every request has a volume ID. Volume information is handled by the volume configuration subsystem. Read and write requests include the LBA. The write request includes data.

  Dynamic data allocation passes requests to several external layers depending on the volume configuration. Remote replication passes the request to the front end, but the destination is the remote destination volume. The RAID interface passes the request to the RAID. Copy / mirror / swap returns the request to dynamic data allocation for the destination volume.

  FIG. 22 illustrates one embodiment of internal data flow in the subsystem. Internal data flow begins with caching. When caching occurs, the write request is cached or the request is passed directly to the data plug-in. The cache supports direct DMA from the front end HBA device. The request is completed quickly and a response is returned to the server. The data plug-in manager is central to the request flow below the cache. For each volume, the data plug-in manager invokes the registered subsystem object for each request.

  Dynamic data allocation subsystems that affect data integrity may require support for controller coherency. As shown in FIG. 23, each subsystem maintains coherency independently. Coherency updates avoid copying data blocks over coherency links. For cache coherency, the data may need to be copied to a peer controller.

Disk Storage System Controller FIG. 14A illustrates a plurality of disk storage system controllers 1402 and a plurality of disk storages for dynamically allocating data in the system in accordance with the principles of the present invention. Shows a disk storage system 1400 with a matrix of disk storage blocks or virtual volumes 1404 controlled by the system controller 1402. FIG. 14B illustrates one embodiment of dynamic data allocation in a virtual volume matrix 1404 of disk storage blocks or virtual volumes.

  In one operation, the disk storage system 1400 generates a snapshot of a matrix 1404 of disk storage blocks or virtual volumes over time at predetermined time intervals, and generates a snapshot or delta address index on the disk. Save to a storage block or virtual volume matrix 1404 so that a snapshot or delta of the disk storage block or virtual volume matrix 1404 can be immediately found by the stored address index.

  Further, in some operations, the disk storage system controller 1402 monitors the frequency of data usage from a snapshot of the disk storage block matrix 1404, and the less frequently used or accessed data is less expensive Apply aging rules to move to the RAID subsystem. Similarly, as data in a less expensive RAID subsystem begins to be used more frequently, the controller moves that data to a more expensive RAID subsystem. Thus, the user can select the desired RAID subsystem portfolio to meet the user's own storage needs. Thus, the cost of the disk drive system is significantly reduced and dynamic control by the user is possible.

RAID-to-Disk Mapping
The RAID subsystem and disk manager dynamically allocates data across the disk space of multiple disk drives based on RAID-to-disk mapping. In one embodiment, the RAID subsystem and disk manager determine whether additional disk drives are needed and send notifications if additional disk drives are needed.

  FIG. 15 illustrates an example of three disk drives 108 (FIG. 1) mapped to a plurality of disk storage blocks 1502-1512 within a RAID-5 subsystem 1500 in accordance with the principles of the present invention. .

  FIG. 16 shows an example of disk drive storage block remapping 1600 after adding a disk drive 1602 to the three disk drives 108 shown in FIG.

Disk Manager The disk manager 106 shown in FIG. 1 generally manages disks and disk arrays, including grouping / resource pooling, disk attribute abstraction, formatting, and disk addition ( Includes tracking of addition / subtraction and disk service time and error rate. The disk manager 106 does not distinguish between the various models of disks and presents a comprehensive storage device for the RAID component. The disk manager 106 also provides a grouping function, which facilitates the creation of RAID groups with specific characteristics such as, for example, 10,000 RPM disks.

  In one embodiment of the present invention, the disk manager 106 is at least triple. The triples are abstraction, configuration, and I / O optimization. The disk manager 106 presents “disks” to an upper layer, which may be, for example, a local or remote attached physical disk drive, a remotely attached disk system, or the like.

  A common underlying feature is that any of these devices can be the target of I / O operations. Abstraction services provide a uniform data path interface, especially for higher layers such as RAID subsystems, and provide administrators with a comprehensive mechanism for managing target devices. provide.

  The disk manager 106 of the present invention also provides a disk grouping function to simplify management and configuration. You can name disks and put them in groups, and you can name groups. Grouping is a powerful feature that allows you to perform tasks such as moving a volume from one group of disks to another, dedicating a group of disks to a specific function, or a group of disks. Simplify tasks such as specifying a spare as a spare.

  The disk manager also interfaces with devices such as a SCSI device subsystem that is responsible for detecting the presence of external devices. The SCSI device subsystem has the ability to determine a subset of devices that are block target devices, at least for Fiber Channel / SCSI type devices. It is these devices that are managed and abstracted by the disk manager.

  In addition, the disk manager is responsible for responding to flow control from the SCSI device layer. The disk manager has a queuing function that provides an opportunity to aggregate I / O requests as a way to optimize the throughput of the disk drive system.

  Furthermore, the disk manager of the present invention manages a plurality of disk storage system controllers. A plurality of redundant disk storage system controllers can also be implemented to cover the failure of the disk storage system controller to be operated. This redundant disk storage system controller is also managed by the disk manager.

Relationship between the disk manager and other subsystems The disk manager interacts with several other subsystems. The RAID subsystem is an important client of services provided by the disk manager with respect to data path activity. The RAID subsystem uses the disk manager as an exclusive path to the disk for I / O. The RAID system also listens for events from the disk manager to determine the presence and operating state of the disk. The RAID subsystem also cooperates with the disk manager to allocate extents for the construction of RAID devices. The administrative control listens for disk events to obtain information about the presence of the disk and to obtain information about changes in operating conditions. In one embodiment of the present invention, the RAID subsystem 104 may include a combination of at least one of RAID types such as RAID-0, RAID-1, RAID-5, RAID-10. It will also be appreciated that other RAID types may be used in alternative RAID subsystems, such as RAID-3, RAID-4, RAID-6, RAID-7, etc.

  In one embodiment of the present invention, the disk manager uses a configuration access service to store persistent configurations and to transfer transient read-only information such as statistics etc. to the presentation layer. To present. The disk manager registers a handler for configuration access to access these parameters.

  The disk manager also uses SCSI device layer services to obtain information about the presence and operational status of block devices and has an I / O path to such block devices. The disk manager queries the SCSI device subsystem for devices as an auxiliary method for uniquely identifying a disk.

Data Instant Replay and Data Instant Fusion The present invention also provides methods for data instant replay and data instant fusion. FIGS. 3A and 3B show schematic diagrams of RAID subsystem disk storage block snapshots at multiple time intervals in accordance with the principles of the present invention. FIG. 3C illustrates a data instant replay method 300 that defines a default size for logical blocks or disk storage blocks, where the disk space of the RAID subsystem is a storage page pool or A step 302 of forming a matrix of disk storage blocks; a step 304 of generating a snapshot of a page pool volume or a matrix of disk storage blocks in time at predetermined time intervals; , Store snapshot or delta address index in storage page pool or disk storage block matrix, And a step of immediately Mitsukedaseru as by the saved address index a snapshot or delta.

  As shown in FIG. 3B, predetermined time intervals such as T1 (12:00 pm), T2 (12:05 pm), T3 (12:10 pm), T4 (12:15 pm), etc. Every 5 minutes, a snapshot of a matrix of storage page pools or disk storage blocks is automatically generated. The snapshot or delta address index in the storage page pool or disk storage block matrix is stored in the storage page pool or disk storage block matrix and stored in the storage page pool or disk storage block matrix. A snapshot or delta of the disk storage block matrix is made immediately discoverable by its stored address index.

  Therefore, in this data instant replay method, a snapshot of the RAID subsystem can be created using a user-defined time interval or a dynamic time stamp such as every few minutes or hours configured by the user. Or, it is generated temporally at the time indicated by the server. In the event of a system failure or virus attack, virtual snapshots with these time stamps allow data instant replay and data instant recovery in about minutes or hours. This technique is also called instant replay fusion. That is, the data immediately before the crash or attack is fused without delay, and the snapshot saved before the crash or attack can be used immediately in subsequent operations.

  FIG. 4 further illustrates a schematic diagram of a data instant fusion function 400 by using multiple snapshots of a RAID subsystem disk storage block in accordance with the principles of the present invention. At T3, a snapshot parallel chain T3′-T5 ′ is generated, which is used to replace the data fused and / or recovered by the fused data T3 ′ with the data fused at T4 Can do. Similarly, multiple parallel chains T3 ", T4" "of the snapshot can be generated and replaced with the data fused at T4'-T5 'and T4" -T5 ". In an alternative embodiment, snapshots at T4, T4'-T5 ', T5 "can still be stored in a page pool or matrix.

  Snapshots can be saved to a local RAID subsystem or a remote RAID subsystem, which can affect data integrity if a serious system crash occurs, for example, due to a terrorist attack Data can be recovered immediately. FIG. 5 shows a schematic diagram of a local-remote data replication and instant replay function 500 by using a RAID subsystem disk storage block snapshot in accordance with the principles of the present invention.

  Remote replication provides a service for replicating volume data to a remote system. Remote replication attempts to keep the local and remote volumes as synchronized as possible. In one embodiment, the remote volume data may not mirror a complete copy of the local volume data. Due to network connectivity and performance, remote volumes may be out of sync with local volumes.

  Another feature of the data instant replay and data instant fusion methods is that the snapshot can be used for inspection while the system is operating. Live data can be used for real-time inspection.

Snapshot and point-in-time copy (PITC)
One example of data instant replay is the use of RAID subsystem disk storage block snapshots in accordance with the principles of the present invention. Snapshots record volume write operations and allow views to be created for viewing past volume contents. Snapshots can also be used for data recovery by creating a view of the volume's previous point-in-time copy (PITC). Provide support.

  The snapshot core implements snapshot creation, coalesce, management, and I / O operations. The snapshot monitors writes to the volume and creates a point-in-time copy (PITC) for access through a view of the volume. Snapshot adds a logical block address (LBA) remapping layer to the data path in the virtualization layer. This is another layer of virtual LBA mapping in the I / O path. PITC may not copy all volume information and may simply change the table used for remapping.

  Snapshots provide the ability to track changes to volume data and view volume data from previous point in time. Snapshots perform this function by maintaining a list of delta writes for each PICT.

  Snapshots provide multiple methods for PITC profiles, including those initiated by applications and those initiated by time. Snapshot provides functionality that allows an application to create a PITC. The application controls creation through an API on the server, which is sent to the snapshot API. The snapshot also provides a function for creating a time profile.

  Snapshots may not implement journaling systems or recover all writes to volumes. A snapshot may only keep the last write to one address in the PITC window. The snapshot enables the user to create a PITC that covers a defined short period such as minutes or hours. A snapshot writes all information to disk to handle failures. The snapshot maintains a volume data page pointer that contains delta writes. The table provides a map to the volume data, and without it, the volume data is inaccessible, so the table information must handle the case of a controller failure.

  A view volume function (a function to view a volume) provides access to the PITC. The view volume function can be attached to any PITC in the volume except the active PITC. Appending to PITC is a relatively quick operation. Uses of the view volume function include inspection, training, backup, and recovery. The view volume function allows write operations and does not change the underlying PITC that is based on it.

  In one embodiment, the snapshot is designed to use disk space to optimize performance and facilitate use, as follows:

  ・ Snapshots increase response time for user requests. User requests include I / O, PITC creation, and view volume creation / deletion. To accomplish this, the snapshot uses more disk space than necessary to store table information. For I / O, the snapshot summarizes the current state of the volume into a single table so that all read and write requests can be satisfied by a single table. Snapshots reduce the impact on normal I / O operations as much as possible. Second, for view volume operations, the snapshot uses the same table mechanism as the main volume data path.

  • Snapshot minimizes the amount of data copied. For this, the snapshot maintains a table of pointers for each PITC. Snapshots copy and move pointers, but do not move data on the volume.

  • Snapshot manages volumes using fixed-size data pages. Tracking individual sectors requires a huge amount of memory, even with a single affordable volume. By using a data page that is larger than a sector, one type of page can contain some of the information that is replicated directly from another page.

  Snapshot uses the data space on the volume to store the data page table. The lookup table is replicated after a controller failure. The lookup table allocates pages and further divides them.

  -Snapshots handle controller failures, but in one embodiment, a volume that uses snapshots is required to run on one controller. In this embodiment, coherency is not required. All changes to the volume are recorded to disk or reliable cache for recovery by the alternate controller. Upon recovery from a controller failure, in one embodiment, it is necessary to read snapshot information from the disk.

  -Snapshot uses the virtual RAID interface to access the storage. Snapshots can use multiple RAID devices as a single data space.

  Snapshot supports “n” PITCs per volume and “m” views per volume. The limits on “n” and “m” are a function of disk space and controller memory.

Volumes and volume allocation / layout Snapshot adds an LBA remapping layer to the volume. The remapping uses an I / O request LBA and a look-up table to translate addresses into data pages. As shown in FIG. 6, a presented volume using a snapshot behaves the same as a volume without a snapshot. It has a linear LBA space and handles I / O requests. The snapshot performs I / O using a RAID interface, and includes a plurality of RAID devices in one volume. In one embodiment, the size of the RAID device for the snapshot volume is not the size of the presented volume. The RAID device allows snapshots to expand the space for data pages within the volume.

  The new volume only needs to include space for new data pages if the snapshot is initially enabled. Snapshots do not create a list of pages to put in the bottom level PITC. The bottom level PITC is empty in this case. At the time of allocation, all PITC pages are on the free list. By initially creating a volume with snapshots enabled, less physical space can be allocated than the volume presents. Snapshots track writes to the volume. In one embodiment of the present invention, NULL volumes are not copied and / or stored in the page pool or matrix, thereby increasing the efficiency of storage space usage.

  In one embodiment, with either allocation scheme, PITC places the virtual NULL volume at the bottom of the list. A read on a NULL volume returns a zero block. A NULL volume handles sectors that the server has not written so far. Writing to a NULL volume cannot occur. The volume uses a NULL volume for reading to unwritten sectors.

  The number of free pages depends on the size of the volume, the number of PITCs, and the expected data change rate. The system determines the number of pages to allocate for a given volume. The number of data pages can grow over time. Expansion can support more rapid data changes, more PITC, or larger volumes than expected. New pages are added to the free list. Adding pages to the free list may be done automatically.

  Snapshot uses data pages for volume space management. Each data page may contain several megabytes of data. Using an operating system tends to write many sectors to the same area of the volume. Memory requirements also require that snapshots use pages for volume management. Maintaining one 32-bit pointer for each sector of a 1 terabyte volume may require 8 gigabytes of RAM. Different volumes can have different page sizes.

  FIG. 7 illustrates one embodiment of a snapshot structure. Snapshots add many objects to the volume structure. Added objects include PITC, pointer to active PITC, data page free list, child view volume, and PITC coalesced object.

  The active PITC (AP) pointer is maintained by the volume. The AP handles the mapping of read and write requests to the volume. The AP contains a summary of the current location of all data in the volume.

  Data page free list keeps track of available pages on the volume.

  An optional child view volume provides access to the volume PITC. The view volume contains its own AP to record writes to the PITC, but does not change the underlying data. One volume may support multiple child view volumes.

  A snapshot coalescing object temporarily links two PITCs for the purpose of removing the previous PITC. PITC coalescing involves transferring ownership of a data page and releasing the data page.

  PITC contains tables and data pages for pages written while the PITC was active. The PITC includes a freeze time stamp at which the PITC stops accepting write requests. The PITC also includes a Time-to-Live (Time to Live) value that determines at what time the PITC will merge.

  Snapshots also summarize data page pointers for the entire volume at the time the PITC is taken to provide predictable performance of reads and writes. Other solutions may require reading to examine multiple PITCs to find the latest pointer. Such a solution requires a table caching algorithm, but performance may be minimal.

  Also, by summarizing snapshots in the present invention, the worst memory usage of the table is reduced. In that case, it may be required to load the entire table into memory, but only a single table may be required to be loaded.

  The summary (list) includes pages owned by the current PITC, but can include pages from all previous PITCs. Track page ownership for each data page to determine which page PITC writes to. The list also tracks ownership of the coalescing process. To handle this, the data page pointer contains a page index.

  FIG. 8 illustrates one embodiment of a PITC life cycle. Each PITC goes through a number of the following states before being stored (committed) as read-only.

  1. Create table-Creates a table.

  2. Commit to disk-this creates storage on disk for PITC. Writing the table at this point ensures that the space required to store the table information is allocated before the PITC is taken. At the same time, the PITC object is also stored (committed) on the disk.

  3. Accept I / O-This is the PITC is the active PITC (AP). Here, PITC handles read and write requests to the volume. This is the only state that accepts a write request to the table. PITC generates an event that is now active.

  4). Store table on disk-PITC is no longer an AP and no longer accepts more pages. A new AP is taking over. After this point, the table is not changed unless it is removed during the coalescing operation. This is read-only. At this point, PITC generates an event that is frozen and stored. Any service can listen to the event.

  5. Freeing table memory (release)-Frees memory required by a table. This step also clears the log to indicate that all changes have been written to disk.

  The top level PITC for a volume or view volume is called active PITC (AP). The AP fulfills all read and write requests to the volume. The AP is the only PITC that can accept write requests for that volume. The AP contains a list of data page pointers for the entire volume.

  An AP can be a destination but not a source for a coalescing process. If it is a destination, the AP increases the number of pages it owns, but does not change the view of the data.

  For volume expansion, the AP grows immediately with the volume. The new page points to a NULL volume. A PITC that is not an AP does not need to change for volume expansion.

  Each PITC maintains a table that maps incoming LBAs to data page pointers to the underlying volume. The table contains pointers to data pages. The table needs to address more physical disk space than presented logical space. FIG. 9 illustrates one embodiment of a table structure having a multi-level index. This structure decrypts the volume LBA into a data page pointer. At each level, as shown in FIG. 9, LSB (less significant bit) with increasing address is decoded. The structure of this table provides fast lookup and volume expansion capabilities. For fast lookups, the multi-level index structure keeps the table shallow and has multiple entries at each level. This index performs an array lookup at each level. In order to support volume expansion, a multi-level index structure allows another layer to be added to support expansion. The expansion of the volume in this case is an expansion of the LBA count (count) presented to the upper layer, not the actual amount of storage space allocated to the volume.

  The multi-level index contains a list (summary) of data page remapping across the volume. Each PITC contains a complete remapping list for this volume at the stored point-in-time (time (specific time)).

  A multi-level index structure uses different entry types for the levels of the table. Different entry types support the need to read information from disk and store the information in memory. The bottom level entry may only contain a data page pointer. The top and middle level entries contain two arrays, one is the LBA of the next level table entry, and contains a memory pointer to the table.

  As the presented volume size expands, the size of the previous PITC table does not need to be increased and the table need not be changed. Since the information in the table is read-only, it does not change and the expansion process modifies the table by adding a NULL page pointer to the end. The snapshot does not present the table from the previous PITC directly to the user.

  When there is an I / O operation, the table is required to map the LBA to the data page pointer. In that case, in the I / O, the data page pointer is multiplied by the data page size to obtain the LBA of the underlying RAID. In one embodiment, the data page size is a power of two.

  The table provides APIs for LBA remapping, page addition, and table coalescence.

  Snapshot uses data pages to store PITC objects and LBA mapping tables. The table accesses the RAID interface directly for I / O to that table entry. The table minimizes changes when reading and writing the table to the RAID device. If there is no change, the table information can be read and written directly to the table entry structure. This reduces the copy required for I / O. Snapshots can use a change log to prevent hot spots on the disk. A hot spot is a place that is used repeatedly to track updates to a volume. The change log records updates to the PITC table and a free list for the volume. During recovery, the snapshot uses the change log to recreate in-memory APs and free lists. FIG. 10 shows an embodiment of table recovery, which shows the relationship between an in-memory AP, an on-disk AP, and a change log. This figure shows the same relationship for the free list. The in-memory AP table can be reconstructed from the on-disk AP table and log. In the event of any controller failure, the AP is rebuilt by reading the AP on disk and applying log changes to it. The change log uses different physical resources depending on the system configuration. In a multiple controller system, the change log relies on battery-backed cache memory for storage. By using cache memory, snapshots can reduce the number of table writes to disk and maintain data integrity. The change log is replicated to the backup controller for recovery. In a single controller system, the change log writes all information to disk. This has the side effect of creating a hot spot at the log location on disk. This allows a number of changes to be written to a single device block.

  The snapshot periodically writes the PITC table and free list to disk to create and clear checkpoints in the log. This period can vary depending on the number of updates to the PITC. The coalescing process does not use a change log.

  Snapshot data page I / O may require requests to be within data page boundaries. If a snapshot is an I / O request that crosses a page boundary, the request is split. The snapshot then passes the request to the request handler. The write and read sections assume that the I / O is within page boundaries. The AP fulfills the I / O request by providing LBA remapping.

  The AP satisfies all write requests. Snapshots support two different write sequences for owned and non-owned pages. Different sequences allow adding pages to the table. FIG. 11 illustrates one embodiment of a write process with an owned page sequence and a non-owned page sequence.

In the owning page sequence, the process includes:
1) Find the table mapping.
2) Page Owned Write-Remap LBA and write data to RAID interface.

  Previously written pages are pure write requests. A snapshot writes data to the page and overwrites the current contents. Only data pages owned by the AP are written. Pages owned by other PITCs are read-only.

For a non-owned page sequence, the process includes:
1) Find the table mapping.
2) Read previous page-Read the data page so that the entire page is completed with a write request and read data. This is the start of the copy on write process.
3) Combining data-Put the payload of data page read and write requests into a single contiguous block
4) Free list allocation-Get a new data page pointer from the free list.
5) Write the combined data to a new data page.
6) Store new page information in the log.
7) Update the table-change the LBA remapping in the table to reflect the new data page pointer. The data page is now owned by PITC.

  Adding a page may require blocking read and write requests until the page is added to the table. Snapshots achieve controller coherency by writing table updates to disk and maintaining multiple cached copies of the log.

  For read requests, the AP satisfies all read requests. The read request uses the AP table to remap the LBA to the LBA for that data page. The remapped LBA is passed to the RAID interface to satisfy the request. A volume can satisfy read requests for data pages that have not been previously written to the volume. These pages are marked with a NULL address (all 1s) in the PITC table. A request for this address is satisfied by the NULL volume and returns a constant data pattern. Pages owned by different PITC tables can satisfy read requests across page boundaries.

  A snapshot uses a NULL volume to satisfy a read request for a previously written data page. This returns all zeros for each sector read. It has no RAID device and no allocated space. It is expected that all zero blocks will be placed in memory to meet the data requirements for reads to a NULL volume. All volumes share a NULL volume to satisfy read requests.

  In one embodiment, the coalescing process removes the PITC and some of its owned pages from the volume. Removing the PITC creates more space that can be used to track new differences. In coalescence, two adjacent tables are compared for differences and only new differences are maintained. Coalescence occurs periodically or manually according to the configuration by the user.

This process includes two PITCs: a source and a destination. In one embodiment, the rules for eligible objects are as follows:
1) The source must be the previous PITC for the destination-the source must be created before the destination.
2) The destination must not be the source at the same time.
3) The source must not be referenced by multiple PITCs. Multiple references occur when the view volume is created from PITC.
4) A destination may support multiple references.
5) AP can be a destination but not a source.

  The coalescing process writes all changes to disk and does not require coherency. If the controller fails, the volume recovers PITC information from the disk and resumes the coalescing process.

  This process marks two PITCs for coalescing and includes the following steps:

  1) Source state is set to coalesced source-This state is stored to disk for controller failure recovery. At this point, the source is no longer accessed because the data page is invalid. The data page is returned to the free list or ownership is transferred to the destination.

  2) Destination state is set to coalesced destination-This state is stored on disk for controller failure recovery.

  3) Load and compare tables-This process moves the data page pointer. Freed data pages are immediately added to the free list.

  4) The destination state is set normally-the process is complete.

  5) Adjust the list-change the source's next pointer previous to the destination. This effectively removes the source from the list.

  6) Release source-Return any data page used for control information to the free list.

  The above process supports a combination of two PITCs. Those skilled in the art understand that coalescing can be designed to delete multiple PITCs and create multiple sources in a single pass.

  As shown in FIG. 2, the page pool maintains a data page free list for use by all volumes associated with the page pool. The free list manager uses data pages from the page pool to store the free list in permanent storage. There are a number of things that cause free list updates: the writing process allocates pages, the control page manager allocates pages, and the coalescing process returns pages.

  The free list can maintain a trigger to automatically expand itself at some threshold. This trigger uses the page pool extension method to add pages to the page pool. Automatic expansion can also be a function of the volume policy. Allows data volumes with high importance to be expanded and forces less important volumes to be merged.

  View volumes provide access to previous point-in-time and support normal volume I / O operations. PITC tracks differences between PITCs, and view volumes allow users access to information contained within PITCs. The view volume branches off from the PITC. View volumes support recovery, inspection, backup operations, and so on. View volume creation occurs almost instantaneously because no data copy is required. A view volume may require its own AP to support writing to the view volume.

  In a view taken from the current state of the volume, the AP can be copied from the AP of the current volume. Using the AP, the view volume allows a write operation to the view volume to be performed without changing the underlying data. The OS may require a file system or file reconstruction in order to use the data. The view volume allocates space from the parent volume for APs and written data pages. View volumes do not have associated RAID device information. Deleting the view volume frees space and returns it to the parent volume.

  FIG. 12 illustrates an exemplary snapshot operation showing the transition for a volume using snapshots. FIG. 12 shows a volume with 10 pages. Each state includes a read request fulfillment list for the volume. The shaded block indicates the owned data page pointer.

  The transition from the left side of the figure (i.e., the initial state) to the center of the figure indicates writing to pages 3 and 8. Writing to page 3 requires a change to PITC I (AP). PITC I adds page 3 to the table according to the new page write process. PITC reads the unchanged information from page J and saves the page using drive page B. All subsequent writes to page 3 of this PITC are handled without moving the page. The writing to page 8 shows a second case relating to writing to the page. Because PITC I already contains page 8, PITC I writes to that portion of that data on page 8. In this case, it exists in drive page C.

  The transition from the center of the figure to the right side of the figure (ie, the final state) indicates the coalescence of PITC II and III. Each coalescing of snapshots involves deleting old pages and maintaining all changes in both PITCs. Both PITCs contain pages 3 and 8. The process keeps the newer page from PITC II, releases the page from PITC III, and returns pages A and D to the free list.

  Snapshot allocates data pages from the page pool to preserve free list and PITC table information. Control page allocation further sub-allocates data pages to fit the size required for the object.

  The volume includes a page pointer for the head of the control page information. All other information can be read from this page.

  Snapshots track the number of pages in use at specific time intervals. Thereby, the snapshot predicts when the user needs to add more physical disk space to the system and prevents the snapshot from running out.

Data Progression In one embodiment of the present invention, data progression (DP) is used to move data gradually to the appropriate cost storage space. The present invention allows a user to add a drive when the drive is actually needed. This significantly reduces the overall cost of the disk drive.

  Data progression moves data that has not been recently accessed and historical snapshot data to less expensive storage. For data that has not been accessed recently, the storage cost for pages that have not been accessed recently is gradually reduced. Data is not immediately moved to the lowest cost storage. For historical snapshot data, move the read-only page to a more efficient storage space such as RAID 5 and to the cheapest storage if the page is no longer accessible from the volume. Moved.

  Other advantages of data progression of the present invention include maintaining high speed I / O access to currently accessed data and reducing the need to purchase a fast but expensive disk drive. .

  In data progression, during operation, the cost of storage is determined using the cost of physical media and the efficiency of the RAID device used for data protection. In data progression, storage efficiency is determined, and data is moved according to it. For example, data progression can convert a RAID 10 device to a RAID 5 device in order to use physical disk space more efficiently.

  Data progression defines accessible data as data that the server can currently read or write. In data progression, accessibility is used to determine the class of storage that a page should use. A page is read-only if it belongs to a history PITC. If the server has not updated the page in the most recent PITC, the page is still accessible.

  FIG. 17 illustrates one embodiment of accessible data pages in a data progression operation. Accessible data pages are divided into the following categories:

  • Recently accessed, accessible-these are the active pages that are most commonly used by the volume.

  Accessible, not recently accessed-a read / write page that has not been used recently.

  -History accessible-Read-only page that can be read by volume--This applies to snapshot volumes.

  -History inaccessible-Read-only data page not currently accessed by volume--This applies to snapshot volumes. Snapshots maintain these pages for recovery purposes, and pages are typically placed in the lowest cost storage possible.

  FIG. 17 shows three PITCs with various owned pages for snapshot volumes. The dynamic capacity volume is represented only by PITC C. All pages are accessible and read / write. Pages can have different access times.

  The table below shows the various storage devices in order of increasing efficiency or decreasing monetary expenditure. The list of storage devices can also follow the general order of slower write I / O accesses. Data progression calculates the efficiency of the logical protection space divided by the total physical space of the RAID device.

  The efficiency of RAID 5 increases as the number of stripe drives increases. As the number of striped disks increases, the fault domain increases. As the number of stripe drives increases, the minimum number of disks required to create a RAID device also increases. In one embodiment, data progression does not use a RAID 5 stripe size greater than 9 drives. This is because the fault domain size increases and the increase in efficiency is limited. Data progression uses a RAID 5 stripe size that is an integer multiple of the snapshot page size. This allows data progression to perform full stripe writing when moving a page to RAID 5 and increases the efficiency of the movement. All RAID5 configurations have the same write I / O characteristics for data progression purposes. For example, RAID 5 on a 2.5 inch (6.5 cm) FC disk may not efficiently use the performance of such a disk. To prevent this combination, data progression needs to support the ability to prevent certain RAID types from operating on specific disk types. Data progression configuration may also prevent the system from using RAID 10 or RAID 5 space.

  The disk type is shown in the following table.

  Data progression includes the ability to automatically classify disk drives relative to the drives in the system. The system examines the disk and determines its performance relative to other disks in the system. Faster disks are classified into the high rating category, and slower disks are classified into the low rating category. When a disk is added to the system, the system automatically rebalances the disk rating classification. This approach handles both systems that never change and systems that change frequently as new disks are added. This automatic classification may place multiple drive types within the same evaluation category. If the drive ratings are determined to be fairly close, the ratings will be the same.

In one embodiment, the system includes the following drives: That is,
High-10K FC drive Low-SATA drive

When adding a 15K FC drive, Data Progression automatically reclassifies the disk and demotes the 10K FC drive. This results in the following classification:
High-15K FC drive Medium-10K FC drive Low-SATA drive

In another embodiment, the system may have the following drive types:
High-25K FC drive Low-15K FC drive

  Therefore, the 15K FC drive is classified as a low evaluation category, while the 15K FC drive is classified as a high evaluation category.

When a SATA drive is added to the system, data progression reclassifies the disk in time. The result is the following classification:
High-25K FC drive Medium-15K FC drive Low-SATA drive

  The data progression can include a waterfall progression. Typically, waterfall progression moves data to a less expensive resource, but only when the resource is fully used. Waterfall progression effectively maximizes the use of the most expensive system resources. It also minimizes system costs. Adding a cheap disk to the lowest pool creates a larger pool at the bottom.

  A typical waterfall progression uses RAID 10 space and then uses the next in RAID space, such as RAID 5 space. This forces the waterfall to go directly to RAID 10 of the next class of disks. As an alternative, the data progression may include a mixed RAID waterfall progression as shown in FIG. This alternative data progression method solves the problem of maximizing disk space and performance and allows storage to be transformed into a more efficient form in the same disk class. This alternative method also supports the requirement that RAID 10 and RAID 5 share the entire resources of a certain disk class. This may require configuring a certain percentage of the disk space to be used for a class of disks with a certain RAID level. Thus, this alternative data progression method maximizes the use of expensive storage and leaves room for another RAID class to coexist.

  In the mixed RAID waterfall, the page is moved to a cheaper storage only when the storage is limited. A threshold, such as a percentage of the total disk space, limits the amount of storage of a particular RAID type. This maximizes the use of the most expensive storage in the system. As storage approaches its limit, data progression automatically moves pages to lower cost storage. Data progression can provide a buffer for write spikes.

  It is understood that the waterfall method described above can quickly move pages to the lowest cost storage, as in some cases, and move historical and inaccessible pages to less expensive storage in a timely manner. There may be a need to It is also possible to immediately move the history page to a cheaper storage.

  FIG. 18 shows a flow diagram of a data progression process 1800. Data progression continually examines each page of the system for its access pattern and storage cost to determine if there are data pages to be moved. Data progression can also determine whether the storage has reached its maximum quota.

  The data progression process determines whether the page is accessible from any volume. The process checks the PITC for each volume added to the history to determine if the page is referenced. If the page is actively in use, it is desirable to promote or slow down the demotion. If the page is not accessible from any volume, it is moved to the lowest cost storage available. Data progression also takes time into account before PITC expires. If the snapshot is scheduled to expire soon, the page will not progress. If the page pool is operating in aggressive mode, page progression may occur.

  Recent access detection for data progression requires removing a lump of activity from page promotion operations. Data progression separates read and write access tracking. This allows data progression to hold data on accessible RAID 5 devices. Operations such as virus scanning and reporting only read data. Data progression changes the credential for recent access when storage is running low. This allows data progression to more aggressively demote pages. This also helps to pack the system from the bottom when storage is running low.

  Data progression can move data pages aggressively when system resources are low. More disk or configuration changes are still necessary in all of these cases. Data progression extends the amount of time that the system can operate in a timeless situation. Data progression attempts to keep the system operational as long as possible. That time is when all the storage class spaces are gone.

  If the RAID 10 space is low and the entire available disk space is low, the data progression can take the RAID 10 disk space and move it to a more efficient RAID 5. This increases the overall capacity of the system at the expense of write performance. More disks are still needed. If a particular storage class is fully used, data progression can continue to operate the system with unacceptable pages. For example, if a volume is configured to use RAID 10-FC for its accessible information, RAID 5-FC or RAID 10-SATA until more RAID 10-FC space is available. Allocate pages from.

  Data progression also supports compression to increase the perceived capacity of the system. Compression is used for history pages that have not been accessed or for storage of recovery information. Compression appears as another class of storage near the bottom of storage costs.

  As shown in FIG. 25, the page pool essentially includes a free list and device information. The page pool must support multiple free lists, enhanced page allocation schemes, and free list classification. The page pool maintains a separate free list for each class of storage. This allocation scheme allocates pages from one of many pools and sets the minimum or maximum class allowed. The free list classification comes from the configuration of the device. Each free list provides its own counter for statistics collection and display. Each free list also provides RAID device efficiency information for collection of storage efficiency statistics.

  In one embodiment, the device list may require additional functionality to track storage class costs. The storage class is determined by the combination. This happens when the user wants the granularity of the configured class to be coarser or finer.

  FIG. 26 illustrates one embodiment of a high performance database where all accessible data is on 2.5FC drives, even if not recently accessed. Inaccessible historical data is moved to a RAID 5 fiber channel.

  FIG. 27 illustrates one embodiment of an MRI image volume, where the accessible storage is SATA RAID 10 and RAID 5 for this dynamic volume. If the image has not been accessed recently, the image is moved to RAID5. In that case, a new write is initially directed to RAID 10. FIG. 19 illustrates one embodiment of a compressed page layout. Data progression implements compression by further allocating fixed size data pages. Further sub-allocation information tracks the location of the free part of the page and the assigned part of the page. Data progression cannot predict compression efficiency, but can handle variable-sized pages within its sub-allocation range.

  Compressed pages can significantly affect CPU performance. For write access, a compressed page should require decompression and recompression of the entire page. Accordingly, the page being actively accessed is not compressed and returned to an uncompressed state. Writing may be necessary in situations where storage is extremely limited.

  The PITC remapping table points to sub-allocation information and is marked to indicate the page being compressed. Accessing a compressed page may require a higher I / O count than an uncompressed page. Access requires reading out the sub-allocation information in order to retrieve the actual data location. The compressed data can be read from the disk and decompressed by the processor.

  In data progression, compression may be required to be able to decompress some of the entire page. This allows data progression read access to decompress only a small portion of the page. The read-ahead feature of the read cache can help with compression delays. A single decompression can handle several server I / Os. Data progression marks pages that are not suitable candidates for compression, thereby avoiding continued attempts to compress a page.

  FIG. 20 illustrates one embodiment of data progression in a high level disk drive system in accordance with the principles of the present invention. Data progression does not change the external behavior of the volume or the behavior of the data path. Data progression may require changes to the page pool. The page pool essentially contains a free list and device information. The page pool must support multiple free lists, enhanced page allocation schemes, and free list classification. The page pool maintains a separate free list for each class of storage. The allocation scheme allows pages to be allocated from one of many pools while setting the minimum and maximum classes allowed. Free list classification can be derived from the configuration of the device. Each free list provides its own counter for statistics collection and display. Each free list also provides RAID device efficiency information for collection of storage efficiency statistics.

  PITC identifies candidates for movement and prevents I / O to accessible pages during the movement. Data progression continually examines the PITC for candidates. The accessibility of pages changes continuously due to server I / O, new snapshot page updates, and view volume creation / deletion. Data progression also continuously checks for volume configuration changes and summarizes the current list of page classes and counts. Thus, the data progression can evaluate the summary (list) and determine whether there is a page to be moved.

  Each PITC presents a counter for the number of pages used for each class of storage. Data progression uses this information to identify PITCs that are good candidates for moving pages when a threshold is reached.

  RAID allocates devices from a set of disks based on the cost of the disks. RAID also provides an API that retrieves the efficiency of a device or potential device. It also needs to return information about the number of I / Os required for the write operation. Data progression may also require RAID NULL to use a third party RAID controller as part of the data progression. RAID NULL can consume the entire disk and can simply act as a pass-through layer.

  The disk manager can also automatically determine and save the disk classification. Changes to the SCSI initiator may be required to automatically determine the disk classification.

  From the foregoing description and drawings, those skilled in the art will appreciate that the specific embodiments shown and described are intended for purposes of illustration only and are not intended to limit the scope of the invention. Those skilled in the art will appreciate that the invention may be practiced in other specific forms without departing from its spirit or essential characteristics. References to details of particular embodiments do not limit the scope of the invention.

Claims (28)

A disk drive system capable of dynamically allocating data in a pool of storage,
A data storage subsystem;
A plurality of user volumes comprising storage space from the data storage subsystem;
A disk manager having at least one disk storage system controller;
The disk manager
Configured to maintain a pool of storage including a matrix of virtual volumes of the data storage subsystem, each of the virtual volumes being an abstraction of a plurality of physical storage devices and pre-determined Having a size of
Configured to dynamically allocate free virtual volumes from the matrix of virtual volumes to the plurality of user volumes;
Configured to write data to the allocated virtual volume;
system.   The system of claim 1, wherein the data storage subsystem and the disk manager determine whether additional disk drives are required and the additional disk drives are required. A system in which notifications are sent in some cases.   3. A system according to claim 1 or 2, wherein the disk manager manages a plurality of disk storage system controllers.   4. A system according to any of claims 1 to 3, further comprising a plurality of redundant disk storage system controllers to cover failures of the operated disk storage system controller.   5. The system according to claim 1, wherein the data storage subsystem is at least one of RAID types such as RAID-0, RAID-1, RAID-5, RAID-10, etc. The system further comprising a combination including one.   6. The system of claim 5, wherein the data storage subsystem further comprises RAID types including RAID-3, RAID-4, RAID-6, and RAID-7.   The system according to any one of claims 1 to 6, wherein the virtual volume is an abstraction of a plurality of RAID devices.   8. A system as claimed in any preceding claim, wherein the at least one disk storage system controller automatically generates a snapshot for each of the virtual volumes at predetermined time intervals, Storing the address index of each snapshot in the associated virtual volume so that the location of the snapshot of the virtual volume can be immediately found via the stored address index; A system that contains changes to the data in the associated virtual volume since the previous snapshot taken at the previous time interval.   8. A system as claimed in any preceding claim, wherein the at least one disk storage system controller automatically generates a snapshot for each of the virtual volumes, At least one snapshot that stores an address index, accesses at least one of the snapshots via the stored address index, and provides a view of the contents of the associated virtual volume at a first time point A temporary volume from each of the at least one of the snapshots, the snapshot being related since the previous snapshot taken at the second point in time To virtual volume It is intended to include changes to the kicking data, system. A method of dynamically allocating data in a data storage subsystem,
Generating a plurality of user volumes of storage space from a plurality of data storage devices of the data storage subsystem;
Managing a pool of storage that includes a matrix of virtual volumes of the data storage subsystem, each of the virtual volumes being an abstraction of a plurality of physical storage devices and predetermined. A step having a specified size;
Dynamically assigning free virtual volumes from the matrix of virtual volumes to the plurality of user volumes using a pool of storage;
Writing data to the allocated virtual volume. The method of claim 10, comprising:
Determining a disk space occupancy of the data storage subsystem based on a history of the disk space occupancy of the data storage subsystem;
Determining whether the data storage subsystem requires an additional disk drive based on the occupancy; and
Sending the notification to the data storage subsystem if the additional disk drive is needed.   The method of claim 11, further comprising sending the notification by email.   13. A method according to any of claims 10 to 12, wherein the predetermined size of a virtual volume is a default size and can be changed by a user.   14. The method according to claim 10, wherein the plurality of user volumes are at least one of a plurality of RAID types of RAID-0, RAID-1, RAID-5, and RAID-10. A method comprising one disk space block. A method according to any of claims 10 to 14, comprising
Automatically generating a snapshot for each of the virtual volumes at a predetermined time interval, each snapshot being associated with a virtual that was taken since the previous snapshot taken at the previous time interval. A step that includes changes to the data in the volume; and
Storing the address index of each snapshot, the method further comprising: instantly finding the location of the snapshot of the virtual volume via the stored address index. .   The method of claim 15, wherein each snapshot includes a record of write operations to the associated virtual volume.   The method according to claim 15 or 16, wherein the predetermined time interval is determined by a user.   18. A method as claimed in any of claims 15 to 17, wherein the time interval ranges from every few seconds to every few hours. 19. A method as claimed in any of claims 15 to 18, wherein the snapshot has no effect on the integrity of the data and the data can be recovered immediately if the main system is broken. Stored in the storage subsystem,
Method.   20. A method as claimed in any of claims 15 to 19, further comprising the step of generating a snapshot for at least one virtual volume in response to a user request.   21. A method as claimed in any of claims 15 to 20, further comprising the step of coalescing at least two snapshots for the associated virtual volume. A method according to any of claims 10 to 14, comprising
Generating a plurality of snapshots for the virtual volume;
Storing an address index of the snapshot;
Accessing at least one snapshot of the plurality of snapshots via the stored address index;
Generating a temporary volume from the accessed at least one snapshot, the temporary volume providing a view of the contents of the associated virtual volume at a particular point in time; A method further comprising:   The method of claim 22, wherein the snapshots are generated at predetermined time intervals.   24. The method according to claim 22 or 23, further comprising the step of changing the temporary volume without changing the data underlying the virtual volume.   25. A method as claimed in any one of claims 22 to 24, wherein the temporary volume is used for inspection, and the inspection is at the point in time without changing the underlying data of the virtual volume. The method that is performed on the data.   26. A method as claimed in any of claims 22 to 25, wherein the temporary volume is used for training, and the training is at the point in time without changing the underlying data of the virtual volume. The method that is performed on the data.   27. A method according to any of claims 22 to 26, wherein the temporary volume is used for backup of the data.   28. A method as claimed in any of claims 22 to 27, wherein the temporary volume is used for recovery of the data.

Images