WO2017131729A1

WO2017131729A1 - Managing a storage system

Info

Publication number: WO2017131729A1
Application number: PCT/US2016/015562
Authority: WO
Inventors: Mykel John Kramer
Original assignee: Hewlett Packard Enterprise Development Lp
Priority date: 2016-01-29
Filing date: 2016-01-29
Publication date: 2017-08-03

Abstract

Techniques for managing a storage system are disclosed. An example system includes a first controller coupled to a storage device pool and a second controller coupled to the storage device pool. The first controller and the second controller are configured to process storage requests received from a client device. The system also includes a manager to communicate with the first controller and the second controller over a network. The manager is to determine which controller is ready to process storage transactions and communicate to the client device which controller is ready to process storage transactions.

Description

MANAGING A STORAGE SYSTEM

BACKGROUND

[0001] Many large-scale storage systems are configured as highly-available systems. Such storage systems incorporate a high level of redundancy to improve the availability and accessibility of stored data. For example, a storage system can include two or more storage controllers that may provide failover capabilities.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Certain exemplary embodiments are described in the following detailed description and in reference to the drawings, in which:

[0003] Fig. 1 is an example block diagram of a distributed storage system with two-phase locking;

[0004] Fig. 2 is an example state diagram for a controller that is designated as the preferred controller of a controller pair;

[0005] Fig. 3 is an examples state diagram for a controller that is designated as the non-preferred controller of a controller pair;

[0006] Figs. 4A through 4Y shows all the states maintained by the database manager and the transitions between those states according to an example;

[0007] Fig. 5 is an example process flow diagram of a method of initializing a storage system with two storage controllers;

[0008] Fig. 6 is an example process flow diagram summarizing a method of managing a storage system; and

[0009] Fig. 7 is an example block diagram showing a tangible, non-transitory, computer-readable medium that stores code configured to manage a storage system.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

[0010] In a storage system with redundant storage controllers, two or more storage controllers may have access to the same storage space. Data corruption can occur if two or more controllers attempt to access the same storage object at the same time. To prevent two controllers from having access to the storage objects at the same time, a centralized database is maintained by a database manager. One of the fields in the database is a lock field referred to herein as the "going ready" field. This field indicates that the corresponding node is being authorized to take control of the storage device pool. Only one controller can be designated with the status "going ready." After updating a controller's status in the database to "going ready," a corresponding "going ready" message is sent to the controller. The controller is then able to issue a backend lock to the storage devices so the storage devices will not be accessible to other controllers. After the backend lock is achieved, the controller indicates a ready status to the database manager and the controller's status in the database is set to ready. The ready status in the database indicates to the clients that the controller is ready to receive storage

transactions.

[0011] Fig. 1 is an example block diagram of a distributed storage system with two-phase locking. It will be appreciated that the storage system 100 shown in Fig. 1 is only one example of a storage system in accordance with embodiments. In an actual implementation, the storage system 100 may include various additional storage devices and networks, which may be interconnected in any suitable fashion, depending on the design considerations of a particular implementation. For example, a large storage system will often have many more client computers and storage devices than shown in this illustration.

[0012] The storage system 100 provides a pool of data storage resources to any number of client computers 102, which may be general purpose computers, workstations, mobile computing devices, and the like. The client computers 102 can be coupled to the storage system 100 through a network 104, which may be a local area network (LAN), wide area network (WAN), a storage area network (SAN), or other suitable type of network. The storage system 100 includes storage controllers 106, referred to herein as controller N and controller N+1 .

[0013] The storage system 100 also includes a storage device pool 1 08, which are controlled by the controllers 106. In the present example, only two controllers 106 are shown. However, it will be appreciated that the storage device pool may be controlled by any suitable number of controller 106, including 2, 4, 6, 10, or more. The storage device pool 108 may include any suitable type of storage devices, including hard disk drives, solid state drives, tape drives, storage arrays, and others. In examples, the storage device pool 108 is configured as block based storage, which is virtualized as one or more storage volumes by the controllers 106. Furthermore, the storage system 1 00 can include more than one type of storage component. The storage network system 100 may also include additional storage devices in addition to what is shown in Fig. 1 .

[0014] The client computers 102 can access the storage space of the storage device pool 108 by sending Input/Output (I/O) requests, including write requests and read requests, to the controllers 106. The controllers 106 process the I/O requests so that user data is written to or read from the appropriate storage locations in the storage device pool 108. As used herein, the term "user data" refers to data that a person might use in the course of business, performing a job function, or for personal use, such as business data and reports, Web pages, user files, image files, video files, audio files, software applications, or any other similar type of data that that a user may wish to save to long term storage.

[0015] Only one of controller N and controller N+1 will have access to particular volume within the storage device pool 108 at any given time. For example, controller N may be a default controller for a particular volume, and controller N+1 may be a standby controller than can assume active control of the volume if controller N fails. Such a process may be referred to as failover, and it helps to improve availability and reliability of data storage systems. The controllers do not have a sideband channel that enables direct communication between the controller N and the controller N+1 .

[0016] The storage system 100 may also include a database 1 10 and a database manager 1 1 2. The database 1 10 stores information regarding the configuration and status of the storage system 100 as described in further detail below. The database manager 1 12 maintains the status information by communicating with the controllers 106. Each client 1 02 may refer to the database 1 10 to determine the data layout of the store system 100. The data layout may include the location of specific data and the identity of the controller 106 that currently has access to specific data.

[0017] The distributed nature of the storage system means that time is required to disburse data regarding changes to the data layout. In order to protect the integrity of the data, the storage system 100 ensures that the client 102 knows the correct controller to send data to, that the controller knows when it should advertise itself as being ready to accept data requests, and that the two controllers 106 do not operate on the same storage devices at the same time. The storage system is also able to survive any losses in communication between the clients, the controllers, and the database. The storage system 100 provides these features in part by implementing a two stage locking process by which a specific controller is designated as the active controller for a particular volume. The two stage process, which is described further below, helps to ensure that the client 102 is sending data requests to the correct controller 1 06 and only one controller 106 at a time is using the devices. Communication between any client, controller and database manager can be lost at any point permanently or temporarily.

[0018] The database manager 1 12 is configured to maintain a gossip with each controller 1 06. As used herein, the term gossip refers to messaging that occurs at regular intervals between the controllers 106 and the database manager 1 12 wherein each controller can report its perceived status and the database manager 1 1 2 can issue instructions. The gossip messages enable each controller 1 06 to report its perceived controller condition. The database manager will then process that information and update the database fields accordingly. The clients 102 are configured to query the database 1 1 0 to identify the controller 106 that is currently configured to process storage transactions for the desired data.

[0019] The storage system uses a two stage lock system to preserve data integrity across time outs and delays while allowing the two controllers 106 to share the same storage device pool 108. In the first stage, the database manager sets a GOING_READY state for a selected controller 106 to true and instructs the controller to advance to the ready state. The controller 106 then acquires reservations for the storage devices. In the second phase, the database manager 1 1 2 receives a gossip from the controller 106 that the controller 106 is in the READY STATE, meaning that the controller is ready to process storage instructions, and the database manager sets the condition state of the controller to READY. A more detailed explanation of the process is described below.

[0020] Fig. 2 is an example state diagram for a controller that is designated as the preferred controller of a controller pair. The preferred controller is preferred in the sense that it that will be instructed by the database manager to advance to the ready state if possible, while the non-preferred controller will only be instructed to advance to the ready state if the preferred controller is unavailable. The controller maintains two status fields, the preferred field and the condition field. As the preferred controller, the preferred field is true. At regular intervals, the controller sends out a gossip message to the database manager to indicate the controller's perceived condition.

[0021] The condition field can be set to a value of READY, NOT READY, or STANDBY. The READY condition is a controller status that indicates that the controller is ready to receive and process storage transactions. The database manager gossips to the controller to instruct the controller to transition to a READY condition or drop out of a READY condition.

[0022] The STANDBY condition indicates that the controller is operational and available to assume the status of READY when commanded by the database manager. The NOT READY condition indicates that the controller is not available to assume the status of READY. For example, the controller may be experiencing some technical impediment or may be booting up.

[0023] Three possible states of the controller are shown in Fig. 2. The initial state of a controller when it boots up is state 1 , wherein the condition has a value of NOT READY. After the controller finishes all of the tasks associated with booting up, the controller state advances to state 2, wherein the condition has a value of STANDBY. If the controller restarts or experiences some type of fault, the state will move back to state 1 and the condition returns to

NOT READY.

[0024] While at state 2, the controller can advance to state 3 if instructed to by the database manager. In this case, the controller receives gossip from the database manager indicating that the GOING_READY field maintained by the database manager has been set to a value of true. The controller then issues persistent reservations to the relevant storage devices in the storage device pool. Once the reservations are established, the condition of the controller is set to READY.

[0025] While at stage 3, if the controller restarts or experiences some type of fault, the state will move back to state 1 , and the controller will reboot back to the state 1 wherein the condition has a value of NOT READY. Additionally, while at stage 3, the controller may receive gossip from the database manager indicating that a status value for the controller is set to JOINING, in which case the controller will return to state 2, wherein the condition of the controller is STANDBY. In this way, the database manager gossip to the controller can instruct the controller to drop out of condition READY and transition back to condition STANDBY. If the database manager transitions the controller from READY to JOINING, this may signify that the database manager and controller do not agree on what state the controller should be in.

[0026] Fig. 3 is an example state diagram for a controller that is designated as the non-preferred controller of a controller pair. As shown in Fig. 3, the preferred field is set to false, indicating that the controller is the non-preferred controller. The non-preferred controller is able to transition between the

READY, NOT READY, or STANDBY conditions. The three possible states of the controller are shown in Fig. 3 as state 4 wherein the condition has a value of NOT READY, state 5 wherein the condition has a value of STANDBY, and state 6 wherein the condition of the controller have a value of READY. The transitions between states 4, 5, and 6 occur in the same manner as discussed above regarding the transitions between states shown in Fig. 2 for the preferred controller. [0027] The differences between the state diagrams shown in Figs. 2 and 3 relate to the conditions under which the database manager will instruct the controller to advance to the READY condition or return to the STANDBY condition. For the non-preferred controller of Fig. 3, the database manager will instruct the controller to advance from STANDY (state 5) to READY (state 6) only if the condition of the other controller is NOT READY. Furthermore, if the condition of the non-preferred controller is READY (state 6) and the preferred controller attains a condition of STANDBY (state 2), the database manager will instruct the non-preferred controller to advance from READY (state 6) to

STANDY (state 5).

[0028] Figs. 4A through 4Y shows all the states maintained by the database manager and the transitions between those states according to an example. In the following example, there are 25 possible states, each represented by a separate state table with a unique state number. Each state table represents one possible combination of controller states for controller N, the preferred controller, and controller N+1 , the non-preferred controller. Each of the transitions between states is represented by an arrow and is controlled by a gossip message received from the two controllers.

[0029] Each state table includes four fields for each controller. The four fields, also referred to as states, include condition, status, going ready, and preferred. The condition fields refers to whether the controller has a condition of READY, STANDBY, or NOT READY. This field is reported by the controller as discussed above.

[0030] The status field refers to whether the controller is operational. The status fields can have a value of UP, DOWN, or JOINING. The UP status indicates that the controller is operational, and the DOWN status indicates that the controller is not operational. The JOINING status indicates that controller is transitioning to the STANDBY condition, either from the NOT READY condition after startup or from the READY condition after commanded to return to the STANDBY condition.

[0031] The Going Ready field is a lock field that indicates that the database manager has commanded the relevant controller to transition to the READY condition. The going ready field can have a value of TRUE or FALSE, which indicates whether the controller has been given the lock, i.e., commanded to transition to the READY condition. Only one controller may be given the lock at any given time.

[0032] The preferred field may have the value TRUE or FALSE and indicates whether the controller is the preferred controller or the non-preferred controller. The preferred field is discussed above in relation to Figs. 2 and 3.

[0033] The gossip messages received by the database manager from the controllers indicate the condition of the controller as NOT READY, STANDBY, or READY. The message NOT READY indicates that the controller is not ready to process storage transactions, but is operational. For example, the controller may have just powered up and may be performing startup tasks. Accordingly, as shown in Fig. 1 , the receipt of the NOT READY gossip will cause the status of the corresponding controller to transition from DOWN to UP as shown in the transition from state 1 to state 2 and state 1 to state 3.

[0034] The receipt of the STANDBY condition indicates that the controller is prepared to receive further instruction from the database manager to take control of the storage device pool, i.e., to go ready. Receipt of the READY condition indicates that the storage device has taken control of the storage device pool and is ready to process storage transactions.

[0035] The database manager updates controller states stored to the database as follows. Both controllers will start with condition NOT READY and Status DOWN. One of the Controllers in each pair will have preferred TRUE while the other Controller will have preferred FALSE. As each controller establishes and losses gossip with the manager, the manager will update the database with the received condition that each controller reports, unless a conflict is identified. The database manager will maintain a lock in the form of the Going Ready field and will notify one of the controllers through gossip when that controller has taken the lock (Going Ready = TRUE).

[0036] As mentioned above, there may be instances when the database manager and the controller do not agree on the state that the controller should be in. As shown in the transition from state 1 to state 17 (Fig. 1 A), the database manager has the controller in a status of DOWN and a condition of

NOT READY. In this case, the database manager will not acknowledge the controller's alleged READY condition despite the fact that the controller has reported itself as being in the READY condition. To correct this discrepancy, the database manager instructs the controller through a gossip message to change its status to JOINING. In this case, the JOINING status means the database manager and controller are communicating, but the controller state can't be trusted. The database manager waits for the controller to indicate that it is not servicing storage requests before the controller's status will be moved to UP and allowed to go ready. The status change of the controller is reflected in Fig. 2 by the transition from state 3 to state 2.

[0037] The database manager will maintain the controller status as JOINING until the database manager and the controller are in agreement about the controller's state. For example, the database manager and the controller will be in agreement if the controller indicates through gossip messages that the controller is in the NOT READY or STANDBY condition as shown in Fig. 4Q. When the database manager and controller are in agreement about the controller's state, the controller's status will move to UP.

[0038] As shown by the transition from state 2 to state 1 in Fig. 4B, the database manager may lose communication with controller for various reasons. This is shown in the figures with the label "gossip lost," which indicates that the gossip signal from that controller has been lost. As shown in Fig. 4B, loss of gossip results in the status of the corresponding controller to transition to DOWN.

[0039] The clients are configured to query the database to identify the controller that has a condition of READY and a status of UP to determine where to send data requests. Given that only the latest controller to get the lock will have taken out a backend lock with a persistent reservation, this guarantees that if any controller is presented as condition READY and status UP, it will be the only controller capable of accessing the storage devices. The two lock system will preserve data integrity across time outs and delays in a distributed system while allowing two controllers to share the same storage device pool. [0040] Fig. 5 is an example process flow diagram of a method of initializing a storage system with two storage controllers. The method 500 will be described in reference to the state diagrams of Figs. 4A-Q. The method 500 may be performed by the system 100 of Fig. 1 . The method 500 starts with the system at state 1 (Fig. 4A), wherein controller N and controller N+1 are both down.

[0041] At block 502, the two controllers, N and N+1 , both boot up. At boot up, the controllers perform one or more startup tasks. While performing the startup tasks, each controller will being gossiping to the database manager, starting with a Not_Ready message. Once the database manager receives the Not_Ready message, the database manager will advance that controller status to UP. This is reflected in the state transition from state 1 to state 2 and from state 1 to state 3, as shown in Fig. 4A. When both controllers have an UP status, the system state will be at state 6 (Fig. 4F).

[0042] At block 504, each controller advances to the standby condition as it completes its startup tasks. Once each controller finishes its own startup tasks, the controller begins indicating through gossip messages that it is in the standby condition. Depending on which controller indicates standby first, the system state will advance from state 6 to state 7 or state 8 (See Fig. 4 F).

[0043] At block 506, the controller that is in standby first will be given the lock, meaning that Going Ready field will be set to true for that controller. For example, if the database manager receives the standby message from controller N, the system state will advance from state 6 to state 8, where the controller N condition field is set to standby and the controller N going ready field is set to true.

[0044] At block 508, the controller that is given the lock acquires a backend lock on the devices of the storage device pool. When the backend lock is achieved, the controller sends the READY message through gossip to the database manager. The database manager then advances the system state from state 8 to state 14. At this time, the condition field of controller N is set to ready and the Going Ready field of controller N is set to false. Controller N is not ready to accept storage requests from client devices. The client can access the database to determine which controller has the ready condition to determine which controller to send storage requests.

[0045] At block 510, the remaining controller advances to the standby condition. In this example, controller N+1 sends a standby gossip message to the database manager, and the database manager sets the controller N+1 condition to standby. At this time, system is at state 15 and controller N+1 is ready to take active control of storage request processing in the event that controller N fails.

[0046] In the event that controller N experiences power loss or some other failure, the database manager can initiate a failover process that will cause the controller N+1 to be the active controller. Failure of a controller can be indicated by receiving a NOT READY gossip message from the controller or a failure of the database manager to receive gossip messages from the controller at all. For example, from system state 1 5 (Fig. 40), if controller N gossip signal is lost, the system state returns to state 5, and if the NOT READY message is received from controller N, the system state moves to state 7. In both cases, the controller N condition field is changed to NOT READY and the controller N+1 is given the lock, i.e., the Going Ready field for controller N+1 is set to true.

[0047] It is to be understood that the process flow diagram of Fig. 5 is not intended to indicate that the method is to include all of the blocks shown in Fig. 5 in every case. Further, any number of additional blocks can be included within the method, depending on the details of the specific implementation. In addition, it is to be understood that the process flow diagram of Fig. 5 is not intended to indicate that the method is only to proceed in the order indicated by the blocks shown in Fig. 5 in every case. As indicated by the various possible system state transitions shown in Figs. 4A to 4Y, any number of additional process flows may be performed by the storage system depending on the timing by which gossip messages are received from the storage controllers.

[0048] Fig. 6 is an example process flow diagram summarizing a method of managing a storage system. The method 600 may be performed by the database manager 1 1 2 of Fig. 1 . [0049] At block 602, state information is received from a first storage controller and a second storage controller over a network. The received state information may be stored to a database. The first storage controller and second storage controller are both configured to process storage requests received from a client device. Only one storage controller will have control over the storage device pool at any time. The storage controller that has control of the storage device pool and is ready to process storage transactions may be referred to herein as the active storage controller. Client devices can access the database to determine which storage controller is ready to process storage transactions.

[0050] At block 604, the first storage controller is commanded to take control of the storage device pool. For example, the database manager may command the first storage controller to take control of the storage device pool by setting a Going Ready field to true. This information may then be communicated to the controller in the next gossip message. The Going Ready field can only be true for one controller at a time, and therefore acts as a lock field. Setting the Going Ready field to true for a particular controller may be referred to herein as giving that controller the lock.

[0051] At block 606, a ready message is received from the first storage controller. The ready message indicates that the first storage controller has control of the storage device pool. In response to receiving the command to take control of the storage device pool, the storage controller issues a backend lock on the devices of the storage device pool. Once that is accomplished the controller may issue the READY condition through the next gossip message.

[0052] At block 608, the READY condition of the first storage controller is communicated to a client device to indicate that the first storage controller is ready to process storage transactions. To communicate this information to the client device, the database manager may respond to a request of the client device and obtain the information from the database.

[0053] Fig. 7 is an example block diagram showing a tangible, non-transitory, computer-readable medium that stores code configured to manage a storage system. The computer-readable medium is referred to by the reference number 700. The computer-readable medium 700 can include RAM, a hard disk drive, an array of hard disk drives, an optical drive, an array of optical drives, a nonvolatile memory, a flash drive, a digital versatile disk (DVD), or a compact disk (CD), among others. The computer-readable medium 700 may be accessed by a processor 702 over a computer bus 704. Furthermore, the computer-readable medium 700 may include code configured to perform the methods described herein.

[0054] The various software components discussed herein may be stored on the computer-readable medium 700. A region 706 can include a database manager that controls two or more storage controllers as described, for example, by the state diagrams of Figs 4A-Y. A region 708 can include a storage controller that processes storage transactions and implements state diagrams described in Figs. 2 and 3.

[0055] Although shown as contiguous blocks, the software components can be stored in any order or configuration. For example, if the tangible, non- transitory, computer-readable medium is a hard drive, the software components can be stored in non-contiguous, or even overlapping, sectors.

[0056] While the present techniques may be susceptible to various modifications and alternative forms, the exemplary examples discussed above have been shown only by way of example. It is to be understood that the technique is not intended to be limited to the particular examples disclosed herein. Indeed, the present techniques include all alternatives, modifications, and equivalents falling within the true spirit and scope of the appended claims.

Claims

CLAIMS What is claimed is:

1 . A storage system, comprising:

a first controller coupled to a storage device pool and a second controller coupled to the storage device pool, the first controller and the second controller to process storage requests received from a client device; and

a manager to communicate with the first controller and the second controller over a network, the manager to determine which controller is ready to process storage transactions and communicate to the client device which controller is ready to process storage transactions.

2. The storage system of claim 1 , wherein the manager is a database manager, the storage system comprising a database to store a state of the first controller and the second controller, wherein the client device accesses the database to determine which controller is ready to process storage transactions.

3. The storage system of claim 2, wherein the database manager maintains a lock field in the database for each of the first controller and the second controller, the database manager to:

update the lock field to indicate that the first controller is going ready; and send a going-ready message to the first controller authorizing the first controller to take control of the storage device pool, wherein the first controller issues a backend lock to the storage device pool in response to receiving the going-ready message.

4. The storage system of claim 1 , wherein the first controller and the second controller are to communicate with the manager by gossip signals that are transmitted periodically to indicate status.

5. The storage system of claim 1 , wherein the first controller and the second controller do not have a sideband channel that enables direct communication between the first controller and the second controller.

6. A method for managing a storage system, the method comprising: receiving state information from a first storage controller and a second storage controller over a network, the first storage controller and second storage controller both configured to process storage requests received from a client device;

commanding the first storage controller to take control of a storage device pool;

receiving a READY message from the first storage controller, the READY message indicating the first storage controller has control of the storage device pool; and

in response to the READY message, communicating to a client device that the first storage controller is ready to process storage transactions.

7. The method of claim 6, comprising storing a state of the first storage controller and the second storage controller to a database, wherein the client device accesses the database to determine which storage controller is ready to process storage transactions.

8. The method of claim 7, comprising:

maintaining a lock field in the database for each of the first storage controller and the second storage controller;

updating the lock field to indicate that the first storage controller is going ready; and

wherein commanding the first storage controller to take control of the storage device pool comprises sending a going-ready message to the first storage controller, wherein the first storage controller issues a backend lock to the storage device pool in response to receiving the going-ready message.

9. The method of claim 7, comprising, in response to losing communication with the first storage controller, updating the database to indicate that the first storage controller is down and commanding the second storage controller to take control of the storage device pool.

10. The method of claim 6, wherein the first storage controller and the second storage controller do not have a sideband channel that enables direct communication between the first storage controller and the second storage controller.

1 1 . A tangible, non-transitory, computer-readable medium comprising instructions for managing a storage system and that direct a processor to: receive state information from a first storage controller and a second storage controller over a network, the first storage controller and second storage controller both configured to process storage requests received from a client device;

command the first storage controller to take control of a storage device pool;

receive a ready state from the first storage controller, the ready state indicating the first storage controller has control of the storage device pool; and in response to the ready state, communicate to a client device that the first storage controller is ready to process storage transactions.

12. The computer-readable medium of claim 1 1 , comprising instructions that direct the processor to store a state of the first storage controller and the second storage controller to a database, wherein the client device accesses the database to determine which storage controller is ready to process storage transactions.

13. The computer-readable medium of claim 12, comprising instructions that direct the processor to:

maintain a lock field in the database for each of the first storage controller and the second storage controller;

update the lock field to indicate that the first storage controller is going ready; and

send a going-ready message to the first storage controller to command the first storage controller to take control of the storage device pool, wherein the first storage controller issues a backend lock to the storage device pool in response to receiving the going-ready message.

14. The computer-readable medium of claim 12, comprising instructions that direct the processor to, in response to losing communication with the first storage controller, update the database to indicate that the first storage controller is down and command the second storage controller to take control of the storage device pool.

15. The computer-readable medium of claim 1 1 , wherein the first storage controller and the second storage controller do not have a sideband channel that enables direct communication between the first storage controller and the second storage controller.