JP6731553B2 - Distributed storage system and distributed storage control method - Google Patents

Description

The present invention relates generally to storage control in distributed storage systems.

As a technique related to distributed storage, for example, the techniques disclosed in Patent Documents 1 and 2 are known.

Patent Document 1 discloses the following, for example. That is, of the plurality of servers that form an SDS (Software Defined Storage) grid, the first server receives an I/O (Input/Output) request from the host computer. If the first server identifies that the second server will handle the I/O request based on a local grid data map that shows the location of all data managed by the SDS grid, then the I/O request is processed. Transfer to the second server.

Patent Document 2 discloses the following, for example. Multiple storage devices are connected to the storage system. Each of the plurality of storage devices has a plurality of storage blocks. A storage system buffers multiple write requests and writes data to a defined group of storage blocks.

US2016/0173598 Japanese Unexamined Patent Publication No. 2010-079928

In the following description, a computer as an element of the distributed storage system may be referred to as a “node”. Any computer having computing resources such as a processor, memory or communication interface device may be able to be a node. The node may be a physical computer (for example, a general-purpose computer or a physical storage device), or a virtual computer that operates based on at least part of the computational resources of the physical computer. Good. One physical computer executes a virtual computer such as a host that issues an I/O request and a virtual computer (for example, SDS) such as a storage device that receives and processes the I/O request. You may.

Also, in the following description, a redundant configuration group is formed by a plurality of nodes. Examples of the redundant configuration include erasure coding, RAIN (Redundant Array of Independent Nodes), inter-node mirroring, and RAID (Redundant Array of Independent (or Inexpensive) Disks) in which a node is regarded as one drive, and any of them may be used. .. Other methods (methods of forming a redundant configuration group between nodes) may be adopted.

Therefore, in the following description, the “redundant configuration group” may be a group configured by two or more storage areas provided by two or more nodes to store data.

Further, the definition of each of the plurality of types of storage areas in the following description is as follows.
-The "redundant configuration area" is a logical storage area provided by the redundant configuration group.
-A "node area" is a logical storage area provided by each of a plurality of nodes. A plurality of node areas provided by a plurality of nodes respectively configure a redundant configuration area.
-A "strip" is a part of a node area. The strip stores the user data set or parity. The strip in which the user data set is stored can be called a “user strip”, and the strip in which the parity is stored can be called a “parity strip”. The “user data set” is a part of the user data unit as at least a part of the user data (write target data) that complies with the write request. A "user data unit" is a set of all user data sets corresponding to stripes. "Parity" is a data set generated based on user data units. A "data set" is data stored in a strip, which in the following description is a user data set or parity. That is, the data set is data in strip units.
A “stripe” is a storage area composed of two or more strips (for example, two or more strips having the same logical address) respectively existing in two or more node areas in the redundant configuration area.

In a distributed storage system (for example, a scale-out type storage system), each node performs inter-node transfer if the write-destination strip of the updated data set is not the strip at that node, that is, the write-destination strip. The updated data set is transferred to the node having.

Transfer between nodes is performed in strip units. Therefore, each node does not have N write-destination strips respectively corresponding to N (N is a natural number) updated data set, and if none of the N write-destination strips exists in the node, the write-destination strips of each N are written. , Transfer between nodes. That is, transfer between nodes is performed N times. The inter-node transfer is one of the causes for lowering the performance (for example, I/O performance) of the distributed storage system due to the communication overhead.

A first node, which is any one of the plurality of nodes, manages, for each node, presence/absence of a transfer target data set, which is a data set to which a strip in a node area of the node is a write destination, in the first node. Holds node management information. The first node, for each second node that is each node other than the first node of the plurality of nodes,
(A) Two or more transfer target data sets to which two or more strips (that is, two or more strips corresponding to the second node of the two or more stripes) in the node area of the second node are write destinations When it is specified based on the node management information that there is a position, two or more positions in the node respectively corresponding to the two or more transfer target data sets are specified based on the node management information,
(B) One transfer command in which two or more transfer target data sets respectively existing in the identified two or more node positions are transferred is transmitted to the second node.

It is possible to reduce performance degradation of the distributed storage system.

An outline of the embodiment is shown. 1 shows a physical configuration of the entire system including a distributed storage system. 1 shows a logical configuration of the entire system including a distributed storage system. The logical configuration of a node is shown. 7 shows a flow of host write processing. An example of a log structured light is shown. The flow of asynchronous transfer processing is shown. The flow of maximum transfer length control processing is shown.

An embodiment will be described below.

In the following description, the “interface unit” includes one or more interfaces. The one or more interfaces may be one or more same type interface devices (for example, one or more NICs (Network Interface Cards)) or two or more different types of interface devices (for example, NICs and HBAs (Host Bus Adapters)). It may be.

Further, in the following description, the “storage unit” includes at least the memory unit of the memory unit and the PDEV unit. The PDEV unit includes at least one PDEV. The memory unit includes one or more memories. The at least one memory may be a volatile memory or a non-volatile memory. The storage unit is mainly used during processing by the processor unit.

Further, in the following description, the “processor unit” includes one or more processors. At least one processor is typically a CPU (Central Processing Unit). The processor may include a hardware circuit that performs a part or all of the processing.

Further, in the following description, the information may be described by an expression such as “xxx table”, but the information may be expressed by any data structure. That is, the “xxx table” can be referred to as “xxx information” to indicate that the information does not depend on the data structure. Further, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or a part of the two or more tables may be one table. Good.

Further, in the following description, the process may be described by using “program” as a subject, but the program is executed by a processor (for example, a CPU (Central Processing Unit)) to appropriately perform a predetermined process. Since the processing is performed using the storage unit (for example, memory) and/or the interface device (for example, communication port), the subject of the process may be the processor (or the device or system having the processor). Further, the processor may include a hardware circuit that performs a part or all of the processing. The program may be installed in a device such as a computer from the program source. The program source may be, for example, a program distribution server or a computer-readable (for example, non-transitory) recording medium. Further, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

In the following description, “PDEV” means a physical storage device, and is typically a non-volatile storage device (for example, auxiliary storage device), for example, HDD (Hard Disk Drive) or SSD (Solid State). Drive).

Further, the definition of each of the plurality of types of storage areas is as follows.
“Cash strip” is a storage area corresponding to a strip and is a storage area on a CM (cache memory). It should be noted that both the cash strip and the strip may have a fixed size.
A “cache stripe” is a storage area corresponding to a stripe and is a storage area on a CM. The cache stripe is composed of two or more cache strips respectively corresponding to two or more strips forming a stripe corresponding to the cache stripe. The data set in the cache strip (user data set or parity) is written to the node area which has a strip corresponding to that cache strip.
The “cache node area” is a storage area corresponding to the node area and is a storage area on the CM. The cache node area is composed of two or more cache strips respectively corresponding to two or more strips forming the node area corresponding to the cache node area.
-"VOL" is an abbreviation for logical volume, which is a logical storage area provided to the host. The VOL may be a substantial VOL (RVOL) or a virtual VOL (VVOL). The “RVOL” may be a VOL based on a physical storage resource (for example, one or more PDEV) included in the storage system that provides the RVOL. The “VVOL” may be a VOL that is configured by a plurality of virtual areas (virtual storage areas) and that complies with the capacity virtualization technology (typically Thin Provisioning).

Further, in the following description, when the same type of element is described without distinction, a reference numeral is used, and when the same type of element is described separately, an element ID (for example, an identification number) may be used. is there. For example, when description is made without particularly distinguishing nodes, it is described as "node 101", and when description is made by distinguishing individual nodes, it is described as "node 0" or "node 1". is there. In addition, in the following description, by adding n to the name of the element in the node n (n is an integer of 0 or more), which node is the element (or which node corresponds to which node) Can be distinguished.

FIG. 1 shows an outline of the embodiment.

The distributed storage system 100 has a plurality of nodes 101, for example, nodes 0-4. Each node 101 provides a node area 52. The node areas 0 to 4 form the redundant configuration area 53. Also, the node areas 0 to 4 are associated with the VOL 54 provided by the nodes 0 to 4.

Each node 101 has a CM (cache memory) 51. The CM 51 may be one or more memories or a partial area of one or more memories. The CM 51 temporarily stores, for example, a write target user data set according to a write request, a read target user data set according to a read request, parity, and a data set according to a transfer command from another node. The storage capacity of the CM 51 is typically smaller than the storage capacity of the node area 52. At least a part of the CM 51 is logically a plurality of cash strips in a matrix (hereinafter, cash strip matrix). The plurality of cache strip rows are the plurality of cache stripes 56 (ie, each of the plurality of cache strip rows corresponds to a plurality of stripes). The plurality of cache strip rows are the plurality of cache node areas 57 (that is, the plurality of cache strip rows correspond to the plurality of nodes 101, respectively). Note that the correspondence between the area address and the strip address in the CM 51 changes dynamically. For example, in the cache node area 0, the second cache strip and the third cache strip are continuous, but in the node area 0, the strip corresponding to the second cache strip and the third cache strip are associated. It is not always continuous with the strip.

Each node 101 has a controller 60. The controller 60 is an example of a function exerted by executing one or more computer programs. The controller 60 controls input/output of a data set and the like.

In each node 101, the controller 60 manages the node management bitmap 102. The controller 60 updates the node management bitmap 102 according to the update of the CM 51. The node management bitmap 102 is an example of node management information that manages, for each node, an in-node position where a transfer target data set exists. The node management bitmap 102 is composed of a plurality of sub-bitmaps 70 respectively corresponding to a plurality of nodes. The sub-bitmap 70 is an example of sub-node management information. The plurality of sub-bitmaps 70 respectively correspond to the plurality of cache node areas 57. For each sub-bitmap 70, two or more bits respectively correspond to two or more cache strips forming the cache node area 57 corresponding to the sub-bitmap 70. Bit "0" means that the data set in the cache strip corresponding to that bit is not transferred. Bit "1" means that the data set in the cache strip corresponding to that bit is to be transferred. A cache strip in which a data set to be transferred is stored can be called a “transfer target cache strip”. Further, the sub-bitmap 70 corresponding to the node n can be referred to as “sub-bitmap n”, and the cache node area 57 corresponding to the node n can be referred to as “cache node area n”. As will be described later in detail, the length of the sub-bitmap 70 (in other words, the number of bits forming the sub-bitmap) can be changed. The length of the sub-bitmap 70 corresponds to the maximum transfer length. The “maximum transfer length” may be the total amount of transferable data sets. Further, in the present embodiment, since the data set is written in the CM 51 by the log structured method described later, it can be expected that two or more transfer target cache strips tend to be continuous in each node.

Hereinafter, in order to make the description easy to understand, the node 0 is mainly taken as an example for describing each node 101. That is, in the following description, the node 0 is an example of the first node (or its own node), and each of the nodes 1 to 3 is an example of the second node (or another node).

According to this embodiment, the node 0 has the node management bitmap 102 for managing the position of the transfer target cache strip for each node. By referring to the node management bitmap 0, the controller 0 can specify all the transfer target cache strips for each of the cache node areas 0 to 3.

The controller 0 transfers to all of the nodes 1 to 3 in one transfer between nodes (in other words, with one transfer command (command for transfer between nodes)) in all the specified transfer target cache strips. Can be transferred regardless of whether all the specified transfer target cache strips are continuous cache strips. That is, it is possible to reduce the number of inter-node transfers (in other words, the number of transfer commands issued). In other words, multiple inter-node transfers can be combined into one inter-node transfer. As a result, the performance degradation of the distributed storage system 100 can be reduced.

Specifically, for example, the controller 0 has three consecutive caches in the cache node area 1 corresponding to three consecutive bits “1” (first bit to third bit) in the sub-bitmap 1. The strip is identified as three transfer target cache strips. The controller 0 makes one transfer command (for example, the address and transfer length of the head write destination strip of the node area 1) in which three data sets D1 in three consecutive transfer target cache strips are transfer targets (write targets). Command that specifies and) is transferred to the node 1. The "transfer length" is a length equal to or shorter than the maximum transfer length. As described above, it can be expected that the transfer target cache strips tend to be continuous, so that it is easy to transfer two or more transfer target data sets with one transfer command. Specifically, the start address and the transfer length may be specified with one transfer command.

Similarly, for example, the controller 0 creates three consecutive cache strips in the cache node area 2 corresponding to three consecutive bits “1” (second bit to fourth bit) in the sub-bitmap 2, respectively. It is specified as three transfer target cash strips. The controller 0 transfers to the node 2 one transfer command in which three data sets D2 in three consecutive transfer target cache strips are transferred.

As described above, two or more data sets (two or more data sets each having two or more strips as write destinations) to be transferred by one transfer command for one node are two or more consecutive cache strips. It is not limited to two or more data sets in the above, and may be two or more data sets in two or more non-contiguous cash strips. In that case, the method of the scatter-gather list (SGL) can be applied (the address for the non-transfer target data set can be designated by the transfer command). Specifically, for example, the controller 0 corresponds to two non-consecutive bits “1” (first bit and fourth bit) in the sub-bitmap 3 in the cache node area 3 respectively. One cash strip is specified as two transfer target cash strips. The controller 0 uses a single transfer command (for example, the address and transfer length of the head write destination strip of the node area 3 and A command specifying the start address and the data length (offset) of the write destination strip is transferred to the node 3. That is, in the case where the transfer target cache strips are non-continuous, the number of parameters specified for transferring two or more transfer target data sets with one transfer command increases in comparison with the case where the transfer target cache strips are continuous. However, even if the transfer target cache strips are discontinuous, it is possible to transfer two or more transfer target data sets with one transfer command.

Further, the controller 0 can also write two or more data sets to two or more strips with one write (that is, one write command) to the node area 0 as well. Specifically, for example, the controller 0 has two consecutive caches in the cache node area 0 corresponding to two consecutive bits “1” (the second bit and the third bit) in the sub-bitmap 0, respectively. The strip is specified as two write target cache strips. The controller 0 issues one write command for writing two data sets D0 in two consecutive cache strips for writing. Thereby, the two data sets D0 are written in the two strips in the node area 0.

Hereinafter, this embodiment will be described in detail.

FIG. 2 shows the physical configuration of the entire system including the distributed storage system 100.

One or more hosts 201, management system 203, and distributed storage system 100 are connected to network 240. The network 240 may include at least one of an FC (Fibre Channel) network and an IP (Internet Protocol) network, for example.

The host 201 issues an I/O request for user data. The host 201 may be a physical computer or a virtual computer executed by a physical computer. The host 201 as a virtual computer may be executed in the node 101. Specifically, for example, in the same node 101, a virtual computer as the host 201 and a virtual computer (eg, SDS) as a storage (controller 60) that receives and processes an I/O request from the host 201. (Software Defined Storage) and may be executed.

The management system 203 manages the distributed storage system 100. The management system 203 may be composed of one or more computers (one or more physical computers or virtual computers). Specifically, for example, when the management computer has a display device and the management computer displays information on its own display device, the management computer may be the management system 203. Also, for example, when a management computer (for example, a server) sends display information to a remote display computer (for example, a client) and the display computer displays the information (when the management computer displays information on the display computer) ), a system including at least the management computer of the management computer and the display computer may be the management system 203.

The distributed storage system 100 includes a plurality of nodes 101 connected to a network 240. Each node 101 has an interface unit 251, a PDEV unit 252, a memory unit 253, and a processor unit 254 connected to them. For example, the interface unit 251 is connected to the network 240, and inter-node transfer is performed via the interface unit 251. The logical storage area based on the PDEV unit 252 is a node area. The memory unit 253 stores one or more programs and the node management bitmap 102 described above. The processor unit 254 executes one or more programs.

FIG. 3 shows a logical configuration of the entire system including the distributed storage system 100.

Two or more VOLs (for example, VOL A1 and VOL A2) that the two or more nodes 101 respectively have are provided to the host 201 as one VOL (for example, VOL A). The host 201 sends an I/O request that specifies VOL A to node 0 that provides VOL A1 or node 1 that provides VOL A2.

Each node 101 has the controller 60 as described above. The controller 60 has a data plane 311 and a control plane 312. The data plane 311 provides the VOL and performs processing according to the I/O request from the host 201. The control plane 312 performs various controls. The control plane 312 has a control master 322 and a control agent 321. The control master 322 receives an instruction from the management system 203 and sends a control command according to the instruction to one or more control agents 321. The control agent 321 controls according to a control command.

Each node 101 specifies and transfers a transfer target data set for each node based on the node management bitmap 102. In the distributed storage system 100, data consistency can be maintained by, for example, one of the following methods.
-Each node becomes the responsible node for each stripe. If the node in charge of the write-destination stripe corresponding to the write request is the self node, the node 101 writes the write-target user data to the CM 51, and transfers the transfer-target data set for that stripe to another node. The node 101 transfers the write request to the responsible node (other node) if the responsible node of the write destination stripe corresponding to the write request is not the own node.
Each node 101 has, for its own node, a first CM part in which a data set complying with a write request is written and a second CM part in which a data set complying with the received transfer command is written. The one storing the newer data set is adopted as the write target for the strip in the node area of the own node.

FIG. 4 shows a logical configuration of the node 101.

In the node 101, the data plane 311 includes a front end program 421, a control program 422, a cache program 423, an address conversion program 424, a data transfer program 425, and a back end program 426. The data plane 0 manages the node management bitmap 102. The front-end program 421 receives an I/O request and returns a response to the I/O request. The control program 422 executes the processing of the received I/O request or executes the transfer processing asynchronously with the processing of the I/O request. The cache program 423 updates the CM 51 and the node management bitmap 102. The address conversion program 424 converts the cache address (CM logical address) into a strip address (strip logical address). The data transfer program 425 sends a transfer command that specifies one or more transfer target data sets, or sends a write command that specifies one or more write target data sets. In response to the write command, the back-end program 426 writes the one or more write target data sets designated by the write command to one or more strips.

In the node 101, the control plane 312 includes a CLI (Command Line Interface) program 431, a GUI (Graphical User Interface) program 432, a REST (REpresentational State Transfer) server program 433, a control agent 321, a control master 322, and a maintenance program 434. .. The CLI program 431 receives an instruction from the user of the host 201 via the CLI. The GUI program 432 receives an instruction from the user of the host 201 via the GUI. The REST server program 433 receives an instruction in the node 101 from an external program (for example, an application program (not shown)) which is at least one program outside the controller 60 (for example, SDS). The REST server program 433 can issue an instruction to the maintenance program 434 according to an instruction from an external program, for example. The control agent 321 receives an instruction from the control master 322 in at least one node 101. The control master 322 issues an instruction to the control agent 321 in at least one node 101. The maintenance program 434 receives an instruction from the REST server program 433 and performs maintenance according to the instruction (for example, changing the maximum transfer length (length of sub-bitmap) corresponding to at least one node).

Hereinafter, an example of the processing performed in this embodiment will be described. Also, the node 0 is taken as an example.

FIG. 5 shows the flow of host write processing.

The front-end program 0 receives a user data write request from the host 201 (S501). The write request includes write destination information that is information indicating the write destination. The write destination information includes, for example, the ID of the write destination VOL (for example, LUN (Logical Unit Number)) and the logical address of the write destination area in the write destination VOL.

The front end program 0 transfers the write request to the control program 0 (S502).

The control program 0 analyzes the write request (S503). For example, the control program 0 specifies that the request is a write request and the write destination information in the write request.

The control program 0 instructs the cache program 0 to cache the user data according to the write request (S504).

In response to the instruction, the cache program 0 performs the log structured write of the user data to the CM 0, that is, writes the user data to the CM 0 by the log structured method (S505). Note that the log structured write may be performed in the process performed in response to the write request as shown in FIG. 5, or may be performed asynchronously with the process performed in response to the write request ( For example, after writing user data in CM0, log structured write may be performed so that the discrete data sets in CM0 are continuous).

Further, the cache program 0 updates the node management bitmap 0 with the update of CM0 in S505 (S506). For example, when a data set according to user data having a new VOL area (area in VOL) as a write destination is written in the cache strip, the cache program 0 changes the value of the bit corresponding to the cache strip from “0” to “0”. Update to 1".

The cache program 0 returns a response to the instruction of S504 to the control program 0 (S507). When the control program 0 receives the response, it returns a response to the request of S502 to the front end program 0 (S508). When receiving the response, the front-end program 0 returns a completion response to the write request of S501 to the host 201 (S509).

An example of the log structured write will be described with reference to FIG.

When the new user data unit X is the write target, the cache program 0 sets the user data sets x1, x2, and x3 forming the user data unit X and the parity xP based on these user data sets as the first data. Write to a cache stripe (consecutive cache strips).

Next, when the new user data unit Y is the write target, the cache program 0 sets the user data sets y1, y2, and y3 that form the user data unit Y and the parity yP based on these user data sets. Writing is performed to a second cache stripe that is a cache stripe next to the first cache stripe (specifically, a second cache stripe starting with a cache strip next to the end cache strip of the first cache stripe).

Next, when the user data unit X'for updating all the user data units X is a write target, the cache program 0 sets the user data sets x1', x2', and x3' that compose the user data unit X'. And the parity xP′ based on those user data sets are written in the third cache stripe, which is the cache stripe next to the second cache stripe. Further, the cache program 0 manages three cache strips respectively storing the user data sets x1, x2, and x3 forming the user data unit X before update as free areas. The cache strip in which the pre-update parity xP is stored may also be managed as a free area.

In this way, according to the log structured write, the area in which the pre-update data set is stored is not overwritten with the post-update data set, but a new area is reserved for the post-update data set, and the area However, it will be a new cash strip. As a result, even if random write is performed with non-contiguous write destination addresses, a data area with continuous addresses (transfer target cache strip with continuous addresses) can be obtained in CM0 having a constant address length.

Note that the three write destination strips respectively corresponding to the updated user data sets x1′, x2′, and x3′ are the same as the three write destination strips respectively corresponding to the pre-update user data sets x1, x2, and x3. The program 0 may not update the node management bitmap 0 (for example, the value of the bit corresponding to the cache strip storing the pre-update user data set x1 may remain “1”). However, since the address of the cache strip in which the transfer target data set is stored has been changed, the cache program 0 stores the cache address corresponding to bit “1” and the pre-update user data set (eg, x1). The address of the cache strip may be changed to the address of the cache strip that stores the updated user data set (eg, x1′).

FIG. 7 shows the flow of asynchronous transfer processing. The asynchronous transfer process is a transfer process that is performed asynchronously with the host write process, and includes a process of transferring the transfer target data to the transfer destination node.

The control program 0 instructs the cache program 0 to acquire the lock (S701).

The cache program 0 performs the following processing in response to the instruction (S702).
The cache program 0 refers to the node management bitmap 102 and specifies the transfer target cache strip for each of the nodes 0 to 3. Hereinafter, the node 1 will be taken as an example.
The cache program 0 makes a lock determination for the node 1, which is a determination as to whether or not consecutive cache strips including all specified transfer target cache strips can be locked.
-If the result of the lock judgment is true, the cache program 0 locks consecutive cache strips. Data sets in locked cache strips are never updated during host write operations.

The cache program 0 returns a response indicating the result of S702 to the control program 0 as a response to the instruction of S701 (S703).

The control program 0 specifies the cache address of each transfer target cache strip among the locked consecutive cache strips (S704). The control program 0 instructs the address translation program 0 to translate the cache address specified in S704. In response to the instruction, the address translation program 0 identifies the strip address corresponding to the cache address (S706) and returns the identified strip address to the control program 0 (S707).

The control program 0 reads the pre-update data set from the node 1 as needed (for example, when the node 0 is in charge of parity generation for the stripe including the strip) and generates parity (S708).

The control program 0 instructs the data transfer program 0 to cache the node 1 (S709). In the instruction, the number of strips and the strip address group are designated. The number of strips is, for example, the number of cache strips locked in S702 for the node 1 or the number of transfer target cache strips of the cache strips locked in S702 for the node 1. The strip address group may be addresses of one or more strips in the node area 1, or may be a set of strip addresses and transfer lengths. In response to the instruction, the data transfer program 0 transmits a lock request to the node 1 (S710). The lock request also specifies the number of strips and the strip address group. That is, the lock request is a request for locking (securing) the same number of cache strips as the designated number of strips from the CM of the node 1. In response to the lock request, the node 1 (for example, the controller 1) locks the same number of areas (cache strips) as the specified number of strips from the CM 1 (for example, locks the cache strip from the cache node area 1). , And the strip address group is associated with these areas (S711). The node 1 (for example, the controller 1) returns a response to the instruction of S710 to the data transfer program 0 (S712). The data transfer program 0 returns a response to the instruction of S709 to the control program 0 (S713).

The control program 0 instructs the data transfer program 0 to transfer data (S714).

In response to that instruction:
The data transfer program 0 generates a write command for all transfer target data sets to the node area 0 (S715). In the write command, a strip address group (addresses of strips in the node area 0) is designated. The data transfer program 0 instructs the backend program 0 to send the write command. In response to the instruction, the backend program 0 transmits a write command for the node area 0 (S716). The backend program 0 returns a response to the data transfer program 0 (S718).
The data transfer program 0 generates a transfer command for all transfer target data sets to the node 1 (S715). The transfer command specifies the strip address group (addresses of strips in the node area 1). The data transfer program 0 transfers the transfer command to the node 1 (S717). In response to the transfer command, the node 1 (for example, the controller 1) writes the transfer target data set according to the transfer command in the cache strip locked in S711, and returns the response to the data transfer program 0 (S719). .. In the node 1, the controller 1 writes the data set to be transferred from the cache strip to the strip in the node area 1.

The data transfer program 0 returns a response to the instruction of S714 to the control program 0 (S720).

The control program 0 instructs the cache program 0 to release the lock (S721). In response to the instruction, the cache program 0 releases the lock acquired in S702 (S722). The cache program 0 returns a response to the instruction of S721 to the control program 0 (S723).

The above is the asynchronous transfer process.

In the asynchronous transfer process, the transfer source node sends a transfer command to the transfer destination node to transfer the transfer target data set from the transfer source node to the transfer destination node (push type transfer). The transfer target data set may be transferred from the transfer source node to the transfer destination node by transmitting a transfer request to the transfer source node (pull type transfer).

Further, the result of the above-mentioned lock determination (determination as to whether or not consecutive cache strips including all specified transfer target cache strips can be locked) is false in the following cases.
• At least one of the locked cash strips is already locked.
When the cache strip to be locked is locked, the lock ratio (ratio of the total capacity of locked cache strips to the capacity of CM) exceeds the threshold value.

When the result of the lock determination is false, the controller 0 may perform processing different from the asynchronous transfer processing (for example, lock determination regarding the cache strip in the cache node area corresponding to another node).

The maximum transfer length (the length of the sub-bitmap 70 in the node management table 102) does not have to be the same for all nodes 101. The maximum transfer length can be set or changed as follows, for example.

FIG. 8 shows the flow of maximum transfer length control processing. The maximum transfer length control process is executed in the node 101. Hereinafter, the node 0 will be taken as an example.

When a setting event for setting the maximum transfer length occurs (S801: YES), the maximum transfer length is set in the node 0 (S802). That is, the sub-bitmap 70 is set in the node 0 for each of the nodes 0 to 3. The “setting event” mentioned here may be, for example, one of the following.
The CLI program 0 has received a setting instruction specifying the maximum transfer length from the host 201.
The GUI program 0 receives the setting instruction designating the maximum transfer length from the host 201.
The maintenance program 434 receives a setting instruction designating the maximum transfer length from an external program via the REST server program 0.
The control master 0 receives a setting instruction designating the maximum transfer length from the management system 203.

The designated maximum transfer length may be determined by any of the management system 203, the external program, and the controller 0.

The maximum transfer length is based on the performance value of the controller 0 (for example, SDS). Specifically, the maximum transfer length is based on at least one of the following, for example. In the following description, “K is relatively large (or small)” means “a certain K value k2 is larger (or smaller) than a certain K value k1”. To do. Further, "the maximum transfer length is relatively small (or large)" means "the maximum transfer length t2 corresponding to another certain K value k2 than the maximum transfer length t1 corresponding to a certain K value k1. Means smaller (or larger)".
Target I/O processing performance for node 0 (amount of data input/output per unit time). For example, when the I/O processing performance is relatively high, the maximum transfer length is relatively small. This is to reduce the influence of the asynchronous transfer processing on the performance of the host I/O processing. The “I/O processing performance” is the processing performance of an I/O request (I/O request received from the host in this embodiment). The “I/O” may be a write, a read, or both a write and a read.
-Network bandwidth of the interface unit installed in node 0. For example, the smaller the network bandwidth, the smaller the maximum transfer length. This is because transfer that exceeds the performance of the physical hardware called the interface unit cannot be performed.
-Multiplicity of host write processing. For example, when the multiplicity is relatively large, the maximum transfer length is relatively small. This is because if the multiplicity is large, the amount of resources available for asynchronous transfer processing is stressed.
-The number of nodes configuring the redundant configuration area 53. For example, when the number of nodes is relatively large, the maximum transfer length is relatively small. This is because when the number of nodes is large, the number of sub-bitmaps increases, and the size of the node management bitmap tends to increase accordingly.

When the change event for changing the set maximum transfer length for at least one node occurs after the operation (S803) (S804: YES), the set maximum transfer length for the node 0 is changed (S805), The changed maximum transfer length is set (S806). For example, the sub-bitmap 70 becomes longer or shorter. The “change event” referred to here may be, for example, one of the following.
The CLI program 0 has received from the host 201 a change instruction designating the maximum transfer length after the change.
The GUI program 0 receives the change instruction designating the maximum transfer length after the change from the host 201.
The maintenance program 434 receives a change instruction specifying the maximum transfer length after the change from the external program via the REST server program 0.
The maintenance program 434 determines that the load of the node 0 (for example, the load of computing resources (hardware) or the operating status of one or more external programs) is greater than or equal to any one of the thresholds of 1 or more, or It was detected that the value fell below either threshold.
The control master 0 receives a change instruction designating the maximum transfer length after the change from the management system 203.
-A certain amount of time has passed since the previous determination in S804.
-The current time has reached the specified time.

The changed maximum transfer length is based on at least one of the following, for example.
-Performance value of the calculation resource (hardware) of node 0. For example, the controller 0 (for example, the maintenance program 0) predicts the limit value of the band based on the performance value of the calculation resource of the node 0 (for example, the free capacity of CM0) (or the predicted value of the limit value of the band is calculated). The maximum transfer length is changed based on the predicted value of the limit value (received from the outside such as the management system 203).
-Operation status of one or more external programs. For example, when one or a plurality of external programs (for example, a certain external program) is newly started or stopped, the controller 0 (for example, REST server program 0) notifies the new start or stop of the one or a plurality of external programs. The notification is received from at least one of the one or more external programs, or the controller 0 (for example, the maintenance program 0) receives the notification of the new operation or stop of the one or more external programs from the management system 203. The controller 0 (for example, the REST server program 0 or the maintenance program 0) that has received the notification makes the maximum transfer length relatively small if one or more external programs newly operate, and the one or more external programs If it is newly stopped, the maximum transfer length is made relatively large.

Although some embodiments have been described above, these are merely examples for explaining the present invention, and the scope of the present invention is not limited only to these embodiments. The present invention can be implemented in various other forms.

For example, the log structured write does not necessarily have to be adopted as the write for the CM 51.

Further, for example, in order to maintain the redundancy of the untransferred transfer target data set, the node 101 writes the transfer target data set in the PDEV unit, and after the inter-node transfer of the transfer target data set is completed, the transfer target data set is transferred. May be deleted from the PDEV section.

Further, for example, the transfer target data set can be referred to as a “dirty data set” that is a data set that is not destaged from the CM. Also, for example, a cache strip that stores a transfer target data set can be referred to as a “dirty cache strip”.

101... Node

Claims (14)

Has multiple nodes ,
Each of the plurality of nodes has a node area which is a logical storage area composed of one or more strips,
The plurality of nodes form a redundant configuration area from two or more node areas,
The redundant area is a computer program that the two or more nodes region performs write target data transfer between the nodes a stripe composed of two or more strips each having at including distributed storage system,
The computer program is executed by a first node, which is any one of the plurality of nodes,
The first node is
A cache memory for storing a data set written in the node area of each of the plurality of nodes;
The first node of the transfer target data set, which is the data set stored in the cache memory, to which the strip in the node area of each of the plurality of second nodes other than the first node among the plurality of nodes becomes a write destination holds, and node management information for managing the presence in,
The computer program, in the first node,
(A) When it is specified based on the node management information that there are two or more transfer target data sets to which two or more strips in the node area of the one second node are write destinations, respectively, Specify the cache strip of the data set to be transferred based on the node management information,
(B) One transfer command in which two or more transfer target data sets respectively existing in two or more node positions corresponding to the specified cash strip are transferred is transmitted to the second node.
A computer program for executing two or more to transmit two or more write target data to the second node by one transfer command . In the first node, for each of the second nodes, in (B),
(B1) lock a plurality of cache strips corresponding to the second node, the cache strips including two or more cache strips respectively storing the two or more transfer target data sets,
(B2) transmitting the transfer command of the two or more transfer target data sets in the plurality of locked cache strips to the second node,
The computer program according to claim 1, which executes the program. The first node is configured to write a data set to the cache memory in the first node in a log structured manner,
In (b1), the plurality of cash strips to be locked are the two or more consecutive cash strips.
The computer program according to claim 2 . The first node is configured to write a data set to the cache memory in the first node in a log structured manner,
In (b1), the plurality of cash strips to be locked are continuous and include the two or more non-continuous cash strips.
The computer program according to claim 2 . The node management information includes subnode management information for each node,
The position in each node managed by the subnode management information is a cache strip corresponding to the node corresponding to the information, and a cache strip corresponding to the write destination strip of the transfer target data set,
For each node, the maximum number of cache strips that the first node can lock depends on the size of the subnode management information corresponding to that node,
For each node, the size of the subnode management information depends on the maximum transfer length, which is the total amount of data sets that the first node can transfer,
The computer program according to claim 2 . In the first node, for at least one of the plurality of nodes, the maximum transfer length is based on at least one of the following:
(P1) I/O processing performance targeted for the node,
(P2) Network bandwidth of the node,
(P3) I/O processing multiplicity, and
(P4) the number of nodes that provide a plurality of node areas forming the redundant configuration area,
The computer program according to claim 5 . When (p1) is adopted, the maximum transfer length is relatively small when the I/O processing performance is relatively high,
When (p2) is adopted, the maximum transfer length is relatively small when the network bandwidth is relatively small,
When (p3) is adopted, the maximum transfer length is relatively small when the I/O processing multiplicity is relatively large,
When (p4) is adopted, the maximum transfer length is relatively small when the number of nodes is relatively large,
The computer program according to claim 6 . After the operation of the first node, the first node causes the first node to change the maximum transfer length for at least one of the plurality of nodes based on at least one of the following:
(Q1) the performance value of the computational resource of the first node, and
(Q2) Operation status of one or more external programs that are programs executed by the first node and are one or more programs other than the computer program,
The computer program according to claim 5 . When (q1) is adopted, the maximum transfer length is changed based on the predicted value of the bandwidth limit value based on the performance value of the calculation resource of the first node,
When (q2) is adopted, the maximum transfer length is made relatively small in response to the one or more external programs newly operating, and the one or more external programs are newly stopped. , Make the maximum transfer length relatively large,
7. The computer program according to claim 6, which causes the first node to execute. In the first node, for each of the second nodes, in (B),
(B3) transmits a lock request, which is a request for locking, from the cache memory of the second node, the same number of cache strips as the number of strips according to the number of transfer target data sets, to the second node,
After completing (b1) and (b3), execute (b2),
The computer program according to claim 1, which executes the program. The node management information includes subnode management information for each node,
For each node, the size of the subnode management information depends on the maximum transfer length, which is the total amount of data sets that the first node can transfer,
The computer program according to claim 1. Has multiple nodes,
Each of the plurality of nodes has a node area which is a logical storage area composed of one or more strips,
The plurality of nodes form a redundant configuration area from two or more node areas,
In the distributed storage system, wherein the redundant configuration area includes stripes composed of two or more strips respectively included in the two or more node areas , and transfers write target data between the nodes,
A first node, which is any one of the plurality of nodes,
A cache memory storing a data set written in the node area of each of the plurality of nodes;
The first node of the transfer target data set, which is the data set stored in the cache memory, to which the strip in the node area of each of the plurality of second nodes other than the first node among the plurality of nodes becomes a write destination holds, and node management information for managing the presence in,
The first node is
(A) When it is specified based on the node management information that there are two or more transfer target data sets to which the two or more strips in the node area of the one second node are write destinations, the two or more transfer operations are performed. Specify the cache strip of the target data set based on the node management information ,
(B) One transfer command in which two or more transfer target data sets respectively existing in two or more node positions corresponding to the specified cash strip are transferred is transmitted to the second node .
As a result, the distributed storage system that transmits two or more write target data to the second node by one transfer command . Have a plurality of nodes,
Each of the plurality of nodes has a node area which is a logical storage area composed of one or more strips,
The plurality of nodes form a redundant configuration area from two or more node areas,
In the method of controlling a distributed storage system, wherein the redundant configuration area includes stripes composed of two or more strips respectively possessed by the two or more node areas , and transfers write target data between the nodes,
A first node, which is any one of the plurality of nodes ,
A cache memory storing a data set written in the node area of each of the plurality of nodes;
The first node of the transfer target data set, which is the data set stored in the cache memory, to which the strip in the node area of each of the plurality of second nodes other than the first node among the plurality of nodes becomes a write destination holds, and node management information for managing the presence in,
In the first node ,
(A) When it is specified based on the node management information that there are two or more transfer target data sets to which the two or more strips in the node area of the one second node are write destinations, the two or more transfer operations are performed. The cache strip of the target data set is specified based on the node management information ,
(B) A single transfer command for transferring two or more transfer target data sets respectively existing at two or more node positions corresponding to the specified cash strip is transmitted to the second node .
Thus, the distributed storage control method of transmitting two or more write target data to the second node by one transfer command . The computer program according to claim 1, wherein
The cache memory of the first node is
A plurality of cash strip rows including a plurality of the cash strips,
Each of the cash strip columns stores a data set to be transferred to the different nodes,
A computer program for transferring a plurality of the transfer target data sets included in one cache strip sequence to the node (single) corresponding to the cache strip sequence by the one transfer command.