WO2016088372A1

WO2016088372A1 - Access device, migration device, distributed storage system, access method, and computer-readable recording medium

Info

Publication number: WO2016088372A1
Application number: PCT/JP2015/005987
Authority: WO
Inventors: 真樹菅; 鈴木　順; 佑樹林
Original assignee: 日本電気株式会社
Priority date: 2014-12-05
Filing date: 2015-12-02
Publication date: 2016-06-09
Also published as: JPWO2016088372A1

Abstract

The present invention prevents access performance in a distributed storage system from declining during a data re-balance process when the number of storage units has increased. An access device provided with: a reception unit for receiving, from a migration device for re-balancing data in accordance with a change in the number of storage units included in a storage device by executing a data addition process on the storage units included in the storage device and then executing a data deletion process, a notification that the addition process has been completed; and an access unit which, when the storage unit that is to hold the data to be accessed changes, accesses the storage unit included in the storage device, on the basis of information indicating the storage unit that is to hold the data before and after the change, until the notification that the addition process has been completed is received, and which, when the notification that the addition process has been completed is received, accesses the storage unit that is to hold the data after change.

Description

Access device, migration device, distributed storage system, access method, and computer-readable recording medium

The present invention relates to an access device, a migration device, a distributed storage system, an access method, and a computer-readable recording medium, and in particular, a distributed storage system in which the number of storage units can be increased or decreased, an access device that accesses such a distributed storage system, and a storage unit The present invention relates to a migration device that performs data migration as the number of files increases and decreases, an access method using an access device, and a computer-readable recording medium.

Various technologies have been developed for controlling data access to storage devices and storage systems. For example, a data store system (for example, a database system, a file system, a cache system, etc.) configured by a single computer or a plurality of computers is known.

In recent years, a distributed storage system is mainly used as such a system. The distributed storage system includes a plurality of general-purpose computers connected via a network. The distributed storage system stores data and provides data using a storage unit (or storage device) mounted on these computers. Here, the storage unit is, for example, a hard disk (HDD: Hard Disk Drive), main memory (for example, DRAM: Dynamic Random Access Memory), or the like.

In the distributed storage system as described above, data is allocated to which computer, and which computer processes the data is determined by software or special hardware. As a method for determining a computer corresponding to data, for example, a method using a random number or a hash value is adopted.

Cluster-based distributed storage and distributed database technologies have been developed on the premise of server architecture. According to the server architecture, in order to access resources of a server other than itself, it is necessary to access via the corresponding server. On the other hand, in the resource separation architecture, each resource (memory, storage, etc.) is connected via an interconnect network. In other words, according to the resource separation type architecture, it is possible to physically share each resource from each CPU (Central Processing Unit), and it is not necessary to access via a server. Such architectural changes bring innovation to distributed storage and distributed database technologies.

In a distributed storage system, it is necessary to allow a dynamic change in the number of storage units due to a failure or performance expansion of a computer or storage device that is a component of the system. For example, when a method of determining a responsible storage unit based on a hash value is used, when the number of storage units is changed, the data distributed arrangement state is changed, so that a data rebalancing process is required. In general, the determination of the responsible storage unit during the data rebalancing process is more complicated than normal.

As a related technique, Patent Document 1 discloses a technique for constructing a distributed storage by consistent hashing and minimizing data rebalancing processing.

Patent Document 2 discloses a technique for quickly recovering the number of replications when the number of information storage nodes storing data is reduced.

JP 2014-142945 A JP 2012-22214 A

The entire disclosure of the above Patent Documents 1 and 2 is incorporated herein by reference. The following analysis was made by the present inventors.

According to the distributed storage system described in Patent Document 1, there is a problem that data cannot be accessed or performance is significantly deteriorated during data migration (rebalance) processing. In the distributed storage system described in Patent Document 1, since data migration (rebalancing) processing takes time, no responsible data exists on a computer or a storage device determined by a data arrangement algorithm. This is because data cannot be read, and writing processing cannot be executed until migration is completed.

In addition, when data is stored in a plurality of storage units in order to improve reliability, it is necessary to read / write data from / to these storage units. By changing the number of storage units, the data placement destination is changed. At this time, it is necessary to change the storage unit to be accessed according to whether or not the data rebalancing is completed.

In a distributed storage with a normal distributed cluster, it is possible to transfer access to the storage unit of the change destination. However, in the resource separation type architecture, it is difficult to provide a function related to the storage unit. Even in a normal distributed cluster, the access performance is degraded due to the communication latency required to transfer access.

Further, according to the technique described in Patent Document 2, since the flag (life / death information) indicating the status of data migration is centrally managed in the node management table on the management node, the client is in the process of data migration. When accessing the information storage node, there is a possibility that the process in the management node becomes a bottleneck.

Therefore, in a distributed storage system (for example, operating on a general distributed cluster or resource separation type architecture), it is a problem to prevent a decrease in access performance during data rebalancing processing when the number of storage units increases or decreases It becomes. An object of the present invention is to provide an access device, a migration device, a distributed storage system, an access method, and a program that contribute to solving such a problem.

According to the first aspect of the present invention, the data deletion process is performed after the data addition process is performed on the storage unit included in the storage apparatus in accordance with the change in the number of storage units included in the storage apparatus. An access device is provided that includes receiving means for receiving a notification that the additional processing has been completed from a migration device that performs data rebalancing by executing. When the storage means that should hold the data to be accessed fluctuates, the access device, based on information indicating the storage means that should hold the data before and after the change, until the notification of completion is received Access means for accessing the storage means that should hold the data after the change when the storage means included in the storage is accessed and the notification of the completion is received.

According to the second aspect of the present invention, the data deletion process is performed after the data addition process is performed on the storage unit included in the storage apparatus in accordance with the change in the number of storage units included in the storage apparatus. A migration apparatus is provided that includes data movement means for performing data rebalance by executing. The migration apparatus includes a notification unit that notifies an access device that accesses a storage unit included in the storage apparatus that the addition process has been completed.

According to the third aspect of the present invention, a distributed storage system including a migration device and an access device is provided. The migration device executes a data deletion process after executing a data addition process on the storage unit included in the storage apparatus in accordance with a change in the number of storage units included in the storage apparatus. Data rebalancing is performed, and the access device is notified that the additional processing has been completed. Further, when the storage unit that should hold the data to be accessed changes, the access device is based on the information indicating the storage unit that should hold the data before and after the change until the notification of completion is received. When the storage means included in the storage device is accessed and the notification of the completion is received, the storage means that should hold the data after the change is accessed.

According to the fourth aspect of the present invention, an access method by an access device is provided. The access method executes a data deletion process after executing a data addition process on the storage means included in the storage apparatus according to a change in the number of storage means included in the storage apparatus. The access device receives a notification to the effect that the additional processing has been completed from a migration device that performs data rebalancing. In the access method, when the storage means that should hold the data to be accessed changes, the access device shows the storage means that the access device should hold the data before and after the change until the notification of the completion is received. Based on the information, the storage means included in the storage device is accessed. Further, when the access method is notified of the completion, the access device accesses the storage means that should hold the data after the change.

According to the fifth aspect of the present invention, the data deletion process is performed after the data addition process is performed on the storage unit included in the storage apparatus in accordance with the change in the number of storage units included in the storage apparatus. A program is provided that causes a computer to execute a process of receiving a notification that the additional process has been completed from a migration apparatus that performs data rebalancing by executing. Further, when the storage means that should hold the data to be accessed fluctuates, the program stores the storage based on information indicating the storage means that should hold the data before and after the change until the notification of completion is received. The computer is caused to execute processing for accessing storage means included in the apparatus. Further, when the program receives the notification of completion, the program causes the computer to execute processing for accessing the storage means that should hold the data after the change.

Note that the program can also be provided as a program product recorded in a non-transitory computer-readable storage medium.

According to the access device, the migration device, the distributed storage system, the access method, and the computer-readable recording medium according to the present invention, the access performance during the data rebalancing process when the number of storage units increases or decreases in the distributed storage system. Decline can be prevented.

It is a block diagram which illustrates the composition of the access device concerning one embodiment of the present invention. It is a block diagram which illustrates the composition of the migration device concerning one embodiment of the present invention. It is a figure which illustrates the structure of the distributed storage system which concerns on the 1st Embodiment of this invention. 1 is a block diagram illustrating a detailed configuration of a distributed storage system according to a first embodiment of the present invention. FIG. 6 is a sequence diagram illustrating an operation during data migration processing in the distributed storage system according to the first embodiment of the invention. It is a figure for demonstrating the data access destination when a memory | storage part decreases in the distributed storage system which concerns on the 1st Embodiment of this invention. In the distributed storage system according to the first embodiment of the present invention, it is a diagram for explaining the data access destination when the storage unit increases.

First, an outline of one embodiment will be described. Note that the reference numerals of the drawings attached to this summary are merely examples for facilitating understanding, and are not intended to limit the present invention to the illustrated embodiment. Moreover, the arrow shown in a figure shows an example of the flow of data. However, the flow of data and the like is not limited to this.

FIG. 1 is a block diagram illustrating the configuration of an access device 2 according to an embodiment. Referring to FIG. 1, the access device 2 includes a receiving unit 25 and an access unit 26. The receiving unit 25 performs data addition processing on the storage unit included in the storage apparatus in accordance with the change in the number of storage units included in the storage apparatus. Thereafter, the reception unit 25 receives a notification that the addition processing is completed from the migration apparatus that performs data rebalancing by executing data deletion processing. When the storage unit that should hold the data to be accessed changes, the access unit 26 stores the storage device based on information indicating the storage unit that should hold the data before and after the change until receiving a notification that the addition process has been completed. Is accessed. Further, when the access unit 26 receives a notification that the addition process has been completed, the access unit 26 accesses the storage unit that should hold the data after the change.

FIG. 2 is a block diagram illustrating the configuration of the migration apparatus 1 according to an embodiment. Referring to FIG. 2, the migration apparatus 1 includes a data moving unit 11 and a notification unit 14. The data moving unit 11 executes data addition processing on the storage unit included in the storage device in accordance with fluctuations in the number of storage units included in the storage device, and then executes data deletion processing. Rebalance data. The notification unit 14 notifies the access device that accesses the storage unit included in the storage device that the addition process has been completed.

According to the access device 2 and the migration device 1, it is possible to prevent a decrease in access performance during data rebalancing processing when the number of storage units increases or decreases in the distributed storage system.

In the following, further details will be described with reference to the drawings. FIG. 4 is a block diagram illustrating the configuration of a distributed storage system according to an embodiment. Referring to FIG. 4, as the number of external storage units 200 to 20N increases or decreases, the access device 2 switches the data read / write algorithm, while the migration device 1 executes only the data addition processing necessary for data rearrangement.

Also, after executing all the data addition processing, the migration device 1 issues an algorithm switching command to the access device 2 group, and performs only data relocation deletion processing after the switching is completed. Furthermore, the access device group 2 performs data access using the old and new data arrangement information when the external storage units 200 to 20N increase or decrease, and after receiving the migration device switching command, the new data arrangement information is used in the normal data access method. Operate.

According to such a distributed storage system, the access performance during the data rebalancing process when the external storage units 200 to 20N increase or decrease does not deteriorate.

The reason is that it is not necessary to wait for the data read (READ) process at a timing when it is unknown whether the migration process is completed. Further, according to the operations of the migration apparatus 1 and the access apparatus 2 described above, a data inconsistency state does not occur during the migration process. That is, data inconsistency (difference in version) does not occur between data replicas in data update processing during migration, and further, data inconsistency does not occur between data replicas even after migration processing is completed. Therefore, inconsistency repair processing or the like is not required at the time of reading (READ).

Furthermore, according to such a distributed storage system, for data migration, it is possible to prevent the occurrence of communication latency such as accessing an external storage unit in which necessary data does not exist and re-communication, thereby improving access performance.

The reason is that the migration apparatus 1 notifies the transition information of migration to the access apparatus 2 so that it is not necessary to access data based on the old data arrangement information.

<Embodiment 1>
Next, the distributed storage system according to the first embodiment of the present invention will be described in detail with reference to the drawings.

[Constitution]
FIG. 3 is a diagram illustrating a schematic configuration of the distributed storage system according to the present embodiment. Referring to FIG. 3, the distributed storage system of this embodiment includes CPUs (Central Processing Units) 100 to 10M, external storage units 200 to 20N, and an interconnect network 300 that couples them.

1 or 2 or more CPUs 100 to 10M and external storage units 200 to 20N are provided, respectively. In the resource separation architecture, a computer server is constructed by coupling the CPUs 100 to 10M and the external storage units 200 to 20N (and other resources) via the interconnect network 300.

The CPUs 100 to 10M are devices including an arithmetic processing device (for example, a CPU) constituting a server, an interface (IF) for connecting to the interconnect network 300, a high-speed storage circuit (register), a CPU cache, and the like. Of course, the CPUs 100 to 10M may have other storage devices such as a memory.

The external storage units 200 to 20N include an interface for coupling with the CPUs 100 to 10M (that is, an IF to the interconnect network 300), a storage device (flash memory, DRAM (Dynamic Random Access Memory), MRAM (Magnetoresistive Random Access Memory). , HDD (Hard Disk Disk Drive), etc.), a function of performing access processing to the storage device, power supply means, and the like.

The interconnect network 300 connects the CPUs 100 to 10M and the external storage units 200 to 20N, and exchanges data, control messages, and other messages. The interconnect network 300 can be realized by, for example, an optical cable and a switch. The interconnect network 300 can also be realized by a PCI-e (Peripheral-Component-Interconnect-Express) cable or the like.

Further, a computer based on a conventional architecture may be handled as a CPU in this embodiment. At this time, the interconnect network 300 is realized by an Ethernet, a PCI-e network, or the like. According to ExpEther, the computer interconnect network PCI-e can be expanded. That is, an architecture similar to the resource separation architecture can be realized by holding the IF having the ExpEther function in the CPU 10X. At this time, the external storage unit 20X includes a card having an ExpEther function, an arbitrary PCI-e device, and the like. The PCI-e device only needs to have some storage device. For example, PCI-e Flash, RAID (Redundant Arrays of Inexpensive Disks) cards (via multiple HDDs and SSDs (Solid State Drive)), GPGPU (General Purpose computing computing on Graphics) Processing Units An arithmetic board based on a MIC (Many Integrated Core) architecture such as Intel Xeon Phi (including storage means) can be used.

Also, if the resources to be separated are limited to the conventional storage system, the interconnect network 300 is realized by Fiber Channel or FCoE (Fibre Channel over Ethernet (registered trademark)). On the other hand, the CPUs 100 to 10M are realized as a storage apparatus / system provided with a host bus adapter (or Ethernet card) in a conventional computer, and the external storage units 200 to 20N are provided with IFs for these networks. In such an architecture, a network between the CPUs 100 to 10M is often prepared separately. For example, a form in which servers are connected by Ethernet (TCP / IP (TransmissionTransControl Protocol / Internet Protocol)) and servers and storages are connected by Fiber Channel is used.

As described above, various modifications can be made to the hardware architecture for realizing the present embodiment.

FIG. 4 is a block diagram illustrating a more detailed configuration of the CPUs 100 to 10M. In the following, a description will be mainly given of a method of performing data access during data migration processing after realizing a distributed storage database system by the computer shown in FIG.

FIG. 4 illustrates a configuration in which a plurality of computers and external storage units exist. The migration device 1 and the access device 2 in FIG. 4 are assumed to operate as software that operates on the CPUs 100 to 10M in FIG. 3 or arbitrary hardware connected to the interconnect network 300, respectively.

Further, the participating host storage unit 3 and the participating device storage unit 4 in FIG. 4 operate as software for realizing arbitrary storage means and storage means, and are realized by any one or more of the CPUs 100 to 10M in FIG. The participating host storage unit 3 and the participating device storage unit 4 in FIG. 4 may also be realized by a computer and storage means connected to the interconnect network 300 in FIG. 3 or other arbitrary network.

Referring to FIG. 4, the distributed storage system according to this embodiment includes a migration device 1, one or more access devices 2, a participating host storage unit 3, a participating device storage unit 4, an interconnect network 300, and an external storage unit. 200 to 20N.

The access device 2 is software that implements a data store (database, KVS (Key Value Store), file system, etc.) realized in the present embodiment and provides a data access function. The CPU 10X that implements the access device 2 has hardware and software that perform communication processing according to an arbitrary communication protocol via the interconnect network 300.

The migration apparatus 1 is software that executes data movement for correcting the data arrangement state when a change occurs in the number, capacity, etc. of the external storage units 200 to 20N constituting the distributed storage system. Similarly, the CPU 10X that implements the migration apparatus 1 also has hardware and software that perform communication processing according to an arbitrary communication protocol via the interconnect network 300.

In this embodiment, it is necessary to switch the data location determination method according to the transition of migration. Therefore, the migration apparatus 1 needs to grasp the access apparatus 2 that accesses the distributed storage system as a notification destination that the switching has been performed. The participating host storage unit 3 holds information related to the access device 2 that accesses the distributed storage system. The migration device 1 refers to the participating host storage unit 3 and grasps the access device 2 that accesses the distributed storage system.

Note that information related to the access device 2 that accesses the distributed storage system is not necessarily managed centrally as a storage means. However, in FIG. 4, for the sake of simplicity, it is assumed that the participating host storage unit 3 is provided. In addition to the centralized management method, a method for dynamically searching for participating hosts in the network when necessary is conceivable.

As a specific example of the information held in the participating host storage unit 3, a list of IP addresses and process port numbers of servers of the CPUs 100 to 10M that operate the access device 2 can be considered.
Further, as a specific embodiment, it is conceivable that the participating host storage unit 3 is operated as database software, and the participation notification unit 21 and the transition notification unit 13 each access the participating host storage unit 3 by an SQL command or the like. In addition, a shared file system or the like may be used as the participating host storage unit 3.

The participating device storage unit 4 is a storage unit that holds information necessary for realizing a distributed storage system that determines a data arrangement location based on a hash or a random number. In such a distributed storage system, it is necessary to hold list information of storage means (a computer when a computer is a storage means) constituting the distributed storage system.

As a method of realizing the participating device storage unit 4, for example, a method of holding data by any one or more computers, a method of holding data by each CPU 10X, and updating them synchronously, distributed among CPUs 100 to 10M A storage system can be realized and a method of holding the storage system can be used.

Hereinafter, details of the operation of each unit in FIG. 4 will be described.

Referring to FIG. 4, the migration apparatus 1 includes a data movement unit 11, a device selection unit 12, and a transition notification unit 13.

The data moving unit 11 uses the data arrangement destination information determined by the device selection unit 12 and executes necessary data movement between the external storage units 200 to 20N in order to reflect the change of the data arrangement state. Here, the data movement processing includes data read from the migration source external storage unit 20Y, data write to the migration destination external storage unit 20Z, and deletion of the original data from the migration source external storage unit 20Y.

The device selection unit 12 calculates information for determining which external storage unit 20Y stores a specific data range from the storage unit list information constituting the distributed storage system held by the participating device storage unit 4. And hold. In the present embodiment, when the state of the participating device storage unit 4 changes, the device selection unit 12 holds both state information before and after the change.

For example, in the case of a key value store (KVS: Key Value Store) system, the external storage unit 20Y as a storage destination is specified for a key using these pieces of information held by the device selection unit 12, or further external storage is performed. The physical address to be accessed in the unit 20Y can be specified. The device selection unit 12 operating in the migration apparatus 1 calculates the change state because the storage destination of each data is changed according to the change in the number of external storage units 20X.

The transition notification unit 13 notifies the migration transition from the participating host storage unit 3 to the access device 2. The transition notification unit 13 notifies the access device 2 after the data write process in the migration is completed. The transition notification unit 13 receives the migration transition information from the data migration unit 11 and refers to the information in the participating host storage unit 3 to notify the access device 2.

Referring to FIG. 4, the access device 2 includes a participation notification unit 21, a device selection unit 24, an algorithm switching unit 23, and a read / write execution unit 22. The access device 2 receives a storage access request from a higher-level application program.

The access request varies depending on the function of the storage software. For example, in the case of a database, the access request is a data operation instruction as specified by SQL. On the other hand, in the case of KVS (Key Value Store), the access request is a processing request for obtaining, registering, or updating a value corresponding to the key.

The participation notification unit 21 notifies the participation / leaving of the system in order for the access device 2 to realize the function of accessing the distributed storage. Specifically, the participation notification unit 21 registers or deletes its own management IP address and port number in the participation host storage unit 3. However, the information registered by the participation notification unit 21 may be information that allows the transition notification unit 13 to identify the access device 2. When the influence on the system is considered to be small, the deletion (leaving) process may be omitted. However, it is preferable to perform processing such as deleting unused information later.

The device selection unit 24 of the access device 2 has the same function as the device selection unit 12 of the migration device 1. In particular, the device selection unit 24 in the access device 2 selects the external storage unit 20Y as an access destination in response to an access request from a higher-order application, and arranges the necessary data on the external storage unit 20Y (that is, the address) ) Is calculated.

The read / write execution unit 22 issues a data access command to the access destination selected by the device selection unit 24. In the distributed storage system, the read / write execution unit 22 issues instructions to the plurality of external storage units 20Y in order to ensure redundancy and ensure reliability. Further, the read / write execution unit 22 changes the algorithm for executing the read / write in accordance with the instruction of the algorithm switching unit 23. Details of the algorithm will be described later.

The algorithm switching unit 23 issues an algorithm change request to the read / write execution unit 22 based on the information notified from the transition notification unit 13.

[Operation]
Next, the operation of the distributed storage system according to this embodiment will be described with reference to the drawings.

{Migration procedure}
The data moving unit 11 performs rebalancing processing of data generated by changing the number of external storage units (devices) 200 to 20N according to the following procedure. First, along with the device change, the transition notification unit 13 notifies the access device 2 of the start of migration. The notification of migration start may be implicitly performed by reflecting the increase / decrease in devices.

Next, the device selection unit 12 calculates a data arrangement change state. Further, the data moving unit 11 executes all necessary data addition processing. Thereafter, the transition notification unit 13 notifies the algorithm switching unit 23 of the algorithm change, and changes the algorithm of the read / write execution unit 22. After that, the data moving unit 11 executes all data deletion processes that are no longer necessary due to the change in the data arrangement.

FIG. 5 is a sequence diagram illustrating the operation of the migration process in the distributed storage system according to this embodiment.

Referring to FIG. 5, first, the external storage unit 20Y is added to or deleted from the distributed storage system due to a failure of the external storage units 200 to 20N or system performance expansion. The migration device 1 (for example, the CPU 10X) notifies the plurality of access devices 2 of addition or deletion of the external storage unit 20Y (step S1).

After the configuration of the external storage units 200 to 20N participating in the distributed storage system is changed, the access device 2 provides information on both the data allocation destination determination method before the configuration change and the data allocation destination determination method after the configuration change. And read (READ) / write (WRITE) processing is performed (step S2). For example, in the case of a distributed storage system based on the consistent hashing method, the access device 2 holds two types of consistent hash rings, an old state and a new state. A specific algorithm for the read (READ) / write (WRITE) access will be described later.

After each access device 2 switches to a method that uses old and new data location determination information, the migration device 1 starts data addition processing accompanying migration (step S3). By changing the data arrangement destination, an area where new data is copied and an area where data is deleted appear. Here, the migration apparatus 1 executes only a new copy process (addition process).

The migration device 1 issues a data placement destination determination information switching request to the access device 2 group after completion of the data addition processing (step S4).

When the access device 2 receives the switching request and completes the switching process, the access device 2 sends a switching completion notification to the migration device 1 (step S5).

When the migration apparatus 1 receives the notification of completion of the switching process, the migration apparatus 1 executes a data deletion process associated with the migration (step S6).

The access device group 2 executes normal read / write access processing using only the new data location determination information after completion of switching (step S7).

{Data access algorithm}
Next, an access algorithm of the read / write execution unit 22 will be described. Here, the determination method of the external storage unit 20X will be described by taking as an example the case where the key-value store system is realized by the consistent hash ring method. However, the present invention is not limited to the key-value store system or the consistent hash method.

Hereinafter, a period before the data moving unit 11 completes the data addition processing after the increase / decrease of the external storage unit 20Y is referred to as “before switching notification reception”. On the other hand, a period after the data moving unit 11 completes the data addition process is referred to as “after switching notification is received”. In the present embodiment, different data access algorithms are used between these two periods.

"Lead processing"
First, the read processing algorithm will be described. FIG. 6 is an image diagram of a consistent hash ring in the consistent hash method. In the consistent hash method, the arrangement destination external storage unit 20Y is determined using a hash value of a key. If the value of the hash value is mapped onto the circumference, the hash value of the key is placed on one of the circumferences. Similarly, the external storage units 200 to 20N can be mapped onto the circumference.

In FIG. 6, as an example, the external storage units 200 to 20N are mapped on the circumference as nodes D0 to D7. In an actual system, a single external storage unit can be mapped to a plurality of nodes using a mechanism called a virtual node. However, here, for simplicity of explanation, the external storage units 200 to 20N and the nodes D0 to D7 correspond to 1: 1.

At this time, the external storage unit 20Y corresponding to the key can be selected as follows. As shown in FIG. 6, when the hash value of the key is between the node D0 and the node D7, the read / write execution unit 22 selects the external storage unit 200 corresponding to the node D0 as the storage unit. Further, assuming that a total of three copies are held for reliability, the read / write execution unit 22 selects the external storage units 201 and 202 corresponding to the next nodes D1 and D2 as the copy destination. In the description of the present embodiment, a case will be described in which such a method is used as an algorithm for selecting an external storage unit in the read / write execution unit 22. However, the duplication destination selection algorithm in the present invention is not limited to this.

In this way, the read / write execution unit 22 accesses the data by issuing a read or write command to the external storage unit 20Y at the placement destination. The read / write execution unit 22 may access the plurality of external storage units 20Y, read a predetermined number of data and find no inconsistency in the data, and return a response to the client. For example, in the majority voting method, the read / write execution unit 22 considers that the processing is successful at the stage where two data are read for three replicas.

However, depending on the reliability required by the system, the read / write execution unit 22 may return a response after reading out three pieces of data, or may return a response when only one piece of data is read out. Good. The read / write execution unit 22 also performs the same process as the read process in order to ensure reliability regarding the write process. Here, the number of accesses required for the read / write execution unit 22 to regard an access (read or write) as a success is referred to as a “minimum access number”.

Here, as shown in FIG. 6, consider a case where the external storage units 201 and 202 corresponding to the nodes D1 and D2 leave due to a failure. At this time, the data allocation destination storage unit after the failure becomes the external storage units 200, 203, and 204 corresponding to the nodes D0, D3, and D4, and the two external storage units are changed. Therefore, a copy for data rearrangement is required. Specifically, the migration apparatus 1 needs to copy the data in the external storage unit 200 corresponding to the node D0 to the external storage units 203 and 204 corresponding to the nodes D3 and D4.

In the read processing algorithm of the present embodiment, before receiving the switching notification from the migration apparatus 1, the read / write execution unit 22 executes read processing based on the data arrangement information before the storage unit reduction. At this time, the read / write execution unit 22 does not access the reduced external storage unit 20Y. That is, the read / write execution unit 22 excludes the external storage units 201 and 202 corresponding to the nodes D1 and D2 from the access destination.

Here, it is necessary to read data from the storage unit with the minimum access number or more in the read processing. The read / write execution unit 22 performs processing by reducing the number of storage units that are reduced from the minimum number of accesses (however, only the storage unit that stores data).

In the example of FIG. 6, since the nodes D1 and D2 are deleted in the arrangement state of the nodes D0, D1, and D2, the read / write execution unit 22 reads the data in the external storage unit 200 corresponding to the node D0. Returns a response.

Even when the two external storage units 20Y are disconnected from the system, the read / write execution unit 22 is minimum when only one of the three external storage units 20Y holding the copy is deleted. Only 1 is subtracted from the number of accesses. The minimum value of the number of external storage units actually accessed by the read / write execution unit 22 is 1. Note that in a system that holds three replicas, data loss cannot be avoided if three external storage units fail simultaneously.

After receiving the switching notification, the read / write execution unit 22 issues a read (READ) command based on the state after the external storage unit is reduced. That is, the read / write execution unit 22 accesses the external storage units 200, 203, and 204 corresponding to the nodes D0, D3, and D4. Here, the minimum access number used by the read / write execution unit 22 remains the system default value (3 in the case of FIG. 6).

On the other hand, in the case where the external storage unit 20Y is added, the read / write execution unit 22 determines the access destination based on the old data arrangement state (before addition) before the switching notification is received. In addition, after receiving the switching notification, the read / write execution unit 22 determines the access destination based on the new data arrangement state (after addition).

"Light processing"
Next, the write algorithm will be described. FIG. 7 is an image diagram when one external storage unit 20Y is added (here, the external storage unit 201 corresponding to the node D1 is added). Again, the number of replicas is three. By adding the external storage unit, the state of the storage destination of a certain key is changed from the nodes D0, D2, and D3 to the nodes D0, D1, and D2.

Before the switching notification is received, the read / write execution unit 22 writes data in both the data arrangement state before the change and the data arrangement state after the change. That is, the read / write execution unit 22 writes data to the external storage units 200 to 203 corresponding to the four nodes D0 to D3. Therefore, the read / write execution unit 22 operates so as to temporarily increase the number of replicas by the number of additional external storage units. On the other hand, after receiving the switching notification, the read / write execution unit 22 performs writing only in the added state. That is, the read / write execution unit 22 performs a write process on the external storage units 200 to 202 corresponding to the nodes D0 to D2.

Here, the reason why the read / write execution unit 22 writes (writes) to the external storage unit (that is, the external storage unit 203 corresponding to the node D3) in which writing is performed only in the data arrangement state before the change This is to prevent inconsistency between the read algorithm and the data migration algorithm.

In the majority method (Quorum method), when the number of reads (READ) is defined to be smaller than the number of copies, the read / write execution unit 22 omits writing to the external storage unit 203 corresponding to the node D3. May be. However, if the writing to the external storage unit 203 corresponding to the node D3 is omitted, it is necessary to perform version management so that old data is not written in the migration process.

The reason why the read / write execution unit 22 also writes to the external storage unit 201 corresponding to the new storage destination node D1 before receiving the switching notification is for consistency with the migration apparatus 1. . As a result, processing can be performed regardless of whether or not the migration apparatus 1 has completed processing for the corresponding key. Here, since it takes a load on the computer to check the history of the migration apparatus 1, it is considered that it is faster to execute the write process.

Note that, as shown in FIG. 6, even in the case where the number of external storage units is reduced, the read / write execution unit 22 performs the same processing as when the number of external storage units increases. That is, before the switching notification is received, the read / write execution unit 22 writes to both external storage units 20Y before and after the reduction. On the other hand, after receiving the switching notification, the read / write execution unit 22 writes only to the external storage unit 20Y in the reduced state.

As illustrated in FIG. 6, when the external storage units 201 and 202 corresponding to the nodes D1 and D2 leave due to a failure, the external storage unit 20Y that stores data before the reduction is an external storage unit corresponding to the nodes D0 to D2. 200 to 202. On the other hand, external storage units that store data after reduction are external storage units 200, 203, and 204 corresponding to nodes D0, D3, and D4.

In this case, before receiving the switching notification, the read / write execution unit 22 writes to both the external storage units 20Y before and after the reduction. That is, the read / write execution unit 22 performs writing to the external storage units 200 to 204 corresponding to the nodes D0 to D4 before and after the reduction, based on the information regarding both external storage units 20Y before and after the reduction. However, since the external storage units 201 and 202 corresponding to the nodes D1 and D2 are not reduced, they are excluded from the write destination. As a result, the read / write execution unit 22 performs writing to the external storage units 200, 203, and 204 corresponding to the nodes D0, D3, and D4.

On the other hand, after receiving the switching notification, the read / write execution unit 22 writes only to the state after the reduction, that is, only the external storage units 200, 203, and 204 corresponding to the nodes D0, D3, and D4.

After all, in the example shown in FIG. 6, before and after the reduction of the external storage units, the write destinations are the external storage units 200, 203, and 204 corresponding to the nodes D0, D3, and D4, and do not change.

Although not shown, the external storage units to be accessed before reduction are the external storage units 207, 201, 202 corresponding to the nodes D7, D1, D2, and the external storage units 201, 202 corresponding to the nodes D1, D2 are If the reduced external storage units to be accessed are the external storage units 200, 203, and 204 corresponding to the nodes D0, D3, and D4, the read / write execution unit 22 performs before and after the reduction notification before receiving the switching notification. Are written in the external storage units 207, 200, 203, and 204 corresponding to both the nodes D7, D0, D3, and D4. On the other hand, after receiving the switching notification, the read / write execution unit 22 writes only to the external storage units 200, 203, and 204 corresponding to the nodes D0, D3, and D4.

Examples of utilization of the present invention include a database system using a storage unit outside a computer, a key-value store system, and a shared distributed data store system sharing a common storage unit among a plurality of computers. The present invention can also be used when a storage system is operated on a server realized by a resource separation architecture. However, the application destination of the present invention is not limited to these.

In the present invention, the following modes are possible.
[Form 1]
As in the access device according to the first aspect.
[Form 2]
In the access device, when the storage unit that should hold the data to be read is reduced, the access unit includes the storage unit that should hold the data before the reduction until the notification of the completion is received. If reading is performed from a storage unit other than the reduction target and a notification of completion is received, the data may be read from the storage unit that should hold the data after reduction.
[Form 3]
In the access device, when a storage unit that should hold data to be read is added, the access unit reads data from the storage unit that should hold the data before addition until receiving a notification of completion. When the notification of completion is received, the data may be read from the storage unit that should hold the data after the addition.
[Form 4]
In the access device, when the storage unit to hold the data to be written is added, the access unit writes the data to the storage unit to hold the data before and after the addition until the notification of the completion is received. When the notification of completion is received, the data may be written to the storage unit that should hold the data after the addition.
[Form 5]
In the access device, when the storage unit to hold data to be written is reduced, the access unit is a storage unit to hold the data before and after the reduction until the completion notification is received. If writing is performed to a storage unit other than the reduction target and a notification of the completion is received, writing may be performed to the storage unit that should hold the data after reduction.
[Form 6]
In the access device, a storage unit that should hold data may be determined based on a consistent hash method.
[Form 7]
This is the same as the migration apparatus according to the second aspect.
[Form 8]
The distributed storage system according to the third aspect is as described above.
[Form 9]
In the distributed storage system, when the storage unit to hold the data to be read is reduced, the access device includes the storage unit to hold the data before the reduction until the notification of the completion is received. When reading is performed from a storage unit other than the reduction target and a notification of the completion is received, the data may be read from the storage unit that should hold the data after the reduction.
[Mode 10]
In the distributed storage system, when a storage unit to hold data to be read is added, the access device reads from the storage unit to hold the data before adding until receiving a notification of completion. When the notification of completion is received, the data may be read from the storage unit that should hold the data after the addition.
[Form 11]
In the distributed storage system, when a storage unit to hold data to be written is added, the access device writes the data to the storage unit to hold the data before and after the addition until receiving notification of completion. When the notification of completion is received, the data may be written in the storage unit that should hold the data after the addition.
[Form 12]
In the distributed storage system, when the storage unit to hold the data to be written is reduced, the access device is a storage unit to hold the data before and after the reduction until the notification of the completion is received. Then, the data may be written to a storage unit other than the reduction target, and when the notification of completion is received, the data may be written to the storage unit that should hold the data after the reduction.
[Form 13]
In the distributed storage system, a storage unit that should hold data may be determined based on a consistent hash method.
[Form 14]
The access method according to the fourth aspect is as described above.
[Form 15]
In the access method, when the storage unit that should hold the data to be read is reduced, the access device includes the storage unit that should hold the data before the reduction until the notification of the completion is received. If reading is performed from a storage unit other than the reduction target and a notification of completion is received, the data may be read from the storage unit that should hold the data after reduction.
[Form 16]
In the access method, when a storage unit to hold data to be read is added, the access device reads from the storage unit to hold the data before adding until receiving a notification of completion. When the notification of completion is received, the data may be read from the storage unit that should hold the data after the addition.
[Form 17]
In the access method, when a storage unit to hold data to be written is added, the access device writes data to the storage unit to hold the data before and after the addition until receiving a notification of completion. When the notification of completion is received, the data may be written to the storage unit that should hold the data after the addition.
[Form 18]
In the access method, when the storage unit to hold data to be written is reduced, the access device is a storage unit to hold the data before and after the reduction until receiving a notification of completion. If writing is performed to a storage unit other than the reduction target and a notification of the completion is received, writing may be performed to the storage unit that should hold the data after reduction.
[Form 19]
In the access method, the storage unit to hold data may be determined based on the consistent hash method.
[Mode 20]
The migration device executes data addition processing on the storage units included in the storage device in accordance with fluctuations in the number of storage units included in the storage device, and then executes data deletion processing to execute data deletion. The migration method includes a step of performing rebalancing, and a step of notifying the access device that accesses the storage unit included in the storage device that the addition process has been completed.
[Form 21]
A program for causing a computer to execute each step of the access method according to any one of Forms 14 to 19 or the migration method according to Form 20 is provided.

It should be noted that the entire disclosure of the above patent document is incorporated herein by reference. Within the scope of the entire disclosure (including claims) of the present invention, the embodiment can be changed and adjusted based on the basic technical concept. Further, various combinations or selections of various disclosed elements (including each element of each claim, each element of each embodiment, each element of each drawing, etc.) are possible within the framework of the entire disclosure of the present invention. is there. That is, the present invention of course includes various variations and modifications that could be made by those skilled in the art according to the entire disclosure including the claims and the technical idea. In particular, with respect to the numerical ranges described in this document, any numerical value or small range included in the range should be construed as being specifically described even if there is no specific description.

This application claims priority based on Japanese Patent Application No. 2014-246506 filed on Dec. 5, 2014, the entire disclosure of which is incorporated herein.

DESCRIPTION OF SYMBOLS 1 Migration apparatus 2 Access apparatus 3 Participating host storage part 4 Participating device storage part 11 Data movement part 12 Device selection part 13 Transition notification part 14 Notification part 21 Participation notification part 22 Read / write execution part 23 Algorithm switching part 24 Device selection part 25 Reception unit 26 Access unit 100 to 10M CPU
200 to 20N External storage unit 300 Interconnect network D0 to D7 Node

Claims

According to the change in the number of storage units included in the storage apparatus, after performing the data addition process on the storage unit included in the storage apparatus, the data rebalancing is performed to execute the data rebalancing. Receiving means for receiving a notification to the effect that the additional processing has been completed from the migration device to be performed;
Storage means included in the storage device based on information indicating the storage means to hold the data before and after the change until the notification of completion is received when the storage means to hold the data to be accessed changes. An access means for accessing the storage means to hold the data after the fluctuation when receiving the notification of the completion,
An access device.
When the storage means that should hold the data to be read is reduced, the access means stores the data other than the reduction target among the storage means that should hold the data before the reduction until receiving the notification of completion. When reading from the means and receiving the notification of completion, reading from the storage means that should hold the data after reduction,
The access device according to claim 1.
When the storage means that should hold the data to be read is added, the access means reads from the storage means that should hold the data before addition until receiving the notification of completion, and the access is completed. Upon receiving the notification to the effect, reading from the storage means that should hold the data after addition,
The access device according to claim 1 or 2.
When the storage unit that should hold the data to be written is added, the access unit writes the storage unit that should hold the data before and after the addition until the notification of the completion is received. When notified to the effect, writing to the storage means that should hold the data after addition,
The access device according to any one of claims 1 to 3.
When the storage means that should hold the data to be written is reduced, the access means is a storage means that should hold the data before and after the reduction until it receives a notification of completion, and stores the data other than the reduction target Write to the means, and upon receiving the notification of completion, write to the storage means to hold the data after reduction,
The access device according to any one of claims 1 to 4.
The storage means to hold the data is determined based on the consistent hash method.
The access device according to any one of claims 1 to 5.
According to the change in the number of storage units included in the storage apparatus, after performing the data addition process on the storage unit included in the storage apparatus, the data rebalancing is performed to execute the data rebalancing. Data movement means to perform;
Notification means for notifying the access device that accesses the storage means included in the storage device that the addition process has been completed,
A migration apparatus characterized by that.
It has a migration device and an access device,
The migration device executes a data deletion process after executing a data addition process on the storage unit included in the storage apparatus in accordance with a change in the number of storage units included in the storage apparatus. Rebalance the data, notify the access device that the additional processing is complete,
When the storage means that should hold the data to be accessed fluctuates, the access device, based on information indicating the storage means that should hold the data before and after the change, until the notification of completion is received When the storage means included in is accessed and the notification of the completion is received, the storage means that should hold the data after the change is accessed.
A distributed storage system characterized by that.
According to the change in the number of storage units included in the storage apparatus, after performing the data addition process on the storage unit included in the storage apparatus, the data rebalancing is performed to execute the data rebalancing. The access device receives a notification that the additional processing is completed from the migration device to perform,
When the storage unit that should hold the data to be accessed changes, the storage device is based on information indicating the storage unit that should hold the data before and after the change until the notification of completion is received. Access the storage means contained in
Upon receiving the notification of completion, the access device accesses the storage means that should hold the data after the change,
An access method characterized by that.
According to the change in the number of storage units included in the storage apparatus, after performing the data addition process on the storage unit included in the storage apparatus, the data rebalancing is performed to execute the data rebalancing. A process of receiving a notification that the addition process has been completed from the migration apparatus to be performed;
Storage means included in the storage device based on information indicating the storage means to hold the data before and after the change until the notification of completion is received when the storage means to hold the data to be accessed changes. The process of accessing
Receiving the notification of completion, causing the computer to execute processing for accessing the storage means that should hold the data after the change,
The computer-readable recording medium which stored the program characterized by the above-mentioned.