JP5969315B2 - Data migration processing system and data migration processing method - Google Patents

Description

  The present invention relates to a distributed processing system in which servers distributed in a network are clustered to store data, and when the server constituting the distributed processing system has failed from the cluster due to a failure or the like. The present invention relates to a data migration processing system and a data migration processing method for performing data migration processing (re-redundancy processing or rearrangement processing) when a server to be configured is added.

  In recent years, with the rise of cloud computing, it has been required to efficiently process and retain a large amount of data. Thus, distributed processing technology has been developed that realizes efficient processing by operating a plurality of servers in a coordinated manner.

  When performing distributed processing, it is necessary to distribute data to be processed to each server (hereinafter referred to as “cluster member” or “node”) that constitutes the cluster. At this time, in order to increase the processing capacity of the entire cluster, it is desirable that the number of data (data amount) handled by each node is averaged.

  As a typical data management (distribution) technique, a value obtained by multiplying the key of each data by a hash function (hereinafter referred to as “hash (key)”) by the number of nodes N, that is, “hash (key)”. There is a method for distributing data to nodes having “mod N” as a number. In this case, it is assumed that numbers “0” to “N−1” are assigned to each node in advance. When such a management (distribution) method is used, if a node is added, the value of N changes, and for many data, the node that is responsible for storing the data is changed. It is necessary to arrange.

  Therefore, a management (distribution) method using a consistent hashing method (see Non-Patent Document 1) is used as a method for suppressing the number of data to be changed by a node in charge with addition of cluster members to about 1 / N. There is. This consistent hash method is used in Amazon Dynamo (see Non-Patent Document 2) and the like.

  In this data management (distribution) method using the consistent hash method, IDs (IDentifiers) are assigned to both nodes and data. Then, when the ID space closed from the data ID is traced clockwise, the first node encountered is taken charge of the data. An example of how to give an ID to a node is a value (hash (IP address)) obtained by multiplying an IP address by a hash function.

  In a distributed processing system with a cluster configuration, when the processing performance of each node is equal, the amount of data handled by each node is made equal, that is, the distance between nodes (hereinafter referred to as “node” in the ID space of the consistent hash method). It is desirable to make them equal to each other. In order to solve this point, a method of virtually giving a plurality of IDs to each node is used (see Non-Patent Document 1). Since each node has a plurality of IDs, even if the assigned area for each virtual ID is different, the assigned areas of the nodes are averaged according to the law of large numbers.

  When a large number of data management is performed in a clustered distributed processing system, data redundancy is achieved by maintaining a copy of the data so that processing can be continued on other nodes even if a failure occurs on one node doing. Even in a distributed processing system using a data management method based on the consistent hash method, data redundancy is required, and there is a method of arranging replicated data as shown in FIG.

  As shown in FIG. 10, in the consistent hash method, IDs are assigned to both nodes (nodes “1” to “4”) and data (data A to D, indicated by black circles (●)), and the ID of the data is calculated. The ID space is traced clockwise and the first node encountered is determined to be responsible for the data. Then, the node that is further to the right of the node in charge (next in the clockwise direction) is assigned the duplicate data.

  For example, in FIG. 10, for data A, the node “1” that first meets in the clockwise direction on the ID space is in charge, and the duplicate data is a node that is adjacent to the node “1” on the right side in the ID space. "2" will be assigned. By determining the node in charge of the original data / replicated data in this way, even if the node leaves, there is an advantage that processing can be continued because the node that owns the replicated data becomes the node in charge of the new data. is there. When a plurality of pieces of duplicate data are taken, the duplicate data is assigned to a node further to the right of the node responsible for the duplicate data.

David karger et al., “Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web”, [online], 1997, ACM, [August 8, 2012 search], Internet <URL: http://www.akamai.com/dl/technical_publications/ConsistenHashingandRandomTreesDistributedCachingprotocolsforrelievingHotSpotsontheworldwideweb.pdf> Giuseppe DeCandia, et al., “Dynamo: Amazon's Highly Available Key-value Store,” SOSP'07, October 14-17, 2007, Stevenson, Washington, USA, [online], [searched August 8, 2012] Internet <URL: http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf>

  Consider a case in which a failure or the like occurs in a node and the node leaves the cluster in a distributed processing system that employs a data management method based on the consistent hash method and adopts a replication data arrangement method as shown in FIG. At this time, since the data held by the detached node is lost, the redundancy of some data is reduced. In the example shown in FIG. 11, since node “2” has left the cluster, the redundancy of data A and data B held by node “2” decreases.

  It is desirable that recovery of data redundancy is performed quickly because data A and data B may be completely lost from the cluster if the node detachment occurs with redundancy reduced. . On the other hand, since each node actually holds a large amount of data, if all the data is re-redundant immediately after the node leaves, the load becomes high and the normal processing that the cluster is performing is affected. There is a possibility of effect.

  Further, consider a case where a node is added to a cluster due to insufficient cluster performance or the like in a distributed processing system using a data management method based on the consistent hash method and adopting a method for arranging replicated data as shown in FIG. The added node takes over part of the data handled by the existing node according to the ID space information of the consistent hash method. At this time, until the data transfer is completed, the amount of data held in each node becomes non-uniform even though a new node is added. On the other hand, each node holds a lot of data, so if you relocate all the data immediately after adding the node, the load will be high and the normal processing that the cluster is performing may be affected. There is.

  The present invention has been made in view of such a background, and in the present invention, when a node constituting a cluster is detached or added, data can be migrated while suppressing the processing load of the node. It is an object of the present invention to provide a migration processing system and a data migration processing method.

In order to solve the above-described problem, the invention according to claim 1 is characterized in that any of a plurality of nodes constituting a cluster stores an owner node that stores data for providing a service to a client as original data, or A data migration processing system that is allocated and stored as one or more replication nodes that store duplicate data of the data, wherein each of the plurality of nodes constituting the data migration processing system is a unique identifier For each of the plurality of nodes to which is attached, a storage unit that stores node identifier management information in which the data, the owner node, and the duplicate node are associated with each other, and the owner node that stores the original data A node identifier of the replication node storing the replication data and the node identifier of the replication data, A node information adding unit to be added as node information to each of the data and the duplicated data, and detection of the node leaving or adding, and the node identifier management information in accordance with the node leaving or adding And a node identifier management unit that changes and stores a new association between the owner node and the replication node, and when the node is detected to be detached, based on the changed node identifier management information, When the original data stored by itself and duplicate data stored by itself are lost, the original data is lost, and the owner node or the duplicate node is changed. Therefore, it is extracted as determination target data indicating data to be determined whether or not data migration to be performed is necessary. If known, based on the changed node identifier management information, the original data stored therein is extracted as the determination target data, and the process of the node itself by the data migration At a predetermined timing based on a parameter set to suppress a load, an owner node and a replication node corresponding to the changed node identifier management information are specified for the extracted determination target data, and the specification is performed. If the node identifier of each of the owner node and the duplicated node does not match with the node information, the extracted determination target data is detected as data migration target data indicating the data that needs to be migrated, The detected data migration target data is migrated to the specified owner node and replication node A data migration processing unit, and the parameter is a data migration processing execution interval indicating the number of data migration processing threads indicating the maximum number of threads capable of executing the data migration in parallel, and a waiting time after the data migration is executed. A simulation maximum number of times indicating the number of times that the data migration target data detection process is continuously executed , including a simulation that is a process of identifying an owner node and a replication node corresponding to the changed node identifier management information, The data migration processing system is characterized by being one.

According to a fourth aspect of the present invention, any one of a plurality of nodes constituting a cluster stores an owner node that stores data for providing services to a client as original data, or duplicate data of the data. A data migration processing method of a data migration processing system that is allocated and stored as one or more replication nodes to store, wherein each of the plurality of nodes constituting the data migration processing system has a unique identifier For each of the plurality of attached nodes, the storage unit stores node identifier management information in which the data, the owner node, and the duplicate node are associated with each other, and stores the original data A node identifier of the replicating node and a node identifier of the replicating node storing the replicated data Adding to each of the original data and the duplicated data as node information; detecting the withdrawal or addition of the node; and detecting the node identifier management information in accordance with the withdrawal or addition of the node; Step of changing to a new association between the owner node and the duplicate node and storing it, and when the detachment of the node is detected, it is stored based on the changed node identifier management information. The original data that is stored, and the duplicate data that is stored in the original data, and when the original data is lost, the duplicate data is used to change the owner node or the duplicate node. Extracted as judgment target data indicating data to be judged whether or not migration is necessary, and addition of the node is detected In other words, based on the changed node identifier management information, the original data stored therein is extracted as the determination target data, and the processing load of the node itself due to the data migration is suppressed. At the predetermined timing based on the parameters set in the above, the owner node and the replication node corresponding to the changed node identifier management information are identified for the extracted determination target data, and the identified owner node and When the node identifier of each replication node does not match with the node information, the extracted determination target data is detected as data migration target data indicating data that needs to be migrated, and the detected data migration Migrating target data to the identified owner node and replica node; , It is executed, the parameter, the data data migration processing thread number indicating the maximum number of threads that can run the migration in parallel, the data migration processing execution interval indicating the wait time after execution of the data migration, is the changed It is at least one of the maximum number of simulations indicating the number of times that the data migration target data detection process is continuously executed , including a simulation that is a process for identifying an owner node and a replication node corresponding to the node identifier management information A data migration processing method characterized by

In this way, the data migration process (re-redundancy process or rearrangement process) is not executed immediately after the nodes constituting the cluster leave or are added, but the data migration process is performed at a predetermined timing. The process for detecting the data (data migration target data) is executed, and the data migration process is executed for the data detected there. Therefore, data can be migrated (re-redundant / rearranged) while suppressing the load on the node.
Further, at least one of the number of data migration processing threads, the data migration processing execution interval, and the maximum number of simulations can be set as a parameter, and data migration can be executed gradually.

According to a second aspect of the present invention, each of the plurality of nodes monitors the processing load of the node itself, and outputs processing interruption information to the data migration processing unit when the processing load exceeds a predetermined value. the node load monitoring unit, further wherein the data migration processing unit receives the process interruption information and the data migration processing system according to claim 1, characterized in that interrupting the data migration.
The invention according to claim 5 further comprises the step of each of the plurality of nodes monitoring the processing load of the node itself, and interrupting the data migration when the processing load exceeds a predetermined value. The data migration processing method according to claim 4, wherein the data migration processing method is executed.

  In this way, each node can interrupt data migration when the processing load exceeds a predetermined value by monitoring its own processing load. Therefore, in addition to data migration processing at a predetermined timing by parameter setting, data migration can be interrupted even when the processing load of the node exceeds a predetermined value, so the node load is reliably suppressed Data can be transferred while

According to a third aspect of the present invention, when each of the plurality of nodes receives a message requesting the provision of the service by the data from the client, the node performs processing of the message, and sets the target of the service. A message processing unit for migrating the data to be serviced to an owner node and a replica node corresponding to the changed node identifier management information when the data to be migrated is the data to be migrated; The data migration processing system according to claim 1 or claim 2 is provided.

  In this way, each node performs message processing on data that is predicted to be highly usable after receiving a message, and performs data migration processing (re-redundancy). Recovery processing or relocation processing) can be executed, and the reduction in redundancy and proper data placement can be quickly achieved.

  According to the present invention, it is possible to provide a data migration processing system and a data migration processing method for migrating data while suppressing the processing load of a node when a node constituting a cluster is detached or added.

It is a figure which shows the whole structure of the distributed processing system containing the data migration processing system which concerns on this embodiment. It is a figure for demonstrating the outline | summary of the data redundancy process by each node which comprises the data migration processing system which concerns on this embodiment. It is a functional block diagram which shows the structural example of the node which concerns on this embodiment. It is a figure which shows the data structural example of the node identifier management table which concerns on this embodiment. 5 is a flowchart showing an overall flow of data migration processing executed by each server in the data migration processing system according to the present embodiment. It is a flowchart which shows the flow of the detection process of the data migration object data which the data migration process part of the node which concerns on this embodiment performs. It is a figure for demonstrating the predetermined timing of the data migration process set by the data migration parameter management part of the node which concerns on this embodiment. It is a functional block diagram which shows the structural example of the node which concerns on the modification 1 of this embodiment. It is a functional block diagram which shows the structural example of the node which concerns on the modification 2 of this embodiment. It is a figure for demonstrating the data management method by the consistent hash method. It is a figure which shows the example which the node left | separated in the data management method by the consistent hash method.

  Next, a data migration processing system and the like in a mode for carrying out the present invention (hereinafter referred to as “the present embodiment”) will be described.

<Overall configuration of distributed processing system>
First, the overall configuration of the distributed processing system 1000 including the data migration processing system 100 according to the present embodiment will be described.
FIG. 1 is a diagram showing an overall configuration of a distributed processing system 1000 including a data migration processing system 100 according to the present embodiment.

  The distributed processing system 1000 includes a load balancer 3 that receives messages from each client 2, a distribution device 4, and a plurality of nodes 1 that form a cluster. The load balancer 3 distributes the message from the client 2 to each distribution device 4 by simple round robin or the like. The distribution device 4 distributes the received message to each node 1 based on, for example, a consistent hash method. Each node 1 performs message processing and provides a service to the client 2. In the present embodiment, a plurality of nodes 1 constituting a cluster will be described as the data migration processing system 100.

  In FIG. 1, the distribution device 4 and the node 1 are described as separate devices, but can be operated as separate functions on the same server. The distribution device 4 can also take a cluster configuration as shown in FIG. Further, the load balancer 3 does not exist, and a message can be transmitted from the client 2 to an arbitrary distribution device 4.

In the present embodiment, as a data management method of the distributed processing system 1000, a data management method based on the consistent hash method, which is less affected when the node 1 leaves or is added, will be described as an example. However, it is not limited to the consistent hash method. Further, according to the replica data arrangement method shown in FIG. 10, the node 1 on the right side (next clockwise) in the ID space of the consistent hash method is assigned the replica data.
Further, in the data migration processing system 100 according to the present embodiment, data re-redundancy processing and rearrangement processing are executed with the same mechanism both when the node 1 is detached and when it is added. An example of executing the conversion processing will be mainly described.

<Process overview>
Next, an outline of processing of the data migration processing system 100 according to the present embodiment will be described.
FIG. 2 is a diagram for explaining the outline of the data re-redundancy processing (the number of copies: 1) by each node 1 constituting the data migration processing system 100 according to the present embodiment. 2A shows an initial state before the re-redundancy processing, and FIG. 2B shows a state after the node 1 (here, the node “4”) is removed (detached). (C) shows a state in which the data redundancy process is executed.

First, as shown in FIG. 2A, in the initial state, for the data X (ID of the data X (data identifier)), the node “1” that first meets the ID space in the clockwise direction is the responsible node. Become. That is, the original data of the data X is stored in the node “1”. The node 1 that stores and manages the original data may be hereinafter referred to as an “owner node”. Then, the node “4” on the right side of the node “1” that is the owner node stores the duplicate data of the data X. Note that the node 1 (replica node) that stores and manages the replicated data may be hereinafter referred to as a “buddy”.
At this time, each node 1 adds an identifier of the owner node and an identifier of the buddy to each data stored in itself (step S1). The identifier of the owner node and the identifier of the buddy (duplicate node) are collectively referred to as node information.

  Next, as shown in FIG. 2B, it is assumed that one of the nodes 1 constituting the cluster, here, the node “4” has left due to a failure or the like. When each node 1 detects the detachment of the node 1 (node “4”) constituting the cluster, (1) data managed as an original (original data), and (2) managed as a replica Data that is the same as node 1 from which the owner node that manages the original data of the replicated data, that is, the replicated data from which the original data has been lost, is removed It is extracted as data (hereinafter referred to as “determination target data”) as a determination target of whether or not redundancy processing is necessary (step S2).

Subsequently, each node 1 simulates the nodes (owner nodes and buddies) that should be in charge after the reduction with respect to the extracted determination target data at a predetermined timing, and the simulation result and the node information of the determination target data Compare Then, as a result of the comparison, each node 1 detects determination target data that does not match the owner node identifier or buddy identifier as data migration target data, and performs data migration processing (here, re-redundancy processing). Execute (Step S3).
The predetermined timing at which each node 1 performs the simulation is determined by setting a plurality of parameters. This parameter includes, for example, the number of data migration processing threads (the maximum number of threads that can execute data migration processing (re-redundancy processing or rearrangement processing) in parallel), the data migration processing execution interval (immediately after executing the data migration processing) standby time), the simulation maximum number (each thread is determined by the maximum number of times) or the like running a simulation continuously. Details of these parameters will be described later.

  As described above, in the data migration processing system 100 or the like according to the present embodiment, the data migration processing (re-redundancy processing or rearrangement processing) is not performed immediately after the node 1 constituting the cluster leaves or is added. Instead, a process for detecting data (data migration target data) to be subjected to the data migration process is executed at a predetermined timing, and the data migration process is executed for the detected data. Therefore, data can be migrated (re-redundant or rearranged) while suppressing the load on the node 1.

<Node configuration>
Hereinafter, a configuration example of the node 1 configuring the data migration processing system 100 according to the present embodiment will be specifically described.

FIG. 3 is a functional block diagram illustrating a configuration example of the node 1 according to the present embodiment.
As shown in FIG. 1, the node 1 is communicably connected to the sorting device 4 and is also communicably connected to other nodes 1 other than itself constituting the cluster. Then, it receives a message from the client 2 and provides a service. The node 1 executes re-redundancy processing or rearrangement processing on data that needs to be migrated when the node 1 constituting the cluster is disconnected or added.
As shown in FIG. 3, the node 1 includes a control unit 10, an input / output unit 11, a memory unit 12, and a storage unit 13.

  The input / output unit 11 (input unit) inputs / outputs information to / from the distribution device 4 and other nodes 1 other than itself. The input / output unit 11 includes a communication interface that transmits and receives information via a communication line, and an input / output interface that performs input / output between an input unit such as a keyboard (not shown) and an output unit such as a monitor. Consists of

  The control unit 10 controls the entire node 1, and includes a node identifier management unit 101, a message processing unit 102, an alive monitoring unit 103, a node information addition unit 104, a data extraction unit 105, a data migration processing unit 106, and a data migration parameter management. The unit 107 is configured to be included. In addition, this control part 10 is implement | achieved when CPU (Central Processing Unit) expand | deploys and executes the program stored in the memory | storage part 13 on RAM (Random Access Memory) which is the memory part 12, for example.

  The node identifier management unit 101 manages identification information regarding each node 1 constituting the cluster as a node identifier management table 400 (node identifier management information).

FIG. 4 is a diagram showing a data configuration example of the node identifier management table 400 (node identifier management information) according to the present embodiment. As shown in FIG. 4, the node identifier management table 400 is configured to include a node identifier 401 and an address 402 of each node 1 constituting the cluster.
This node identifier 401 corresponds to the node ID on the consistent hash method ID space. Further, when a virtual ID is used in the consistent hash method, the node identifier 401 is assigned for each virtual ID and registered in the node identifier management table 400. In the node identifier management table 400, for example, by arranging the node identifiers 401 in ascending order, IDs (or virtual IDs) in the ID space of the consistent hash method can be arranged and managed in ascending order. That is, in the node identifier management table 400, the next node 1 when the node identifiers are arranged in ascending order is the node 1 on the right side (next clockwise) in the ID space.
For example, in FIG. 4, for data whose data identifier based on the ID space of the consistent hash method is “0” to “56”, the node (node identifier “56”, address The node “192.168.0.24”) is in charge as the “owner node”, and the node indicated by the next second line (node identifier “172”, address “192.168.1.25”) Indicates that he will be in charge as a buddy. Similarly, for data whose data identifier is “57” to “172” obtained by adding 1 to “56”, the node pointed to by the second row is in charge as the “owner node”, and the next third row Indicates that the node pointed to by is in charge as a buddy.
Thus, based on this node identifier management table 400, data, its owner node, and buddies are associated with each other.
The node identifier 401 can be assigned to each node 1 by the node identifier management unit 101, or the node identifier management table 400 generated by another node 1 or an external device (for example, the distribution device 4). Can be received and stored.

  When a node 1 leaves a plurality of nodes 1 constituting a cluster, the node identifier management unit 101 deletes a record including the node identifier 401 and the address 402 of the node 1. In addition, when a node 1 is added to a plurality of nodes 1 constituting the cluster, the node identifier management unit 101 newly registers a record including the node identifier 401 and the address 402 of the node 1.

Returning to FIG. 3, the message processing unit 102 provides the service by receiving the message distributed from the distribution device 4, executing the processing of the message, and returning the processing result to the client 2. The processing executed by the message processing unit 102 by this message is, for example, data registration, update, search, and deletion. Further, when receiving a message such as data registration or update, the message processing unit 102 receives a node other than itself (here, the next node when node identifiers are arranged in ascending order, that is, a consistent hash). Data redundancy is realized by duplicating data on the right adjacent node in the legal ID space. In addition, when the node 1 itself does not hold data necessary for message processing, the message processing unit 102 can acquire the data by requesting the other node 1 or the like. is there.
Note that the message processing unit 102 receives a message regarding data to be migrated before re-redundancy or rearrangement of data in the case where the node 1 constituting the cluster is disconnected or added. If so, the message processing is executed and the data migration processing (re-redundancy processing or rearrangement processing) of the data is executed.

  The alive monitoring unit 103 monitors the detachment or addition of the nodes 1 constituting the cluster by executing transmission / reception of alive monitoring signals at predetermined time intervals with other nodes 1 other than itself. When the alive monitoring unit 103 detects the detachment or addition of the node 1, the alive monitoring unit 103 notifies the node identifier management unit 101 of the own node 1 or the other node 1 or an external device for setting the node identifier, and manages the node identifier. This is reflected in the table 400. That is, the plurality of nodes 1 constituting the cluster are always provided with the node identifier management table 400 having the same contents.

The node information assigning unit 104 assigns node information to each data in association with the data identifier of each data. The node information is information on the node identifier of the node holding the original data (owner node) and the node identifier of the node holding the duplicate data (buddy: duplicate node) as described above. . The node information adding unit 104 adds new node information to the data when the data migration processing unit 106 executes a data re-redundancy process or a rearrangement process by the removal or addition of the node 1.
Therefore, the node information adding unit 104 applies the node to the target data when registering new data and when executing the data re-redundancy process or the rearrangement process by the removal or addition of the node 1 or the like. Give information.

The data extraction unit 105 is triggered by the change of the node identifier management table 400 (see FIG. 4) by the node identifier management unit 101 (1) data (original data) managed by itself as an original, and (2) Of the data (copy data) that is managed as a copy by itself, the data that matches the node 1 from which the node 1 (owner node) that manages the original data of the copy data leaves, that is, The duplicate data in which the original data is lost is extracted as data (determination target data) that is subject to determination as to whether or not data migration processing (re-redundancy processing or relocation processing) is necessary, and the data identifier of the extracted data is And stored in the extracted data management table 200.
As described above, the trigger of this processing is a change in the node identifier management table 400 (see FIG. 4). However, in another embodiment, some nodes 1 or external devices (for example, system management devices) are others. By transmitting a re-redundancy processing or relocation processing start request message to each node 1, the other nodes 1 may receive this message as a trigger for this processing. .
When node 1 is added, since there is no duplicate data in which the original data in (2) has been lost, only data (original data) managed by the node 1 as the original is extracted.

  The data migration processing unit 106 changes the node that has been changed with respect to the determination target data extracted by the data extraction unit 105 and stored in the extracted data management table 200 at a predetermined timing based on the parameters set by the data migration parameter management unit 107. Based on the identifier management table 400, an owner node and a buddy when a predetermined data management method such as a consistent hash method is used are specified (hereinafter, this process is referred to as “simulation”), and each data is specified. Compare with the node information given to (determination target data). Then, as a result of this comparison, the data migration processing unit 106 detects data whose owner node identifier and buddy identifier do not match (completely match) as data migration target data, and performs data migration processing (re-redundancy processing or Relocation processing).

  When the redundancy number is 3 or more (duplicate data is 2 or more), a plurality of nodes 1 that manage the replicated data (from the node close to the owner node that manages the original data on the ID space, buddy “1”, buddy “2”,...)) May perform data migration processing on the same data. Therefore, when there are a plurality of buddies, the buddy with the smallest number is in charge of the data migration process. Details of the data migration target data detection process will be described later with reference to FIG.

  The data migration parameter management unit 107 sets a predetermined timing at which the data migration processing unit 106 executes the data migration processing using one or more parameters. By adjusting the execution timing of the data migration processing by the data migration parameter management unit 107, the load is adjusted so that the load of the data migration processing is concentrated at a time and the normal processing is not disturbed and the service quality is not deteriorated. However, the data migration process can be executed.

The parameters set in the data migration parameter management unit 107 are, for example, the number of data migration processing threads, the data migration processing execution interval, and the maximum number of simulations.
The number of data migration processing threads is the maximum number of threads that can execute re-redundancy processing and rearrangement processing in parallel. Setting a large value for the number of data migration processing threads can reduce the time required for the data migration processing, but the processing load on the node 1 increases. On the other hand, if a small value is set for the number of data migration processing threads, the time required for the data migration processing increases, but the processing load on the node 1 decreases.
The data migration processing execution interval is the result of the data migration processing unit 106 executing the data migration target data detection process (FIG. 6) in each thread, so that the data migration target data is detected and the data migration processing unit 106 It refers to the time to wait after executing data migration processing (re-redundancy processing or relocation processing).
The maximum number of simulations means that each thread continuously executes the data migration target data detection process including the above-described simulation (identification of owner node and buddy based on the changed node identifier management table 400 (see FIG. 4)). The number of times to do. The data migration processing unit 106 continuously executes the data migration target data detection process, waits for a predetermined time, and then starts the data migration target data detection process including simulation again.
By setting the maximum number of simulations small or setting the data migration processing execution interval long, the node 1 can gradually execute the data migration processing while suppressing the processing load.
The data migration processing thread number, the data migration processing execution interval, and the simulation maximum number of parameters may be all set by the data migration parameter management unit 107, or any one of them may be set. Any combination may be set.
Details of the data migration process at a predetermined timing adjusted by the parameters set by the data migration parameter management unit 107 will be described later with reference to FIG.

  The memory unit 12 includes a primary storage device such as a RAM, and temporarily stores information necessary for data processing by the control unit 10. The memory unit 12 stores the extracted data management table 200 that stores the data identifier of the determination target data extracted by the data extraction unit 105.

The storage unit 13 includes a storage device such as a hard disk or a flash memory, and stores data 300 including original data and copy data to be serviced, the node identifier management table 400 (see FIG. 4), and the like. The storage unit 13 stores parameter values (not shown) set by the data migration parameter management unit 107.
Each data of the data 300 includes a data identifier of the data, a node identifier of the owner node, and a node identifier of the buddy.

<Processing flow of data migration processing system>
Next, the flow of processing when executing data migration processing in the data migration processing system 100 according to the present embodiment will be described.

(Overall flow of data migration processing)
FIG. 5 is a flowchart showing the overall flow of data migration processing executed by each node 1 in the data migration processing system 100 according to the present embodiment.

First, the alive monitoring unit 103 of the node 1 transmits / receives a alive monitoring signal to / from other nodes 1 other than the node 1 at a predetermined time interval, so that the node 1 constituting the cluster is disconnected or added. Whether or not (step S10). If the life and death monitoring unit 103 determines that the node 1 has been detached or added (step S10 → Yes), the information on the withdrawal or addition is output to the node identifier management unit 101. On the other hand, if the life and death monitoring unit 103 determines that the node 1 is not detached or added (step S10 → No), the determination process of step S10 is repeated.
The life / death monitoring unit 103 does not transmit / receive a life / death monitoring signal to / from another node 1, but leaves or adds a node 1 detected by an external device (for example, the distribution device 4) or another node 1. By receiving the information, the node 1 may be removed or additional information may be acquired.

Next, the node identifier management unit 101 changes the node identifier management table 400 (see FIG. 4) based on the information about the detachment or addition of the node 1 received from the alive monitoring unit 103 (step S11).
Specifically, the node identifier management unit 101 deletes a record including the node identifier 401 and the address 402 of the node 1 when a node 1 leaves the plurality of nodes 1 constituting the cluster. Further, when a new node 1 is added to the plurality of nodes 1 constituting the cluster, the node identifier management unit 101 newly registers a record including the node identifier 401 and the address 402 of the node 1. .

Subsequently, the data extraction unit 105 extracts the determination target data when the node identifier management unit 400 changes the node identifier management table 400 (see FIG. 4), and extracts the data identifier of the determination target data. The data is stored in the data management table 200 (step S12).
Specifically, the data extraction unit 105 includes (1) original data managed as original data (original data) and (2) original data among data managed as duplicates (replicated data). Is extracted as data (determination target data) that is subject to determination as to whether data migration processing (re-redundancy processing or relocation processing) is necessary, and the data identifier of the extracted data is extracted Store in the data management table 200.

Then, the data migration processing unit 106 determines whether it is a predetermined timing set by the data migration parameter management unit 107 (step S13). If it is a predetermined timing (step S13 → Yes), the data migration processing unit 106 proceeds to the next step S14, and if it is not the predetermined timing (step S13 → No), it waits until the predetermined timing is reached.
The predetermined timing set by the data migration parameter management unit 107 will be described with reference to FIG.

  In step S14, the data migration processing unit 106 refers to the extracted data management table 200 and extracts one determination target data (specifically, a data identifier of the determination target data) (step S14).

  Next, the data migration processing unit 106 detects whether or not the determination target data extracted in step S14 is data (data migration target data) that is a target of data migration processing (re-redundancy processing or rearrangement processing). Processing (data migration target data detection processing) is executed (step S15). Details of the data migration target data detection process will be described with reference to FIG.

  Subsequently, the data migration processing unit 106 determines whether or not data migration target data has been detected in step S15 (step S16). If the data migration target data is detected (step S16 → Yes), the data migration processing unit 106 proceeds to step S17. If not detected (step S16 → No), the data migration processing unit 106 proceeds to step S18. move on.

  In step S17, the data migration processing unit 106 executes data migration processing (re-redundancy processing or rearrangement processing). The data migration processing by the data migration processing unit 106 is performed by the node information adding unit 104 simulating the node information given to the data to be migrated in the data migration target data detection process in step S15 (node identifier after change). This is executed after changing the owner node identifier and buddy identifier based on the management table 400 (see FIG. 4).

  Next, in step S <b> 18, the data migration processing unit 106 determines whether all the determination target data (data identifiers) stored in the extracted data management table 200 have been processed. Here, if the extracted data management table 200 stores determination target data (data identifier) that has not yet been processed (step S18 → No), the data migration processing unit 106 returns to step S13 to perform the processing. to continue. On the other hand, if the data migration processing unit 106 has finished processing all the determination target data (data identifiers) stored in the extracted data management table 200 (step S18 → Yes), the data migration processing ends.

Even while the data migration processing unit 106 of the node 1 gradually executes the data migration processing in steps S13 to S18 in FIG. 5, the node 1 receives the message from the client 2 and receives the message processing unit. 102 executes processing for the message. The message processing unit 102 executes processing on the received message and determines whether or not the data targeted by the message is data migration target data with reference to the node information attached to the data. Specifically, if any of the owner node and the buddy in the node information is different from the own node and the right adjacent node in the ID space, it is determined as the data migration target data. When the data is data migration target data, the message processing unit 102 executes data migration processing (re-redundancy processing or rearrangement processing) of the data. When the data related to the message received by the own node is not held (which may occur when a node is added to the right side of the ID space), the message processing unit 102 receives data from other nodes. Message processing is executed with relocation processing including acquisition of. The data acquisition destination is specified with reference to the node identifier management table 400.
In this way, the node 1 performs message processing on the data that is predicted to be highly usable after receiving the message, and also performs data migration processing (re-redundancy). Or the rearrangement process), the reduction of the redundancy can be recovered and the proper arrangement of the data can be quickly achieved.

(Data migration target data detection process)
Next, the data migration target data detection process executed by the data migration processing unit 106 of the node 1 in step S15 in FIG. 5 will be described in detail.
FIG. 6 is a flowchart showing a flow of data migration target data detection processing performed by the data migration processing unit 106 of the node 1 according to the present embodiment.

  First, the data migration processing unit 106 of the node 1 executes a simulation (described as “A process” in FIG. 6) for the determination target data extracted from the extracted data management table 200 (step S20). Specifically, the data migration processing unit 106 uses the owner of the extracted determination target data in accordance with a predetermined data management method such as a consistent hash method based on the changed node identifier management table 400. Identify nodes and buddies.

  Next, the data migration processing unit 106 extracts node information (owner node identifier and buddy identifier) attached to the determination target data extracted from the extracted data management table 200 (in FIG. 6, “B processing”). (Step S21).

  Subsequently, as a result of the simulation (A process), it is determined whether or not the own node is the owner node (step S22). Here, as a result of the simulation (A process), when the own node is the owner node (step S22 → Yes), the process proceeds to the next step S23, while when the own node is not the owner node (step S22-> No), it progresses to step S25.

  In step S23, the data migration processing unit 106 determines whether or not the own node is the owner node as a result of the node information extraction (B process). Here, when the node information is extracted (B process), if the own node is not the owner node (step S23 → No), the data (determination target data) is converted into data (data migration target). Data). That is, regarding the same data, since the owner node does not match between the A process and the B process, the data needs to be migrated (described as “data migration target” in FIG. 6). On the other hand, as a result of the node information extraction (B process), when the own node is the owner node (step S23 → Yes), the process proceeds to the next step S24.

  In step S24, the data migration processing unit 106 determines whether or not the buddies match between the result of the simulation (A process) and the result of the node information extraction (B process). Here, when the buddies match (step S24 → Yes), the data (determination target data) is data that does not require data migration (described as “not subject to data migration” in FIG. 6). On the other hand, if the buddies do not match (step S24 → No), the data (determination target data) is determined as data (data transfer target data) to be subjected to data transfer processing. That is, even if the owner nodes match, the buddies do not match, so the data becomes the data migration target data.

  Next, in step S22, when the result of the simulation (A process) is that the own node is not the owner node (step S22 → No), the data migration processing unit 106 obtains the result of the node information extraction (B process). It is determined whether or not the node is the owner node (step S25). Here, when the node information is extracted (B process) and the own node is the owner node (step S25 → Yes), the data (determination target data) is converted into the data (data migration target) for the data migration process. Target data). That is, since the owner node does not match between the A process and the B process, it becomes a data migration target. On the other hand, as a result of the node information extraction (B process), if the own node is not the owner node (step S25 → No), the process proceeds to the next step S26.

  In step S26, the data migration processing unit 106 determines whether the owner node exists in the node identifier management table 400 (see FIG. 4) after the change as a result of the node information extraction (B process). If it exists (step S26 → Yes), the data (determination target data) is not subject to data migration. On the other hand, if it does not exist (step S26 → No), the process proceeds to the next step S27.

  In step S27, as a result of the node information extraction (B process), the data migration processing unit 106 has a buddy younger than itself (a buddy with a younger number when node identifiers are arranged in ascending order), and the node identifier after the change In the management table 400 (see FIG. 4), it is determined whether there is a buddy younger than itself. That is, when there are a plurality of buddies, it is determined whether or not it is the youngest buddy. Here, if there is a buddy younger than itself (step S27 → Yes), the data (determination target data) is transferred to the data transfer target in order to avoid performing the data transfer process on the same data redundantly. It is outside. On the other hand, when there is no buddy younger than itself (step S27 → No), the data (determination target data) is determined as data (data migration target data) to be subjected to data migration processing.

  In this way, the data migration processing unit 106 determines whether or not the data migration processing should be executed for each determination target data extracted in the extracted data management table 200, and detects the data migration target data. Can do. Even if there are multiple buddies, the data migration process is executed by the owner node and the buddy with the lowest number on both sides of the node 1 that has been detached or added. It is possible to avoid duplicating the data migration process.

(Data migration process at a predetermined timing)
Next, a predetermined timing of the data migration process based on the parameters set by the data migration parameter management unit 107 will be described with reference to FIG.
FIG. 7 shows an example in which the number of data migration processing threads, the data migration processing execution interval, and the maximum number of simulations are set as examples of parameters set in the data migration parameter management unit 107.

FIG. 7 shows an example in which the number of data migration processing threads (symbol P1) is set to “3”.
In each thread, when the data migration processing unit 106 acquires the determination target data from the extracted data management table 200 (step S30), the data migration target data is detected (step S31). If detected, data migration processing (re-redundancy processing or rearrangement processing) is executed as it is (step S32).

  Here, for the data migration target data, after the data migration processing is executed in step S32, the data migration processing unit 106 sets the data migration processing execution interval (reference P2), and therefore, for a predetermined time, The process waits without performing the determination target data acquisition process (step S30).

In addition, as shown in the thread “1” in FIG. 7, when the parameter of the maximum simulation count (symbol P3) is set to “5”, the data migration processing unit 106 determines from the extracted data management table 200. The target data is acquired, and the data migration target data detection process is continuously executed five times. The data migration processing unit 106 waits for a predetermined time when the data migration target data is not detected even if the data migration target data is detected the maximum number of times of simulation (here, “5” times).

  In this way, the data migration target data detection process is executed at a predetermined timing set by each parameter, and the data migration process can be executed when the data migration target data is detected. Therefore, immediately after the node 1 constituting the cluster leaves or is added, the other existing nodes 1 do not execute the data migration process at a time, so that the node 1 gradually performs the data migration process while reducing the processing load. (Re-redundancy processing or rearrangement processing) can be executed.

  As described above, according to the data migration processing system 100 and the data migration processing method according to the present embodiment, when the node 1 constituting the cluster is detached or added, the processing load on the node 1 is suppressed. Data can be migrated.

<Modification 1>
Next, a first modification of the data migration processing system 100 according to the present embodiment will be described.
FIG. 8 is a functional block diagram illustrating a configuration example of the node 1a according to the first modification of the present embodiment.
A difference from the node 1 according to the present embodiment illustrated in FIG. 3 is that a node load monitoring unit 108 is additionally provided in the control unit 10.

The node load monitoring unit 108 monitors the processing load (for example, CPU usage rate, memory usage rate, etc.) of the node 1a itself, and when the processing load exceeds a predetermined value set in advance, the data migration processing unit By outputting the processing interruption information to 106, the data migration processing by the data migration processing unit 106 is suspended. Specifically, for example, the data migration processing unit 106 interrupts the data migration processing by canceling the acquisition of the determination target data from the extracted data management table 200 shown in step S30 of FIG.
Then, the node load monitoring unit 108 outputs the processing start information to the data migration processing unit 106 when the processing load of the node 1a itself becomes a predetermined value or less, thereby performing the data migration processing by the data migration processing unit 106. Let it resume. For example, the data migration processing unit 106 executes the data migration processing by resuming the determination target data acquisition processing shown in step S30 of FIG.

  Thus, even if the data migration processing unit 106 is executing the data migration processing at a predetermined timing based on the parameters set by the data migration parameter management unit 107, for some reason (for example, When a large amount of messages are received from the client 2 and the processing load of the node 1a exceeds a predetermined value, the data migration processing (re-redundancy processing or rearrangement processing) can be interrupted. Therefore, it is possible to suppress the processing load on the node 1a.

<Modification 2>
Next, a second modification of the data migration processing system 100 according to the present embodiment will be described.
FIG. 9 is a functional block diagram illustrating a configuration example of the node 1b according to the second modification of the present embodiment.
A difference from the node 1 a according to the first modification of the present embodiment illustrated in FIG. 8 is that the control unit 10 does not include the data migration parameter management unit 107.

In this case, since the node 1b does not include the data migration parameter management unit 107, the determination target data is extracted from the extracted data management table 200 shown in steps S13 and S14 in FIG. 5 at a predetermined timing based on the set parameters. The data migration processing unit 106 does not execute the process to obtain. The data migration processing unit 106 immediately acquires the determination target data from the extracted data management table 200 in step S14 without executing step S13, executes the data migration target data detection process, and detects the data migration target data. If so, the data migration processing of the data is executed. In this case, if step S18 → No in the flow of FIG. 5, the process returns to step S14.
Then, the node load monitoring unit 108 monitors the processing load (for example, CPU usage rate, memory usage rate, etc.) of the node 1b itself, and when the processing load exceeds a predetermined value set in advance, data migration is performed. By outputting the processing interruption information to the processing unit 106, the data migration processing by the data migration processing unit 106 is interrupted. Further, the node load monitoring unit 108 outputs the processing start information to the data migration processing unit 106 when the processing load of the node 1b itself becomes a predetermined value or less, thereby performing the data migration processing by the data migration processing unit 106. Let it resume.

  In this way, the node 1b interrupts the data migration process when its own processing load exceeds a predetermined value, and resumes the data migration process when the processing load falls below the predetermined value. Can do. Thus, the node 1b can directly monitor the processing load of itself and execute the data migration process after suppressing the processing load from exceeding a predetermined value.

1, 1a, 1b Node 2 Client 3 Load balancer 4 Distribution device 10 Control unit 11 Input / output unit (input unit)
DESCRIPTION OF SYMBOLS 12 Memory part 13 Storage part 100 Data migration processing system 101 Node identifier management part 102 Message processing part 103 Life / death monitoring part 104 Node information provision part 105 Data extraction part 106 Data migration processing part 107 Data migration parameter management part 108 Node load monitoring part 200 Extracted data management table 300 data (service target data)
400 Node identifier management table (node identifier management information)
1000 Distributed processing system

Claims (5)

One of a plurality of nodes constituting a cluster is assigned as an owner node that stores data for providing services to clients as original data, or one or more replication nodes that store duplicate data of the data A data migration processing system for storing
Each of the plurality of nodes constituting the data migration processing system is
A storage unit that stores node identifier management information in which the data, the owner node, and the duplicate node are associated with each other, with respect to each of the plurality of nodes to which a node identifier that is a unique identifier is attached;
A node information giving unit that gives, as node information, a node identifier of the owner node that stores the original data and a node identifier of the duplicate node that stores the duplicate data, respectively, to the original data and the duplicate data;
Detecting the removal or addition of the node, and storing the node identifier management information by changing the data to the new association between the data and the owner node and the replication node according to the removal or addition of the node A node identifier management unit
When the separation of the node is detected, based on the changed node identifier management information, the original data stored by itself and the duplicate data stored by itself, the original data Is lost, the replication data is extracted as determination target data indicating data to be determined whether or not the data migration performed to change the owner node or the replication node is necessary,
A data extraction unit that extracts the original data stored therein as the determination target data based on the changed node identifier management information when the addition of the node is detected;
The owner node corresponding to the changed node identifier management information for the extracted determination target data at a predetermined timing based on a parameter set to suppress the processing load of the node itself due to the data migration If the node identifier of each of the identified owner node and duplicate node does not match the node information, the extracted determination target data is designated as the data that needs to be migrated. A data migration processing unit that detects the data migration target data shown and migrates the detected data migration target data to the identified owner node and replication node,
The parameter is
The number of data migration processing threads indicating the maximum number of threads that can execute the data migration in parallel, the data migration processing execution interval indicating the waiting time after executing the data migration, the owner corresponding to the changed node identifier management information A data migration processing system comprising at least one of a simulation maximum number of times indicating the number of times the data migration target data detection process is continuously executed , including a simulation that is a process of identifying a node and a replication node . Each of the plurality of nodes
A node load monitoring unit that monitors the processing load of the node itself, and outputs processing interruption information to the data migration processing unit when the processing load exceeds a predetermined value;
The data migration processing system according to claim 1, wherein the data migration processing unit suspends the data migration when receiving the processing interruption information. Each of the plurality of nodes
When a message requesting the provision of the service by the data is received from the client, the processing of the message is executed, and when the data targeted for the service is the data migration target data, The data migration according to claim 1, further comprising a message processing unit that migrates target data to an owner node and a replication node corresponding to the changed node identifier management information. Processing system. One of a plurality of nodes constituting a cluster is assigned as an owner node that stores data for providing services to clients as original data, or one or more replication nodes that store duplicate data of the data A data migration processing method of a data migration processing system for storing
Each of the plurality of nodes constituting the data migration processing system is
For each of the plurality of nodes to which a node identifier that is a unique identifier is attached, the storage unit stores node identifier management information in which the data, the owner node, and the duplicate node are associated with each other.
A node identifier of the owner node that stores the original data and a node identifier of the replica node that stores the duplicate data are provided as node information to the original data and the duplicate data, respectively.
Detecting the removal or addition of the node, and storing the node identifier management information by changing the data to the new association between the data and the owner node and the replication node according to the removal or addition of the node Step to
When the separation of the node is detected, based on the changed node identifier management information, the original data stored by itself and the duplicate data stored by itself, the original data Is lost, the replication data is extracted as determination target data indicating data to be determined whether or not the data migration performed to change the owner node or the replication node is necessary,
When the addition of the node is detected, based on the changed node identifier management information, extracting the original data stored therein as the determination target data;
The owner node corresponding to the changed node identifier management information for the extracted determination target data at a predetermined timing based on a parameter set to suppress the processing load of the node itself due to the data migration If the node identifier of each of the identified owner node and duplicate node does not match the node information, the extracted determination target data is designated as the data that needs to be migrated. Detecting as the data migration target data shown, and migrating the detected data migration target data to the identified owner node and replication node,
The parameter is
The number of data migration processing threads indicating the maximum number of threads that can execute the data migration in parallel, the data migration processing execution interval indicating the waiting time after executing the data migration, the owner corresponding to the changed node identifier management information A data migration processing method comprising: at least one of a simulation maximum number of times indicating the number of times the data migration target data detection process is continuously executed , including a simulation that is a process of identifying a node and a replication node . Each of the plurality of nodes
The data migration processing method according to claim 4, further comprising the step of monitoring the processing load of the node itself and suspending the data migration when the processing load exceeds a predetermined value.

Images