JP6322161B2 - Node, data relief method and program - Google Patents

Description

  The present invention relates to a technique for relieving data so as not to be affected by a disaster or the like in order to continue the service even when a large-scale disaster or the like occurs in a service using a highly available system having a cluster configuration.

  In a conventional highly available system with a cluster configuration, each server (hereinafter also referred to as a “node”) that constitutes a cluster can be added and removed in accordance with service usage and load fluctuations. Computational resource optimization (maintenance increase / decrease) is executed. For example, the following processing procedure (maintenance reduction) is executed when a server holding the original data (hereinafter sometimes referred to as “original data”) fails or when there is a high possibility of failure. The

(1) Promoting a copy of the original data (hereinafter sometimes referred to as “replicated data”) to the original data.
(2) Rearrange original data replicas (replicated data) so as to maintain a preset redundancy number.
(3) Disconnect a failed server (a server with a high possibility of failure) from the cluster.
By performing such maintenance reduction, the original data is relieved and the service is continued. A server with a high possibility of failure is detected when there is a progressive failure such as a temperature rise due to hardware deterioration or abnormal heat generation, or failure information such as SNMP (Simple Network Management Protocol) trap or syslog. Or being detected by acquiring an alarm or the like.

  However, when a large-scale disaster or the like occurs, the original data and the duplicated data that are made redundant are the same base (an area where a data center exists (hereinafter sometimes referred to as a “data center area”)). When it is installed in the original data, the original data and the duplicate data may be lost at the same time. In view of this, there has been proposed a highly available system having a cluster configuration in which original data and replicated data are placed on servers at different locations so that redundant original data and replicated data are not lost at the same time (patents). Reference 1).

Japanese Patent Laying-Open No. 2015-082129

In the high availability system described in Patent Document 1, even when a large-scale disaster or the like occurs, either the original data or the replicated data is not lost, and the service can be continued. However, when a large-scale failure occurs due to a large-scale disaster and the original data is lost, the server failure detection processing is executed, and for the failed server, in addition to the above-described maintenance reduction processing, Traffic increases due to an increase in requests for using the service itself for normality confirmation. As a result, the performance of the system will be insufficient, and the network cost (communication cost and infrastructure cost) will increase due to the duplication and relocation of data to remote servers, and the system ( The problem is that there is a delay until the redundant system) is reconfigured.
Also, for example, when the time between a disaster forecast and a disaster is short, such as an earthquake early warning, and data must be relieved suddenly, the relief will be in time if a complicated procedure is followed, such as maintenance reduction. Not being an issue is also an issue.

  The present invention has been made in view of such a background. The present invention suppresses an increase in network cost and a delay in system reconfiguration even when a large-scale failure occurs due to a large-scale disaster or the like. Therefore, it is an object to provide a node, a data relief method, and a program that can rescue original data.

  In order to solve the above-described problem, the invention according to claim 1 is a distribution in which each of a plurality of nodes constituting a cluster holds data for processing as original data or duplicate data that is a duplicate of the original data. Each node of the processing system is in charge of processing in association with position information indicating an area where each of the nodes is physically installed and an identifier of the node installed in the area. A storage unit for storing node management information stored as replicated data, in which data is stored in the original data or in a node installed in a region different from the node holding the original data; Receiving failure probability information indicating failure probability at a predetermined time for each of the nodes associated with the information, and holding the original data A node that refers to the failure probability information, extracts a node having the failure probability exceeding a predetermined threshold as a relief node, and holds a duplicate data corresponding to the original data held by the extracted relief node, The node management information is detected with reference to the failure probability information, and the node having the lowest failure probability is determined as a replacement destination node among the nodes holding the detected duplicate data, and the rescue node and A node comprising: a data relief unit for exchanging a function for processing the original data and a function for processing the duplicate data with the exchange destination node.

  Further, according to the present invention, each of a plurality of nodes constituting a cluster holds the data in charge of processing as original data or duplicate data that is a duplicate of the original data. In the data relief method, each of the nodes performs processing in association with position information indicating an area where each of the nodes is physically installed and an identifier of the node installed in the area. A storage unit in which the data in charge is stored with the original data or node management information stored as the duplicated data arranged in a node installed in a region different from the node holding the original data. Failure probability information indicating failure probability at a predetermined time for each of the nodes associated with the position information is received. And referring to the failure probability information for a node holding the original data, extracting a node having the failure probability exceeding a predetermined threshold as a relief node, and corresponding to the original data held by the extracted relief node The node that holds the duplicate data to be detected is detected with reference to the node management information, and among the nodes that hold the detected duplicate data, the failure probability information is referred to, and the node with the lowest failure probability is exchanged A step of determining as a node, and a step of exchanging a function of processing the original data and a function of processing the duplicated data between the relief node and the exchange destination node. Data relief method was adopted.

  By doing so, for example, for a node whose failure probability exceeds a predetermined threshold before a large-scale failure due to a large-scale disaster or the like occurs, a function for processing original data and a function for processing duplicate data Are exchanged between nodes in different regions, so that even if a large-scale failure occurs thereafter, the processing can be continued without losing the original data. In addition, since only the function of processing the original data and the function of processing the replicated data are exchanged, there is no need to replicate or relocate the data, thereby increasing the network cost and delaying the system reconfiguration. The original data can be relieved after the suppression.

  The invention described in claim 3 is a program for causing a computer to execute the data rescue method according to claim 2.

  According to such a program, the data rescue method according to claim 2 can be realized by a general computer.

  According to the present invention, even when a large-scale failure occurs due to a large-scale disaster or the like, a node and a data relief method for relieving original data while suppressing an increase in network cost and a delay in system reconfiguration And can provide programs.

It is a figure which shows the whole structure of the distributed processing system containing the node which concerns on this embodiment. It is a figure for demonstrating the process outline | summary of the distributed processing system containing the node which concerns on this embodiment. It is a functional block diagram which shows the structural example of the node which concerns on this embodiment. It is a figure which shows the data structural example of the node management table which concerns on this embodiment. It is a figure which shows the example of a data structure of the life and death monitoring table which concerns on this embodiment. It is a figure which shows the data structural example of the failure probability information which concerns on this embodiment. It is a flowchart which shows the flow of the data relief process which the node concerning this embodiment performs. It is a figure which shows the example which lose | disappears both original data and replication data, when each node of the distributed processing system which performs the data management based on a consistent hash is arrange | positioned on ID space. It is a flowchart which shows the flow of the node insertion position determination process in ID space at the time of node participation which the distributed processing system of a comparative example performs. It is a figure for demonstrating the node insertion position determination process in ID space at the time of node participation which the distributed processing system of a comparative example performs.

<Distributed processing system of comparative example>
First, the above-described distributed processing system according to Patent Document 1 will be described as a comparative example of the cluster configuration high availability system (hereinafter also referred to as “distributed processing system”) according to the present embodiment.

In the distributed processing system according to Patent Document 1, data handled by each server constituting the cluster (hereinafter sometimes referred to as “node”) is determined based on the consistent hash method. Specifically, in the consistent hash method, IDs (IDentifiers) are assigned to both nodes and data. Each node is mapped to a consistent hash ID space (hereinafter, sometimes simply referred to as “ID space”), so that each node is also in charge of the range of hash IDs that it is in charge of (also referred to as “responsible region”). Say). When processing data, key information that uniquely identifies the data is applied to the hash function, and the first encounter is made by proceeding in a predetermined method (for example, clockwise) from the position in the ID space of the derived consistent hash. The node that performs the processing of the data (the node that holds the original data). The duplicated data is created in the adjacent node in the clockwise direction (or counterclockwise) in the ID space (when the redundancy number is “2”). When the redundancy number is “3” or more, duplicate data is created in the next node in the clockwise direction. Note that the redundancy number is the total number of nodes holding original data and nodes holding duplicate data.
In this way, even if the node that holds the original data leaves the cluster due to a failure or the like, the process can be continued by distributing the inquiry to the data to the node that holds the duplicate data. It becomes possible.

  As described above, the data management method based on the consistent hash method dynamically changes the cluster configuration because the data migration associated with the addition or removal of the nodes constituting the cluster is limited to a part of the data. This is effective for systems in which node addition / removal) occurs frequently. Further, in preparation for a failure of a node constituting the cluster, fault tolerance is improved by holding one or more nodes other than the node holding the original data to hold the duplicate data. However, without considering the physical arrangement of nodes (location information of areas, etc.), if a large-scale failure occurs due to a large-scale disaster, a failure may occur simultaneously in all nodes that hold the corresponding data There is sex. This will be specifically described below.

  FIG. 8 shows an example in which each node of the distributed processing system that performs data management based on the consistent hash is physically arranged. FIG. 8A shows a state before a large-scale disaster or the like occurs, and FIG. 8B shows a state after a large-scale disaster or the like occurs. As shown in FIG. 8A, a plurality of nodes belonging to each base (data center area) in Japan are arranged on the consistent hash ID space to form a cluster.

When a large-scale disaster or the like occurs in the Kanto area, a node in the Kanto area on the consistent hash ID space fails and leaves the ID space. At this time, in the normal data management method as shown in FIG. 8, since the physical arrangement of nodes (position information of areas and the like) is not considered, it is assumed that nodes in the same Kanto area are adjacent to each other. As shown in FIG. 8B, there is a possibility that both the original data and the duplicated data may be lost at the same time (in the case of the redundancy number “2”).
In this way, the distributed data processing system according to Patent Document 1 is a method in which the original data and the duplicated data are arranged on different servers so that the original data and the duplicated data that are made redundant are not lost at the same time. Proposed in

  In the distributed processing system according to Patent Document 1, a large-scale failure occurs due to a large-scale disaster or the like, and M (M ≧ 2, M is “redundant number”) continuous nodes in the consistent hash ID space are present. In order to prevent both the original data and the duplicated data from being lost even if they are separated at the same time, at least M−X copies of the duplicated data to nodes at different bases from the newly inserted node base (data center area). The node insertion position determination process in the ID space is executed in such a manner as to be arranged. Here, X (1 ≦ X <M) is the number of data set with respect to certain data (a total of M of original data and replicated data) that can be simultaneously lost due to a large-scale failure such as a large-scale disaster. Note that M-X corresponds to the number of nodes with different bases that are not lost even after a large-scale failure.

The node insertion position determination process in the distributed processing system according to Patent Document 1 will be described below with reference to FIGS. 9 and 10.
FIG. 9 is a flowchart showing a flow of node insertion position determination processing in the ID space performed by the distributed processing system according to Patent Document 1 when a node joins. FIG. 10 is a diagram for explaining node insertion position determination processing in the ID space when a node joins. Note that this node insertion position determination process may be performed by a specific node (privileged node) selected from each node of the distributed processing system according to Patent Document 1, or a management device that manages the entire distributed processing system However, here, it is assumed that the privileged node (hereinafter simply referred to as “node”) constituting the distributed processing system performs.

  First, for each base (data center area), a total value of the number of all nodes constituting the cluster (or the number of virtual nodes when a plurality of virtual nodes are assigned to one physical node) is calculated. (Step S1).

Subsequently, the server (node) of the base having the smallest base value calculated in step S1 is selected (step S2).
FIG. 10 shows that a standby server belonging to the base (data center area) of “◯ (white circle)” is selected as an example (T1 in FIG. 10).

Next, candidates for IDs (insertion positions) in the ID space are selected according to a predetermined ID assignment method (for example, a method of dividing the maximum region in the ID space at random or into two) (step S3).
FIG. 10 shows that the ID between the base server indicated by “Δ (white triangle)” and the base server indicated by “□ (white square)” is selected as the insertion position as an example. (T2 in FIG. 10).

Then, from the selected insertion position, the bases to which the surrounding (left and right (both sides)) M-1 nodes belong are confirmed, and whether the bases to which M−X or more nodes belong are different from the bases of the newly inserted nodes. It is determined whether or not (step S4, T3 in FIG. 10). In addition, the fact that “the base where the M−X or more nodes belong is different from the base of the newly inserted node” may be hereinafter referred to as “node insertion condition for withstanding a large-scale failure”.
Here, when the node insertion condition for withstanding this large-scale failure is not satisfied (step S4 → No), the process returns to step S3, and the ID (insertion position) selected so far using a predetermined ID allocation method. ), The insertion position selection process is executed again, and the process is repeated. On the other hand, when the node insertion condition for withstanding this large-scale failure is satisfied (step S4 → Yes), the ID selected in step S3 is determined as the insertion position in the ID space of the newly participating node (step S4). S5). Then, the process ends.

By doing so, in the distributed processing system according to Patent Document 1, when a new server joins the cluster, M−X or more of the bases to which the M−1 nodes around the insertion position belong are included. The bases to which the nodes belong can be arranged differently from the bases of the newly inserted nodes. Therefore, when a node (privileged node) joins a new server (node), the arrangement based on this node insertion position determination process is repeated in the cluster, so that nodes at the same site are adjacent in the ID space. Can be suppressed, and the occurrence of data acquisition processing at the time of replication deviation and node departure can be suppressed.
However, as described above, in the distributed processing system according to Patent Document 1, when a large-scale failure occurs due to a large-scale disaster or the like and the original data is lost, the traffic is increased to perform maintenance reduction, etc. In addition to insufficient system performance, data replication and relocation to remote servers increase network costs and cause delays before reconfiguring a redundant system. An embodiment according to the present invention for solving this problem will be described below.

<Distributed processing system of this embodiment>
Next, a distributed processing system 1000 (see FIG. 1) including the node 1 (see FIG. 3) according to a mode for carrying out the present invention (hereinafter referred to as “this embodiment”) will be described.

≪Overall configuration of distributed system≫
First, the overall configuration of the distributed processing system 1000 including the node 1 according to the present embodiment will be described.
FIG. 1 is a diagram showing an overall configuration of a distributed processing system 1000 including a node 1 according to the present embodiment.

  The distributed processing system 1000 includes a load balancer 3 that receives messages from each client 2, a distribution device 4, and a plurality of nodes 1 that form a cluster. The load balancer 3 distributes the message from the client 2 to each distribution device 4 by a simple round robin method or the like. The distribution device 4 distributes the received message to each node 1 based on, for example, a consistent hash method. Each node 1 performs message processing and provides a service to the client 2.

  In FIG. 1, the distribution device 4 and the node 1 are described as separate devices, but can be operated as separate functions on the same server. The distribution device 4 can also take a cluster configuration as shown in FIG. Further, the load balancer 3 does not exist, and a message can be transmitted from the client 2 to an arbitrary distribution device 4.

≪Process outline≫
The distributed processing system 1000 according to the present embodiment performs server placement on an ID space in consideration of physical placement of nodes (position information such as areas) in the distributed processing system of the comparative example described above. The following processing is executed in preparation for the occurrence of a large-scale failure due to a scale disaster or the like.

  First, in the distributed processing system 1000 according to the present embodiment, information indicating the probability that a server at each site will fail due to a large-scale disaster that is predicted to occur in the future (hereinafter referred to as “failure probability information”) is an external device. Receive from etc. Then, a server whose failure probability exceeds a predetermined threshold (a “relief server” (relief node) described later) is extracted (see symbol a in FIG. 2). Next, a server holding duplicate data corresponding to the original data held by the extracted server (a server located in a different region) is searched, and a server having the lowest failure probability among the servers holding the duplicate data is replaced. It is determined as a destination server (exchange destination node) (see symbol b in FIG. 2). Then, the role of the original and the copy is exchanged between the server (replacement destination server) that holds the determined copy data and the server (relief server) that holds the original data (see symbol c in FIG. 2). That is, the replicated data is promoted to the original data, and the server (exchange destination server) that holds the replicated data is in charge of processing the data (processing for the original data). On the other hand, the server (relief server) that has demoted the original data to replicated data and retained the original data does not process the data (processing for the original data) after the role change, and does not process the retained original data. Process as duplicate data.

In FIG. 2, two replicated data corresponding to the original data “Original (indicated as“ O ”in FIG. 2)” held by the server (relief server) extracted as the failure probability exceeds a predetermined threshold value. Of “Replica” (described as “R ₁ ” and “R ₂ ” in FIG. 2) ”, an example in which the server holding the replicated data of“ R ₁ ”as the server with the lowest failure probability is determined as the replacement server Is shown. Then, the role of the original and the copy is exchanged between the server holding the original data “O” (relief server) and the server holding the duplicate data “R ₁ ” (exchange destination server).

In this way, in the distributed processing system 1000 including the node 1 according to the present embodiment, the role of the original and the copy is appropriately set between nodes in different regions before a large-scale failure due to a large-scale disaster or the like occurs. Therefore, even if a large-scale failure occurs thereafter, the processing can be continued without losing the original data. In addition, since only the roles of the original and the copy are exchanged, there is no need to copy or relocate the data. Therefore, in the event of a large-scale failure, an increase in network costs and a delay in system reconfiguration The original data can be relieved after the suppression.
Hereinafter, a specific configuration and processing of the node 1 according to the present embodiment will be described.

<Node>
Next, the node 1 constituting the distributed processing system 1000 according to the present embodiment will be specifically described. Note that the node 1 according to the present embodiment is a privileged node that manages a node management table 100 (node management information) to be described later among the plurality of nodes 1 of the distributed processing system 1000, and the node management table from the privileged node. There is a case where it is not a privileged node that receives the information of 100 and updates its own node management table 100. The processing performed by the privileged node will be described later.

FIG. 3 is a functional block diagram illustrating a configuration example of the node 1 according to the present embodiment.
As shown in FIG. 1, the node 1 is communicably connected to each sorting device 4 and is also communicably connected to other nodes 1 other than itself constituting the cluster. Then, it receives a message from the client 2 and provides a service. Further, the node 1 transmits the information held as the original data to the other node 1 according to the preset redundancy number, thereby causing the other node 1 to hold the duplicate data.
Note that, as an example of the data management method of the node 1 according to the present embodiment, an example in which the data distribution destination and the replication destination are determined based on the consistent hash method will be described below. However, the present invention is limited to the consistent hash method. However, the present invention is applicable to any system in which a plurality of nodes 1 are configured in a cluster and the original data and the corresponding replicated data are distributed and processed.
As illustrated in FIG. 3, the node 1 includes a control unit 10, an input / output unit 11, and a storage unit 12.

  The input / output unit 11 inputs and outputs information to and from the distribution device 4 and other nodes 1 other than itself. The input / output unit 11 performs input / output between a communication interface (not shown) that transmits and receives information via a communication line and an input means such as a keyboard (not shown) and an output means such as a monitor. And an output interface.

  The storage unit 12 includes storage means such as a hard disk, a flash memory, and a RAM (Random Access Memory), and original data and duplicated data (both not shown) to be processed, a node management table 100 (see FIG. 4), A life / death monitoring table 200 (see FIG. 5), failure probability information 300 (see FIG. 6), and the like are stored. Details of the node management table 100, the alive monitoring table 200, and the failure probability information 300 will be described later. In addition, the storage unit 12 stores the value of each parameter (M: number of redundancy, X: number of data set to be simultaneously lost due to a large-scale failure among M, a predetermined threshold used for determination to be described later). Is done.

The control unit 10 controls the entire node 1 and, as shown in FIG. 3, a node management unit 101, a node arrangement determination unit 102, a message processing unit 103, a data replication processing unit 104, an alive monitoring unit 105, and a data rescue unit 106 is comprised. Among these, the data rescue unit 106 further includes functions of a failure information receiving unit 107, a data search unit 108, and an original / duplicate exchange processing unit 109.
The control unit 10 is realized by, for example, a CPU (Central Processing Unit) developing and executing a program stored in the storage unit 12 in a RAM.

The node management unit 101 includes identification information regarding each node 1 constituting the cluster, information on data in charge, information regarding physical location (location information such as data center area), original / replication exchange information (details will be described later), It is managed as a node management table 100 (node management information: see FIG. 4). When the node management unit 101 is a privileged node, the node management unit 101 receives information on addition or removal of the node 1 from the cluster and updates the information in the node management table 100 held by the node management unit 101. Then, the node management unit 101 transmits the updated update information of the node management table 100 to other nodes 1 and the distribution device 4 other than itself in the cluster.
Further, when the node management unit 101 is not a privileged node, the node management unit 101 receives information (update information) of the node management table 100 from the privileged node, and updates the node management table 100 held by itself.
In this way, each node 1 and each distribution device 4 in the cluster always have the node management table 100 having the same contents.

  Note that when the node management table 100 is updated, that is, when the node 1 is added or removed, the node management unit 101 relocates the original data and the duplicated data to the data replication processing unit 104. Outputs data replication instructions for execution.

FIG. 4 is a diagram showing a data configuration example of the node management table 100 (node management information) according to the present embodiment. As shown in FIG. 4, the node management table 100 is configured to include a node identifier 110 of each node 1 constituting the cluster, responsible data 120, a server name (address) 130, a region ID 140, and original / replica exchange information 150. The
FIG. 4A shows a state in which data relief processing (details will be described later) by the data relief unit 106 is not executed. Specifically, no information is stored in the data item of the original / copy exchange information 150. On the other hand, FIG. 4B shows a state where the data relief process by the data relief unit 106 is being executed. Specifically, “information indicating that the roles of the original and the copy have been exchanged” is stored in the data item of the original / copy exchange information 150.
Here, the node management table 100 in the state of FIG. 4A will be described, and the node management table 100 in the state of FIG. 4B will be described later.

The node identifier 110 corresponds to the node ID on the ID space of the consistent hash method. Further, when a virtual ID is used in the consistent hash method, the node identifier 110 is assigned for each virtual ID and registered in the node management table 100. In the node management table 100, for example, IDs (or virtual IDs) in the ID space of the consistent hash are arranged and managed in ascending order. That is, in the node management table 100, when the node identifier 110 (node ID) is arranged in ascending order, the node 1 of the next row of the node 1 of its own is next to the right in the ID space (next clockwise). Node 1.
For example, in FIG. 4, for data whose data identifier based on the ID space of the consistent hash is “0” to “56”, the node (node identifier “56”, server name shown in the first row of FIG. The node “server A”) is in charge of processing related to the data (including storage and update of original data, data extraction, etc.). Similarly, for data whose data identifier is “57” to “172” obtained by adding 1 to “56”, the node shown in the second row (the node having the node identifier “172” and the server name “server B”) ) Is in charge of processing related to the data. The same applies hereinafter.

In charge data 120, an ID of data handled by the server as original data is stored. As described above, the data IDs of data in which the node shown in the first line (the node having the node identifier “56” and the server name “server A”) is handled as original data are stored as “0” to “56”. The In FIG. 4, data IDs “0” to “56” and data managed by “server A” as original data are collectively expressed as “data a”.
Similarly, “data b” (data IDs “57” to “172”) is the data that the node shown in the second row (the node having the node identifier “172” and the server name “server B”) takes charge of as the original data. ") Is stored. The same applies hereinafter.

  The server name (address) 130 represents the identifier of each node 1 constituting the cluster. The server name 130 is stored in association with each node 1 address (for example, IP address).

The region ID 140 represents an identifier of a base (K location) that is information relating to a physical position (position information such as a data center area). For example, the area ID 140 “00” represents “base α (Kyushu area)”, the area ID 140 “01” represents “base β (Kansai area)”, and the area ID 140 “02” represents “base γ ( Kanto area) ”.
The original / duplicate exchange information (exchange destination server) 150 will be described later, but no information is stored in a state where the data relief processing by the data relief unit 106 is not executed.

Note that the node identifier 110 of the node management table 100 can be given to each node 1 by the node management unit 101 of the privileged node, or an external device (for example, a network management device or the like) It is also possible to receive and store the node identifier 110 assigned in the above. In addition, a privileged node is not provided, and a management apparatus for the distributed processing system 1000 is provided. The management apparatus manages the detachment and addition (participation) of the node 1 with respect to the node management table 100, and the updated node management table 100 is stored in each You may make it deliver to the node 1. FIG.
Furthermore, it is also possible to add other additional information (for example, the date and time of joining each node 1 to the cluster) necessary for processing to the node management table 100.

When the node management unit 101 is a privileged node, the node management unit 101 receives information about the detachment of the node 1 from its own alive monitoring unit 105, another node 1 that is not a privileged node, or an external device (such as a network management device). In this case, in the node management table 100, a record including information on the node 1 to be detached (node identifier 110, responsible data 120, server name (address) 130, area ID 140, and original / replica exchange information 150) is deleted.
In addition, when the node management unit 101 is a privileged node, the node management table 100 adds information on the node 1 to be newly added (node identifier 110, responsible data 120, server name (address) 130, region ID 140, and The record including the original / duplicate exchange information 150) is inserted at the position determined by the node arrangement determining unit 102.

When the node allocation determining unit 102 is a privileged node, the node allocation determining unit 102 adds (participates) node 1 from its own alive monitoring unit 105, another node 1 that is not a privileged node, or an external device (such as a network management device). When information is received and a new node is to be added to the ID space (node management table 100), the following node insertion position determination process is performed. The node insertion position determination process is the same process as the process described with reference to FIGS.
The node arrangement determination unit 102 includes a base (data center area) to which MX or more nodes out of M-1 nodes around the node ID to which a new node is added belong (the data center area). Arrangement is determined so as to be different bases (data center areas). That is, the node arrangement determining unit 102 determines the insertion position of the ID space such that the base of the M−X nodes and the base of the node to be added are different bases as the insertion position of the newly added node. . Then, the node arrangement determination unit 102 causes the node management unit 101 to insert information on the newly added node 1 at the determined insertion position of the node management table 100.
By repeating the arrangement based on the node insertion position determining process executed by the node arrangement determining unit 102 in the cluster, the number of cases where nodes at the same site are adjacent in the ID space is reduced, and there is a bias in duplicate data or when the node leaves The generation of the data acquisition process can be suppressed.

The message processing unit 103 provides the service by receiving the message distributed from the distribution device 4, executing the processing of the message, and returning the processing result to the client 2. In addition, when the node 1 itself does not hold data necessary for processing the message, the message processing unit 103 determines that another node 1 (node 1 in the next row of the node management table 100, further The data can be acquired by making a request to the node 1 in the next row.
In addition, when the original message data is newly stored or updated by processing of the received message, the message processing unit 103 duplicates the newly stored original data or the updated original data to copy the original data. A data duplication instruction is output to the data duplication processing unit 104 so as to be stored as data in another node.

  Further, when the message processing unit 103 receives the message, the message processing unit 103 refers to the node management table 100 and stores “information indicating that the roles of the original and the replica have been exchanged” in the original / replica exchange information 150. In this case, depending on the information, the process as the node 1 having the role of holding the original data or the process as the node 1 having the role of holding the duplicate data is executed as an exchanged role. Details will be described later.

  The data replication processing unit 104 refers to the node management table 100 for the data (original data) held by itself, and refers to the node in the next (lower) row of its own record (that is, the node on the right in the ID space). ) Is selected as a node for holding duplicate data, a duplicate of the original data is generated and transmitted as duplicate data. When the redundancy number is “M”, the data replication processing unit 104 refers to the node management table 100 and, as described above, the node in the next row of its own record and the node in the next row ( In the ID space, “M−1” nodes that hold the duplicate data are determined in the order of the right neighbor and then the right neighbor), and the duplicate data is transmitted.

  The data replication processing unit 104 indicates the role of the original when “information indicating that the roles of the original and the replica have been exchanged” is stored in the original / replication exchange information 150 of the node management table 100. When the information is stored, it transmits the replicated data to the server stored as the rescue server, and the server that has transmitted the replicated data to the server other than itself, The duplicate data is transmitted. In other words, instead of the server that holds the original data, a copy data transmission process for holding the data redundancy is executed.

  The life and death monitoring unit 105 refers to the life and death monitoring table 200 (FIG. 5 to be described later) and exchanges life and death monitoring signals at a predetermined time interval with a designated node 1 (for example, a node in the next row of itself). The failure of the node 1 constituting the cluster is detected. When a failure of the node 1 is detected, the alive monitoring unit 105 notifies the node management unit 101 when the node itself is a privileged node and notifies the privileged node when the node itself is not a privileged node (failure occurrence notification). If a privileged node is not provided and a management device that manages the node management table 100 is provided, the alive monitoring unit 105 of each node 1 transmits a failure occurrence notification to the management device.

FIG. 5 is a diagram illustrating a data configuration example of the life and death monitoring table 200 according to the present embodiment.
The life and death monitoring table 200 is created in units of one physical device, and lists nodes 1 (servers) to be monitored. In the alive monitoring table 200, for example, a server name and an address (IP address) associated with the server name are stored.

  The alive monitoring table 200 is set so that a physical device that constructs a logical device becomes a monitoring target at least once in consideration of a pattern in which nodes are configured in units of logical devices (virtual nodes). In addition, when there is an addition or withdrawal to the node 1 constituting the cluster, it is assumed that the node management table 100 is updated synchronously. Therefore, when the node identifier 110 of the node management table 100 is not based on a virtual ID configured in units of logical devices but is an ID in units of physical devices, the alive monitoring table 200 and the node management table 100 are the same. May be used. In this case, the alive monitoring unit 105 may perform the alive monitoring of each node 1 using the node management table 100 without generating the alive monitoring table 200.

Here, processing for determining a privileged node from among a plurality of nodes 1 in the cluster will be described.
As described above, when each node 1 stores the date and time of participation in the cluster of each node 1 as additional information in the node management table 100, the privileged nodes are selected in descending order of the participation date and time. It may be. Each node 1 may be set to be a privileged node in order from the top row of the alive monitoring table 200 with reference to the alive monitoring table 200.
When node 1 newly becomes a privileged node, information indicating that it is a privileged node is transmitted to each node 1 or the like. Then, when the node 1 in the cluster has left or added (participated), the privileged node updates its own node management table 100 and distributes the update information to each node 1, the distribution device 4, and the like.

Returning to FIG. 3, the data rescue unit 106 will be described.
The data rescue unit 106 executes “data rescue processing” that prevents original data from being lost due to a large-scale failure caused by a large-scale disaster or the like. In this data relief process, the data relief unit 106 receives information (failure probability information) indicating the probability that a server at each site will fail due to a large-scale disaster that is predicted to occur in the future from an external device (not shown). Then, a server (relief server (relief node)) whose failure probability exceeds a predetermined threshold is extracted. Then, the data rescue unit 106 searches for a server holding duplicate data corresponding to the original data held by the extracted server (relief server), and selects a server with the lowest failure probability from among them as a replacement server (replacement node) ). The data rescue unit 106 exchanges the role of processing the original data and the replicated data for the server (replacement destination server) that retains the determined replicated data and the server (rescue server) that retains the original data.
The data rescue unit 106 includes a failure information reception unit 107, a data search unit 108, and an original / duplicate exchange processing unit 109.

  The failure information reception unit 107 receives the failure probability information 300 from an external device or the like via the input / output unit 11. The failure information reception unit 107 stores the failure probability information 300 in the storage unit 12. When the failure information reception unit 107 receives the failure probability information 300 at predetermined time intervals, the failure information reception unit 107 updates the failure probability information 300 stored in the storage unit 12 to the latest information. Further, the failure information reception unit 107 receives failure probability information 300 that is received irregularly, for example, failure probability information 300 that is calculated in association with a sudden emergency earthquake warning, and the failure probability that is stored in the storage unit 12. The information 300 may be updated.

FIG. 6 is a diagram illustrating a data configuration example of the failure probability information 300 according to the present embodiment.
The failure probability information 300 is information indicating the probability of failure of each node 1 (server) at each site due to a large-scale disaster or the like that is predicted to occur in the future (probability of failure). Large-scale disasters include, for example, typhoons, storms, tornadoes, lightning strikes, heavy rains, river floods, tsunamis, earthquakes, etc., and in the areas where the bases are located several days or seconds to hours later, Information that indicates the predicted probability that the server will become unusable due to physical damage (including network disconnection, etc.), power outages, or failures such as the server administrator not getting close to the server installation facility It is.

The failure probability information 300 includes data items of area ID 310, server name 320, time 330, and failure probability 340.
The region ID 310 is the same information as the region ID 140 of the node management table 100 shown in FIG. The server name 320 is the same information as the server name 130 in the node management table 100 shown in FIG.
The time 330 and the failure probability 340 are information stored in association with the area ID 310 and the server name 320. The failure probability 340 stores the failure probability (%) of the server at time 330 (predetermined time). FIG. 6 shows an example in which the time 330 is set every hour.
For example, as shown in the first line of the failure probability information 300, “Server A” of the server name 320 whose area ID 310 is “00 (base α)” is “June 1, 2015” ", The failure probability 340 between 13:00 and 14:00 is" 10 (%) ", and the failure probability 340 between 14:00 and 15:00 is" 30 (%) ".

Returning to FIG. 3, the data search unit 108 refers to the received failure probability information 300 for each server holding the original data, for example, when the failure information reception unit 107 receives the failure probability information 300. Based on the current time, the latest failure probability is extracted, and it is determined whether or not a predetermined threshold is exceeded.
In the example of the failure probability information 300 shown in FIG. 6, when the current time is 12:50 on June 1, 2015, the failure probability 340 is referred to by referring to “20150601 (13: 00-)” at time 330. "10 (%)" is extracted. Then, the data search unit 108 determines whether or not a predetermined threshold value (for example, 20 (%)) is exceeded. Here, the data search unit 108 determines that the predetermined threshold value is not exceeded.
Further, when the current time is 13:50 on June 1, 2015, the data search unit 108 refers to “20150601 (14: 00-)” at time 330 and sets “30 ( %) ”. Then, the data search unit 108 determines whether or not a preset threshold value (for example, 20 (%)) is exceeded. Here, the data search unit 108 determines that the predetermined threshold value is exceeded.

  When the data search unit 108 determines that there is a server (relief server) exceeding a predetermined threshold, the data search unit 108 uses the original data held by the server (relief server) as the assigned data 120 of the node management table 100 (FIG. 4). Extract by reference. Further, the data search unit 108 searches for a server that holds duplicate data of the original data held by the server (relief server). Then, the data search unit 108 refers to the failure probability information 340 at the same time of the server holding the searched replicated data in the failure probability information 300, and determines the server with the lowest failure probability as the replacement server.

The original / duplicate exchange processing unit 109 selects a server (relief server) that the data search unit 108 determines to exceed a predetermined threshold and a server that holds duplicate data of original data held by the server (relief server). Based on the determined information of the exchange destination server, a process for exchanging the roles of the original and the copy is executed.
Specifically, the original / duplicate exchange processing unit 109 handles the original data as duplicated data for the server designated as the rescue server in the column of the original / duplicate exchange information 150 in the node management table 100 (see FIG. 4). Information to that effect (in FIG. 4B, “original copy → replication”) is stored. At this time, the original / copy exchange processing unit 109 stores the identification information of the exchange destination server in association with each other (“exchange destination server: server B” in FIG. 4B).
Further, the original / duplicate exchange processing unit 109 has information indicating that the replicated data is handled as original data for the server determined as the exchange destination server in the original / duplicate exchange information 150 column of the node management table 100 (FIG. 4). In (b), “replication → original”) is stored. At this time, the original / copy exchange processing unit 109 stores the relief server identification information in association with each other (“relief server: server A” in FIG. 4B).

  In this way, in the node management table 100, information of the server whose role of the original and the copy is exchanged is stored. When receiving a new message, the message processing unit 103 of the node 1 refers to the original / replica exchange information 150 of the node management table 100 for the server that stores the original data of the data, and plays the role of the original and the replica. It is confirmed whether or not the information indicating the exchange is set. If the server's role is “original → replicated” for the data to be processed as original data by its own server, it is determined that the request is not processed as the original data but is processed as the replicated data. . On the other hand, the message processing unit 103 refers to the original / copy exchange information 150 of the node management table 100 for the data, and exchanges the role of the original and the copy for the data that is not the data that the server itself should process as the original data. If the information indicating that the server has been set is confirmed, the role of the server is “original → replicated”, and its own server is set as the exchange destination server, The data is processed as original data.

  At this time, the data replication processing unit 104 confirms the rescue server identification information of the original / copy exchange information 150 stored in the record of its own server in the node management table 100, and Data updated by message processing (replicated data) is transmitted. Further, the data replication processing unit 104 extracts the server to which the rescue server has transmitted the replicated data with reference to the node management table 100, and is updated for servers other than the extracted server. The process of transmitting the duplicated data is executed. By doing so, original data and duplicate data are not rearranged, and the redundancy of the entire system can be maintained.

  In addition, the data retrieval unit 108 performs a predetermined process when the next data relief process is executed for a server that has been determined to be a relief server in the previous process after a time has elapsed since the data relief process was executed. If it is determined that the value is equal to or less than the threshold, the information indicating that the role of the original and the copy stored in the original / copy exchange information 150 of the node management table 100 has been exchanged is deleted via the original / copy exchange processing unit 109. To do. Thereby, the roles of the original and the copy can be easily returned to the original state.

<Process flow>
Next, data relief processing executed by the node 1 according to the present embodiment will be described with reference to FIG. FIG. 7 is a flowchart showing the flow of data relief processing executed by the node 1 according to this embodiment.

  First, the failure information reception unit 107 of the node 1 receives failure probability information 300 from an external device or the like (step S10). Then, the failure information receiving unit 107 stores the received failure probability information 300 in the storage unit 12.

  Subsequently, the data search unit 108 of the node 1 refers to the received failure probability information 300 for each server holding the original data, triggered by the failure information reception unit 107 receiving the failure probability information 300, etc. For example, the most recent failure probability is extracted based on the current time, and it is determined whether or not a predetermined threshold (for example, 20 (%)) is exceeded (step S11).

If the data search unit 108 determines that the server holding the original data does not exceed the predetermined threshold (step S11 → No), the data search unit 108 ends the process. On the other hand, when the data search unit 108 determines that the server holding the original data exceeds a predetermined threshold (step S11 → Yes), the data search unit 108 determines that the server needs a relief of the original data (relief server) ), The original data (identifier) held by the server (relief server) is extracted with reference to the data 120 in charge in the node management table 100 (FIG. 4) (step S12).
Note that the processing from step S12 to step S16 is processing executed for each server determined as the rescue server.

  Subsequently, the data search unit 108 searches for a server that holds duplicate data of the original data held by the server (relief server) with reference to the node management table 100 (step S13). Specifically, the data search unit 108 uses the node management table 100 (see FIG. 4A) as a reference to the next “M−1” rows based on the row where the server (relief server) is located. The node described in (1) is searched as a server that holds duplicate data.

  Next, the data search unit 108 refers to the failure probability information 340 at the same time of the server holding the searched replicated data in the failure probability information 300, and determines the server with the lowest failure probability (step S14). Note that when there are a plurality of servers having the lowest failure probability, the data search unit 108 may determine, for example, at random from among them, or the server ( It may be determined to be a server adjacent to the ID space.

  Then, the data search unit 108 determines whether or not the failure probability of the server determined in step S14 exceeds the predetermined threshold (for example, 20 (%)) (step S15). Here, if the determined failure probability of the server exceeds a predetermined threshold value (step S15 → Yes), the process is terminated. On the other hand, if the failure probability of the determined server does not exceed the predetermined threshold value (step S15 → No), the data search unit 108 sets the server determined in step S14 as the replacement server, and proceeds to the next step S16.

In step S16, the original / copy exchange processing unit 109 of the node 1 copies the server (relief server) that the data search unit 108 has determined to exceed the threshold value in step S11 and the original data held by the server (relief server). Based on the information of the exchange destination server determined from among the servers holding the data (the server determined in step S14), a process of exchanging the roles of the original and the copy is executed.
Specifically, as shown in FIG. 4B, the original / duplicate exchange processing unit 109 includes a server designated as a rescue server in the column of original / duplicate exchange information 150 in the node management table 100. Information that the original data is handled as duplicate data (in FIG. 4B, “original → duplicate”) is stored. At this time, the original / copy exchange processing unit 109 stores the identification information of the exchange destination server in association with each other.
Further, the original / duplicate exchange processing unit 109 has information indicating that the replicated data is handled as original data for the server determined as the exchange destination server in the original / duplicate exchange information 150 column of the node management table 100 (FIG. 4). In (b), “replication → original”) is stored. At this time, the original / copy exchange processing unit 109 stores the identification information of the rescue server in association with each other.
In this way, the process is executed for all the nodes determined as the repair server, and the data repair process by the node 1 is ended.

Note that if the data search unit 108 of the node 1 determines in step S11 that the server holding the original data exceeds a predetermined threshold (step S11 → Yes), the failure probability information 300 is determined from the current time. When the time up to the time set at time 330 (time to reach the corresponding failure occurrence probability) is longer than the time required for exchanging the roles of the original and the copy (time for executing step S16) Alternatively, the server exceeding the predetermined threshold may be used as a relief server, and the processes after step S12 may be executed. Here, the time required for exchanging the roles of the original and the copy is set in advance based on, for example, the average of the actual measurement values so far.
In the above description of the processing flow, the predetermined threshold value in step S15 has been described as the same threshold value used when determining the server that is the target of the original data relief in step S11. However, the threshold values in step S11 and step S15 do not have to be the same, and arbitrary values may be set separately.

  As described above, according to the node 1, the data relief method, and the program according to the present embodiment, the role of the original and the copy between nodes in different regions is changed before a large-scale failure due to a large-scale disaster or the like occurs. Therefore, even if a large-scale failure occurs thereafter, the processing can be continued without losing the original data. In addition, since only the roles of the original and the copy are exchanged, there is no need to copy or relocate the data. Therefore, in the event of a large-scale failure, an increase in network costs and a delay in system reconfiguration The original data can be relieved after the suppression.

  As described above, the node data management method according to the present embodiment is not limited to the consistent hash method, and a plurality of nodes are clustered to distribute the original data and the corresponding replicated data. Thus, any system that processes data can be applied. For example, each node includes a list of data arranged on the server at each site. In the list of data, the data arranged in each server is given identification information indicating whether it is original data or duplicated data, and the original / replica exchange information shown in FIG. 150 is attached. In this way, each node can process the data after referring to the information indicating that the role of the original and the copy has been exchanged.

1 Node 2 Client 3 Load Balancer 4 Distribution Device 10 Control Unit 11 Input / Output Unit 12 Storage Unit 100 Node Management Table (Node Management Information)
DESCRIPTION OF SYMBOLS 101 Node management part 102 Node arrangement | positioning determination part 103 Message processing part 104 Data replication processing part 105 Life / death monitoring part 106 Data rescue part 107 Failure information reception part 108 Data search part 109 Original / replication exchange processing part 200 Life / death monitoring table 300 Failure probability information 1000 Distributed processing system

Claims (3)

Each of a plurality of nodes constituting the cluster is the node of the distributed processing system that holds data in charge of processing as original data or replicated data that is a duplicate of the original data,
Corresponding to the location information indicating the area where each of the nodes is physically installed and the identifier of the node installed in the area, the data each node is responsible for processing is the original data, or A storage unit for storing node management information stored as the duplicated data arranged in a node installed in a different area from the node holding the original data;
Receiving failure probability information indicating failure probability at a predetermined time for each of the nodes associated with the location information,
For the node holding the original data, refer to the failure probability information, extract a node having the failure probability exceeding a predetermined threshold as a relief node, and copy data corresponding to the original data held by the extracted relief node A node to be held is detected by referring to the node management information, and among the nodes holding the detected duplicate data, the failure probability information is referred to, and the node having the lowest failure probability is determined as a replacement destination node. ,
A data relief unit for exchanging a function for processing the original data and a function for processing the duplicate data between the relief node and the exchange destination node;
A node characterized by comprising: Each of a plurality of nodes constituting a cluster is a data recovery method for the nodes in a distributed processing system that holds data in charge of processing as original data or replicated data that is a copy of the original data,
The node is
Corresponding to the location information indicating the area where each of the nodes is physically installed and the identifier of the node installed in the area, the data each node is responsible for processing is the original data, or A node for storing node management information stored as the duplicate data arranged in a node installed in a different area from the node holding the original data;
Receiving failure probability information indicating failure probability at a predetermined time for each of the nodes associated with the position information;
For the node holding the original data, refer to the failure probability information, extract a node having the failure probability exceeding a predetermined threshold as a relief node, and copy data corresponding to the original data held by the extracted relief node A node to be held is detected by referring to the node management information, and a node having the lowest failure probability is determined as a replacement destination node by referring to the failure probability information among nodes holding the detected duplicate data. Steps,
Exchanging a function for processing the original data and a function for processing the duplicated data between the rescue node and the exchange destination node;
A data relief method comprising:   A program for causing a computer to execute the data rescue method according to claim 2.

Images