JPH07311748A - Fault recovery system of decentralized data base system - Google Patents

Abstract

PURPOSE:To provide a fault recovery system of the decentralized data base which minimizes the time required for fault recovery even when the decentralized data base is large in scale and complex, decreases the number of bypasses evading a recovery failure due to a communication path fault, and further enables the fault recovery state of the whole decentralized data base to be confirmed at each node. CONSTITUTION:Plural nodes 11 which constitute the decentralized data base are divided logically into recovery node groups 13. A recovery control node 15 which is positioned above the respective nodes 11 and controls the fault recovery of the recovery node group 13 is connected to the respective nodes 11 by communication lines 14 which exchange recovery information and determination information. Plural recovery control nodes 15 are divided logically into recovery node groups 17 and connected to recovery control nodes positioned one layer above through communication lines 18 to form a hierarchical structure, and the respective recovery control nodes control the fault recovery of nodes positioned below.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention, in a large-scale distributed database system composed of network nodes connected by a communication network or parallel processors loosely connected, causes a system failure including a failure of a communication path or a node. In this case, the present invention relates to a failure recovery method that quickly recovers the failure of the entire system.

[0002]

2. Description of the Related Art FIG. 10 shows, for example, Japanese Patent Laid-Open No. 1-19404.
It is a block diagram of the distributed database shown by 0 publication.
Processors 3a and 3b of a client node and a server node are coupled to the communication network 4, respectively.
Each node has a database management system 2a,
2b, databases 5a and 5b, history information 6a and 6b, and transaction status management tables 1a and 1b.

In the conventional recovery method of a distributed database system, the end status of each transaction is held in the transaction status management tables 1a and 1b, and at the time of failure recovery, the transaction status management table 1a,
Restore 1b. After the communication is restarted, messages are exchanged between the nodes based on the contents of the transaction status management tables 1a and 1b to recover the history information 6a and 6b of the nodes.

[0004]

However, in the conventional recovery method of the distributed database system, since message exchange is performed between the two nodes that directly accessed the database at the time of failure, system failure occurs in a large-scale distributed database. If the number of nodes accessing the database is large, the amount of message transfer increases accordingly, and there is a problem that the time until the failure recovery processing of the entire system ends increases.

Looking at the time required for failure recovery in terms of the number of message exchanges, when the number of nodes forming a distributed database is n in the past, at least 2n times when all pairs of nodes being accessed are different. , N (n-1) when at most all nodes are accessing other nodes
The number of message exchanges is required. Therefore, there is a problem that the larger the number of complicated accesses, the greater the number of message exchanges.

Furthermore, since the time required for recovery depends on both the number of nodes and the complexity of the distributed database access, the fluctuation range of the time required is large, and the recovery time is required until the recovery process is completed. There was a problem that it could not be specified.

Further, in the conventional method for performing direct database access between two nodes,
If there is a failure in the communication path between the two nodes, a detour for avoiding the failure must be provided between all the nodes that may access the distributed database. Therefore, there is a problem that a huge number of detours of n (n-1) / 2 is required to completely avoid the communication failure via the detour.

Further, since the conventional failure recovery method is carried out by exchanging messages between two nodes that directly access the database at the time of failure, recovery between the parties who directly access the database can be confirmed. However, there was no way to check the status of the entire distributed database, such as whether recovery was completed for the entire distributed database, or whether there was an inaccessible node because recovery was not completed.

The present invention has been made in order to solve such a problem, and even in a large-scale and complicated distributed database, the time required for failure recovery is minimized, and the bypass for avoiding the unrecoverability due to the communication path failure is avoided. Reduce the number of paths,
It is another object of the present invention to provide a distributed database failure recovery method in which the failure recovery status of the entire distributed database can be confirmed at each node.

[0010]

In order to achieve the above object, the invention according to claim 1 is such that a plurality of nodes each having a processor and a database are connected by a network, and a process in each node is a database of another node. In a distributed database system that accesses a node, a plurality of nodes are logically divided into groups, and a recovery control node that controls the failure recovery is provided for each group to perform failure recovery control of each node in a tree structure. It is characterized by

According to a second aspect of the present invention, in the disaster recovery method for a distributed database system according to the first aspect, a plurality of the recovery control nodes are logically divided into groups, and failure recovery control is performed for each of the groups. By providing a higher recovery control node to perform, failure recovery control of each node is performed hierarchically.

According to a third aspect of the present invention, in the failure recovery method of the distributed database system according to the second aspect, a hierarchy is formed until the number of the upper recovery control nodes finally becomes one.

The invention according to a fourth aspect is characterized in that, in the failure recovery method of the distributed database system according to the first or second aspect, a detour is provided between nodes located at a higher level and a lower level of the tree structure.

According to a fifth aspect of the present invention, in the failure recovery method of the distributed database system according to the first to fourth aspects, the recovery control node sends recovery information for establishing a state of an indefinite transaction to a communication partner. It is characterized by having a mechanism for inquiring to the node when it cannot be obtained from the node.

According to a sixth aspect of the present invention, in the disaster recovery method of the distributed database system according to the first to fifth aspects, the recovery control node has a mechanism for notifying a lower level node of a failure recovery state. Characterize.

[0016]

In the distributed database failure recovery method having the above-mentioned configuration, when the system failure of the distributed database occurs, the information related to the recovery of the distributed database accessed by each node is logically located above the node. Send to control node. The upper level recovery control node notifies each node in the group of the result of a failure that can be resolved in the group that controls failure recovery. On the other hand, regarding a failure that cannot be resolved within the group, information regarding recovery is transmitted to a higher-level recovery control node. By performing this hierarchically up to the highest-level recovery control node, all recovery information of uncertain transactions is collected and the transaction status is confirmed. The state of the confirmed transaction is notified to each node by tracing back the hierarchy, and the faulty node recovers.

Further, even when the communication path between the nodes located at the upper and lower positions fails and the message cannot be exchanged via the communication path, the detour is provided between the nodes. Messages such as recovery information reports and confirmation information can be exchanged.

Further, even when the upper recovery control node cannot obtain the recovery information for establishing the state of the indefinite transaction from the node of the communication partner, it sends a message inquiring the transaction state to the node. As a result, a response message indicating the transaction status can be obtained, so that the status of an indefinite transaction can be fixed and recovery can be performed.

Further, at the time of full or partial recovery from a failure, the upper recovery control node notifies the lower nodes constituting the distributed database of the recovery status of the distributed database system, and the service restart of the distributed database is used. Can inform others.

[0020]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described below with reference to the drawings.

Example 1. FIG. 1 is a conceptual diagram of a distributed database for explaining the gist of the first embodiment. A plurality of nodes 11 having a processor and a database 12 and forming a distributed database are logically divided into recovery node groups 13. The recovery control node 15 having a processor and a database 16 positioned above each node 11 controls failure recovery of the recovery node group 13. The recovery control node 15 is logically positioned higher in the tree structure than the node 11, but may be one of the nodes 11 included in the recovery node group 13. The recovery control node 15 and each node 11 included in the recovery node group 13 are connected by a communication path 14 that exchanges information regarding recovery.

Further, the plurality of recovery control nodes 15 logically recover the recovery node group 1 like the node 11 described above.
Each recovery control node 15 is divided into seven, and each recovery control node 15 is controlled by the recovery control node located one layer higher than the other through the communication path 18. In this way, a hierarchy is formed until the number of upper recovery control nodes becomes one.

The characteristic feature of this embodiment is that when a system failure occurs in the distributed database, the information related to the recovery of the distributed database accessed by each node 11 is transmitted to the recovery control node 15 located at a higher level, For a failure that can be resolved in the recovery node group 13, 15 notifies each node 11 of the result, and for a failure that cannot be resolved in the recovery node group 13, sends information about the recovery to a higher-level recovery control node. By hierarchically performing the recovery control node at the highest level, all the information related to the recovery of an indefinite transaction is collected and the state of the transaction is confirmed. Furthermore, the state of the confirmed transaction is traced backwards to notify the node concerned and recovery is performed. In this way, by hierarchically controlling the recovery of each node, even if the distributed database system is large in scale, it is possible to minimize the increase in the number of messages exchanged in the system.

FIG. 2 is a block diagram of a distributed database for explaining the first embodiment. The plurality of lowest nodes 21 forming the distributed database are divided into the lowest recovery node group 22. Each node 21 is connected via a communication path 23 to a first layer recovery control node 24 that controls recovery of the lowest layer recovery node group 22. Further, the plurality of recovery control nodes 24 of the first layer are divided into the recovery node group 25 of the first layer. Each of the recovery control nodes 24 of the first layer communicates with the recovery control node 27 of the second layer that controls the recovery of the recovery node group 25 through the communication path 26.
Connected via. The plurality of recovery control nodes 27 in the second layer are divided into recovery node groups 28 in the second layer. Then, each recovery control node 27 of the second hierarchy is the third hierarchy that controls recovery of the recovery node group 28, and the recovery control node 30 located at the highest level in this embodiment.
To the communication path 29.

3, 4 and 5, the node 21 located at the lowest level, the recovery control node 24 located at the middle layer,
27 is a flowchart of a recovery process at the time of a failure in the recovery control node 30 located at 27 and the highest level.
The recovery operation of the embodiment will be described with reference to the drawings.

First, the recovery process at the time of a failure in the lowest node 21 will be described with reference to FIG.

At the time of the system failure of the distributed database, the lowest node 21 determines the
If the own node is accessing the distributed database with another node at the time of failure, in step 32, an identifier that can specify the addresses and transactions of the own node and the partner node for all transactions being accessed, and the recovery protocol are specified. A message including information indicating the transaction status of the own node (hereinafter, referred to as recovery information) indicating any of commit, rollback, and indeterminate based on the communication is transmitted to the recovery control node 24 of the first layer through the communication path 23. In step 33, if there is an indefinite transaction in the own node, step 3
In step 4, the transaction control node 24 waits for notification of transaction confirmation information, and after receiving the confirmation information, the transaction is committed by committing or rolling back the database according to the confirmation information.

Next, with reference to FIG. 4, a recovery process at the time of a failure in the recovery control nodes 24 and 27 located in the intermediate layer will be described.

The recovery control node 24 of the first layer performs the receiving process according to the step 41 and the judgment 42, and the node which becomes the communication partner at the time of the failure of the transaction which received the report of the indeterminate state by the judgment 43 is the recovery control node 24. Or the node exists in the recovery node group 22 in the lowest layer controlled by the recovery control node 24 of the first layer and the state of commit or rollback of the transaction can be confirmed, the node having the transaction in the indeterminate state according to step 46. Notify the confirmed transaction status to. If the communication partner at the time of the failure of each node is not the recovery control node 24 but the recovery node group 22 according to the determination in step 43, the recovery control node 24 determines all the recovery information and recovery control of those nodes in step 44. When the node 24 itself accesses the distributed database with another node that does not belong to the recovery node group 22, a message to which recovery information of the recovery control node 24 itself is added is recovered through the communication path 26 to the upper second hierarchy. It is transmitted to the control node 27. Step 45
In step 2, the transaction control information 27 from the recovery control node 27 of the second upper layer is waited for, and after the confirmation information is received, the node which has reported the transaction in the indetermination state is notified of the confirmed transaction state according to step 46.

In the recovery control node 27 of the second hierarchy, the first
Recovery control node 24 in the hierarchy is recovery control node group 2
The same operation as performed for the second node 21 is performed.
That is, the node with which each node communicates at the time of failure is the recovery control node 27, or the recovery node group 25 located in the lower layer of the recovery control node 27, or the recovery node group in the same layer as the recovery node group 22 in the lower layer. When the state of commit or rollback of the transaction can be confirmed, the confirmed transaction state is notified to the node having the transaction in the indeterminate state according to the processing of step 46. On the other hand, if there is no communication partner node in the lower layer from the recovery control node 27, in step 44, the recovery control node 27 recovers all the nodes located below the recovery control node 27 and the recovery control node 27 itself. The message with the information added is transmitted to the uppermost recovery control node 30 via the communication path 29.

Next, the recovery process at the time of a failure in the recovery control node 30 located at the top will be described with reference to FIG.

In the first embodiment, since the recovery control node 30 is the recovery control node of the highest hierarchy, all recovery node groups are hierarchically provided under the recovery control node 30. Therefore, in steps 51 and 52, the recovery information from all the nodes 27, 24 and 21 belonging to the recovery node groups 28, 25 and 22 in the lower layer is received during a predetermined waiting time. In step 53, if the recovery information is reported during the waiting time, basically all the communication partners of each node are confirmed, the indeterminate state of each transaction is also confirmed, and the transaction in the indeterminate state is reported. Notify the node of the confirmed transaction status.
However, the recovery information of the communication partner may not be obtained due to a failure of the communication paths 23, 26, 29. In that case, the recovery information from the partner node is not recovered in the transaction whose recovery information is reported in step 54. For unreported transactions, the recovery information is retained so that definite information can be sent later when recovery information is obtained. Regarding the confirmed state of the transaction in the indefinite state, in step 55, the confirmation information is transmitted to the node having the transaction in the indefinite state via the communication path 29. The recovery control nodes 24 and 27 located in the middle tier follow the hierarchical path opposite to the one in which the recovery information is reported in steps 45 and 46, and the definite information from the uppermost recovery control node 30 is in an uncertain state. Notify the node that has the transaction. Further, each node having the transaction in the indeterminate state commits the transaction by receiving the committing information in step 34.

In the recovery method according to this embodiment, even if a large-scale access is made to the entire distributed database, when the number of nodes constituting the distributed database is n, it depends on the complexity of the distributed database access. It is possible to always recover by 2n times or less of message exchanges, and in the case of the conventional recovery between two nodes, depending on the complexity of the distributed database access, at least 2n times and at most a quadratic function n of the number of nodes. (N-1)
The number of message exchanges can be greatly reduced compared to the case where the number of times was required.

Example 2. FIG. 6 is a conceptual diagram of a distributed database for explaining the gist of the second embodiment. In addition,
The same elements as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

A characteristic of the second embodiment is that there is a detour between nodes located at the upper and lower levels of the tree structure. In the second embodiment, the bypass communication path 19 directly connected between the node 11 and the recovery control node 15 and the bypass communication path 20 indirectly connected via the other node 11 in the same recovery node group 13. have.

As a result, some communication failure occurs in the communication path 14 with the recovery control node 15 during the recovery operation in the first embodiment, and a message with the recovery control node 15 is transmitted using the communication path 14. Even when the exchange becomes impossible, the recovery control node 15 reports the recovery information and receives the confirmation information by using the direct detour communication path 19 or the indirect detour communication path 20, and the adjacent node 11 Can be done through. Note that the failure recovery control processing can be performed in the same processing procedure as in the first embodiment.

When the number of nodes constituting the distributed database is set to n by such detour setting, the number of detours required to be n (n-1) / 2 in the conventional case of recovery between two nodes is According to the present embodiment, the direct detour communication path 1
Since the number n of 9 is sufficient, the difference becomes clear as the distributed database system becomes large in scale.

In the present embodiment, the node 11 located at the bottom and the recovery control node 1 located above it.
Although the configuration of the detour, which is a feature of this embodiment, has been described with reference to the relationship with No. 5, it is not limited to this, and the detour can be similarly set between other layers or in the recovery node group.

Example 3. FIG. 7 is a configuration diagram of a distributed database for explaining the third embodiment. The same elements as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

A feature of this embodiment is that a detour is provided between the node and a node other than the recovery control node that controls the recovery of a certain node. In the third embodiment, another recovery control node 2 belonging to the same hierarchy as a higher recovery control node 27a of a node, for example, the node 24a.
The bypass communication path 71 is set between the recovery control node 27 and the recovery control node 30 belonging to a higher hierarchy than the recovery control node 27a at the higher level.

Thus, when there is a failure in the communication path 26 between the nodes 24a and 27a, or the recovery control node 27a of the recovery node group 25a to which the node 24a belongs.
Even when the recovery control node is down, a message is exchanged with another recovery control node 27b or the recovery control node 30 via the bypass communication path 71 or the bypass communication path 72, and the recovery control operation is delegated to those other recovery control nodes. By doing so, recovery can be performed. The failure recovery control processing can be performed in the same processing procedure as in the second embodiment.

Further, in this embodiment, the configuration of the detour, which is a feature of this embodiment, is used by using the relationship between the recovery control node 24 located in the first layer and the recovery control nodes 27 and 30 located above it. However, the detour can be similarly set between other layers or in the recovery node group.

Example 4. A characteristic of this embodiment is that in the configuration of FIG. 2 or 7, the highest-level recovery control node 30 provides recovery information for establishing the state of an indefinite transaction with the recovery control node 27 of the communication partner.
It is to have a mechanism to inquire to the node when it cannot be obtained from the node. This makes it possible to later obtain recovery information that was not obtained due to a failure of the communication path 29 during normal processing.

FIG. 8 is a flowchart of the recovery process of the highest-level node in the fourth embodiment, and the inquiry process in this embodiment will be described below with reference to FIG.

The highest recovery control node 30 basically establishes all communication partners of each node in steps 51, 52, and 55 in the same way as the processing shown in FIG. 5, and basically sets the indeterminate state of each transaction. Is also confirmed, and the confirmed transaction state is notified to the node that has reported the transaction in the indeterminate state.

However, the recovery information of the communication partner may not be obtained in some cases due to the failure of the communication path or the delay of the processing of the communication partner.
At, a query message is transmitted to a node for which recovery information cannot be obtained, either directly or via each recovery control node 27 having a tree structure. The node that has received the inquiry message transmits the recovery information, so that the recovery control node 30 can obtain the recovery information from the node for which the recovery information has not been obtained. If there is a node that does not respond to the inquiry, the information is held as an unrecoverable node.

As a result, except when a failure of the communication path including the detour or a down of the communication partner node occurs,
If the processing is simply delayed due to the processing delay of the communication partner, the delay can be minimized to obtain the recovery information of the entire distributed database system, and the failure recovery processing is completed in the minimum time. In addition, the node that cannot be recovered at that time can be surely specified.

In this embodiment, the recovery control node 30 located at the highest level in order to obtain the recovery information of the entire distributed database system has been described, but the recovery control node of another hierarchy may be used.

Example 5. A feature of this embodiment is that in the configuration of FIG. 2 or FIG. 7, the highest recovery control node 30 has a mechanism for notifying the lower node 27 or the like of the failure recovery status. This allows all nodes to be notified of the recovery status of the distributed database system.

FIG. 9 is a flow chart of the recovery process of the highest node in the fifth embodiment, and the process of notifying the failure recovery status in this embodiment will be described below with reference to FIG.

The recovery control node 30 at the highest level basically establishes all the communication partners of each node in steps 51, 52 and 55 as in the processing shown in FIG. Is also confirmed, and the confirmed transaction state is notified to the node that has reported the transaction in the indeterminate state. Also, step 84
At, the recovery information is obtained by transmitting an inquiry message to the node for which the recovery information is not acquired.

In step 95, all nodes are notified of a message including system recovery status information indicating the recovery status of the entire distributed database system, such as the presence or absence of unrecoverable nodes grasped in step 84. A message is sent together with the confirmation information to the node that has reported the transaction in the indeterminate state.

As a result, the recovery status of the entire distributed database system, which cannot be conventionally grasped by each node, can be confirmed at all nodes.

By the way, in each of the above-mentioned embodiments, the distributed database system constituted by a network such as a line has been described, but it goes without saying that it can also be applied to a distributed database system constructed in a loosely coupled parallel computer system.

[0055]

As described above, the present invention has the following effects.

The number of nodes constituting the distributed database is n
When a large-scale access is made to the entire distributed database system at the time of failure, the number of message exchanges required for recovery is always 2n times or less as a primary function of n, no matter how complicated access is made at the time of failure. Therefore, the recovery time can be significantly reduced.

Further, since the time required for recovery depends on the linear function of the number of nodes, it is possible to estimate the end time of the recovery process.

When the number of nodes forming the distributed database is n, the number of detours that should be set so as not to hinder recovery at the time of a communication path failure is n, and the number of detours is significantly reduced. It becomes possible to do.

Further, even when the information necessary for the recovery cannot be obtained, the communication mechanism is simply provided by the provision of the inquiry mechanism, except when the failure of the communication path including the detour or the failure of the partner node occurs. When the processing is delayed due to the processing delay of the other party, the delay can be minimized to obtain the recovery information, and the failure recovery processing can be completed in the minimum time. In addition, the node that cannot be recovered at that time can be surely specified.

Further, by providing a mechanism for notifying the lower level nodes of the failure recovery status, it is possible to confirm the recovery status of the entire distributed database system in all the nodes, which could not be grasped in the conventional node. It will be possible. As a result, it is possible to reliably notify the user of the full or partial resumption of business using the distributed database.

[Brief description of drawings]

FIG. 1 is a conceptual diagram of a distributed database for explaining the gist of a first embodiment according to the present invention.

FIG. 2 is a configuration diagram of a distributed database for explaining the first embodiment.

FIG. 3 is a flowchart of a recovery process at the time of a failure in the lowest node in the first embodiment.

FIG. 4 is a flowchart of a recovery process at the time of a failure in the recovery control node located in the middle tier in the first embodiment.

FIG. 5 is a flowchart of a recovery process at the time of a failure in the recovery control node located at the highest level in the first embodiment.

FIG. 6 is a conceptual diagram of a distributed database for explaining the gist of the second embodiment according to the present invention.

FIG. 7 is a configuration diagram of a distributed database for explaining a third embodiment according to the present invention.

FIG. 8 is a flowchart of a recovery process of the highest node in the fourth embodiment according to the present invention.

FIG. 9 is a flowchart of a recovery process of the highest node in the fifth embodiment according to the present invention.

FIG. 10 is a configuration diagram of a conventional distributed database.

[Explanation of symbols]

11, 21 nodes, 12, 16 databases, 1
3, 17, 22, 25, 28 Recovery node group, 1
4, 18, 23, 26, 29 Channel, 15, 24, 2
7, 30 Recovery control node, 19, 20, 71, 72
Detour communication path.

Claims (6)

[Claims] 1. In a distributed database system in which a plurality of nodes each having a processor and a database are connected by a network, and a process in each node accesses a database of another node, the plurality of nodes are logically divided into groups, and A failure recovery method for a distributed database system characterized in that a failure control node for controlling failure recovery is provided for each group so that failure recovery control for each node is performed in a tree structure. 2. A failure recovery method for a distributed database system according to claim 1, wherein a plurality of said recovery control nodes are logically divided into groups, and a higher recovery control node for controlling failure recovery for each group. A disaster recovery method for distributed database systems, in which the failure recovery control of each node is performed hierarchically by providing the. 3. The failure recovery method for a distributed database system according to claim 2, wherein a hierarchy is formed until the number of the upper recovery control nodes finally becomes one. . 4. The failure recovery method for a distributed database system according to claim 1, wherein a detour is provided between nodes located at an upper level and a lower level of the tree structure. 5. The failure recovery method for a distributed database system according to any one of claims 1 to 4, wherein the recovery control node cannot obtain recovery information for establishing the state of an indefinite transaction from a node of a communication partner. A failure recovery method for a distributed database system, characterized in that it has a mechanism for inquiring about the node. 6. The failure recovery method for a distributed database system according to claim 1, wherein the recovery control node has a mechanism for notifying a lower level node of the failure recovery status. Disaster recovery method.