WO2017041671A1

WO2017041671A1 - Method and apparatus for recovering fault

Info

Publication number: WO2017041671A1
Application number: PCT/CN2016/097957
Authority: WO
Inventors: 李龙; 龚学文; 胡琳
Original assignee: 华为技术有限公司
Priority date: 2015-09-10
Filing date: 2016-09-02
Publication date: 2017-03-16
Also published as: CN106528324A

Abstract

Provided are a method and apparatus for recovering a fault, which can improve the reliability of fault recovery. The method comprises: determining the size of a recovery file corresponding to each failure process in N failure processes, and determining a running state of each recovery node in M recovery nodes, where N ≥ 1 and M ≥ 2; according to the size of the recovery file corresponding to each failure process and the running state of each recovery node, determining a recovery node corresponding to each failure process, wherein the running state comprises a resource usage state or a communication state; and performing control according to the recovery node corresponding to each failure process, so as to perform fault recovery on each failure process at the recovery node corresponding to each failure process.

Description

Method and device for fault recovery

The present application claims priority to Chinese Patent Application No. 2015-A No. No. No. No. No. No. No. No. No. No. No. No. No. No. No. No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No

Technical field

The present invention relates to the field of computers and, more particularly, to methods and apparatus for fault recovery.

Background technique

With the rapid development of computer technology, more and more industries are using computer technology to improve the efficiency of the industry.

In a computer system, the computer mainly relies on the execution process to realize its function. When the process fails or even fails, it may affect the normal operation of the program and even the entire computer system. Therefore, how to realize the process recovery of the process becomes an urgent problem to be solved.

At present, a method for recovering a fault is known, which periodically records a recovery file that records a calculation state when a process is normal, and corresponds to the node when a node running the process fails and the process fails. The recovery node recovers the process based on the saved recovery file.

However, when the recovery node corresponding to the node also fails, the recovery of the process cannot be achieved, which seriously affects the reliability of the fault recovery.

Summary of the invention

Embodiments of the present invention provide a method and apparatus for fault recovery, which can improve the reliability of fault recovery.

In a first aspect, a method for fault recovery is provided, the method comprising: determining a size of a recovery file corresponding to each of the N failed processes, and determining an operating state of each of the M recovery nodes, Wherein, N≥1, M≥2; determining a recovery node corresponding to each invalidation process according to the size of the recovery file corresponding to each failure process and the running state of each recovery node, wherein the running state includes a resource usage state Controlling the recovery nodes corresponding to each failed process to perform a failure process for each failed process Recovery.

In conjunction with the first aspect, in a first implementation manner of the first aspect, the recovery file corresponding to the first invalid process of the N failed processes is stored in the at least two storage nodes.

With reference to the first aspect and the foregoing implementation manner, in the second implementation manner of the first aspect, the recovery files corresponding to the first invalidation process stored in each storage node are the same.

With reference to the first aspect and the foregoing implementation manner, in a third implementation manner of the first aspect, the recovery file corresponding to the first invalidation process includes at least two sub-recovery files, and the sub-recovery files stored in each storage node different.

With reference to the first aspect and the foregoing implementation manner, in a fourth implementation manner of the first aspect, when N≥2, the size of the recovery file corresponding to each failure process and the running state of each recovery node are determined. The recovery node corresponding to each failure process includes: determining, according to the running state of each recovery node, the size of the recovery file corresponding to each failure process, in descending order, determining each failure process in turn Recovery node.

With reference to the first aspect and the foregoing implementation manner, in the fifth implementation manner of the first aspect, the recovery node corresponding to a failed process is different from the storage node corresponding to the same invalidation process.

With reference to the first aspect and the foregoing implementation manner, in a sixth implementation manner of the first aspect, the controlling, by the recovery node corresponding to each failure process, includes: performing, according to the recovery node corresponding to each failure process The state and the size of the recovery file corresponding to each failed process, the recovery time of each failed process is estimated; and the control is performed according to the recovery time of each failed process.

In a second aspect, a device for fault recovery is provided, the device comprising: a determining unit, configured to determine a size of a recovery file corresponding to each of the N failed processes, and each of the M recovery nodes The running state, and determining the recovery node corresponding to each invalid process according to the size of the recovery file corresponding to each failure process and the running state of the M recovery nodes, wherein the running state includes the resource usage state, N≥1 , M≥2; a processing unit, configured to control a recovery node corresponding to each failed process to perform fault recovery for each failed process in the recovery node corresponding to each failed process.

With reference to the second aspect, in a first implementation manner of the second aspect, the recovery file corresponding to the first invalid process of the N failed processes is stored in the at least two storage nodes.

With reference to the second aspect and the foregoing implementation manner, in the second implementation manner of the second aspect, the recovery files corresponding to the first invalidation process stored in each storage node are the same.

In combination with the second aspect and the foregoing implementation manner, in a third implementation manner of the second aspect, the The recovery file corresponding to the first invalidation process includes at least two sub-recovery files, and the sub-recovery files stored in each of the storage nodes are different.

With reference to the second aspect and the foregoing implementation manner, in a fourth implementation manner of the second aspect, when N≥2, the determining unit is specifically configured to be configured according to each failure process according to an operating state of each recovery node. The size of the restored files, in descending order, determines the recovery nodes corresponding to each failed process.

With reference to the second aspect and the foregoing implementation manner, in the fifth implementation manner of the second aspect, the recovery node corresponding to one failure process is different from the storage node corresponding to the same invalidation process.

With reference to the second aspect and the foregoing implementation manner, in a sixth implementation manner of the second aspect, the processing unit is specifically configured to: restore, according to an operating state of the recovery node corresponding to each failure process, and recovery corresponding to each failure process. The size of the file, the recovery time of each failed process is estimated, and control is performed according to the recovery time of each failed process.

It can be seen that, according to the method for recovering faults according to the embodiment of the present invention, the recovery of the failure recovery of the failed process is determined from the at least two recovery nodes according to the size of the recovery file corresponding to the failed process and the running status of the at least two recovery nodes. A node is more reliable than only one recovery node, and at the same time can ensure that the determined recovery node can achieve fault recovery for the failed process to further improve the reliability of the fault recovery.

DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings to be used in the embodiments of the present invention will be briefly described below. It is obvious that the drawings described below are only some embodiments of the present invention, Those skilled in the art can also obtain other drawings based on these drawings without paying any creative work.

1 is a schematic flow chart of a method of fault recovery according to an embodiment of the present invention.

2 is a schematic architectural diagram of a method of applying fault recovery in accordance with an embodiment of the present invention.

3 is a schematic block diagram of an apparatus for fault recovery in accordance with an embodiment of the present invention.

4 is a schematic structural diagram of an apparatus for fault recovery according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly described below with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are a part of the embodiments of the present invention, and not all of them. Example. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

The method and apparatus for fault recovery provided by the embodiments of the present invention can be applied to a computer, which includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a CPU, a Memory Management Unit (MMU), and a memory (also referred to as a memory). The operating system may be any one or more computer operating systems that implement business processing through processes, such as a Linux system, a Unix system, an Android system, an iOS system, or a Windows system. The application layer includes applications such as browsers, contacts, word processing software, and instant messaging software. It should be understood that the above-listed computer devices are merely illustrative and the invention is not particularly limited.

1 is a schematic flowchart of a method 100 for fault recovery according to an embodiment of the present invention. As shown in FIG. 1, the method 100 includes:

S110, determining a size of a recovery file corresponding to each of the N failed processes, and determining an operation state of each of the M recovery nodes, where N≥1, M≥2;

It should be noted that, as an example and not by way of limitation, in the embodiment of the present invention, the failure process refers to the result of the operation (or processing) failing to reach the desired process, for example, the operation is aborted due to the failure. The process, that is, the process does not run the result (it can also be seen that the running result of the process is not as expected), and for example, the process seems to be successful but the running result is not in accordance with the expected process.

S120. Determine, according to a size of the recovery file corresponding to each failure process and an operation state of each recovery node, a recovery node corresponding to each failure process, where the operation state includes a resource usage state.

It should be noted that, as an example and not by way of limitation, the resource usage status includes hardware utilization on the recovery node, such as CPU utilization and/or memory utilization. For example, if the CPU usage of a recovery node is already high, that is, the recovery node is already busy, and the recovery file corresponding to the invalidation process is large, that is, the recovery process needs to consume more resources, then the recovery is performed. The node is not suitable as a recovery node for the failed process.

Optionally, in other embodiments, the running state may further include a communication state of the recovery node. Specifically, the communication state refers to whether the communication state of the recovery node and other nodes is communicable or not. For example, if a recovery node cannot communicate with the storage node of the recovery file corresponding to the storage invalidation process A, the recovery node cannot be the recovery node of the invalidation process A.

S130. Control a recovery node corresponding to each failed process to perform fault recovery for each failed process in the recovery node corresponding to each failed process.

2 shows a schematic architectural diagram of a computer system 200 to which the method 100 is applied. As shown in FIG. 2, the computer system 200 includes a management node 210, a plurality of storage nodes 220, and a plurality of recovery nodes 230 and a plurality of computing nodes 240. For ease of understanding and illustration, only one computing node 240 is shown in FIG.

Each computing node 240 runs one or more processes.

Moreover, each computing node 240 is in communication with one or more storage nodes 220 such that the computing node 240 can transfer the recovered files of the running processes to the connected storage node 220 for backup. By way of example and not limitation, in an embodiment of the invention, the recovery file may be data when its process is running in a normal state.

In addition, the management node 210 is in communication with each of the computing nodes 240 such that the management node 210 can monitor the operational status of each of the processes running in each computing node 240.

Further, the management node 210 is communicatively coupled to each of the recovery nodes 230 such that the management node 210 can monitor the operational status of each of the recovery nodes 230 and send control instructions to the recovery node 230.

Optionally, in the embodiment of the present invention, each recovery node 230 and each storage node 220 can be communicably connected, so that the recovery node 230 can obtain the recovery file from the storage node 220 when recovering the process.

It should be understood that the connection relationship between each of the recovery nodes 230 and the storage nodes 220 enumerated above is only an example. For example, the management node 210 may be communicably connected to each storage node 220, and the recovery node 230 may be passed through the management node 210. The recovery file is obtained in the storage node 220.

Further, in the computer system 200, a bus system 250 communicably connected to the nodes may be provided, so that the communication connection between the above-described nodes can be realized by the bus system 250.

In the embodiment of the present invention, one node may be an independently configured computer entity, or multiple nodes may be configured in the same computer entity, or multiple computer entities may constitute one node, and the present invention is not special. limited.

In the embodiment of the present invention, the method 100 may be performed by the management node 210, which may be independent of each node in the computer system (including the failed computing node 240, the storage node for storing the recovery file of the failed process). 220 and recovery node 230 for failover failure. And, the management node is in communication connection with each node to transmit information such as control instructions or data; or the method 100 may be integrated with the management node in the computer system 200. The function of one or more other nodes is executed.

Moreover, the method 100 implemented by the present invention is applied to a computer system including at least two recovery nodes, each of which is capable of providing computing resources (e.g., a central processing unit and memory, etc.) to enable recovery of the failed process.

In addition, the management node can instruct the recovery node to perform recovery for the failed process.

Hereinafter, in order to facilitate understanding and explanation, the processing procedure of the method 100 will be described in detail with the management node as the execution subject of the method 100.

Specifically, in the implementation of the present invention, in order to prevent the process from being invalid and affecting the service service, the recovery file of the process may be periodically stored, for example, the execution state or the calculation state, and when the process fails, roll back to the previously saved one. The status is restarted.

The recovery file in the embodiment of the present invention is used to record data when the process is running in a normal state, so as to perform fault recovery for each invalid process according to the content recorded by the recovery file. In some embodiments, the recovery file may be a checkpoint file that is generated based on a checkpoint technique; in other embodiments, the recovery file may be a log file that is generated based on a logging technique of. It should be noted that the generation of the checkpoint file or the log file may be implemented by the prior art, and is not included in the scope of the present invention, and therefore will not be described in detail herein.

It should be understood that the form of the recovery file listed above and the recorded content are merely exemplary descriptions, and the present invention is not limited thereto, and other specific embodiments capable of implementing the function of restoring files of the embodiments of the present invention are all included in the present invention. Within the scope of protection.

In the implementation of the present invention, the number of invalid processes may be one or multiple, and the present invention is not particularly limited, and when the number of failed processes is plural, the processing for each failed process is similar, and the following is For ease of understanding and explanation, the processing of the method 100 implemented by the present invention will be described in detail by taking the processing of the invalidation process #A as an example.

Optionally, the recovery file corresponding to the first invalid process of the N failed processes is stored in at least two storage nodes.

Specifically, in the implementation of the present invention, for the invalidation process #A (ie, an example of the first invalidation process), the corresponding recovery file may be stored in two or more corresponding to the invalidation process #A. In the storage node.

Here, when there are multiple invalid processes, the storage nodes in which the recovery files of any two failed processes are stored may be the same or different, and the present invention is not particularly limited.

In addition, the above "identical" may include: identical, for example, the recovery text of the invalidation process #A The pieces may be stored in the storage node #α and the storage node #β, and the recovery file of the invalidation process #B may be stored in the storage node #α and the storage node #β; or, partially, for example, the recovery file of the invalidation process #C may be stored. For the storage node #n and the storage node #θ, the recovery file of the invalidation process #D can be stored in the storage node #θ and the storage node #λ).

Similarly, the above "different" may include: completely different, for example, the recovery file of the invalidation process #A may be stored in the storage node #α and the storage node #β, and the recovery file of the invalidation process #B may be stored in the storage node #γ and Storage node #δ; or, partially different, for example, the recovery file of the invalidation process #C may be stored in the storage node #n and the storage node #θ, and the recovery file of the invalidation process #D may be stored in the storage node #θ and the storage node # λ).

Moreover, the number of storage nodes in which the recovery files of one of the above-mentioned failed processes are stored is merely exemplary. The present invention is not limited thereto. For example, a recovery file of a failed process may be stored only in one storage node. Moreover, when there are multiple invalid processes, the number of storage nodes in which the recovery files of the failed processes are stored may be the same or different, and the present invention is not particularly limited.

In the implementation of the present invention, the recovery file corresponding to the invalidation process #A may be stored in a plurality of (at least two) storage nodes in the following manner.

Mode 1

Optionally, the recovery files corresponding to the first invalidation process stored in each storage node are the same.

Specifically, in the implementation of the present invention, a complete recovery file corresponding to the invalidation process #A may be stored in a plurality of storage nodes (hereinafter, for ease of understanding and distinction, it is recorded as: recovery file #A). In addition, the "complete recovery file" means that the failure process #A can be handled by the recovery file #A stored in one storage node.

The method for fault processing according to the present invention, by making the recovery files corresponding to the first invalidation process stored in each storage node the same (ie, each storage node stores a consistent, complete recovery file), When one or more storage nodes fail, the recovery files can still be obtained from other non-failed storage nodes, thereby further improving the reliability of the fault handling.

Mode 2

Optionally, the recovery file corresponding to the first invalidation process includes at least two sub-recovery files, and the sub-recovery files stored in each storage node are different. .

Specifically, in the practice of the present invention, the recovery file #A can be divided into a plurality of sub-recovery files (for ease of understanding and distinction, it is recorded as: sub-recovery file #A ₁ - sub-recovery file #A _X ). And the sub-recovery file #A ₁ - sub-recovery file #A _X are respectively stored in a plurality of storage nodes, wherein one sub-recovery file may be stored in one storage node, or may be stored (repeatedly or dividedly) in multiple Among the storage nodes, the present invention is not particularly limited, and any two sub-recovery files are stored in different storage nodes, or the sub-recovery files stored in the storage nodes are different. Here, "different" may be completely different. For example, for the sub-recovery file #A ₁ and the sub-recovery file #A ₂ , the sub-recovery file #A ₁ may be stored in the storage node #1 and the storage node #2, and the sub-recovery file #A ₂ may be stored in the storage node #3 and Storage node #4; or, "different" may also be partially different, for example, for sub-recovery file #A ₃ and sub-recovery file #A ₄ , sub-recovery file #A ₃ may be stored in storage node #5 and storage node #6, Sub-recovery file #A ₄ can be stored in storage node #6 and storage node #7.

According to the method for fault processing provided by the embodiment of the present invention, by dividing the recovery file corresponding to the failed process into multiple sub-recovery files and storing each sub-recovery file in a different storage node, it is possible to simultaneously perform the fault processing. The plurality of storage nodes acquire the sub-recovery files, thereby reducing the time required for transmitting the restored files and improving the efficiency of troubleshooting.

Optionally, in some embodiments, the storage of the recovery file of the "failed process" is performed periodically when the process is valid (or normal operation), that is, in the implementation of the present invention, Each process running in a computer system periodically stores its recovery files while the process is active (or, in normal operation).

In other embodiments, the recovery file may also be stored in the computing node running after the failure process fails, and uploaded to the storage node before the failure, that is, the backup of the recovery file may also be performed periodically. of.

The storage methods of the recovery files listed above are merely exemplary, and the present invention is not limited thereto. For example, the recovery files of the processes may be uniformly stored in one storage node.

The method for fault processing implemented by the present invention may be executed when the management node determines that the process is invalid. As a method for determining the process failure, for example, a process running on each node in the computer system may periodically send a heartbeat message to the management node, if the management node If the heartbeat message of a process is not received within the specified time, the process may be considered invalid. It should be understood that the above-listed methods for determining the process failure are merely exemplary, and the present invention is not limited thereto, and the methods for determining the process failure in the prior art all fall within the protection scope of the present invention.

Therefore, after determining that the process is invalid as described above, at S110, the management node may determine the size of the recovery file corresponding to each failed node, for example, for the invalidation process #A (ie, an example of the invalidation process), the management node may fail. The storage device(s) corresponding to the process #A acquires information indicating the size of the restored file #A (or, the sub-recovery file #A ₁ to the sub-recovery file #A _X ), and determines based on the information. The size of the recovery file corresponding to the invalidation process #A.

For another example, each process may determine the size of the restored file when generating the recovery file, and send information for indicating the size of the restored file to the management node, and the management node may store the information according to the identifier of the process, that is, may be based on the process. And mapping the information indicating the size of each recovery file to the information indicating the size of the recovery file from the process, and storing and indexing the obtained information indicating the size of each recovery file, so that when the management node finds that the process is invalid, the The identifier of the process finds information indicating the size of the recovery file corresponding to the process.

And, at S110, the management node can determine the operational status of each recovery node.

In some embodiments, the running state may include restoring a resource usage state of the node. Specifically, the recovery node may report the central processor of the recovery node to the management node according to the indication of the management node or periodically (CPU, Central Processing) Unit) Load information such as usage rate or memory usage, so that the management node can determine the resource usage status based on the load information from the recovery node.

In other embodiments, the operational state may include restoring a communication state of the node. Specifically, the recovery node may report to the management node a communication status indicating the recovery node and other nodes in the computer system (eg, the status may include Communication status information of normal communication, inability to communicate, or communication delay, etc., whereby the management node can determine its communication status based on the communication status information from the recovery node.

It should be noted that, in the implementation of the present invention, each storage node may constitute a storage network (or a storage grid), and the storage network provides data to each recovery node in the computer system through a unified external interface (for example, recovery). File), and each recovery node and the storage network can communicate through a message queue or the like, so that each recovery node does not need to know the specific address of the storage node storing the required recovery file, for example, Internet Protocol (IP, Internet Protocol) address or media access control address (MAC, Media Access Control) address, etc.

For example, the recovery node only needs to send the identifier of the invalidation process #A, which is the object of the fault processing, to the storage network, and the interface device of the storage network can map the relationship between the identifier of each process and the storage node where the recovery file is stored according to the pre-stored identifier. The storage node corresponding to the received invalidation process #A is found, and the recovery file #A corresponding to the invalidation process #A can be obtained.

Therefore, the communication state can be a communication state between the recovery node and the interface device of the storage network.

In addition, in the implementation of the present invention, there are cases where a plurality of processes (for example, the plurality of processes include the above-mentioned failed process #A and process #B) need to communicate to implement a service function, in this case, for The recovery node that fails the process #A for recovery needs to communicate with the node running Process #B (ie, the associated node of the recovery node).

Therefore, the communication state may also be a communication state of the recovery node and the associated node.

After determining the size of the recovery file and the running status of each recovery node, as described above, S120 determines the recovery corresponding to each invalidation process according to the size of the recovery file corresponding to each invalidation process and the operating state of each recovery node. node.

For example, when N=1, there is only one process that needs to be restored (for example, the above-mentioned invalidation process #A). In this case, the node whose current running state can satisfy the operating condition required by the invalidation process #A can be selected as the node. The recovery node corresponding to the invalid process #A.

As the operating conditions, for example,

A. The computing resource or the storage resource can satisfy the processing requirement of the invalid process #A, that is, the idle resource of the node can ensure the running of the invalid process #A, for example, the memory, CPU, storage, and the like of the recovery node satisfy the process #A. Operational requirements.

B. The communication state can satisfy the processing requirement of the invalidation process #A, that is, the recovery node can communicate with the associated node of the failed process #A, or the recovery node can communicate with the storage node of the failed process #A to obtain the invalidation process# A recovery file.

It should be understood that the method and process for determining the recovery node corresponding to the invalidation process according to the operating state of the recovery node and the size of the recovery file enumerated above are merely exemplary descriptions, and the present invention is not limited thereto.

For another example, when N≥2, the management node may determine the recovery order for each invalidation process according to the size of each recovery file, that is,

Optionally, when N≥2, the recovery node corresponding to each invalidation process is determined according to the size of the recovery file corresponding to each failure process and the running state of each recovery node, including:

According to the running state of each recovery node, based on the size of the recovery file corresponding to each invalidation process, the recovery nodes corresponding to each invalidation process are sequentially determined in descending order.

Specifically, when there are multiple failed processes, it is necessary to find a suitable recovery node for the multiple failed processes, and ensure that the sum of recovery times of the multiple failed processes is the shortest.

In the embodiment of the present invention, the recovery node corresponding to each invalid process may be sequentially determined according to the size of the restored file, and the recovery node corresponding to the largest invalidation process of the recovery file is preferentially determined.

The following describes the specific determination process of the recovery node corresponding to each failure process when N≥2.

Step 1: The management node may determine the set of the invalid processes (including the invalid process #1 to the invalid process #K) according to the size of the restored file, in descending order, wherein P ₁ ≥ P ₂ ≥ ... ≥ P _K , wherein , P represents the recovery file size, K is the number of invalid processes to be restored, and the count variable i=1;

Step 2: The management node may acquire a node capable of providing recovery processing (or a virtual machine capable of placing a failure to be restored), and set the set of these nodes to N, and the allocated set Nu=0;

Step 3, if i>K, then perform step 6, otherwise, perform step 4;

Step 4: For the invalidation process #i, if the order of the nodes in the set Nu is arranged (for example, it may be arranged in the chronological order of the set, or may be arranged according to the remaining available resources, the invention is not particularly limited) as N ₁ , N ₂ ... N _j , firstly try to determine whether the node in the set Nu can satisfy the running condition of the invalid process #i, if not, select from the set N, and the node that can be satisfied (for example, the set N can The node with the smallest subscript that satisfies the operating condition of the failed process #i is placed in the combined Nu as N _j+1 . And, the management node can perform update processing, let i=i+1, and go to Step3.

It should be understood that the foregoing enumerated methods for determining the recovery nodes corresponding to each failure process are merely exemplary, and the present invention is not limited thereto, and for example, first adaptation, optimal adaptation, first-order adaptation in descending order, and optimal descending order may be used. A heuristic algorithm such as an adaptive algorithm determines the recovery node corresponding to each failed process.

After the recovery node corresponding to each failure process is determined as described above, at S130, the management node may perform recovery processing on each failure according to the recovery node corresponding to each failure process.

For example, the management node may directly instruct the recovery node to obtain a recovery file of the invalidation process from the storage node, and perform recovery based on the recovery file. Here, the method and the process of the recovery node performing the recovery process based on the recovery file pair may be similar to the prior art, and a detailed description thereof will be omitted herein to avoid redundancy.

For another example, the management node may estimate the recovery time according to the recovery node corresponding to each failure process, and determine the recovery strategy according to the recovery time, that is,

Optionally, the fault recovery process is performed on each failed process according to the recovery node corresponding to each failure process, including:

Estimating the recovery time of each failed process according to the running state of the recovery node corresponding to each failed process and the size of the recovery file corresponding to each failed process;

According to the recovery time of each failed process, each failed process is subjected to failure recovery processing.

Specifically, the management node can sample the recovery time of the failed process in the following manner.

The management node can match the failed process and the recovery node by using multiple schemes (for example, preset), or the management node can simulate and place the failed process with a recovery by using multiple schemes (for example, preset). Node, and calculate the recovery time under the scenario.

Among them, each program needs to meet the following conditions:

Condition 1

The recovery node matched by the invalidation process #K needs to meet the requirement of the failure recovery process of the failure process #K for the CPU processing capability, that is, the above condition 1 can be expressed as the following formula (1)

among them,

Indicates the requirement of CPU recovery capability for failure recovery of failure process #K; a _Ki ∈[0,1], ie, if failure process #K is placed at recovery node i, then a _Ki =1, otherwise a _Ki =0;

Indicates the processing power of the CPU of the recovery node #i (for example, the number of virtual CPUs that can be provided).

Condition 2

The recovery node matched by the invalidation process #K needs to meet the memory recovery requirement of the failure process #K, that is, the above condition 2 can be expressed as the following formula (2)

among them,

Represents the memory recovery requirement for failure process #K; a _Ki ∈[0,1], ie, if the failure process K is placed at recovery node i, a _Ki =1, otherwise a _Ki =0;

Indicates the memory that can be provided by the recovery node #i.

And above

It can be determined according to the following formula (3):

Where m _K represents the size of the invalidation process #K application memory (or the size of the recovery file of the invalidation process #K), μ, ε are preset coefficients, determined by experiments, and v represents the number of failures.

Condition 3

The failure recovery of a failed process is only performed in one recovery node, for example, the CPU processing capability of the recovery node for performing failure recovery for the failed process #1 (for example, The number of virtual CPUs needs to meet the CPU processing capability requirements for failure recovery of failure process #1, and the recovery node that can be used for failure recovery for failure process #1 can provide memory recovery that needs to satisfy failure process #1. Memory requirements.

The quantized relationship between the failed process recovery time T(n _i ) on the recovery node #i and the size m _K of the recovery file of the invalidation process #K can be expressed by the following equation (4):

Among them, a and β are coefficients, which are determined by experiments.

The failure node recovery time of the jth placement scheme takes the maximum value of the recovery time of the failure process set on all nodes. R _j can be expressed by the following formula (5):

R _j =max(T _j (n _j )), i=1,2,...s (5)

Thus, based on the above conditions 1 to 3, it is possible to determine a matching scheme in which the objective function min(R _j ) of the recovery time overhead quantization model is minimized, and the recovery time of each invalidation process in the scheme. .

After determining the recovery time of each failure process as described above, the management node may determine whether the estimated recovery time of each failure process satisfies the recovery time requirement of the process, and perform recovery processing according to the determination result, for example, if the estimated The recovery time is less than or equal to the maximum value of the recovery time required by the process, and the management node may instruct the recovery node to recover the invalidation process. For another example, if the estimated recovery time is greater than the maximum value of the recovery time required by the process, the management node may perform troubleshooting processing or the like on the original node running the failed process.

Optionally, the recovery node corresponding to a failed process is different from the storage node corresponding to the same invalid process.

Specifically, in the implementation of the present invention, the storage node corresponding to the invalidation process #A is different from the recovery node corresponding to the invalidation process #A, that is, in the computer system of the embodiment of the present invention, it can be used for storage. The storage node that restores the file is independent of the recovery node used to perform the recovery process, thereby facilitating maintenance and reducing the burden on each node.

It should be understood that the relationship between the storage node and the recovery node enumerated above is only an exemplary description, and the storage node and the recovery node corresponding to one failure process may also be the same node, and the present invention is not particularly limited.

According to the method for recovering a fault according to an embodiment of the present invention, a recovery node that performs fault recovery on the failed process is determined from at least two recovery nodes according to a size of the recovery file corresponding to the failed process and an operating state of the at least two recovery nodes. More reliable than just one recovery node, It can ensure to a certain extent that the determined recovery node can achieve fault recovery for the failed process, thereby further improving the reliability of the fault recovery.

The method of fault recovery implemented by the present invention has been described in detail above with reference to FIGS. 1 and 2. Hereinafter, the apparatus for fault recovery of the present invention will be described in detail with reference to FIG.

FIG. 3 shows a schematic block diagram of an apparatus 300 for fault recovery in accordance with an embodiment of the present invention. As shown in FIG. 3, the apparatus 300 includes:

The determining unit 310 is configured to determine a size of the recovery file corresponding to each of the N failed processes, and an operating state of each of the M recovery nodes, and according to the recovery file corresponding to each invalid process The size and the running status of the M recovery nodes determine the recovery node corresponding to each failed process, wherein the running status includes a resource usage status or a communication status, N≥1, M≥2;

The processing unit 320 is configured to perform, according to the recovery node corresponding to each invalidation process, to perform fault recovery for each failed process in the recovery node corresponding to each failed process.

Optionally, the recovery file corresponding to the first invalidation process includes at least two sub-recovery files, and the sub-recovery files stored in each storage node are different.

Optionally, when N≥2, the determining unit is specifically configured to sequentially determine each of the recovery files according to the running state of each recovery node according to the size of the recovery file corresponding to each invalidation process, in descending order. The recovery node corresponding to the failed process.

Optionally, the processing unit is specifically configured to estimate, according to an operation state of the recovery node corresponding to each failure process and a size of the recovery file corresponding to each failure process, a recovery time of each failure process, and according to each failure The recovery time of the process, and the failure recovery process is performed for each failed process.

The apparatus 300 for fault processing according to an embodiment of the present invention may correspond to an execution subject in a method of the embodiment of the present invention, for example, a management node, and each unit in the apparatus 300 of the fault processing, that is, a module and the other operations described above and/or For the sake of brevity, the functions of the method 100 in FIG. 1 are not described here.

According to the apparatus for fault recovery according to the embodiment of the present invention, the recovery node that recovers the failure process from the at least two recovery nodes is determined according to the size of the recovery file corresponding to the failure process and the operation state of the at least two recovery nodes, Compared with only one recovery node, the reliability is higher, and at the same time, it can ensure that the determined recovery node can achieve fault recovery for the failed process, thereby further improving the reliability of the fault recovery.

The method of fault recovery implemented by the present invention has been described in detail above with reference to FIG. 1 and FIG. 2. The apparatus for fault recovery of the invention will be described in detail below with reference to FIG.

FIG. 4 shows a schematic block diagram of a device 400 for fault recovery in accordance with an embodiment of the present invention. As shown in FIG. 4, the device 400 includes:

Bus system 410;

a processor 420 connected to the bus system 410;

a memory 430 connected to the bus system 410;

The processor, by using the bus, invokes a program stored in the memory to determine a size of a recovery file corresponding to each of the N failed processes, and determines each of the M recovery nodes. Operating state, wherein N≥1, M≥2;

And determining, by the size of the recovery file corresponding to each failure process, and the operation state of each recovery node, the recovery node corresponding to each failure process, where the operation state includes a resource usage state or a communication state;

It is used to control according to the recovery node corresponding to each failure process, to recover the failure process for each failure process in the recovery node corresponding to each failure process.

Optionally, the recovery file corresponding to the first invalid process of the N failed processes is stored in at least two storage nodes corresponding to the first invalid process.

Optionally, when N≥2, the processor is specifically configured to sequentially determine each of the recovery files according to the running state of each recovery node according to the size of the recovery file corresponding to each invalidation process, in descending order. The recovery node corresponding to the failed process.

Optionally, the processor is specifically configured to estimate, according to an operating state of the recovery node corresponding to each failed process and a size of the recovery file corresponding to each failed process, a recovery time of each invalid process;

It is used to perform fault recovery processing for each failed process according to the recovery time of each failed process.

The processor can also be referred to as a CPU. The memory can include read only memory and random access memory and provides instructions and data to the processor. A portion of the memory may also include a Non-Volatile Random Access Memory (NVRAM). In a specific application, the device 400 may be embedded or may be a computer device. The bus includes a power bus, a control bus, and a status signal bus in addition to the data bus. However, for the sake of clarity, various buses are labeled as bus system 410 in the figure. The decoder in a specific different product may be integrated with the processing unit.

The processor may implement or perform the steps and logic blocks disclosed in the method embodiments of the present invention. The general purpose processor may be a microprocessor or the processor or any conventional processor, decoder or the like. The steps of the method disclosed in the embodiments of the present invention may be directly implemented by the hardware processor, or may be performed by a combination of hardware and software modules in the decoding processor. The software module can be located in a conventional storage medium such as random access memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, registers, and the like.

It should be understood that, in the embodiment of the present invention, the processor 420 may be a central processing unit ("CPU"), and the processor 420 may also be other general-purpose processors and digital signal processors (digital signals). Processor, DSP, Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), etc. The general purpose processor may be a microprocessor or the processor or any conventional processor or the like.

The memory 430 can include read only memory and random access memory and provides instructions and data to the processor 420. A portion of the memory 430 may also include a non-volatile random access memory. For example, the memory 430 can also store information of the device type.

The bus system 410 may include a power bus, a control bus, a status signal bus, and the like in addition to the data bus. However, for clarity of description, various buses are labeled as bus system 410 in the figure. It should be noted that, in the embodiment of the present invention, "connected to the bus system 410" may include direct connection or indirect connection.

In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 420 or an instruction in a form of software. The steps of the method disclosed in the embodiments of the present invention may be directly implemented as a hardware processor, or may be performed by a combination of hardware and software modules in the processor. The software module can be located in a conventional storage medium such as random access memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, registers, and the like. The storage medium is located in the memory 430, and the processor 420 reads the information in the memory 430 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

The device 400 for failure recovery according to an embodiment of the present invention may correspond to an execution body (eg, a management node) in the method of the embodiment of the present invention, and each unit in the device 400 of the failure recovery, that is, the module and the other operations described above and/or For the sake of brevity, the functions of the method 100 in FIG. 1 are not described here.

According to the embodiment of the present invention, the fault recovery device determines, according to the size of the recovery file corresponding to the invalidation process and the operating state of the at least two recovery nodes, the recovery node that recovers the failed process from the at least two recovery nodes, Compared with only one recovery node, the reliability is higher, and at the same time, it can ensure that the determined recovery node can achieve fault recovery for the failed process, thereby further improving the reliability of the fault recovery.

It should be noted that the meanings of A and/or B mentioned in the embodiments of the present invention include A, B, and A and B.

It should be understood that, in various embodiments of the present invention, the size of the sequence numbers of the above processes does not mean the order of execution, and the order of execution of each process should be determined by its function and internal logic, and should not be taken to the embodiments of the present invention. The implementation process constitutes any limitation.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the various examples described in connection with the implementations disclosed herein can be implemented in electronic hardware, or in combination with computer hardware and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods for implementing the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present invention.

A person skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the system, the device and the unit described above can refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the device embodiments described above are merely illustrative For example, the division of the unit is only a logical function division, and the actual implementation may have another division manner, for example, multiple units or components may be combined or may be integrated into another system, or some features may be Ignore, or not execute. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

The functions may be stored in a computer readable storage medium if implemented in the form of a software functional unit and sold or used as a standalone product. Based on such understanding, the technical solution of the present invention, which is essential or contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product, which is stored in a storage medium, including The instructions are used to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like. .

The above is only a specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of changes or substitutions within the technical scope of the present invention. It should be covered by the scope of the present invention. Therefore, the scope of the invention should be determined by the scope of the appended claims.

Claims

A method for fault recovery, characterized in that the method comprises:

Determining a size of a recovery file corresponding to each of the N failed processes, and determining an operating state of each of the M recovery nodes, wherein N≥1, M≥2;

Determining, according to the size of the recovery file corresponding to each failure process and the running state of each recovery node, the recovery node corresponding to each failure process, wherein the operation state includes a resource usage state;

The recovery nodes corresponding to each failed process are controlled to perform fault recovery for each failed process at the recovery node corresponding to each failed process.
The method according to claim 1, wherein the recovery file corresponding to the first invalid process of the N failed processes is stored in at least two storage nodes.
The method according to claim 2, wherein the recovery files corresponding to the first invalidation process stored in each of the storage nodes are the same.
The method according to claim 2, wherein the recovery file corresponding to the first invalidation process comprises at least two sub-recovery files, and the sub-recovery files stored in each of the storage nodes are different.
The method according to any one of claims 1 to 4, wherein, when N ≥ 2, each of the size of the recovery file corresponding to each failure process and the operation state of each recovery node is determined. The recovery node corresponding to the invalidation process, including:

According to the running state of each recovery node, based on the size of the recovery file corresponding to each invalidation process, the recovery nodes corresponding to each invalidation process are sequentially determined in descending order.
The method according to any one of claims 1 to 5, characterized in that the recovery node corresponding to a failed process is different from the storage node corresponding to the same invalidation process.
The method according to any one of claims 1 to 6, wherein the controlling according to the recovery node corresponding to each failure process comprises:

Estimating the recovery time of each failed process according to the running state of the recovery node corresponding to each failed process and the size of the recovery file corresponding to each failed process;

Control is performed according to the recovery time of each failed process.
A device for fault recovery, characterized in that the device comprises:

a determining unit, configured to determine a size of a recovery file corresponding to each of the N failed processes, and an operating state of each of the M recovery nodes, and according to each invalid process Determining a recovery node corresponding to each failure process, wherein the operation state includes a resource usage state, N≥1, M≥2;

The processing unit is configured to control the recovery node corresponding to each failed process to perform fault recovery for each failed process in the recovery node corresponding to each failed process.
The apparatus according to claim 8, wherein the recovery file corresponding to the first invalidation process of the N failed processes is stored in at least two storage nodes.
The apparatus according to claim 9, wherein the recovery files corresponding to the first invalidation process stored in each of the storage nodes are the same.
The apparatus according to claim 9, wherein the recovery file corresponding to the first invalidation process comprises at least two sub-recovery files, and the sub-recovery files stored in each of the storage nodes are different.
The device according to any one of claims 8 to 11, characterized in that, when N≥2, the determining unit is specifically configured to recover according to each failure process according to an operating state of each recovery node. The size of the file, in descending order, determines the recovery node corresponding to each failed process.
The apparatus according to any one of claims 8 to 12, characterized in that the recovery node corresponding to a failure process is different from the storage node corresponding to the same failure process.
The apparatus according to any one of claims 8 to 13, wherein the processing unit is specifically configured to: according to an operating state of the recovery node corresponding to each failure process and a recovery file corresponding to each failure process Size, estimate the recovery time of each failed process, and control according to the recovery time of each failed process.