CN110659148A - Distributed cluster data recovery method, device, equipment and storage medium - Google Patents
Distributed cluster data recovery method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN110659148A CN110659148A CN201910867318.2A CN201910867318A CN110659148A CN 110659148 A CN110659148 A CN 110659148A CN 201910867318 A CN201910867318 A CN 201910867318A CN 110659148 A CN110659148 A CN 110659148A
- Authority
- CN
- China
- Prior art keywords
- node
- storage
- recovery speed
- recovery
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000011084 recovery Methods 0.000 title claims abstract description 244
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000004590 computer program Methods 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 7
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 7
- 238000001514 detection method Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000002457 bidirectional effect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 101100121776 Arabidopsis thaliana GIG1 gene Proteins 0.000 description 1
- 101100267551 Saccharomyces cerevisiae (strain ATCC 204508 / S288c) YME1 gene Proteins 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 235000012054 meals Nutrition 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0727—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a storage system, e.g. in a DASD or network based storage system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distributed cluster data recovery method, which comprises the following steps: detecting the distributed storage cluster, and judging whether the distributed storage cluster fails; if yes, acquiring the number of objects to be recovered of nodes corresponding to each storage node in a target storage pool in the distributed storage cluster; acquiring the maximum recovery speed of the target storage pool, and calculating the node recovery speed corresponding to each storage node by using the number of objects to be recovered of each node and the maximum recovery speed; restoring the data in the corresponding storage nodes according to the restoring speed of each node; the method calculates the node recovery speed of the node, and recovers the data in the storage node according to the node recovery speed, thereby solving the problem that the existing distributed cluster data recovery method cannot balance the reading and writing of the user service without being influenced and the failure recovery speed is improved; in addition, the invention also provides a distributed cluster data recovery device, equipment and a readable storage medium, and the distributed cluster data recovery device, the equipment and the readable storage medium also have the beneficial effects.
Description
Technical Field
The present application relates to the field of distributed storage, and in particular, to a distributed cluster data recovery method, a distributed cluster data recovery apparatus, a distributed cluster data recovery device, and a computer-readable storage medium.
Background
With the rise and popularization of cloud computing technology, distributed shared storage systems are getting more and more concerned by the industry.
After the hard disk fails, data in the hard disk can be partially or completely lost, so data recovery needs to be performed after the hard disk fails, and the conventional data recovery technology sets a uniform data recovery speed for the whole storage cluster. In the fields of video monitoring, broadcast and television media resources and the like requiring continuous and stable storage systems, after a hard disk failure, great influence is generated on the use of users when data recovery is carried out, and sometimes, abnormal access to the hard disk is caused because the data recovery speed is too high, so that the data recovery speed is controlled, and the influence of the data recovery on the hard disk reading and writing service is reduced. However, when a disk failure and a node failure occur in the distributed storage cluster, in order to avoid data loss, data recovery needs to be completed as soon as possible, so that the data recovery speed needs to be increased to ensure that data recovery can be completed in a short time. Therefore, the current data recovery technology cannot balance the situation that the read-write service of the user is not affected and the fault recovery speed is improved, and cannot realize the situation that the use of the user is not affected and the data can be quickly recovered.
Therefore, how to solve the problem that the existing distributed cluster data recovery method cannot balance the read-write service of the user without being affected and improve the fault recovery speed is a technical problem to be solved by technical personnel in the field.
Disclosure of Invention
In view of the above, the present invention provides a distributed cluster data recovery method, a distributed cluster data recovery apparatus, a distributed cluster data recovery device, and a computer readable storage medium, which solve the problem that the existing distributed cluster data recovery method cannot achieve balance in that the read-write service of a user is not affected and the failure recovery speed is increased.
In order to solve the above technical problem, the present invention provides a distributed cluster data recovery method, including:
detecting a distributed storage cluster, and judging whether the distributed storage cluster fails or not;
if yes, acquiring the number of objects to be recovered of nodes corresponding to each storage node in a target storage pool in the distributed storage cluster;
acquiring the maximum recovery speed of the target storage pool, and calculating the node recovery speed corresponding to each storage node by using the number of objects to be recovered of each node and the maximum recovery speed;
and restoring the data in the corresponding storage node according to the node restoring speed respectively.
Optionally, calculating a node recovery speed corresponding to each storage node by using the number of objects to be recovered of each node and the maximum recovery speed includes:
acquiring the number of objects to be recovered in the target storage pool;
dividing the number of objects to be restored of the nodes corresponding to the storage nodes by the number of the objects to be restored to obtain the weight of the restoration speed corresponding to the storage nodes;
and multiplying the recovery speed weight by the maximum recovery speed of the target storage pool to obtain the node recovery speed corresponding to each storage node.
Optionally, the restoring the data in the corresponding storage node according to the node restoring speeds respectively includes:
sending each node recovery speed to the corresponding storage node;
and the storage node acquires backup data according to the corresponding node recovery speed and recovers the data.
Optionally, the obtaining of the maximum restoration speed corresponding to the target storage pool includes:
acquiring the read-write frequency of the target storage pool, and determining a target read-write frequency interval where the read-write frequency is located;
acquiring a target recovery speed threshold corresponding to the target read-write frequency interval by using a read-write frequency table; the reading and writing frequency table is a corresponding table of different reading and writing frequency intervals and different recovery speed thresholds;
determining the target recovery speed threshold as the maximum recovery speed.
The invention also provides a distributed cluster data recovery device, which comprises:
the fault judgment module is used for detecting the distributed storage cluster and judging whether the distributed storage cluster has a fault;
a node object number to be restored acquisition module, configured to acquire, if yes, the number of node objects to be restored corresponding to each storage node in a target storage pool in the distributed storage cluster;
the node recovery speed calculation module is used for acquiring the maximum recovery speed of the target storage pool and calculating the node recovery speed corresponding to each storage node by using the number of objects to be recovered of each node and the maximum recovery speed;
and the data recovery module is used for recovering the data in the corresponding storage node according to the node recovery speed respectively.
Optionally, the node recovery speed calculating module includes:
a to-be-restored object number obtaining unit, configured to obtain the number of objects to be restored in the target storage pool;
a recovery speed weight obtaining unit, configured to divide the number of objects to be recovered of the node corresponding to each storage node by the number of objects to be recovered to obtain a recovery speed weight corresponding to each storage node;
and the node recovery speed obtaining unit is used for multiplying the maximum recovery speed of the target storage pool by the recovery speed weight to obtain the node recovery speed corresponding to each storage node.
Optionally, the data recovery module includes:
a recovery speed sending unit, configured to send each node recovery speed to the corresponding storage node;
and the recovery unit is used for acquiring backup data by the storage node according to the corresponding node recovery speed and recovering the data.
Optionally, the node recovery speed calculating module includes:
a target read-write frequency interval determining unit, configured to obtain a read-write frequency of the target storage pool, and determine a target read-write frequency interval where the read-write frequency is located;
a recovery speed threshold obtaining unit, configured to obtain a target recovery speed threshold corresponding to the target read-write frequency interval by using the read-write frequency table; the reading and writing frequency table is a corresponding table of different reading and writing frequency intervals and different recovery speed thresholds;
a determining unit configured to determine the target recovery speed threshold as the maximum recovery speed.
The invention also provides a distributed cluster data recovery device, which comprises a memory and a processor, wherein:
the memory is used for storing a computer program;
the processor is configured to execute the computer program to implement the above-mentioned distributed cluster data recovery method.
The present invention also provides a computer readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the distributed cluster data recovery method described above.
The distributed cluster data recovery method provided by the invention detects the distributed storage cluster and judges whether the distributed storage cluster fails. And if so, acquiring the number of the objects to be recovered of the nodes corresponding to each storage node in the target storage pool in the distributed storage cluster. And acquiring the maximum recovery speed of the target storage pool, and calculating the node recovery speed corresponding to each storage node by using the number of objects to be recovered of each node and the maximum recovery speed. And respectively restoring the data in the corresponding storage nodes according to the restoring speed of each node.
It can be seen that, when it is determined that the distributed storage cluster has a failure, that is, when a storage pool in the distributed storage cluster has a failure, the method obtains the number of objects to be restored of the node in each storage node, that is, the number of objects to be restored in each node, calculates the node restoration speed of the node by using the number of objects to be restored of the node and the maximum restoration speed of the target storage pool, and restores the data in the storage node according to the node restoration speed. The maximum recovery speed of the storage pool can be determined according to the read-write frequency of the user, namely, a lower maximum recovery speed is determined for the storage pool with higher read-write frequency of the user, and a higher maximum recovery speed is determined for the storage pool with lower read-write frequency. Therefore, the data recovery speed of the whole distributed storage cluster can be improved, the use of users cannot be influenced, and the problem that the existing distributed cluster data recovery method cannot achieve balance in the aspects that the reading and writing of user services are not influenced and the fault recovery speed is improved is solved.
In addition, the invention also provides a distributed cluster data recovery device, distributed cluster data recovery equipment and a computer readable storage medium, and the distributed cluster data recovery device, the distributed cluster data recovery equipment and the computer readable storage medium also have the beneficial effects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a distributed cluster data recovery method according to an embodiment of the present invention;
fig. 2 is a flowchart of another distributed cluster data recovery method according to an embodiment of the present invention;
fig. 3 is a flowchart of another distributed cluster data recovery method according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a distributed cluster data recovery apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a distributed cluster data recovery device according to an embodiment of the present invention;
fig. 6 is a schematic working diagram of a distributed cluster data recovery device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart of a distributed cluster data recovery method according to an embodiment of the present invention. The method comprises the following steps:
s101: and detecting the distributed storage cluster, and judging whether the distributed storage cluster fails.
The distributed storage cluster comprises a plurality of storage pools, and each storage pool comprises a plurality of storage nodes with the same or different numbers. It should be noted that, in the present application, data may be respectively placed in different storage pools according to different read/write services of users for different data, that is, according to different read/write frequencies of users for different data. Specifically, data with high user read-write frequency may be placed in one or more storage pools, and data with low user read-write frequency may be placed in one or more storage pools. Further, in order to prevent the user from changing the read-write frequency of some data due to a change in the center of gravity of the service, the embodiment is preferable to establish a read-write frequency table, which includes a recovery speed threshold corresponding to each read-write frequency interval, so as to implement a function of automatically adjusting the data recovery speed.
The embodiment does not limit the frequency of detecting the distributed storage cluster, and for example, the distributed storage cluster may be detected in real time; or the distributed storage cluster can be detected once every preset detection time. And detecting the distributed storage cluster to judge whether the distributed storage cluster fails. It should be noted that, a detection method used when detecting the distributed storage cluster is related to a determination method for determining whether a failure occurs, and a specific content is not limited in this embodiment, for example, when the method for determining whether a failure occurs in the distributed storage cluster is to determine whether a bidirectional connection with the distributed storage cluster is disconnected, when the bidirectional connection is disconnected, it is determined that a failure occurs, the detection method is a connection signal between the detection method and the distributed storage cluster, and when the connection signal disappears, it may be determined that the bidirectional connection is disconnected, that is, the distributed storage cluster fails; when the method for judging whether the distributed storage cluster has the fault is to judge whether the distributed storage cluster sends an error signal, and when the error signal is sent, the fault is determined to occur, and the detection method is to detect whether the error signal sent by the distributed storage cluster is received.
When determining that the distributed storage cluster fails, the method may proceed to step S102; when it is determined that the distributed storage cluster has not failed, no operation may be performed, or the distributed storage cluster may be re-detected.
S102: and acquiring the number of objects to be recovered of the nodes corresponding to each storage node in the target storage pool in the distributed storage cluster.
In this embodiment, the target storage pool is a storage pool in which data is lost by a storage node, and any storage pool in which data needs to be recovered by a storage node may be used as the target storage pool. The number of the objects to be restored in the node is the number of the objects to be replied in each storage node, and the specific size of the number of the objects to be restored in the node is not limited in this embodiment. After determining that the distributed storage cluster has a fault, counting the number of objects lost in each storage node in the target storage pool, namely the number of objects to be recovered by the node corresponding to each storage node in the target storage pool, so as to calculate the node recovery speed of each storage node in the follow-up process. The frequency of acquiring the number of the objects to be restored of the node is not limited in this embodiment, and for example, the number of the objects to be restored of the node may be acquired in real time; or the operation of acquiring the number of the objects to be restored of the node may be performed according to a predetermined frequency.
S103: and acquiring the maximum recovery speed of the target storage pool, and calculating the node recovery speed corresponding to each storage node by using the number of objects to be recovered of each node and the maximum recovery speed.
The maximum recovery speed of the target storage pool is obtained, and it should be noted that there may be a variety of specific obtaining methods, for example, a technician may preset the maximum recovery speed for each storage pool according to the access frequency of the user to the data in the storage pool, and directly obtain the maximum recovery speed during obtaining; or corresponding recovery speed thresholds can be set according to different storage pool read-write frequencies, and further, the read-write frequency can be divided into a plurality of intervals, and different read-write frequency intervals correspond to different recovery speed thresholds. When the maximum recovery speed of the target storage pool needs to be obtained, the read-write frequency of the current target storage pool can be detected. And acquiring a corresponding recovery speed threshold value by using the read-write frequency. Specifically, referring to fig. 2, fig. 2 is a flowchart of another distributed cluster data recovery method according to an embodiment of the present invention, where the method includes:
s201: and acquiring the read-write frequency of the target storage pool, and determining a target read-write frequency interval where the read-write frequency is located.
In this embodiment, this step may be performed when a signal for acquiring the read-write frequency of the target storage pool is detected; or the read-write frequency of the target storage pool may be obtained immediately after it is determined that the meal is not a storage cluster failure, and the timing for obtaining the read-write frequency of the target storage pool is executed, which is not limited in this embodiment.
The embodiment does not limit the specific content of the read-write frequency, for example, the read-write frequency may be the read-write frequency of the target storage pool at the current time; or the read-write frequency may be an average value of all read-write frequencies of the target storage pool within a preset statistical time period, for example, the preset statistical time period is one hour, that is, when the read-write frequency of the target storage pool is obtained, the average value of the read-write frequencies of the target storage pool within one hour before the current time may be obtained; or the read-write frequency may be a weighted average of the read-write frequency at a particular time in the target storage pool within a predetermined statistical time period, for example, the predetermined statistical time period may be 24 hours, the particular time is three points in the morning, nine points in the morning, ten points in the morning, two points in the afternoon, five points in the afternoon, eight points in the afternoon, and ten points in the afternoon every day, the weights of the read-write frequencies of the target storage pools at nine points in the morning, ten points in the morning, and two points in the afternoon are set to 0.2, the weights of the read-write frequencies of the target storage pools at the remaining time are set to 0.1, and when it is determined that the distributed storage cluster has a failure, the weighted average is calculated by using the. And determining a target read-write frequency interval in which the read-write frequency is positioned by using the read-write frequency of the target storage pool.
S202: and acquiring a target recovery speed threshold corresponding to the target read-write frequency interval by using the read-write frequency table.
The read-write frequency table is a corresponding table of different read-write frequency intervals and the speed threshold value which does not pass the recovery. It should be noted that, when the read-write frequency is low, it indicates that the user access frequency is low, so that the data can be recovered at a higher recovery speed; when the read-write frequency is higher, it indicates that the user accesses more frequently, so a lower recovery speed is needed to recover the data, so as to avoid affecting the user. Therefore, a lower read-write frequency interval may correspond to a higher recovery speed threshold, and a higher read-write frequency interval may correspond to a higher recovery speed threshold. The specific corresponding relationship, the upper and lower thresholds and the interval width of the read-write frequency interval, and the size of the recovery speed threshold corresponding to each read-write frequency interval are not limited in this embodiment, and can be set according to actual conditions.
After the target read-write frequency interval corresponding to the read-write frequency is determined, a target recovery speed threshold corresponding to the target read-write frequency interval can be further determined.
S203: the target recovery speed threshold is determined to be the maximum recovery speed.
And determining the target recovery speed threshold as the maximum recovery speed of the target storage pool, and further calculating the node recovery speed of each storage node.
After obtaining the maximum restoration speed of the target storage pool, the node restoration speed corresponding to each storage node may be calculated by using the number of objects to be restored of each node and the maximum restoration speed, and a specific calculation process refers to fig. 3, where fig. 3 is a flowchart of another distributed cluster data restoration method provided by the embodiment of the present invention, and includes:
s301: and acquiring the number of objects to be restored in the target storage pool.
The embodiment does not limit the specific method for acquiring the number of the objects to be restored, and for example, the target storage pool may be directly detected, and the number of the objects to be restored in the whole target storage pool may be counted; or the number of the objects to be restored of the nodes corresponding to the storage nodes in the target storage pool can be added to obtain the number of the objects to be restored.
S302: and dividing the number of the objects to be recovered of the nodes corresponding to the storage nodes by the number of the objects to be recovered to obtain the recovery speed weight corresponding to each storage node.
And dividing the number of the objects to be recovered of the nodes of each storage node by the number of the objects to be recovered of the target storage pool to obtain the corresponding recovery speed weight of each storage node. For example, if the number of objects to be restored in the target storage pool is 300, the number of objects to be restored in the node of the first storage node is 100, the number of objects to be restored in the node of the second storage node is 50, and the number of objects to be restored in the node of the third storage node is 150, the restoration speed weight corresponding to the first storage node is 0.33, the restoration speed weight corresponding to the second storage node is 0.17, and the restoration speed weight corresponding to the third storage node is 0.5.
S303: and multiplying the recovery speed weight by the maximum recovery speed of the target storage pool respectively to obtain the node recovery speed corresponding to each storage node.
And multiplying the recovery speed weight corresponding to each storage node by the maximum recovery speed of the target storage pool to obtain the node recovery speed corresponding to each storage node. For example, when the maximum restoration rate is 500M/s, the node restoration rate of the first storage node is 165M/s, the node restoration rate of the second storage node is 85M/s, and the node restoration rate of the third storage node is 250M/s.
The present embodiment does not limit the frequency of calculating the node recovery speed, and for example, the node recovery speed of each storage node may be calculated in real time; or the node recovery speed of each storage node can be calculated according to a preset period; or the recovery speed of each individual node stored in the storage can be calculated once every time the number of the objects to be recovered of the node is acquired.
S104: and respectively restoring the data in the corresponding storage nodes according to the restoring speed of each node.
Specifically, after the node recovery speed corresponding to each storage node is obtained through calculation, the node recovery speed is sent to the corresponding storage node, and the storage node acquires backup data according to the corresponding node recovery speed and recovers the data in the storage node by using the backup data.
By applying the distributed cluster data recovery method provided by the embodiment of the invention, when the distributed storage cluster is determined to have a fault, namely when a storage pool in the distributed storage cluster has a fault, the number of objects to be recovered of nodes in each storage node, namely the number of objects needing data recovery in each node, is obtained, the node recovery speed of the node is calculated by using the number of the objects to be recovered of the node and the maximum recovery speed of a target storage pool, and the data in the storage nodes are recovered according to the node recovery speed. The maximum recovery speed of the storage pool can be determined according to the read-write frequency of the user, namely, a lower maximum recovery speed is determined for the storage pool with higher read-write frequency of the user, and a higher maximum recovery speed is determined for the storage pool with lower read-write frequency. Therefore, the data recovery speed of the whole distributed storage cluster can be improved, the use of users cannot be influenced, and the problem that the existing distributed cluster data recovery method cannot achieve balance in the aspects that the reading and writing of user services are not influenced and the fault recovery speed is improved is solved.
In the following, the distributed cluster data recovery apparatus provided in the embodiment of the present invention is introduced, and the distributed cluster data recovery apparatus described below and the distributed cluster data recovery method described above may be referred to correspondingly.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a distributed cluster data recovery apparatus according to an embodiment of the present invention, including:
a failure determining module 410, configured to detect the distributed storage cluster and determine whether the distributed storage cluster fails;
a node object to be restored number obtaining module 420, configured to, if yes, obtain the number of node object to be restored corresponding to each storage node in the target storage pool in the distributed storage cluster;
the node recovery speed calculation module 430 is configured to obtain a maximum recovery speed of the target storage pool, and calculate a node recovery speed corresponding to each storage node by using the number of objects to be recovered of each node and the maximum recovery speed;
and the data recovery module 440 is configured to recover the data in the corresponding storage node according to the recovery speed of each node.
Optionally, the node recovery speed calculating module 430 includes:
the device comprises a to-be-recovered object number obtaining unit, a to-be-recovered object number obtaining unit and a recovery unit, wherein the to-be-recovered object number obtaining unit is used for obtaining the to-be-recovered object number of a target storage pool;
a recovery speed weight obtaining unit, configured to divide the number of objects to be recovered by the number of nodes corresponding to each storage node by the number of objects to be recovered to obtain a recovery speed weight corresponding to each storage node;
and the node recovery speed acquisition unit is used for multiplying the maximum recovery speed of the target storage pool by the recovery speed weight respectively to obtain the node recovery speed corresponding to each storage node.
Optionally, the data recovery module 440 includes:
the recovery speed sending unit is used for sending the recovery speed of each node to the corresponding storage node;
and the recovery unit is used for the storage node to acquire the backup data according to the corresponding node recovery speed and recover the data.
Optionally, the node recovery speed calculating module 430 includes:
the target read-write frequency interval determining unit is used for acquiring the read-write frequency of the target storage pool and determining a target read-write frequency interval where the read-write frequency is located;
a recovery speed threshold obtaining unit, configured to obtain a target recovery speed threshold corresponding to a target read-write frequency interval by using the read-write frequency table; the reading and writing frequency table is a corresponding table of different reading and writing frequency intervals and different recovery speed thresholds;
a determination unit for determining the target recovery speed threshold as the maximum recovery speed.
In the following, the distributed cluster data recovery device provided in the embodiment of the present invention is introduced, and the distributed cluster data recovery device described below and the distributed cluster data recovery method described above may be referred to correspondingly.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a distributed cluster data recovery device according to an embodiment of the present invention, where the distributed cluster data recovery device includes a memory and a processor, where:
a memory 510 for storing a computer program;
processor 520 is configured to execute a computer program to implement the above-described distributed cluster data recovery method.
In practical use, reference may also be made to fig. 6, where fig. 6 is a schematic working diagram of a distributed cluster data recovery device provided in an embodiment of the present invention, and the schematic working diagram includes: the system comprises a distributed cluster data recovery device MON, a user machine Client, a storage pool A and a storage pool B. MON, Client, A and B are all connected via a public network. The storage pool a comprises three storage nodes of OSD1, OSD3 and OSD4, and the storage pool B comprises three storage nodes of OSD2, OSD5 and OSD 6.
In the following, the computer-readable storage medium provided by the embodiment of the present invention is introduced, and the computer-readable storage medium described below and the distributed cluster data recovery method described above may be referred to correspondingly.
The present invention also provides a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the above-mentioned distributed cluster data recovery method.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it should also be noted that, herein, relationships such as first and second, etc., are intended only to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
The distributed cluster data recovery method, the distributed cluster data recovery device, the distributed cluster data recovery apparatus, and the computer readable storage medium provided by the present invention are described in detail above, and specific examples are applied in this document to explain the principle and the implementation of the present invention, and the description of the above embodiments is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. A distributed cluster data recovery method is characterized by comprising the following steps:
detecting a distributed storage cluster, and judging whether the distributed storage cluster fails or not;
if yes, acquiring the number of objects to be recovered of nodes corresponding to each storage node in a target storage pool in the distributed storage cluster;
acquiring the maximum recovery speed of the target storage pool, and calculating the node recovery speed corresponding to each storage node by using the number of objects to be recovered of each node and the maximum recovery speed;
and restoring the data in the corresponding storage node according to the node restoring speed respectively.
2. The distributed cluster data recovery method of claim 1, wherein calculating the node recovery speed corresponding to each storage node by using the number of objects to be recovered of each node and the maximum recovery speed comprises:
acquiring the number of objects to be recovered in the target storage pool;
dividing the number of objects to be restored of the nodes corresponding to the storage nodes by the number of the objects to be restored to obtain the weight of the restoration speed corresponding to the storage nodes;
and multiplying the recovery speed weight by the maximum recovery speed of the target storage pool to obtain the node recovery speed corresponding to each storage node.
3. The distributed cluster data recovery method according to claim 2, wherein recovering the data in the corresponding storage node according to each node recovery speed includes:
sending each node recovery speed to the corresponding storage node;
and the storage node acquires backup data according to the corresponding node recovery speed and recovers the data.
4. The distributed cluster data recovery method of any of claims 1-3, wherein obtaining the maximum recovery speed corresponding to the target storage pool comprises:
acquiring the read-write frequency of the target storage pool, and determining a target read-write frequency interval where the read-write frequency is located;
acquiring a target recovery speed threshold corresponding to the target read-write frequency interval by using a read-write frequency table; the reading and writing frequency table is a corresponding table of different reading and writing frequency intervals and different recovery speed thresholds;
determining the target recovery speed threshold as the maximum recovery speed.
5. A distributed cluster data recovery apparatus, comprising:
the fault judgment module is used for detecting the distributed storage cluster and judging whether the distributed storage cluster has a fault;
a node object number to be restored acquisition module, configured to acquire, if yes, the number of node objects to be restored corresponding to each storage node in a target storage pool in the distributed storage cluster;
the node recovery speed calculation module is used for acquiring the maximum recovery speed of the target storage pool and calculating the node recovery speed corresponding to each storage node by using the number of objects to be recovered of each node and the maximum recovery speed;
and the data recovery module is used for recovering the data in the corresponding storage node according to the node recovery speed respectively.
6. The distributed cluster data recovery apparatus of claim 5, wherein the node recovery speed calculation module comprises:
a to-be-restored object number obtaining unit, configured to obtain the number of objects to be restored in the target storage pool;
a recovery speed weight obtaining unit, configured to divide the number of objects to be recovered of the node corresponding to each storage node by the number of objects to be recovered to obtain a recovery speed weight corresponding to each storage node;
and the node recovery speed obtaining unit is used for multiplying the maximum recovery speed of the target storage pool by the recovery speed weight to obtain the node recovery speed corresponding to each storage node.
7. The distributed cluster data recovery module of claim 5, wherein the data recovery module comprises:
a recovery speed sending unit, configured to send each node recovery speed to the corresponding storage node;
and the recovery unit is used for acquiring backup data by the storage node according to the corresponding node recovery speed and recovering the data.
8. The distributed cluster data recovery module of claim 5, wherein the node recovery speed calculation module comprises:
a target read-write frequency interval determining unit, configured to obtain a read-write frequency of the target storage pool, and determine a target read-write frequency interval where the read-write frequency is located;
a recovery speed threshold obtaining unit, configured to obtain a target recovery speed threshold corresponding to the target read-write frequency interval by using the read-write frequency table; the reading and writing frequency table is a corresponding table of different reading and writing frequency intervals and different recovery speed thresholds;
a determining unit configured to determine the target recovery speed threshold as the maximum recovery speed.
9. A distributed cluster data recovery device comprising a memory and a processor, wherein:
the memory is used for storing a computer program;
the processor is configured to execute the computer program to implement the distributed cluster data recovery method according to any one of claims 1 to 4.
10. A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the distributed cluster data recovery method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910867318.2A CN110659148A (en) | 2019-09-12 | 2019-09-12 | Distributed cluster data recovery method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910867318.2A CN110659148A (en) | 2019-09-12 | 2019-09-12 | Distributed cluster data recovery method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110659148A true CN110659148A (en) | 2020-01-07 |
Family
ID=69038095
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910867318.2A Withdrawn CN110659148A (en) | 2019-09-12 | 2019-09-12 | Distributed cluster data recovery method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110659148A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112269532A (en) * | 2020-10-15 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Statistical method, system and device for reconstruction progress of distributed storage cluster |
| CN113535477A (en) * | 2021-07-15 | 2021-10-22 | 上海上讯信息技术股份有限公司 | Method and equipment for data disaster recovery |
| CN114064362A (en) * | 2021-11-16 | 2022-02-18 | 北京志凌海纳科技有限公司 | Data recovery method, system and computer readable storage medium for distributed storage |
| CN114428760A (en) * | 2021-12-30 | 2022-05-03 | 北京云宽志业网络技术有限公司 | Cluster storage system and metadata recovery method |
| CN114443431A (en) * | 2022-01-25 | 2022-05-06 | 苏州浪潮智能科技有限公司 | Distributed storage system fault recovery method, device, equipment and medium |
| CN117851132A (en) * | 2024-03-07 | 2024-04-09 | 四川省华存智谷科技有限责任公司 | Data recovery optimization method for distributed object storage |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108958975A (en) * | 2018-06-29 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of method, device and equipment controlling data resume speed |
| CN109117306A (en) * | 2018-07-24 | 2019-01-01 | 广东浪潮大数据研究有限公司 | A kind of method and device based on object read-write time delay adjustment data resume speed |
-
2019
- 2019-09-12 CN CN201910867318.2A patent/CN110659148A/en not_active Withdrawn
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108958975A (en) * | 2018-06-29 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of method, device and equipment controlling data resume speed |
| CN109117306A (en) * | 2018-07-24 | 2019-01-01 | 广东浪潮大数据研究有限公司 | A kind of method and device based on object read-write time delay adjustment data resume speed |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112269532A (en) * | 2020-10-15 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Statistical method, system and device for reconstruction progress of distributed storage cluster |
| CN112269532B (en) * | 2020-10-15 | 2022-07-29 | 苏州浪潮智能科技有限公司 | Statistical method, system and device for reconstruction progress of distributed storage cluster |
| CN113535477A (en) * | 2021-07-15 | 2021-10-22 | 上海上讯信息技术股份有限公司 | Method and equipment for data disaster recovery |
| CN114064362A (en) * | 2021-11-16 | 2022-02-18 | 北京志凌海纳科技有限公司 | Data recovery method, system and computer readable storage medium for distributed storage |
| CN114064362B (en) * | 2021-11-16 | 2022-08-05 | 北京志凌海纳科技有限公司 | Data recovery method, system and computer readable storage medium for distributed storage |
| CN114428760A (en) * | 2021-12-30 | 2022-05-03 | 北京云宽志业网络技术有限公司 | Cluster storage system and metadata recovery method |
| CN114443431A (en) * | 2022-01-25 | 2022-05-06 | 苏州浪潮智能科技有限公司 | Distributed storage system fault recovery method, device, equipment and medium |
| CN114443431B (en) * | 2022-01-25 | 2024-05-10 | 苏州浪潮智能科技有限公司 | Distributed storage system fault recovery method, device, equipment and medium |
| CN117851132A (en) * | 2024-03-07 | 2024-04-09 | 四川省华存智谷科技有限责任公司 | Data recovery optimization method for distributed object storage |
| CN117851132B (en) * | 2024-03-07 | 2024-05-07 | 四川省华存智谷科技有限责任公司 | Data recovery optimization method for distributed object storage |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110659148A (en) | Distributed cluster data recovery method, device, equipment and storage medium | |
| CN104519032B (en) | A kind of security strategy and system of internet account number | |
| CN106201829B (en) | Monitor Threshold and device, monitoring alarm method, apparatus and system | |
| CN107819631B (en) | Equipment anomaly detection method, device and equipment | |
| CN106548402B (en) | Resource transfer monitoring method and device | |
| CN107481090A (en) | A kind of user's anomaly detection method, device and system | |
| CN104506392B (en) | A kind of delay machine detection method and equipment | |
| CN110601900A (en) | Network fault early warning method and device | |
| CN110650060A (en) | Processing method, equipment and storage medium for flow alarm | |
| CN108540533B (en) | Request answering method and device | |
| CN107026708B (en) | Data processing method and device | |
| CN108924202B (en) | Distributed cluster data disaster tolerance method and related device | |
| CN107402851B (en) | Data recovery control method and device | |
| CN112636979B (en) | Cluster alarm method and related device | |
| CN109597800B (en) | Log distribution method and device | |
| CN110580234A (en) | Fusing method suitable for micro server and external system | |
| CN109144782B (en) | Data recovery method and device | |
| CN113903389A (en) | Slow disk detection method and device and computer readable and writable storage medium | |
| CN110740064A (en) | Distributed cluster node fault processing method, device, equipment and storage medium | |
| CN110955544A (en) | Method, device and system for detecting usability of web system | |
| CN108256311B (en) | Authorization authentication method and device and electronic equipment | |
| CN112100029B (en) | Client state monitoring method and related components | |
| CN109861843A (en) | Complete acquisition confirmation method, device and the equipment of journal file | |
| CN114666390B (en) | Page monitoring method and device of application program, electronic equipment and storage medium | |
| CN113676365A (en) | Access request processing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200107 |
|
| WW01 | Invention patent application withdrawn after publication |