CN111400098B - Copy management method and device, electronic equipment and storage medium - Google Patents
Copy management method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111400098B CN111400098B CN202010181063.7A CN202010181063A CN111400098B CN 111400098 B CN111400098 B CN 111400098B CN 202010181063 A CN202010181063 A CN 202010181063A CN 111400098 B CN111400098 B CN 111400098B
- Authority
- CN
- China
- Prior art keywords
- copy
- host
- data
- instruction
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007726 management method Methods 0.000 title claims abstract description 116
- 238000003860 storage Methods 0.000 title claims abstract description 20
- 230000001360 synchronised effect Effects 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 8
- 238000011156 evaluation Methods 0.000 claims description 8
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000000034 method Methods 0.000 description 37
- 230000008569 process Effects 0.000 description 18
- 230000007246 mechanism Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 238000013508 migration Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1469—Backup restoration techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a copy management method, which comprises the steps of determining a data copy in a file system and setting the currently accessed data copy as a read source copy; setting the data copy except the read source copy as a target data copy; if a copy management instruction is received, copying the target data copy to a host corresponding to the copy management instruction; the copy management instruction comprises a copy reconstruction instruction, a copy synchronization instruction or a load balancing instruction. The application can improve the copy management efficiency and reduce the influence of copy management operation on executing normal business. The application also discloses a copy management device, a storage medium and an electronic device, which have the beneficial effects.
Description
Technical Field
The present application relates to the field of data storage technologies, and in particular, to a method and apparatus for managing copies, an electronic device, and a storage medium.
Background
Distributed storage (Distributed Storage) is where data is stored in a decentralized manner on a plurality of individual devices. The distributed network storage system adopts an expandable system structure, utilizes a plurality of storage servers to share the storage load, and utilizes the position servers to position the storage information, thereby improving the reliability, availability and access efficiency of the system and being easy to expand.
In the related art, if the distributed file system receives a copy management instruction (such as a copy reconstruction instruction, a copy synchronization instruction, or a load balancing instruction), it needs to determine a target copy corresponding to the copy management instruction, then copy the data from another good copy to the newly created copy, and the reconstruction is completed after the data copy is completed. However, the service cannot be interrupted in the process of executing the copy management instruction, which means that the service is always operated in the reconstruction process, if the good copy selected at the moment is exactly the copy being accessed by the virtual machine, the disk corresponding to the copy can become random IO (virtual machine reading and reconstruction reading are simultaneously performed), the copy management efficiency is reduced, and the normal execution of the service is also affected.
Therefore, how to improve the efficiency of copy management and reduce the influence of copy management operations on performing normal services is a technical problem that needs to be solved by those skilled in the art.
Disclosure of Invention
The application aims to provide a copy management method, a copy management device, electronic equipment and a storage medium device, which can improve copy management efficiency and reduce the influence of copy management operation on normal business execution.
In order to solve the technical problems, the present application provides a copy management method, which includes:
Determining a data copy in a file system, and setting the currently accessed data copy as a read source copy;
Setting the data copy except the read source copy as a target data copy;
if a copy management instruction is received, copying the target data copy to a host corresponding to the copy management instruction; the copy management instruction comprises a copy reconstruction instruction, a copy synchronization instruction or a load balancing instruction.
Optionally, if the copy management instruction is the copy reconstruction instruction, copying the target data copy to the host corresponding to the copy management instruction includes:
determining a target host in which the data copy is located, and determining an idle host from hosts in the file system except the target host;
and selecting a copy reconstruction source from the target data copy, and copying data in the copy reconstruction source to the idle host to complete the data copy reconstruction operation.
Optionally, determining an idle host from hosts in the file system other than the target host includes:
Performing priority evaluation on hosts except the target host in the file system according to the host capacity parameter and the host performance parameter;
And determining the idle host according to the priority evaluation result.
Optionally, if the copy management instruction is the copy synchronization instruction, copying the target data copy to the host corresponding to the copy management instruction includes:
Selecting a copy synchronization source from the target data copy, and comparing the data in the copy synchronization source with the data of the data copy in the host to be synchronized;
executing data synchronization operation on the data copies in the host to be synchronized according to the data comparison result; and the data of the data copy in the host to be synchronized after the data synchronization operation is the same as the data in the copy synchronization source.
Optionally, if the copy management instruction is the load balancing instruction, copying the target data copy to the host corresponding to the copy management instruction includes:
Determining an original host and a capacity expansion host according to the load balancing instruction; the original host is the host with the largest disk load in the hosts where all the data copies are located;
copying the data copy in the original host to the capacity expansion host, and deleting the data copy in the original host after the data copy is copied.
Optionally, the method further comprises:
if the target host where the data copy is located fails, judging whether the failed target host is the host where the read source copy is located;
If yes, switching the current accessed data copy, and redefining a new read source copy and a new target data copy, and generating a copy reconstruction instruction or a copy synchronization instruction according to the fault duration;
And if not, generating the copy reconstruction instruction or the copy synchronization instruction according to the fault duration.
Optionally, generating the duplicate reconstruction instruction or the duplicate synchronization instruction according to the fault duration includes:
Judging whether the fault duration time is longer than a preset duration time or not;
If yes, generating the copy reconstruction instruction;
and if not, generating the copy synchronization instruction.
The application also provides a copy management device, which comprises:
The read source copy setting module is used for determining the data copy in the file system and setting the currently accessed data copy as the read source copy;
the target copy setting module is used for setting the data copy except the read source copy as a target data copy;
The copy management module is used for copying the target data copy to a host corresponding to the copy management instruction if the copy management instruction is received; the copy management instruction comprises a copy reconstruction instruction, a copy synchronization instruction or a load balancing instruction.
The present application also provides a storage medium having stored thereon a computer program which, when executed, implements the steps of the above-described copy management method.
The application also provides an electronic device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the steps executed by the copy management method when calling the computer program in the memory.
The application provides a copy management method, which comprises the steps of determining a data copy in a file system and setting the currently accessed data copy as a read source copy; setting the data copy except the read source copy as a target data copy; if a copy management instruction is received, copying the target data copy to a host corresponding to the copy management instruction; the copy management instruction comprises a copy reconstruction instruction, a copy synchronization instruction or a load balancing instruction.
The application firstly determines the data copy existing in the distributed system, and takes the currently accessed data copy as a read source copy. After receiving the copy management instruction, selecting a target data copy from other copies except the read source copy, copying the target data copy to a host corresponding to the copy management instruction, and further completing copy management operation. The application does not take the current accessed data copy as the target data copy for data copy, and can avoid the condition that the data read by the service and the data copied in the copy management process are the data in the same copy, thereby improving the copy management efficiency and reducing the influence of the copy management operation on the execution of normal service. The application also provides a copy management device, a storage medium and an electronic device, which have the beneficial effects and are not described in detail herein.
Drawings
For a clearer description of embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described, it being apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to the drawings without inventive effort for those skilled in the art.
FIG. 1 is a flowchart of a copy management method according to an embodiment of the present application;
FIG. 2 is a flowchart of a copy reconstruction method according to an embodiment of the present application;
FIG. 3 is a flowchart of a method for synchronizing copies according to an embodiment of the present application;
fig. 4 is a flowchart of a load balancing method according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a distributed file system according to an embodiment of the present application;
Fig. 6 is a schematic structural diagram of a copy management apparatus according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Referring to fig. 1, fig. 1 is a flowchart of a copy management method according to an embodiment of the present application.
The specific steps may include:
s101: determining a data copy in a file system, and setting the currently accessed data copy as a read source copy;
The embodiment can be applied to a file system, wherein the file system can be a distributed file system or a cluster file system, the file system can comprise a plurality of hosts, each host can be operated with a virtual machine, and the virtual machine can read data copies stored in the hosts so as to provide corresponding services. As a possible implementation manner, the distributed file system mentioned in this embodiment may be a system adopting a super-fusion architecture (Hyper-Converged Infrastructure, HCI), where the super-fusion architecture not only includes resources and technologies such as computing, networking, storage, and server virtualization in the same unit device (x 86 server), but also includes elements such as cache acceleration, data de-duplication, online data compression, backup software, and snapshot technology, and multiple nodes may be aggregated through the network to implement modularized seamless lateral expansion (scale-out) to form a unified resource pool. In a distributed file system, a high reliability method can be obtained by saving data in multiple copies. For example, the data is stored in three copies in three different hosts. The present embodiment does not limit the number of copies of the storage manner employed by the distributed file system.
In this step, first, the data copies in the file system are determined, the number of the data copies may be multiple, and under the condition that the hosts where all the data copies are located operate normally, the data stored in any two data copies are the same. The copy of the data currently being accessed referred to in this step is the copy being accessed by the virtual machine running in the host, and may be set as the read source copy. The read source copy refers to a copy being read in the copy group, for example, the copy being accessed by the virtual machine, or the copy being currently read by a process.
S102: setting data copies except the read source copy as target data copies;
The data copy except the read source copy is set as the target data copy in the step, and the operation mode of the step is illustrated. For example, the file system includes a host a, a host B, and a host C, where the host a stores a data copy a, the host B stores a data copy B, and the host C stores a data copy C, and if the virtual machine in the host a is reading the data copy a at this time, the data copy B and the data copy C may be set as target data copies at this time.
As a possible implementation manner, the present embodiment may have an operation of updating the target data copy according to a preset period, so as to determine the current read source copy and the current target data copy.
S103: if the copy management instruction is received, copying the target data copy to a host corresponding to the copy management instruction;
Before this step, there may be an operation of determining whether a copy management instruction is received, where the copy management instruction may be a copy rebuilding instruction, a copy synchronization instruction, or a load balancing instruction. Specifically, if the copy management instruction is a copy reconstruction instruction, a new host that does not store the data copy may be used as a host corresponding to the copy reconstruction instruction, and the target data copy may be copied to the new host. If the copy management instruction is a copy synchronization instruction, the data copy to be synchronized can be determined, and the target data copy is copied into the data copy to be synchronized. If the copy management instruction is a load balancing instruction, determining an original host and an expansion host according to the load balancing instruction, wherein the original host is the host where the target data copy is located, and the expansion host is the newly added host of the distributed file system, so that the data on the original host can be copied into the expansion host. It should be noted that, because the data stored in the target data copies are the same, the embodiment may select any normal data copy from all the target data copies to copy to the host corresponding to the copy management instruction.
Further, copy reconstruction (Rebuild) refers to that under the multi-copy mechanism, when one copy is in a failure of a resource, the copy data is re-created on other available resources, and the multi-copy redundancy state is finally achieved by copying correct data from other available copies to the newly created copy. The load balancing instruction needs to implement a copy migration operation, where copy migration (Migrate) refers to the purpose of moving data of one copy to another resource group under a multi-copy mechanism, so as to release the source resource group space. The replica synchronization (Heal) refers to that under the multi-replica mechanism, when the replicas fail, the replica data is inconsistent, and after the failure recovery, the data needs to be synchronously copied from the good replicas to the failed replicas, so that the purpose of data consistency is achieved.
The embodiment first determines the data copy existing in the distributed system, and takes the currently accessed data copy as the read source copy. After receiving the copy management instruction, selecting a target data copy from other copies except the read source copy, copying the target data copy to a host corresponding to the copy management instruction, and further completing copy management operation. Because the current accessed data copy is not used as the target data copy for data copying, the situation that the data read by the service and the data copied in the copy management process are the data in the same copy can be avoided, so that the copy management efficiency can be improved, and the influence of the copy management operation on the execution of normal service can be reduced.
As a further introduction to the corresponding embodiment of fig. 1, this embodiment may further include an operation of determining whether the target host fails, and if the target host where the data copy is located fails, determining whether the failed target host is the host where the read source copy is located; if yes, switching the current accessed data copy, and redefining a new read source copy and a new target data copy, and generating a copy reconstruction instruction or a copy synchronization instruction according to the fault duration; if not, generating a copy reconstruction instruction or a copy synchronization instruction according to the fault duration.
Specifically, the procedure for generating the copy reconstruction command or the copy synchronization command according to the fault duration is specifically as follows: judging whether the duration time of the fault is longer than a preset duration time; if yes, generating a copy reconstruction instruction; if not, generating a copy synchronization instruction.
Referring to fig. 2, fig. 2 is a flowchart of a copy reconstruction method provided in an embodiment of the present application, where the embodiment is further described with reference to S103 in the corresponding embodiment of fig. 1, if the copy management instruction is a copy reconstruction instruction, the operation of copying the target data copy to the host corresponding to the copy management instruction may include the following steps:
s201: determining a target host where the data copy is located, and determining an idle host from hosts except the target host in the file system;
The step can determine the target host where all the data copies are located, and thus the corresponding relationship between the data copies and the target host is obtained. The file system may include a plurality of hosts, one part of the hosts having data copies stored thereon and another part of the hosts not having data copies stored thereon, and the step determines a free host from the hosts not having data copies stored thereon, that is, determines a free host from hosts in the file system other than the target host.
S202: selecting a copy reconstruction source from the target data copies;
any one of all target data copies can be selected as a copy reconstruction source in the step.
S203: copying the data in the copy reconstruction source to an idle host to finish the data copy reconstruction operation.
In this embodiment, the copy reconstruction source is not the current read data copy, so that the situation that the data of the data copy is accessed and copied at the same time can be avoided.
As a further introduction to the above embodiment, the present embodiment may also determine an idle host by: and carrying out priority evaluation on the hosts except the target host in the file system according to the host capacity parameter and the host performance parameter, and determining an idle host according to the priority evaluation result. In the above manner, the host with the largest host capacity and the optimal host performance can be preferentially selected as the idle host.
Referring to fig. 3, fig. 3 is a flowchart of a copy synchronization method provided in an embodiment of the present application, where the embodiment is further described with reference to S103 in the corresponding embodiment of fig. 1, if the copy management instruction is a copy synchronization instruction, the operation of copying the target data copy to the host corresponding to the copy management instruction may include the following steps:
step S301: a copy synchronization source is selected from the target data copy.
Step S302: and comparing the data in the copy synchronization source with the data of the data copy in the host to be synchronized.
Step S303: and executing data synchronization operation on the data copies in the host to be synchronized according to the data comparison result.
The data stored in the target data copies are the same, so that any normal data copy can be selected from all the target data copies to serve as a copy synchronization source. The data to be synchronized can be determined by comparing the data in the copy synchronization source with the data of the data copy in the host to be synchronized, and a data comparison result is obtained. And after the data synchronization operation is carried out on the data copies in the host to be synchronized according to the data comparison result, the data of the data copies in the host to be synchronized are identical to the data in the copy synchronization source.
Referring to fig. 4, fig. 4 is a flowchart of a load balancing method according to an embodiment of the present application, where the embodiment is further described with reference to S103 in the corresponding embodiment of fig. 1, if the copy management instruction is a load balancing instruction, the operation of copying the target data copy to the host corresponding to the copy management instruction may include the following steps:
s401: receiving a load balancing instruction;
s402: determining an original host and a capacity expansion host according to the load balancing instruction;
The original host is the host with the largest disk load in the hosts where all the data copies are located;
S403: copying the data copy in the original host to the capacity expansion host, and deleting the data copy in the original host after the data copy is copied.
In this embodiment, the host with the largest disk load is used as the original host, and the newly added capacity-expanding host is used for remembering load balancing, so as to improve the overall stability of the distributed file system.
The flow described in the above embodiment is explained below by way of an embodiment in practical application.
The existing super fusion system is more and more widely applied to governments, enterprises, schools and the like, and plays an important role in various production business systems. There are incomparable advantages over conventional pure hardware systems and conventional compute-store-separated virtualized systems. Firstly, a distributed architecture is provided, and single-point faults are avoided; secondly, the intelligent HDD/SSD is mixed for storage, so that I/O performance is extremely high; thirdly, a platform is unified and managed, and operation and maintenance are simplified. For financial industry and medical, large enterprise and government clients, the reliability requirement is generally higher, the capability of 'fault tolerance 2' is generally required, the medical, large enterprise and government clients also have higher reliability requirement on a core business system, the fault tolerance 2 is required, and most manufacturers now provide three cost mechanisms. For a host scene of a three-copy mechanism, 5 copies are uniformly distributed on 5 different physical disks, so that the service operation is not influenced even if 2 disks are physically damaged; for a five-host scene, 5 copies are uniformly distributed on 5 different physical hosts, so that service operation is not affected even if 2 hosts are physically damaged, and the fault tolerance of the 3+2 copies is fully embodied.
The service cannot be interrupted in the process of reconstructing the copy of the distributed file system, which means that the service in the reconstruction process is always operated, and at the moment, the key point is how to select the copy of the read source (namely, the good copy of the read data in the above description), if the good copy selected at the moment is exactly the copy being accessed by the virtual machine, the disk corresponding to the copy can be changed into random IO (the virtual machine reading and the reconstruction reading are simultaneously carried out), and the service of the virtual machine is greatly influenced due to the large reconstruction flow; and the reconstruction itself is a large block sequential reading, at this time, the read of the service can cause the IO of the disk where the reconstruction is located to become random, thereby causing the reconstruction speed to be greatly reduced, and the reconstruction service is greatly affected. Based on the problem, the embodiment provides a service perception scheme in the reconstruction process, wherein a service access copy corresponding to a reconstruction fragment is found at first when reconstruction starts, and then the copy is removed when a reconstruction read source copy is selected, so that the reconstruction read source and the service read source can be completely isolated, the sequential throughput of a reconstructed block can be ensured, and the service can be ensured not to be influenced by reconstruction tasks.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a distributed file system according to an embodiment of the present application, and virtual machine copies are distributed on the hosts 2-6. Wherein, the host 2-host 4 are data copies, virtual machine data is stored, the host 5 and the host 6 are arbitration copies, and the control information of the data is stored; the virtual machine is running on top of the host 3 and is reading the copy data on top of the host 3.
When the virtual machine is pulled up by the host 3, the copy is marked by storing the copy of the data, which identifies the read source of the virtual machine as the host 3, and the copy of the file accessed by the service is recorded on the host 3. It will be appreciated that copies will not be marked if the virtual machine is not running.
If a failure occurs in the host 4 during the running process of the virtual machine, the virtual machine has two complete data copies and two complete arbitration copies, the storage can still provide service according to a fault tolerance algorithm, but when the failure reaches a certain threshold value, a reconstruction strategy is triggered, at this time, the copies are reconstructed to the hosts which can provide service in the cluster, a distribution algorithm under the scene selects the host 1 as a reconstruction purpose according to the conditions such as capacity and the like, and then the stored data plane reconstruction logic is triggered.
The duplicate reconstruction process is as follows: firstly, whether a copy is marked as being read or not is acquired (for example, the copy on the host 3 is acquired under the scene and is being read by a virtual machine), the copy which is being read is removed from the copy which can be used as a reconstruction source (for example, under the scene, the copy which can be reconstructed originally is the copy on the host 2 and the host 3, then the host 3 in the step 1 is removed), the copy in the host 2 is selected as the reconstruction source, and the data is completely copied from the reconstruction source to the reconstruction destination.
When the distributed file system works, single node faults are normal in client rebuilding, under a mechanism of three-copy and two-arbitration, two copies of data can be used as rebuilding/repairing sources all the time, and only one copy is asked when the service runs, so that the optimization can run perfectly under the scene of three-copy and two-arbitration. In the above embodiment, the virtual machine reads the copy on the host 3, and the rebuilding will rebuild from the copy on the host 2 to the host 1, so that the whole rebuilding process has no influence on the service, the service still operates as before, and the rebuilding process will not be influenced by the service.
In the embodiment, when the service is fully considered to run, the reconstruction source is reasonably selected by sensing the traffic of the service, so that the aim that the service is not influenced in the reconstruction process is finally achieved, and the reconstruction task is not influenced by the service. The principle of the embodiment can be also applied to balance tasks after capacity expansion and synchronization tasks after fault recovery, and the purposes of not affecting the service and not affecting the service per se can be achieved.
Referring to fig. 6, fig. 6 is a schematic structural diagram of a copy management apparatus according to an embodiment of the present application;
The apparatus may include:
The read source copy setting module 100 is configured to determine a data copy in the file system, and set the currently accessed data copy as a read source copy;
a target copy setting module 200, configured to set a data copy other than the read source copy as a target data copy;
the copy management module 300 is configured to copy, if a copy management instruction is received, the target data copy to a host corresponding to the copy management instruction; the copy management instruction comprises a copy reconstruction instruction, a copy synchronization instruction or a load balancing instruction.
The embodiment first determines the data copy existing in the distributed system, and takes the currently accessed data copy as the read source copy. After receiving the copy management instruction, selecting a target data copy from other copies except the read source copy, copying the target data copy to a host corresponding to the copy management instruction, and further completing copy management operation. Because the current accessed data copy is not used as the target data copy for data copying, the situation that the data read by the service and the data copied in the copy management process are the data in the same copy can be avoided, so that the copy management efficiency can be improved, and the influence of the copy management operation on the execution of normal service can be reduced.
Further, if the copy management instruction is the copy reconstruction instruction, the copy management module 300 includes:
The idle host determining unit is used for determining a target host in which the data copy is located and determining idle hosts from hosts except the target host in the file system;
And the copy reconstruction unit is used for selecting a copy reconstruction source from the target data copy, copying the data in the copy reconstruction source to the idle host, and completing the data copy reconstruction operation.
In this embodiment, the copy reconstruction source is not the current read data copy, so that the situation that the data of the data copy is accessed and copied at the same time can be avoided.
Further, an idle host determining unit, configured to perform priority evaluation on hosts in the file system except for the target host according to a host capacity parameter and a host performance parameter; and the idle host is also used for determining the idle host according to the priority evaluation result.
Further, if the copy management instruction is the copy synchronization instruction, the copy management module 300 includes:
The data comparison unit is used for selecting a copy synchronization source from the target data copy, and comparing the data in the copy synchronization source with the data of the data copy in the host to be synchronized;
the copy synchronization unit is used for executing data synchronization operation on the data copies in the host to be synchronized according to the data comparison result; and the data of the data copy in the host to be synchronized after the data synchronization operation is the same as the data in the copy synchronization source.
The data stored in the target data copies are the same, so that any normal data copy can be selected from all the target data copies to serve as a copy synchronization source. The data to be synchronized can be determined by comparing the data in the copy synchronization source with the data of the data copy in the host to be synchronized, and a data comparison result is obtained. And after the data synchronization operation is carried out on the data copies in the host to be synchronized according to the data comparison result, the data of the data copies in the host to be synchronized are identical to the data in the copy synchronization source.
Further, if the copy management instruction is the load balancing instruction, the copy management module 300 includes:
The balancing host determining unit is used for determining an original host and a capacity expansion host according to the load balancing instruction; the original host is the host with the largest disk load in the hosts where all the data copies are located;
And the load balancing unit is used for copying the data copy in the original host to the capacity expansion host and deleting the data copy in the original host after the data copy is copied.
According to the method, the host with the largest disk load is used as the original host, and the newly added capacity expansion host is used for carrying out the memory load balancing, so that the overall stability of the distributed file system is improved.
Further, the method further comprises the following steps:
The fault judging module is used for judging whether the target host with the fault is the host with the read source copy if the target host with the data copy is faulty; if yes, switching the current accessed data copy, and redefining a new read source copy and a new target data copy, and generating a copy reconstruction instruction or a copy synchronization instruction according to the fault duration; and if not, generating the copy reconstruction instruction or the copy synchronization instruction according to the fault duration.
Correspondingly, the fault judging module comprises:
A duration judging unit, configured to judge whether the duration of the fault is greater than a preset duration; if yes, generating the copy reconstruction instruction; and if not, generating the copy synchronization instruction.
The embodiment first determines the data copy existing in the distributed system, and takes the currently accessed data copy as the read source copy. After receiving the copy management instruction, selecting a target data copy from other copies except the read source copy, copying the target data copy to a host corresponding to the copy management instruction, and further completing copy management operation. Because the current accessed data copy is not used as the target data copy for data copying, the situation that the data read by the service and the data copied in the copy management process are the data in the same copy can be avoided, so that the copy management efficiency can be improved, and the influence of the copy management operation on the execution of normal service can be reduced.
Since the embodiments of the apparatus portion and the embodiments of the method portion correspond to each other, the embodiments of the apparatus portion are referred to the description of the embodiments of the method portion, and are not repeated herein.
The present application also provides a storage medium having stored thereon a computer program which, when executed, performs the steps provided by the above embodiments. The storage medium may include: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The application also provides an electronic device, which can comprise a memory and a processor, wherein the memory stores a computer program, and the processor can realize the steps provided by the embodiment when calling the computer program in the memory. Of course the electronic device may also include various network interfaces, power supplies, etc.
In the description, each embodiment is described in a progressive manner, and each embodiment is mainly described by the differences from other embodiments, so that the same similar parts among the embodiments are mutually referred. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section. It should be noted that it will be apparent to those skilled in the art that various modifications and adaptations of the application can be made without departing from the principles of the application and these modifications and adaptations are intended to be within the scope of the application as defined in the following claims.
It should also be noted that in this specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Claims (8)
1. The copy management method is characterized by being applied to a file system, wherein the file system comprises a plurality of hosts, each host is provided with a virtual machine, and the virtual machines read data copies stored in the hosts so as to provide corresponding services, and the copy management method comprises the following steps:
determining a data copy in a file system, and setting the currently accessed data copy as a read source copy; the current accessed data copy is a copy which is accessed by a virtual machine running in the host;
Setting the data copy except the read source copy as a target data copy;
if a copy management instruction is received, copying the target data copy to a host corresponding to the copy management instruction; the copy management instruction comprises a copy reconstruction instruction, a copy synchronization instruction or a load balancing instruction;
if the target host where the data copy is located fails, judging whether the failed target host is the host where the read source copy is located;
If yes, switching the current accessed data copy, and redefining a new read source copy and a new target data copy, and generating a copy reconstruction instruction or a copy synchronization instruction according to the fault duration;
If not, generating the copy reconstruction instruction or the copy synchronization instruction according to the fault duration;
Wherein generating the replica reconstruction instruction or the replica synchronization instruction according to the failure duration comprises:
Judging whether the fault duration time is longer than a preset duration time or not;
If yes, generating the copy reconstruction instruction;
and if not, generating the copy synchronization instruction.
2. The copy management method of claim 1, wherein if the copy management instruction is the copy reconstruction instruction, copying the target data copy to the host corresponding to the copy management instruction comprises:
determining a target host in which the data copy is located, and determining an idle host from hosts in the file system except the target host;
and selecting a copy reconstruction source from the target data copy, and copying data in the copy reconstruction source to the idle host to complete the data copy reconstruction operation.
3. The copy management method of claim 2, wherein determining an idle host from hosts in the file system other than the target host comprises:
Performing priority evaluation on hosts except the target host in the file system according to the host capacity parameter and the host performance parameter;
And determining the idle host according to the priority evaluation result.
4. The copy management method of claim 1, wherein if the copy management instruction is the copy synchronization instruction, copying the target data copy to the host corresponding to the copy management instruction comprises:
Selecting a copy synchronization source from the target data copy, and comparing the data in the copy synchronization source with the data of the data copy in the host to be synchronized;
executing data synchronization operation on the data copies in the host to be synchronized according to the data comparison result; and the data of the data copy in the host to be synchronized after the data synchronization operation is the same as the data in the copy synchronization source.
5. The copy management method of claim 1, wherein if the copy management instruction is the load balancing instruction, copying the target data copy to the host corresponding to the copy management instruction comprises:
Determining an original host and a capacity expansion host according to the load balancing instruction; the original host is the host with the largest disk load in the hosts where all the data copies are located;
copying the data copy in the original host to the capacity expansion host, and deleting the data copy in the original host after the data copy is copied.
6. The utility model provides a copy management device which characterized in that is applied to file system, and the file system includes a plurality of hosts, and each host is operated with virtual machine, and the virtual machine reads the data copy that stores in the host and then provides corresponding service, and copy management device includes:
the read source copy setting module is used for determining the data copy in the file system and setting the currently accessed data copy as the read source copy; the current accessed data copy is a copy which is accessed by a virtual machine running in the host;
the target copy setting module is used for setting the data copy except the read source copy as a target data copy;
the copy management module is used for copying the target data copy to a host corresponding to the copy management instruction if the copy management instruction is received; the copy management instruction comprises a copy reconstruction instruction, a copy synchronization instruction or a load balancing instruction;
The fault judging module is used for judging whether the target host with the fault is the host with the read source copy if the target host with the data copy is faulty; if yes, switching the current accessed data copy, and redefining a new read source copy and a new target data copy, and generating a copy reconstruction instruction or a copy synchronization instruction according to the fault duration; if not, generating the copy reconstruction instruction or the copy synchronization instruction according to the fault duration;
wherein, the trouble judgement module includes:
A duration judging unit, configured to judge whether the duration of the fault is greater than a preset duration; if yes, generating the copy reconstruction instruction; and if not, generating the copy synchronization instruction.
7. An electronic device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the copy management method of any of claims 1 to 5 when the computer program in the memory is invoked by the processor.
8. A storage medium having stored therein computer executable instructions which when loaded and executed by a processor implement the steps of the copy management method of any of the preceding claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010181063.7A CN111400098B (en) | 2020-03-16 | 2020-03-16 | Copy management method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010181063.7A CN111400098B (en) | 2020-03-16 | 2020-03-16 | Copy management method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111400098A CN111400098A (en) | 2020-07-10 |
| CN111400098B true CN111400098B (en) | 2024-05-28 |
Family
ID=71436337
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010181063.7A Active CN111400098B (en) | 2020-03-16 | 2020-03-16 | Copy management method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111400098B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116302680B (en) * | 2023-01-15 | 2024-01-23 | 北京志凌海纳科技有限公司 | Recovery system and method for reducing fault influence of super fusion system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE10055603A1 (en) * | 1999-12-03 | 2001-06-21 | Ibm | Accessing method for data sets of data memory libraries involves providing rest time status to data memory library and accessing database by data memory library having largest rest time status |
| CN102385537A (en) * | 2011-10-25 | 2012-03-21 | 曙光信息产业(北京)有限公司 | Disk failure processing method of multi-copy storage system |
| CN103139302A (en) * | 2013-02-07 | 2013-06-05 | 浙江大学 | Real-time copy scheduling method considering load balancing |
| CN104360953A (en) * | 2014-10-16 | 2015-02-18 | 华为技术有限公司 | Data copying method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013093994A1 (en) * | 2011-12-19 | 2013-06-27 | 富士通株式会社 | Storage system, data rebalancing program and data rebalancing method |
-
2020
- 2020-03-16 CN CN202010181063.7A patent/CN111400098B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE10055603A1 (en) * | 1999-12-03 | 2001-06-21 | Ibm | Accessing method for data sets of data memory libraries involves providing rest time status to data memory library and accessing database by data memory library having largest rest time status |
| CN102385537A (en) * | 2011-10-25 | 2012-03-21 | 曙光信息产业(北京)有限公司 | Disk failure processing method of multi-copy storage system |
| CN103139302A (en) * | 2013-02-07 | 2013-06-05 | 浙江大学 | Real-time copy scheduling method considering load balancing |
| CN104360953A (en) * | 2014-10-16 | 2015-02-18 | 华为技术有限公司 | Data copying method and device |
Non-Patent Citations (1)
| Title |
|---|
| 一种基于文件支持度的动态副本管理机制;肖中正 等;《计算机研究与发展》;20160229;第53卷(第2期);第431-442页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111400098A (en) | 2020-07-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109729129B (en) | Configuration modification method of storage cluster system, storage cluster and computer system | |
| US20220103622A1 (en) | Commissioning and decommissioning metadata nodes in a running distributed data storage system | |
| US8060703B1 (en) | Techniques for allocating/reducing storage required for one or more virtual machines | |
| US20160062853A1 (en) | Preventing migration of a virtual machine from affecting disaster recovery of replica | |
| JP2019101703A (en) | Storage system and control software arrangement method | |
| US10534796B1 (en) | Maintaining an active-active cloud across different types of cloud storage services | |
| CN111078667B (en) | Data migration method and related device | |
| WO2013117002A1 (en) | Method, device and system for data reconstruction | |
| CN113220729B (en) | Data storage method and device, electronic equipment and computer readable storage medium | |
| CN114466027B (en) | Cloud primary database service providing method, system, equipment and medium | |
| JP4693540B2 (en) | Database reconfiguration device and database reconfiguration program | |
| US11740925B2 (en) | Method and apparatus for online migration of multi-disk virtual machine into different storage pools | |
| CN110825543B (en) | Method for quickly recovering data on fault storage device | |
| CN109582686B (en) | Method, device, system and application for ensuring consistency of distributed metadata management | |
| CN112181736A (en) | Distributed storage system and configuration method thereof | |
| CN112328365A (en) | Virtual machine migration method, device, equipment and storage medium | |
| CN112230853A (en) | Storage capacity adjusting method, device, equipment and storage medium | |
| CN110377664B (en) | Data synchronization method, device, server and storage medium | |
| CN111400098B (en) | Copy management method and device, electronic equipment and storage medium | |
| CN112579550B (en) | Metadata information synchronization method and system of distributed file system | |
| WO2024041363A1 (en) | Serverless-architecture-based distributed fault-tolerant system, method and apparatus, and device and medium | |
| CN115982101B (en) | Machine room data migration method and device based on multi-machine room copy placement strategy | |
| WO2015196692A1 (en) | Cloud computing system and processing method and apparatus for cloud computing system | |
| CN115955488B (en) | Distributed storage copy cross-machine room placement method and device based on copy redundancy | |
| CN115098300B (en) | Database backup method, disaster recovery method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |