CN110309224B - Data copying method and device - Google Patents
Data copying method and device Download PDFInfo
- Publication number
- CN110309224B CN110309224B CN201810209543.2A CN201810209543A CN110309224B CN 110309224 B CN110309224 B CN 110309224B CN 201810209543 A CN201810209543 A CN 201810209543A CN 110309224 B CN110309224 B CN 110309224B
- Authority
- CN
- China
- Prior art keywords
- proposal
- node device
- synchronization
- queue
- aging
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data replication method and a data replication device, relates to the field of data processing, and can complete synchronous data replication across clusters and guarantee atomic consistency when data replication is submitted. The method comprises the following steps: the method comprises the steps that a first main node device creates a first proposal; storing the first proposal to a synchronization queue, wherein the first main node equipment and the second main node equipment belong to different clusters; the first main node equipment reads the first proposal from the synchronous queue and sends the first proposal to the second main node equipment so that the second main node equipment can create a second proposal according to the first proposal; the first main node equipment reads a first proposal from the synchronous queue and stores the first proposal in a first aging queue; the first main node equipment receives a synchronization success instruction sent by the second main node equipment, wherein the second main node equipment stores the second proposal to a second aging queue and generates a synchronization success instruction; and the first main node equipment activates or stores the first proposal in the first aging queue into a disk according to the synchronization success instruction.
Description
Technical Field
The embodiment of the application relates to the field of data processing, in particular to a data copying method and device.
Background
Atomic consistency refers to the situation where only the success or failure of data is fed back for the data solidification case. The current software related to message passing consistency is generally based on Paxos algorithm or enhancement of the algorithm. At present, after a method of majority elections in a cluster is mainly adopted, unified submission and refreshing are realized to realize the consistency of data atoms in the same cluster. For multiple message clusters (or Key-value-based clusters), there is currently no effective solution for how to accomplish synchronous data replication across clusters and ensure atomic consistency when data replication commits.
Disclosure of Invention
Embodiments of the present application provide a data replication method and apparatus, which can complete synchronous data replication across clusters and ensure atomic consistency when data replication is submitted.
In a first aspect, a data replication method is provided, which includes: the method comprises the steps that a first main node device creates a first proposal; when determining that a first proposal is synchronized to a second main node device, a first main node device stores the first proposal to a synchronization queue, wherein the first main node device and the second main node device belong to different clusters; the first main node equipment reads the first proposal from the synchronous queue and sends the first proposal to the second main node equipment so that the second main node equipment can create a second proposal according to the first proposal; the first main node equipment reads a first proposal from the synchronous queue and stores the first proposal in a first aging queue; the first main node equipment receives a synchronization success instruction sent by the second main node equipment, wherein the second main node equipment stores the second proposal to a second aging queue and generates a synchronization success instruction; and the first main node equipment activates or stores the first proposal in the first aging queue into a disk according to the synchronization success instruction. In the above scheme, after the first master node device creates the first proposal, when the first master node device determines to synchronize the first proposal to the second master node device, the first master node device first stores the first proposal to a synchronization queue, where the first master node device and the second master node device belong to different clusters; the first main node equipment reads the first proposal from the synchronous queue and sends the first proposal to the second main node equipment, and the second main node equipment creates a second proposal according to the first proposal; the first main node equipment reads a first proposal from the synchronous queue and stores the first proposal in a first aging queue; the second host node equipment stores the second proposal to a second aging queue and generates a synchronization success instruction, and the first host node equipment receives the synchronization success instruction sent by the second host node equipment; the first master node equipment activates or stores the first proposal in the first aging queue into a disk according to the synchronization success instruction only after acquiring the synchronization success instruction; meanwhile, the second host node equipment activates or stores the second proposal in the second aging queue into a disk, so that cross-cluster synchronous data replication is realized and atomic consistency during data replication and submission is ensured.
In one possible design, after the first master node device activates the first proposal, the method further includes: the first master node device deletes the first proposal in the synchronization queue and the first aging queue.
In one possible design, the first proposal includes a data object; before the first master node device stores the first proposal in the synchronization queue, the method further comprises: the first host node device adds metadata to the data object of the first proposal. Wherein the metadata includes an aging parameter; the method further comprises the following steps: and when the current time is determined to meet the aging parameters, if a synchronization success instruction is not received, generating synchronization error information. And when the first main node equipment creates a third proposal, refreshing the aging parameters corresponding to the first proposal. The metadata includes synchronization parameters; before the first main node equipment receives a synchronization success instruction sent by the second main node equipment, the synchronization parameter is not synchronized; after the first master node device receives the synchronization success instruction sent by the second master node device, the method further includes: the synchronization parameter is modified to be synchronized. The metadata includes an identification of a cluster to which the first proposal belongs. Since the first proposal adds metadata to the data object when it is stored in the synchronization queue, it can be determined which cluster has the latest data according to the metadata, and when a communication failure occurs between clusters, the cluster with the latest data can be used as the only surviving cluster, and the other cluster terminates the operation. Therefore, the aging queue is checked in the cluster with the latest data, the data which are not the cluster and are not synchronized are deleted, the data which are not the clusters and are not synchronized are deleted in the aging queue in another cluster, and therefore the data copying operation can be carried out again after communication is recovered, and resources are prevented from being occupied by the data which are not copied successfully.
In a second aspect, a data replication method is provided, the method comprising: the second main node equipment receives a first proposal sent by the first main node equipment; the second host node device creates a second proposal according to the first proposal, wherein the first host node device and the second host node device belong to different clusters; the second main node equipment stores the second proposal to a second aging queue and generates a synchronization success instruction; and the second main node equipment sends the synchronization success instruction to the first main node equipment, so that the first main node equipment activates or stores the first proposal into a disk according to the synchronization success instruction. The beneficial effects achieved by the method can refer to the beneficial effects in the first aspect, and are not described herein again.
In one possible design, further comprising: the method further comprises the following steps: and the second main node equipment activates or stores the second proposal in the second aging queue into a disk.
In one possible design, further comprising: after the second master node device activates the second proposal, the method further includes: the second master node device deletes the second proposal in the second aging queue.
In a third aspect, an embodiment of the present application provides a data copying apparatus. The data replication apparatus may be adapted to perform any of the methods provided by the first aspect above. The data copying device may specifically be the first master node device described in the first and second aspects, or the data copying device may be a functional entity, for example, a chip, on the first master node device provided in the first and second aspects, for implementing the method provided in the first aspect.
In one possible design, the data copying apparatus may be divided into functional modules according to the method provided in the first aspect, for example, the functional modules may be divided for each function, or two or more functions may be integrated into one functional module.
In another possible design, the data replication device may include: a processor for executing the computer program in the memory such that any of the methods provided by the first aspect is performed.
In a fourth aspect, an embodiment of the present application provides a data copying apparatus. The data replication apparatus may be adapted to perform any of the methods provided by the second aspect above. The data copying apparatus may specifically be the second master node device described in the first and second aspects, or the functional entity of the data copying apparatus, which is provided on the second master node device provided in the first and second aspects and implements the method provided in the second aspect, may be, for example, a chip.
In one possible design, the data copying apparatus may be divided into functional modules according to the method provided in the second aspect, for example, the functional modules may be divided according to the functions, or two or more functions may be integrated into one functional module.
In another possible design, the data replication device may include: a processor for executing the computer program in the memory such that any of the methods provided by the second aspect is performed.
Embodiments of the present application also provide a computer-readable storage medium, on which computer instructions are stored, which, when executed on a computer, cause the computer to perform any one of the possible methods of the first and second aspects.
Embodiments of the present application also provide a computer program product, which when run on a computer causes any one of the methods provided in the first and second aspects to be performed.
It is understood that any one of the apparatuses, computer storage media or computer program products provided above is used for executing the corresponding method provided above, and therefore, the beneficial effects achieved by the apparatuses, computer storage media or computer program products can refer to the beneficial effects in the corresponding methods, and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments will be briefly described below.
Fig. 1 is a schematic diagram of an application scenario of a data replication method provided in an embodiment of the present application;
fig. 2 is a schematic structural diagram of a data copying apparatus according to an embodiment of the present application;
FIG. 3 is a schematic flow chart diagram illustrating a data replication method according to an embodiment of the present application;
FIG. 4 is a diagram illustrating a proposed data structure according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a data copying apparatus according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a data copying apparatus according to another embodiment of the present application.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present application.
In the description of the present application, "/" indicates an OR meaning, for example, A/B may indicate A or B; "and/or" herein is merely an association describing an associated object, and means that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. Also, in the description of the present application, "a plurality" means two or more than two unless otherwise specified. In addition, for convenience of clearly describing the technical solutions of the embodiments of the present application, "first" and "second" and the like in the embodiments of the present application are used for distinguishing different objects or distinguishing different processes on the same object, and are not used for describing a specific order of the objects. For example, the first master node device and the second master node device are master node devices in different clusters.
The data replication method provided by the embodiment of the application can be applied to the data replication process among different clusters. Fig. 1 is a simplified schematic diagram of a system architecture according to an embodiment of the present application. The system architecture includes at least two clusters, where each cluster includes several devices, where the cluster may be a distributed cluster, and the devices in each cluster may determine one master node device (leader) and multiple slave node devices (follower) by electing leader, where two clusters are shown in fig. 1, where an exemplary cluster of the first cluster shows three slave node devices being joined to one master node device (first master node device 11), an exemplary cluster of the second cluster shows three slave node devices being joined to one master node device (second master node device 12), and a cache and a data curing disk of the master node device are further shown. Wherein, in the same cluster, the master node device is responsible for the creation of a proposal, the initiation and resolution of a proposal vote, and the activation and synchronization of the proposal.
Based on the system architecture, the solution provided by the embodiment of the present application is executed by a data replication device, where the data replication device may be the first master node device itself or a functional entity (e.g., a chip) configured inside the first master node device, and in another example, the data replication device may be the second master node device itself or a functional entity (e.g., a chip) configured inside the second master node device.
Fig. 2 is a schematic diagram illustrating a data copying apparatus according to an embodiment of the present disclosure, and as shown in fig. 2, the data copying apparatus may include at least one processor 21, a communication interface 22, and a bus 23.
The following describes each component of the data copying apparatus in detail with reference to fig. 2:
the processor 21 is a control center of the data copying apparatus, and may be a single processor or a collective term for a plurality of processing elements. For example, the processor 21 is a Central Processing Unit (CPU), and may be an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, such as: one or more microprocessors (digital signal processors, DSPs), or one or more Field Programmable Gate Arrays (FPGAs).
Wherein the processor 21 may perform various functions of the data copying apparatus by running or executing software programs or instructions.
Optionally, the data copying apparatus may further include a memory 24 for storing the above software programs or instructions, and further, may also store data, for example, for generating data required for data copying.
In particular implementations, processor 21 may include one or more CPUs such as CPU0 and CPU1 shown in fig. 2 as one example.
In particular implementations, the data replication device may include a plurality of processors, such as processor 21 and processor 25 shown in FIG. 2, as one embodiment. Each of these processors may be a single-Core Processor (CPU) or a multi-Core Processor (CPU). A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).
The memory 24 may be, but is not limited to, a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a Random Access Memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, optical disk storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory 24 may be separate and coupled to the processor 21 via a bus 24. The memory 24 may also be integrated with the processor 21.
The memory 24 is used for storing software programs for executing the scheme of the application, and is controlled by the processor 21 to execute.
A communication interface 22 for communicating with other devices. Such as for communicating with a second master node device, with other slave node devices in the cluster one or two.
The bus 23 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an extended ISA (enhanced industry standard architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 2, but it is not intended that there be only one bus or one type of bus.
The device structure shown in fig. 2 does not constitute a limitation of the data copying apparatus and may include more or fewer components than those shown, or some of the components may be combined, or a different arrangement of components. Alternatively, the software program in the embodiment of the present application may also be referred to as a computer execution instruction, which is not specifically limited in the embodiment of the present application. Alternatively, the software program in the embodiment of the present application may also be referred to as a computer execution instruction, which is not specifically limited in the embodiment of the present application. It will be appreciated that the data replication apparatus described above may be applied to either the first master node device or the second master node device.
Hereinafter, the data replication method provided in the present application will be described with reference to fig. 1 and 3. It should be noted that the name of the instruction or data between the devices in the following embodiments of the present application is only an example, and other names may also be used in the specific implementation, which are described herein in a unified manner, and the embodiments of the present application are not limited to this.
101. The first master node device creates a first proposal.
It should be noted that the first proposal may be a proposal created by any slave node device or the first master node device in the first cluster according to the written data, where after the first proposal is created by the first master node device, a vote needs to be initiated in the first cluster to which the first master node device belongs, and only after each slave node passes through the first proposal according to a certain rule (for example, a minority obeys a majority), the first proposal creation is successful, which is different from the prior art.
102. When determining to synchronize the first proposal to the second master node device, the first master node device stores the first proposal to a synchronization queue, wherein the first master node device and the second master node device belong to different clusters.
And if not, directly activating the first proposal or solidifying and storing the first proposal into a disk when the first proposal is abnormally activated. Wherein the first proposal comprises a data object; before the first master node device stores the first proposal in the synchronization queue, the method further includes: the first host node device adds metadata to the data object of the first proposal.
103. The first master node device reads the first proposal from the synchronization queue and sends the first proposal to the second master node device.
Illustratively, referring to fig. 4, the metadata includes one or more of an identifier of a cluster to which the first proposal belongs, a synchronization parameter, and an aging parameter, where the aging parameter is a parameter related to time and operation, and there is a unique characteristic within a certain time, for example, the aging parameter may be a timing length value, when the first proposal is stored in the synchronization queue and starts to be timed according to the timing length value, and when the timing exceeds the timing length value, a specified operation is triggered, for example: and when the current time is determined to meet the aging parameters, if a synchronization success instruction is not received, generating synchronization error information. In addition, when the first master node device creates a new proposal, the aging parameter corresponding to the first proposal is refreshed, for example, the timing length value is shortened. When the metadata includes the synchronization parameter, the synchronization parameter is not synchronized before the first host node device receives the synchronization success instruction sent by the second host node device. The first master node device may send the first proposal through any slave node device in the first cluster, or send the first proposal directly through itself.
104. The first master node device reads the first proposal from the synchronization queue and stores the first proposal in the first aging queue.
And if not, reading the first proposal from the synchronous queue again until the first proposal is completely synchronized to the first aging queue.
105. The second main node equipment receives the first proposal sent by the first main node equipment.
The second master node device may receive the first proposal through any slave node device in the second cluster (as shown in fig. 1), or directly receive the first proposal through itself. In the process of step 105, the first master node device will take out data from the synchronization queue according to the First In First Out (FIFO) principle and transmit the data to another distributed cluster, and the second master node device creates a second proposal after parsing and stores the second proposal in a second aging queue and returns a synchronization success instruction.
106. The second master node device creates a second proposal according to the first proposal.
107. And the second main node equipment stores the second proposal to a second aging queue and generates a synchronization success instruction.
108. And the second main node equipment sends the synchronization success instruction to the first main node equipment.
After the first master node device receives the synchronization success instruction sent by the second master node device, the first master node device is further configured to modify the synchronization parameter to be synchronized.
109. And the first main node equipment receives the synchronization success instruction sent by the second main node equipment.
110. And the first main node equipment activates or stores the first proposal in the first aging queue into a disk according to the synchronization success instruction.
In addition, the first master node device, after activating the first proposal, further includes: the first main node equipment deletes the first proposal in the synchronization queue and the first aging queue; and the synchronized but inactive proposal is cured to disk in the event of an anomaly. Similarly, the second master node device activates or stores the second proposal in the second aging queue to disk. And after the second host node device activates the second proposal, the second host node device deletes the second proposal in the second aging queue.
Since the first proposal adds metadata to the data object when being stored in the synchronization queue, the first proposal can judge that the cluster has the latest data according to the metadata, when the first cluster and the second cluster have communication failure, the first cluster can be used as the only surviving cluster, and the second cluster stops operating. Therefore, the first aging queue is checked in the first cluster, the data which are not the first cluster and are not synchronized are deleted, and the data which are not the unsynchronized data are deleted in the second cluster, so that the data copying operation can be carried out again after the communication is recovered, and the resource occupation of the data which are not successfully copied is avoided.
In the above scheme, after the first master node device creates the first proposal, when the first master node device determines to synchronize the first proposal to the second master node device, the first master node device first stores the first proposal to a synchronization queue, where the first master node device and the second master node device belong to different clusters; the first main node equipment reads the first proposal from the synchronous queue and sends the first proposal to the second main node equipment, and the second main node equipment creates a second proposal according to the first proposal; the first main node equipment reads a first proposal from the synchronous queue and stores the first proposal in a first aging queue; the second host node equipment stores the second proposal to a second aging queue and generates a synchronization success instruction, and the first host node equipment receives the synchronization success instruction sent by the second host node equipment; the first master node equipment activates or stores the first proposal in the first aging queue into a disk according to the synchronization success instruction only after acquiring the synchronization success instruction; meanwhile, the second host node equipment activates or stores the second proposal in the second aging queue into a disk, so that cross-cluster synchronous data replication is realized and atomic consistency during data replication and submission is ensured.
The scheme provided by the embodiment of the application is mainly introduced from the perspective of a method. To implement the above functions, it includes hardware structures and/or software modules for performing the respective functions. Those of skill in the art would readily appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiment of the present application, the data copying apparatus may be divided into the functional modules according to the above method examples, for example, each functional module may be divided according to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that, in the embodiment of the present application, the division of the module is schematic, and is only one logic function division, and there may be another division manner in actual implementation.
Fig. 5 is a schematic structural diagram of a data copying apparatus according to an embodiment of the present application. The data replication arrangement shown in fig. 5 may be used to perform the steps performed by the respective first master node device in any of the data replication methods provided above. The data replication device 50 may include: a sending module 51, a processing module 52 and a receiving module 53. Wherein, the processing module 52 is configured to create a first proposal; the processing module 52 is further configured to, when it is determined that the first proposal is synchronized to a second master node device, store the first proposal in a synchronization queue, where the first master node device and the second master node device belong to different clusters; a sending module 51, configured to read the first proposal from the synchronization queue and send the first proposal to a second host node device, so that the second host node device creates a second proposal according to the first proposal; the processing module 52 is further configured to read the first proposal from the synchronization queue and store the first proposal in a first aging queue; a receiving module 53, configured to receive a synchronization success instruction sent by the second master node device, where the second master node device stores the second proposal in a second aging queue, and generates the synchronization success instruction; the processing module 52 is further configured to activate or store the first proposal in the first aging queue to a disk according to the synchronization success instruction. All relevant contents of each step related to the above method embodiment may be referred to as a functional description of the corresponding functional module, and explanation of relevant contents in this embodiment may refer to the above method embodiment, which is not described herein again.
As an example, in conjunction with the data copying apparatus shown in fig. 2, the above-mentioned sending module 51 and receiving module 53 may correspond to the communication interface 22 in fig. 2; the processing module 52 may correspond to the processors 21, 25 in fig. 2.
Fig. 6 is a schematic structural diagram of a data copying apparatus 60 according to an embodiment of the present disclosure. The data replication arrangement 60 shown in fig. 6 may be used to perform the steps performed by the respective second master node device in any of the data replication methods provided above. The data replication device 60 may include: a sending module 61, a processing module 62 and a receiving module 63. The receiving module 63 is configured to receive a first proposal sent by a first master node device; a processing module 62, configured to create a second proposal according to the first proposal received by the receiving module, where the first host node device and the second host node device belong to different clusters; the processing module 62 is further configured to store the second proposal in a second aging queue, and generate a synchronization success instruction; a sending module 61, configured to send the synchronization success instruction generated by the processing module to the first master node device, so that the first master node device activates or stores the first proposal in a disk according to the synchronization success instruction. All relevant contents of each step related to the above method embodiment may be referred to as a functional description of the corresponding functional module, and explanation of relevant contents in this embodiment may refer to the above method embodiment, which is not described herein again.
As an example, in conjunction with the data copying apparatus shown in fig. 2, the sending module 61 and the receiving module 63 may correspond to the communication interface 22 in fig. 2; the processing module 62 may correspond to the processors 21, 25 in fig. 2.
It should be understood that, in the various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the units is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented using a software program, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions described in accordance with the embodiments of the present application are all or partially generated upon loading and execution of computer program instructions on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or can comprise one or more data storage devices, such as a server, a data center, etc., that can be integrated with the medium. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (20)
1. A data copying method, characterized in that,
the method comprises the steps that a first main node device creates a first proposal;
when the first host node equipment determines to synchronize the first proposal to second host node equipment, storing the first proposal to a synchronization queue, wherein the first host node equipment and the second host node equipment belong to different clusters;
the first main node equipment reads the first proposal from the synchronous queue and sends the first proposal to second main node equipment so that the second main node equipment can create a second proposal according to the first proposal;
the first master node equipment reads the first proposal from the synchronous queue and stores the first proposal in a first aging queue;
the first main node equipment receives a synchronization success instruction sent by the second main node equipment, wherein the second main node equipment stores the second proposal to a second aging queue and generates the synchronization success instruction;
and the first main node equipment activates or stores the first proposal in the first aging queue into a disk according to the synchronization success instruction.
2. The method of claim 1, wherein after the first host node device activates the first proposal, further comprising:
the first master node device deletes the first proposal in the synchronization queue and the first aging queue.
3. The method of claim 1 or 2, wherein the first proposal comprises a data object;
before the first master node device stores the first proposal to a synchronization queue, the method further comprises: the first master node device adds metadata to the data object of the first proposal.
4. The method of claim 3, wherein the metadata comprises an aging parameter; the method further comprises the following steps:
and when the current time is determined to meet the aging parameters, if the synchronization success instruction is not received, generating synchronization error information.
5. The method of claim 3, further comprising refreshing an aging parameter corresponding to the first proposal when the first master node device creates a third proposal.
6. The method of claim 3, wherein the metadata comprises synchronization parameters;
before the first master node device receives a synchronization success instruction sent by the second master node device, the synchronization parameter is not synchronized;
after the first master node device receives the synchronization success instruction sent by the second master node device, the method further includes:
modifying the synchronization parameter to be synchronized.
7. The method of claim 3, wherein the metadata comprises an identification of a cluster to which the first proposal belongs.
8. A data copying method, characterized in that,
the second main node equipment receives a first proposal sent by the first main node equipment;
the second host node device creates a second proposal according to the first proposal, wherein the first host node device and the second host node device belong to different clusters;
the second main node equipment stores the second proposal to a second aging queue and generates a synchronization success instruction;
and the second main node equipment sends the synchronization success instruction to the first main node equipment, so that the first main node equipment activates or stores the first proposal into a disk according to the synchronization success instruction.
9. The data replication method of claim 8, the method further comprising:
and the second main node equipment activates or stores the second proposal in the second aging queue into a disk.
10. The data replication method of claim 9, wherein after the second master node device activates the second proposal, the method further comprises:
the second master node device deletes the second proposal in the second aging queue.
11. A data copying apparatus, characterized in that,
a processing module for creating a first proposal;
the processing module is further configured to store the first proposal in a synchronization queue when it is determined that the first proposal is synchronized to a second master node device, where the first master node device and the second master node device belong to different clusters;
a sending module, configured to read the first proposal from the synchronization queue and send the first proposal to a second host node device, so that the second host node device creates a second proposal according to the first proposal;
the processing module is further configured to read the first proposal from the synchronization queue and store the first proposal in a first aging queue;
a receiving module, configured to receive a synchronization success instruction sent by the second host node device, where the second host node device stores the second proposal in a second aging queue, and generates the synchronization success instruction;
the processing module is further configured to activate or store the first proposal in the first aging queue to a disk according to the synchronization success instruction.
12. The apparatus of claim 11, wherein the processing module is further configured to delete the first proposal in the synchronization queue and the first aging queue after activating the first proposal.
13. The apparatus of claim 11 or 12, wherein the first proposal comprises a data object;
the processing module is further configured to add metadata to the data object of the first proposal before storing the first proposal in the synchronization queue.
14. The apparatus of claim 13, wherein the metadata comprises an aging parameter; and the processing module is further used for generating synchronization error information if the synchronization success instruction is not received when the current time is determined to meet the aging parameter.
15. The apparatus of claim 13, wherein the processing module is further configured to refresh an aging parameter corresponding to the first proposal when creating a third proposal.
16. The apparatus of claim 13, wherein the metadata comprises synchronization parameters;
before the first master node device receives a synchronization success instruction sent by the second master node device, the synchronization parameter is not synchronized;
the processing module is further configured to modify the synchronization parameter to be synchronized after receiving the synchronization success instruction sent by the second master node device.
17. The apparatus of claim 13, wherein the metadata comprises an identification of a cluster to which the first proposal belongs.
18. A data copying apparatus, characterized in that,
the receiving module is used for receiving a first proposal sent by first main node equipment;
a processing module, configured to create a second proposal according to the first proposal received by the receiving module, where the first host node device and the second host node device belong to different clusters;
the processing module is further configured to store the second proposal to a second aging queue and generate a synchronization success instruction;
a sending module, configured to send the synchronization success instruction generated by the processing module to the first host node device, so that the first host node device activates or stores the first proposal in a disk according to the synchronization success instruction.
19. The apparatus of claim 18, wherein the processing module is further configured to activate or store the second proposal in the second aging queue to disk.
20. The apparatus of claim 19, wherein the processing module is further configured to delete the second proposal in the second aging queue after activating the second proposal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810209543.2A CN110309224B (en) | 2018-03-14 | 2018-03-14 | Data copying method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810209543.2A CN110309224B (en) | 2018-03-14 | 2018-03-14 | Data copying method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110309224A CN110309224A (en) | 2019-10-08 |
| CN110309224B true CN110309224B (en) | 2021-08-31 |
Family
ID=68073318
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810209543.2A Active CN110309224B (en) | 2018-03-14 | 2018-03-14 | Data copying method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110309224B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021168697A1 (en) * | 2020-02-26 | 2021-09-02 | 深圳市欢太科技有限公司 | Data synchronization method and apparatus, data storage system, and computer readable medium |
| CN113703954A (en) * | 2020-05-21 | 2021-11-26 | 中移(苏州)软件技术有限公司 | Message backup method and device, electronic equipment and computer storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102629268A (en) * | 2012-03-09 | 2012-08-08 | 华为技术有限公司 | Data synchronization method, system and date access device |
| CN103875229A (en) * | 2013-12-02 | 2014-06-18 | 华为技术有限公司 | Asynchronous replication method, device and system |
| CN104202375A (en) * | 2014-08-22 | 2014-12-10 | 广州华多网络科技有限公司 | Method and system for synchronous data |
| CN105512297A (en) * | 2015-12-10 | 2016-04-20 | 中国测绘科学研究院 | Distributed stream-oriented computation based spatial data processing method and system |
| CN105786959A (en) * | 2016-01-11 | 2016-07-20 | 北京京东尚科信息技术有限公司 | Synchronization method and device of primary database and spare database |
| CN106034080A (en) * | 2015-03-10 | 2016-10-19 | 中兴通讯股份有限公司 | Metadata migration method and metadata migration device in distributed system |
| CN106445409A (en) * | 2016-09-13 | 2017-02-22 | 郑州云海信息技术有限公司 | Distributed block storage data writing method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102511031B (en) * | 2009-09-24 | 2015-11-25 | 国际商业机器公司 | The data of bitmap are used to store |
-
2018
- 2018-03-14 CN CN201810209543.2A patent/CN110309224B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102629268A (en) * | 2012-03-09 | 2012-08-08 | 华为技术有限公司 | Data synchronization method, system and date access device |
| CN103875229A (en) * | 2013-12-02 | 2014-06-18 | 华为技术有限公司 | Asynchronous replication method, device and system |
| CN104202375A (en) * | 2014-08-22 | 2014-12-10 | 广州华多网络科技有限公司 | Method and system for synchronous data |
| CN106034080A (en) * | 2015-03-10 | 2016-10-19 | 中兴通讯股份有限公司 | Metadata migration method and metadata migration device in distributed system |
| CN105512297A (en) * | 2015-12-10 | 2016-04-20 | 中国测绘科学研究院 | Distributed stream-oriented computation based spatial data processing method and system |
| CN105786959A (en) * | 2016-01-11 | 2016-07-20 | 北京京东尚科信息技术有限公司 | Synchronization method and device of primary database and spare database |
| CN106445409A (en) * | 2016-09-13 | 2017-02-22 | 郑州云海信息技术有限公司 | Distributed block storage data writing method and device |
Non-Patent Citations (2)
| Title |
|---|
| 分布式系统中数据同步机制的研究与实现;崔伟等;《计算机工程与设计》;20070523(第10期);第2259-2261页 * |
| 基于消息中间件的数据同步更新方法;袁震;《兵工自动化》;20130715(第07期);第93-96页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110309224A (en) | 2019-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108234641B (en) | Data reading and writing method and device based on distributed consistency protocol | |
| CN106843749B (en) | Write request processing method, device and equipment | |
| CN110597910A (en) | Remote data synchronization method, device and system | |
| CN106648440B (en) | Control method for operating storage device and storage device | |
| WO2018054079A1 (en) | Method for storing file, first virtual machine and namenode | |
| CN106878382B (en) | Method and device for dynamically changing cluster scale in distributed arbitration cluster | |
| CN115599747B (en) | Metadata synchronization method, system and equipment of distributed storage system | |
| US20180139280A1 (en) | Methods and apparatuses for cluster switching | |
| US20160034332A1 (en) | Information processing system and method | |
| CN113094430B (en) | Data processing method, device, equipment and storage medium | |
| CN110309224B (en) | Data copying method and device | |
| CN114185558A (en) | Native application master selection method and device based on K8s and storage medium | |
| CN111400086B (en) | Method and system for realizing fault tolerance of virtual machine | |
| CN112015591A (en) | Log management method, server and database system | |
| JP2021168123A (en) | Systems and method for distributed read/write locking with network key values for storage devices | |
| CN102904946B (en) | Method and device for managing nodes in cluster | |
| CN112363762A (en) | Fusion command processing method, system, device and medium | |
| US9952941B2 (en) | Elastic virtual multipath resource access using sequestered partitions | |
| CN115629901A (en) | Log playback method and device, data recovery method and device and electronic equipment | |
| EP4170519A1 (en) | Data synchronization method and device | |
| CN114036164A (en) | High-concurrency transaction locking method and system combining optimistic lock and pessimistic lock | |
| CN116594551A (en) | Data storage method and device | |
| CN114168636A (en) | Data processing method, device and equipment based on cache consistency and storage medium | |
| CN115705269A (en) | Data synchronization method, system, server and storage medium | |
| CN113239059A (en) | Switching method and device of distributed lock, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |