CN117874136A - Method and device for synchronizing copies of distributed database and nonvolatile storage medium - Google Patents

Abstract

The invention discloses a copy synchronization method and device of a distributed database and a nonvolatile storage medium. Wherein the method comprises the following steps: acquiring change data in a target main copy; determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data; selecting at least one first slave copy from a plurality of target slave copies of the target master copy to store change data; at least one second slave copy is selected from among a plurality of target slave copies of the target master copy to store the oplog. The invention solves the technical problem of higher data storage cost in the existing distributed storage mode.

Description

Method and device for synchronizing copies of distributed database and nonvolatile storage medium

Technical Field

The present invention relates to the field of distributed databases, and in particular, to a method and an apparatus for synchronizing copies of a distributed database, and a nonvolatile storage medium.

Background

The principal principle of the master-slave architecture is to divide nodes in the system into two types, namely a master node and a slave node. The master node plays a core role of the system and is responsible for controlling and managing the operation of the whole system, and the slave node is responsible for storing and processing data.

In a master-slave architecture, the master node is usually a single node, and is responsible for coordinating and managing the running state of the whole system, including read-write operation of control data, load balancing, data backup and the like. A slave node is a plurality of nodes in the system that are responsible for storing and processing data. The data synchronization and backup between the slave nodes are controlled by the master node to ensure the data consistency and reliability of the system.

However, in the multi-copy storage system of the master-slave architecture, since the master node is a control center of the whole system, once the master node fails, the whole system cannot work normally, which may cause a data loss or service unavailability. In the master-slave architecture, the master node needs to process all metadata and coordinate the work of all nodes, which can cause the master node to become a performance bottleneck of the whole system and affect the expansibility and performance of the system.

In a master-slave architecture, there is a problem of data synchronization between the master and slave nodes, since data is only modified on the master node. If errors or delays occur in the synchronization process, data inconsistency can be caused; in the master-slave architecture, the expansibility of the system is limited to a certain extent due to the limitation of the master node, and the requirement of large-scale data processing is difficult to meet.

Furthermore, each node in the general multi-copy storage system of the distributed consistency protocol comprises a plurality of copies, and a master copy can be selected from each node to process data, wherein the plurality of copies are communicated through the distributed consistency protocol so as to ensure the consistency of the data. Data is stored in a distributed manner among a plurality of nodes so as to ensure the reliability and high availability of the data. Each node contains a complete data set and can provide service when other nodes are down.

However, under the condition that the master node or the master copy is determined, election needs to be carried out based on at least three nodes or copies, namely at least more than half of nodes or copies need to agree to carry out data modification operation, and each node or copy needs to store the same data.

Aiming at the problem that the existing distributed storage mode has higher data storage cost, no effective solution is proposed at present.

Disclosure of Invention

The embodiment of the invention provides a method and a device for synchronizing copies of a distributed database and a nonvolatile storage medium, which at least solve the technical problem of high data storage cost of the existing distributed storage mode.

According to an aspect of an embodiment of the present invention, there is provided a replica synchronization method of a distributed database, including: acquiring change data in a target main copy; determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data; selecting at least one first slave copy from a plurality of target slave copies of the target master copy to store the change data; at least one second slave copy is selected from a plurality of target slave copies of the target master copy to store the operation log.

Optionally, obtaining the change data in the target primary copy includes: selecting the target master copy from a plurality of preset copies of a preset copy group; the change data is generated by the target primary replica in response to at least one data operation.

Optionally, selecting the target primary copy from a plurality of preset copies of a preset copy group includes: counting the received voting votes of each preset copy in a preset copy group, wherein the voting votes represent agreeing to the preset copy as a main copy; detecting the number of votes of the votes; and determining the preset copies with the voting number meeting a preset election rule as the target main copy, wherein the preset election rule is that the voting number is the total number of the preset copies in the preset copy group or exceeds half of the total number of the preset copies in the preset copy group.

Optionally, selecting the target primary copy from a plurality of preset copies of a preset copy group includes: identifying a preset copy group without a main copy as a target copy group, wherein the target copy group comprises: a plurality of target copies; determining a plurality of target nodes for storing the target copy group in a plurality of preset nodes of a distributed database, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a target node and a storage unit, wherein each preset data block is used for storing a preset copy, the preset copy comprises the target copy, and the preset data blocks used for storing the target copy exist in the target node; monitoring the number of primary copies in each target node, wherein a plurality of preset copies belonging to different preset copy groups exist in the target node, and the number of primary copies is the number of primary copies in the plurality of preset copies of the same target node; and determining the target copy as the target master copy in the target node with the lowest master copy number.

Optionally, before electing the target primary copy among the plurality of preset copies of the preset copy group, the method further includes: determining the total number of the preset copies in the preset copy group; selecting preset nodes with the number of target nodes from a plurality of preset nodes of a distributed database as target nodes, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a target node and a storage unit, wherein each preset data block is used for storing preset copies, the number of the target nodes is the same as that of the total copies, and each target node is used for storing one preset copy; determining a target data block in each target node, wherein the target data block is an idle preset data block; and respectively placing a plurality of preset copies in the preset copy group into target data blocks in the target node.

Optionally, selecting at least one second slave backup from among the plurality of target slave copies of the target master copy to store the operation log includes: determining a plurality of slave replica nodes for storing the target slave replica in a plurality of preset nodes of a distributed database, wherein each preset node comprises: the target slave copy comprises a target slave copy node, wherein the target slave copy node comprises a target slave copy node and a target slave copy node, and the target slave copy node comprises a plurality of target slave copy nodes; monitoring the log storage quantity in each slave copy node, wherein a plurality of preset slave copies belonging to different preset copy groups exist in the slave copy nodes, the preset slave copies keep synchronous with a third slave copy of a master copy based on copy data or keep synchronous with a fourth slave copy of the master copy based on operation logs, and the log storage quantity is the quantity of the fourth slave copies in the same slave copy node; and selecting the second slave copy from slave copy nodes with the log storage quantity lower than a preset quantity threshold value.

Optionally, after determining the operation log of the change data, the method further comprises: obtaining a commit vote of a preset copy group where the primary copy is located, wherein the preset copy group comprises: a plurality of pre-set copies, each of the pre-set copies being either the master copy or a slave copy, the commit vote representing consent to commit the oplog; detecting the number of submissions of the submissions of votes; and submitting the operation log to the slave copy under the condition that the number of submissions meets a preset submission rule.

According to another aspect of the embodiment of the present invention, there is also provided a replica synchronization apparatus for a distributed database, including: the acquisition module is used for acquiring the change data in the target main copy; the determining module is used for determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data; the first synchronization module is used for selecting at least one first slave copy from a plurality of target slave copies of the target master copy to store the change data; and the second synchronization module is used for selecting at least one second slave copy from a plurality of target slave copies of the target master copy to store the operation log.

According to another aspect of the embodiment of the present invention, there is further provided a nonvolatile storage medium, where the nonvolatile storage medium is configured to store a program, and when the program runs, control a device where the nonvolatile storage medium is located to execute a copy synchronization method of the distributed database.

According to another aspect of the embodiment of the present invention, there is also provided an electronic device, including: the system comprises a memory and a processor, wherein the processor is used for running a program stored in the processor, and the program runs to execute a copy synchronization method of the distributed database.

In the embodiment of the invention, the change data in the target main copy is acquired; determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data; selecting at least one first slave copy from a plurality of target slave copies of the target master copy to store change data; at least one second slave copy is selected from a plurality of target slave copies of the target master copy, so that the master copy and the slave copies can be synchronized based on both the change data and the operation log, and the data storage cost of the operation log is lower than that of the change data because the data volume of the operation log is smaller than that of the change data.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:

FIG. 1 is a flow chart of a method of replica synchronization for a distributed database according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a portion of a preset node in a distributed database according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a replica synchronization apparatus for a distributed database according to an embodiment of the present invention;

fig. 4 is a block diagram of a computer terminal according to an embodiment of the present invention.

Detailed Description

In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

First, partial terms or terminology appearing in describing embodiments of the present application are applicable to the following explanation:

distributed storage system: a storage system for storing data in a distributed manner on a plurality of nodes achieves the purpose of storing the data and generally has different degrees of data redundancy so as to ensure that the data is not lost.

Node (node): typically composed of hardware and software, may be an instance in a server, computer, storage device, or cloud service.

Consistency protocol: protocols for ensuring data consistency among multiple nodes in a distributed system.

Duplicate (duplicate): logical concepts in the coherence protocol refer to virtual instances of data held in the protocol, and the amount of data held is determined according to physical concepts corresponding to the virtual instances.

Log (log): in this document, "log" refers to an ordered sequence of records storing software operations, located in each copy, in a coherence protocol. Operating system log files or console logs, etc. in a not common sense.

Election: 1 or more copies according to a certain election rule, by mutually sending voting messages, the process of electing the master copy.

In accordance with an embodiment of the present invention, there is provided a replica synchronization method embodiment of a distributed database, it being noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system, such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.

FIG. 1 is a flowchart of a method for replica synchronization of a distributed database according to an embodiment of the present invention, as shown in FIG. 1, the method comprising the steps of:

step S102, obtaining change data in a target main copy;

step S104, determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data;

step S106, selecting at least one first slave copy to store change data in a plurality of target slave copies of the target master copy;

step S108, selecting at least one second slave copy from a plurality of target slave copies of the target master copy to store the operation log.

In the embodiment of the invention, the change data in the target main copy is acquired; determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data; selecting at least one first slave copy from a plurality of target slave copies of the target master copy to store change data; at least one second slave copy is selected from a plurality of target slave copies of the target master copy, so that the master copy and the slave copies can be synchronized based on both the change data and the operation log, and the data storage cost of the operation log is lower than that of the change data because the data volume of the operation log is smaller than that of the change data.

Optionally, the first slave copy and the second slave copy are different copies, and the target master copy has at least two slave copies, e.g., the slave copy of the target master copy may include: a first slave copy and a second slave copy.

Alternatively, the second slave copy may be a pure log copy.

Optionally, the data amount of the operation log is smaller than the data amount of the change data.

It should be noted that, the data storage cost may be expressed as the amount of space occupied by the storage, and the more space occupied by the storage data, the higher the cost of data storage.

As an alternative embodiment, obtaining the change data in the target primary copy includes: selecting a target main copy from a plurality of preset copies in a preset copy group; and generating change data by the target main copy in response to at least one data operation.

According to the embodiment of the invention, the target primary copy is in the preset copy group, the target primary copy for responding to the data operation can be selected from the multiple preset copies in an election mode, and the target primary copy responds to the change data of the data operation, so that the determination of the change data is realized.

As an alternative embodiment, in the plurality of preset copies of the preset copy group, electing the target primary copy includes: counting that each preset copy in the preset copy group receives a voting vote, wherein the voting vote represents agreement of the preset copy as a main copy; detecting the number of votes of the votes; and determining the preset copies with the voting number meeting the preset election rule as target main copies, wherein the preset election rule is that the voting number is the total number of the preset copies in the preset copy group or exceeds half of the total number of the preset copies in the preset copy group.

According to the embodiment of the invention, the target master copy obtains the voting confirmation according to each preset copy in the preset copy group, and when the voting number of the voting confirmation obtained by a certain preset copy is the total number of the preset copies in the preset copy group or exceeds half of the total number of the preset copies in the preset copy group, the preset copy is confirmed as the target master copy in the preset copy group, and other preset copies in the preset copy group are confirmed as target slave copies of the target master copy, so that the confirmation of the target master copy and the target slave copies is realized.

Optionally, the set of pre-set copies includes at least 3 pre-set copies.

Optionally, the number of votes is the total number of the preset copies in the preset copy group, which indicates that all preset copies in the preset copy group agree to the specified preset copy as the target slave copy.

Optionally, the number of votes exceeds half of the total number of the preset copies in the preset copy group, which means that more than half of the preset copies in the preset copy group agree to the specified preset copy as the target slave copy.

As an alternative embodiment, in the plurality of preset copies of the preset copy group, electing the target primary copy includes: identifying a preset copy group without a master copy as a target copy group, wherein the target copy group comprises: a plurality of target copies; determining a plurality of target nodes for storing a target copy group in a plurality of preset nodes of the distributed database, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a target node and a storage unit, wherein each preset data block is used for storing a preset copy, the preset copy comprises a target copy, and the preset data block used for storing the target copy exists in the target node; monitoring the number of primary copies in each target node, wherein a plurality of preset copies belonging to different preset copy groups exist in the target node, and the number of the primary copies is the number of serving as the primary copies in the plurality of preset copies of the same target node; and determining the target copy as the target master copy in the target node with the lowest master copy number.

According to the embodiment of the invention, the plurality of preset nodes of the distributed database respectively store the plurality of preset copies through the plurality of preset data blocks, the distributed database can comprise storing the plurality of preset copy groups, under the condition of determining the target master copy, the preset copy group which is not selected as the target copy group can be determined in the distributed database, and the target copy group is distributed in the preset nodes of the distributed database, because each preset node can store the preset copies of the plurality of preset copy groups, and the preset copies in each preset node can be the master copy of the corresponding preset copy group, the number of the master copy serving as the master copy in each preset node can be counted, and then the target copy of the target copy group stored by the target node is determined in the target node and serves as the master copy in the target copy group, so that the master copy of the plurality of preset copy groups can be uniformly distributed in the plurality of preset nodes of the distributed database.

It should be noted that, when the primary copy of each preset copy group is used for responding to a data operation, in the case of changing the data in the primary copy based on the data operation, the more the computing resources of the preset node where the primary copy is located are occupied, the lower the efficiency of the primary copy in the preset node in responding to the data operation, the primary copies of the multiple preset copy groups are uniformly distributed in multiple preset nodes of the distributed database, so that each primary copy can have optimal response efficiency to the data operation.

As an alternative embodiment, before electing the target primary replica among the plurality of preset replicas of the preset replica group, the method further includes: determining the total number of the preset copies in the preset copy group; selecting preset nodes with the number of target nodes from a plurality of preset nodes of a distributed database as target nodes, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a plurality of target nodes and a plurality of storage nodes, wherein each preset data block is used for storing preset copies, the number of the target nodes is the same as that of the total copies, and each target node is used for storing one preset copy; in each target node, determining a target data block, wherein the target data block is an idle preset data block; and respectively placing a plurality of preset copies in the preset copy group into target data blocks in the target node.

According to the embodiment of the invention, the plurality of preset nodes of the distributed database respectively store the plurality of preset copies through the plurality of preset data blocks, and the plurality of preset copies of the same preset copy group are respectively stored in different preset nodes, so that if the preset nodes stop running due to faults, other available preset copies in the distributed database can be ensured, and the reliability and high availability of the data are ensured.

As an alternative embodiment, selecting at least one second slave backup storage operation log among the plurality of target slave copies of the target master copy includes: determining a plurality of slave replica nodes for storing target slave replicas among a plurality of preset nodes of the distributed database, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a target slave copy node and a target slave copy node, wherein each preset data block is used for storing a preset copy; monitoring the log storage quantity in each slave copy node, wherein a plurality of preset slave copies belonging to different preset copy groups exist in the slave copy node, and the preset slave copies are based on a third slave copy of which the copy data are kept synchronous with the master copy or based on a fourth slave copy of which the operation log is kept synchronous with the master copy, wherein the log storage quantity is the quantity of the fourth slave copies in the same slave copy node; and selecting a second slave copy from slave copy nodes with the log storage quantity lower than a preset quantity threshold value.

According to the embodiment of the invention, the plurality of preset nodes of the distributed database respectively store a plurality of preset copies through the plurality of preset data blocks, the slave copy nodes with the log storage quantity lower than the preset quantity threshold value are selected from the plurality of preset nodes, and the target slave copy in the slave copy nodes is determined to be the second slave copy, so that the excessive quantity stored in the preset nodes where the slave copy nodes are located can be avoided.

Alternatively, the preset number threshold may be determined according to a lowest value of the log storage number among a plurality of preset nodes of the distributed database.

As an alternative embodiment, after determining the oplog of the change data, the method further comprises: obtaining a commit vote of a preset copy group where a main copy is located, wherein the preset copy group comprises: the system comprises a plurality of preset copies, wherein each preset copy is a master copy or a slave copy, and the submitting vote represents agreement to submit an operation log; detecting the number of submissions of the submitted votes; and submitting the operation log to the slave copy in the condition that the number of submissions meets the preset submission rule.

According to the embodiment of the invention, a mode similar to the master copy is adopted, whether the master copy and the slave copy in the preset copy group agree with each other or not is indicated by the preset copies in the preset copy group through submitting votes, and further, under the condition that the submitting votes of the submitted votes meet preset submitting rules, the fact that the preset copies in the preset copy group agree with each other based on the change data is determined, and then the operation log can be submitted to the slave copy.

Optionally, the preset commit rule is that the number of commit is the total number of copies of the preset copies in the preset copy group, or the number of commit exceeds half of the total number of copies of the preset copies in the preset copy group.

Optionally, the submitted number is the total number of the preset copies in the preset copy group, which indicates that all preset copies in the preset copy group are agreed.

Optionally, the number of votes exceeds half of the total number of the preset copies in the preset copy group, indicating that more than half of the preset copies in the preset copy group have been granted.

The present invention also provides a preferred embodiment that provides a method of optimizing storage space in distributed storage that uses a consistency protocol based on an operation log (e.g., operation log); the data copies using the protocol ensure data consistency between the copies by copying the oplog.

Optionally, the consistency protocol determines the main copy by an election mode to provide relevant services to the outside; the submitting of the log is limited according to preset submitting rules, for example, the submitting rule is limited to obtain the agreements of more than half of the copies to submit new logs; or consent to obtain all copies may submit a new log.

Alternatively, the pre-set election rules for the master copy may be changed as desired, for example, selecting the pre-set commit rule to obtain consent-selectable master copies for more than half of the copies, or to obtain consent-selectable master copies for all of the copies.

Alternatively, the copy may choose whether to apply the log to complete the data drop (choose to synchronize based on the change data, or synchronize based on the oplog), and if the log is applied (i.e., synchronize based on the oplog), then the storage space occupation of the copy changes substantially, which is a more specific copy (referred to as a pure log copy).

It should be noted that, according to the characteristics of the above-mentioned consistency protocol, the redundancy of the minimum data copy in the distributed database may be less than 3.

Optionally, the multiple copy groups may be interleaved among multiple nodes of the distributed database, the master nodes of the multiple copy groups are uniformly distributed on different nodes, and the uniform distribution of the master copies may be achieved by performing master copy election on the copy groups without master copies in each node one by one; the main copies of the multi-copy group are distributed uniformly among the nodes, any node can execute the client request, the performance bottleneck of a single node is avoided,

in the above embodiment of the present invention, the number of copies of different copy groups may be different, and may be specified by configuring a consistency protocol in the copy group, and cooperate with a "pure log copy" (such as a second slave copy) in the copy group, so as to achieve the purpose that the data redundancy is less than 3.

FIG. 2 is a schematic diagram of a portion of nodes in a distributed database, as shown in FIG. 2, where the distributed database uses a storage system to block managed physical storage units, each block of data is a copy, and the data between the different copies of the different nodes forms a copy group using the consistency protocol, where multiple copy groups exist simultaneously in the distributed database from the perspective of the entire storage system.

Optionally, as shown in fig. 2, the distributed database has at least 3 nodes and 5 replica groups, where the replicas in the replica group No. 1 include: copies (1-1), the copies in copy group number 2 include: copies (2-1, 2-2), copy group 3 has copies including: copies (3-1, 3-2, 3-3), the copies in copy group 4 include: copies (4-1, 4-2), copies of copy group 5 include: copy (5-1).

As an alternative example, each replica group in the distributed database uses a consistency protocol to ensure data consistency. After the distributed database is started, the primary copy interacted with the client is selected through an election mechanism. For example, a preset election rule (e.g., obtain consent of all copies to select a master copy) may be used, and then sequentially electing a master copy on a table in the non-leader copy groups stored in each node in turn, as shown in fig. 2, so that the leader of the 5 copy groups are relatively uniformly distributed on 3 different nodes, respectively.

As an alternative example, logs are synchronized to (e.g., from) other copies via a consistency protocol, which selects whether to apply the logs and drop the data according to the configuration, i.e., based on change data, or based on operation logs.

Alternatively, as shown in FIG. 2, the copies (2-1), (3-2), and (4-2) are pure log copies, and no log drop will apply.

Optionally, the number of the most configured nodes according to the scheme of the multi-copy storage system of the common distributed protocol is 3, namely, the data redundancy is 3; the technology provided by the application can synchronize the slave copies based on the operation log, so that the data quantity required by storage can be reduced, and the data redundancy is calculated to be 2.

As an alternative example, the data may be evenly distributed by controlling whether the log is applied to the copies between each node, so that the data is evenly distributed between the nodes.

Alternatively, as shown in FIG. 2, the data is evenly distributed among the 6 copies (1-1), (3-1), (4-1), (2-2), (5-1), (5-3), and because the leader copies are also relatively evenly distributed across the different nodes. Each node can respond to the request of the client, the cluster resources are fully utilized, and the problem that the overall performance is limited due to the performance bottleneck of a single node is avoided.

The embodiment of the invention can achieve the aim of optimizing the storage space, not only can reduce duplicate redundancy, but also can uniformly distribute the data and request balanced response; the reliability and usability of the distributed database can be improved and excellent in performance can be exhibited.

According to the embodiment of the invention, an embodiment of a copy synchronization device of a distributed database is further provided, and it should be noted that the copy synchronization device of the distributed database may be used to execute the copy synchronization method of the distributed database in the embodiment of the invention, and the copy synchronization method of the distributed database in the embodiment of the invention may be executed in the copy synchronization device of the distributed database.

FIG. 3 is a schematic diagram of a replica synchronization apparatus for a distributed database according to an embodiment of the present invention, as shown in FIG. 3, the apparatus may include: an obtaining module 32, configured to obtain the change data in the target primary copy; a determining module 34, configured to determine an operation log of the change data, where the operation log is used to record at least one data operation for generating the change data; a first synchronization module 36 for selecting at least one first slave copy among a plurality of target slave copies of the target master copy to store change data; a second synchronization module 38 for selecting at least one second slave copy among the plurality of target slave copies of the target master copy to store the oplog.

It should be noted that, the acquiring module 32 in this embodiment may be used to perform step S102 in the embodiment of the present application, the determining module 34 in this embodiment may be used to perform step S104 in the embodiment of the present application, the first synchronizing module 36 in this embodiment may be used to perform step S106 in the embodiment of the present application, and the second synchronizing module 38 in this embodiment may be used to perform step S108 in the embodiment of the present application. The above modules are the same as examples and application scenarios implemented by the corresponding steps, but are not limited to what is disclosed in the above embodiments.

In the embodiment of the invention, the change data in the target main copy is acquired; determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data; selecting at least one first slave copy from a plurality of target slave copies of the target master copy to store change data; at least one second slave copy is selected from a plurality of target slave copies of the target master copy, so that the master copy and the slave copies can be synchronized based on both the change data and the operation log, and the data storage cost of the operation log is lower than that of the change data because the data volume of the operation log is smaller than that of the change data.

As an alternative embodiment, the obtaining module includes: the selecting unit is used for selecting the target main copy from a plurality of preset copies in the preset copy group; and the generating unit is used for responding to at least one data operation by the target main copy and generating change data.

As an alternative embodiment, the election unit comprises: the counting unit is used for counting the fact that each preset copy in the preset copy group receives the voting, wherein the voting represents that the preset copy is agreed to be used as a main copy; the detection unit is used for detecting the number of votes of the votes; the first determining unit is used for determining the preset copies with the voting number meeting the preset election rule as target main copies, wherein the preset election rule is that the voting number is the total number of the preset copies in the preset copy group or exceeds half of the total number of the preset copies in the preset copy group.

As an alternative embodiment, the election unit comprises: the identifying unit is used for identifying a preset copy group without a main copy as a target copy group, wherein the target copy group comprises: a plurality of target copies; a second determining unit, configured to determine, among a plurality of preset nodes in the distributed database, a plurality of target nodes for storing a target copy group, where each preset node includes: the system comprises a plurality of preset data blocks, a target node and a storage unit, wherein each preset data block is used for storing a preset copy, the preset copy comprises a target copy, and the preset data block used for storing the target copy exists in the target node; the monitoring unit is used for monitoring the number of the main copies in each target node, wherein a plurality of preset copies belonging to different preset copy groups exist in the target node, and the number of the main copies is the number of the main copies in the plurality of preset copies of the same target node; and the third determining unit is used for determining the target copy as the target master copy in the target nodes with the lowest master copy number.

As an alternative embodiment, the apparatus further comprises: a first determining subunit, configured to determine, among a plurality of preset copies in the preset copy group, a total number of copies of the preset copies in the preset copy group before electing the target primary copy; the selecting subunit is configured to select, from a plurality of preset nodes in the distributed database, a preset node of the number of target nodes as a target node, where each preset node includes: the system comprises a plurality of preset data blocks, a plurality of target nodes and a plurality of storage nodes, wherein each preset data block is used for storing preset copies, the number of the target nodes is the same as that of the total copies, and each target node is used for storing one preset copy; a second determining subunit, configured to determine, in each target node, a target data block, where the target data block is an idle preset data block; and the storage subunit is used for respectively placing a plurality of preset copies in the preset copy group into the target data blocks in the target node.

As an alternative embodiment, the second synchronization module comprises: a determining submodule, configured to determine, among a plurality of preset nodes in the distributed database, a plurality of slave copy nodes for storing target slave copies, where each preset node includes: the system comprises a plurality of preset data blocks, a target slave copy node and a target slave copy node, wherein each preset data block is used for storing a preset copy; the monitoring submodule is used for monitoring the log storage quantity in each slave copy node, wherein a plurality of preset slave copies belonging to different preset copy groups exist in the slave copy nodes, the preset slave copies keep synchronous with the master copy based on copy data, or keep synchronous with the master copy based on operation logs, and the log storage quantity is the quantity of fourth slave copies in the same slave copy node; and the selecting sub-module is used for selecting a second slave copy from the slave copy nodes with the log storage quantity lower than a preset quantity threshold value.

As an alternative embodiment, the apparatus further comprises: the acquisition sub-module is used for acquiring the commit vote of the preset copy group where the main copy is located after the operation log of the change data is determined, wherein the preset copy group comprises: the system comprises a plurality of preset copies, wherein each preset copy is a master copy or a slave copy, and the submitting vote represents agreement to submit an operation log; the detection sub-module is used for detecting the number of submitted votes; and the submitting sub-module is used for submitting the operation log to the slave copy under the condition that the number of submissions meets the preset submitting rule.

Embodiments of the present invention may provide a computer terminal, which may be any one of a group of computer terminals. Alternatively, in the present embodiment, the above-described computer terminal may be replaced with a terminal device such as a mobile terminal.

Alternatively, in this embodiment, the above-mentioned computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

In this embodiment, the above-mentioned computer terminal may execute the program code of the following steps in the replica synchronization method of the distributed database: acquiring change data in a target main copy; determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data; selecting at least one first slave copy from a plurality of target slave copies of the target master copy to store change data; at least one second slave copy is selected from among a plurality of target slave copies of the target master copy to store the oplog.

Alternatively, fig. 4 is a block diagram of a computer terminal according to an embodiment of the present invention. As shown in fig. 4, the computer terminal 10 may include: one or more (only one is shown) processors 42, and memory 44.

The memory may be used to store software programs and modules, such as program instructions/modules corresponding to the method and apparatus for synchronizing copies of a distributed database in the embodiments of the present invention, and the processor executes the software programs and modules stored in the memory, thereby executing various functional applications and data processing, that is, implementing the method for synchronizing copies of a distributed database. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include memory located remotely from the processor, which may be connected to the terminal 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: acquiring change data in a target main copy; determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data; selecting at least one first slave copy from a plurality of target slave copies of the target master copy to store change data; at least one second slave copy is selected from among a plurality of target slave copies of the target master copy to store the oplog.

Optionally, the above processor may further execute program code for: selecting a target main copy from a plurality of preset copies in a preset copy group; and generating change data by the target main copy in response to at least one data operation.

Optionally, the above processor may further execute program code for: counting that each preset copy in the preset copy group receives a voting vote, wherein the voting vote represents agreement of the preset copy as a main copy; detecting the number of votes of the votes; and determining the preset copies with the voting number meeting the preset election rule as target main copies, wherein the preset election rule is that the voting number is the total number of the preset copies in the preset copy group or exceeds half of the total number of the preset copies in the preset copy group.

Optionally, the above processor may further execute program code for: identifying a preset copy group without a master copy as a target copy group, wherein the target copy group comprises: a plurality of target copies; determining a plurality of target nodes for storing a target copy group in a plurality of preset nodes of the distributed database, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a target node and a storage unit, wherein each preset data block is used for storing a preset copy, the preset copy comprises a target copy, and the preset data block used for storing the target copy exists in the target node; monitoring the number of primary copies in each target node, wherein a plurality of preset copies belonging to different preset copy groups exist in the target node, and the number of the primary copies is the number of serving as the primary copies in the plurality of preset copies of the same target node; and determining the target copy as the target master copy in the target node with the lowest master copy number.

Optionally, the above processor may further execute program code for: determining the total copy number of the preset copies in the preset copy group before selecting a target main copy in a plurality of preset copies in the preset copy group; selecting preset nodes with the number of target nodes from a plurality of preset nodes of a distributed database as target nodes, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a plurality of target nodes and a plurality of storage nodes, wherein each preset data block is used for storing preset copies, the number of the target nodes is the same as that of the total copies, and each target node is used for storing one preset copy; in each target node, determining a target data block, wherein the target data block is an idle preset data block; and respectively placing a plurality of preset copies in the preset copy group into target data blocks in the target node.

Optionally, the above processor may further execute program code for: determining a plurality of slave replica nodes for storing target slave replicas among a plurality of preset nodes of the distributed database, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a target slave copy node and a target slave copy node, wherein each preset data block is used for storing a preset copy; monitoring the log storage quantity in each slave copy node, wherein a plurality of preset slave copies belonging to different preset copy groups exist in the slave copy node, and the preset slave copies are based on a third slave copy of which the copy data are kept synchronous with the master copy or based on a fourth slave copy of which the operation log is kept synchronous with the master copy, wherein the log storage quantity is the quantity of the fourth slave copies in the same slave copy node; and selecting a second slave copy from slave copy nodes with the log storage quantity lower than a preset quantity threshold value.

Optionally, the above processor may further execute program code for: after determining the operation log of the change data, acquiring a commit vote of a preset copy group where the main copy is located, wherein the preset copy group comprises: the system comprises a plurality of preset copies, wherein each preset copy is a master copy or a slave copy, and the submitting vote represents agreement to submit an operation log; detecting the number of submissions of the submitted votes; and submitting the operation log to the slave copy in the condition that the number of submissions meets the preset submission rule.

By adopting the embodiment of the invention, a copy synchronization scheme of the distributed database is provided. In the embodiment of the invention, the change data in the target main copy is acquired; determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data; selecting at least one first slave copy from a plurality of target slave copies of the target master copy to store change data; at least one second slave copy is selected from a plurality of target slave copies of the target master copy, so that the master copy and the slave copies can be synchronized based on both the change data and the operation log, and the data storage cost of the operation log is lower than that of the change data because the data volume of the operation log is smaller than that of the change data.

It will be appreciated by those skilled in the art that the configuration shown in fig. 4 is only illustrative, and the computer terminal may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm-phone computer, a mobile internet device (Mobile Internet Devices, MID), a PAD, etc. Fig. 4 is not limited to the structure of the electronic device. For example, the computer terminal 40 may also include more or fewer components (e.g., network interfaces, display devices, etc.) than shown in FIG. 4, or have a different configuration than shown in FIG. 4.

Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program for instructing a terminal device to execute on hardware associated with the terminal device, the program may be stored in a nonvolatile storage medium, and the nonvolatile storage medium may include: flash disk, read-Only Memory (ROM), random-access Memory (Random Access Memory, RAM), magnetic or optical disk, and the like.

Embodiments of the present invention also provide a nonvolatile storage medium. Alternatively, in this embodiment, the above-described nonvolatile storage medium may be used to store the program code executed by the copy synchronization method of the distributed database provided in the above-described embodiment.

Alternatively, in this embodiment, the above-mentioned nonvolatile storage medium may be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

Optionally, in the present embodiment, the non-volatile storage medium is arranged to store program code for performing the steps of: acquiring change data in a target main copy; determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data; selecting at least one first slave copy from a plurality of target slave copies of the target master copy to store change data; at least one second slave copy is selected from among a plurality of target slave copies of the target master copy to store the oplog.

Optionally, in the present embodiment, the non-volatile storage medium is arranged to store program code for performing the steps of: selecting a target main copy from a plurality of preset copies in a preset copy group; and generating change data by the target main copy in response to at least one data operation.

Optionally, in the present embodiment, the non-volatile storage medium is arranged to store program code for performing the steps of: counting that each preset copy in the preset copy group receives a voting vote, wherein the voting vote represents agreement of the preset copy as a main copy; detecting the number of votes of the votes; and determining the preset copies with the voting number meeting the preset election rule as target main copies, wherein the preset election rule is that the voting number is the total number of the preset copies in the preset copy group or exceeds half of the total number of the preset copies in the preset copy group.

Optionally, in the present embodiment, the non-volatile storage medium is arranged to store program code for performing the steps of: identifying a preset copy group without a master copy as a target copy group, wherein the target copy group comprises: a plurality of target copies; determining a plurality of target nodes for storing a target copy group in a plurality of preset nodes of the distributed database, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a target node and a storage unit, wherein each preset data block is used for storing a preset copy, the preset copy comprises a target copy, and the preset data block used for storing the target copy exists in the target node; monitoring the number of primary copies in each target node, wherein a plurality of preset copies belonging to different preset copy groups exist in the target node, and the number of the primary copies is the number of serving as the primary copies in the plurality of preset copies of the same target node; and determining the target copy as the target master copy in the target node with the lowest master copy number.

Optionally, in the present embodiment, the non-volatile storage medium is arranged to store program code for performing the steps of: determining the total copy number of the preset copies in the preset copy group before selecting a target main copy in a plurality of preset copies in the preset copy group; selecting preset nodes with the number of target nodes from a plurality of preset nodes of a distributed database as target nodes, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a plurality of target nodes and a plurality of storage nodes, wherein each preset data block is used for storing preset copies, the number of the target nodes is the same as that of the total copies, and each target node is used for storing one preset copy; in each target node, determining a target data block, wherein the target data block is an idle preset data block; and respectively placing a plurality of preset copies in the preset copy group into target data blocks in the target node.

Optionally, in the present embodiment, the non-volatile storage medium is arranged to store program code for performing the steps of: determining a plurality of slave replica nodes for storing target slave replicas among a plurality of preset nodes of the distributed database, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a target slave copy node and a target slave copy node, wherein each preset data block is used for storing a preset copy; monitoring the log storage quantity in each slave copy node, wherein a plurality of preset slave copies belonging to different preset copy groups exist in the slave copy node, and the preset slave copies are based on a third slave copy of which the copy data are kept synchronous with the master copy or based on a fourth slave copy of which the operation log is kept synchronous with the master copy, wherein the log storage quantity is the quantity of the fourth slave copies in the same slave copy node; and selecting a second slave copy from slave copy nodes with the log storage quantity lower than a preset quantity threshold value.

Optionally, in the present embodiment, the non-volatile storage medium is arranged to store program code for performing the steps of: after determining the operation log of the change data, acquiring a commit vote of a preset copy group where the main copy is located, wherein the preset copy group comprises: the system comprises a plurality of preset copies, wherein each preset copy is a master copy or a slave copy, and the submitting vote represents agreement to submit an operation log; detecting the number of submissions of the submitted votes; and submitting the operation log to the slave copy in the condition that the number of submissions meets the preset submission rule.

The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

In the foregoing embodiments of the present invention, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed technology content may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, for example, may be a logic function division, and may be implemented in another manner, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a non-volatile storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a non-volatile storage medium, including instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned nonvolatile storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.

Claims (10)

1. A method for synchronizing copies of a distributed database, comprising:

acquiring change data in a target main copy;

determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data;

selecting at least one first slave copy from a plurality of target slave copies of the target master copy to store the change data;

at least one second slave copy is selected from a plurality of target slave copies of the target master copy to store the operation log.

2. The method of claim 1, wherein obtaining change data in the target primary copy comprises:

selecting the target master copy from a plurality of preset copies of a preset copy group;

the change data is generated by the target primary replica in response to at least one data operation.

3. The method of claim 2, wherein electing the target primary replica among a plurality of pre-set replicas of a pre-set replica group comprises:

Counting the received voting votes of each preset copy in a preset copy group, wherein the voting votes represent agreeing to the preset copy as a main copy;

detecting the number of votes of the votes;

and determining the preset copies with the voting number meeting a preset election rule as the target main copy, wherein the preset election rule is that the voting number is the total number of the preset copies in the preset copy group or exceeds half of the total number of the preset copies in the preset copy group.

4. The method of claim 2, wherein electing the target primary replica among a plurality of pre-set replicas of a pre-set replica group comprises:

identifying a preset copy group without a main copy as a target copy group, wherein the target copy group comprises: a plurality of target copies;

determining a plurality of target nodes for storing the target copy group in a plurality of preset nodes of a distributed database, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a target node and a storage unit, wherein each preset data block is used for storing a preset copy, the preset copy comprises the target copy, and the preset data blocks used for storing the target copy exist in the target node;

Monitoring the number of primary copies in each target node, wherein a plurality of preset copies belonging to different preset copy groups exist in the target node, and the number of primary copies is the number of primary copies in the plurality of preset copies of the same target node;

and determining the target copy as the target master copy in the target node with the lowest master copy number.

5. The method of claim 2, wherein prior to electing the target primary replica among a plurality of pre-set replicas of a pre-set of replicas, the method further comprises:

determining the total number of the preset copies in the preset copy group;

selecting preset nodes with the number of target nodes from a plurality of preset nodes of a distributed database as target nodes, wherein each preset node comprises: the system comprises a plurality of preset data blocks, a target node and a storage unit, wherein each preset data block is used for storing preset copies, the number of the target nodes is the same as that of the total copies, and each target node is used for storing one preset copy;

determining a target data block in each target node, wherein the target data block is an idle preset data block;

And respectively placing a plurality of preset copies in the preset copy group into target data blocks in the target node.

6. The method of claim 1, wherein selecting at least one second slave backup among a plurality of target slave copies of the target master copy to store the oplog comprises:

determining a plurality of slave replica nodes for storing the target slave replica in a plurality of preset nodes of a distributed database, wherein each preset node comprises: the target slave copy comprises a target slave copy node, wherein the target slave copy node comprises a target slave copy node and a target slave copy node, and the target slave copy node comprises a plurality of target slave copy nodes;

monitoring the log storage quantity in each slave copy node, wherein a plurality of preset slave copies belonging to different preset copy groups exist in the slave copy nodes, the preset slave copies keep synchronous with a third slave copy of a master copy based on copy data or keep synchronous with a fourth slave copy of the master copy based on operation logs, and the log storage quantity is the quantity of the fourth slave copies in the same slave copy node;

And selecting the second slave copy from slave copy nodes with the log storage quantity lower than a preset quantity threshold value.

7. The method of claim 1, wherein after determining the oplog of change data, the method further comprises:

obtaining a commit vote of a preset copy group where the primary copy is located, wherein the preset copy group comprises: a plurality of pre-set copies, each of the pre-set copies being either the master copy or a slave copy, the commit vote representing consent to commit the oplog;

detecting the number of submissions of the submissions of votes;

and submitting the operation log to the slave copy under the condition that the number of submissions meets a preset submission rule.

8. A replica synchronization apparatus for a distributed database, comprising:

the acquisition module is used for acquiring the change data in the target main copy;

the determining module is used for determining an operation log of the change data, wherein the operation log is used for recording at least one data operation for generating the change data;

the first synchronization module is used for selecting at least one first slave copy from a plurality of target slave copies of the target master copy to store the change data;

And the second synchronization module is used for selecting at least one second slave copy from a plurality of target slave copies of the target master copy to store the operation log.

9. A non-volatile storage medium, wherein the non-volatile storage medium is configured to store a program, and wherein the program, when executed, controls a device in which the non-volatile storage medium is located to perform the copy synchronization method of the distributed database according to any one of claims 1 to 7.

10. An electronic device, comprising: a memory and a processor for executing a program stored in the processor, wherein the program is run to perform the replica synchronization method of the distributed database of any of claims 1 to 7.