CN111083192B - Data consensus method and device and electronic equipment - Google Patents
Data consensus method and device and electronic equipment Download PDFInfo
- Publication number
- CN111083192B CN111083192B CN201911073047.XA CN201911073047A CN111083192B CN 111083192 B CN111083192 B CN 111083192B CN 201911073047 A CN201911073047 A CN 201911073047A CN 111083192 B CN111083192 B CN 111083192B
- Authority
- CN
- China
- Prior art keywords
- node
- nodes
- data
- learning
- fragment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the disclosure provides a data consensus method, a data consensus device and electronic equipment, belonging to the technical field of data processing, wherein the method comprises the following steps: setting a plurality of fragmentation nodes for data storage; performing role management on the plurality of fragment nodes to enable a group of fragment nodes to comprise a main node, one or more following nodes and one or more learning nodes; and according to a preset data consensus strategy, enabling the following node and the learning node to perform data synchronization from the main node. The scheme of the disclosure can improve the efficiency of data consensus.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a data consensus method, apparatus, and electronic device.
Background
With the development of the internet, more and more applications need message interaction functions, such as a common instant messaging tool, a live broadcast room, and the like. The message service is a fundamental stone of user interaction in a live scene and is the most important service for all people to interact online at the same time. However, the most important key points of the live broadcast scene, which are different from instant chat tools such as WeChat, are as follows:
1. loose chat groups can be accessed at any time, and the relationship is weak.
2. The number of people in a chat group (room) who are online at the same time is unlimited and may reach the millions or even tens of millions instantaneously.
For the above reasons, the IM solution and architecture of the live scene are very different from the traditional IM. The instantaneous message reading flow can be increased sharply, so that the pressure of a server is very large, and for tens of millions of simultaneous online audiences, the pressure is great for the data support of a database.
Disclosure of Invention
In view of this, embodiments of the present disclosure provide a data consensus method, apparatus and electronic device, which at least partially solve the problems in the prior art.
In a first aspect, an embodiment of the present disclosure provides a data consensus method, including:
setting a plurality of fragmentation nodes for data storage;
performing role management on the plurality of fragment nodes to enable a group of fragment nodes to comprise a main node, one or more following nodes and one or more learning nodes;
and according to a preset data consensus strategy, enabling the following node and the learning node to perform data synchronization from the main node.
According to a specific implementation manner of the embodiment of the present disclosure, the setting a plurality of sharded nodes for data storage includes:
and deploying the fragmentation nodes for data storage according to a row-column arrangement architecture mode.
According to a specific implementation manner of the embodiment of the present disclosure, the performing role management on the plurality of sharded nodes includes:
and setting a plurality of fragmentation nodes into a master-slave structure by using the meta-server, so that only one master node exists at the same time.
According to a specific implementation manner of the embodiment of the present disclosure, the performing role management on the plurality of sharded nodes includes:
initializing all the fragment nodes in the cluster to be added through a meta-server, and uniformly distributing role states through the meta-server;
setting the state of the fragment node which is started when the fragment node is not initialized as standby;
and placing all the fragment node instances to be deployed and expanded in a cluster of a pool to wait for deployment and expansion.
According to a specific implementation manner of the embodiment of the present disclosure, the performing role management on the plurality of sharded nodes includes:
acquiring fragmentation nodes which are added into a cluster and are uniformly distributed;
utilizing the meta-server to uniformly distribute the state fragmentation nodes for timing activation detection;
and performing secondary role management on the fragment nodes based on the timing activation result.
According to a specific implementation manner of the embodiment of the present disclosure, the performing secondary role management on a segment node based on a result of timing activation includes:
when the main node activity detection failure exceeds a preset time, the meta-server actively switches the main nodes;
and after the old master node recovers the exploration, setting the old master node as a learning node for carrying out full-scale copy operation.
According to a specific implementation manner of the embodiment of the present disclosure, the performing secondary role management on a segment node based on a result of timing activation includes:
acquiring a learning node switching request sent by a following node;
determining whether to switch the following node sending the switching request into a learning node by using the meta-server;
and when the gap between the synchronous id of the following node and the id of the main node is larger than a preset threshold value, using the meta-server to set the following node with the gap between the synchronous id and the id of the main node larger than the preset threshold value as a learning node.
According to a specific implementation manner of the embodiment of the present disclosure, the synchronizing the following node and the learning node from the master node according to a preset data consensus policy includes:
and after the following node and the main node are switched to be learning nodes, performing full-quantity copy and incremental copy data synchronization operation on the switched learning nodes.
In a second aspect, an embodiment of the present disclosure provides a data consensus apparatus, including:
the device comprises a setting module, a data storage module and a data transmission module, wherein the setting module is used for setting a plurality of fragment nodes for data storage;
the management module is used for carrying out role management on the plurality of fragment nodes, so that a group of fragment nodes comprise a main node, one or more following nodes and one or more learning nodes;
and the synchronization module is used for synchronizing the data of the following node and the data of the learning node from the main node according to a preset data consensus strategy.
In a third aspect, an embodiment of the present disclosure further provides an electronic device, where the electronic device includes:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method of Ren Di as described above in any implementation of the one aspect or the first aspect.
In a fourth aspect, the disclosed embodiments also provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the data consensus method in the first aspect or any implementation manner of the first aspect.
In a fifth aspect, the present disclosure also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the data consensus method in the first aspect or any implementation manner of the first aspect.
The data consensus scheme in the embodiment of the disclosure comprises setting a plurality of fragmentation nodes for data storage; performing role management on the plurality of fragment nodes to enable a group of fragment nodes to comprise a main node, one or more following nodes and one or more learning nodes; and according to a preset data consensus strategy, enabling the following node and the learning node to perform data synchronization from the main node. By the scheme, the consistency of data in the nodes in the distributed system can be guaranteed.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present disclosure, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic diagram of a data consensus process provided by an embodiment of the present disclosure;
fig. 2 is an interaction diagram of a data consensus system provided by an embodiment of the present disclosure;
fig. 3 is a schematic diagram of another data consensus process provided by the embodiments of the present disclosure;
fig. 4 is a schematic diagram of another data consensus process provided by the embodiments of the present disclosure;
fig. 5 is a schematic structural diagram of a data consensus device according to an embodiment of the present disclosure;
fig. 6 is a schematic diagram of an electronic device provided in an embodiment of the present disclosure.
Detailed Description
The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
The embodiments of the present disclosure are described below with specific examples, and other advantages and effects of the present disclosure will be readily apparent to those skilled in the art from the disclosure in the specification. It is to be understood that the described embodiments are merely illustrative of some, and not restrictive, of the embodiments of the disclosure. The disclosure may be embodied or carried out in various other specific embodiments, and various modifications and changes may be made in the details within the description without departing from the spirit of the disclosure. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without inventive step, are intended to be within the scope of the present disclosure.
It is noted that various aspects of the embodiments are described below within the scope of the appended claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is merely illustrative. Based on the disclosure, one skilled in the art should appreciate that one aspect described herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method practiced using any number of the aspects set forth herein. Additionally, such an apparatus may be implemented and/or such a method may be practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present disclosure, and the drawings only show the components related to the present disclosure rather than the number, shape and size of the components in actual implementation, and the type, amount and ratio of the components in actual implementation may be changed arbitrarily, and the layout of the components may be more complicated.
In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects may be practiced without these specific details.
The embodiment of the disclosure provides a data consensus method. The data consensus method provided by the embodiment can be executed by a computing device, which can be implemented as software or as a combination of software and hardware, and can be integrally arranged in a server, a terminal device, and the like.
In a distributed system, consistency refers to: given a series of operations, a number of service nodes in the system attempt to bring them into some degree of agreement on the processing results, under the guarantee of an agreement (often by some consensus algorithm). Consistency does not represent whether the result is correct or not, but whether the state presented by the system to the outside is consistent or not, for example, whether all nodes reach a failure state is consistent.
Network communications between nodes in a distributed system are unreliable, including any delays and content failures. The processing of the nodes may be erroneous or even the nodes themselves may be down at any time, in which case the synchronous invocation may make the system non-scalable.
In order to ensure the consistency of data in the distributed system, the consistency should satisfy the following conditions for the distributed system:
termination (Termination) consistent results can be completed in a limited time;
consensus (Consensus) that the final decision-making results of different nodes should be the same;
validity-the outcome of a decision is necessarily a proposal by other processes.
To ensure that the system meets different degrees of consistency, consensus algorithms are often needed. The consensus algorithm solves the process of agreeing on a proposal and nodes in the system.
Referring to fig. 1 and fig. 2, a data consensus method provided by the embodiment of the present disclosure includes the following steps:
s101, setting a plurality of slicing nodes for data storage.
The sharded nodes are basic units for data storage, and data related to the system can be stored on each sharded unit. In order to manage the fragmentation nodes, a message server may be provided, and a plurality of fragmentation nodes are managed by the message server.
The fragmentation nodes can be deployed in various ways, and as one way, the fragmentation nodes can be deployed in a row + column architecture way, that is, a plurality of fragmentation nodes and a plurality of replica nodes are correspondingly arranged.
Data replication on message server (msgserver) fragments can be realized by adopting a master-slave replication mechanism, so that the consistency of data of fragment nodes on the fragment message server is ensured.
S102, performing role management on the plurality of fragment nodes, so that a group of fragment nodes comprises a main node, one or more following nodes and one or more learning nodes.
To manage a fragment node, a plurality of role states may be set for the fragment node: the main node (leader), the following node (follower) and the learning node (leaner) can effectively synchronize data on the fragment nodes by carrying out role management on a plurality of fragment nodes.
For this purpose, a meta-server (metaserver) is provided, through which the role of the fragment node is set. The meta-server sets a master-slave architecture for the fragment nodes, and the locking mechanism ensures that only one master node is arranged at the same time.
The meta information on the meta server is stored by, for example, byteKV and Redis doubly, and specifically, the meta information is subject to ByteKV, and the Redis makes weak dependence. And (4) supporting degradation, switching the meta-server is not allowed during the degradation, and data update is written into the redis. When the degradation is canceled, the data is restored to the ByteKV
The instance providing service on all the fragment nodes needs to be added into the cluster through the meta-server as initialization, and the meta-server uniformly distributes role states for the fragment nodes. The fragment node on the message server is started when not initialized, and the state is a Standby state (Standby). All message server instances to be deployed and expanded are placed in a cluster (cluster) of a pool (pool) to wait for deployment and expansion.
After the meta-server uniformly distributes the states, the fragment nodes added into the cluster perform timed activation (ping) on all the fragment nodes (examples), and the message server returns the sync _ id inside the current state machine in the activation response result.
And when the main node fails to detect the activity for a certain time, the meta-server actively switches the main nodes, and selects the node with the largest sync _ id in the ping result as a new main node. And setting the old master node as a learning node. And when the old master node recovers the exploration, the old master node is used as a learning node to perform full-scale replication.
And when the id gap (lag) between the following node and the main node is greater than a certain threshold value, actively applying for switching the learning node to the meta-server. The meta-server decides whether switching can be carried out or not, and when the meta-server finds that the gap between the sync _ id of a certain following node and the id of the main node is larger than a certain threshold value, the node is set as a learning node actively.
And S103, synchronizing the data of the following node and the learning node from the main node according to a preset data consensus strategy.
After the following node and the main node are switched to learning nodes, data synchronization can be performed from the main node in a full replication and incremental replication mode. In addition, it may be configured that only the following node provides a data reading request to the outside, the master node is only used to store a WAL (Write-ahead log), and conversion and storage of a real data structure are not performed.
Through the scheme, all the instances in the whole fragment node can achieve consensus, and the final consistency of data can be ensured through incremental synchronization and full synchronization. Meanwhile, the meta-server can ensure that under the condition that network partitions occur, nodes out of synchronization with the master node in the partitions are quickly shielded, and therefore usability is ensured.
According to a specific implementation manner of the embodiment of the present disclosure, the setting a plurality of sharded nodes for data storage includes: and deploying the fragmentation nodes for data storage according to a row-column arrangement architecture mode. By the mode, the fragment nodes can be managed conveniently.
According to a specific implementation manner of the embodiment of the present disclosure, the performing role management on the plurality of sharded nodes includes: and setting a plurality of fragmentation nodes into a master-slave structure by using the meta-server, so that only one master node exists at the same time. By the method, data can be synchronized conveniently.
Referring to fig. 3, according to a specific implementation manner of the embodiment of the present disclosure, the performing role management on the sharded nodes includes:
s301, initializing all the fragmentation nodes in the cluster to be added through the meta-server, and uniformly distributing role states by the meta-server.
The meta-server can uniformly manage all the nodes added into the cluster, and the role management can be performed on the fragment nodes in the cluster through the mode.
S302, the state of the fragment node which is started when the fragment node is not initialized is set to be standby.
And S303, placing all the fragment node instances to be deployed and expanded in a cluster of a pool to wait for deployment and expansion.
By the method, the nodes in the pool can be uniformly managed.
Referring to fig. 4, according to a specific implementation manner of the embodiment of the present disclosure, the performing role management on the sharded nodes includes:
s401, the fragmentation nodes which are added into the cluster and are in the uniformly distributed state are obtained.
S402, the partitioned nodes in the state distributed uniformly by the meta-server are utilized to carry out timing activation detection.
By carrying out the activity detection operation on the fragmentation node, whether the node is in a normal communication state can be checked, so that the fragmentation node with abnormal communication can be found out conveniently.
And S403, performing secondary role management on the fragment nodes based on the timing detection result.
Specifically, when the failure of the master node activity detection exceeds a preset time, the meta-server actively switches the master node; and after the old master node recovers the exploration, setting the old master node as a learning node for carrying out full-scale copy operation. By the method, the state of the fragment node can be searched in time, and role management is performed.
According to a specific implementation manner of the embodiment of the present disclosure, the performing secondary role management on a segment node based on a result of timing activation includes: acquiring a learning node switching request sent by a following node; determining whether to switch the following node sending the switching request into a learning node by using the meta-server; and when the gap between the synchronous id of the following node and the id of the main node is larger than a preset threshold value, using the meta-server to set the following node with the gap between the synchronous id and the id of the main node larger than the preset threshold value as a learning node.
According to a specific implementation manner of the embodiment of the present disclosure, the synchronizing the following node and the learning node from the master node according to a preset data consensus policy includes: and after the following node and the main node are switched to be learning nodes, performing full-quantity copy and incremental copy data synchronization operation on the switched learning nodes. By the method, effective data synchronization operation can be performed on the fragment nodes.
Corresponding to the above method embodiment, referring to fig. 5, an embodiment of the present disclosure further provides a data consensus apparatus 50, including:
a setting module 501, configured to set a plurality of sharded nodes for data storage.
The sharded nodes are basic units for data storage, and data related to the system can be stored on each sharded unit. In order to manage the fragmentation nodes, a message server may be provided, and a plurality of fragmentation nodes are managed by the message server.
The fragmentation nodes can be deployed in various ways, and as one way, the fragmentation nodes can be deployed in a row + column architecture way, that is, a plurality of fragmentation nodes and a plurality of replica nodes are correspondingly arranged.
Data replication on a message server (msgserver) fragment can be realized by adopting a master-slave replication mechanism, so that the consistency of data of fragment nodes on the fragment message server is ensured.
A management module 502, configured to perform role management on the multiple segment nodes, so that a group of segment nodes includes a master node, one or more follower nodes, and one or more learning nodes.
To manage a fragment node, a plurality of role states may be set for the fragment node: the main node (leader), the following node (follower) and the learning node (leaner) can effectively synchronize data on the fragment nodes by carrying out role management on a plurality of fragment nodes.
For this, a meta-server (metaserver) is provided, and the role of the fragmentation node is set through the meta-server. The meta-server sets a master-slave architecture for the fragment nodes, and the locking mechanism ensures that only one master node is arranged at the same time.
The meta information on the meta server is stored by, for example, byteKV and Redis doubly, and specifically, the meta information is subject to ByteKV, and the Redis makes weak dependence. And (4) supporting degradation, switching the meta-server is not allowed during the degradation, and data update is written into the redis. When the degradation is canceled, the data is restored to the ByteKV
The instance providing service on all the fragment nodes needs to be added into the cluster through the meta-server as initialization, and the meta-server uniformly distributes role states for the fragment nodes. The fragment node on the message server is started when not initialized, and the state is a Standby state (Standby). All message server instances to be deployed and expanded are placed in a cluster (cluster) of a pool (pool) to wait for deployment and expansion.
After the meta-server uniformly distributes the states, the fragment nodes added into the cluster perform timed activation (ping) on all the fragment nodes (examples), and the message server returns the sync _ id inside the current state machine in the activation response result.
And when the main node fails to detect the activity for a certain time, the meta-server actively switches the main nodes, and selects the node with the largest sync _ id in the ping result as a new main node. And setting the old master node as a learning node. And when the old master node recovers the exploration, the old master node is used as a learning node to perform full-scale replication.
And when the id gap (lag) between the following node and the main node is larger than a certain threshold value, actively applying for switching the learning node to the meta-server. The meta-server decides whether switching can be carried out or not, and when the meta-server finds that the gap between the sync _ id of a certain following node and the id of the main node is larger than a certain threshold value, the node is set as a learning node actively.
A synchronization module 503, configured to synchronize, according to a preset data consensus policy, data of the following node and the learning node from the master node.
After the following node and the main node are switched to learning nodes, data synchronization can be performed from the main node in a full replication and incremental replication mode. In addition, it may be configured that only the following node provides a data reading request to the outside, the master node is only used to store a WAL (Write-ahead log), and conversion and storage of a real data structure are not performed.
Through the scheme, all the instances in the whole fragment node can achieve consensus, and the final consistency of data can be ensured through incremental synchronization and full synchronization. Meanwhile, the meta-server can ensure that under the condition of network partition, the nodes which are not synchronous with the host node in the partition are quickly shielded, so that the usability is ensured.
The apparatus shown in fig. 5 may correspondingly execute the content in the above method embodiment, and details of the part not described in detail in this embodiment refer to the content described in the above method embodiment, which is not described again here.
Referring to fig. 6, an embodiment of the present disclosure also provides an electronic device 60, including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data consensus method of the preceding method embodiment.
Embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the foregoing method embodiments.
Embodiments of the present disclosure also provide a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the data consensus method in the aforementioned method embodiments.
Referring now to FIG. 6, a schematic diagram of an electronic device 60 suitable for use in implementing embodiments of the present disclosure is shown. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., car navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 6, the electronic device 60 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 60 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 60 to communicate with other devices wirelessly or by wire to exchange data. While the figures illustrate an electronic device 60 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, the processes described above with reference to the flow diagrams may be implemented as computer software programs, according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 609, or installed from the storage means 608, or installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of embodiments of the present disclosure.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring at least two internet protocol addresses; sending a node evaluation request comprising the at least two internet protocol addresses to node evaluation equipment, wherein the node evaluation equipment selects the internet protocol addresses from the at least two internet protocol addresses and returns the internet protocol addresses; receiving an internet protocol address returned by the node evaluation equipment; wherein the obtained internet protocol address indicates an edge node in the content distribution network.
Alternatively, the computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: receiving a node evaluation request comprising at least two internet protocol addresses; selecting an internet protocol address from the at least two internet protocol addresses; returning the selected internet protocol address; wherein the received internet protocol address indicates an edge node in the content distribution network.
Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of a unit does not in some cases constitute a limitation of the unit itself, for example, the first retrieving unit may also be described as a "unit for retrieving at least two internet protocol addresses".
It should be understood that portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.
The above description is only for the specific embodiments of the present disclosure, but the scope of the present disclosure is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present disclosure should be covered within the scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (10)
1. A data consensus method, comprising:
setting a plurality of fragmentation nodes for data storage;
performing role management on the plurality of fragment nodes to enable a group of fragment nodes to comprise a main node, one or more following nodes and one or more learning nodes;
according to a preset data consensus strategy, enabling the following node and the learning node to perform data synchronization from the main node;
the preset data consensus strategy enables the following node and the learning node to perform data synchronization from the main node, and comprises the following steps:
the main node is used for storing a pre-written log;
the one or more following nodes provide data reading requests to the outside; and
and the one or more learning nodes perform data synchronization from the main node in a full-quantity replication or incremental replication mode.
2. The method according to claim 1, wherein the setting a plurality of sharded nodes for data storage comprises:
and deploying the fragmentation nodes for data storage according to a row-column arrangement architecture mode.
3. The method of claim 1, wherein the managing the roles of the sharded nodes comprises:
and setting a plurality of fragmentation nodes into a master-slave structure by using the meta-server, so that only one master node exists at the same time.
4. The method of claim 3, wherein the role managing the plurality of sharded nodes comprises:
initializing all fragment nodes needing to be added into a cluster through a meta-server, and uniformly distributing role states through the meta-server;
setting the state of the fragment node which is started when the fragment node is not initialized as standby;
and placing all the fragment node instances to be deployed and expanded in a cluster of a pool to wait for deployment and expansion.
5. The method of claim 3, wherein the managing the roles of the plurality of sharded nodes comprises:
acquiring fragmentation nodes which are added into a cluster and are uniformly distributed;
utilizing the meta-server to uniformly distribute the state fragmentation nodes for timing activation detection;
and performing secondary role management on the fragment nodes based on the timing activation result.
6. The method of claim 5, wherein performing secondary role management on the fragmented nodes based on the timing activation result comprises:
when the main node activity detection failure exceeds a preset time, the meta-server actively switches the main nodes;
and after the old master node recovers the exploration, setting the old master node as a learning node for carrying out full-scale copy operation.
7. The method of claim 5, wherein performing secondary role management on the fragmented nodes based on the timing activation result comprises:
acquiring a learning node switching request sent by a following node;
determining whether to switch the following node sending the switching request into a learning node by using the meta-server;
and when the gap between the synchronous id of the following node and the id of the main node is larger than a preset threshold value, the following node, of which the gap between the synchronous id and the id of the main node is larger than the preset threshold value, is set as a learning node by using the meta-server.
8. A data consensus apparatus, comprising:
the device comprises a setting module, a data storage module and a data transmission module, wherein the setting module is used for setting a plurality of fragment nodes for data storage;
the management module is used for carrying out role management on the plurality of fragment nodes, so that a group of fragment nodes comprise a main node, one or more following nodes and one or more learning nodes;
the synchronization module is used for enabling the following node and the learning node to perform data synchronization from the main node according to a preset data consensus strategy;
wherein, the preset data consensus strategy for synchronizing the following node and the learning node from the master node comprises:
the main node is used for storing a pre-written log;
the one or more following nodes provide data reading requests to the outside; and
and the one or more learning nodes perform data synchronization from the main node in a full-quantity copying and incremental copying mode.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and (c) a second step of,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data consensus method of any one of the preceding claims 1-7.
10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the data consensus method of any one of the preceding claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911073047.XA CN111083192B (en) | 2019-11-05 | 2019-11-05 | Data consensus method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911073047.XA CN111083192B (en) | 2019-11-05 | 2019-11-05 | Data consensus method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111083192A CN111083192A (en) | 2020-04-28 |
| CN111083192B true CN111083192B (en) | 2023-02-17 |
Family
ID=70310686
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911073047.XA Active CN111083192B (en) | 2019-11-05 | 2019-11-05 | Data consensus method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111083192B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111935320B (en) * | 2020-09-28 | 2021-01-05 | 腾讯科技（深圳）有限公司 | Data synchronization method, related device, equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103973725A (en) * | 2013-01-28 | 2014-08-06 | 阿里巴巴集团控股有限公司 | Distributed collaboration method and collaboration device |
| CN106936931A (en) * | 2017-04-26 | 2017-07-07 | 华为技术有限公司 | The implementation method of distributed lock, relevant device and system |
| CN107919977A (en) * | 2016-10-11 | 2018-04-17 | 阿里巴巴集团控股有限公司 | A kind of on-line rapid estimation of the distributed consensus system based on Paxos agreements, the method and apparatus of online capacity reducing |
| CN108347455A (en) * | 2017-01-24 | 2018-07-31 | 阿里巴巴集团控股有限公司 | Metadata exchange method and system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10643147B2 (en) * | 2016-05-31 | 2020-05-05 | International Business Machines Corporation | Coordinated version control system, method, and recording medium for parameter sensitive applications |
| CN108073656A (en) * | 2016-11-17 | 2018-05-25 | 杭州华为数字技术有限公司 | A kind of method of data synchronization and relevant device |
| CN110377570B (en) * | 2017-10-12 | 2021-06-11 | 腾讯科技（深圳）有限公司 | Node switching method and device, computer equipment and storage medium |
| CN108243253B (en) * | 2018-01-11 | 2020-05-12 | 杭州复杂美科技有限公司 | Block chain link point synchronization method |
| CN109978528B (en) * | 2019-03-15 | 2020-05-12 | 北京世纪诚链科技有限公司 | Pluggable consensus protocol framework model, consensus protocol and implementation method thereof |
-
2019
- 2019-11-05 CN CN201911073047.XA patent/CN111083192B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103973725A (en) * | 2013-01-28 | 2014-08-06 | 阿里巴巴集团控股有限公司 | Distributed collaboration method and collaboration device |
| CN107919977A (en) * | 2016-10-11 | 2018-04-17 | 阿里巴巴集团控股有限公司 | A kind of on-line rapid estimation of the distributed consensus system based on Paxos agreements, the method and apparatus of online capacity reducing |
| CN108347455A (en) * | 2017-01-24 | 2018-07-31 | 阿里巴巴集团控股有限公司 | Metadata exchange method and system |
| CN106936931A (en) * | 2017-04-26 | 2017-07-07 | 华为技术有限公司 | The implementation method of distributed lock, relevant device and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111083192A (en) | 2020-04-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107295080B (en) | Data storage method applied to distributed server cluster and server | |
| EP4033374A1 (en) | Method and device for synchronizing node data | |
| CN109951331B (en) | Method, device and computing cluster for sending information | |
| CN109522363B (en) | Cloud platform synchronization method, system, equipment and storage medium based on block chain | |
| CN111639061B (en) | Data management method, device, medium and electronic equipment in Redis cluster | |
| CN110795503A (en) | Multi-cluster data synchronization method and related device of distributed storage system | |
| CN111104069B (en) | Multi-region data processing method and device of distributed storage system and electronic equipment | |
| US20170193070A1 (en) | System and method for a distributed replication lock for active-active geo-redundant systems | |
| CN110909521B (en) | Online document information synchronous processing method and device and electronic equipment | |
| EP3533198B1 (en) | Highly available and reliable secret distribution infrastructure | |
| CN103701913A (en) | Data synchronization method and device | |
| CN113051110A (en) | Cluster switching method, device and equipment | |
| CN109471647B (en) | Data updating method and device, electronic equipment and readable medium | |
| CN111338834B (en) | Data storage method and device | |
| CN111259022A (en) | Information synchronization method, synchronization system, computer equipment and medium | |
| CN113407637A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN111083192B (en) | Data consensus method and device and electronic equipment | |
| CN114780615A (en) | Error code management method and device thereof | |
| CN113259476B (en) | Message pushing method and system | |
| US10481833B2 (en) | Transferring data encoding functions in a distributed storage network | |
| CN111177260A (en) | Database remote copying method and device and electronic equipment | |
| CN112363815A (en) | Redis cluster processing method and device, electronic equipment and computer readable storage medium | |
| CN113391882B (en) | Virtual machine memory management method and device, storage medium and electronic equipment | |
| CN105511808A (en) | Data operation method, system and related device | |
| CN113420400B (en) | Routing relation establishment method, request processing method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |