CN106951443B - Method, equipment and system for synchronizing copies based on distributed system - Google Patents
Method, equipment and system for synchronizing copies based on distributed system Download PDFInfo
- Publication number
- CN106951443B CN106951443B CN201710082040.9A CN201710082040A CN106951443B CN 106951443 B CN106951443 B CN 106951443B CN 201710082040 A CN201710082040 A CN 201710082040A CN 106951443 B CN106951443 B CN 106951443B
- Authority
- CN
- China
- Prior art keywords
- log
- copy
- information
- version
- replica
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The invention aims to provide a method, equipment and a system for copy synchronization based on a distributed system. The distributed system comprises a plurality of common copies with log information located locally and at least one read-only copy with log information located in a file system, wherein the common copies comprise a master copy and at least one slave copy. The advantages according to the invention are: the system needs fewer copies, and has no special requirements on the random read-write performance of the copies, so that the cost of the whole system is reduced, and meanwhile, the performance indexes of the system, such as delay, throughput capacity and the like, can still be the same as the copy synchronization mode based on the distributed consistency algorithm. Moreover, the system can be smoothly upgraded to the scheme of the invention on the basis of the existing synchronization scheme based on the remote file system, thereby having wide implementation and application scenes.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, and a system for copy synchronization based on a distributed system.
Background
In the prior art, two copy synchronization modes are generally available. One is a synchronization method based on a distributed consistency algorithm, and the other is a synchronization method based on a remote file system.
The copy synchronization method based on the distributed consistent algorithm has low time delay and high throughput because different copies can process data concurrently, but the method has higher requirement on the number of the copies, generally needs at least 3 copies, and most services require the copies to have better random read-write performance, thereby causing higher overall cost.
Based on the copy synchronization method of the remote file system, the updating operation is stored in the distributed file system in a log mode, and the updating operation is only in charge of business logic. The distributed file system adopted by the system stores log data, the log data is mainly read and written sequentially, and the overall performance is not affected even if a cheap storage medium is adopted. Considering master-slave switching when a replica fails, 2 replicas of the system itself are sufficient.
This feature makes the cost advantage of this type of system obvious. However, since the system needs to store the additional log at the remote end and then apply the log, the delay and throughput performance are poor, and the reliability and availability of the system are limited by the distributed file system.
Disclosure of Invention
The invention aims to provide a method, equipment and a system for copy synchronization based on a distributed system.
According to an aspect of the present invention, there is provided a method for performing backup in a copy of a distributed system, wherein the distributed system includes a plurality of normal copies having log information located locally and at least one read-only copy having log information located in a file system, wherein the plurality of normal copies includes a master copy and at least one slave copy, and wherein, in the method, each copy in the distributed system performs the following steps:
-comparing the log index information with the respective copies to determine a master copy in the distributed system based on the log index information of the respective copies, wherein the log index information comprises a copy version and a log version.
According to an aspect of the present invention, there is further provided a distributed system, wherein the distributed system includes a plurality of common copies of log information located locally and at least one read-only copy of log information located in a file system, and the plurality of common copies includes one master copy and at least one slave copy.
According to another aspect of the present invention, there is also provided a copy for implementing backup in a distributed system, wherein the distributed system includes a plurality of normal copies having log information located locally and at least one read-only copy having log information located in a file system, wherein the plurality of normal copies includes a master copy and at least one slave copy, and wherein the copy includes:
means for comparing log index information to the respective copies to determine a master copy in the distributed system based on the log index information of the respective copies, wherein the log index information includes a copy version and a log version.
According to another aspect of the present invention, there is also provided a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method when executing the program.
According to a further aspect of the invention, a computer-readable storage medium is also provided, on which a computer program is stored, characterized in that the program realizes the method when executed by a processor.
Compared with the prior art, the invention has the following advantages: since the read-only copy is a logical abstraction of the remote file system, and is located on the same physical machine as the normal copy, no new network delay is introduced, so the delay according to the scheme is the same as the copy synchronization method based on the distributed consistency algorithm, depending on the copy with the median delay value. Moreover, the file system in the scheme is not a single point of the system any more, so that the overall read-write performance of the system cannot be reduced, and the overall performance of the system cannot be influenced or write failure cannot be caused under the condition that less than most copies fail or slow down.
According to the scheme of the invention, when the local of the copy to be restored has no log information, the log information is only directly obtained from the file system for application, and the data restoration efficiency is improved.
In general, according to the distributed system of the present solution, due to the adoption of the structure including both the copy of the distributed structure, the read-only copy and the corresponding file system, the number of copies required by the system is small, and no special requirement is imposed on the random read-write performance of the read-only copy, and a cheaper storage medium can be adopted, so that the cost of the whole system is reduced, and meanwhile, the performance indexes of the system, such as delay, throughput capability and the like, can still be the same as the copy synchronization mode based on the distributed consistency algorithm. Moreover, the system can be smoothly upgraded to the scheme of the invention on the basis of the existing synchronization scheme based on the remote file system, thereby having wide implementation and application scenes.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments made with reference to the following drawings:
FIG. 1 illustrates a system architecture diagram of a distributed system for replica synchronization according to the present invention;
FIG. 2 illustrates a method flow diagram for replica synchronization in a distributed system in accordance with the present invention;
fig. 3 is a schematic diagram illustrating a structure of an apparatus for replica synchronization in a distributed system according to the present invention.
The same or similar reference numbers in the drawings identify the same or similar elements.
Detailed Description
The present invention is described in further detail below with reference to the attached drawing figures.
Referring to fig. 1, fig. 1 illustrates a system structure diagram of a distributed system for replica synchronization according to the present invention.
The distributed system comprises a plurality of common copies with log information located locally and at least one read-only copy with log information located in a file system, wherein the common copies comprise a master copy and at least one slave copy.
The log information according to the present invention includes a log file, and preferably, the log information further includes a log snapshot generated based on the log file. The primary copy is used for providing read-write service for the client, and the secondary copy is used for data backup serving as the primary copy.
Wherein the plain copy and the read-only copy are both implemented by a computer device. The computer device includes an electronic device capable of automatically performing numerical calculation and/or information processing according to instructions set or stored in advance, and hardware thereof includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a programmable gate array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like. The computer device comprises a network device.
Wherein the network device includes, but is not limited to, a single network server, a server group consisting of a plurality of network servers, or a Cloud Computing (Cloud Computing) based Cloud consisting of a large number of hosts or network servers, wherein Cloud Computing is one of distributed Computing, a super virtual computer consisting of a collection of loosely coupled computers.
The network where the network device is located includes, but is not limited to, the internet, a wide area network, a metropolitan area network, a local area network, a VPN network, and the like. It should be noted that the user equipment, the network device and the network are only examples, and other existing or future network devices and networks may be applicable to the present invention, and are included in the scope of the present invention and are included by reference.
According to a preferred embodiment of the present invention, a common copy and a read-only copy may be located on the same computer device, and the computer device implements the logic functions of the common copy and the read-only copy simultaneously.
Preferably, when the common copy becomes the master copy, the computer device is simultaneously activated to perform the logical function of the read-only copy, so as to realize that the master copy and the read-only copy are on the same computer device.
Wherein each replica, including a read-only replica, according to aspects of the present invention can determine a primary replica in the distributed system based on the log index information of each replica by comparing the log index information with other respective replicas.
Wherein the log index information includes a copy version and a log version. The copy version and the log version are both incremental.
In particular, the replica version is expressed in terms of a term value of the replica, wherein the term value represents the version number of the replica set, and the term value is increased by 1 each time an operation of selecting the primary replica is performed.
The log version is usually expressed by a log index, wherein the log index is increased by 1 every time a log is newly created by the request issuing end (i.e., the client or the primary replica).
Wherein, the term value and logIndex value both need to be persisted with the log to ensure that they are not lost.
Specifically, in the master selection operation, each copy including the read-only copy and the other copies respectively compare respective copy version information to determine one or more common copies with the highest copy version, and then each copy of the one or more copies compares respective log versions with each other and determines the copy with the highest log version as the master copy; preferably, the log index information of other copies is received by a copy and compared to determine one or more copies with the highest copy version, and then the copy continues to compare the respective log versions of the determined copies of the one or more copies and determines the copy with the highest log version as the primary copy.
Preferably, when the selected copy is a read-only copy, any normal copy is determined as the primary copy.
Reference is made to fig. 2 according to an aspect of the present invention. Fig. 2 illustrates a schematic flow diagram for log distribution in a distributed system according to the present invention. The method according to the scheme comprises a step S101, a step S102 and a step S103 which are executed by a request distribution terminal; and step S201, step S202, step S203, and step S204 performed by the normal copy, and step S201', step S202', and step S203' performed by the read-only copy.
Wherein, the request distributing terminal can be any one of the following:
1) a client;
2) a primary replica.
Preferably, according to the solution of the invention, the request-issuing side is the primary replica.
In step S101, the request distribution side generates log update information based on the user request.
Specifically, the request distribution terminal packs the write request of the client into a piece of log information, and increases the log version LogIndex of the log information by 1.
Next, in step S102, the request distribution terminal transmits the log update information to each copy of the distributed system.
In step S201, the common copy receives log update information from the request issuing end;
next, in step S202, the ordinary copy updates the local log based on the obtained log update information.
Next, in step S203, the normal copy replies to the request issuing end to receive a commit instruction from the request issuing end.
Meanwhile, in step S201', the read-only copy receives log update information from the request issuing end; and in step S202', the read-only copy updates the log information of the corresponding file system based on the received log update information.
Next, in step S203', the read-only copy replies to the request issuing end.
Next, in step S103, when the request issuing side determines that the received copy replies more than the predetermined number, it is determined that the log becomes the commit state, and a commit instruction is sent to each copy.
Preferably, when the slave copies in the distributed system do not provide services, the request issuing end may first send a commit instruction to the master copy through step S103, and send the commit instruction to each slave copy piggybacked in a log update message of a new request by executing step S103 while sending a log update request next time.
Preferably, the predetermined number is half of all copies including the read-only copy.
Next, in step S204, the common copy applies the current log information and feeds back that the request issuing end successfully submits.
Preferably, when the copy is a primary copy, the scheme according to the present invention further includes step S205 (not shown).
In step S205, after the master copy starts to apply the log information, the master copy notifies the request issuing end so that the request issuing end replies that the data update of the client is completed; and when the application is completed, allowing the client to read the applied log information.
More preferably, the master copy may notify the request issuing end after the application log information is completed, so that the request issuing end replies that the client data update is completed.
According to the method of the invention, since the read-only copy is a logical abstraction of the remote file system, and is located on the same physical machine as the primary copy, no new network latency is introduced, so the latency according to the scheme is the same as that of the copy synchronization method based on the distributed consistency algorithm, depending on the copy with the median latency value. Moreover, the file system in the scheme is not a single point of the system any more, so that the overall read-write performance of the system cannot be reduced, and the overall performance of the system cannot be influenced or write failure cannot be caused under the condition that less than most copies fail or slow down.
According to a preferred embodiment of the present invention, when a copy is added as a copy to be recovered to the distributed system according to the present invention, the copy to be recovered further performs step S204 '(not shown) and step S205' (not shown) to recover the data of the copy to be recovered.
In step S204', the copy to be restored is loaded and the latest log information in the file system is applied.
Next, in step S205', the to-be-restored copy searches, in the log information of the file system, for the submitted log information whose log version is greater than the latest log information, and sequentially performs application search on the obtained log information.
Preferably, when the local log exists in the copy to be restored, the method according to this embodiment further includes step S206 (not shown), step S207 (not shown), step S208 (not shown), and step S209 (not shown) executed by the primary copy; and step S206' (not shown), step S207' (not shown), step S208' (not shown), step S209' (not shown), and step S210' (not shown) performed by the copy to be restored.
In step S206', the to-be-restored copy sends a registration request to the primary master copy. Wherein the registration request includes log version information of the copy to be restored.
Preferably, the registration request includes log index information of the copy to be recovered.
Next, in step S206, the primary replica receives a registration request from the replica to be restored.
Next, in step S207, the master copy sends, to the copy to be restored, log index information in the master copy, where the log version is greater than the maximum log version of the copy to be restored, based on the registration request.
Next, in step S207', the copy to be restored receives the log index information of the log information of which the log version from the primary copy is greater than the locally maximum log version of the copy to be restored.
Then, in step S208', the to-be-recovered copy compares the local log information with the log information of the same log version in the index information in sequence to obtain a log with a different copy version and a minimum log version, and discards all log information whose log version is higher than that of the log.
Wherein the minimum log information includes: and in the local log information of the copy to be recovered, the log information with the smallest log version is the log information with the smallest log version in the part of the log information which has the same log version and is different from the index information.
Next, in step S209', the to-be-restored copy sends the minimum log version to the primary copy to receive the log information from the primary copy, which is transmitted from the minimum log version.
Next, in step S208, the primary replica receives a minimal log version from the to-be-restored replica.
Next, in step S209, the primary copy sends the local log larger than the minimum log version to the copy to be restored, so that the copy to be restored is updated.
Subsequently, in step S210', the copy to be recovered updates the log information local to the copy to be recovered based on the received log information from the primary copy.
And, in step S211', the to-be-recovered copy determines whether a difference between the latest log version and the latest log version of the obtained primary copy in the local log information is less than a predetermined threshold, and when the difference is less than the predetermined threshold, stops updating the local log information.
For example, the predetermined threshold is 2, and when the to-be-recovered copy determines that the difference between the maximum LogIndex in the local log information and the maximum LogIndex in the log information from the primary copy is less than 2, the local log information is stopped from being updated.
According to a preferred embodiment of the present invention, the copy may locally maintain a periodically updated archive point file containing the maximum journal version that the copy has applied, the corresponding journal information of the journal version in the file system, and the journal file offset.
When the copy is added to the system again as the copy to be restored, in step S204', the log version of the copy to be restored from the local archive point of the copy to be restored is used as the start, and the log information of the file system larger than the log version is loaded and applied.
According to the scheme of the invention, when the local of the copy to be restored has no log information, the log information is only directly obtained from the file system for application, and the data restoration efficiency is improved.
In general, according to the distributed system of the present solution, due to the adoption of the structure including the copy of the distributed structure, the read-only copy and the corresponding file system, the number of copies required by the system is small, and no special requirement is imposed on the random read-write performance of the copy, so that the cost of the whole system is reduced, and meanwhile, the performance indexes of the system, such as delay, throughput and the like, can still be the same as the copy synchronization mode based on the distributed consistency algorithm. Moreover, the system can be smoothly upgraded to the scheme of the invention on the basis of the existing synchronization scheme based on the remote file system, thereby having wide implementation and application scenes.
Next, fig. 3 is referred to. Fig. 3 shows a schematic structural diagram of an apparatus for log distribution in a distributed system according to the present invention. The request distribution terminal according to the present scheme includes: means for generating log update information based on a user request (hereinafter referred to as "generating means 101"), means for transmitting the log update information to each copy of the distributed system (hereinafter referred to as "first transmitting means 102"), means for determining that the log becomes a commit state and transmitting a commit instruction to each copy (hereinafter referred to as "second transmitting means 103") when the request issuing side determines that the received copy replies more than a predetermined number; the common copy according to the present solution comprises: means for receiving log update information from a request issuing terminal (hereinafter referred to as "first receiving means 201"), means for updating a local log based on the obtained log update information (hereinafter referred to as "first updating means 202"), means for replying to the request issuing terminal to receive a commit instruction from the request issuing terminal (hereinafter referred to as "first replying means 203"), and means for applying current log information and feeding back a success of commit of the request issuing terminal (hereinafter referred to as "feedback means 204"); the read-only copy according to the present scheme comprises: means for receiving log update information from the request issuing side (hereinafter referred to as "second receiving means 201'"); means for updating log information of the corresponding file system based on the received log update information (hereinafter referred to as "second updating means 202 '"), and means for replying to the request issuing side (hereinafter referred to as "second replying means 203'").
Wherein, the request distributing terminal can be any one of the following:
1) a client;
2) a primary replica.
Preferably, according to the solution of the invention, the request-issuing side is the primary replica.
The generation means 101 of the request distribution side generates log update information based on a user request.
Specifically, the generation apparatus 101 packages the write request of the client into one piece of log information, and increases the log version LogIndex of the log information by 1.
The first sending means 102 then sends the log update information to each copy of the distributed system.
Next, the first receiving device 201 of the ordinary copy receives the log update information from the request issuing side.
Next, the first updating means 202 updates the local log based on the obtained log update information.
Then, the first reply device 203 replies to the request issuing end to receive the commit instruction from the request issuing end.
Meanwhile, the second receiving device 201' in the read-only copy receives the log update information from the request issuing end; and the second updating means 202' updates the log information of the corresponding file system based on the received log update information.
Then, the second replying device 203' replies to the request issuing end.
Next, when the request issuing side determines that the received copy replies exceed the predetermined number, the second sending means 103 of the request issuing side determines that the log becomes the commit state, and sends a commit instruction to each of the normal copies.
Preferably, when the secondary copies in the distributed system are not served, the second sending means may perform a partial operation to send a commit instruction to the primary copy, and send the commit instruction to each secondary copy piggybacked while sending a log update request next time.
Preferably, the predetermined number is half of all copies including the read-only copy.
Then, the feedback device 204 in the common copy applies the current log information and feeds back the success of the submission of the request issuing end.
Preferably, when the copy is a primary copy, according to the solution of the present invention, the primary copy further includes a means for notifying the request issuing end after the primary copy starts to apply the log information, so that the request issuing end replies that the client data update is completed. And the primary copy allows the client to read the applied log information when the application is completed.
More preferably, the master copy may notify the request issuing end after the application log information is completed, so that the request issuing end replies that the client data update is completed.
According to the method of the invention, since the read-only copy is a logical abstraction of the remote file system, and is located on the same physical machine as the primary copy, no new network latency is introduced, so the latency according to the scheme is the same as that of the copy synchronization method based on the distributed consistency algorithm, depending on the copy with the median latency value. Moreover, the file system in the scheme is not a single point of the system any more, so that the overall read-write performance of the system cannot be reduced, and the overall performance of the system cannot be influenced or write failure cannot be caused under the condition that less than most copies fail or slow down.
According to a preferred embodiment of the present invention, when a copy is added to the distributed system according to the present invention as a copy to be restored, the copy to be restored is loaded and the latest log information in the file system is applied; then, the copy to be restored searches submitted log information of which the log version is larger than the latest log information in the log information of the file system, and sequentially searches the obtained log information by application.
Preferably, when the copy to be restored has a local log, the copy to be restored sends a registration request to the master copy. Wherein the registration request includes log version information of the copy to be restored.
Preferably, the registration request includes log index information of the copy to be recovered.
The primary replica then receives a registration request from the copy to be restored.
Then, the master copy sends log index information, which is larger than the maximum log version of the copy to be restored, in the master copy to the copy to be restored based on the registration request.
Then, the copy to be restored receives log index information of the log information of which the log version from the primary copy is larger than the locally maximum log version of the copy to be restored.
Then, the to-be-recovered copy compares the log information with the same log version in the index information in the local log information in sequence to obtain the log with the copy version different and the log version minimum, and discards all the log information of which the log version is higher than the log.
Wherein the minimum log information includes: and in the local log information of the copy to be recovered, the log information with the smallest log version is the log information with the smallest log version in the part of the log information which has the same log version and is different from the index information.
Then, the copy to be recovered sends the minimum log version to the primary copy to receive the log information transmitted from the primary copy from the minimum log version.
Then, the primary replica receives a minimum log version from the to-be-restored replica.
And then, the main copy sends the local log with the version larger than the minimum log version to the copy to be restored so as to update the copy to be restored.
Subsequently, the copy to be recovered updates the log information local to the copy to be recovered based on the received log information from the primary copy.
And the to-be-recovered copy judges whether the difference value between the latest log version and the latest log version of the obtained main copy in the local log information is smaller than a preset threshold value, and when the difference value is smaller than the preset threshold value, the local log information is stopped being updated.
For example, the predetermined threshold is 2, and when the to-be-recovered copy determines that the difference between the maximum LogIndex in the local log information and the maximum LogIndex in the log information from the primary copy is less than 2, the local log information is stopped from being updated.
According to a preferred embodiment of the present invention, the copy may locally maintain a periodically updated archive point file containing the maximum journal version that the copy has applied, the corresponding journal information of the journal version in the file system, and the journal file offset.
When the copy is added into the system again as the copy to be restored, the log version of the copy to be restored from the local archive point of the copy to be restored is used as the start, and the log information of the file system, which is larger than the log version, is loaded and applied.
According to the scheme of the invention, when the local of the copy to be restored has no log information, the log information is only directly obtained from the file system for application, and the data restoration efficiency is improved.
In general, according to the distributed system of the present solution, due to the adoption of the structure including the copy of the distributed structure, the read-only copy and the corresponding file system, the number of copies required by the system is small, and no special requirement is imposed on the random read-write performance of the copy, so that the cost of the whole system is reduced, and meanwhile, the performance indexes of the system, such as delay, throughput and the like, can still be the same as the copy synchronization mode based on the distributed consistency algorithm. Moreover, the system can be smoothly upgraded to the scheme of the invention on the basis of the existing synchronization scheme based on the remote file system, thereby having wide implementation and application scenes.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Claims (16)
1. A method for synchronizing among replicas in a distributed system, wherein the distributed system comprises a plurality of common replicas with log information located locally and at least one read-only replica with log information located in a file system, wherein the plurality of common replicas comprise a master replica and at least one slave replica, and wherein, in the method, each replica in the distributed system performs the following steps:
-comparing the log index information with the respective copies to determine a master copy in the distributed system based on the log index information of the respective copies, wherein the log index information comprises a copy version and a log version;
when the copy is a copy to be restored, the method further comprises the following steps:
-sending a registration request to a master copy of the distributed system, wherein the registration request comprises a log version of the copy to be restored;
wherein the method further comprises the steps of:
-receiving log index information from the log information of the master copy having a log version greater than the local maximum log version;
comparing log information of local log information with the same log version in the index information in sequence to obtain logs with different copy versions and minimum log versions, and discarding all log information of log versions higher than the log; wherein the minimum log information includes: in the local log information of the copy to be recovered, the log information with the smallest log version is the log information of partial log information which has the same log version as the index information and is different from the copy version;
-sending the minimal log version to the primary copy to receive log information from the primary copy transmitted starting from the minimal log version to update the local log information;
-stopping the updating of the local log information if the difference between the current latest log version and the latest log version of the primary copy is smaller than a predetermined threshold.
2. The method of claim 1, wherein when the copy is a normal copy, the method further comprises the steps of:
-receiving log update information from a requesting issuer;
-updating its own local log based on the log update information;
-replying to the request issuing peer to receive a commit instruction from the request issuing peer;
wherein the method further comprises the steps of:
-applying log information local to the common copy based on the commit instruction.
3. The method of claim 2, wherein when the common replica is a master replica, wherein the method further comprises the steps of:
-notifying the requesting publisher that the data update is complete after starting to apply the log information local to the primary replica.
4. The method of claim 1, wherein when the copy is a read-only copy, the method further comprises the steps of:
-receiving log update information from a requesting issuer;
-updating log information in the file system based on the log update information;
-replying to the request issuing end.
5. The method of claim 4, wherein the request issuing end comprises any one of:
-a primary replica;
-a client.
6. The method of claim 5, when the request issuing end is the primary replica, wherein the method further comprises the following steps:
-sending commit instructions to all copies after receiving replies for the majority of copies.
7. The method according to any one of claims 1 to 6, wherein, when the copy is a copy to be restored, wherein the method further comprises the step performed by the copy to be restored of:
-loading and applying the latest log information in the file system;
-looking up submitted log information in the log information of the file system with a log version larger than the latest log information, and applying.
8. The method of claim 1, wherein the method further comprises the following steps performed by the primary replica:
-receiving the registration request from the copy to be restored;
-based on the registration request, sending log index information in the primary copy, of which the log version is greater than the maximum log version of the copy to be restored, to the copy to be restored;
-receiving the minimal log version from the copy to be restored;
-sending the local log larger than the minimum log version to the copy to be restored for updating by the copy to be restored.
9. The method of claim 1, wherein the primary copy is located on the same device as the read-only copy.
10. A replica synchronized in a distributed system, wherein the distributed system comprises a plurality of common replicas with log information located locally and at least one read-only replica with log information located in a file system, wherein the plurality of common replicas comprise a master replica and at least one slave replica, and wherein the replica comprises:
means for comparing log index information to respective copies to determine a master copy in the distributed system based on the log index information of the respective copies, wherein the log index information includes a copy version and a log version;
when the copy is a copy to be restored, the copy to be restored is further configured to:
-sending a registration request to a master copy of the distributed system, wherein the registration request comprises a log version of the copy to be restored;
-receiving log index information from the log information of the master copy having a log version greater than the local maximum log version;
comparing log information of local log information with the same log version in the index information in sequence to obtain logs with different copy versions and minimum log versions, and discarding all log information of log versions higher than the log; wherein the minimum log information includes: in the local log information of the copy to be recovered, the log information with the smallest log version is the log information of partial log information which has the same log version as the index information and is different from the copy version;
-sending the minimal log version to the primary copy to receive log information from the primary copy transmitted starting from the minimal log version to update the local log information;
-stopping the updating of the local log information if the difference between the current latest log version and the latest log version of the primary copy is smaller than a predetermined threshold.
11. The copy of claim 10, wherein when the copy is a normal copy, the normal copy further comprises:
means for receiving log update information from a request issuing side;
means for updating its local log based on the log update information;
means for replying to the request issuing terminal to receive a commit instruction from the request issuing terminal;
means for applying log information local to the common copy based on the commit instruction.
12. The copy of claim 10, wherein when the copy is a read-only copy, the read-only copy comprises:
means for receiving log update information from a request issuing side;
means for updating log information in the file system based on the log update information;
means for replying to the request issuing end.
13. The replica of any one of claims 10 to 12, wherein when the replica is a to-be-recovered replica, the to-be-recovered replica is for:
-loading and applying the latest log information in the file system;
-looking up submitted log information in the log information of the file system with a log version larger than the latest log information, and applying.
14. The replica of claim 10, wherein when the replica is a primary replica, said primary replica is further for:
-receiving the registration request from the copy to be restored;
-based on the registration request, sending log index information in the primary copy, of which the log version is greater than the maximum log version of the copy to be restored, to the copy to be restored;
-receiving the minimal log version from the copy to be restored;
-sending the local log larger than the minimum log version to the copy to be restored for updating by the copy to be restored.
15. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1-9 when executing the program.
16. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710082040.9A CN106951443B (en) | 2017-02-15 | 2017-02-15 | Method, equipment and system for synchronizing copies based on distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710082040.9A CN106951443B (en) | 2017-02-15 | 2017-02-15 | Method, equipment and system for synchronizing copies based on distributed system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106951443A CN106951443A (en) | 2017-07-14 |
| CN106951443B true CN106951443B (en) | 2020-03-13 |
Family
ID=59466455
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710082040.9A Active CN106951443B (en) | 2017-02-15 | 2017-02-15 | Method, equipment and system for synchronizing copies based on distributed system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106951443B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109165230A (en) * | 2018-09-11 | 2019-01-08 | 郑州云海信息技术有限公司 | A kind of method, device and equipment that distributed type assemblies management software data are synchronous |
| CN110351386B (en) * | 2019-07-23 | 2022-09-16 | 华云工业互联网有限公司 | Increment synchronization method and device between different copies |
| CN111597197B (en) * | 2020-06-29 | 2022-02-08 | 腾讯科技（深圳）有限公司 | Data reconciliation method and device between databases, storage medium and electronic equipment |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102024022B (en) * | 2010-11-04 | 2013-08-28 | 曙光信息产业(北京)有限公司 | Method for copying metadata in distributed file system |
| CN102368267A (en) * | 2011-10-25 | 2012-03-07 | 曙光信息产业(北京)有限公司 | Method for keeping consistency of copies in distributed system |
| CN104239310A (en) * | 2013-06-08 | 2014-12-24 | 中国移动通信集团公司 | Distributed database data synchronizing method and device |
| CN104281506B (en) * | 2014-07-10 | 2017-02-15 | 中国科学院计算技术研究所 | Data maintenance method and system for file system |
| CN105550229B (en) * | 2015-12-07 | 2019-05-03 | 北京奇虎科技有限公司 | The method and apparatus of distributed memory system data reparation |
-
2017
- 2017-02-15 CN CN201710082040.9A patent/CN106951443B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN106951443A (en) | 2017-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9170892B2 (en) | Server failure recovery | |
| CN109995813B (en) | Partition expansion method, data storage method and device | |
| CN105814544B (en) | System and method for supporting persistent partition recovery in a distributed data grid | |
| US10831741B2 (en) | Log-shipping data replication with early log record fetching | |
| CN106776130B (en) | Log recovery method, storage device and storage node | |
| US20180150501A1 (en) | Database system, server device, computer program product, and information processing method | |
| US10296254B2 (en) | Method and device for synchronization in the cloud storage system | |
| US8843581B2 (en) | Live object pattern for use with a distributed cache | |
| CN107809326B (en) | Data consistency processing method, device and equipment | |
| CN106951443B (en) | Method, equipment and system for synchronizing copies based on distributed system | |
| CN111597197B (en) | Data reconciliation method and device between databases, storage medium and electronic equipment | |
| CN113268472B (en) | Distributed data storage system and method | |
| CN112052230B (en) | Multi-machine room data synchronization method, computing device and storage medium | |
| CN110121694B (en) | Log management method, server and database system | |
| US10169441B2 (en) | Synchronous data replication in a content management system | |
| CN105323271B (en) | Cloud computing system and processing method and device thereof | |
| CN107547605B (en) | message reading and writing method based on node queue and node equipment | |
| CN109992447B (en) | Data copying method, device and storage medium | |
| CN110196788B (en) | Data reading method, device and system and storage medium | |
| CN113220500A (en) | Recovery method, apparatus and program product based on reverse differential recovery | |
| CN107045426B (en) | Multi-copy reading method and system | |
| CN113297008B (en) | Data processing method and system | |
| CN112947847B (en) | Method, apparatus and computer program product for storing data | |
| CN115698955A (en) | Fault tolerance of transaction images | |
| JP2016009216A (en) | Redundant system and redundancy method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |