CN108205588B - Data synchronization method and device based on master-slave structure - Google Patents
Data synchronization method and device based on master-slave structure Download PDFInfo
- Publication number
- CN108205588B CN108205588B CN201711477361.5A CN201711477361A CN108205588B CN 108205588 B CN108205588 B CN 108205588B CN 201711477361 A CN201711477361 A CN 201711477361A CN 108205588 B CN108205588 B CN 108205588B
- Authority
- CN
- China
- Prior art keywords
- log
- data
- length
- node
- synchronization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Hardware Redundancy (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization method and device based on a master-slave structure. The method comprises the following steps: when the reading abnormality occurs when the main node reads a first log positioned at the main node side, determining the log transition length according to the reading abnormality; the master node sends a log skipping synchronization command to the slave node according to the log shifting length; and the slave node skips the synchronous command according to the log, and fills the blank log content matched with the log transition length after the current synchronous offset of the second log at the slave node side. By adopting the scheme, the master node and the slave node can be ensured to have the same log synchronization offset in the data synchronization process of the master node and the slave node, so that the data loss in the synchronization process is avoided, and particularly, the data loss can be greatly reduced under the condition of frequent master node and slave node switching, so that the data consistency of the master node and the slave node is improved.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data synchronization method and device based on a master-slave structure.
Background
In a database system, a master-slave structure mode is usually adopted to realize data read-write separation, data recovery and the like, so that the reliability of data is improved, the load of a service node is reduced, and the read-write efficiency of the data is improved.
In a database system with a master-slave structure, in order to realize data synchronization of master-slave nodes, a master node can sequentially send log files located at a master node side, and slave nodes update the log files located at the slave node side according to the log files sent by the master node, so that log synchronization is realized, and further, the data synchronization of the master-slave nodes is realized according to the log files.
However, when the master node reads the log file recorded on the master node side and the reading is abnormal, the master node directly skips the abnormal area to continue sending the log file to the slave node, and the slave node directly receives the log file after the abnormal area after receiving the log file before the abnormal area, so that the offset of the log file synchronization on the master node side is inconsistent with the offset of the log file synchronization on the slave node side, which easily causes data loss, and particularly after the master node and the slave node are frequently switched, the data loss frequently occurs, so that the data consistency of the master node and the slave node is reduced.
Disclosure of Invention
In view of the above, the present invention has been made to provide a data synchronization method and apparatus based on a master-slave structure that overcomes or at least partially solves the above-mentioned problems.
According to an aspect of the present invention, there is provided a data synchronization method based on a master-slave structure, including:
when the reading abnormality occurs when the main node reads a first log positioned at the main node side, determining the log transition length according to the reading abnormality;
the master node sends a log skipping synchronization command to the slave node according to the log shifting length;
and the slave node skips the synchronous command according to the log, and fills the blank log content matched with the log transition length after the current synchronous offset of the second log at the slave node side.
According to another aspect of the present invention, there is provided a data synchronization apparatus based on a master-slave structure, including:
the determining module is suitable for determining the log transition length according to the abnormal reading condition when the main node reads a first log positioned at the main node side and the reading is abnormal;
the command sending module is suitable for the master node to send a log skipping synchronization command to the slave node according to the log pushing length;
and the filling module is suitable for the slave node to skip the synchronous command according to the log, and fill the blank log content matched with the log transition length after the current synchronous offset of the second log at the slave node side.
According to yet another aspect of the present invention, there is provided a computing device comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the data synchronization method based on the master-slave structure.
According to still another aspect of the present invention, a computer storage medium is provided, where at least one executable instruction is stored, and the executable instruction causes a processor to perform an operation corresponding to the data synchronization method based on the master-slave structure.
According to the data synchronization method and device based on the master-slave structure, when the master node reads a first log positioned at the master node side and is abnormal in reading, the log transition length is determined according to the abnormal reading condition; the master node sends a log skipping synchronization command to the slave node according to the log shifting length; and the slave node skips the synchronous command according to the log, and fills blank log content matched with the log transition length after the current synchronous offset of a second log at the slave node side. By adopting the scheme, the master node and the slave node can be ensured to have the same log synchronization offset in the data synchronization process of the master node and the slave node, so that the data loss in the synchronization process is avoided, and particularly, the data loss can be greatly reduced under the condition of frequent master node and slave node switching, so that the data consistency of the master node and the slave node is improved.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1 is a flow chart illustrating a data synchronization method based on a master-slave structure according to an embodiment of the present invention;
FIG. 2 is a flow chart of a data synchronization method based on a master-slave structure according to another embodiment of the present invention;
fig. 3 is a block diagram illustrating a structure of a data synchronization apparatus based on a master-slave structure according to an embodiment of the present invention;
FIG. 4 illustrates a schematic structural diagram of a computing device provided in accordance with an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Fig. 1 is a flowchart illustrating a data synchronization method based on a master-slave structure according to an embodiment of the present invention. As shown in fig. 1, the method includes:
step S110, when the main node reads the first log at the main node side and the reading is abnormal, determining the log transition length according to the abnormal reading condition.
The master node records a first log, and the first log records information such as operations executed on data in the master node and/or changes of the data in the master node. In this embodiment, the type of the first log is not limited, for example, the first log may be a binlog (binary log), or a redo log (redo log), or the like.
Specifically, log data damage may occur in the log due to artificial damage or abnormal power failure, and thus a reading abnormality may occur when the master node reads the first log located on the master node side. When the main node reads the first log positioned at the main node side and has reading abnormity, the log transition length can be determined according to the reading abnormity. For example, when the read data content is abnormal, the data length corresponding to the data content can be determined, and the log transition length can be determined according to the data length.
And step S120, the master node sends a log skipping synchronization command to the slave node according to the log transition length.
The master node transmits a log skip synchronization command to the slave node according to the log passage length determined in step S110. In the specific implementation process, when the master node and the slave node synchronize the logs, the master node usually sends a log synchronization command to the slave node in a sync command, and in the step, when the master node reads the first log on the master node side and a read exception occurs, a log skip synchronization command is sent to the slave node, which is different from the sync command sent in the normal case of reading, and the log skip synchronization command contains log transition length information.
And step S130, the slave node skips the synchronous command according to the log, and fills blank log content matched with the log transition length after the current synchronous offset of the second log at the slave node side.
Specifically, the slave node records a second log, and after receiving a log skip synchronization command sent by the master node, the slave node fills blank log content matched with the log transition length after the current synchronization offset of the second log. For example, if it is determined that the log transition length is 8 bytes and the current synchronization offset of the second log is 100 bytes, the 101 th byte to the 108 th byte of the second log are filled with blank log contents, and the current offset of the second log is refreshed to 108 bytes.
Optionally, when there are multiple slave nodes, each slave node skips the synchronization command according to the received log, and fills the blank log content matching the log transition length after the current synchronization offset of the respective second log.
According to the data synchronization method based on the master-slave structure, when a master node reads a first log positioned at the master node side and is abnormal in reading, the log transition length is determined according to the abnormal reading condition; the master node sends a log skipping synchronization command to the slave node according to the log shifting length; and the slave node skips the synchronous command according to the log, and fills blank log content matched with the log transition length after the current synchronous offset of a second log at the slave node side. By adopting the scheme, the master node and the slave node can be ensured to have the same log synchronization offset in the master-slave node log synchronization process, so that the data loss in the synchronization process is avoided, and particularly, the data loss can be greatly reduced under the condition of frequent master-slave node switching, so that the data consistency of the master-slave node is improved.
Fig. 2 is a flowchart illustrating a data synchronization method based on a master-slave structure according to another embodiment of the present invention. As shown in fig. 2, the method includes:
step S210, when the first log read by the master node is abnormal, determining the log transition length according to the abnormal reading condition.
The master node records a first log, and the first log records information such as operations executed on data in the master node and/or changes of the data in the master node. For example, the first log may be a binlog log, or a redo log, or the like.
Specifically, a data bad block occurs in the log due to artificial damage or abnormal power failure, which may cause a reading abnormality when the master node reads the first log located on the master node side. When the main node reads a first log positioned at the main node side and has reading abnormity, determining the log transition length according to the reading abnormity.
The method for determining the log lapse length includes, but is not limited to, one or a combination of at least two of the following embodiments.
In the first embodiment, the data length and the data content of each piece of data are recorded in the first log. And when the read data content is abnormal, determining the log transition length according to the data length corresponding to the read abnormal data content. For example, the data in the first log is stored in the following manner: the data length of each piece of data is stored at the start position of the piece of data, and the data content of the piece of data is stored immediately after the data length. If the data length of the piece of data which can be read normally is 80 bytes when the data5 data is read, and if the reading abnormality occurs when the piece of data content is read, the data length corresponding to the piece of data content is determined to be 80 bytes, so that the log transition length is further determined to be 80 bytes.
In the second embodiment, in order to manage the log data, the first log is divided into a plurality of data blocks, each data block includes at least one piece of data, and the length of each data block may be the same or different. The data length and the data content of each data block are recorded in the first log. Therefore, the log transition length can be determined according to the data length of the data block with the abnormal reading. For example, when a read abnormality occurs in a certain piece of data in the read Block2, the log transition length may be determined according to the data length of the Block 2. Optionally, the data lengths of the data blocks recorded in the first log are all preset lengths, and the log transition length may be determined according to the number of the data blocks with abnormal reading and the preset lengths. For example, the preset length is 40 bytes, when a read abnormality occurs to a certain piece of data in the read Block2, the master node skips the Block2 to read the data in the Block3, when a read abnormality also occurs to the data in the read Block3, the master node skips the Block3 to read the data in the Block4, and when a read abnormality does not occur to the data in the read Block4, the log transition length is determined to be 2 times of the preset length, that is, 80 bytes.
Optionally, in a specific implementation process, the data length and the data content of each data block are recorded in the first log, and the data length and the data content of each piece of data are also recorded in the first log. When the data length of the main node reading a certain piece of data in the first log is not read abnormally, and when the data content reading the piece of data is read abnormally, the data length corresponding to the data content is determined to be the log transition length. When the main node reads the data length of certain data in the first log, the main node can determine the data block corresponding to the data, and determine the log transition length according to the length of the data block.
In step S220, the master node sends a log skip synchronization command to the slave node according to the log passage length.
The master node transmits a log skip synchronization command to the slave node according to the log passage length determined in step S210. In a specific implementation process, during log synchronization of a master node and a slave node, a master node generally sends a log synchronization command to the slave node in a sync command, where the sync command includes a key value and a value of a specific operation and operation data. In this step, a log skip synchronization command is sent to the slave node according to the log transition length. Optionally, the log skipping synchronization command is an SKIP command, and the SKIP command includes log passage length information.
And step S230, the slave node skips the synchronous command according to the log, and fills blank log content matched with the log transition length after the current synchronous offset of the second log at the slave node side.
Specifically, the slave node records a second log, and after receiving a log skip synchronization command sent by the master node, the slave node fills blank log content matched with the log transition length after the current synchronization offset of the second log. For example, if it is determined that the log transition length is 8 bytes and the current synchronization offset of the second log is 100 bytes, the 101 th byte to the 108 th byte of the second log are filled with blank log contents, and the current offset of the second log is refreshed to 108 bytes.
Optionally, when there are multiple slave nodes, each slave node skips the synchronization command according to the received log, and fills the blank log content matching the log transition length after the current synchronization offset of the respective second log.
Optionally, the consistency between the first log and the second log may be checked by a checksum algorithm at preset time intervals or after filling blank log contents matching the log transition length each time. And when the first log is inconsistent with the second log, checking the second log to ensure the consistency of the first log and the second log.
Further, in a specific implementation process, the slave node sequentially performs corresponding operations on data in the slave node according to operations recorded in a second log on the slave node side, so as to implement data synchronization between the master node and the slave node.
Step S240, the master node updates the current synchronization offset of the first log, and performs data synchronization according to the updated current synchronization offset of the first log.
Specifically, after the master node reads the first log and has read exception, the master node updates the current synchronization offset of the first log, determines the log transition length and sends a log skip synchronization command to the slave node. And after the log skipping synchronization command is sent, sending a sync command to the slave node according to the current synchronization offset of the updated first log so as to enable the master node and the slave node to perform subsequent data synchronization.
According to the data synchronization method based on the master-slave structure, when a master node reads a first log positioned at the master node side and is read abnormally, the log transition length is determined according to the data length corresponding to the abnormal data content, or the log transition length is determined according to the data length of an abnormal data block; the master node sends a log skipping synchronization command to the slave node according to the log shifting length; the slave node skips a synchronization command according to the log, and fills blank log content matched with the log transition length after the current synchronization offset of a second log at the slave node side; and the master node updates the current synchronization offset of the first log and synchronizes the log according to the updated current synchronization offset of the first log. By adopting the scheme, the master node and the slave node can be ensured to have the same log synchronization offset in the data synchronization process of the master node and the slave node, so that the data loss in the synchronization process is avoided, and particularly, the data loss can be greatly reduced under the condition of frequent master node and slave node switching, so that the data consistency of the master node and the slave node is improved.
Fig. 3 is a block diagram illustrating a data synchronization apparatus based on a master-slave structure according to an embodiment of the present invention. As shown in fig. 3, the apparatus includes: a determination module 31, a command transmission module 32, and a padding module 33.
The determining module 31 is adapted to determine a log transition length according to a reading abnormal condition when the master node reads a first log located on the master node side and the reading abnormal condition occurs.
And the command sending module 32 is suitable for the master node to send the log skipping synchronous command to the slave node according to the log transition length.
And the filling module 33 is adapted to fill the blank log content matched with the log transition length after the current synchronization offset of the second log at the slave node side according to the log skip synchronization command of the slave node.
Optionally, the data length and the data content of each piece of data are recorded in the first log; the determination module 31 is further adapted to: and determining the log transition length according to the data length corresponding to the read abnormal data content.
Optionally, the data length and the data content of each data block are recorded in the first log; the determination module 31 is further adapted to: and determining the log transition length according to the data length of the abnormal data block.
Further optionally, the data lengths of the data blocks recorded in the first log are all preset lengths; the determination module 31 is further adapted to: and determining the log transition length according to the number of the abnormal data blocks and the preset length.
Optionally, the apparatus further includes an updating module (not shown in the figure), adapted to update, by the master node, the current synchronization offset of the first log according to the log shift length, and perform data synchronization according to the updated current synchronization offset of the first log.
Optionally, the apparatus further comprises: and a data synchronization module (not shown in the figure) adapted to perform data synchronization according to the second log by the slave node.
The specific implementation of the data synchronization apparatus based on the master-slave structure in this embodiment may refer to the corresponding descriptions in the embodiments of the data synchronization method based on the master-slave structure shown in fig. 1 and fig. 2, which are not described herein again.
According to the data synchronization device based on the master-slave structure, when the master node reads a first log positioned at the master node side and is abnormal in reading, the determining module determines the log transition length according to the abnormal reading condition; the command sending module is used for enabling the master node to send a log skipping synchronization command to the slave node according to the log shifting length; and finally, the slave node skips the synchronous command according to the log through a filling module, and fills the blank log content matched with the log transition length after the current synchronous offset of a second log at the slave node side. By adopting the scheme, the master node and the slave node can be ensured to have the same log synchronization offset in the data synchronization process of the master node and the slave node, so that the data loss in the synchronization process is avoided, and particularly, the data loss can be greatly reduced under the condition of frequent master node and slave node switching, so that the data consistency of the master node and the slave node is improved.
According to an embodiment of the present invention, a non-volatile computer storage medium is provided, where at least one executable instruction is stored, and the computer executable instruction can execute the data synchronization method based on the master-slave structure in any of the above method embodiments.
Fig. 4 is a schematic structural diagram of a computing device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the computing device.
As shown in fig. 4, the computing device may include: a processor (processor)402, a Communications Interface 404, a memory 406, and a Communications bus 408.
Wherein:
the processor 402, communication interface 404, and memory 406 communicate with each other via a communication bus 408.
A communication interface 404 for communicating with network elements of other devices, such as clients or other servers.
The processor 402 is configured to execute the program 410, and may specifically execute relevant steps in the above-described data synchronization method embodiment based on the master-slave structure.
In particular, program 410 may include program code comprising computer operating instructions.
The processor 402 may be a central processing unit CPU or an application Specific Integrated circuit asic or one or more Integrated circuits configured to implement embodiments of the present invention. The computing device includes one or more processors, which may be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And a memory 406 for storing a program 410. Memory 406 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 410 may specifically be configured to cause the processor 402 to perform the following operations:
when the reading abnormality occurs when the main node reads a first log positioned at the main node side, determining the log transition length according to the reading abnormality;
the master node sends a log skipping synchronization command to the slave node according to the log shifting length;
and the slave node skips the synchronous command according to the log, and fills the blank log content matched with the log transition length after the current synchronous offset of the second log at the slave node side.
In an optional implementation manner, the data length and the data content of each piece of data are recorded in the first log; the program 410 may specifically be adapted to cause the processor 402 to perform the following operations:
and determining the log transition length according to the data length corresponding to the read abnormal data content.
In an optional implementation manner, the data length and the data content of each data block are recorded in the first log;
the program 410 may specifically be adapted to cause the processor 402 to perform the following operations:
and determining the log transition length according to the data length of the abnormal data block.
In an optional implementation manner, the data lengths of the data blocks recorded in the first log are all preset lengths;
the program 410 may specifically be adapted to cause the processor 402 to perform the following operations:
and determining the log transition length according to the number of the abnormal data blocks and the preset length.
In an alternative embodiment, the program 410 may be specifically configured to cause the processor 402 to perform the following operations:
and the main node updates the current synchronization offset of the first log and performs data synchronization according to the updated current synchronization offset of the first log.
In an alternative embodiment, the program 410 may be specifically configured to cause the processor 402 to perform the following operations:
and the slave node performs data synchronization according to the second log.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.
The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in a data synchronization apparatus based on a master-slave architecture according to embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
Claims (10)
1. A data synchronization method based on a master-slave structure comprises the following steps:
when the reading abnormality occurs when the main node reads a first log positioned at the main node side, determining the log transition length according to the reading abnormality;
the master node sends a log skipping synchronization command to the slave node according to the log shifting length;
the slave node skips a synchronization command according to the log, and fills blank log content matched with the log shift length after the current synchronization offset of a second log at the slave node side;
the main node updates the current synchronous offset of the first log and performs data synchronization according to the updated current synchronous offset of the first log;
and the slave node performs data synchronization according to the second log.
2. The method according to claim 1, wherein the data length and the data content of each piece of data are recorded in the first log;
the determining the log shift length according to the read abnormal condition further comprises:
and determining the log transition length according to the data length corresponding to the read abnormal data content.
3. The method of claim 1, wherein the first log has recorded therein a data length and a data content of each data block;
the determining the log shift length according to the read abnormal condition further comprises:
and determining the log transition length according to the data length of the abnormal data block.
4. The method of claim 3, wherein the data lengths of the data blocks recorded in the first log are all preset lengths;
the determining the log shift length according to the data length of the data block with the abnormal reading further comprises: and determining the log transition length according to the number of the abnormal data blocks and the preset length.
5. A data synchronization apparatus based on a master-slave structure, comprising:
the determining module is suitable for determining the log transition length according to the abnormal reading condition when the main node reads a first log positioned at the main node side and the reading is abnormal;
the command sending module is suitable for the master node to send a log skipping synchronization command to the slave node according to the log pushing length;
the filling module is suitable for the slave node to skip the synchronous command according to the log, and fill the blank log content matched with the log shift length after the current synchronous offset of the second log at the slave node side;
the updating module is suitable for the main node to update the current synchronous offset of the first log according to the log shift length and carry out data synchronization according to the updated current synchronous offset of the first log;
and the data synchronization module is suitable for performing data synchronization on the slave node according to the second log.
6. The apparatus according to claim 5, wherein the first log records a data length and a data content of each piece of data;
the determination module is further adapted to: and determining the log transition length according to the data length corresponding to the read abnormal data content.
7. The apparatus of claim 5, wherein the first log has recorded therein a data length and a data content of each data block;
the determination module is further adapted to: and determining the log transition length according to the data length of the abnormal data block.
8. The apparatus of claim 7, wherein the data lengths of the data blocks recorded in the first log are all preset lengths;
the determination module is further adapted to: and determining the log transition length according to the number of the abnormal data blocks and the preset length.
9. A computing device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation corresponding to the data synchronization method based on the master-slave structure in any one of claims 1-4.
10. A computer storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the master-slave structure based data synchronization method of any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711477361.5A CN108205588B (en) | 2017-12-29 | 2017-12-29 | Data synchronization method and device based on master-slave structure |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711477361.5A CN108205588B (en) | 2017-12-29 | 2017-12-29 | Data synchronization method and device based on master-slave structure |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108205588A CN108205588A (en) | 2018-06-26 |
| CN108205588B true CN108205588B (en) | 2021-04-09 |
Family
ID=62605122
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711477361.5A Active CN108205588B (en) | 2017-12-29 | 2017-12-29 | Data synchronization method and device based on master-slave structure |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108205588B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109388677B (en) * | 2018-08-23 | 2022-10-11 | 顺丰科技有限公司 | Method, device and equipment for synchronizing data among clusters and storage medium thereof |
| CN110688254B (en) * | 2019-09-06 | 2022-06-03 | 北京达佳互联信息技术有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN113515574B (en) * | 2021-05-17 | 2023-05-19 | 聚好看科技股份有限公司 | Data synchronization method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103500229A (en) * | 2013-10-24 | 2014-01-08 | 北京奇虎科技有限公司 | Database synchronization method and database system |
| CN104504062A (en) * | 2014-12-22 | 2015-04-08 | 浙江宇视科技有限公司 | Data synchronization method and device of primary and standby databases |
| CN105912628A (en) * | 2016-04-07 | 2016-08-31 | 北京奇虎科技有限公司 | Synchronization method and device for master database and slave database |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9152686B2 (en) * | 2012-12-21 | 2015-10-06 | Zetta Inc. | Asynchronous replication correctness validation |
-
2017
- 2017-12-29 CN CN201711477361.5A patent/CN108205588B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103500229A (en) * | 2013-10-24 | 2014-01-08 | 北京奇虎科技有限公司 | Database synchronization method and database system |
| CN104504062A (en) * | 2014-12-22 | 2015-04-08 | 浙江宇视科技有限公司 | Data synchronization method and device of primary and standby databases |
| CN105912628A (en) * | 2016-04-07 | 2016-08-31 | 北京奇虎科技有限公司 | Synchronization method and device for master database and slave database |
Non-Patent Citations (2)
| Title |
|---|
| "MySQL数据库主从复制的实现";何小川;《广东通信技术》;20171015(第10期);第54-56页 * |
| 谈书才 等." 基于Replication和LVS的MySQL分布式数据库研究".《中国科技信息》.2015,第61-62页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108205588A (en) | 2018-06-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104881494B (en) | The methods, devices and systems synchronous with Redis server progress data | |
| CN108123851B (en) | Survival detection method and device for master-slave node synchronous link in distributed system | |
| CN108205588B (en) | Data synchronization method and device based on master-slave structure | |
| CN111124755B (en) | Fault recovery method and device for cluster nodes, electronic equipment and storage medium | |
| CN107391634B (en) | Data migration method and device | |
| EP3933573B1 (en) | Zero differential upgrade method, non-volatile storage medium and electronic device | |
| CN108234630A (en) | Method for reading data and device based on distributed consensus protocol realization | |
| CN110784348A (en) | Firmware upgrading method and device, electronic equipment and storage medium | |
| CN107391033B (en) | Data migration method and device, computing equipment and computer storage medium | |
| CN108228789B (en) | Synchronous abnormity recovery method and device triggered by slave node | |
| CN108228812B (en) | Self-adaptive main node switching method and device | |
| CN111049928A (en) | Data synchronization method, system, electronic device and computer readable storage medium | |
| CN112052230B (en) | Multi-machine room data synchronization method, computing device and storage medium | |
| CN106817387B (en) | Data synchronization method, device and system | |
| CN107368563B (en) | Database data deleting method and device, electronic equipment and storage medium | |
| CN104580425A (en) | Client data synchronization method and system | |
| CN108200157B (en) | Log synchronization method and device for triggering rollback by master node | |
| CN112015595B (en) | Master-slave database switching method, computing device and storage medium | |
| CN107741965B (en) | Database synchronous processing method and device, computing equipment and computer storage medium | |
| CN112636987B (en) | Cross-chain gateway determination method and system for block chain and terminal equipment | |
| CN113468143A (en) | Data migration method, system, computing device and storage medium | |
| CN105550230A (en) | Method and device for detecting failure of node of distributed storage system | |
| CN107864209A (en) | The method, apparatus and server of data write-in | |
| CN112035418A (en) | Multi-computer room synchronization method, computing device and computer storage medium | |
| CN112000850A (en) | Method, device, system and equipment for data processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |