CN112417045A - Data synchronization method and device for master node and slave node of database - Google Patents
Data synchronization method and device for master node and slave node of database Download PDFInfo
- Publication number
- CN112417045A CN112417045A CN202011314944.8A CN202011314944A CN112417045A CN 112417045 A CN112417045 A CN 112417045A CN 202011314944 A CN202011314944 A CN 202011314944A CN 112417045 A CN112417045 A CN 112417045A
- Authority
- CN
- China
- Prior art keywords
- database
- node
- slave
- master
- output log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 230000001360 synchronised effect Effects 0.000 claims abstract description 46
- 230000004044 response Effects 0.000 claims abstract description 24
- 230000006870 function Effects 0.000 claims description 11
- 230000002123 temporal effect Effects 0.000 claims 2
- 230000008569 process Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
Abstract
The invention discloses a data synchronization method and a device for a database master node and a database slave node, wherein the method comprises the following steps: starting an output log function of a time sequence type database on a main node and determining the format of the output log; configuring and starting a master-slave synchronizer of the time sequence database to continuously monitor an output log of a master node; analyzing all editing operations aiming at the time sequence type database based on the time sequence from the output log; in response to the slave node going offline, caching the editing operation as an operation to be synchronized in a lightweight database of the master node; the editing operation and the cached operation to be synchronized are synchronized to the slave node in response to the slave node being online. The invention can normally execute the master-slave data synchronization and the master-slave switching in the off-line way of the master-slave database, thereby ensuring the data integrity.
Description
Technical Field
The present invention relates to the field of storage, and in particular, to a method and an apparatus for synchronizing data of a master node and a slave node of a database.
Background
There are a number of disadvantages associated with using conventional relational database storage time series. For example, when the data volume is large, the query speed is slow, the user is not friendly, the storage is not optimized, and the occupied disk space is large. The inflixdb is used as open source time sequence data, focuses on inquiring and storing the time sequence data with high performance, but does not have a master-slave synchronization function, namely, does not support the cluster. If the master-slave data synchronization needs to be realized, data is written into the master-slave nodes at the same time basically when the data is written, and the method can solve the problem of master-slave synchronization, but is not an optimal method because the simultaneous storage of the data into the master-slave database occupies a large amount of bandwidth, and once the master/slave database is offline, the data is lost, which is very serious.
For the problem that master-slave synchronization in the prior art may cause data loss, no effective solution is available at present.
Disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide a method and an apparatus for data synchronization of a master node and a slave node of a database, which can normally perform master-slave data synchronization and master-slave switching in an offline manner on a master database and a slave database, so as to ensure data integrity.
Based on the above object, a first aspect of the embodiments of the present invention provides a data synchronization method for a master node and a slave node of a database, including the following steps:
starting an output log function of a time sequence type database on a main node and determining the format of the output log;
configuring and starting a master-slave synchronizer of the time sequence database to continuously monitor an output log of a master node;
analyzing all editing operations aiming at the time sequence type database based on the time sequence from the output log;
in response to the slave node going offline, caching the editing operation as an operation to be synchronized in a lightweight database of the master node;
the editing operation and the cached operation to be synchronized are synchronized to the slave node in response to the slave node being online.
In some embodiments, the output log includes a plurality of operations, each operation having a sequence number determined based on timing; analyzing all editing operations for the time-series database based on the time-series sequence from the output log comprises the following steps: the chronological order of all editing operations is determined based on the sequence number of each operation.
In some embodiments, synchronizing the editing operation and the cached operation to be synchronized to the slave node comprises: and synchronizing each operation in the editing operation and the operation to be synchronized to the slave node in turn based on the sequence number sequence of each operation.
In some embodiments, the method further comprises: in response to a master-slave switch occurring between the master node and the slave node, the output logs of the slave node are adaptively parsed and the editing operation is synchronized to the master node.
In some embodiments, the editing operations include adding data, deleting data, and modifying data.
In some embodiments, the method further comprises: after synchronizing the operation to be synchronized to the slave node, the operation to be synchronized in the lightweight database is also deleted.
In some embodiments, the time-ordered database is influxdb and the lightweight database is sqlite.
A second aspect of the embodiments of the present invention provides a data synchronization apparatus for a master node and a slave node of a database, including:
a processor; and
a memory storing program code executable by the processor, the program code when executed performing the steps of:
starting an output log function of a time sequence type database on a main node and determining the format of the output log;
configuring and starting a master-slave synchronizer of the time sequence database to continuously monitor an output log of a master node;
analyzing all editing operations aiming at the time sequence type database based on the time sequence from the output log;
in response to the slave node going offline, caching the editing operation as an operation to be synchronized in a lightweight database of the master node;
the editing operation and the cached operation to be synchronized are synchronized to the slave node in response to the slave node being online.
In some embodiments, the output log includes a plurality of operations, each operation having a sequence number determined based on timing;
analyzing all editing operations for the time-series database based on the time-series sequence from the output log comprises the following steps: determining a time sequence order of all editing operations based on the sequence number of each operation;
synchronizing the editing operations and the cached operations to be synchronized to the slave node comprises: and synchronizing each operation in the editing operation and the operation to be synchronized to the slave node in turn based on the sequence number sequence of each operation.
In some embodiments, the steps further comprise: in response to a master-slave switch occurring between the master node and the slave node, the output logs of the slave node are adaptively parsed and the editing operation is synchronized to the master node.
The invention has the following beneficial technical effects: according to the data synchronization method and device for the master node and the slave node of the database, the output log function of the time sequence type database is started on the master node, and the format of the output log is determined; configuring and starting a master-slave synchronizer of the time sequence database to continuously monitor an output log of a master node; analyzing all editing operations aiming at the time sequence type database based on the time sequence from the output log; in response to the slave node going offline, caching the editing operation as an operation to be synchronized in a lightweight database of the master node; the technical scheme of synchronizing the editing operation and the cached operation to be synchronized to the slave node in response to the online slave node can normally execute master-slave data synchronization and master-slave switching in an offline manner in a master-slave database, and ensure the integrity of data.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a data synchronization method for a master node and a slave node of a database according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
Based on the above objectives, a first aspect of the embodiments of the present invention provides an embodiment of a data synchronization method for a database master-slave node, where master-slave data synchronization and master-slave switching are normally performed offline in a master-slave database. Fig. 1 is a schematic flow chart illustrating a data synchronization method for a database master node and a database slave node according to the present invention.
The data synchronization method of the database master-slave node, as shown in fig. 1, includes the following steps:
step S101, starting an output log function of a time sequence type database on a main node and determining the format of an output log;
step S103, configuring and starting a master-slave synchronizer of the time-sequence database to continuously monitor the output log of the master node;
step S105, analyzing all editing operations aiming at the time sequence type database based on the time sequence from the output log;
step S107, responding to the offline of the slave node, caching the editing operation as the operation to be synchronized into a lightweight database of the master node;
step S109, responding to the online of the slave node, and synchronizing the editing operation and the cached operation to be synchronized to the slave node.
The invention monitors the operation log in the master database in real time, and synchronizes the operations of adding, deleting and changing to the slave database through the tool. The tool guarantees uniqueness of data through log time sequence serial numbers based on binary logs generated by analyzing infiluxdb, supports the backup node to store the data into sqlite after being offline, and synchronizes the stored data to the backup node after the backup node is online.
Specifically, the method uses the inflixdb master-slave synchronization tool based on the overview, does not need to modify the inflixdb source code, and has good compatibility support for later-stage inflixdb upgrading. Analyzing the inflixdb binary log, extracting key information, and synchronizing the key information into the inflixdb; when the slave infixtb is offline, the storage of the key information modified by the master infixtb into the sqlite is supported, and after the slave infixtb database is online, the offline stored data is stored into the slave infixtb database.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a Random Access Memory (RAM), or the like. Embodiments of the computer program may achieve the same or similar effects as any of the preceding method embodiments to which it corresponds.
In some embodiments, the output log includes a plurality of operations, each operation having a sequence number determined based on timing; analyzing all editing operations for the time-series database based on the time-series sequence from the output log comprises the following steps: the chronological order of all editing operations is determined based on the sequence number of each operation.
In some embodiments, synchronizing the editing operation and the cached operation to be synchronized to the slave node comprises: and synchronizing each operation in the editing operation and the operation to be synchronized to the slave node in turn based on the sequence number sequence of each operation.
In some embodiments, the method further comprises: in response to a master-slave switch occurring between the master node and the slave node, the output logs of the slave node are adaptively parsed and the editing operation is synchronized to the master node.
In some embodiments, the editing operations include adding data, deleting data, and modifying data.
In some embodiments, the method further comprises: after synchronizing the operation to be synchronized to the slave node, the operation to be synchronized in the lightweight database is also deleted.
In some embodiments, the time-ordered database is influxdb and the lightweight database is sqlite.
The following further illustrates embodiments of the invention in terms of specific examples:
(1) modifying the primary node infiluxdb configuration file to support log file output and setting the format of an output log;
(2) configuring an infiluxdb master-slave synchronization tool configuration file;
(3) starting an influxdb master-slave synchronization tool, and continuously monitoring an operation log of a master influxdb database;
(4) analyzing the operation log, adding, deleting and changing operations, and synchronizing the operations to the influxdb slave database;
(5) if the slave influxdb database is offline, adding, deleting and changing the master influxdb database into the sqlite database;
(6) after detecting that the slave infilxdb database is online, storing the data in the sqlite into the slave infixdb database, and deleting the data stored in the sqlite;
(7) if the master-slave structure is switched, the tool also supports dynamic switching, so that master-slave data consistency is ensured.
It can be seen from the foregoing embodiments that, in the data synchronization method for a master node and a slave node of a database provided in the embodiments of the present invention, an output log function of a time-series database is enabled on a master node, and a format of the output log is determined; configuring and starting a master-slave synchronizer of the time sequence database to continuously monitor an output log of a master node; analyzing all editing operations aiming at the time sequence type database based on the time sequence from the output log; in response to the slave node going offline, caching the editing operation as an operation to be synchronized in a lightweight database of the master node; the technical scheme of synchronizing the editing operation and the cached operation to be synchronized to the slave node in response to the online slave node can normally execute master-slave data synchronization and master-slave switching in an offline manner in a master-slave database, and ensure the integrity of data.
It should be particularly noted that, the steps in the embodiments of the data synchronization method for the database master-slave nodes may be mutually intersected, replaced, added, and deleted, so that these data synchronization methods for the database master-slave nodes that are reasonably arranged and combined to transform shall also belong to the scope of the present invention, and shall not limit the scope of the present invention to the described embodiments.
In view of the foregoing, a second aspect of the embodiments of the present invention provides an embodiment of a data synchronization apparatus for a database master-slave node, where the master-slave data synchronization and the master-slave switching are normally performed offline in a master-slave database. The data synchronization device of the database master-slave node comprises:
a processor; and
a memory storing program code executable by the processor, the program code when executed performing the steps of:
starting an output log function of a time sequence type database on a main node and determining the format of the output log;
configuring and starting a master-slave synchronizer of the time sequence database to continuously monitor an output log of a master node;
analyzing all editing operations aiming at the time sequence type database based on the time sequence from the output log;
in response to the slave node going offline, caching the editing operation as an operation to be synchronized in a lightweight database of the master node;
the editing operation and the cached operation to be synchronized are synchronized to the slave node in response to the slave node being online.
In some embodiments, the output log includes a plurality of operations, each operation having a sequence number determined based on timing;
analyzing all editing operations for the time-series database based on the time-series sequence from the output log comprises the following steps: determining a time sequence order of all editing operations based on the sequence number of each operation;
synchronizing the editing operations and the cached operations to be synchronized to the slave node comprises: and synchronizing each operation in the editing operation and the operation to be synchronized to the slave node in turn based on the sequence number sequence of each operation.
In some embodiments, the steps further comprise: in response to a master-slave switch occurring between the master node and the slave node, the output logs of the slave node are adaptively parsed and the editing operation is synchronized to the master node.
It can be seen from the foregoing embodiments that, in the data synchronization apparatus for a master node and a slave node of a database provided in the embodiments of the present invention, an output log function of a time-series database is enabled on a master node, and a format of the output log is determined; configuring and starting a master-slave synchronizer of the time sequence database to continuously monitor an output log of a master node; analyzing all editing operations aiming at the time sequence type database based on the time sequence from the output log; in response to the slave node going offline, caching the editing operation as an operation to be synchronized in a lightweight database of the master node; the technical scheme of synchronizing the editing operation and the cached operation to be synchronized to the slave node in response to the online slave node can normally execute master-slave data synchronization and master-slave switching in an offline manner in a master-slave database, and ensure the integrity of data.
It should be particularly noted that, the above-mentioned embodiment of the data synchronization apparatus of the database master-slave node specifically describes the working process of each module by using the embodiment of the data synchronization method of the database master-slave node, and those skilled in the art can easily think that these modules are applied to other embodiments of the data synchronization method of the database master-slave node. Of course, since the steps in the data synchronization method embodiments of the database master-slave nodes may be intersected, replaced, added, or deleted, these data synchronization apparatuses that are reasonably arranged, combined, and transformed in the database master-slave nodes also belong to the scope of the present invention, and the scope of the present invention should not be limited to the embodiments.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of an embodiment of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A data synchronization method of a database master node and a database slave node is characterized by comprising the following steps:
starting an output log function of a time sequence type database on a main node and determining the format of the output log;
configuring and enabling a master-slave synchronizer of a time-ordered database to continuously monitor the output log of the master node;
analyzing all editing operations aiming at the time sequence type database based on time sequence from the output log;
caching the editing operation as an operation to be synchronized in a lightweight database of the master node in response to a slave node going offline;
synchronizing the editing operation and the cached operation to be synchronized to the slave node in response to the slave node being online.
2. The method of claim 1, wherein the output log includes a plurality of operations, each operation having a sequence number determined based on timing; analyzing all editing operations for the time-sequence database based on the time-sequence from the output log comprises: determining a temporal order of all of the editing operations based on the sequence number of each operation.
3. The method of claim 2, wherein synchronizing the editing operation and the cached operation to be synchronized to the slave node comprises: sequentially synchronizing each of the editing operations and the operations to be synchronized to the slave node based on the sequence number order of each operation.
4. The method of claim 1, further comprising: adaptively parsing the output log of the slave node and synchronizing the editing operation to the master node in response to a master-slave switch occurring between the master node and the slave node.
5. The method of claim 1, wherein the editing operations include adding data, deleting data, and modifying data.
6. The method of claim 1, further comprising: after synchronizing the operation to be synchronized to the slave node, the operation to be synchronized in the lightweight database is also deleted.
7. The method of claim 1, wherein the time-ordered database is infiluxdb and the lightweight database is sqlite.
8. A data synchronization device for a master node and a slave node of a database is characterized by comprising:
a processor; and
a memory storing program code executable by the processor, the program code when executed performing the steps of:
starting an output log function of a time sequence type database on a main node and determining the format of the output log;
configuring and enabling a master-slave synchronizer of a time-ordered database to continuously monitor the output log of the master node;
analyzing all editing operations aiming at the time sequence type database based on time sequence from the output log;
caching the editing operation as an operation to be synchronized in a lightweight database of the master node in response to a slave node going offline;
synchronizing the editing operation and the cached operation to be synchronized to the slave node in response to the slave node being online.
9. The apparatus of claim 8, wherein the output log comprises a plurality of operations, each operation having a sequence number determined based on timing;
analyzing all editing operations for the time-sequence database based on the time-sequence from the output log comprises: determining a temporal order of all of the editing operations based on the sequence number of each operation;
synchronizing the editing operation and the cached operation to be synchronized to the slave node comprises: sequentially synchronizing each of the editing operations and the operations to be synchronized to the slave node based on the sequence number order of each operation.
10. The apparatus of claim 8, wherein the steps further comprise: adaptively parsing the output log of the slave node and synchronizing the editing operation to the master node in response to a master-slave switch occurring between the master node and the slave node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011314944.8A CN112417045A (en) | 2020-11-20 | 2020-11-20 | Data synchronization method and device for master node and slave node of database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011314944.8A CN112417045A (en) | 2020-11-20 | 2020-11-20 | Data synchronization method and device for master node and slave node of database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112417045A true CN112417045A (en) | 2021-02-26 |
Family
ID=74776930
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011314944.8A Withdrawn CN112417045A (en) | 2020-11-20 | 2020-11-20 | Data synchronization method and device for master node and slave node of database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112417045A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113515574A (en) * | 2021-05-17 | 2021-10-19 | 聚好看科技股份有限公司 | Data synchronization method and device |
-
2020
- 2020-11-20 CN CN202011314944.8A patent/CN112417045A/en not_active Withdrawn
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113515574A (en) * | 2021-05-17 | 2021-10-19 | 聚好看科技股份有限公司 | Data synchronization method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105447156A (en) | Resource description framework distributed engine and incremental updating method | |
| CN110399356B (en) | Online data migration method and device, computing equipment and storage medium | |
| US20170031948A1 (en) | File synchronization method, server, and terminal | |
| CN108121782A (en) | Distribution method, database middleware system and the electronic equipment of inquiry request | |
| CN108829720B (en) | Data processing method and device | |
| US11640261B2 (en) | Log processing method to avoid log collision, and related device and system | |
| CN112181306A (en) | Method for generating distributed ID (identity) in number segment mode | |
| CN112163038A (en) | Cross-cluster data synchronization method, device, equipment and storage medium | |
| CN113946583A (en) | Primary key generation method and device | |
| CN112417045A (en) | Data synchronization method and device for master node and slave node of database | |
| CN113064903A (en) | Global ID generation method, device, equipment and storage medium | |
| CN115146002A (en) | Cross-data-center data synchronization method and device | |
| CN106603610B (en) | A kind of data acquisition and synchronous method, device | |
| WO2021082925A1 (en) | Transaction processing method and apparatus | |
| CN111552701B (en) | Method for determining data consistency in distributed cluster and distributed data system | |
| CN112000850A (en) | Method, device, system and equipment for data processing | |
| CN113360319B (en) | Data backup method and device | |
| CN115189931A (en) | Distributed key management method, device, equipment and storage medium | |
| CN114064717A (en) | Data processing method, device, equipment and storage medium | |
| CN115481158B (en) | Automatic loading and converting method for data distributed cache | |
| CN113934792A (en) | Processing method and device of distributed database, network equipment and storage medium | |
| CN116662290B (en) | Read optimization method and device for stateful server non-perceptual function | |
| CN117453730B (en) | Data query method, device, equipment and storage medium | |
| US20210216379A1 (en) | Method, device, and computer program product for managing a task in an application node | |
| CN116257583A (en) | Database data synchronization method, electronic device, storage medium, and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210226 |