CN106354830B - Method and device for data synchronization between database cluster nodes - Google Patents
Method and device for data synchronization between database cluster nodes Download PDFInfo
- Publication number
- CN106354830B CN106354830B CN201610777767.4A CN201610777767A CN106354830B CN 106354830 B CN106354830 B CN 106354830B CN 201610777767 A CN201610777767 A CN 201610777767A CN 106354830 B CN106354830 B CN 106354830B
- Authority
- CN
- China
- Prior art keywords
- lock
- data synchronization
- information
- deleting
- metadata server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and a device for data synchronization among database cluster nodes, wherein the method comprises the following steps: acquiring information of a table to be synchronized from a metadata server; checking the lock queue to determine whether a lock to delete the table is included; and if the lock queue does not comprise the deletion information of the table, performing data synchronization. The table can be deleted before the table is modified, so that the data exchange synchronization is reduced, and the waste of time and system resources is avoided. The database synchronization is efficiently completed.
Description
Technical Field
The invention belongs to the field of database clusters, and particularly relates to a method and a device for data synchronization among database cluster nodes.
Background
When the database cluster is used, a user needs to frequently perform operations of table building, calculation and table deletion. Before deleting a table, if a table on a certain node of the database cluster is damaged, the table on the database node needs to be synchronized.
The following methods can be currently used for synchronization: and locking the synchronized table, and deleting the table after the synchronization is finished. If the user has already done the calculation during the synchronization and needs to delete the table, the user needs to wait until the deleted table is synchronized before deleting the table, which wastes a lot of time and system resources.
Disclosure of Invention
The embodiment of the invention provides a method and a device for data synchronization among database cluster nodes, which are used for solving the existing technical problems.
In one aspect, an embodiment of the present invention provides a method for synchronizing data among database cluster nodes, including:
acquiring information of a table to be synchronized from a metadata server;
checking the lock queue to determine whether a lock to delete the table is included;
and if the lock queue does not comprise the deletion information of the table, performing data synchronization.
Further, the method further comprises:
and if the lock queue does not comprise the deletion information of the table, abandoning the data synchronization.
Further, after the data synchronization, the method further includes:
confirming whether the data synchronization is successful, and if the data synchronization is unsuccessful, writing the name of the table and the node corresponding to the failure into a metadata server to generate damage information;
adding a delete lock to the lock queue;
deleting the delete lock after the table is successfully deleted;
further, after the data synchronization, the method further includes:
and deleting the damage information of the table in the metadata server.
On the other hand, an embodiment of the present invention further provides a device for data synchronization between database cluster nodes, including:
an information acquisition unit for acquiring information of a table to be synchronized from a metadata server;
a lock checking unit for checking a lock queue to confirm whether a lock for deleting the table is included;
and the synchronization unit is used for carrying out data synchronization if the deletion information of the table is not included in the lock queue.
Further, the apparatus further comprises:
and the abandoning unit is used for abandoning the data synchronization if the lock queue comprises the deletion information of the table.
Further, the apparatus further comprises:
a writing unit, configured to determine whether data synchronization is successful, and if the data synchronization is unsuccessful, write the name of the table and a node corresponding to the failure into the metadata server, so as to generate damage information;
an adding unit, configured to add a delete lock to a lock queue;
and the lock deleting unit is used for deleting the deleting lock after the table is successfully deleted.
Further, the apparatus further comprises:
and the damage information deleting unit is used for deleting the damage information of the table in the metadata server.
The embodiment of the invention acquires the information of the table needing to be synchronized from the metadata server; checking a lock queue to determine whether the lock of the table is deleted; and if the lock queue does not comprise the deletion information of the table, performing data synchronization. The table can be deleted before the table is modified, so that the data exchange synchronization is reduced, and the waste of time and system resources is avoided. The database synchronization is efficiently completed.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.
Fig. 1 is a schematic flowchart of a method for synchronizing data between database cluster nodes according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of a method for synchronizing data between database cluster nodes according to a second embodiment of the present invention;
fig. 3 is a schematic flowchart of a method for synchronizing data between database cluster nodes according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data synchronization apparatus between database-partitioned cluster nodes according to a fourth embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
Fig. 1 is a schematic flowchart of a method for synchronizing data between database cluster nodes according to an embodiment of the present invention. The method can be executed by a data synchronization device between the database cluster nodes, can be realized by a software/hardware mode, and can be integrated in the database cluster nodes.
Referring to fig. 1, the distributed database data backup segment deployment method includes:
s110, obtaining information of the table to be synchronized from the metadata server.
Data in any file system is divided into data and metadata. Data refers to actual data in a general file, and metadata refers to system data used to describe characteristics of a file, such as access rights, file owner, and distribution information of file data blocks, etc. A node in the system is specially managed by the metadata, and all the metadata is stored on a storage device of the node. All clients request the metadata from the metadata manager before requesting the file. Information of the tables that need to be synchronized can be obtained from the metadata server. Illustratively, a storage location for the synchronization table is included.
S120, checking the lock queue, and confirming whether the lock of the table is deleted or not.
The database cluster comprises a lock queue manager which is used for processing SQL locking requests of the database, queuing and storing the requests and providing a query interface for a synchronization tool. When the database executes data change operation, if some database nodes fail and the corresponding data copies on the failed nodes are successfully executed on other nodes, the failure information (table name and database node name) is recorded in the metadata server. The database performs a delete table operation while adding the delete lock to the lock queue manager. By querying the lock queue manager, it may be determined whether a lock to delete the table is included.
S130, if the lock queue does not include the deletion information of the table, data synchronization is carried out.
If the deletion information of the table is not included in the lock queue, the table is not damaged, and a synchronization tool can be used for carrying out data synchronization on the table.
The embodiment of the invention acquires the information of the table needing to be synchronized from the metadata server; checking a lock queue to determine whether the lock of the table is deleted; and if the lock queue does not comprise the deletion information of the table, performing data synchronization. The table can be deleted before the table is modified, so that the data exchange synchronization is reduced, and the waste of time and system resources is avoided. The database synchronization is efficiently completed.
In a preferred implementation of this embodiment, the method further includes: and if the lock queue comprises the deletion information of the table, abandoning the data synchronization. And if the lock queue comprises the deletion information of the table, the table is damaged, and a synchronization tool is not required to be adopted to perform data synchronization on the table. Or terminate synchronization when synchronizing a batch of data, avoiding wasting time and system resources.
Example two
Fig. 2 is a schematic flow chart of a method for synchronizing data between database cluster nodes according to a second embodiment of the present invention, where the second embodiment of the present invention is based on the foregoing embodiment, and further, the following steps are added after data synchronization is performed: confirming whether the data synchronization is successful, and if the data synchronization is unsuccessful, writing the name of the table and the node corresponding to the failure into a metadata server to generate damage information; adding a delete lock to the lock queue; and deleting the deletion lock after the table is deleted successfully.
Referring to fig. 2, the method for data synchronization between database cluster nodes includes:
s210, obtaining the information of the table needing to be synchronized from the metadata server.
S220, checking the lock queue, and confirming whether the lock of the table is deleted or not.
S230, if the lock queue does not include the deletion information of the table, data synchronization is carried out.
S240, confirming whether the data synchronization is successful, and if the data synchronization is unsuccessful, writing the name of the table and the node corresponding to the failure into the metadata server to generate damage information.
The database is used for receiving the SQL request, and if the table on a certain node is damaged in the request execution process, the database records the damaged table name and the database node name into the metadata server.
S250, adding the deletion lock to the lock queue.
Since the table has been corrupted, the table may be deleted and the database performs the delete table operation while the delete lock is added to the lock queue manager.
S260, deleting the deletion lock after the table is deleted successfully.
If the table is deleted successfully, the metadata server is informed to delete the corresponding table damage information in the metadata, and the lock queue server is informed to delete the deletion lock of the table in the lock queue.
The present embodiment adds the following steps after performing data synchronization: confirming whether the data synchronization is successful, and if the data synchronization is unsuccessful, writing the name of the table and the node corresponding to the failure into a metadata server to generate damage information; adding a delete lock to the lock queue; and deleting the deletion lock after the table is deleted successfully. The list with unsuccessful synchronization can be deleted to avoid the time and system resource waste caused by synchronization again in the later period.
EXAMPLE III
Fig. 3 is a schematic flow chart of a method for synchronizing data between database cluster nodes according to a third embodiment of the present invention, where the third embodiment of the present invention is based on the foregoing embodiment, and further, the following steps are added after data synchronization is performed: and deleting the damage information of the table in the metadata server.
Referring to fig. 3, the method for data synchronization between database cluster nodes includes:
s310, obtain the information of the table to be synchronized from the metadata server.
S320, checking the lock queue, and confirming whether the lock of the table is deleted or not.
S330, if the lock queue does not include the deletion information of the table, data synchronization is carried out.
S340, deleting the damage information of the table in the metadata server.
During synchronous operation, after a batch of data is synchronized, whether synchronization is completed is judged: if the synchronization is complete, the table corruption information on the metadata server is deleted. Since the data of the table is complete and correct due to the synchronization, the corruption information of the table in the metadata server can be deleted.
The present embodiment adds the following steps after performing data synchronization: and deleting the damage information of the table in the metadata server. After the table synchronization is successful, the metadata corresponding to the table can be modified in time, and the table which is successfully synchronized is prevented from being deleted by mistake.
Example four
Fig. 4 is a schematic structural diagram of an apparatus for synchronizing data between database cluster nodes according to a fourth embodiment of the present invention, as shown in fig. 4, the apparatus includes:
an information acquisition unit 410 for acquiring information of a table that needs synchronization from a metadata server;
a lock checking unit 420 for checking the lock queue to see if a lock to delete the table is included;
a synchronizing unit 430, configured to perform data synchronization if the lock queue does not include the deletion information of the table.
The embodiment of the invention acquires the information of the table needing to be synchronized from the metadata server; checking a lock queue to determine whether the lock of the table is deleted; and if the lock queue does not comprise the deletion information of the table, performing data synchronization. The table can be deleted before the table is modified, so that the data exchange synchronization is reduced, and the waste of time and system resources is avoided. The database synchronization is efficiently completed.
Further, the apparatus further comprises:
and the abandoning unit is used for abandoning the data synchronization if the lock queue comprises the deletion information of the table.
Further, the apparatus further comprises:
a writing unit, configured to determine whether data synchronization is successful, and if the data synchronization is unsuccessful, write the name of the table and a node corresponding to the failure into the metadata server, so as to generate damage information;
an adding unit, configured to add a delete lock to a lock queue;
and the lock deleting unit is used for deleting the deleting lock after the table is successfully deleted.
Still further, the apparatus further comprises:
and the damage information deleting unit is used for deleting the damage information of the table in the metadata server.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. A method for data synchronization between database cluster nodes is characterized by comprising the following steps:
acquiring information of a table to be synchronized from a metadata server;
checking the lock queue to determine whether a lock to delete the table is included; when a database executes data change operation, if partial database nodes fail and copies of corresponding data on the failed nodes are successfully executed on other nodes, recording failure information to a metadata server, and adding a lock for deleting a table to a lock queue manager when the database executes table deletion operation;
and if the lock queue does not comprise the deletion information of the table, performing data synchronization.
2. The method of claim 1, further comprising:
and if the lock queue comprises the deletion information of the table, abandoning the data synchronization.
3. The method of claim 1, further comprising, after performing data synchronization:
confirming whether the data synchronization is successful, and if the data synchronization is unsuccessful, writing the name of the table and the node corresponding to the failure into a metadata server to generate damage information;
adding a delete lock to the lock queue;
and deleting the deletion lock after the table is deleted successfully.
4. The method of claim 1, further comprising, after performing data synchronization:
and deleting the damage information of the table in the metadata server.
5. An apparatus for synchronizing data between database cluster nodes, comprising:
an information acquisition unit for acquiring information of a table to be synchronized from a metadata server;
a lock checking unit for confirming whether a lock for deleting the table is included; when a database executes data change operation, if partial database nodes fail and copies of corresponding data on the failed nodes are successfully executed on other nodes, recording failure information to a metadata server, and adding a lock for deleting a table to a lock queue manager when the database executes table deletion operation;
and the synchronization unit is used for carrying out data synchronization if the deletion information of the table is not included in the lock queue.
6. The apparatus of claim 5, further comprising:
and the abandoning unit is used for abandoning the data synchronization if the lock queue comprises the deletion information of the table.
7. The apparatus of claim 5, further comprising:
a writing unit, configured to determine whether data synchronization is successful, and if the data synchronization is unsuccessful, write the name of the table and a node corresponding to the failure into the metadata server, so as to generate damage information;
an adding unit, configured to add a delete lock to a lock queue;
and the lock deleting unit is used for deleting the deleting lock after the table is successfully deleted.
8. The apparatus of claim 5, further comprising:
and the damage information deleting unit is used for deleting the damage information of the table in the metadata server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610777767.4A CN106354830B (en) | 2016-08-31 | 2016-08-31 | Method and device for data synchronization between database cluster nodes |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610777767.4A CN106354830B (en) | 2016-08-31 | 2016-08-31 | Method and device for data synchronization between database cluster nodes |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106354830A CN106354830A (en) | 2017-01-25 |
| CN106354830B true CN106354830B (en) | 2020-12-04 |
Family
ID=57856596
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610777767.4A Active CN106354830B (en) | 2016-08-31 | 2016-08-31 | Method and device for data synchronization between database cluster nodes |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106354830B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106991606B (en) * | 2017-03-29 | 2020-09-18 | 恒生电子股份有限公司 | Transaction data processing method and device |
| CN108650320B (en) * | 2018-05-15 | 2021-05-11 | 北京中创腾锐技术有限公司 | Method for synchronizing configuration files of isomorphic cascading equipment in cluster environment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101610276A (en) * | 2008-06-16 | 2009-12-23 | 华为技术有限公司 | Data soft delete, recovery and method for synchronous and terminal and system |
| CN102685257A (en) * | 2012-06-11 | 2012-09-19 | 深圳中兴网信科技有限公司 | Data synchronous processing system, data synchronous processing method and server system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4171171B2 (en) * | 2000-10-25 | 2008-10-22 | 株式会社日立グローバルストレージテクノロジーズ | Data synchronization detection method and data synchronization detection device, information recording method and information recording device, information reproduction method and information reproduction device, information recording format, signal processing device, information recording and reproduction device, information recording medium |
| CN102023979B (en) * | 2009-09-09 | 2013-02-13 | 中国工商银行股份有限公司 | Meta-data management method and system |
| CN105718484A (en) * | 2014-12-04 | 2016-06-29 | 中兴通讯股份有限公司 | File writing method, file reading method, file deletion method, file query method and client |
| CN104967660B (en) * | 2015-05-11 | 2018-01-05 | 华中科技大学 | A kind of network performance method for improving towards more cloud frameworks |
| CN105045938A (en) * | 2015-09-17 | 2015-11-11 | 浪潮(北京)电子信息产业有限公司 | Metadata concurrent access method and system |
-
2016
- 2016-08-31 CN CN201610777767.4A patent/CN106354830B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101610276A (en) * | 2008-06-16 | 2009-12-23 | 华为技术有限公司 | Data soft delete, recovery and method for synchronous and terminal and system |
| CN102685257A (en) * | 2012-06-11 | 2012-09-19 | 深圳中兴网信科技有限公司 | Data synchronous processing system, data synchronous processing method and server system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106354830A (en) | 2017-01-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110532247B (en) | Data migration method and data migration system | |
| US11023448B2 (en) | Data scrubbing method and apparatus, and computer readable storage medium | |
| US10567542B2 (en) | Method for state based snapshot difference with restart capability | |
| CN109284073B (en) | Data storage method, device, system, server, control node and medium | |
| CN108509462B (en) | Method and device for synchronizing activity transaction table | |
| CN107515874B (en) | Method and equipment for synchronizing incremental data in distributed non-relational database | |
| US10489378B2 (en) | Detection and resolution of conflicts in data synchronization | |
| CN105824846B (en) | Data migration method and device | |
| CN111078667B (en) | Data migration method and related device | |
| EP2461615A1 (en) | Method for backing up terminal data and system thereof | |
| CN107291768B (en) | Index establishing method and device | |
| CN105450682B (en) | Method, device and system for synchronously storing data and synchronizing data to client | |
| US11500812B2 (en) | Intermediate file processing method, client, server, and system | |
| JP6940645B2 (en) | Replica processing methods, nodes, storage systems, servers and readable storage media | |
| CN110602165A (en) | Government affair data synchronization method, device, system, computer equipment and storage medium | |
| CN110928728A (en) | Virtual machine copying and switching method and system based on snapshot | |
| US20180121531A1 (en) | Data Updating Method, Device, and Related System | |
| CN106354830B (en) | Method and device for data synchronization between database cluster nodes | |
| CN114968966A (en) | Distributed metadata remote asynchronous replication method, device and equipment | |
| CN106874343B (en) | Data deletion method and system for time sequence database | |
| US11061889B2 (en) | Systems and methods of managing manifest refresh in a database | |
| WO2021082925A1 (en) | Transaction processing method and apparatus | |
| CN109783274B (en) | Disk snapshot management method and device and readable storage medium | |
| CN116049306A (en) | Data synchronization method, device, electronic equipment and readable storage medium | |
| US8630976B2 (en) | Fast search replication synchronization processes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |