CN105786959A - Synchronization method and device of primary database and spare database - Google Patents
Synchronization method and device of primary database and spare database Download PDFInfo
- Publication number
- CN105786959A CN105786959A CN201610013943.7A CN201610013943A CN105786959A CN 105786959 A CN105786959 A CN 105786959A CN 201610013943 A CN201610013943 A CN 201610013943A CN 105786959 A CN105786959 A CN 105786959A
- Authority
- CN
- China
- Prior art keywords
- transaction
- major key
- synchronized
- write
- write queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention discloses a synchronization device of a primary database and a spare database. The synchronization device comprises a synchronous transaction acquisition unit and a synchronous transaction execution unit, wherein the synchronous transaction acquisition unit is used for reading a database transaction in the primary database, obtaining database transaction information which needs to be synchronized, adding the database transaction information into a database transaction write sequence and taking an amount of database transaction primary keys as a primary key processing progress value; and the synchronous transaction execution unit is used for adopting a parallel write way to write the database transaction in the database transaction write sequence into the spare database, and deleting the database transaction on the top end of the database transaction write sequence. The invention also discloses a synchronization method of the primary database and the spare database. The synchronization device and method can greatly improve the cluster performance of MariaDB, effectively expands the applicable range of the MariaDB and can support a higher database visitor volume. In addition, a multi-thread parallel writing way reduces synchronization execution time, and the execution efficiency of the MariaDB is greatly improved.
Description
Technical field
The present invention relates to data processing field, relate more specifically to synchronous method and the device in a kind of master/slave data storehouse.
Background technology
MySQL is the widely used PostgreSQL database software of internet industry, but due to there is the risk that MySQL is closed source in the inscriptions on bones or tortoise shells, many application based on MySQL all suffer from needing to use substituting PostgreSQL database software, and MariaDB is the replacement scheme that MySQL is compatible best.But for large-scale website, use MariaDB data base to there is the problem that a MariaDB data base cannot support huge visit capacity.
It is set up MariaDB data-base cluster to this effective solution, it it is wherein the complete mirror image of MDL for data base, first write request performs on MDL, and read request can be distributed on all data bases, MDL is write in reception and write request is synchronized to according to the mode that single-threaded order performs on standby data base by synchronization means after data complete, thus providing more powerful data base read-write service.
In the version of the active and standby synchronization that current official provides, what active and standby synchronization adopted is single-threaded synchronization, when MDL load is higher, can cause that synchronizing relay increases, and standby data base can not read latest data in time, reduces the availability of data-base cluster.
Summary of the invention
In view of this, present invention is primarily targeted at a kind of master/slave data storehouse synchronous method of offer and device, in order to solve existing master/slave data storehouse when standby data base is more, the problem that synchronous efficiency is low.
To achieve these goals, as one aspect of the present invention, the invention provides a kind of master/slave data storehouse synchronizer, including:
Synchronized transaction acquiring unit, for by being read out the db transaction in MDL, it is thus achieved that needs the db transaction information synchronized, is added in db transaction write queue, and as major key, db transaction major key quantity is processed progress value；And
Synchronized transaction performance element, for adopting concurrent writing to be written to by the described db transaction in described db transaction write queue in standby data base, and deletes the described db transaction being positioned at described db transaction write queue top.
Wherein, described synchronized transaction performance element includes synchronized transaction process assembly, synchronized transaction write assembly and synchronized transaction and completes assembly, wherein said synchronized transaction processes each pending db transaction major key of assembly training in rotation and processes progress value, enters described synchronized transaction executive module when major key issued transaction progress value is 0；Major key is processed the db transaction that progress value is 0 and distributes and write thread and perform by described synchronized transaction executive module, adopts concurrent writing the affairs of write queue to be write in standby data base；Described synchronized transaction completes assembly after the standby wide area information server affairs of write complete, delete the db transaction being complete, delete this db transaction being positioned at major key write queue top, the Article 2 db transaction of major key write queue automatically becomes the pending affairs of major key write queue, and notifies that described db transaction major key processes progress value and subtracts one.
As another aspect of the present invention, present invention also offers a kind of master/slave data storehouse synchronous method, comprise the following steps:
By the db transaction in MDL is read out, it is thus achieved that need the db transaction information synchronized, obtain the major key of db transaction, the major key of recording synchronism db transaction, and db transaction major key quantity is processed progress value as major key；
According to db transaction major key information, by the db transaction got according to sequentially increasing in db transaction major key correspondence major key write queue in MDL；Wherein, create new major key write queue for not having the major key of write queue to need, and these affairs are set to write queue top, notify that this major key processes progress value and subtracts one；
Each pending db transaction major key of training in rotation processes progress value, enters synchronized transaction executive module when major key issued transaction progress value is 0；
Major key processes the db transaction that progress value is 0 distribute and write thread and perform, adopt concurrent writing the affairs of write queue to be write in standby data base；
The standby wide area information server affairs of write are deleted the db transaction being complete, are deleted this db transaction being positioned at major key write queue top after completing, and the Article 2 db transaction of major key write queue automatically becomes the pending affairs of major key write queue；Notify that this db transaction major key processes progress value and subtracts one.
Based on technique scheme it can be seen that the synchronizer of the present invention and method are relative to prior art, it is possible to the cluster performance of MariaDB be greatly promoted, effectively extend the scope on probation of MariaDB, it would be preferable to support higher database access amount；Additionally, using affairs as the least unit synchronized, affairs itself have atomicity, and the mode that simultaneous multi-threading is written in parallel to decreases the execution time of synchronization, is greatly improved the execution efficiency of MariaDB.
Accompanying drawing explanation
Fig. 1 is the FB(flow block) of the synchronous method based on MariaDB master/slave data storehouse of the present invention.
Detailed description of the invention
For making the object, technical solutions and advantages of the present invention clearly understand, below in conjunction with specific embodiment, and with reference to accompanying drawing, the present invention is described in further detail.
MariaDB is a MySQL branch version adopting Maria storage engines, is the database server freely increased income developed of the company established by the author MichaelWidenius of original MySQL.
Db transaction (DatabaseTransaction), refers to as the sequence of operations that single logical unit of work performs, or is completely executed, or fully do not perform.Issued transaction may insure that unless all operations in transactional unit is all successfully completed, and otherwise will not forever update data-oriented resource.By one group of associative operation being combined as all success or an all failed unit, it is possible to simplify Fault recovery and make application program relatively reliable.
The problem low in order to solve existing master/slave data storehouse synchronous efficiency, the invention discloses a kind of master/slave data storehouse synchronizer based on MariaDB data base, and it is made up of two parts, is synchronized transaction acquiring unit and synchronized transaction performance element respectively.
Synchronized transaction performance element is made up of three assemblies again, is that synchronized transaction processes assembly, synchronized transaction writes assembly, synchronized transaction completes assembly respectively.
Owing to synchronous working is performed continuously over, therefore in real process, the new affairs needing to synchronize constantly are given synchronized transaction performance element and are processed by synchronized transaction acquiring unit.Meanwhile, synchronized transaction performance element also due to being continuously increased of synchronized transaction and repeating to process assembly according to synchronized transaction, synchronized transaction write assembly, synchronized transaction complete assembly and carry out issued transaction.
In detail below each ingredient of the master/slave data storehouse synchronizer of the present invention is described in detail and sets forth.
Synchronized transaction acquiring unit
By the db transaction in MDL is read out, it is thus achieved that need the db transaction information synchronized.
For the db transaction got, it is necessary to obtain the major key of db transaction, the major key of recording synchronism db transaction, and db transaction major key quantity is processed progress value as major key.
The db transaction got, according to db transaction major key information, according to sequentially increasing in db transaction major key correspondence major key write queue in MDL, major key write queue performs according to the sequential system of first in first out.Create new major key write queue for not having the major key of write queue to need, and these affairs are set to write queue top, notify that this major key processes progress value and subtracts one.
Synchronized transaction performance element
Synchronized transaction performance element is made up of three assemblies, is that synchronized transaction processes assembly, synchronized transaction writes assembly and synchronized transaction completes assembly respectively.
Synchronized transaction processes assembly
Each pending db transaction major key of training in rotation processes progress value, enters synchronized transaction executive module when major key issued transaction progress value is 0.
Wherein, there are several pending db transaction major keys etc. pending inside major key write queue, and what these affairs major keys were ordered into, only the affairs major key before this affairs major key just can take turns to new affairs major key execution after having performed, in time taking turns to this affairs major key and process, it is 0 that this affairs major key processes progress value, execution can remove all affairs major keys of poll after completing, the process progress value of this affairs major key is set to-1, finds the pending affairs major key that affairs major key progress value is 0 simultaneously.
Synchronized transaction executive module
Major key processes the db transaction that progress value is 0 distribute and write thread and perform, adopt concurrent writing the affairs of write queue to be write in standby data base.
Synchronized transaction completes assembly
The standby wide area information server affairs of write are deleted the db transaction being complete, are deleted this db transaction being positioned at major key write queue top after completing, and the Article 2 db transaction of major key write queue automatically becomes the pending affairs of major key write queue.Notify that all affairs major keys of this data base process progress value and subtract one.
As another aspect of the present invention, the invention also discloses a master/slave data storehouse synchronous method, comprise the following steps:
By the db transaction in MDL is read out, it is thus achieved that need the db transaction information synchronized, obtain the major key of db transaction, the major key of recording synchronism db transaction, and db transaction major key quantity is processed progress value as major key；
According to db transaction major key information, by the db transaction got according to sequentially increasing in db transaction major key correspondence major key write queue in MDL, major key write queue performs according to the sequential system of first in first out；Wherein, create new major key write queue for not having the major key of write queue to need, and these affairs are set to write queue top, notify that this major key processes progress value and subtracts one；
Each pending db transaction major key of training in rotation processes progress value, enters synchronized transaction executive module when major key issued transaction progress value is 0；
Major key processes the db transaction that progress value is 0 distribute and write thread and perform, adopt concurrent writing the affairs of write queue to be write in standby data base；
The standby wide area information server affairs of write are deleted the db transaction being complete, are deleted this db transaction being positioned at major key write queue top after completing, and the Article 2 db transaction of major key write queue automatically becomes the pending affairs of major key write queue；Notify that this db transaction major key processes progress value and subtracts one.
Particular embodiments described above; the purpose of the present invention, technical scheme and beneficial effect have been further described; it it should be understood that; the foregoing is only specific embodiments of the invention; it is not limited to the present invention; all within the spirit and principles in the present invention, any amendment of making, equivalent replacement, improvement etc., should be included within protection scope of the present invention.
Claims (9)
1. a master/slave data storehouse synchronizer, including:
Synchronized transaction acquiring unit, for by being read out the db transaction in MDL, it is thus achieved that needs the db transaction information synchronized, is added in db transaction write queue, and as major key, db transaction major key quantity is processed progress value；And
Synchronized transaction performance element, for adopting concurrent writing to be written to by the described db transaction in described db transaction write queue in standby data base, and deletes the described db transaction being positioned at described db transaction write queue top.
2. master/slave data storehouse synchronizer according to claim 1, wherein said synchronized transaction acquiring unit needs to obtain the major key of described db transaction the major key of recording synchronism db transaction when obtaining the db transaction information needing to synchronize.
3. master/slave data storehouse synchronizer according to claim 1, wherein said synchronized transaction acquiring unit, by when needing the db transaction information synchronized to add in described db transaction write queue, is that the order in described MDL of the major key information according to described db transaction reads.
4. master/slave data storehouse synchronizer according to claim 1, wherein said synchronized transaction acquiring unit when needing the db transaction information synchronized to add in described db transaction write queue, will adopt the write of first in first out pattern.
5. master/slave data storehouse synchronizer according to claim 1, wherein said synchronized transaction performance element includes synchronized transaction process assembly, synchronized transaction write assembly and synchronized transaction and completes assembly, wherein:
Described synchronized transaction processes each pending db transaction major key of assembly training in rotation and processes progress value, enters described synchronized transaction executive module when major key issued transaction progress value is 0；
Major key is processed the db transaction that progress value is 0 and distributes and write thread and perform by described synchronized transaction executive module, adopts concurrent writing the affairs of write queue to be write in standby data base；And
Described synchronized transaction completes assembly after the standby wide area information server affairs of write complete, delete the db transaction being complete, delete this db transaction being positioned at major key write queue top, the Article 2 db transaction of major key write queue automatically becomes the pending affairs of major key write queue, and notifies that described db transaction major key processes progress value and subtracts one.
6. master/slave data storehouse synchronizer according to claim 1, the synchronizer of wherein said master/slave data storehouse adopts the data base based on MariaDB.
7. a master/slave data storehouse synchronous method, comprises the following steps:
By the db transaction in MDL is read out, it is thus achieved that need the db transaction information synchronized, obtain the major key of db transaction, the major key of recording synchronism db transaction, and db transaction major key quantity is processed progress value as major key；
According to db transaction major key information, by the db transaction got according to sequentially increasing in db transaction major key correspondence major key write queue in MDL；Wherein, create new major key write queue for not having the major key of write queue to need, and these affairs are set to write queue top, notify that this major key processes progress value and subtracts one；
Each pending db transaction major key of training in rotation processes progress value, enters synchronized transaction executive module when major key issued transaction progress value is 0；
Major key processes the db transaction that progress value is 0 distribute and write thread and perform, adopt concurrent writing the affairs of write queue to be write in standby data base；
The standby wide area information server affairs of write are deleted the db transaction being complete, are deleted this db transaction being positioned at major key write queue top after completing, and the Article 2 db transaction of major key write queue automatically becomes the pending affairs of major key write queue；Notify that this db transaction major key processes progress value and subtracts one.
8. master/slave data storehouse according to claim 7 synchronous method, wherein said according to db transaction major key information, by the db transaction got according to, in the step sequentially increased in db transaction major key correspondence major key write queue in MDL, adopting the write of first in first out pattern.
9. master/slave data storehouse according to claim 7 synchronous method, wherein said master/slave data storehouse all adopts the data base based on MariaDB.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610013943.7A CN105786959A (en) | 2016-01-11 | 2016-01-11 | Synchronization method and device of primary database and spare database |
| CN201610532157.8A CN106959989A (en) | 2016-01-11 | 2016-07-07 | The synchronous method and device in master/slave data storehouse |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610013943.7A CN105786959A (en) | 2016-01-11 | 2016-01-11 | Synchronization method and device of primary database and spare database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105786959A true CN105786959A (en) | 2016-07-20 |
Family
ID=56402317
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610013943.7A Pending CN105786959A (en) | 2016-01-11 | 2016-01-11 | Synchronization method and device of primary database and spare database |
| CN201610532157.8A Pending CN106959989A (en) | 2016-01-11 | 2016-07-07 | The synchronous method and device in master/slave data storehouse |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610532157.8A Pending CN106959989A (en) | 2016-01-11 | 2016-07-07 | The synchronous method and device in master/slave data storehouse |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN105786959A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109977168A (en) * | 2019-03-18 | 2019-07-05 | 武汉达梦数据库有限公司 | The method for synchronizing data of database and equipment preloaded based on data page |
| CN110008284A (en) * | 2019-03-18 | 2019-07-12 | 武汉达梦数据库有限公司 | Method for synchronizing data of database and equipment based on data page preloading and rollback |
| CN110309224A (en) * | 2018-03-14 | 2019-10-08 | 华为技术有限公司 | A kind of data copy method and device |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109918178B (en) * | 2019-03-06 | 2021-04-30 | 恒生电子股份有限公司 | Transaction submitting method and related device |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2003006020A (en) * | 2001-06-26 | 2003-01-10 | Hitachi Ltd | Duplicate data base access device and its accessing method |
| CN102098342B (en) * | 2011-01-31 | 2013-08-28 | 华为技术有限公司 | Transaction level-based data synchronizing method, device thereof and system thereof |
| CN102629268B (en) * | 2012-03-09 | 2014-12-17 | 华为技术有限公司 | Data synchronization method, system and date access device |
| CN103885986B (en) * | 2012-12-21 | 2017-12-19 | 阿里巴巴集团控股有限公司 | The synchronous method and apparatus in master/slave data storehouse |
| CN104111962B (en) * | 2013-04-22 | 2018-09-18 | Sap欧洲公司 | Enhanced affairs cache with batch operation |
| CN105117304A (en) * | 2015-07-29 | 2015-12-02 | 北京京东尚科信息技术有限公司 | MariaDB-based database recovery system and method |
-
2016
- 2016-01-11 CN CN201610013943.7A patent/CN105786959A/en active Pending
- 2016-07-07 CN CN201610532157.8A patent/CN106959989A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110309224A (en) * | 2018-03-14 | 2019-10-08 | 华为技术有限公司 | A kind of data copy method and device |
| CN110309224B (en) * | 2018-03-14 | 2021-08-31 | 华为技术有限公司 | Data copying method and device |
| CN109977168A (en) * | 2019-03-18 | 2019-07-05 | 武汉达梦数据库有限公司 | The method for synchronizing data of database and equipment preloaded based on data page |
| CN110008284A (en) * | 2019-03-18 | 2019-07-12 | 武汉达梦数据库有限公司 | Method for synchronizing data of database and equipment based on data page preloading and rollback |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106959989A (en) | 2017-07-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109101627B (en) | Heterogeneous database synchronization method and device | |
| US9779128B2 (en) | System and method for massively parallel processing database | |
| CN103780638B (en) | Method of data synchronization and system | |
| EP3185143B1 (en) | Decentralized transaction commit protocol | |
| CN102402596B (en) | A kind of reading/writing method of master slave separation database and system | |
| US8527459B2 (en) | System and method for data replication between heterogeneous databases | |
| US9881041B2 (en) | Multiple RID spaces in a delta-store-based database to support long running transactions | |
| CN105786959A (en) | Synchronization method and device of primary database and spare database | |
| CN108509462B (en) | Method and device for synchronizing activity transaction table | |
| US20160142249A1 (en) | Method, medium, and system for configuring a new node in a distributed memory network | |
| CN104615759A (en) | Data synchronization method for different business system platforms | |
| CN106777270A (en) | A kind of Heterogeneous Database Replication parallel execution system and method based on submission point time line locking | |
| US20140280394A1 (en) | Multi-Threaded Message Passing Journal | |
| US20160088077A1 (en) | Seamless binary object and metadata sync | |
| CN103514212A (en) | Data writing method and system | |
| CN104346373A (en) | Partition log queue synchronization management method and device | |
| CN106873902B (en) | File storage system, data scheduling method and data node | |
| CN104462342B (en) | database snapshot synchronization processing method and device | |
| CN111694863B (en) | Database cache refreshing method, system and device | |
| CN109977168A (en) | The method for synchronizing data of database and equipment preloaded based on data page | |
| CN103106200A (en) | Synchronization system of non-relational type database and double-writing synchronization method | |
| WO2015085901A1 (en) | File transmission method and device, and related system | |
| US8812468B2 (en) | Database management method | |
| CN106155838A (en) | A kind of database back-up data restoration methods and device | |
| CN110008284A (en) | Method for synchronizing data of database and equipment based on data page preloading and rollback |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20160720 |