CN103885986B - The synchronous method and apparatus in master/slave data storehouse - Google Patents
The synchronous method and apparatus in master/slave data storehouse Download PDFInfo
- Publication number
- CN103885986B CN103885986B CN201210564590.1A CN201210564590A CN103885986B CN 103885986 B CN103885986 B CN 103885986B CN 201210564590 A CN201210564590 A CN 201210564590A CN 103885986 B CN103885986 B CN 103885986B
- Authority
- CN
- China
- Prior art keywords
- affairs
- write
- queue
- major key
- virtual record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The present invention relates to a kind of synchronous method and apparatus in master/slave data storehouse, and for the data syn-chronization between primary database and standby database, this method includes：Affairs distribute step, and the mark of each affairs read from primary database or each affairs is distributed to write queue；Affairs write step, affairs corresponding to the affairs or mark respectively write in write queue corresponding to thread are write into standby database using multiple thread parallels of writing.The present invention writes thread parallel execution write operation simultaneously using multiple, reduces time delay, can realize the synchronization of master/slave data storehouse in time, improve synchronous efficiency.
Description
Technical field
The application is related to data processing field, the synchronous method and apparatus in especially a kind of database master/slave data storehouse.
Background technology
MySQL is the widely used PostgreSQL database software of internet industry.Many large-scale websites are all using MySQL numbers
According to storehouse, as business scale constantly increases, a MySQL database can not support huge visit capacity.
It is exactly to establish the active and standby clusters of MySQL to have an effective solution, and standby database is the complete mirror of primary database
Picture.Write request only falls in primary database, and read request balanced can be distributed on all databases, and so whole cluster can carry
For the read-write service of more Large Copacity.MySQL primary databases are after newly-increased change data are received, then by synchronization means, changing
Real time data synchronization is to for database.
Mysql binlog daily records are a kind of binary logs, including three kinds of forms, respectively ROW (OK) form,
Statement (SQL statement) form, MIX (mixing) form.Wherein it is complete to have recorded the full line that each data change for ROW forms
Old value and new value.Abundant information is complete, and possesses repeat replication to from storehouse, obtained final data or consistent spy
Property.
MySQL officials provide Community Edition, and active and standby synchronization is synchronous using single thread.When primary database has higher write-in
During load, the delay increase of master-slave synchronisation, cause timely read newest data from storehouse, reduce whole cluster
Availability.
The content of the invention
The application technical problems to be solved are to provide a kind of synchronous method and apparatus in master/slave data storehouse, existing to solve
The problem of master/slave data storehouse synchronous efficiency is low.
In order to solve the above technical problems, this application provides a kind of synchronous method in master/slave data storehouse, for primary database
Data syn-chronization between standby database, this method include：
Affairs distribute step, and the mark of each affairs read from primary database or each affairs is distributed to write queue；
Affairs write step, using multiple thread parallels of writing by the affairs respectively write in write queue corresponding to thread or mark pair
The standby database of affairs write-in answered.
Further, it is described distribute step before, this method also includes：
Pre-treatment step, the major key of each affairs is extracted successively, list of primary keys is created for each affairs, according to each affairs
List of primary keys, the mark of each affairs is put into major key transaction queues corresponding to the major key of each affairs successively, in master
The affairs of the head of the queue of key transaction queues obtain virtual record corresponding to the major key and locked；
The premise for distributing affairs or write-in affairs is equal to the list of primary keys of the affairs for the virtual record lock sum of the affairs
The sum of middle major key.
Further, in the pre-treatment step, the list of primary keys is created after removing the major key of repetition.
Further, the step that distributes also includes：Affairs for having been written into standby database, deletion are located at major key thing
The Transaction Identifier of the affairs of business queue head of the queue, virtual record lock is passed to the next node of the major key transaction queues.
Further, the affairs distribute step and performed by distributing thread, in pre-treatment step, are created for each affairs empty
Intend record lock counter, the affairs often obtain virtual record lock, and virtual record lock counter adds 1, described to distribute thread
Or write thread and judge that the virtual record lock sum of Current transaction is equal to the major key of the affairs according to virtual record lock counter
The sum of major key in list.
Further, affairs are distributed in step, are selected according to the most short principle of write queue or affairs cryptographic Hash for Current transaction
Write queue.
Further, the cryptographic Hash for stating affairs is that first major key of each affairs is converted into character string, then passes through Kazakhstan
Wish what is be calculated.
Further, after affairs write step, this method also includes delete step, deletes the thing for having been written into standby database
Business object.
In order to solve the above technical problems, present invention also offers a kind of synchronous device in master/slave data storehouse, the device includes：
Dispatch unit, for successively distributing the mark of each affairs read from primary database or each affairs to writing team
Row；
Writing unit, for using multiple thread parallels of writing by the affairs respectively write in write queue corresponding to thread or mark pair
The standby database of affairs write-in answered；
Several write queues, for depositing the mark of affairs that dispatch unit distributes or affairs, a write queue corresponding one
It is individual to write thread.
Compared to prior art, the synchronous least unit of technical scheme is affairs；It ensure that in synchronizing process
The atomicity of affairs；And thread parallel execution write operation simultaneously is write using multiple, reduce time delay, active and standby number can be realized in time
It is synchronous according to storehouse, synchronous efficiency is improved, adds the availability of cluster.
Brief description of the drawings
Fig. 1 is the schematic diagram of the synchronous embodiment of the method 1 in the application master/slave data storehouse；
Fig. 2 is the schematic diagram of the synchronous embodiment of the method 2 in the application master/slave data storehouse；
Fig. 3 is the schematic diagram of the synchronous embodiment of the method 3 in the application master/slave data storehouse；
Fig. 4 is the schematic diagram of the synchronous embodiment of the method 4 in the application master/slave data storehouse；
Fig. 5 is the schematic diagram for the application example that the application provides；
Fig. 6 is the modular structure schematic diagram of the synchronous device in master/slave data storehouse of the present invention；
Fig. 7 is another modular structure schematic diagram of the synchronous device in master/slave data storehouse of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present application, the technical scheme in the embodiment of the present application is carried out clear, complete
Site preparation describes.Obviously, described embodiment is only some embodiments of the present application, and not all embodiment.Based on this
Embodiment in application, those of ordinary skill in the art are obtained every other on the premise of creative work is not made
Embodiment, belong to the scope of the application protection.It should be noted that in the case where not conflicting, the embodiment in the application
And the feature in embodiment can be mutually combined.
Embodiment 1
The synchronous method in the embodiment of the present application master/slave data storehouse, as shown in figure 1, between primary database and standby database
Data syn-chronization, this method include：
Step 101：Affairs distribute step, successively send the mark of each affairs read from primary database or each affairs
It is sent to write queue；
The order of the reading for the affairs that the embodiment of the present application is read from primary database is submitted and write in primary database for it
Enter the order of daily record.
The affairs distribute step and distribute thread realization by one.
Affairs are distributed in step, are that Current transaction selects write queue according to the most short principle of write queue or affairs cryptographic Hash.
It is preferred that using the most short principle of write queue, will affairs currently to be distributed distribute to most short write queue and (appoint
The minimum write queue of business amount), in this way, load balancing can be admirably achieved.
The application is not limited the method for obtaining the cryptographic Hash of affairs, and alternatively, the cryptographic Hash of affairs can be by each thing
First major key of business is converted into character string, then obtained by Hash calculation.
Step 102：Affairs write step, using multiple affairs write thread parallel and will respectively write in write queue corresponding to thread
Or the standby database of affairs write-in corresponding to mark.
In the application, synchronous least unit is affairs；When write-in is for database, it can be ensured that affairs are not broken up, i.e.,
Primary database is the record of an affairs write-in, can also be write from storehouse in an affairs, ensure to get in synchronizing process
The atomicity of business.
Understandably, the mark of affairs serially being distributed to write queue even with single thread, its speed is still very fast, and
When affairs write standby database, thread parallel execution write operation simultaneously is write using multiple, reduces time delay, can realize in time
Master/slave data storehouse is synchronous, improves synchronous efficiency, adds the availability of cluster.
Embodiment 2
The synchronous method in the application master/slave data storehouse, as shown in Fig. 2 same for the data between primary database and standby database
Step, this method include：
Step 201：Pre-treatment step, the major key of each affairs read from primary database is extracted successively, is each affairs
List of primary keys is created, according to the list of primary keys of each affairs, the mark of each affairs is put into the major key pair of each affairs successively
In the major key transaction queues answered, virtual record corresponding to the affairs of the head of the queue in the major key transaction queues major key is locked；
Herein, the corresponding major key transaction queues of a major key, different major keys correspond to different major key transaction queues.
One affairs is before standby database is write, it is necessary to obtain the virtual record corresponding to every record of the affairs corresponding to major key
Lock.
For example, the major key of affairs A N number of row record is a, then major key a major key transaction queues include affairs A
N number of mark, when the head of the queue deposit transaction A of major key transaction queues mark, affairs A obtains the N number of of the major key successively
Virtual record is locked.
Each corresponding major key transaction queues (PK Tran Queue) of major key.The virtual record lock of each major key is by such as
Under type is safeguarded：Transaction Identifier is in the affairs of the head of the queue of major key transaction queues, obtains virtual record corresponding to the major key and locks；When
After affairs corresponding to the Transaction Identifier of head of the queue have been written into standby database, head of the queue is deleted, virtual record lock is passed into the queue
Next node.
Step 202：Affairs distribute step, when the virtual record lock sum of affairs is equal to major key in the list of primary keys of the affairs
Sum when the affairs are distributed to write queue；
The affairs distribute step and distribute thread realization by one.
Affairs are distributed in step, are that Current transaction selects write queue according to the most short principle of write queue or affairs cryptographic Hash.
It is preferred that using the most short principle of write queue, that is, select most short write queue to distribute affairs, can admirably achieve negative
Carry balanced.
The application is not limited the cryptographic Hash method for obtaining affairs, and alternatively, the cryptographic Hash of affairs can be by each affairs
First major key be converted into character string, then obtained by Hash calculation.
Step 203：Affairs write step, using multiple affairs write thread parallel and will respectively write in write queue corresponding to thread
Or the standby database of affairs write-in corresponding to mark.
Understandably, Transaction Identifier serially being distributed to write queue even with single thread, its speed is still very fast, and
When write-in is for database, threads parallel execution write operation simultaneously is write using multiple, reduces time delay, active and standby number can be realized in time
It is synchronous according to storehouse, synchronous efficiency is improved, adds the availability of cluster.
Optimally, in above example 2, in step 201 pre-treatment step virtual record lock meter can be created for each affairs
Number device, the affairs often obtain a virtual record lock, and virtual record lock counter adds 1, and step 202 affairs are distributed in step,
Distribute thread and judge that the virtual record of current event locks whether sum is equal to the affairs according to virtual record lock counter
The sum of major key in list of primary keys, and then decide whether to distribute the affairs to write queue.
Alternatively, in step 201 pre-treatment step of above example 2, before the list of primary keys of establishment, can delete wherein heavy
Multiple major key, for example, the major key of affairs A N number of row record is a, then after the major key for deleting repetition, now major key a is still right
Affairs A N number of row is answered to record.Major key a major key transaction queues include an affairs A mark (such as pointer), phase
Answer, when affairs A mark-row is in the head of the queue of major key transaction queues corresponding to major key a, affairs A obtains 1 of the major key
Virtual record is locked.Step 202 affairs are distributed in step, are distributed thread and are locked always still through the virtual record for judging event acquisition
The sum whether number is equal to major key in the list of primary keys of the affairs distributes to trigger affairs.
Embodiment 3
The synchronous method in the application master/slave data storehouse, as shown in figure 3, same for the data between primary database and standby database
Step, this method include：
Step 301：Pre-treatment step, extracts the major key of each affairs read from primary database successively, and removes repetition
Major key, for each affairs create list of primary keys；According to the list of primary keys of each affairs, successively by the mark (ratio of each affairs
Such as pointer) it is put into major key transaction queues corresponding to each major key of the affairs, Transaction Identifier is in the head of the queue of major key transaction queues
Affairs obtain corresponding to the major key virtual record and lock；
In step 301, major key is extracted from each row record of affairs successively, after the major key for removing repetition, a major key
Corresponding one or several rows record.
The virtual record lock of each major key is safeguarded as follows：Transaction Identifier in major key transaction queues head of the queue is corresponding
Affairs, obtain virtual record corresponding to the major key and lock；After the Transaction Identifier of head of the queue, which corresponds to affairs, has been written into standby database,
Head of the queue is deleted, virtual record lock is passed to the next node of the queue.
Step 302：Affairs distribute step, successively distribute the mark of each affairs or each affairs to write queue；
The affairs distribute step and distribute thread realization by one.
Affairs are distributed in step, are that Current transaction selects write queue according to the most short principle of write queue or affairs cryptographic Hash.
It is preferred that using the most short principle of write queue, that is, select most short write queue to distribute affairs, can admirably achieve negative
Carry balanced.
The application is not limited the cryptographic Hash method for obtaining affairs, and alternatively, the cryptographic Hash of affairs can be by each affairs
First major key be converted into character string, then obtained by Hash calculation.
Step 303：Affairs write step, realized using multiple thread parallels of writing, write thread and judge to be located at write queue head of the queue
The virtual record lock sum of affairs when being equal to the sum of major key in the list of primary keys of the affairs, the affairs are write into standby data
Storehouse.
Understandably, Transaction Identifier serially being distributed to write queue even with single thread, its speed is still very fast, and
When write-in is for database, threads parallel execution write operation simultaneously is write using multiple, reduces time delay, active and standby number can be realized in time
It is synchronous according to storehouse, synchronous efficiency is improved, adds the availability of cluster.
Optimally, in above example 3, in step 301 pre-treatment step virtual record lock meter can be created for each affairs
Number devices, the affairs often obtain virtual record lock, and virtual record lock counter adds 1, in step 303 affairs write step,
Write thread and judge that the virtual record lock sum of Current transaction is equal to the primary key column of the affairs according to virtual record lock counter
The sum of major key in table.
Embodiment 4
Such as Fig. 4, largely used queue (Queue) and Hash (Hash) to calculate in the present embodiment, thus also known as this
Apply as Hash Queue Algorithm.
The present embodiment includes input rank (Input Queue), source queue (Source Queue), major key transaction queues
(PK Tran Queue), write queue (Wrtier Queue), queue to be released (Free Queue), pretreatment thread
(Preprocess Thread), thread (Dispatcherd Thread) is distributed, writes thread (WrtierThread)；Wherein only
It is multiple according to configuration establishment to write thread (Writer Thread) and write queue (Writer Queue), other are single reality
Example；One is write the corresponding write queue of thread.
Such as scheme in each key modules maintenance and management some queues, next according to the process of stream compression, introduce
The implementation process and key modules of this programme：
Step 401, the affairs in input rank (Input Queue) are pre-processed；
The input data of the application is deposited in input rank (Input Queue), i.e. the pointer of affairs one by one, according to
The order of affairs is read in primary database, is first read, is first stored in its affairs pointer, first consumes.
Thread (Preprocess Thread) is pre-processed, affairs corresponding to the affairs pointer in input rank are done as follows
Processing：
Transaction queues node (BinlogItem) is created, including it is such as properties：The affairs of input, the list of primary keys of the affairs
(PK List), the hash value of affairs, virtual record lock counter.
The major key of affairs is extracted, removes and repeats major key, each affairs are created every in a list of primary keys, including affairs
Major key corresponding to row record；
Affairs Hash (Hash) value is calculated, first major key in transaction queues is extracted, he is converted into character string, such as
" library name table name row name values ", then calculated by Hash, the long shaping hash value that major key character string maps are 64, as
Affairs distribute the hash value used in write queue.
The value of initialization of virtual record lock calculator is zero.
Source queue (Source Queue), after the completion of transaction preprocess, just it is put into source queue (SourceQueue).Source
Queue is used only to the temporary transaction queues node (BinlogItem) for finishing pretreatment；
Major key transaction queues (PK Tran Queue), the i.e. corresponding queue of a major key, press advanced elder generation in this queue
The pattern gone out deposits the pointer of the transaction queues node (BinlogItem) pre-processed；The pointer of transaction queues node is located at
The head of the queue of major key transaction queues, that is, indicate that corresponding affairs achieve virtual record corresponding to the major key and locked；
How many different major keys of one affairs, it is put into how many different major key transaction queues；The object of entrance is straight
Connect be transaction queues node (BinlogItem) pointer, this object points to whole affairs, and affairs are by being by all record groups
Into.
According to the list of primary keys of each affairs, it is each that the pointer of the transaction queues node of each affairs is put into the affairs successively
In major key transaction queues corresponding to individual major key, affairs corresponding to the pointer of the transaction queues node in major key transaction queues head of the queue
Virtual record corresponding to the major key is obtained to lock.
The virtual record lock that the record of recovery affairs occupies, after affairs have been written into standby database, discharge major key affairs
Affairs corresponding to transaction queues node pointer in queue (PK Tran Queue).From the affairs list of primary keys of the affairs, by
It is individual to obtain each major key, then major key transaction queues corresponding to the major key (PK Tran Queue) are obtained, and head of the queue is deleted, then
Virtual record lock is passed to next node (BinlogItem) in the major key transaction queues (PKTran Queue), i.e., under
The virtual record lock counter of affairs corresponding to one node adds 1；If the queue has been empty, you can deletes and discharges the team
Row, discharge queue nodes object again after completing this step.
Queue (Free Queue) to be released, fifo queue, for depositing the affairs for having been written into standby database
The pointer of queue nodes；
Distribute the pointer of transaction queues node of the thread in queue to be released, corresponding affairs pair in releasing memory
As now writing thread and corresponding affairs being written with into standby database, complete synchronization.
Virtual record lock, the corresponding virtual record lock of every record, virtual record lock have two states, lock, be unlocked
It is fixed；Lock-out state is to represent that this record has locked in the present system, and the record is write standby database, will not produce data
Conflict；Unlocked state, represent that the record does not possess the prerequisite of the standby database of write-in also.In the application, an affairs exist
, it is necessary to obtain the virtual record lock corresponding to every record of the affairs corresponding to major key before the standby database of write-in；One affairs
In all different records virtual records lock, counter attribute is locked using the virtual record of transaction queues node (BinlogItem)
To describe.The virtual record lock of a record is obtained, that is, represents the virtual record lock status of locking record, the in other words record
It is arranged to lock.
Virtual record locks counter, is a shaping counter, initial value 0, maximum is equal to step 1 pretreatment funeral affairs
The size (i.e. the sum of major key in list of primary keys) of the list of primary keys of business；Corresponding affairs obtain a virtual record lock every time,
Corresponding numerical value just adds 1, when the sum for all virtual records lock that an affairs obtain is equal to the size of list of primary keys, represents
The affairs can write standby database, without producing any data collision.
Virtual record lock calculator is for single affairs, if has had the mark write from the condition in storehouse.Because
One affairs may come the head of the queue of part major key transaction queues, while come in the team of part major key transaction queues, such a feelings
Under condition, according to the application, the affairs just do not possess the precondition for performing and distributing or writing, unless affairs acquisition is all virtual
Record lock.
Step 402：Affairs are distributed to write queue；
Distributing affairs has two ways：
A. all virtual record locks are first obtained, then are distributed：The pointer of transaction queues node is first put into the transaction queues section
In the tail of the queue of major key transaction queues (PK Tran Queue) corresponding to all major keys of point, if the pointer of transaction queues node is
Through coming head of the queue, then corresponding transaction queues node obtains virtual record corresponding to the major key and locked, the transaction queues node
Virtual record lock counter attribute add 1；Otherwise transaction queues node waits for obtaining virtual record lock in major key transaction queues,
After a transaction queues node obtains all virtual record locks, you can be dispatched to write queue, the algorithm distributed can
To be, write queue is traveled through, the pointer or affairs pointer of transaction queues node are put into most short write queue；Can also be according to step
The affairs cryptographic Hash of 1 generation, is distributed by hash algorithm, such as Tran Hash Value%WriteQueue Size act
Business cryptographic Hash modulus number of queues, subscript of the obtained numerical value as write queue array；
B. directly distribute：The pointer of transaction queues node is first put into major key transaction queues (PK corresponding to its all major key
Tran Queue) in, if the pointer of transaction queues node has come head of the queue, then corresponding transaction queues node obtains
Virtual record corresponding to the major key is locked；No matter whether transaction queues node obtains all virtual record locks, all distributes transaction queues
The pointer of node is to write queue.The affairs cryptographic Hash that the algorithm distributed can generate according to step 1, is distributed by hash algorithm.Adopt
Distribute mode with this, write thread each affairs are write before standby database need to check whether the affairs obtain it is all virtual
Record lock, write again after waiting affairs to obtain all virtual record locks without if.
Step 403, affairs are written in parallel to,
Write queue (Wrtier Queue) and first in first out (FIFO) queue, deposit by step 402 distribute Lai
The pointer of transaction queues node；It is each to write the corresponding write queue of thread；
If using a modes in step 403, transaction queues node of the thread in the write queue of oneself is each write
Pointer goes interior access affairs, and writing affairs according to the standard of db transaction (starts affairs, one by one write-in record, submit thing
Business), after the completion of write-in, being removed from write queue for the pointer of transaction queues node, and it is put back into queue (Free to be released
Queue)。
If using b modes in step 403, before write-in, write thread and first check whether the transaction queues node obtains
All virtual record locks, if so writing；The transaction queues node is so waited to obtain all virtual record locks if not
Write operation is performed again
The application has preferable performance using C/C++, but can also use other high level languages.In the application
Implementation experiment in, determine that the application can be so that the write-in of standby database reach capacity；
Affairs are pre-processed and distributed in the application serially to perform to write queue, and write step is to perform parallel, such as
The affairs that have modified identical recordings are serially written standby database by this successively, and what it is for the i.e. modification of incoherent affairs is complete
The affairs of difference record, parallel processing.
As shown in figure 5, input data is three orderly affairs, affairs 1 (Tran 1) and affairs 2 (Tran 2) all update
Same record B, then affairs 1 and affairs 2 should be serial, and order must be first write-in affairs 1, then write affairs 2.
Affairs 3 and affairs 1, affairs 2 all have no relations, so can also be written in parallel to.Using the application mechanism, affairs can be performed simultaneously
1 and the write operation of affairs 3.And affairs 2, because there is no enough virtual record locks, it have to wait until that it obtains foot
Enough virtual record locks, you can the standby database of write-in.So as to having carried out effective control to the parallel and serial of affairs, reaching can be with
Parallel is parallel, it should which serial is serial.
When the transaction queues node of affairs 1 and affairs 2 is not written into same write queue, before the write-in of affairs 1, if
403a, then writing thread and being suspended for affairs 2 is performed, until affairs 1 write, and release virtual record lock；If 403b,
So affairs 2 before the write-in of affairs 1 all without being distributed to write queue.
In above example, input rank is primarily to ensure that affairs corresponding to node therein are according to primary database
The order that middle affairs are read, the node of specific input rank are that affairs, the finger of affairs will not produce shadow for present invention essence
Ring.
Node in the queue of source is the transaction queues node being sequentially generated in input rank, each transaction queues node
The virtual record lock counter of each property parameters and the distinctive affairs of the present invention including affairs.
Each corresponding specific major key of major key transaction queues, wherein the node deposited is corresponding to the major key, basis
The affairs pointer or transaction queues node of affairs sequential storage in input rank or source queue (can guarantee that affairs sequential)
Pointer, it is mainly used in counting the virtual record lock of affairs and affairs obtain the sequential that virtual record is locked and controlled
System.
Node in write queue and queue to be released is the pointer of affairs or the pointer of transaction queues node, is mainly used in referring to
Show that writing thread carries out write operation to which affairs or complete write operation.
It in summary it can be seen, the pointer of affairs or the pointer of transaction queues node are respectively provided with the effect of Identifying transactions, can unite
Claim Transaction Identifier.Caching is only needed from the affairs of primary database output once, treats transaction write corresponding to Transaction Identifier in write queue
After entering standby database, affairs corresponding to Transaction Identifier in queue to be released are deleted.
The occupancy of cache resources can be reduced using the mark of affairs, improve treatment effeciency.
To realize the above method, present invention also offers a kind of synchronous device in master/slave data storehouse, as shown in fig. 6, the dress
Put including：
Dispatch unit, for successively distributing the mark of each affairs read from primary database or each affairs to writing team
Row；
Dispatch unit is that Current transaction selects write queue according to the most short principle of write queue or affairs cryptographic Hash.
The affairs cryptographic Hash is that first major key of each affairs is converted into character string, then is obtained by Hash calculation
's.
Writing unit, for using multiple thread parallels of writing by the affairs respectively write in write queue corresponding to thread or mark pair
The standby database of affairs write-in answered；
Several write queues, for depositing the mark of affairs that dispatch unit distributes or affairs, a write queue corresponding one
It is individual to write thread.
As shown in fig. 7, unlike Fig. 6, described device also includes：
Pretreatment unit, for extracting the major key of each affairs successively, list of primary keys is created for each affairs；It is additionally operable to root
According to the list of primary keys of each affairs, the pointer of each affairs is put into major key transaction queues corresponding to each major key of the affairs successively
In, the affairs that the pointers of affairs is in head of the queue obtain virtual record corresponding to the major key and locked；
Several major key transaction queues, for storing the mark of each affairs；
The dispatch unit or writing unit, when the virtual record lock sum that affairs obtain is equal to the list of primary keys of the affairs
During the sum of middle major key, affairs or write-in affairs are distributed.
Preferably, the pretreatment unit removes the list of primary keys of establishment affairs after the major key repeated.
Preferably, the dispatch unit, be additionally operable to delete each major key transaction queues head of the queue has been written into standby database
Affairs Transaction Identifier, virtual record lock is passed into the next node of the queue.
Preferably, the dispatch unit is realized by distributing thread, and pretreatment unit is additionally operable to create virtually for each affairs
Record lock counter, the affairs often obtain virtual record lock, and virtual record lock counter adds 1, the dispatch unit or
Writing unit locks counter according to the virtual record and judges that the virtual record lock sum of current event is equal to the major key of the affairs
The sum of major key in list.
Technical scheme has advantages below：
Synchronous least unit is affairs；When write-in is for database, it can be ensured that affairs are not broken up, i.e., primary database is
The record of one affairs write-in, can also write in an affairs from storehouse, the atomicity of affairs is ensure that in synchronizing process；
Thread parallel execution write operation simultaneously is write using multiple, reduces time delay, can realize that master/slave data storehouse is same in time
Step, improves synchronous efficiency, adds the availability of cluster；
Locked by virtual record so that standby database can be written in parallel between different affairs and may insure sequential and not
Produce deadlock；
In same recovery, the distribution for distributing processing virtual record lock in thread (Dispatcher Thread), avoid
Multithreading contention record lock, improves performance, reduces program complexity；
Affairs can first be distributed to write queue, allowed and write thread and go whether verification obtains all record locks；Institute can also be obtained
There is record to distribute affairs again after locking and enter write queue, then write direct standby database；After affairs write standby database, it will put back to
To queue to be released；
Using the application, parallel synchronous can be carried out to database synchronization, improve writing speed more than 500%
Simultaneously, maximum delay is controlled to improve handling capacity within 1 second, and reduced time delay, and during also ensuring synchronization
Affairs are not broken up, and have the advantages of affairs strong consistency.
Claims (14)
1. a kind of synchronous method in master/slave data storehouse, for the data syn-chronization between primary database and standby database, it is characterised in that
This method includes：
Affairs distribute step, and the mark of each affairs read from primary database or each affairs is distributed to write queue；
Affairs write step, using multiple thread parallels of writing by corresponding to the affairs or mark respectively write in write queue corresponding to thread
The standby database of affairs write-in；
Distribute affairs or write affairs premise be：The affairs obtain the virtual record lock corresponding to each record of the affairs；Its
In, the virtual record lock that the described affairs obtain corresponding to each record of the affairs represents that the affairs lock each note of the affairs
Record.
2. the method as described in claim 1, it is characterised in that：It is described distribute step before, this method also includes：
Pre-treatment step, the major key of each affairs is extracted successively, list of primary keys is created for each affairs, according to the master of each affairs
Key list, the mark of each affairs is put into major key transaction queues corresponding to the major key of each affairs successively, in major key thing
The affairs of the head of the queue of business queue obtain virtual record corresponding to the major key and locked；
The premise for distributing affairs or write-in affairs is equal to master in the list of primary keys of the affairs for the virtual record lock sum of the affairs
The sum of key.
3. method as claimed in claim 2, it is characterised in that：In the pre-treatment step, created after removing the major key of repetition
The list of primary keys.
4. method as claimed in claim 2, it is characterised in that the step that distributes also includes：For having been written into standby data
The affairs in storehouse, the Transaction Identifier of the affairs positioned at major key transaction queues head of the queue is deleted, virtual record lock is passed into the major key
The next node of transaction queues.
5. method as claimed in claim 4, it is characterised in that：The affairs distribute step and performed by distributing thread, pretreatment
In step, create virtual record for each affairs and lock counter, the affairs often obtain a virtual record lock, virtual record lock
Counter adds 1, it is described distribute thread or write thread the virtual record that counter judges Current transaction is locked according to the virtual record
Lock sum is equal to the sum of major key in the list of primary keys of the affairs.
6. the method as described in claim 1, it is characterised in that：Affairs are distributed in step, according to the most short principle of write queue or thing
Cryptographic Hash of being engaged in is that Current transaction selects write queue.
7. method as claimed in claim 6, it is characterised in that：The cryptographic Hash of the affairs is by first master of each affairs
Key is converted into character string, then obtained by Hash calculation.
8. the method as described in claim 1, it is characterised in that：After affairs write step, this method also includes delete step,
Delete the transaction object for having been written into standby database.
9. the synchronous device in a kind of master/slave data storehouse, it is characterised in that the device includes：
Dispatch unit, for successively distributing the mark of each affairs read from primary database or each affairs to write queue；
Writing unit, for using multiple thread parallels of writing by corresponding to the affairs or mark respectively write in write queue corresponding to thread
The standby database of affairs write-in；The write queue is used to depositing the mark of the affairs that dispatch unit distributes or affairs, a write queue
Corresponding one is write thread；
Wherein, distribute affairs or write affairs premise be：The affairs obtain the virtual record corresponding to each record of the affairs
Lock；Wherein, the virtual record lock that the described affairs are obtained corresponding to each record of the affairs represents that the affairs lock the affairs
Each record.
10. device as claimed in claim 9, it is characterised in that：Described device also includes：
Pretreatment unit, for extracting the major key of each affairs successively, list of primary keys is created for each affairs；It is additionally operable to according to each
The list of primary keys of individual affairs, the pointer of each affairs is put into major key transaction queues corresponding to each major key of the affairs successively,
The affairs that the pointer of affairs is in head of the queue obtain virtual record lock corresponding to the major key；
Several major key transaction queues, for storing the mark of each affairs；
The dispatch unit or writing unit, when the virtual record lock sum that affairs obtain is equal to master in the list of primary keys of the affairs
During the sum of key, affairs or write-in affairs are distributed.
11. device as claimed in claim 10, it is characterised in that the dispatch unit, be additionally operable to deletion and have been written into standby number
According to the Transaction Identifier of major key transaction queues head of the queue corresponding to the affairs in storehouse, virtual record is locked to the next section for passing to the queue
Point.
12. device as claimed in claim 10, it is characterised in that：The dispatch unit realizes that pretreatment is single by distributing thread
Member is additionally operable to create virtual record lock counter for each affairs, and the affairs often obtain a virtual record lock, the virtual record
Lock counter adds 1, and the dispatch unit or writing unit lock counter according to the virtual record and judge the virtual of current event
Record lock sum is equal to the sum of major key in the list of primary keys of the affairs.
13. device as claimed in claim 9, it is characterised in that：Dispatch unit is according to the most short principle of write queue or affairs Hash
It is worth and selects write queue for Current transaction.
14. device as claimed in claim 10, it is characterised in that：The affairs cryptographic Hash is by first master of each affairs
Key is converted into character string, then obtained by Hash calculation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210564590.1A CN103885986B (en) | 2012-12-21 | 2012-12-21 | The synchronous method and apparatus in master/slave data storehouse |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210564590.1A CN103885986B (en) | 2012-12-21 | 2012-12-21 | The synchronous method and apparatus in master/slave data storehouse |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103885986A CN103885986A (en) | 2014-06-25 |
| CN103885986B true CN103885986B (en) | 2017-12-19 |
Family
ID=50954881
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210564590.1A Active CN103885986B (en) | 2012-12-21 | 2012-12-21 | The synchronous method and apparatus in master/slave data storehouse |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103885986B (en) |
Families Citing this family (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105574026B (en) * | 2014-10-15 | 2019-12-13 | 腾讯科技（深圳）有限公司 | Method and device for supporting transaction of non-relational database |
| CN104503986A (en) * | 2014-12-02 | 2015-04-08 | 国家电网公司 | User information input method and equipment |
| CN105893386B (en) * | 2015-01-04 | 2021-08-06 | 伊姆西Ip控股有限责任公司 | Method and apparatus for processing transactions in a synchronous replication system |
| CN105989123A (en) * | 2015-02-13 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Data synchronization method, apparatus and system |
| CN104699541B (en) * | 2015-03-30 | 2018-07-10 | 北京奇虎科技有限公司 | Method, apparatus, data transfer components and the system of synchrodata |
| CN106802897A (en) * | 2015-11-26 | 2017-06-06 | 北京国双科技有限公司 | Lookup table data synchronous method and device |
| CN106815094B (en) * | 2015-12-02 | 2020-12-11 | 阿里巴巴集团控股有限公司 | Method and equipment for realizing transaction submission in master-slave synchronization mode |
| CN106855860A (en) * | 2015-12-08 | 2017-06-16 | 阿里巴巴集团控股有限公司 | Method and apparatus for creating master/slave data storehouse |
| CN106909554B (en) * | 2015-12-22 | 2020-08-04 | 亿阳信通股份有限公司 | Method and device for loading database text table data |
| CN105786959A (en) * | 2016-01-11 | 2016-07-20 | 北京京东尚科信息技术有限公司 | Synchronization method and device of primary database and spare database |
| CN107783975B (en) * | 2016-08-24 | 2021-02-26 | 北京京东尚科信息技术有限公司 | Method and device for synchronous processing of distributed databases |
| CN107918620B (en) | 2016-10-10 | 2022-04-19 | 阿里巴巴集团控股有限公司 | Database writing method and device and electronic equipment |
| CN106776790A (en) * | 2016-11-11 | 2017-05-31 | 北京奇虎科技有限公司 | Concurrent master-slave synchronisation method and device based on token |
| CN108073658B (en) * | 2016-11-16 | 2020-06-12 | 中国移动通信集团广西有限公司 | Data synchronization system and method |
| CN106874341B (en) * | 2016-12-23 | 2022-04-05 | 中科星图股份有限公司 | Database synchronization method |
| CN108304406B (en) * | 2017-01-12 | 2022-10-25 | 阿里云计算有限公司 | Database reading and writing method and device and electronic equipment |
| CN108694199A (en) * | 2017-04-10 | 2018-10-23 | 北京京东尚科信息技术有限公司 | Data synchronization unit, method, storage medium and electronic equipment |
| CN107958023A (en) * | 2017-11-06 | 2018-04-24 | 北京华宇信息技术有限公司 | Method of data synchronization, data synchronization unit and computer-readable recording medium |
| CN109446268A (en) * | 2018-10-09 | 2019-03-08 | 联动优势科技有限公司 | A kind of method of data synchronization and device |
| CN109299136B (en) * | 2018-11-27 | 2021-11-09 | 佛山科学技术学院 | Real-time synchronization method and device for intelligently manufactured database resource pool |
| CN109656936A (en) * | 2018-11-30 | 2019-04-19 | 金蝶软件（中国）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN110134500A (en) * | 2019-04-15 | 2019-08-16 | 平安普惠企业管理有限公司 | Method of data synchronization, device, equipment and storage medium based on multithreading |
| CN111198872B (en) * | 2020-01-06 | 2021-06-11 | 中科驭数（北京）科技有限公司 | Method and device for processing transactions by database |
| US11947427B2 (en) * | 2020-02-26 | 2024-04-02 | EMC IP Holding Company LLC | Method, electronic device, and computer program product for storage management using blockchain |
| CN111858626A (en) * | 2020-06-04 | 2020-10-30 | 武汉达梦数据库有限公司 | Data synchronization method and device based on parallel execution |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101394641A (en) * | 2007-09-18 | 2009-03-25 | 中兴通讯股份有限公司 | Main standby machine switching method oriented to user data |
| CN101706795A (en) * | 2009-11-30 | 2010-05-12 | 上海世范软件技术有限公司 | Method for synchronizing data of database in active/standby server |
| CN102081611A (en) * | 2009-11-26 | 2011-06-01 | 中兴通讯股份有限公司 | Method and device for synchronizing databases of master network management system and standby network management system |
| CN102831156A (en) * | 2012-06-29 | 2012-12-19 | 浙江大学 | Distributed transaction processing method on cloud computing platform |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8768902B2 (en) * | 2010-06-11 | 2014-07-01 | Microsoft Corporation | Unified concurrent changes to data, schema, and application |
-
2012
- 2012-12-21 CN CN201210564590.1A patent/CN103885986B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101394641A (en) * | 2007-09-18 | 2009-03-25 | 中兴通讯股份有限公司 | Main standby machine switching method oriented to user data |
| CN102081611A (en) * | 2009-11-26 | 2011-06-01 | 中兴通讯股份有限公司 | Method and device for synchronizing databases of master network management system and standby network management system |
| CN101706795A (en) * | 2009-11-30 | 2010-05-12 | 上海世范软件技术有限公司 | Method for synchronizing data of database in active/standby server |
| CN102831156A (en) * | 2012-06-29 | 2012-12-19 | 浙江大学 | Distributed transaction processing method on cloud computing platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103885986A (en) | 2014-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103885986B (en) | The synchronous method and apparatus in master/slave data storehouse | |
| US20230100223A1 (en) | Transaction processing method and apparatus, computer device, and storage medium | |
| KR101959153B1 (en) | System for efficient processing of transaction requests related to an account in a database | |
| US20150172412A1 (en) | Managing dependencies between operations in a distributed system | |
| CN102640151A (en) | High throughput, reliable replication of transformed data in information systems | |
| CN103886109B (en) | Method and device for realizing row lock of database | |
| CN103246749A (en) | Matrix data base system for distributed computing and query method thereof | |
| JP7438603B2 (en) | Transaction processing methods, apparatus, computer devices and computer programs | |
| CN106663062A (en) | System and method for providing distributed transaction lock in transactional middleware machine environment | |
| CN107113341A (en) | The system of the high-throughput processing of affairs in the Distributed Relation Database Management System divided for data | |
| CN103440246A (en) | Intermediate result data sequencing method and system for MapReduce | |
| CN109643310A (en) | System and method for fast resampling in database | |
| CN104750720A (en) | Method for achieving high-performance data processing under multithread concurrent access environment | |
| Agrawal et al. | Managing geo-replicated data in multi-datacenters | |
| Waqas et al. | Transaction management techniques and practices in current cloud computing environments: A survey | |
| CN113076304A (en) | Distributed version management method, device and system | |
| JP5331050B2 (en) | Data synchronization system, data synchronization method, information processing apparatus, information processing method, and program | |
| CN103995827B (en) | High-performance sort method in MapReduce Computational frames | |
| Nie et al. | Angel-ptm: A scalable and economical large-scale pre-training system in tencent | |
| CN107276914A (en) | The method of self-service resource allocation scheduling based on CMDB | |
| WO2023065868A1 (en) | Transaction execution method and apparatus, and computer device and storage medium | |
| Perwej et al. | An extensive investigate the mapreduce technology | |
| Sang et al. | A fast implementation of parallel discrete-event simulation on GPGPU | |
| KR102386921B1 (en) | Computer-readable recording medium that recorded block data | |
| CN106708948A (en) | Method and apparatus for implementing transaction operation with primary key uniqueness |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |