CN109165258A - A kind of method of data synchronization and device - Google Patents
A kind of method of data synchronization and device Download PDFInfo
- Publication number
- CN109165258A CN109165258A CN201810919871.1A CN201810919871A CN109165258A CN 109165258 A CN109165258 A CN 109165258A CN 201810919871 A CN201810919871 A CN 201810919871A CN 109165258 A CN109165258 A CN 109165258A
- Authority
- CN
- China
- Prior art keywords
- database
- node
- statement
- broadcast
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/542—Event management; Broadcasting; Multicasting; Notifications
Abstract
The invention discloses a kind of method of data synchronization and device to include: configuration database middleware and make database middleware enabling table broadcast mechanism；Database middleware is set to receive the database statement of data manipulation language；Global table logic whether is needed to be implemented according to configuration determination database statement；When database statement needs to be implemented global table logic, database statement table is broadcast to all database nodes using table broadcast mechanism；When database statement does not need to execute global table logic, all database nodes are sent by database statement and execute database statement.Method of data synchronization proposed by the present invention can be synchronous for disparate databases node or different types of database node progress data with device, maintains the data consistency of global table in database.
Description
Technical field
The present invention relates to database fields, more specifically, particularly relating to a kind of method of data synchronization and device.
Background technique
Mycat is the distributed data base system of an open source, is the database middleware for realizing MySQL agreement.Before
End subscriber can regard it as a database broker, with MySQL client utility and command-line access；Rear end can be used
Primary (Native) agreement of MySQL is communicated with multiple MySQL servers, can also use JDBC agreement and most of mainstream datas
Library server communication.
Affairs in Mycat mainly include SQL not across fragment affairs and across the fragment affairs of SQL.For SQL not across fragment thing
Business, SQL statement only execute on a back end, and the db transaction mode of Mycat transaction mode and standard is complete at this time
Unanimously, it submits and rollback is no can guarantee strong consistency.Affairs across fragment for SQL, the SQL in affairs is at respective point first
On piece executes and returns status code, if the return code on some fragment is wrong (error), Mycat thinks that affairs fail, answers
With end rollback (Rollback) affairs, after Mycat receives rollback instruction, all fragments for being successively related in rollback affairs；If
The execution of all SQL in affairs all returns to the return code of successfully (OK), then application program submit affairs, from Mycat simultaneously to
The node being related in affairs sends the instruction (Commit) for submitting affairs, if some node malfunctions in COMMIT, Mycat
Again, there is part of nodes commit in commit after nodes can not be waited to restore and part of nodes does not have the case where commit.Due to
Second stage across fragment affairs not can guarantee strong consistency, therefore Mycat is a kind of weak XA transaction mode.
There are two types of typical tables in Mycat: one is according to certain given fragment rule, data being carried out horizontal cutting
Point, one big table horizontal segmentation at N number of small table, it is stored in rear end MySQL server or other different databases；It is another
Kind is global table, and Mycat receives the extraneous SQL statement sent, the SQL statement of DML (data manipulation language) is sent respectively
It is successively executed on to each database node.The DML operating process of Mycat global table is as shown in Figure 1.If continually to global table
It carries out additions and deletions and changes operation, due to the weak XA affair mechanism of Mycat, frequent operation global table may result in the hair of abnormal conditions
Raw: some user is modifying global table submission in a flash, some node malfunctions suddenly, it is possible that certain nodes are
Success is modified and the node data that malfunctions does not change, so that the situation that each node data is inconsistent.The problem makes entire database
Lose data consistency, be may be difficult to it is received.
Database middleware is made to cannot be guaranteed global table for the weak XA affair mechanism in the prior art across fragment affairs
Data consistency the problem of, there has been no effective solution schemes at present.
Summary of the invention
In view of this, the purpose of the embodiment of the present invention is to propose a kind of method of data synchronization and device, it can be for not
Data are carried out with database node or different types of database node to synchronize, and maintain the data of global table in database consistent
Property.
Based on above-mentioned purpose, the one side of the embodiment of the present invention provides a kind of method of data synchronization, is applied to database,
The following steps are included:
Configuration database middleware simultaneously makes database middleware enabling table broadcast mechanism；
Database middleware is set to receive the database statement of data manipulation language；
Global table logic whether is needed to be implemented according to configuration determination database statement；
When database statement needs to be implemented global table logic, database statement table is broadcast to institute using table broadcast mechanism
There is database node；
When database statement does not need to execute global table logic, all database nodes are sent simultaneously for database statement
Execute database statement.
In some embodiments, table broadcast mechanism includes:
A database host node is determined in all database nodes, and other database nodes are determined as database
From node；
Database middleware is set to send the broadcast about database statement to database host node；
Make database host node that database statement table is broadcast to all databases from node.
In some embodiments, make database host node that database statement table is broadcast to all databases from node packet
It includes:
Saving database host node includes the binary message file of database statement, and to all databases from node
Push binary message file；
Make all databases that binary message file be dumped to all databases by dedicated worker thread from node
From the respective relaying file of node；
Make database host node and all databases from node respectively according to binary message file and respective relaying text
Part executes database statement.
In some embodiments, binary message file is dumped into corresponding relaying text from node in each database
After in part, dedicated worker thread is recorded in binary message file；All numbers are recorded in binary message file
After all having carried out unloading from node according to library, database host node sends to all databases from node and executes database statement
Broadcast.
In some embodiments, when dedicated worker thread cannot read binary message file, corresponding data
Library enters suspend mode from node and waits new broadcast.
In some embodiments, dedicated worker thread binary message file is written in a serial fashion corresponding
Relay file.
In some embodiments, judge whether database statement needs to be implemented global table logic and include:
Judge whether the type of database statement is one of the following: insertion is updated, is deleted, and is then continued in this way；
Whether the Object table type for judging database statement is global table, is then continued in this way；
Judge whether database statement is configured to enable single node insertion, is that judgement needs to be implemented global table logic, no
Then determine not needing.
In some embodiments, configuration database middleware includes:
The table label for judging table type is added in configuration file into database middleware；
The single node insertion attribute for judging whether enabling table broadcast mechanism is added into table label.
In some embodiments, database middleware Mycat, database MySQL.
The another aspect of the embodiment of the present invention additionally provides a kind of data synchronization unit, including processor and memory, institute
It states memory and is stored with the executable instruction of processor, described instruction realizes the above method when executed.
The another aspect of the embodiment of the present invention additionally provides a kind of computer readable storage medium, computer-readable storage
Media storage has computer program, and above-mentioned method is executed when computer program is executed by processor.
The another aspect of the embodiment of the present invention, additionally provides a kind of computer program product, and computer program product includes
The calculation procedure being stored on computer readable storage medium, calculation procedure include instruction, when executed by the processor, are made
Computer executes the above method.
The present invention has following advantageous effects: method of data synchronization and device provided in an embodiment of the present invention pass through
Using enabling table broadcast mechanism, receive data manipulation language database statement, judge whether database statement needs to be implemented entirely
Office's table logic will count when using table broadcast mechanism database statement table being broadcast to all database nodes when needing, do not needed
All database nodes are sent to according to library sentence and execute the technical solution of database statement, can be directed to disparate databases node
Or different types of database node carries out data and synchronizes, and maintains the data consistency of global table in database.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
It obtains other drawings based on these drawings.
Fig. 1 is the DML operating process schematic diagram of Mycat global table in the prior art；
Fig. 2 is the flow diagram of the embodiment of method of data synchronization provided by the invention；
Fig. 3 is that the table of Mycat global table provided by the invention broadcasts flow diagram；
Fig. 4 is the judgement flow diagram that the table of Mycat global table provided by the invention is broadcasted；
Fig. 5 is the hardware knot of one embodiment of the computer equipment provided by the invention for executing the method for data synchronization
Structure schematic diagram.
Specific embodiment
To make the objectives, technical solutions, and advantages of the present invention clearer, below in conjunction with specific embodiment, and reference
The embodiment of the present invention is further described in attached drawing.
It should be noted that all statements for using " first " and " second " are for differentiation two in the embodiment of the present invention
The non-equal entity of a same names or non-equal parameter, it is seen that " first " " second " only for the convenience of statement, does not answer
It is interpreted as the restriction to the embodiment of the present invention, subsequent embodiment no longer illustrates this one by one.
Based on above-mentioned purpose, the first aspect of the embodiment of the present invention, disparate databases section can be directed to by proposing one kind
Point or different types of database node carry out the embodiment of the synchronous method of data.Fig. 2 shows be number provided by the invention
According to the flow diagram of the embodiment of synchronous method.
The method of data synchronization is optionally applied to database, comprising the following steps:
Step S201, configuration database middleware simultaneously make database middleware enabling table broadcast mechanism；
Step S203 makes database middleware receive the database statement of data manipulation language；
Whether step S205 needs to be implemented global table logic according to configuration determination database statement；
Step S207, when database statement needs to be implemented global table logic, using table broadcast mechanism by database statement
Table is broadcast to all database nodes；
Step S209 sends all numbers for database statement when database statement does not need to execute global table logic
According to library node and execute database statement.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, Ke Yitong
Computer program is crossed to instruct related hardware and complete, the program can be stored in a computer-readable storage medium,
The program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, the storage medium can for magnetic disk,
CD, read-only memory (ROM) or random access memory (RAM) etc..The embodiment of the computer program, Ke Yida
The effect identical or similar to corresponding aforementioned any means embodiment.
The embodiment of the present invention will be acted on by changing the global table logic of Mycat (database middleware) when DML is accessed
It is changed to the operation to host node in the operation of all nodes, then the data between each node are completed by table broadcast mechanism and are synchronized, is come
Realize the data consistency between each node.
In some embodiments, table broadcast mechanism includes:
A database host node is determined in all database nodes, and other database nodes are determined as database
From node；
Database middleware is set to send the broadcast about database statement to database host node；
Make database host node that database statement table is broadcast to all databases from node.
In some embodiments, make database host node that database statement table is broadcast to all databases from node packet
It includes:
Saving database host node includes the binary message file of database statement, and to all databases from node
Push binary message file；
Make all databases that binary message file be dumped to all databases by dedicated worker thread from node
From the respective relaying file of node；
Make database host node and all databases from node respectively according to binary message file and respective relaying text
Part executes database statement.
In addition, disclosed method is also implemented as the computer program executed by CPU according to embodiments of the present invention, it should
Computer program may be stored in a computer readable storage medium.When the computer program is executed by CPU, the present invention is executed
The above-mentioned function of being limited in method disclosed in embodiment.Above method step and system unit also can use controller and
For storing so that controller realizes that the computer readable storage medium of the computer program of above-mentioned steps or Elementary Function is realized.
As shown in figure 3, in embodiments of the present invention, realizing that table broadcast mechanism includes three steps:
(1) user sends DML request, and host receives order, sends and broadcasts to host node.Host node receives message,
By Message Record to binary file, and other are transmitted the message to from node；
(2) message is read from node, and message is written to the relaying log of oneself.
(3) event in relaying log is reformed from node, and changes the data of oneself.
In addition, typically, the embodiment of the present invention discloses described database middleware and node etc. can be whole for various electronics
End equipment, such as mobile phone, personal digital assistant (PDA), tablet computer (PAD), smart television etc., are also possible to large-scale terminal and set
It is standby, such as server, therefore protection scope disclosed by the embodiments of the present invention should not limit as certain certain types of device, set
It is standby.The embodiment of the present invention discloses the client and can be is answered with the combining form of electronic hardware, computer software or both
For in any one of the above electric terminal equipment.
In some embodiments, binary message file is dumped into corresponding relaying text from node in each database
After in part, dedicated worker thread is recorded in binary message file；All numbers are recorded in binary message file
After all having carried out unloading from node according to library, database host node sends to all databases from node and executes database statement
Broadcast.
Host is broadcasted after receiving request to host node transmission, and host node receives message and simultaneously records binary file.Each
Before affairs more new data is completed, host node records these changes in binary file.MySQL thread serially writes affairs
Enter binary file, after the completion of event write-in, host node notifies storage engines to submit affairs and send to other from node wide
It broadcasts.
It host node as described herein and can be volatile memory or nonvolatile memory from node, or can wrap
Include both volatile memory and nonvolatile memory.As an example and not restrictive, nonvolatile memory can wrap
Include read-only memory (ROM), programming ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM
(EEPROM) or flash memory.Volatile memory may include random access memory (RAM), which can serve as outer
Portion's cache memory.As an example and not restrictive, RAM can be obtained in a variety of forms, such as synchronous random access memory
(DRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate SDRAM (DDR SDRAM), enhancing SDRAM
(ESDRAM), synchronization link DRAM (SLDRAM) and directly Rambus RAM (DRRAM).The storage of disclosed aspect is set
The standby memory for being intended to including but not limited to these and other suitable type.
In some embodiments, when dedicated worker thread cannot read binary message file, corresponding data
Library enters suspend mode from node and waits new broadcast.In some embodiments, dedicated worker thread in a serial fashion will
Corresponding relaying file is written in binary message file.
Message is received from node, the binary file of host node is copied to the relaying log of oneself.First from node
Start a worker thread (I/O thread), I/O thread opens a common connection on the primary node, then starts binary system
Unloading.Binary system loading program reads message event from the binary file of host node, if not new message, it can sleep
It sleeps and host node is waited to send new broadcast.SQL from thread from relaying log read message event, and reset event therein with
The data from node are updated, keep it consistent with the data in host node.
In some embodiments, judge whether database statement needs to be implemented global table logic and include:
Judge whether the type of database statement is one of the following: insertion is updated, is deleted, and is to continue next step；
Whether the Object table type for judging database statement is global table, is to continue next step；
Judge whether database statement is configured to enable single node insertion, is that judgement needs to be implemented global table logic, no
Then determine not needing.
As shown in figure 4, the table broadcast mechanism in Mycat in the execution process Mycat of table broadcast mechanism is to be directed to the overall situation
The DML of table is operated and is proposed, and only in writeOneNode=true, its function can just come into force, it is therefore desirable to be increased
New SQL executes logic.
As shown in figure 4, Mycat carries out following judgement to the SQL statement received:
(1) whether the type of SQL statement is insert, update or delete；
(2) whether the table being inserted into is global table；
It (3) whether is single node write-in, i.e. writeOneNode=true.
Only meeting simultaneously there are three condition makes, and original multinode insertions function is just revised as single node insertion, is enabled
Table broadcast capability.
Various illustrative logical blocks, module, circuit and algorithm steps in conjunction with described in disclosure herein can be implemented
For the combination of electronic hardware, computer software or both.In order to clearly demonstrate this interchangeability of hardware and software,
General description has been carried out to it with regard to the function of various exemplary components, square, module, circuit and step.This function is
Software is implemented as also to be implemented as hardware depending on concrete application and be applied to the design constraint of whole system.This field
Technical staff can realize the function in various ways for every kind of concrete application, but determine should not be by for this realization
It is construed to lead to be detached from range disclosed by the embodiments of the present invention.
In some embodiments, configuration database middleware includes:
The table label for judging table type is added in configuration file into database middleware；
The single node insertion attribute for judging whether enabling table broadcast mechanism is added into table label.
Table broadcast mechanism in Mycat is for global table.Configuring Mycat is to have increased an attribute newly for global table,
Change the execution logic of original SQL statement when the attribute value is true.Mycat receives DML order and sends extensively to host node
It broadcasts, host node, which executes, orders and save binary message file simultaneously to from node PUSH message, and it is same to receive broadcast from node
Walk the data of host node.When there is node exception when modifying data in the Mycat that the embodiment of the present invention realizes table broadcast mechanism
Journal file, the synchrodata that storage message can be actively read after releasing extremely, ensure that strong consistency.
Specifically, configuration file is read first when Mycat starts, judged according to the setting information of wherein table label
The type of table.Increase an attribute " writeOneNode " in the table label of configuration file schema.xml as enabling
The mark of table broadcast mechanism just enables table broadcast capability when the type of logical table is global table i.e. " type=global ".Newly
The attribute of increasing is Boolean type, and default value is " false ".As " writeOneNode=true ", the SQL of filtering DML operation
Sentence acts on host node by table broadcast；As " writeOneNode=false ", the SQL statement of DML operation is according to original
Beginning logical action is in all nodes.
When on the other hand, such as by java application accessing XML, Java object is first converted to XML file
(marshall), then by the content transformation in XML file at corresponding Java object (unmarshal), which needs to use
JAXB (the Java architecture of XML binding) explains to mark.Therefore it needs to modify to explain class Schemas.java, inside it
The attribute setting that " writeOneNode " is added in class, make content unmarshal in configuration file " schema.xml " at
It include the corresponding information of " writeOneNode " field when the object of Table type.Initialize the class of XML configuration file
In XmlToYaml.java, increase the function of initializing to parameter " writeOneNode ", the load configuration text in startup program
Part initializes the relevant parameter being arranged in configuration file, by parsing XML configuration file in each interior of Schemas
This class object is instantiated in portion's static class, the Key-value key-value pair configured required for finally being stored again by HashMap,
This class instance object is externally provided.
The step of method in conjunction with described in disclosure herein or algorithm, can be directly contained in hardware, be held by processor
In capable software module or in combination of the two.Software module may reside within RAM memory, flash memory, ROM storage
Device, eprom memory, eeprom memory, register, hard disk, removable disk, CD-ROM or known in the art it is any its
In the storage medium of its form.Illustrative storage medium is coupled to processor, enables a processor to from the storage medium
Information is written to the storage medium in middle reading information.In an alternative, the storage medium can be with processor collection
At together.Pocessor and storage media may reside in ASIC.ASIC may reside in user terminal.It is replaced at one
In scheme, it is resident in the user terminal that pocessor and storage media can be used as discrete assembly.
In some embodiments, database middleware Mycat, database MySQL.
From above-described embodiment as can be seen that method of data synchronization provided in an embodiment of the present invention, by using enabling, table is wide
It broadcasts mechanism, the database statement for receiving data manipulation language, judge whether database statement needs to be implemented global table logic, needs
When send database statement to when database statement table being broadcast to all database nodes, do not needed using table broadcast mechanism
All database nodes and the technical solution for executing database statement can be directed to disparate databases node or different types of number
Data are carried out according to library node to synchronize, and are changed Mycat when DML is accessed to the execution logic of global table, be will act on all nodes
Operation to be changed to the data completing each node by table broadcast mechanism again to the operation of host node synchronous, to maintain database
The data consistency of middle global table.
It is important to note that each step in each embodiment of above-mentioned method of data synchronization can be handed over mutually
It pitches, replace, increase, delete, therefore, should also be as in method of data synchronization of these reasonable permutation and combination transformation belongs to the present invention
Protection scope, and protection scope of the present invention should not be confined on the embodiment.
Based on above-mentioned purpose, the second aspect of the embodiment of the present invention, disparate databases section can be directed to by proposing one kind
Point or different types of database node carry out the embodiment of the synchronous device of data.The data synchronization unit has used above-mentioned
Method of data synchronization.
Various illustrative logical blocks, module and circuit, which can use, in conjunction with described in disclosure herein is designed to
The following component of function described here is executed to realize or execute: general processor, digital signal processor (DSP), dedicated collection
At circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, divide
Any combination of vertical hardware component or these components.General processor can be microprocessor, but alternatively, processing
Device can be any conventional processors, controller, microcontroller or state machine.Processor also may be implemented as calculating equipment
Combination, for example, the combination of DSP and microprocessor, multi-microprocessor, one or more microprocessors combination DSP and/or any
Other this configurations.
Data synchronization unit provided in an embodiment of the present invention by using enabling table broadcast mechanism, receives data manipulation language
The database statement of speech judges will count when whether database statement needs to be implemented global table logic, needs using table broadcast mechanism
All database nodes are sent by database statement when being broadcast to all database nodes according to library statement list, do not need and are executed
It is same can to carry out data for disparate databases node or different types of database node for the technical solution of database statement
Step changes Mycat when DML is accessed to the execution logic of global table, and the operation that will act on all nodes is changed to host node
Operation again by table broadcast mechanism complete the data between each node synchronization, to maintain the data of global table in database consistent
Property.
It is important to note that the embodiment of above-mentioned data synchronization unit uses the implementation of the method for data synchronization
Example illustrates the course of work of each module, and those skilled in the art can be it is readily conceivable that by these module applications to institute
It states in the other embodiments of method of data synchronization.Certainly, due to each step in the method for data synchronization embodiment
To intersect, replace, increase, delete, therefore, these reasonable permutation and combination transformation are also answered in the data synchronization unit
When belonging to the scope of protection of the present invention, and protection scope of the present invention should not be confined on the embodiment.
The data synchronization unit includes memory, at least one processor, and memory, which is stored with, to be transported on a processor
Capable computer program, processor execute any one of the above method when executing program.
As shown in figure 5, the hardware structural diagram of one embodiment for data synchronization unit provided by the invention.
It include several processors 501 and several in the computer equipment by taking computer equipment as shown in Figure 5 as an example
Memory 502, and can also include: input unit 503 and output device 504.
Processor 501, memory 502, input unit 503 and output device 504 can pass through bus or other modes
It connects, in Fig. 5 for being connected by bus.
Memory 502 is used as a kind of non-volatile computer readable storage medium storing program for executing, can be used for storing non-volatile software journey
Sequence, non-volatile computer executable program and module, as the method for data synchronization in the embodiment of the present application is corresponding
Program instruction/module.Non-volatile software program, instruction and the mould that processor 501 is stored in memory 502 by operation
Block realizes the data side of synchronization of above method embodiment thereby executing the various function application and data processing of server
Method.
Memory 502 may include storing program area and storage data area, wherein storing program area can store operation system
Application program required for system, at least one function；Storage data area can be stored to be created according to using for data synchronization unit
Data etc..In addition, memory 502 may include high-speed random access memory, it can also include nonvolatile memory, example
Such as at least one disk memory, flush memory device or other non-volatile solid state memory parts.In some embodiments, it deposits
Optional reservoir 502 includes the memory remotely located relative to processor 501, these remote memories can pass through network connection
To local module.The example of above-mentioned network include but is not limited to internet, intranet, local area network, mobile radio communication and its
Combination.
Input unit 503 can receive the number or character information of input, and generates and set with the user of data synchronization unit
It sets and the related key signals of function control inputs.Output device 504 may include that display screen etc. shows equipment.
Corresponding program instruction/the module of one or more of method of data synchronization is stored in the memory 502,
When being executed by the processor 501, the method for data synchronization in above-mentioned any means embodiment is executed.
Any one embodiment of the computer equipment for executing the method for data synchronization, can achieve and be corresponding to it
The identical or similar effect of aforementioned any means embodiment.
Based on above-mentioned purpose, the third aspect of the embodiment of the present invention proposes a kind of computer readable storage medium, institute
Stating computer-readable recording medium storage has computer executable instructions, which can be performed above-mentioned any side
The data synchronization unit in method of data synchronization and the above-mentioned any device embodiment of realization in method embodiment.The computer can
The embodiment for reading storage medium, it is identical or similar as Installation practice to can achieve corresponding aforementioned any means
Effect.
Based on above-mentioned purpose, the 4th aspect of the embodiment of the present invention proposes a kind of computer program product, the calculating
Machine program product includes the calculation procedure being stored on computer readable storage medium, which includes instruction, when this
Instruction is when being computer-executed, and makes that the computer executes the method for data synchronization in above-mentioned any means embodiment and realization is above-mentioned
Data synchronization unit in any device embodiment.The embodiment of the computer program product can achieve corresponding
The aforementioned any means effect identical or similar with Installation practice.
It is exemplary embodiment disclosed by the invention above, it should be noted that in the sheet limited without departing substantially from claim
Under the premise of inventive embodiments scope of disclosure, it may be many modifications and modify.According to open embodiment described herein
The function of claim to a method, step and/or movement be not required to the execution of any particular order.In addition, although the present invention is implemented
Element disclosed in example can be described or be required in the form of individual, but be unless explicitly limited odd number, it is understood that be multiple.
It should be understood that it is used in the present context, unless the context clearly supports exceptions, singular " one
It is a " it is intended to also include plural form.It is to be further understood that "and/or" used herein refers to including one or one
Any and all possible combinations of a above project listed in association.The embodiment of the present invention discloses embodiment sequence number
Description, does not represent the advantages or disadvantages of the embodiments.
It should be understood by those ordinary skilled in the art that: the discussion of any of the above embodiment is exemplary only, not
It is intended to imply that range disclosed by the embodiments of the present invention (including claim) is limited to these examples；In the think of of the embodiment of the present invention
Under road, it can also be combined between the technical characteristic in above embodiments or different embodiments, and exist as described above
Many other variations of the different aspect of the embodiment of the present invention, for simplicity, they are not provided in details.Therefore, all at this
Within the spirit and principle of inventive embodiments, any omission, modification, equivalent replacement, improvement for being made etc. should be included in this hair
Within the protection scope of bright embodiment.
Claims (10)
1. a kind of method of data synchronization, which is characterized in that be applied to database, comprising the following steps:
Configuration database middleware simultaneously makes the database middleware enabling table broadcast mechanism；
The database middleware is set to receive the database statement of data manipulation language；
Global table logic whether is needed to be implemented according to database statement described in the configuration determination；
When the database statement needs to be implemented the global table logic, using the table broadcast mechanism by the database language
Sentence table is broadcast to all database nodes；
When the database statement does not need to execute the global table logic, send the database statement to described all
Database node simultaneously executes the database statement.
2. the method according to claim 1, wherein the table broadcast mechanism includes:
A database host node is determined in all database nodes, and other database nodes are determined as database
From node；
The database middleware is set to send the broadcast about the database statement to the database host node；
Make the database host node that the database statement table is broadcast to all databases from node.
3. according to the method described in claim 2, it is characterized in that, making the database host node by the database statement table
All databases, which are broadcast to, from node includes:
Saving the database host node includes the binary message file of the database statement, and to all data
Library pushes the binary message file from node；
Make all databases that the binary message file be dumped to all institutes by dedicated worker thread from node
Database is stated from the respective relaying file of node；
Make the database host node and all databases from node respectively according to the binary message file and respectively
The relaying file execute the database statement.
4. according to the method described in claim 3, it is characterized in that, the binary system is disappeared from node in each database
Breath file dump in the corresponding relaying file after, the dedicated worker thread in the binary message file into
Row record；It is recorded in the binary message file after all databases have all carried out unloading from node, the number
The broadcast for executing the database statement is sent to all databases from node according to library host node.
5. according to the method described in claim 3, it is characterized in that, the dedicated worker thread cannot read described two into
When message file processed, the corresponding database enters suspend mode from node and waits new broadcast.
6. according to the method described in claim 3, it is characterized in that, the dedicated worker thread in a serial fashion will be described
The corresponding relaying file is written in binary message file.
7. the method according to claim 1, wherein it is described complete to judge whether the database statement needs to be implemented
Office table logic include:
Judge whether the type of the database statement is one of the following: insertion is updated, is deleted, and is then continued in this way；
Whether the Object table type for judging the database statement is global table, is then continued in this way；
Judge whether the database statement is configured to enable single node insertion, then determines that needing to be implemented the global table patrols in this way
Volume, otherwise determine not needing.
8. the method according to claim 1, wherein configuring the database middleware and including:
The table label for judging table type is added in configuration file into the database middleware；
The single node insertion attribute for judging whether to enable the table broadcast mechanism is added into the table label.
9. method according to claim 1 to 8, which is characterized in that the database middleware is Mycat, institute
Stating database is MySQL.
10. a kind of data synchronization unit, which is characterized in that including processor and memory, the memory is stored with processor
Executable instruction, described instruction realize method as described in any one of claims 1-9 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810919871.1A CN109165258A (en) | 2018-08-14 | 2018-08-14 | A kind of method of data synchronization and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810919871.1A CN109165258A (en) | 2018-08-14 | 2018-08-14 | A kind of method of data synchronization and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109165258A true CN109165258A (en) | 2019-01-08 |
Family
ID=64895465
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810919871.1A Pending CN109165258A (en) | 2018-08-14 | 2018-08-14 | A kind of method of data synchronization and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109165258A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110413687A (en) * | 2019-05-09 | 2019-11-05 | 国网冀北电力有限公司 | The distributed transaction fault handling method and relevant device of verification are mutually demonstrate,proved based on node |
| CN111104409A (en) * | 2019-12-25 | 2020-05-05 | 上海米哈游天命科技有限公司 | Database processing method and device, storage medium and electronic equipment |
| CN111459882A (en) * | 2020-03-30 | 2020-07-28 | 北京百度网讯科技有限公司 | Namespace transaction processing method and device of distributed file system |
| CN113434500A (en) * | 2019-08-20 | 2021-09-24 | 北京奥星贝斯科技有限公司 | Table connection method, device, distributed database system, server and medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9317372B1 (en) * | 2005-04-13 | 2016-04-19 | Progress Software Corporation | Dynamic membership management in a distributed system |
| CN106126600A (en) * | 2016-06-21 | 2016-11-16 | 浪潮电子信息产业股份有限公司 | A kind of perform the method for read and write abruption, Apparatus and system |
| CN106815097A (en) * | 2017-01-18 | 2017-06-09 | 北京许继电气有限公司 | Database disaster tolerance system and method |
| US20180004801A1 (en) * | 2013-05-13 | 2018-01-04 | Amazon Technologies, Inc. | Transaction ordering |
-
2018
- 2018-08-14 CN CN201810919871.1A patent/CN109165258A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9317372B1 (en) * | 2005-04-13 | 2016-04-19 | Progress Software Corporation | Dynamic membership management in a distributed system |
| US20180004801A1 (en) * | 2013-05-13 | 2018-01-04 | Amazon Technologies, Inc. | Transaction ordering |
| CN106126600A (en) * | 2016-06-21 | 2016-11-16 | 浪潮电子信息产业股份有限公司 | A kind of perform the method for read and write abruption, Apparatus and system |
| CN106815097A (en) * | 2017-01-18 | 2017-06-09 | 北京许继电气有限公司 | Database disaster tolerance system and method |
Non-Patent Citations (1)
| Title |
|---|
| 王锦、梁正和、王法强: "表广播机制在MyCat中的实现", 《计算机技术与发展》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110413687A (en) * | 2019-05-09 | 2019-11-05 | 国网冀北电力有限公司 | The distributed transaction fault handling method and relevant device of verification are mutually demonstrate,proved based on node |
| CN113434500A (en) * | 2019-08-20 | 2021-09-24 | 北京奥星贝斯科技有限公司 | Table connection method, device, distributed database system, server and medium |
| CN111104409A (en) * | 2019-12-25 | 2020-05-05 | 上海米哈游天命科技有限公司 | Database processing method and device, storage medium and electronic equipment |
| CN111459882A (en) * | 2020-03-30 | 2020-07-28 | 北京百度网讯科技有限公司 | Namespace transaction processing method and device of distributed file system |
| CN111459882B (en) * | 2020-03-30 | 2023-08-29 | 北京百度网讯科技有限公司 | Namespace transaction processing method and device for distributed file system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109165258A (en) | A kind of method of data synchronization and device | |
| EP4254183A1 (en) | Transaction processing method and apparatus, computer device, and storage medium | |
| US8131670B2 (en) | Techniques to cross-synchronize data | |
| US8364656B2 (en) | Method and system for implementing multiuser cached parameterized cells | |
| CN109710388B (en) | Data reading method and device, electronic equipment and storage medium | |
| CN113396407A (en) | System and method for augmenting database applications using blockchain techniques | |
| EP2797013B1 (en) | Database update execution according to power management schemes | |
| WO2021184761A1 (en) | Data access method and apparatus, and data storage method and device | |
| US20110289514A1 (en) | Sharing and synchronization of objects | |
| CN109997118A (en) | The method of mass data is consistently stored with ultrahigh speed in persistent storage systems | |
| CN110188114A (en) | A kind of optimization method of data manipulation, device, system, equipment and storage medium | |
| JP2008287548A (en) | Control method for database and program | |
| US20230110826A1 (en) | Log execution method and apparatus, computer device and storage medium | |
| WO2022213526A1 (en) | Transaction processing method, distributed database system, cluster, and medium | |
| WO2023077971A1 (en) | Transaction processing method and apparatus, and computing device and storage medium | |
| CN112612850A (en) | Data synchronization method and device | |
| CN113438275B (en) | Data migration method and device, storage medium and data migration equipment | |
| CN113448977A (en) | Distributed cache updating method and device based on database Binlog | |
| CN115495495A (en) | Transaction processing method, distributed database system, cluster and medium | |
| CN112905613A (en) | Data bidirectional synchronization method and device for heterogeneous database | |
| CN112612647B (en) | Log parallel replay method, device, equipment and storage medium | |
| CN114579604B (en) | Database transaction implementation method and system of application layer | |
| CN111881212A (en) | Method, device, equipment and storage medium for executing check point operation | |
| CN109857837A (en) | A kind of dictionary loading method and device that can customize | |
| CN110096389A (en) | A kind of starting method, apparatus, equipment and the storage medium of database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190108 |