CN109885617A - The method of data synchronization and device of Distributed Heterogeneous Database system - Google Patents
The method of data synchronization and device of Distributed Heterogeneous Database system Download PDFInfo
- Publication number
- CN109885617A CN109885617A CN201910084472.2A CN201910084472A CN109885617A CN 109885617 A CN109885617 A CN 109885617A CN 201910084472 A CN201910084472 A CN 201910084472A CN 109885617 A CN109885617 A CN 109885617A
- Authority
- CN
- China
- Prior art keywords
- data
- message
- operation information
- operational order
- operating position
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention discloses the method for data synchronization and device of a kind of Distributed Heterogeneous Database system, wherein method includes: to obtain one or more groups of operational orders and operation information, the operational order and operation information are sent to first database cluster, the operational order includes: insertion data, more new data or deletes data, and the operation information includes: operating position and data value；First database cluster is received according to the operating result returned after the data at every group of operational order and operation information modification respective operations position；According to each operating result, corresponding operational order and operation information are packaged as message to issue, the message is used to carry out data to operating position corresponding in the second data-base cluster synchronous, avoid between distinct type data-base system can not perception data in time variation, effectively improve the accuracy rate of data query and analysis.
Description
Technical field
The present invention relates to the method for data synchronization of computer application field more particularly to Distributed Heterogeneous Database system and
Device.
Background technique
In today of information data explosive growth, big data has become a grand strategy assets of enterprise.Currently, big
Data processing is broadly divided into two classes: the first kind is with on-line analytical processing (On-Line Analytical Processing, abbreviation
It is representative for OLAP), excavation, complicated analytical calculation for mass data；Second class is with Transaction Processing (On-line
Transaction processing, referred to as OLTP) be representative, including conventional transaction type operation and mass data it is real-time
Access.In order to meet the needs of above-mentioned two classes data processing simultaneously, large enterprise, can be according to business field when designing application system
Scape uses different types of distributed data base system.Therefore, same a data distribution it is different physically, table
It is now distributed heterogeneous database system environment.
In existing Distributed Heterogeneous Database system, information is isolated each other between different types of Database Systems,
When local data table content changes, strange land tables of data is difficult to perceive variation in time.For example, user is in OLTP number of types
After being inserted into a data according to library, when being analyzed and processed using OLAP types of database to same batch of data, due to failing in time
It perceives user and has passed through OLTP types of database and data are modified, carried out at analysis using OLAP types of database
When reason, using original unmodified data, this will lead to data analysis inaccuracy；In addition, under the premise of big data quantity into
When row data are synchronous, can all there are pressure, very consumption computing resource, serious meeting to network pressure, magnetic disc i/o, memory capacity
Influence regular traffic operation.
Summary of the invention
The embodiment of the present invention provides a kind of method of data synchronization of Distributed Heterogeneous Database system, to distributed different
Data are synchronized under structure database system environment, avoiding between distinct type data-base system can not timely perception data
Variation, effectively improves the accuracy rate of data query and analysis, this method comprises:
One or more groups of operational orders and operation information are obtained, the operational order and operation information are sent to the first number
According to library cluster, the operational order includes: insertion data, more new data or deletes data, and the operation information includes: operative position
It sets and data value；
First database cluster is received according to the data at every group of operational order and operation information modification respective operations position
The operating result returned afterwards；
According to each operating result, corresponding operational order and operation information are packaged as message and issued, the message is used for
It is synchronous that data are carried out to operating position corresponding in the second data-base cluster.
The embodiment of the present invention is by obtaining one or more groups of operational orders and operation information, by operational order and operation information
It is sent to first database cluster, operational order includes: insertion data, more new data or deletes data, and operation information includes: behaviour
Make position and data value, receives first database cluster and modified at respective operations position according to every group of operational order and operation information
Data after the operating result that returns corresponding operational order and operation information are packaged as disappearing finally according to each operating result
Breath issues, and message is used to carry out data to operating position corresponding in the second data-base cluster to synchronize, and avoids different types of data
Between the system of library can not in time perception data variation, effectively improve the accuracy rate of data query and analysis.
The embodiment of the present invention provides a kind of method of data synchronization of Distributed Heterogeneous Database system, to distributed different
Data are synchronized under structure database system environment, are avoided because can not timely perception data between distinct type data-base system
The analysis inaccuracy of data caused by variation, and while real-time synchronization data, resource consumption is effectively saved, safeguards system
It operates normally, this method comprises:
The message of setting quantity is obtained, the message includes operational order and operation information, the operational order and operation
Information is corresponding with the operating result of first database cluster, and the operating result is first database cluster according to every group of operation
It is returned after data at instruction and operation information modification respective operations position；
The message of same operating position is merged, according to the message after merging to right in Second Type Database Systems
It is synchronous that the operating position answered carries out data.
The embodiment of the present invention sets the message of quantity by obtaining, and merges to the message of same operating position, according to
Message after merging carries out data to operating position corresponding in Second Type Database Systems and synchronizes, and avoids because of different type number
According between the system of library can not data analysis inaccuracy caused by perception data variation in time, and in the same of real-time synchronization data
When, by being merged to the identical message of operating position in operation information, effectively saving resource consumption, safeguards system it is normal
Operation.
The embodiment of the present invention provides a kind of data synchronization unit of Distributed Heterogeneous Database system, to distributed different
Data are synchronized under structure database system environment, avoiding between distinct type data-base system can not timely perception data
Variation, effectively improves the accuracy rate of data query and analysis, which includes:
Operation obtains module, for obtaining one or more groups of operational orders and operation information, by the operational order and behaviour
It is sent to first database cluster as information, the operational order includes: insertion data, more new data or deletes data, described
Operation information includes: operating position and data value；
As a result receiving module is corresponded to for receiving first database cluster according to every group of operational order and operation information modification
The operating result returned after data at operating position；
Message transmission module, for according to each operating result, corresponding operational order and operation information to be packaged as message
It issues, the message is used to carry out data to operating position corresponding in the second data-base cluster synchronous.
The embodiment of the present invention is by obtaining one or more groups of operational orders and operation information, by operational order and operation information
It is sent to first database cluster, operational order includes: insertion data, more new data or deletes data, and operation information includes: behaviour
Make position and data value, receives first database cluster and modified at respective operations position according to every group of operational order and operation information
Data after the operating result that returns corresponding operational order and operation information are packaged as disappearing finally according to each operating result
Breath issues, and message is used to carry out data to operating position corresponding in the second data-base cluster to synchronize, and avoids different types of data
Between the system of library can not in time perception data variation, effectively improve the accuracy rate of data query and analysis.
The embodiment of the present invention provides a kind of data synchronization unit of Distributed Heterogeneous Database system, to distributed different
Data are synchronized under structure database system environment, are avoided because can not timely perception data between distinct type data-base system
The analysis inaccuracy of data caused by variation, and while real-time synchronization data, resource consumption is effectively saved, safeguards system
It operates normally, which includes:
Message reception module, for obtaining the message of setting quantity, the message includes operational order and operation information, institute
It is corresponding with the operating result of first database cluster to state operational order and operation information, the operating result is first database
Cluster after the data at every group of operational order and operation information modification respective operations position according to returning；
Data simultaneous module is merged for the message to same operating position, according to the message after merging to second
It is synchronous to carry out data for corresponding operating position in types of database system.
The embodiment of the present invention sets the message of quantity by obtaining, and merges to the message of same operating position, according to
Message after merging carries out data to operating position corresponding in Second Type Database Systems and synchronizes, and avoids because of different type number
According between the system of library can not data analysis inaccuracy caused by perception data variation in time, and in the same of real-time synchronization data
When, by being merged to the identical message of operating position in operation information, effectively saving resource consumption, safeguards system it is normal
Operation.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
It obtains other drawings based on these drawings.In the accompanying drawings:
Fig. 1 is a kind of schematic diagram of the method for data synchronization of Distributed Heterogeneous Database system in the embodiment of the present invention；
Fig. 2 is the schematic diagram of the method for data synchronization of another Distributed Heterogeneous Database system in the embodiment of the present invention；
Fig. 3 is a kind of structure chart of the data synchronization unit of Distributed Heterogeneous Database system in the embodiment of the present invention；
Fig. 4 is the structure chart of the data synchronization unit of another Distributed Heterogeneous Database system in the embodiment of the present invention.
Specific embodiment
Understand in order to make the object, technical scheme and advantages of the embodiment of the invention clearer, with reference to the accompanying drawing to this hair
Bright embodiment is described in further details.Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but simultaneously
It is not as a limitation of the invention.
In order to synchronize under Distributed Heterogeneous Database system environments to data, distinct type data-base system is avoided
Between can not perception data in time variation, effectively improve the accuracy rate of data query and analysis, the embodiment of the present invention provides one
The method of data synchronization of kind Distributed Heterogeneous Database system, as shown in Figure 1, this method may include:
Step 101 obtains one or more groups of operational orders and operation information, and the operational order and operation information are sent
To first database cluster, the operational order includes: insertion data, more new data or deletes data, the operation information packet
It includes: operating position and data value；
Step 102 receives first database cluster according to every group of operational order and operation information modification respective operations position
The operating result returned after the data at place；
Step 103, according to each operating result, corresponding operational order and operation information are packaged as message and issued, it is described
Message is used to carry out data to operating position corresponding in the second data-base cluster synchronous.
It, will as shown in Figure 1 it is known that the embodiment of the present invention is by obtaining one or more groups of operational orders and operation information
Operational order and operation information are sent to first database cluster, and operational order includes: insertion data, more new data or deletes number
According to operation information includes: operating position and data value, receives first database cluster according to every group of operational order and operation information
The operating result returned after data at modification respective operations position, finally according to each operating result, by corresponding operational order
It is packaged as message with operation information to issue, message is used to carry out data to operating position corresponding in the second data-base cluster same
Step, avoid between distinct type data-base system can not perception data in time variation, effectively improve data query and analysis
Accuracy rate.
It should be noted that first database cluster and the second data-base cluster are two distinct types of data base sets
System, can support the execution of update, insertion, deletion and the inquiry of business datum.It is fixed carrying out function for two data-base clusters
When position, the update, insertion, deletion unification that we design all data enter system, therefore, first by first database cluster
Data-base cluster can flexibly cope with the write-in of data, be responsible for basic, daily issued transaction, this kind of database typical generation
Table is the database HBase of OLTP type；Second data-base cluster handles the complex query processing of big data quantity, stresses to decision
The database of the decision support of personnel and senior management staff, this kind of OLAP type is represented as Hive.Two isomeric data clusters
It can be deployed in the network environment of two isomeries.When executing data insertion, update or delete operation, manager's cluster 1 will be adjusted
With the write order of first database cluster, data are fallen into first in first database cluster 2, then in the second data-base cluster
It is synchronous to carry out data.When executing data query operation, needs to analyze the type and operate most matched data-base cluster, will inquire
Corresponding data-base cluster is forwarded to after request conversion.
When it is implemented, one or more groups of operational orders and operation information are obtained, by the operational order and operation information
It is sent to first database cluster, the operational order includes: insertion data, more new data or deletes data, the operation letter
Breath includes: operating position and data value.
In embodiment, user can issue operational order and operation information in the form of interface is given client first,
One or more groups of operational orders and operation information are obtained, the operational order includes: insertion data, more new data or deletes number
According to the operation information includes: operating position and data value.
In embodiment, operating position includes: application name, data table name and field name.
In embodiment, operating position further include: major key is expert at for the specific data in mark data table.
It must include the data value being inserted into and operation in operation information if operational order is insertion data in embodiment
Position；It must include data value and operating position to be updated in operation information if operational order is more new data；If operation refers to
It enables to delete data, must include operating position to be deleted in operation information；If operational order is inquiry data, operation information
In must include the matching condition of data to be checked and the method name that is analyzed and counted to query result.
In embodiment, one or more groups of operational orders and operation information are obtained, operational order and operation information are converted to
The processing sentence that first database cluster is appreciated that and operates needs to be converted into internal system for operation information to be processed
General data format, i.e. DataObject data object.For the synchronous accuracy of follow-up data, DataObject must be detailed
Operation information each time is recorded to the greatest extent.Data include affiliated application name, table name, major key data and String word in DataObject
(key-value) key assignments group that three symbol, Long integer and Decimal floating type data types are constituted.
Respective operations position is modified according to every group of operational order and operation information when it is implemented, receiving first database cluster
The operating result returned after the data at the place of setting.
In embodiment, first database cluster is modified at respective operations position according to every group of operational order and operation information
Data parse the field of acquisition if operational order is insertion data or more new data from user operation instruction and operation information
List information is compared with the field name of corresponding data table name in metadata table information table and field Value Types one by one, by the every of list
A pair of of field and data value are mapped to the corresponding data group of DataObject；If operational order is to delete data, from metadata table
The corresponding full dose field name of identical table name in information table encloses default null value, is mapped to the corresponding data group of DataObject,
And be added in Long group with " Delete " be key with 1 for value key-value pair, illustrate that the data markers are deleted for logic.
In embodiment, operational order can also be subdivided into basic query and complex query for inquiry data, inquiry, substantially
Several data lines for meeting querying condition are searched in inquiry from mass data, and OLTP database is responsible for the inquiry；Complicated
Data query, i.e. analysis mining batch data obtain certain result of decision, and olap database is responsible for the inquiry.In refinement user
When query type, according to the field name of the table name of offer and matching condition, carried out with metadata table special field information table
It compares, the field name of matching condition appears in tables of data special field information table, illustrates item in operational order and operation information
Part part includes major key or secondary index field, and further, inquiry request does not need analysis statistics, and (request sentence do not include
Statistics numbers, grouping and classifying ,/maximum/minimum of averaging), then it is OLTP type operations, is converted to and calls OLTP data
Library access interface, on the contrary it is converted to and calls olap database access interface.
In embodiment, first database cluster is modified at respective operations position according to every group of operational order and operation information
After data, the operating result that first database cluster returns is received, the operating result may include: the data at operating position
Successfully modification or operating position at data modify not successfully.
When it is implemented, corresponding operational order and operation information are packaged as message and issued according to each operating result, institute
It is synchronous for carrying out data to operating position corresponding in the second data-base cluster to state message.
In embodiment, the operating result that first kind Database Systems return is received, if the data at operating position have become
Function modification, is packaged as message for corresponding operational order and operation information；If the data at operating position are modified not successfully, to
First kind Database Systems send operation failure instruction, heavy after receiving operation failure instruction for first database cluster
New root is according to the data at operational order and operation information modification respective operations position；Continuation is judged according to the above method, and is united
The transmission times of operation failure instruction is counted, if number reaches preset threshold, such as number reaches 5 times, stops judgement, operation is referred to
It enables and the log of operation information write error.
In embodiment, after obtaining one or more groups of operational orders and operation information, every group of operational order and operation are generated
The corresponding operating time stamp of information, then by the corresponding operational order of data and operation letter at the operating position for the modification that succeeded
Breath and operating time stamp are packaged as message.Inventors have found that requiring in data synchronize timing high.Timing is to measure
One of the important indicator of net synchronization capability quality, common synchronization method enter heterogeneous database system timing to data and are difficult to protect
Card, it is understood that there may be a possibility that synchronous sequence malfunctions.It is assumed that service application A and B to the same data line of source database table respectively into
Row updates and delete operation, it is assumed that updates and carries out earlier than deletion.But when the follow-up data corresponding log of synchronous generation, due to being
The reasons such as system network delay are that the log of delete operation is early created on the log for updating operation instead, data are caused to synchronize rear mesh
The same data record deletion of database is marked earlier than execution is updated, causes timing error.Therefore, the embodiment of the present invention passes through generation
Every group of operational order and the corresponding operating time stamp of operation information, and timestamp is packaged together with operational order and operation information
For message, guarantee that the timing of every group of message can track, to guarantee the accuracy of timing.
Based on the same inventive concept, the embodiment of the invention also provides a kind of data of Distributed Heterogeneous Database system are same
Device is walked, as described in the following examples.The data of the principle and Distributed Heterogeneous Database system that are solved the problems, such as due to these
Synchronous method is similar, therefore the implementation of device may refer to the implementation of method, and overlaps will not be repeated.
Fig. 2 is the structure chart of the data synchronization unit of Distributed Heterogeneous Database system in the embodiment of the present invention, such as Fig. 2 institute
Show, the data synchronization unit of the Distributed Heterogeneous Database system includes:
Operation obtains module 201, for obtaining one or more groups of operational orders and operation information, by the operational order and
Operation information is sent to first database cluster, and the operational order includes: insertion data, more new data or deletes data, institute
Stating operation information includes: operating position and data value；
As a result receiving module 202 are modified for receiving first database cluster according to every group of operational order and operation information
The operating result returned after data at respective operations position；
Message transmission module 203, for according to each operating result, corresponding operational order and operation information being packaged as disappearing
Breath issues, and the message is used to carry out data to operating position corresponding in the second data-base cluster synchronous.
It, will as shown in Figure 2 it is known that the embodiment of the present invention is by obtaining one or more groups of operational orders and operation information
Operational order and operation information are sent to first database cluster, and operational order includes: insertion data, more new data or deletes number
According to operation information includes: operating position and data value, receives first database cluster according to every group of operational order and operation information
The operating result returned after data at modification respective operations position, finally according to each operating result, by corresponding operational order
It is packaged as message with operation information to issue, message is used to carry out data to operating position corresponding in the second data-base cluster same
Step, avoid between distinct type data-base system can not perception data in time variation, effectively improve data query and analysis
Accuracy rate.
The embodiment of the present invention provides the method for data synchronization of another Distributed Heterogeneous Database system, in distribution
Data are synchronized under heterogeneous database system environment, are avoided because number can not be perceived between distinct type data-base system in time
Inaccuracy is analyzed according to data caused by variation, and while real-time synchronization data, effectively saving resource consumption, safeguards system
Normal operation, as shown in figure 3, this method may include:
Step 301, the message for obtaining setting quantity, the message includes operational order and operation information, and the operation refers to
Enable and operation information it is corresponding with the operating result of first database cluster, the operating result for first database cluster according to
It is returned after data at every group of operational order and operation information modification respective operations position；
Step 302 merges the message of same operating position, according to the message after merging to Second Type database
It is synchronous to carry out data for corresponding operating position in system.
As shown in Figure 3 it is known that the embodiment of the present invention sets the message of quantity by obtaining, to same operating position
Message merges, and it is same to carry out data to operating position corresponding in Second Type Database Systems according to the message after merging
Step, avoid because between distinct type data-base system can not in time perception data variation caused by data analysis inaccuracy, and
While real-time synchronization data, by merging to the identical message of operating position in operation information, resource is effectively saved
Consumption, the normal operation of safeguards system.
When it is implemented, obtaining the message of setting quantity, the message includes operational order and operation information, the operation
Instruction and operation information are corresponding with the operating result of first database cluster, and the operating result is first database cluster root
According to what is returned after the data at every group of operational order and operation information modification respective operations position.
In embodiment, big data Kafka technology can be used and guarantee that the reliable of data obtains and exchange, guarantee the complete of data
Property and consistency, lose synchronous data not, sequentially transmit, and may span across the network environment of isomery.Specifically, among message
Part cluster is made of several message queues, and the management and access of message are carried out using publish/subscribe mode.Each message queue is deposited
The message of same service application is stored up, all tables of data are grouped by service application in system.Duplication cluster is copied by multiple
Cheng Zucheng, the data that each process is responsible for one or more queues are synchronous, and multiple duplicating process can be deployed in same or more
On platform server, according to the pressure increase of copy job or the number of the corresponding duplicating process of reduction.Message-oriented middleware cluster obtains
After the message for setting quantity, duplication cluster subscribes to the message of the service application, periodically obtains message from the queue, such as
Message is obtained from the queue every 100 milliseconds, parsing message is converted to the processing sentence of the executable write-in of the second data-base cluster,
Complete database synchronization.
When it is implemented, being merged to the message of same operating position, according to the message after merging to Second Type number
It is synchronous that data are carried out according to operating position corresponding in the system of library.Inventors have found that it is synchronous to carry out data under the premise of big data quantity
When, can all there are pressure, very consumption computing resource to network pressure, magnetic disc i/o, memory capacity, it is serious to will affect normal industry
Business operation, it is synchronous for performance requirement with the completion in 1 hour of 1TB data, need to expend two heterogeneous database system cluster 280MB/
Second network bandwidth, magnetic disc i/o and target cluster 1TB memory capacity.Therefore, the embodiment of the present invention by message into
Row merges, and then carries out data to operating position corresponding in Second Type Database Systems according to the message after merging and synchronizes,
While real-time synchronization data, by merging to the identical message of operating position in operation information, resource is effectively saved
Consumption, the normal operation of safeguards system.
In embodiment, the message further includes operating time stamp, and merging to the message of same operating position includes: head
The operating position in each Message Opcode information is first obtained, then by the identical message coalescing of operating position, and presses operating time stamp
Sequencing arranged.Inventors have found that requiring in data synchronize timing high.Timing is to measure net synchronization capability
One of the important indicator of quality, common synchronization method enters heterogeneous database system timing to data and is difficult to ensure, Ke Nengcun
A possibility that synchronous sequence malfunctions.It is assumed that service application A and B is updated and deletes respectively to the same data line of source database table
Except operation, it is assumed that update and carried out earlier than deletion.But when the follow-up data corresponding log of synchronous generation, since grid postpones
Etc. reasons, be instead delete operation log be early created on update operation log, cause data to synchronize rear target database
With data record deletion earlier than execution is updated, timing error is caused.Therefore, the embodiment of the present invention by by timestamp and operation
Instruction and operation information are packaged as message together, the operating position in each Message Opcode information are then obtained, by operating position phase
Same message coalescing, and arranged by the sequencing of operating time stamp, guarantee that the timing of every group of message can track,
To guarantee the accuracy of timing.
In embodiment, each message is traversed, major key information is parsed from message, the message of same major key is divided into one
Group.If the message groups of same major key there are multiple messages, need to merge message content.Message coalescing specific method is such as
Under, the operational order and operating time stamp in message are parsed, message is pressed into the big minispread of operating time stamp, timestamp is small to be come
The head of message groups.Judge in current message group whether containing operational order for the message for deleting data, if current message group is not
It is the message for deleting data containing operational order, it is only necessary to by operational order be that insertion data or the message of more new data are closed
And be a piece of news, merge algorithm and is referred to as Combine algorithm.Combine algorithm is as follows, from small to large time by operating time stamp
Each message of message groups is gone through, the key-value pair for including in outbound message is extracted, key assignments is added in system Map container one by one,
It is duplicate key-value pair due to not allowing to store key key in Map container, the Value value for the Key key being newly put into every time can cover
Same Key key in Map container, thus after traversing operation in Map container remaining key-value pair be message coalescing most
New content can finally update the data of target database.Operational order if it exists is that the message count of deletion data is big
In being equal to one, positioning operation timestamp is newest and operational order is the position for deleting the message of data, if the position
Setting in message groups the last item, then entire message groups only need to retain X；If the position is in the middle part of message groups or head, delete
All message of the operating time stamp earlier than the message position are gone, retaining the operational order is the message for deleting data, furthermore will behaviour
The remaining entire message for making timestamp greater than the message merges into a piece of news also according to Combine algorithm.By above-mentioned conjunction
And repeatedly the message groups of more new scene finally retain a piece of news, the message groups for deleting again more new scene can finally retain two
Message greatly reduces number and data volume in peak traffic period across cluster transmission data.For deleting more new scene again, after
It must be handled in such a way that db transaction operates when continuous synchronous, it is ensured that first delete the sequence accuracy updated afterwards.
In embodiment, the synchronous task that generating to the message after merging concurrently to execute carries out corresponding simultaneously operating.By
It is independent of each other in the simultaneously operating of the message of different major keys, therefore process handles and takes concurrent mode when synchronizing, and will criticize
The synchronous task of amount is distributed in different threads and executes.According to the message after merging, from message extraction operation code, table name,
Major key and data key-value pair information create synchronous task one by one, are committed to thread pool and are handled.It should be noted that if
Combined message is form renewal after first deleting, then is the synchronous task of an atomic transaction form by two message establishings.Line
Cheng Chi is provided with fixed thread pool size, when task processing number is greater than thread pool size, i.e., in peak traffic phase workload
When huge, in order to promote synchronous efficiency, system resource is saved, task queue, which enters, waits until thread in buffering queue
There is vacant thread to carry out task execution in pond.
Based on the same inventive concept, the embodiment of the invention also provides a kind of data of Distributed Heterogeneous Database system are same
Device is walked, as described in the following examples.The data of the principle and Distributed Heterogeneous Database system that are solved the problems, such as due to these
Synchronous method is similar, therefore the implementation of device may refer to the implementation of method, and overlaps will not be repeated.
Fig. 4 is the structure chart of the data synchronization unit of Distributed Heterogeneous Database system in the embodiment of the present invention, such as Fig. 4 institute
Show, the data synchronization unit of the Distributed Heterogeneous Database system includes:
Message reception module 401, for obtaining the message of setting quantity, the message includes operational order and operation letter
Breath, the operational order and operation information are corresponding with the operating result of first database cluster, and the operating result is first
Data-base cluster after the data at every group of operational order and operation information modification respective operations position according to returning；
Data simultaneous module 402 is merged for the message to same operating position, according to the message after merging to
It is synchronous to carry out data for corresponding operating position in two types of database systems.
As shown in Figure 4 it is known that the embodiment of the present invention sets the message of quantity by obtaining, to same operating position
Message merges, and it is same to carry out data to operating position corresponding in Second Type Database Systems according to the message after merging
Step, avoid because between distinct type data-base system can not in time perception data variation caused by data analysis inaccuracy, and
While real-time synchronization data, by merging to the identical message of operating position in operation information, resource is effectively saved
Consumption, the normal operation of safeguards system.
In conclusion the embodiment of the present invention provides a kind of method of data synchronization of Distributed Heterogeneous Database system, pass through
One or more groups of operational orders and operation information are obtained, operational order and operation information are sent to first database cluster, is grasped
Make instruction to include: insertion data, more new data or delete data, operation information includes: operating position and data value, receives first
Data-base cluster modifies the operating result returned after the data at respective operations position according to every group of operational order and operation information,
Finally according to each operating result, corresponding operational order and operation information are packaged as message and issued, message is used for the second number
Data are carried out according to operating position corresponding in the cluster of library to synchronize, and avoid that number can not be perceived between distinct type data-base system in time
According to variation, effectively improve the accuracy rate of data query and analysis.In embodiment, believed by generating every group of operational order and operation
Corresponding operating time stamp is ceased, and timestamp is packaged as message together with operational order and operation information, guarantees every group of message
Timing can track, to guarantee the accuracy of timing.
The embodiment of the present invention also provides the method for data synchronization of another Distributed Heterogeneous Database system, is set by obtaining
The message of fixed number amount merges the message of same operating position, according to the message after merging to Second Type data base set
Corresponding operating position progress data are synchronous in system, avoid because that perception data can not become in time between distinct type data-base system
The analysis inaccuracy of data caused by change, and while real-time synchronization data, by identical to operating position in operation information
Message merge, effectively saving resource consumption, the normal operation of safeguards system.In embodiment, by by timestamp and behaviour
Make instruction and operation information is packaged as message together, the operating position in each Message Opcode information is then obtained, by operating position
Identical message coalescing, and arranged by the sequencing of operating time stamp, guarantee that the timing of every group of message is can to track
, to guarantee the accuracy of timing.
The embodiment of the present invention also provides a kind of data synchronization unit of Distributed Heterogeneous Database system, by obtaining one group
Or multiple groups operational order and operation information, operational order and operation information are sent to first database cluster, operational order packet
Include: insertion data, more new data delete data, and operation information includes: operating position and data value, receive first database collection
Group is according to the operating result returned after the data at every group of operational order and operation information modification respective operations position, last basis
Corresponding operational order and operation information are packaged as message and issued by each operating result, and message is used for the second data-base cluster
In corresponding operating position carry out that data are synchronous, avoid between distinct type data-base system can not perception data in time change
Change, effectively improves the accuracy rate of data query and analysis.
The embodiment of the present invention also provides a kind of data synchronization unit of Distributed Heterogeneous Database system, is set by obtaining
The message of quantity merges the message of same operating position, according to the message after merging to Second Type Database Systems
In corresponding operating position to carry out data synchronous, avoid because can not perception data variation in time between distinct type data-base system
Caused by data analysis inaccuracy, and while real-time synchronization data, by identical to operating position in operation information
Message merges, effectively saving resource consumption, the normal operation of safeguards system.
It should be understood by those skilled in the art that, the embodiment of the present invention can provide as method, system or computer program
Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the present invention
Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the present invention, which can be used in one or more,
The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces
The form of product.
The present invention be referring to according to the method for the embodiment of the present invention, the process of equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
Particular embodiments described above has carried out further in detail the purpose of the present invention, technical scheme and beneficial effects
Describe in detail it is bright, it should be understood that the above is only a specific embodiment of the present invention, the guarantor being not intended to limit the present invention
Range is protected, all within the spirits and principles of the present invention, any modification, equivalent substitution, improvement and etc. done should be included in this
Within the protection scope of invention.
Claims (10)
1. a kind of method of data synchronization of Distributed Heterogeneous Database system characterized by comprising
One or more groups of operational orders and operation information are obtained, the operational order and operation information are sent to first database
Cluster, the operational order include: insertion data, more new data or delete data, the operation information include: operating position with
Data value；
Reception first database cluster returns after modifying the data at respective operations position according to every group of operational order and operation information
The operating result returned；
According to each operating result, corresponding operational order and operation information are packaged as message and issued, the message is used for the
It is synchronous to carry out data for corresponding operating position in two data-base clusters.
2. the method as described in claim 1, which is characterized in that the operating position includes: application name, data table name and field
Name.
3. the method as described in claim 1, which is characterized in that according to each operating result, by corresponding operational order and operation
Information package is message sending, comprising:
Receive the operating result that first kind Database Systems return；
If the data at operating position are successfully modified, corresponding operational order and operation information are packaged as message；
If the data at operating position are modified not successfully, operation failure instruction is sent to first kind Database Systems, is used for
First database cluster modifies respective operations position according to operational order and operation information again after receiving operation failure instruction
Set the data at place；
Continuation is judged according to the above method, and the transmission times of statistical operation failure command is stopped if number reaches preset threshold
Only judge, by operational order and operation information write error log.
4. the method as described in claim 1, which is characterized in that further include:
After obtaining one or more groups of operational orders and operation information, every group of operational order and the corresponding operation of operation information are generated
Timestamp；
By at the operating position for the modification that succeeded the corresponding operational order of data and operation information and operating time stamp be packaged
For message.
5. a kind of method of data synchronization of Distributed Heterogeneous Database system characterized by comprising
The message of setting quantity is obtained, the message includes operational order and operation information, the operational order and operation information
Corresponding with the operating result of first database cluster, the operating result is first database cluster according to every group of operational order
It is returned with after the data at operation information modification respective operations position；
The message of same operating position is merged, according to the message after merging to corresponding in Second Type Database Systems
It is synchronous that operating position carries out data.
6. method as claimed in claim 5, which is characterized in that the message further includes operating time stamp；
The message of same operating position is merged as follows:
Obtain the operating position in each Message Opcode information；
It is arranged by the identical message coalescing of operating position, and by the sequencing of operating time stamp.
7. a kind of data synchronization unit of Distributed Heterogeneous Database system characterized by comprising
Operation obtains module, and for obtaining one or more groups of operational orders and operation information, the operational order and operation are believed
Breath is sent to first database cluster, and the operational order includes: insertion data, more new data or deletes data, the operation
Information includes: operating position and data value；
As a result receiving module modifies respective operations according to every group of operational order and operation information for receiving first database cluster
The operating result returned after data at position；
Message transmission module, for corresponding operational order and operation information being packaged as message and issued according to each operating results,
The message is used to carry out data to operating position corresponding in the second data-base cluster synchronous.
8. a kind of data synchronization unit of Distributed Heterogeneous Database system characterized by comprising
Message reception module, for obtaining the message of setting quantity, the message includes operational order and operation information, the behaviour
Make instruction and operation information is corresponding with the operating result of first database cluster, the operating result is first database cluster
According to what is returned after the data at every group of operational order and operation information modification respective operations position；
Data simultaneous module is merged for the message to same operating position, according to the message after merging to Second Type
It is synchronous to carry out data for corresponding operating position in Database Systems.
9. a kind of computer equipment including memory, processor and stores the meter that can be run on a memory and on a processor
Calculation machine program, which is characterized in that the processor realizes any side of claim 1 to 6 when executing the computer program
Method.
10. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage has perform claim
It is required that the computer program of 1 to 6 any the method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910084472.2A CN109885617A (en) | 2019-01-29 | 2019-01-29 | The method of data synchronization and device of Distributed Heterogeneous Database system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910084472.2A CN109885617A (en) | 2019-01-29 | 2019-01-29 | The method of data synchronization and device of Distributed Heterogeneous Database system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109885617A true CN109885617A (en) | 2019-06-14 |
Family
ID=66927188
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910084472.2A Pending CN109885617A (en) | 2019-01-29 | 2019-01-29 | The method of data synchronization and device of Distributed Heterogeneous Database system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109885617A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110602250A (en) * | 2019-09-29 | 2019-12-20 | 网易（杭州）网络有限公司 | Data synchronization method and device, server and terminal equipment |
| CN110784532A (en) * | 2019-10-25 | 2020-02-11 | 北京天润融通科技股份有限公司 | Bidirectional data synchronization method and system |
| CN110795499A (en) * | 2019-09-17 | 2020-02-14 | 中国平安人寿保险股份有限公司 | Cluster data synchronization method, device and equipment based on big data and storage medium |
| CN111190912A (en) * | 2019-12-27 | 2020-05-22 | 山大地纬软件股份有限公司 | Large-transaction-oriented fragment execution method and device based on row change |
| CN111324668A (en) * | 2020-02-18 | 2020-06-23 | 中国联合网络通信集团有限公司 | Database data synchronous processing method and device and storage medium |
| CN111797166A (en) * | 2020-06-29 | 2020-10-20 | 中国工商银行股份有限公司 | Quasi-real-time resume data synchronization method and device, electronic equipment and medium |
| CN112015812A (en) * | 2020-08-10 | 2020-12-01 | 仁励家网络科技(杭州)有限公司 | Data synchronization method and data synchronization device |
| CN112487097A (en) * | 2020-12-11 | 2021-03-12 | 杭州安恒信息技术股份有限公司 | Method, system and equipment for synchronizing distributed field data |
| CN112699131A (en) * | 2021-01-18 | 2021-04-23 | 中国电子系统技术有限公司 | Mapping connection interaction method and device |
| CN113779048A (en) * | 2020-06-18 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Data processing method and device |
| CN116578647A (en) * | 2023-05-29 | 2023-08-11 | 玖章算术(浙江)科技有限公司 | Data synchronization method, device, system and computer readable storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060095477A1 (en) * | 2004-10-26 | 2006-05-04 | Via Technologies, Inc. | Database synchronizing system and method |
| CN102693324A (en) * | 2012-01-09 | 2012-09-26 | 西安电子科技大学 | Distributed database synchronization system, synchronization method and node management method |
| CN102902654A (en) * | 2012-09-03 | 2013-01-30 | 东软集团股份有限公司 | Method and device for linking data among heterogeneous platforms |
| CN103761318A (en) * | 2014-01-27 | 2014-04-30 | 中国工商银行股份有限公司 | Method and system for data synchronization of relational heterogeneous databases |
| CN105227657A (en) * | 2015-09-29 | 2016-01-06 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of data syn-chronization |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN108696595A (en) * | 2018-05-28 | 2018-10-23 | 郑州云海信息技术有限公司 | Distributed type assemblies method of data synchronization, master node, slave node, system and medium |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
-
2019
- 2019-01-29 CN CN201910084472.2A patent/CN109885617A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060095477A1 (en) * | 2004-10-26 | 2006-05-04 | Via Technologies, Inc. | Database synchronizing system and method |
| CN102693324A (en) * | 2012-01-09 | 2012-09-26 | 西安电子科技大学 | Distributed database synchronization system, synchronization method and node management method |
| CN102902654A (en) * | 2012-09-03 | 2013-01-30 | 东软集团股份有限公司 | Method and device for linking data among heterogeneous platforms |
| CN103761318A (en) * | 2014-01-27 | 2014-04-30 | 中国工商银行股份有限公司 | Method and system for data synchronization of relational heterogeneous databases |
| CN105227657A (en) * | 2015-09-29 | 2016-01-06 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of data syn-chronization |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN108696595A (en) * | 2018-05-28 | 2018-10-23 | 郑州云海信息技术有限公司 | Distributed type assemblies method of data synchronization, master node, slave node, system and medium |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110795499A (en) * | 2019-09-17 | 2020-02-14 | 中国平安人寿保险股份有限公司 | Cluster data synchronization method, device and equipment based on big data and storage medium |
| CN110795499B (en) * | 2019-09-17 | 2024-04-16 | 中国平安人寿保险股份有限公司 | Cluster data synchronization method, device, equipment and storage medium based on big data |
| CN110602250A (en) * | 2019-09-29 | 2019-12-20 | 网易（杭州）网络有限公司 | Data synchronization method and device, server and terminal equipment |
| CN110784532B (en) * | 2019-10-25 | 2021-09-07 | 北京天润融通科技股份有限公司 | Bidirectional data synchronization method and system |
| CN110784532A (en) * | 2019-10-25 | 2020-02-11 | 北京天润融通科技股份有限公司 | Bidirectional data synchronization method and system |
| CN111190912A (en) * | 2019-12-27 | 2020-05-22 | 山大地纬软件股份有限公司 | Large-transaction-oriented fragment execution method and device based on row change |
| CN111324668A (en) * | 2020-02-18 | 2020-06-23 | 中国联合网络通信集团有限公司 | Database data synchronous processing method and device and storage medium |
| CN111324668B (en) * | 2020-02-18 | 2023-11-21 | 中国联合网络通信集团有限公司 | Database data synchronous processing method, device and storage medium |
| CN113779048A (en) * | 2020-06-18 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Data processing method and device |
| CN111797166B (en) * | 2020-06-29 | 2023-08-18 | 中国工商银行股份有限公司 | Method and device for synchronizing quasi-real-time resume data, electronic equipment and medium |
| CN111797166A (en) * | 2020-06-29 | 2020-10-20 | 中国工商银行股份有限公司 | Quasi-real-time resume data synchronization method and device, electronic equipment and medium |
| CN112015812B (en) * | 2020-08-10 | 2021-04-27 | 仁励家网络科技(杭州)有限公司 | Data synchronization method and data synchronization device |
| CN112015812A (en) * | 2020-08-10 | 2020-12-01 | 仁励家网络科技(杭州)有限公司 | Data synchronization method and data synchronization device |
| CN112487097A (en) * | 2020-12-11 | 2021-03-12 | 杭州安恒信息技术股份有限公司 | Method, system and equipment for synchronizing distributed field data |
| CN112699131A (en) * | 2021-01-18 | 2021-04-23 | 中国电子系统技术有限公司 | Mapping connection interaction method and device |
| CN112699131B (en) * | 2021-01-18 | 2021-11-30 | 中国电子系统技术有限公司 | Mapping connection interaction method and device |
| CN116578647A (en) * | 2023-05-29 | 2023-08-11 | 玖章算术(浙江)科技有限公司 | Data synchronization method, device, system and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109885617A (en) | The method of data synchronization and device of Distributed Heterogeneous Database system | |
| CN110622152B (en) | Scalable database system for querying time series data | |
| US11630845B2 (en) | Data replication and data failover in database systems | |
| US11960464B2 (en) | Customer-related partitioning of journal-based storage systems | |
| US10262002B2 (en) | Consistent execution of partial queries in hybrid DBMS | |
| CN104781810B (en) | Capable and object database activity is traced into block grade thermal map | |
| Fragkoulis et al. | A survey on the evolution of stream processing systems | |
| US9589041B2 (en) | Client and server integration for replicating data | |
| US11442920B2 (en) | Graph database system | |
| CN108920698A (en) | A kind of method of data synchronization, device, system, medium and electronic equipment | |
| KR20080102622A (en) | Data replication method and systme for database management system | |
| CN104317928A (en) | Service ETL (extraction-transformation-loading) method and service ETL system both based on distributed database | |
| CN104519103A (en) | Synchronous network data processing method, server and related system | |
| CN102779138A (en) | Hard disk access method of real time data | |
| CN110019469A (en) | Distributed data base data processing method, device, storage medium and electronic device | |
| CN104317957A (en) | Open platform and system for processing reports and report processing method | |
| US10235407B1 (en) | Distributed storage system journal forking | |
| Branco et al. | Managing very large distributed data sets on a data grid | |
| CN111930821A (en) | One-step data exchange method, device, equipment and storage medium | |
| CN108763323A (en) | Meteorological lattice point file application process based on resource set and big data technology | |
| CN115640300A (en) | Big data management method, system, electronic equipment and storage medium | |
| CN109308290B (en) | Efficient data cleaning and converting method based on CIM | |
| US11789973B2 (en) | Software-defined database replication links | |
| Goncalves et al. | DottedDB: Anti-entropy without merkle trees, deletes without tombstones | |
| Zhong et al. | On mixing high-speed updates and in-memory queries: A big-data architecture for real-time analytics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190614 |
|
| RJ01 | Rejection of invention patent application after publication |