CN109739929A - Method of data synchronization, apparatus and system - Google Patents

Method of data synchronization, apparatus and system Download PDF

Info

Publication number
CN109739929A
CN109739929A CN201811552328.9A CN201811552328A CN109739929A CN 109739929 A CN109739929 A CN 109739929A CN 201811552328 A CN201811552328 A CN 201811552328A CN 109739929 A CN109739929 A CN 109739929A
Authority
CN
China
Prior art keywords
data
message
synchronized
subregion
message queue
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201811552328.9A
Other languages
Chinese (zh)
Other versions
CN109739929B (en
Inventor
张娜
鹿慧
何栋
黎锦康
尚玲瑞
李子旺
于灏
欧创新
刘震
杨猛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
PICC PROPERTY AND CASUALTY Co Ltd
Original Assignee
PICC PROPERTY AND CASUALTY Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by PICC PROPERTY AND CASUALTY Co Ltd filed Critical PICC PROPERTY AND CASUALTY Co Ltd
Priority to CN201811552328.9A priority Critical patent/CN109739929B/en
Publication of CN109739929A publication Critical patent/CN109739929A/en
Application granted granted Critical
Publication of CN109739929B publication Critical patent/CN109739929B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

This application discloses a kind of method of data synchronization, comprising: obtains the data to be synchronized in source database, the first data generated for the data to be synchronized change message；According to the data to be synchronized database table affiliated in the source database, the message subject of the first data change message is determined；According to the message subject, first data change message is pushed into the first message queue for only including a subregion corresponding with the message subject；When meeting trigger condition, according to default partitioning strategies, by the first data change message distribution in the first message queue into the first subregion of second message queue；The second message queue obtains data for data synchronisation unit and changes message, changes message according to data and data to be synchronized are synchronized to target database.It solves the problems, such as how more acurrate the change data of source database to be rapidly synchronized to target database.

Description

Method of data synchronization, apparatus and system
Technical field
This application involves technical field of data processing, and in particular to the synchronous system of method of data synchronization, device and a kind of data System.
Background technique
And business demand diversified industry huge for data scale, using micro services system construction data processing system System can adapt to business demand and the user volume increased rapidly.A kind of existing data processing using micro services system construction System includes a main system and multiple subsystems, and wherein main system is based on the main system data library for storing basic data The basic data is handled, using the main system data library as source database；Each subsystem is based on respective independent data Library externally provides business service, using the subsystem database as target database；The basic data of source database is occurring After change, need to be synchronized to each target database in time, to keep data consistency.How by the change data of source database More rapidly accurately sorting to target database, it is ensured that the data of each subsystem can reach consistency in time, be to need to solve The problem of.
Summary of the invention
The application provides a kind of method of data synchronization, how that the change data of source database are rapidly and accurately clear to solve The problem of point to target database.
The application provides a kind of method of data synchronization, comprising:
The data to be synchronized in source database are obtained, the first data generated for the data to be synchronized change message；
According to the data to be synchronized database table affiliated in the source database, the first data change is determined The message subject of message；
According to the message subject, first data change message push-in corresponding with the message subject first is disappeared Cease queue；Wherein, the first message queue is the single partition message queue for only including a subregion；
When meeting trigger condition, according to default partitioning strategies, the first data in the first message queue are changed Message distribution is into the first subregion of second message queue；The second message queue is the multi partition message for including multiple subregions Queue, each subregion in the multiple subregion change message for storing data；First subregion is the multiple point The subregion of the first data change message is stored in area；The second message queue obtains data change for data synchronisation unit Message changes message according to data and data to be synchronized is synchronized to target database.
Optionally, the data to be synchronized obtained in source database generate the first number for the data to be synchronized According to change message, comprising:
Obtain the log of source database；
Modification information is recorded according to the log acquisition；
The data to be synchronized are obtained according to the record modification information；
Using the data to be synchronized, generates the first data and change message.
Optionally, described to use the data to be synchronized, it generates the first data and changes message, comprising:
Processing is formatted to the data to be synchronized according to preset data form, treated counts using formatting According to generation first data change message.
Optionally, the data to be synchronized obtained in source database generate the first number for the data to be synchronized According to change message, comprising:
Determine the message major key for generating the first data change message；
According to the message major key and the data to be synchronized, generates first data and change message.
Optionally, described when meeting trigger condition, according to default partitioning strategies, by the first message queue One data change message distribution into the first subregion of second message queue, comprising:
When meeting trigger condition, the message major key of the first data change message is obtained；
According to the message major key and the default partitioning strategies, the subregion of the corresponding first data change message is determined Mark；
First data change message is sent to the subregion of the instruction of partition identification described in second message queue；It is described The subregion of partition identification instruction is first subregion.
Optionally, described according to the message major key and the default partitioning strategies, determine that corresponding first data become The partition identification of more message, comprising:
Obtain the cryptographic Hash of the message major key；
Obtain the number of partitions of second message queue；
Using the cryptographic Hash of the message major key to the number of partitions modulus, modulus is obtained as a result, by the modulus knot Fruit is as the default partitioning strategies；
The partition identification is obtained according to the default partitioning strategies.
Optionally, described when meeting trigger condition, according to default partitioning strategies, by the first message queue One data change message distribution into the first subregion of second message queue, comprising:
When meeting trigger condition, service class belonging to the corresponding data to be synchronized of the first data change message is obtained Type information or affiliated service area information；
Pair by the service type information or the service area information, at least one subregion of second message queue It should be related to, as the default partitioning strategies；
According to the partition identification of the specified corresponding first data change message of the default partitioning strategies；
According to the partition identification, first data change message is sent to subregion mark described in second message queue Know the subregion of instruction.
Optionally, the method also includes:
For the second message queue, start for providing the server-side of data change message；The server-side is used for The data, which are provided, to data synchronisation unit changes message.
The application also provides a kind of method of data synchronization, comprising:
Data synchronisation unit acquisition, which is used to indicate to obtain the first data in the first subregion of second message queue and change, to disappear The instruction information of breath；Wherein, the data synchronisation unit is used to the data to be synchronized in source database being synchronized to target data Library, the second message queue include multiple subregions, and each subregion in the multiple subregion changes for storing data to disappear Breath, first subregion are to be stored with first data to change message and subregion corresponding with the data synchronisation unit, The first data change message is the data change message for data to be synchronized；
The data synchronisation unit obtains the first data change message from first subregion；
The data synchronisation unit changes message according to first data, obtains the data to be synchronized；
The data to be synchronized are synchronized to the target database by the data synchronisation unit.
Optionally, the method also includes:
The data synchronisation unit obtains the data from second message queue using at least one thread and changes message；
The data synchronisation unit obtains the first data change message from first subregion, comprising:
The data synchronisation unit is obtained from first subregion using the first thread at least one described thread First data change message.
Optionally, the data synchronisation unit obtains the data from second message queue using at least one thread and changes Message, comprising:
The data synchronisation unit starts corresponding line for each subregion in multiple subregions of second message queue Journey；
The data change message is obtained from corresponding subregion using the thread of starting.
Optionally, the method also includes:
Using the thread as client；
Configure the service client information of the client parameter and corresponding second message queue；
Using the client parameter and the service client information, corresponding with the thread point is obtained from the server-side Data in area change message.
Optionally, the data synchronisation unit obtains the data from second message queue using at least one thread and changes Message, comprising:
Determine the message subject of at least one thread process；
According to the message subject, at least one subregion in multiple subregions of the second message queue is monitored；
The data listened to are obtained using at least one described thread changes message.
Optionally, the data to be synchronized are synchronized to the target database by the data synchronisation unit, comprising:
The data synchronisation unit changes message according to first data, generates for first into target database The database synchronization of database table synchrodata instructs；
It is instructed using the database synchronization, the data to be synchronized is synchronized to the first database table；Wherein, institute The structure for stating first database table, different from the structure of synchrodata database table affiliated in source database.
The application also provides a kind of database synchronous system, comprising: source database, message queue flow module, data are synchronous Unit, target database；
The source database needs to be synchronized to the data to be synchronized of the target database for providing, described to same Step data includes the basic data for insurance business；
The message queue flow module is generated for obtaining the data to be synchronized in source database for described to be synchronized First data of data change message；According to the data to be synchronized database table affiliated in the source database, determine The message subject of the first data change message；According to the message subject, by first data change message push-in with The corresponding first message queue of the message subject, wherein the first message queue is to only include the single partition of a subregion Message queue；When meeting trigger condition, according to default partitioning strategies, the first data in the first message queue are changed For message distribution into the first subregion of second message queue, the second message queue is the multi partition message for including multiple subregions Queue, each subregion in the multiple subregion change message for storing data, first subregion be stored with it is described First data change message and subregion corresponding with the data synchronisation unit, and the second message queue is synchronous for data Unit obtains data and changes message, changes message according to data and data to be synchronized are synchronized to the target database；
The data synchronisation unit, for starting and every for each subregion in multiple subregions of second message queue The corresponding thread of a subregion obtains the data using the thread of starting from corresponding subregion and changes message, by source number Target database is synchronized to according to the data to be synchronized in library, wherein the second message queue includes multiple subregions, the multiple Each subregion in subregion changes message for storing data；Acquisition is used to indicate in the first subregion of second message queue The instruction information for obtaining the first data change message is obtained from first subregion using the first thread in the thread of starting The first data are taken to change message, wherein the first data change message is the data change message for data to be synchronized；Root Message is changed according to first data, the data to be synchronized is obtained, the data to be synchronized is synchronized to the target data Library；
The target database, for receiving the synchronous data to be synchronized of the data synchronisation unit.
Optionally, the message queue flow module, including delta data capture submodule, and the delta data captures submodule Block is used for:
Obtain the log of source database；
Modification information is recorded according to the log acquisition；
The data to be synchronized are obtained according to the record modification information；
Using the data to be synchronized, generates the first data and change message.
Optionally, the delta data capture submodule is also used to:
According to the first database table in the source database, the data to be synchronized are obtained, the number to be synchronized is used Message is changed according to the first data are generated；
The data synchronisation unit, is also used to:
Message is changed according to first data, generates the data for the second database table in the target database Library synchronic command；
It is instructed according to the database synchronization and the data to be synchronized is synchronized to second database table；
Wherein, second database table structure is different from the first database table.
Optionally, the message queue flow module, is specifically used for:
Determine the message major key for generating the first data change message；
According to the message major key and the data to be synchronized, generates first data and change message.
Optionally, the message queue flow module, is specifically used for:
When meeting trigger condition, the message major key of the first data change message is obtained；
According to the message major key and the default partitioning strategies, the subregion of the corresponding first data change message is determined Mark；
First data change message is sent to the subregion of the instruction of partition identification described in second message queue；It is described The subregion of partition identification instruction is first subregion.
Optionally, the message queue flow module, is specifically used for:
Obtain the cryptographic Hash of the message major key；
Obtain the number of partitions of second message queue；
Using the cryptographic Hash of the message major key to the number of partitions modulus, modulus is obtained as a result, by the modulus knot Fruit is as the default partitioning strategies；
The partition identification is obtained according to the default partitioning strategies.
Optionally, the message queue flow module, including default partitioning strategies determination unit, the default partitioning strategies are true Order member is used for: when meeting trigger condition, obtaining industry belonging to the corresponding data to be synchronized of the first data change message Service type information or affiliated service area information；
Pair by the service type information or the service area information, at least one subregion of second message queue It should be related to, as the default partitioning strategies；
The message queue flow module, is specifically used for: according to specified corresponding first data of the default partitioning strategies Change the partition identification of message；
According to the partition identification, first data change message is sent to subregion mark described in second message queue Know the subregion of instruction.
Optionally, the message queue flow module further includes service terminals module, and the service terminals module is used for: needle To the second message queue, start for providing the server-side of data change message；The server-side is used for synchronous to data Unit provides the data and changes message；
The data synchronisation unit, is also used to:
Using the thread as client；
Configure the service client information of the client parameter and corresponding second message queue；
Using the client parameter and the service client information, corresponding with the thread point is obtained from the server-side Data in area change message.
Optionally, the data synchronisation unit, is also used to:
Determine the message subject of the thread process of the starting；
According to the message subject, at least one subregion in multiple subregions of the second message queue is monitored；
The data listened to, which are obtained, using the thread of the starting changes message.
Optionally, the system also includes marketing platforms, call service subsystem；
The marketing platform changes place to the basic data for handling the basic data of the source database Reason；The application programming interface for calling the call service subsystem, at the traffic data of the target database Reason；
The call service subsystem for providing application programming interface, and responds the marketing platform for institute The calling for stating application programming interface handles the traffic data of the target database；
The source database is also stored with the basic data comprising the data to be synchronized, receives the marketing platform pair The basic data changes the request of processing, and the request for the exception processes provides the data to be synchronized；
The target database is also stored with traffic data, and using the received data to be synchronized as increment traffic Data are stored.
Optionally, the source database includes: primary database and backup library；
The primary database changes the request of processing for receiving the marketing platform to the basic data, will The change data that request for the exception processes generates are synchronized to the backup library；
The backup library, for receiving the change data synchronized by the primary database, for the change data, to institute It states message queue flow module and the data to be synchronized is provided.
The application also provides a kind of data synchronization unit, comprising:
Data change message generation unit, for obtaining the data to be synchronized in source database, generate for described to same First data of step data change message；
Message subject determination unit, for the database affiliated in the source database according to the data to be synchronized Table determines the message subject of the first data change message；
First message queue unit, for according to the message subject, first data to be changed message push-in and institute State the corresponding first message queue of message subject；Wherein, the first message queue is to only include the single partition of a subregion to disappear Cease queue；
Zoning unit, will be in the first message queue according to default partitioning strategies for when meeting trigger condition First data change message distribution into the first subregion of second message queue；The second message queue be include multiple subregions Multi partition message queue, each subregion in the multiple subregion changes message for storing data；First subregion To be stored with the subregion that the first data change message in the multiple subregion；The second message queue is used for data synchronisation unit It obtains data and changes message, change message according to data and data to be synchronized are synchronized to target database.
The application also provides a kind of data synchronization unit, is applied to data synchronisation unit, and the data synchronisation unit includes:
It indicates acquisition of information subelement, obtains first in the first subregion of second message queue for obtaining to be used to indicate The instruction information of data change message；Wherein, the data synchronisation unit is used for the data to be synchronized in source database are synchronous To target database, the second message queue includes multiple subregions, and each subregion in the multiple subregion is used to store Data change message, first subregion be stored with first data change message and with the data synchronisation unit pair The subregion answered, the first data change message are the data change message for data to be synchronized；
First data change message and obtain subelement, disappear for obtaining the first data change from first subregion Breath；
Data acquisition subelement to be synchronized obtains the data to be synchronized for changing message according to first data；
To the synchronous subelement of target database, for the data to be synchronized to be synchronized to the target database.
Compared with prior art, the application has the following advantages:
The application providing method generates the first data by the data to be synchronized for source database and changes message, according to Data to be synchronized database table affiliated in source database determines the message subject of the first data change message；According to message First data change message distribution is arrived the including multiple subregions using only including the first message queue of a subregion by theme In first subregion of two message queues；The second message queue obtains data for data synchronisation unit and changes message, according to Data change message and data to be synchronized are synchronized to target database, can be same with real-time perfoming data by message queue mechanism Step, and message subject is determined according to database table, it can guarantee the corresponding data change of the data to be synchronized of each database table Message is divided into the same message subject, to guarantee the order of data to be synchronized；Pass through multiple points of second message queue Area can make the data for carrying data to be synchronized change message can be carried out parallel processing, promote treatment effeciency, enable data It is synchronous in time and update, to solve the change data of source database rapidly and accurately sorting asking to target database Topic.
Detailed description of the invention
Fig. 1 is a kind of process flow diagram for method of data synchronization that the application first embodiment provides；
Fig. 2 is to include in the application first embodiment from single partition message queue to the multiple cloth number of multi partition message queue According to the subregion schematic diagram of modified line message；
Fig. 3 is a kind of process flow diagram for method of data synchronization that the application second embodiment provides；
Fig. 4 is a kind of schematic diagram for data synchronous system that the application 3rd embodiment provides；
Fig. 5 is that the source database in the application 3rd embodiment is to what the target database of branch company progress data synchronized System schematic diagram；
Fig. 6 is a kind of data synchronization unit schematic diagram that the application fourth embodiment provides；
Fig. 7 is a kind of data synchronization unit schematic diagram that the 5th embodiment of the application provides.
Specific embodiment
Many details are explained in the following description in order to fully understand the application.But the application can be with Much it is different from other way described herein to implement, those skilled in the art can be without prejudice to the application intension the case where Under do similar popularization, therefore the application is not limited by following public specific implementation.
The application provides method of data synchronization, device and a kind of data synchronous system.In the following embodiments one by one It is described in detail.
The application first embodiment provides a kind of method of data synchronization.
For ease of understanding, the practical application scene of the method for data synchronization is briefly described first.Practical application In, message queue flow module and data synchronisation unit can be disposed in the calculating equipment between source database and target database, Data synchronisation unit is also referred to as sorting module.The equipment that calculates can be the server node of cluster.Message queue flow module For data to be synchronized in such a way that data change message, to be pushed into the single partition message queue of only one subregion, then will be single Data change message in subregion message queue is put into multiple subregions according to default partitioning strategies, each subregion in multi partition Storing data changes message.Specifically the message queue flow module can be realized based on message queue mechanism.The embodiment of the present application In, so-called message queue (Message Queue) mechanism is to transmit data by data unit of message between computing devices A kind of distributed message drive system, for example, Kafka message queue.Then, then by data synchronisation unit from the more of message queue A subregion obtains data and changes message, further obtains data to be synchronized, and be synchronized to target database.First embodiment provides Method of data synchronization, message queue flow module can be deployed in, for the change data of source database are rapidly and accurately clear Divide to target database.
A kind of method of data synchronization that the application first embodiment provides is illustrated below in conjunction with Fig. 1 and Fig. 2.
Method of data synchronization shown in FIG. 1, comprising: step S101 to step S104.
Step S101 obtains the data to be synchronized in source database, generates the first data for being directed to the data to be synchronized Change message.
The data to be synchronized are after the data of source database change, to need to be synchronized to the change of target database Data content.For example, source database is the oracle database A for being stored with the basic data for insurance business, user passes through Marketing platform can operate the basic data, so that basic data changes, the data content of variation is described Data to be synchronized.
Data change message, refer to the message for carrying data content that can be handled based on message queue mechanism, can To be the byte arrays that can be used to store any object.For example, being stored with the content of data to be synchronized and generating the same step number According to corresponding change type.First data change message, to change message comprising the data of the data to be synchronized.
In the embodiment of the present application, the publication of data change message is realized based on message queue mechanism, by data synchronisation unit Data change message is obtained from multiple subregions of message queue to be handled, and data to be synchronized are synchronized to target database.Tool Body, message queue mechanism uses Kafka message queue.The message that Kafka message queue can be used as data change message is slow Pond is rushed, in the biggish situation of the Message Processing efficiency variance of source database side and target database side, realizes that data change disappears The pooling feature of breath.Also, data are changed news release to difference by the graded properties that can also utilize kafka message queue Subregion improves data-handling efficiency so that multiple consumers may be implemented to message parallel processing in the data synchronisation unit.
In the embodiment of the present application, is generated especially by following processing according to the data to be synchronized and be directed to the number to be synchronized According to the first data change message:
Obtain the log of source database；
Modification information is recorded according to the log acquisition；
The data to be synchronized are obtained according to the record modification information；
Using the data to be synchronized, generates the first data and change message.
Continue to use the example of oracle database A.CDC (Changed Data Capture) connection Oracle number can be used According to library A, to obtain data to be synchronized.So-called CDC can use for the Incremental Log analysis tool that can connect oracle database In capture delta data in real time.
Preferably, when generating the first data change message using the data to be synchronized, further includes: according to preset data Format is formatted processing to the data to be synchronized, using treated data are formatted, generates first data and becomes More message.So-called preset data form can be the formats such as String, JSON.For example, by the data to be synchronized with JSON lattice Formula generates data and changes message, and the data of generation change in message, content and change type comprising data to be synchronized.
Preferably, the data to be synchronized obtained in source database generate the first number for the data to be synchronized According to change message, comprising: determine the message major key for generating the first data change message；According to the message major key and The data to be synchronized generate first data and change message.Message binding message major key, Ke Yi are changed to every data Designated position is sent by data change message in subsequent processing steps, is convenient for multidomain treat-ment.
Step S102, according to the data to be synchronized in the source database belonging to database table, determine described the The message subject of one data change message.
The message subject (Topic), to be distributed to classification belonging to every message of message queue.Pass through message master Topic, it may be convenient to the message to be consumed of determination.
For changing message based on the data of Kafka message queue, Kafka message queue includes the message producer and disappears Consumer is ceased, the message producer is the side that data are changed to news release to kafka message queue cluster, such as generates first Data change message and the module for pushing it into first message queue；Message consumer is to obtain from Kafka message queue cluster The side that access is handled according to change message, such as dispose the terminal or service of the data synchronisation unit.Belong to same The data change message of message subject actually may be maintained on one or more cluster servers, when message consumer subscribes to When one message subject, the message subject of data change message need to be only specified, without the practical holding position of data, just It can with cross-node receive all data change message relevant to the message subject.
In the embodiment of the present application, it is preferred that the corresponding message subject of a database table.Correspondingly, if one disappears The message for ceasing theme is needed by the processing of multiple consumption groups, then the message of the message subject can copy as more according to the quantity of consumption group Part, it is supplied to each consumption group, so that each consumption group can get whole and identical data, realizes that message is wide The function of broadcasting.Alternatively, it is also possible to also be able to maintain in subregion and disappear using the form of the corresponding message subject of multiple database tables The order of breath.
Step S103, according to the message subject, by first data change message push-in and the message subject pair The first message queue answered；Wherein, the first message queue is the single partition message queue for only including a subregion.
The subregion, refers to the minimum unit safeguarded in message queue mechanism to message data, in the same subregion Data change message have order.For example, the data change message being in together in a subregion, passes through the unique of setting Offset keeps the order of message.Different subregions can be different cluster servers physically.Each message subject can be with Include one or more subregions.The single partition message queue, for the message queue for only including a subregion, message queue includes Store the container of message.
This step is, according to the message subject of data change message, data change message to be pushed into corresponding single partition and is disappeared It ceases in queue.In practical application, due to multi-pass operation of the operation system to source database of source database, so that source number Successively occur repeatedly to change according to the same record data in library, it includes change type, changed content that change, which generates one, every time The data of the relevant information of synchrodata are waited to change message.Since there are successive dependence, institutes for the operation to source database To be produced the data change message of same message subject by a thread and be pushed into the message for belonging to same message subject Single partition message queue, it may have stringent, the identical order with the operation.When data synchronisation unit is according to generation data The order processing data for changing message change message, data to be synchronized correctly can be synchronized to target database.
Step S104, when meeting trigger condition, according to default partitioning strategies, by first in the first message queue Data change message distribution into the first subregion of second message queue；The second message queue is include multiple subregions more Subregion message queue, each subregion in the multiple subregion change message for storing data；First subregion is institute State the subregion that the first data change message is stored in multiple subregions；The second message queue is obtained for data synchronisation unit Data change message, change message according to data and data to be synchronized are synchronized to target database.
The multi partition message queue is the message queue for including multiple subregions.
This step is that the data in the single partition of first message queue are changed message, are published to second message queue Multiple subregions.During by the news release of single partition to multi partition, according to default partitioning strategies, every data is specified to change News release is to a subregion in the multi partition of second message queue.Specific to the embodiment of the present application, including following processing: will The data to be synchronized at source database end are first pushed to first message queue of only one subregion；To first message team Column carry out continuing monitoring, as long as listening in first message queue has data to change message, pulling data changes message and root The data pulled change message is pushed in the correspondence subregion of second message queue according to default partitioning strategies.
Referring to FIG. 2, Fig. 2 shows the message of the multi partition of the single partition by first message queue to second message queue Schematic diagram is issued, in figure, 201 be the single partition of first message queue, and first message queue, which maintains, to be relied on operation order and generate The order of data change message；202 be that the data of single partition are changed to news release to multi partition；203 be second message team The multi partition of column, number of partitions n.Each subregion maintains data change message in first message queue in second message queue Order.
In one embodiment of the embodiment of the present application, default partitioning strategies are determined according to message major key.Specifically, described When meeting trigger condition, according to default partitioning strategies, the first data in the first message queue are changed into message distribution Into the first subregion of second message queue, comprising:
When meeting trigger condition, the message major key of the first data change message is obtained；
According to the message major key and the default partitioning strategies, the subregion of the corresponding first data change message is determined Mark；
First data change message is sent to the subregion of the instruction of partition identification described in second message queue；It is described The subregion of partition identification instruction is first subregion.
First partitioned storage has first data change message, and obtains for the data synchronisation unit One data change message, change message according to the first data and corresponding data to be synchronized are synchronized to target database.
Preferably, the partition identification of the corresponding first data change message is determined using following processing:
Obtain the cryptographic Hash of the message major key；
Obtain the number of partitions of second message queue；
Using the cryptographic Hash of the message major key to the number of partitions modulus, modulus is obtained as a result, by the modulus knot Fruit is as the default partitioning strategies；
The partition identification is obtained according to the default partitioning strategies.
For example, default partitioning strategies determine as follows:
Partition value=hash (key) %partitionNum；
Wherein, key is message major key；
PartitionNum is the number of partitions of the multi partition of second message queue.
Partition value can be used as the partition identification of specified partition.Partitioning strategies are preset using this, it is ensured that message major key Identical data change message is pushed in the same subregion in multiple subregions of second message queue, ensure that data become More message is able to maintain data change message order in single partition when being published to multi partition by single partition.
In the another embodiment of the embodiment of the present application, for the specified default partitioning strategies of every data change message. Preferably, according to service type information or service area information belonging to data change message, the data change message is specified A subregion in the multi partition for the second message queue being published to.Specifically, it is described when meeting trigger condition, according to default Partitioning strategies, by the first subregion of the first data change message distribution in the first message queue to second message queue In, comprising:
When meeting trigger condition, service class belonging to the corresponding data to be synchronized of the first data change message is obtained Type information or affiliated service area information；
Pair by the service type information or the service area information, at least one subregion of second message queue It should be related to, as the default partitioning strategies；
According to the partition identification of the specified corresponding first data change message of the default partitioning strategies；
According to the partition identification, first data change message is sent to subregion mark described in second message queue Know the subregion of instruction.
By data change message by the single partition of first message queue, second message team is published to according to default partitioning strategies When multiple subregions of column, the data change message in each message subject is rationally published to multiple subregions, to ensure and one The same subregion that relevant all data change message fully enter second message queue is recorded, due to each of multi partition point Message order in area, to maintain the order of the data change message in single partition, therefore can correctly will be to same Step data is synchronized to target database, it is ensured that the consistency of data.And it is possible to parallel processing is carried out to multi partition, it is corresponding every A subregion is obtained using a thread and handles data change message, changes message for corresponding data to be synchronized according to the data It is synchronized to target database, improves data synchronization efficiency.
In the embodiment of the present application, when actual implementation, further includes:
For the second message queue, start for providing the server-side of data change message；The server-side is used for The data, which are provided, to data synchronisation unit changes message.
Based on above-described embodiment, the application second embodiment provides a kind of method of data synchronization.Second embodiment mentions The method of data synchronization of confession can be deployed in data synchronisation unit.Processing by data synchronisation unit as data change message Person obtains data from multiple subregions of message queue and changes message, further obtains data to be synchronized, and be synchronized to target data Library.
A kind of method of data synchronization that the application second embodiment provides is illustrated below in conjunction with Fig. 3.
Method of data synchronization shown in Fig. 3, comprising: step S301 to step S304.
Step S301, data synchronisation unit acquisition is used to indicate obtains the first number in the first subregion of second message queue According to the instruction information of change message；Wherein, the data to be synchronized in source database for being synchronized to by the data synchronisation unit Target database, the second message queue include multiple subregions, and each subregion in the multiple subregion is used to store number According to change message, first subregion is to be stored with first data to change message and corresponding with the data synchronisation unit Subregion, first data change message is to change message for the data of data to be synchronized.
This step is to obtain to be used to indicate to obtain the first data change message in the first subregion of second message queue Indicate information.For example, obtaining the instruction information according to business processing triggering.
In the embodiment of the present application, the data synchronisation unit is using at least one thread from described in second message queue acquisition Data change message.After obtaining the instruction information, first data are obtained from first subregion by following processing Change message: the data synchronisation unit is obtained from first subregion using the first thread at least one described thread Take the first data change message.In actual implementation, the Thread Count of the data synchronisation unit is not less than second message queue The number of partitions improved efficiency to realize multi-threading parallel process data.
Preferably, the data synchronisation unit obtains the data from second message queue using at least one thread and changes Message, comprising: the data synchronisation unit starts corresponding for each subregion in multiple subregions of second message queue Thread；The data change message is obtained from corresponding subregion using the thread of starting.
For example, provided that the message queue flow module of second message queue is to be realized based on Kafka message queue mechanism Computing module, then the per thread of data synchronisation unit starting can be consumer's mould of integrated Kafka message queue The consumer thread of block consumes the data change message of each subregion in the multi partition of second message queue.So-called consumer Module is the client that data change message is read to the broker (succedaneum) of Kafka cluster.In actual implementation, data are same Walking unit may include multiple consumption groups, and each consumption group may include multiple consumer threads.It is every in the same consumption group The data of one subregion of a consumer thread's alignment processing change message, and every data change message can only be by the same consumption group In a consumer thread processing.Due to the order of the data change message in each subregion of second message queue, protect The operation order that source database end generates data to be synchronized for same record is held, also, same records relevant number The same subregion of second message queue, therefore, a consumer thread of data synchronisation unit are distributed to according to change message The data handled in the subregion of the corresponding consumer thread one by one change message, do not upset the original suitable of data change message Data to be synchronized correctly can be synchronized to target database, keep data consistency by sequence.Multiple consumer threads respectively correspond place The data change message for managing multiple subregions, realizes the parallel processing of data change message, improves data synchronization efficiency.It needs Illustrate, in the message queue flow module for providing second message queue, the data in the same message subject change message More parts will be replicated according to the quantity of the consumption group of data synchronisation unit, each consumption group gets whole and identical data and becomes More message realizes the function of data change message broadcast, is convenient for horizontal extension.
Preferably, customer count is more than the number of partitions of second message queue, it is ensured that the data change of each subregion disappears Breath all can be processed, provides high availability.
Step S302, the data synchronisation unit obtain the first data change message from first subregion.
The data synchronisation unit is obtained from the first subregion in the multi partition of the second message queue of message queue flow module It takes and handles the first data change message.
In one embodiment of the embodiment of the present application, data synchronisation unit is pulled as client from second message queue (pull mode) data change message, including following processing:
Using the thread as client；
Configure the service client information of the client parameter and corresponding second message queue；
Using the client parameter and the service client information, corresponding with the thread point is obtained from the server-side Data in area change message.
For example, the first thread of the first subregion of data synchronisation unit alignment processing is as client, one by one from the first subregion It pulls and handles data change message.
In the another embodiment of the embodiment of the present application, data synchronisation unit monitors second message queue in real time and obtains number According to change message, including following processing:
Determine the message subject of at least one thread process；
According to the message subject, at least one subregion in multiple subregions of the second message queue is monitored；
The data listened to are obtained using at least one described thread changes message.
Step S303, the data synchronisation unit change message according to first data, obtain the data to be synchronized.
This step is the data to be synchronized for obtaining the first data change message and including.
In the embodiment of the present application, specifically, including following processing: changing message from the first data according to preset data form It is middle to obtain the data to be synchronized.For example, when generating the first data change message the format that uses for JSON format, then according to JSON format parses the content and change type of data to be synchronized.
The data to be synchronized are synchronized to the target database by step S304, the data synchronisation unit.
In the embodiment of the present application, the source database and the target database can be for disparate databases management systems The heterogeneous database of system.Specifically, the source database and the target database are the isomery of following any forms: running on Different Computer Architecture, with different underlying Operating Systems, with different DBMS (Database Management System).The method of data synchronization may also apply to following two kinds of scenes: the data to be synchronized are synchronized to target data In library belonging to target database table, table structure can be different from the data to be synchronized in the source database belonging to The table structure of source database table；Alternatively, the target database table and the source database table also can have identical table knot Structure.
The data to be synchronized are synchronized to the target database by the data synchronisation unit, comprising:
The data synchronisation unit changes message according to first data, generates for first into target database The database synchronization of database table synchrodata instructs；
It is instructed using the database synchronization, the data to be synchronized is synchronized to the first database table；Wherein, institute The structure for stating first database table, different from the structure of synchrodata database table affiliated in source database.In this way, Realize the synchronization between heterogeneous database.
Based on above-described embodiment, the application 3rd embodiment provides a kind of data synchronous system.
Referring to FIG. 4, it illustrates the schematic diagrames of the data synchronous system.Since 3rd embodiment is with above-described embodiment Based on, so describing fairly simple, relevant portion refers to the corresponding explanation of above-described embodiment.
Data synchronous system shown in Fig. 4, comprising: source database, message queue flow module, data synchronisation unit, target Database；
The source database 401 needs to be synchronized to the data to be synchronized of the target database for providing, it is described to Synchrodata includes the basic data for insurance business.
Source database can be the database for being stored with basic data, such as oracle database.
In one embodiment mode in the embodiment of the present application, the data synchronous system, further includes: marketing platform, words Business service subsystem；Wherein,
The marketing platform changes place to the basic data for handling the basic data of the source database Reason；The application programming interface for calling the call service subsystem, at the traffic data of the target database Reason；
The call service subsystem for providing application programming interface, and responds the marketing platform for institute The calling for stating application programming interface handles the traffic data of the target database；
Correspondingly, the source database, is also stored with the basic data comprising the data to be synchronized, receives the marketing Platform changes the request of processing to the basic data, and the request for the exception processes provides the number to be synchronized According to；The target database, is also stored with traffic data, and using the received data to be synchronized as increment traffic data into Row storage.
For example, the basic data in source database changes, change when marketing platform operates source database Data content and change type can be used as data to be synchronized.
Preferably, the source database includes: primary database and backup library；Wherein, the primary database, for cooperating It states marketing platform to handle the basic data, receives the marketing platform and the basic data is changed, will become More data are synchronized to the backup library；The backup library, for receiving the change data synchronized by the primary database, for institute The change data of basic data are stated, Xiang Suoshu message queue flow module provides the data to be synchronized.The backup library, including it is standby Use database.Data to be synchronized are provided to message queue flow module using backup library, data can be reduced and synchronized to primary database Performance influence.
The target database 402, for receiving the synchronous data to be synchronized of the data synchronisation unit.
Target database can be the database with source database isomery.For example, source database is oracle database, mesh Mark database can be Gauss database (Gaussian data library).The data synchronous system can be realized in two kinds of isomeric datas Data between library are synchronous.
The message queue flow module 403 is generated for described for obtaining the data to be synchronized in source database to same First data of step data change message；According to the data to be synchronized database table affiliated in the source database, really The message subject of the fixed first data change message；According to the message subject, by first data change message push-in First message queue corresponding with the message subject, wherein the first message queue is single point for only including a subregion Area's message queue；When meeting trigger condition, according to default partitioning strategies, the first data in the first message queue are become More message distribution into the first subregion of second message queue, the second message queue be include that the multi partitions of multiple subregions disappears Queue is ceased, each subregion in the multiple subregion changes message for storing data, and first subregion is to store The first data change message and subregion corresponding with the data synchronisation unit are stated, the second message queue is same for data It walks unit and obtains data change message, change message according to data and data to be synchronized are synchronized to the target database.
In one embodiment of the embodiment of the present application, the message queue flow module, including delta data capture submodule Block, the delta data capture submodule are used for the data to be synchronized at source database end, generate the first data and change message, specifically Including following processing:
Obtain the log of source database；
Modification information is recorded according to the log acquisition；
The data to be synchronized are obtained according to the record modification information；
Using the data to be synchronized, generates the first data and change message.
In practical application, the source database and the target database can be for different data base management systems Heterogeneous database.Specifically, the source database and the target database are the isomery of following any forms: running on difference Computer Architecture, with different underlying Operating Systems, with different DBMS (Database Management System).For example, source database is oracle database, purpose database is Gaussian data library (Gauss DB).Preferably, The delta data capture submodule is also used to: according to the first database table in the source database, being obtained described to be synchronized Data generate the first data using the data to be synchronized and change message；Correspondingly, the data synchronisation unit, is also used to:
Message is changed according to first data, generates the data for the second database table in the target database Library synchronic command；
It is instructed according to the database synchronization and the data to be synchronized is synchronized to second database table；
Wherein, second database table structure is different from the first database table.
In an embodiment provided by the embodiments of the present application, the message queue flow module changes message according to data Message major key, determines default partitioning strategies, selects data to change the message message queue to be published to according to default partitioning strategies Specific subregion.Therefore, the message queue flow module, also particularly useful for: it determines and disappears for generating the first data change The message major key of breath；According to the message major key and the data to be synchronized, generates first data and change message.Further , the message queue flow module obtains the message major key of the first data change message when meeting trigger condition；According to The message major key and the default partitioning strategies determine the partition identification of the corresponding first data change message；It will be described First data change message is sent to the subregion of the instruction of partition identification described in second message queue；The partition identification instruction Subregion is first subregion.
Wherein, the partition identification is determined especially by following processing:
Obtain the cryptographic Hash of the message major key；
Obtain the number of partitions of second message queue；
Using the cryptographic Hash of the message major key to the number of partitions modulus, modulus is obtained as a result, by the modulus knot Fruit is as the default partitioning strategies；
The partition identification is obtained according to the default partitioning strategies.
In the another embodiment of the embodiment of the present application, according to type of service or industry belonging to data change message Business region determines the specific subregion for being published to second message queue.Specifically, the message queue flow module, including default subregion Policy determining unit, the default partitioning strategies determination unit are used for: when meeting trigger condition, being obtained first data and become Service type information belonging to the corresponding data to be synchronized of more message or affiliated service area information；
Pair by the service type information or the service area information, at least one subregion of second message queue It should be related to, as the default partitioning strategies；
The message queue flow module, is specifically used for: according to specified corresponding first data of the default partitioning strategies Change the partition identification of message；
According to the partition identification, first data change message is sent to subregion mark described in second message queue Know the subregion of instruction.
In the another embodiment of the embodiment of the present application, the message queue flow module further includes server-side submodule Block, the service terminals module are used for: being directed to the second message queue, started for providing the service of data change message End；The server-side, which is used to provide the data to data synchronisation unit, changes message.
The data synchronisation unit 404, for for second message queue multiple subregions in each subregion, starting with The corresponding thread of each subregion obtains the data using the thread of starting from corresponding subregion and changes message, by source Data to be synchronized in database are synchronized to target database, wherein the second message queue includes multiple subregions, described more Each subregion in a subregion changes message for storing data；Obtain the first subregion being used to indicate in second message queue The middle instruction information for obtaining the first data change message, using the first thread in the thread of starting, from first subregion Obtain the first data change message, wherein the first data change message is the data change message for data to be synchronized； Message is changed according to first data, the data to be synchronized is obtained, the data to be synchronized is synchronized to the number of targets According to library.
In the embodiment of the present application, the data synchronisation unit is also referred to as sorting module, passes through multiple threads message queue Data in multiple subregions of the second message queue of module change message.In a kind of specific embodiment, the data are synchronous Unit, including multithreading consume submodule, and the multithreading consumption submodule is used for:
For each subregion in multiple subregions of second message queue, start corresponding thread；
The data change message is obtained from corresponding subregion using the thread of starting；Wherein, using the thread of starting In first thread, first data change message is obtained from first subregion.In actual implementation, the data are synchronous The Thread Count of unit is improved efficiency not less than the number of partitions of second message queue to realize multi-threading parallel process data.
In one embodiment of the embodiment of the present application, the message queue flow module, further includes server-side as described above Submodule, correspondingly, the data synchronisation unit, is also used to:
Using the thread as client；
Configure the service client information of the client parameter and corresponding second message queue；
Using the client parameter and the service client information, corresponding with the thread point is obtained from the server-side Data in area change message.
In the another embodiment of the embodiment of the present application, the data synchronisation unit supervises second message queue It listens, obtains and handle the data change message listened to, specifically include:
Determine the message subject of the thread process of the starting；
According to the message subject, at least one subregion in multiple subregions of the second message queue is monitored；
The data listened to, which are obtained, using the thread of the starting changes message.
Preferably, data synchronisation unit changes the content and change type that message obtains data to be synchronized according to data, SQL statement is generated, target database is connected to by JDBC, executes the SQL statement, the data to be synchronized are synchronized to mesh Mark database.
It is exemplified below.The source database that Fig. 5 shows certain group company is same to the target database of branch company progress data The system of step.In figure, marketing platform 501 connects the source database 502 for having basic data, and handles basic data； Source database 502 is oracle database, including primary database and backup library；
Call service subsystem 503 connects the target database 504 for having traffic data, and call service subsystem 503 mentions It needs to rely on basic data for call service, therefore makes basic data after marketing platform 501 operates source database Change is generated, the change of basic data needs to be synchronized to target database 503 and stored as increment traffic data；Number of targets Gauss database is used according to library 504；
The backup library of source database 502 is connected to message queue flow module 505, and message queue flow module 505 obtains backup The data to be synchronized in library generate data and change message, and push-in first only includes in the first message queue 505-1 an of subregion, The data in single partition are changed into news release to the second message queue 505-2 including multiple subregions when meeting trigger condition；
Sorting module 506 includes multiple consumer threads, and each consumer thread handles the second message queue 505-2 A subregion data change message, corresponding data to be synchronized are synchronized in target database 504.
Corresponding with a kind of method of data synchronization that the application first embodiment provides, the application fourth embodiment also provides A kind of data synchronization unit.
Referring to Fig. 6, it illustrates the data synchronization unit schematic diagrames.Since Installation practice is substantially similar to method reality Example is applied, so describing fairly simple, relevant part refers to the corresponding explanation of embodiment of the method.
Fourth embodiment provides a kind of data synchronization unit, comprising:
Data change message generation unit 601, for obtaining the data to be synchronized in source database, generate for it is described to First data of synchrodata change message；
Message subject determination unit 602, for the data affiliated in the source database according to the data to be synchronized Library table determines the message subject of the first data change message；
First message queue unit 603, for according to the message subject, will first data change message push-in and The corresponding first message queue of the message subject；Wherein, the first message queue is to only include the single partition of a subregion Message queue；
Zoning unit 604, for when meeting trigger condition, according to default partitioning strategies, by the first message queue In the first data change message distribution into the first subregion of second message queue；The second message queue be include multiple The multi partition message queue of subregion, each subregion in the multiple subregion change message for storing data；Described first Subregion is the subregion for being stored with the first data in the multiple subregion and changing message；The second message queue is synchronous for data Unit obtains data and changes message, changes message according to data and data to be synchronized are synchronized to target database.
Wherein, the data change message generation unit 601, are specifically used for:
Obtain the log of source database；
Modification information is recorded according to the log acquisition；
The data to be synchronized are obtained according to the record modification information；
Using the data to be synchronized, generates the first data and change message.
Wherein, the data change message generation unit 601, are specifically used for: according to preset data form to described to same Step data is formatted processing, using treated data are formatted, generates first data and changes message.
Wherein, the data change message generation unit 601, are specifically used for: determining and become for generating first data The message major key of more message；According to the message major key and the data to be synchronized, generates first data and change message.
Wherein, the zoning unit 604 is specifically used for:
When meeting trigger condition, the message major key of the first data change message is obtained；
According to the message major key and the default partitioning strategies, the subregion of the corresponding first data change message is determined Mark；
First data change message is sent to the subregion of the instruction of partition identification described in second message queue；It is described The subregion of partition identification instruction is first subregion.
Wherein, the zoning unit 604 includes that partition identification determines that subelement, the partition identification determine that subelement is used In:
Obtain the cryptographic Hash of the message major key；
Obtain the number of partitions of second message queue；
Using the cryptographic Hash of the message major key to the number of partitions modulus, modulus is obtained as a result, by the modulus knot Fruit is as the default partitioning strategies；
The partition identification is obtained according to the default partitioning strategies.
Wherein, the zoning unit 604 is specifically used for:
When meeting trigger condition, service class belonging to the corresponding data to be synchronized of the first data change message is obtained Type information or affiliated service area information；
Pair by the service type information or the service area information, at least one subregion of second message queue It should be related to, as the default partitioning strategies；
According to the partition identification of the specified corresponding first data change message of the default partitioning strategies；
According to the partition identification, first data change message is sent to subregion mark described in second message queue Know the subregion of instruction.
Wherein, described device further includes service terminator unit, and the service terminator unit is used for: being directed to the second message Queue starts for providing the server-side of data change message；The server-side is used to provide the number to data synchronisation unit According to change message.
Corresponding with a kind of method of data synchronization that the application second embodiment provides, the 5th embodiment of the application also provides A kind of data synchronization unit.
Referring to Fig. 7, it illustrates the data synchronization unit schematic diagrames.Since Installation practice is substantially similar to method reality Example is applied, so describing fairly simple, relevant part refers to the corresponding explanation of embodiment of the method.
5th embodiment provides a kind of data synchronization unit, is applied to data synchronisation unit, comprising:
It indicates acquisition of information subelement 701, is obtained in the first subregion of second message queue for obtaining to be used to indicate The instruction information of first data change message；Wherein, the data synchronisation unit is used for the data to be synchronized in source database It is synchronized to target database, the second message queue includes multiple subregions, and each subregion in the multiple subregion is used for Storing data changes message, and first subregion is to be stored with first data to change message and list synchronous with the data The corresponding subregion of member, the first data change message are the data change message for data to be synchronized；
First data change message and obtain subelement 702, become for obtaining first data from first subregion More message；
Data acquisition subelement 703 to be synchronized obtains the number to be synchronized for changing message according to first data According to；
To the synchronous subelement 704 of target database, for the data to be synchronized to be synchronized to the target database.
Wherein, described device further includes multithreading subelement, and the multithreading subelement is used for: using at least one thread The data, which are obtained, from second message queue changes message；
Correspondingly, the first data change message obtains subelement 702 and is also used to: using at least one described thread First thread, first data change message is obtained from first subregion.
Wherein, the multithreading subelement, is specifically used for: for each of multiple subregions of second message queue point Area starts corresponding thread；The data change message is obtained from corresponding subregion using the thread of starting.
Wherein, described device further includes client child unit, and the client child unit is used for:
Using the thread as client；
Configure the service client information of the client parameter and corresponding second message queue；
Using the client parameter and the service client information, corresponding with the thread point is obtained from the server-side Data in area change message.
Wherein, the multithreading subelement, is specifically used for:
Determine the message subject of at least one thread process；
According to the message subject, at least one subregion in multiple subregions of the second message queue is monitored；
The data listened to are obtained using at least one described thread changes message.
Wherein, described to be specifically used for the synchronous subelement 704 of target database:
Message is changed according to first data, is generated for the number to target database first database table synchrodata According to library synchronic command；
It is instructed using the database synchronization, the data to be synchronized is synchronized to the first database table；Wherein, institute The structure for stating first database table, different from the structure of synchrodata database table affiliated in source database.
In a typical configuration, calculating equipment includes one or more processors (CPU), input/output interface, net Network interface and memory.
Memory may include the non-volatile memory in computer-readable medium, random access memory (RAM) and/or The forms such as Nonvolatile memory, such as read-only memory (ROM) or flash memory (flash RAM).Memory is computer-readable medium Example.
1, computer-readable medium can be by any side including permanent and non-permanent, removable and non-removable media Method or technology realize that information stores.Information can be computer readable instructions, data structure, the module of program or other numbers According to.The example of the storage medium of computer includes, but are not limited to phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other kinds of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory techniques, CD-ROM are read-only Memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape magnetic disk storage or Other magnetic storage devices or any other non-transmission medium, can be used for storage can be accessed by a computing device information.According to Herein defines, and computer-readable medium does not include non-temporary computer readable media (transitory media), such as modulates Data-signal and carrier wave.
2, it will be understood by those skilled in the art that embodiments herein can provide as the production of method, system or computer program Product.Therefore, complete hardware embodiment, complete software embodiment or embodiment combining software and hardware aspects can be used in the application Form.It can be used moreover, the application can be used in the computer that one or more wherein includes computer usable program code The computer program product implemented on storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) Form.
Although the application is disclosed as above with preferred embodiment, it is not for limiting the application, any this field skill Art personnel are not departing from spirit and scope, can make possible variation and modification, therefore the guarantor of the application Shield range should be subject to the range that the claim of this application defined.

Claims (10)

1. a kind of method of data synchronization characterized by comprising
The data to be synchronized in source database are obtained, the first data generated for the data to be synchronized change message；
According to the data to be synchronized database table affiliated in the source database, the first data change message is determined Message subject；
According to the message subject, first data change message is pushed into first message corresponding with message subject team Column；Wherein, the first message queue is the single partition message queue for only including a subregion；
When meeting trigger condition, according to default partitioning strategies, the first data in the first message queue are changed into message It is distributed in the first subregion of second message queue；The second message queue is the multi partition message team for including multiple subregions It arranges, each subregion in the multiple subregion changes message for storing data；First subregion is the multiple subregion In be stored with the first data change message subregion；The second message queue obtains data change for data synchronisation unit and disappears Breath changes message according to data and data to be synchronized is synchronized to target database.
2. the method according to claim 1, wherein the data to be synchronized obtained in source database, generate Message is changed for the first data of the data to be synchronized, comprising:
Obtain the log of source database；
Modification information is recorded according to the log acquisition；
The data to be synchronized are obtained according to the record modification information；
Using the data to be synchronized, generates the first data and change message.
3. according to the method described in claim 2, it is characterized in that, described use the data to be synchronized, the first data of generation Change message, comprising:
Processing is formatted to the data to be synchronized according to preset data form, it is raw using formatting treated data Message is changed at first data.
4. the method according to claim 1, wherein the data to be synchronized obtained in source database, generate Message is changed for the first data of the data to be synchronized, comprising:
Determine the message major key for generating the first data change message；
According to the message major key and the data to be synchronized, generates first data and change message.
5. according to the method described in claim 4, it is characterized in that, described when meeting trigger condition, according to default subregion plan Slightly, by the first data change message distribution in the first message queue into the first subregion of second message queue, comprising:
When meeting trigger condition, the message major key of the first data change message is obtained；
According to the message major key and the default partitioning strategies, the subregion mark of the corresponding first data change message is determined Know；
First data change message is sent to the subregion of the instruction of partition identification described in second message queue；The subregion The subregion of mark instruction is first subregion.
6. according to the method described in claim 5, it is characterized in that, described according to the message major key and the default subregion plan Slightly, the partition identification of the corresponding first data change message is determined, comprising:
Obtain the cryptographic Hash of the message major key；
Obtain the number of partitions of second message queue；
Using the cryptographic Hash of the message major key to the number of partitions modulus, modulus is obtained as a result, the modulus result is made For the default partitioning strategies；
The partition identification is obtained according to the default partitioning strategies.
7. the method according to claim 1, wherein described when meeting trigger condition, according to default subregion plan Slightly, by the first data change message distribution in the first message queue into the first subregion of second message queue, comprising:
When meeting trigger condition, type of service letter belonging to the corresponding data to be synchronized of the first data change message is obtained Breath or affiliated service area information；
It is corresponding at least one subregion of second message queue to close by the service type information or the service area information System, as the default partitioning strategies；
According to the partition identification of the specified corresponding first data change message of the default partitioning strategies；
According to the partition identification, first data change message is sent to partition identification described in second message queue and is referred to The subregion shown.
8. the method according to claim 1, wherein further include:
For the second message queue, start for providing the server-side of data change message；The server-side is used for number The data, which are provided, according to synchronization unit changes message.
9. a kind of method of data synchronization characterized by comprising
Data synchronisation unit obtains to be used to indicate obtains the first data change message in the first subregion of second message queue Indicate information；Wherein, the data synchronisation unit is used to the data to be synchronized in source database being synchronized to target database, institute Stating second message queue includes multiple subregions, and each subregion in the multiple subregion changes message for storing data, institute Stating the first subregion is to be stored with first data to change message and subregion corresponding with the data synchronisation unit, described the It is the data change message for data to be synchronized that one data, which change message,；
The data synchronisation unit obtains the first data change message from first subregion；
The data synchronisation unit changes message according to first data, obtains the data to be synchronized；
The data to be synchronized are synchronized to the target database by the data synchronisation unit.
10. a kind of database synchronous system characterized by comprising source database, message queue flow module, data synchronize list Member, target database；
The source database needs to be synchronized to the data to be synchronized of the target database for providing, the number to be synchronized According to including basic data for insurance business；
The message queue flow module generates for obtaining the data to be synchronized in source database and is directed to the data to be synchronized The first data change message；According to the data to be synchronized in the source database belonging to database table, determine described in The message subject of first data change message；According to the message subject, by first data change message push-in with it is described The corresponding first message queue of message subject, wherein the first message queue is to only include the single partition message an of subregion Queue；When meeting trigger condition, according to default partitioning strategies, the first data in the first message queue are changed into message It is distributed in the first subregion of second message queue, the second message queue is the multi partition message team for including multiple subregions Column, each subregion in the multiple subregion change message for storing data, and first subregion is to be stored with described the One data change message and subregion corresponding with the data synchronisation unit, and the second message queue synchronizes list for data Member obtains data and changes message, changes message according to data and data to be synchronized are synchronized to the target database；
The data synchronisation unit, for for each subregion in multiple subregions of second message queue, starting to divide with each The corresponding thread in area obtains the data using the thread of starting from corresponding subregion and changes message, by source database In data to be synchronized be synchronized to target database, wherein the second message queue includes multiple subregions, the multiple subregion In each subregion change message for storing data；Acquisition is used to indicate to be obtained in the first subregion of second message queue First data change the instruction information of message, and using the first thread in the thread of starting, the is obtained from first subregion One data change message, wherein the first data change message is the data change message for data to be synchronized；According to institute The first data change message is stated, the data to be synchronized is obtained, the data to be synchronized is synchronized to the target database；
The target database, for receiving the synchronous data to be synchronized of the data synchronisation unit.
CN201811552328.9A 2018-12-18 2018-12-18 Data synchronization method, device and system Active CN109739929B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811552328.9A CN109739929B (en) 2018-12-18 2018-12-18 Data synchronization method, device and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811552328.9A CN109739929B (en) 2018-12-18 2018-12-18 Data synchronization method, device and system

Publications (2)

Publication Number Publication Date
CN109739929A true CN109739929A (en) 2019-05-10
CN109739929B CN109739929B (en) 2021-03-16

Family

ID=66360484

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811552328.9A Active CN109739929B (en) 2018-12-18 2018-12-18 Data synchronization method, device and system

Country Status (1)

Country Link
CN (1) CN109739929B (en)

Cited By (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110321387A (en) * 2019-07-10 2019-10-11 中国联合网络通信集团有限公司 Method of data synchronization, equipment and terminal device
CN110471896A (en) * 2019-06-21 2019-11-19 深圳中琛源科技股份有限公司 A kind of data processing method, system and server
CN110516124A (en) * 2019-08-09 2019-11-29 济南浪潮数据技术有限公司 A kind of document analysis method, apparatus and computer readable storage medium
CN110807067A (en) * 2019-09-29 2020-02-18 北京淇瑀信息科技有限公司 Data synchronization method, device and equipment for relational database and data warehouse
CN110852778A (en) * 2019-09-30 2020-02-28 口口相传(北京)网络技术有限公司 Data processing method and device for business object
CN110941623A (en) * 2019-11-12 2020-03-31 北京达佳互联信息技术有限公司 Data synchronization method and device
CN111026774A (en) * 2019-12-03 2020-04-17 深圳前海环融联易信息科技服务有限公司 Data sequence synchronization method and device, computer equipment and storage medium
CN111013133A (en) * 2019-11-29 2020-04-17 北京奇艺世纪科技有限公司 Data processing method and device
CN111209126A (en) * 2020-01-03 2020-05-29 北京明略软件系统有限公司 Data transmission method and device between microservices and electronic equipment
CN111400407A (en) * 2020-04-10 2020-07-10 浙江大华技术股份有限公司 Data synchronization method and device, storage medium and electronic device
CN111752910A (en) * 2020-06-24 2020-10-09 上海微盟企业发展有限公司 Data synchronization method, system and related device for heterogeneous platform
CN111813868A (en) * 2020-08-13 2020-10-23 中国工商银行股份有限公司 Data synchronization method and device
CN111831748A (en) * 2020-06-30 2020-10-27 北京小米松果电子有限公司 Data synchronization method, device and storage medium
CN111880955A (en) * 2020-08-04 2020-11-03 北京立通智联科技有限公司 Message processing method, device, terminal and medium for Internet of things equipment
CN112333083A (en) * 2020-10-30 2021-02-05 平安付科技服务有限公司 Transaction information processing method and device, computer equipment and computer readable medium
CN112463886A (en) * 2020-11-30 2021-03-09 浙江大华技术股份有限公司 Data processing method and device, electronic equipment and storage medium
CN112597247A (en) * 2020-12-25 2021-04-02 杭州数梦工场科技有限公司 Data synchronization method and device
CN112612799A (en) * 2020-12-08 2021-04-06 福建天泉教育科技有限公司 Data synchronization method and terminal
CN112612583A (en) * 2020-12-16 2021-04-06 平安消费金融有限公司 Data synchronization method and device, computer equipment and readable storage medium
CN112685426A (en) * 2021-01-21 2021-04-20 浪潮云信息技术股份公司 NiFi-based Kafka consumption NewSQL CDC stream data conversion method
CN112905706A (en) * 2021-03-19 2021-06-04 平安消费金融有限公司 Database synchronization method and device, storage medium and computer equipment
CN112925743A (en) * 2021-02-07 2021-06-08 中国工商银行股份有限公司 File generation method and device and storage medium
CN113064950A (en) * 2021-03-18 2021-07-02 北京沃东天骏信息技术有限公司 Data synchronization method, device, equipment and storage medium
CN113220791A (en) * 2021-06-03 2021-08-06 西安热工研究院有限公司 Data cascade synchronization system and method
CN113312192A (en) * 2021-06-07 2021-08-27 平安证券股份有限公司 Data synchronization method and device based on window, electronic equipment and storage medium
CN113407637A (en) * 2021-07-13 2021-09-17 上海浦东发展银行股份有限公司 Data synchronization method and device, electronic equipment and storage medium
CN113609199A (en) * 2021-07-27 2021-11-05 远景智能国际私人投资有限公司 Database system, server, and storage medium
CN113722390A (en) * 2021-09-03 2021-11-30 小马国炬(玉溪)科技有限公司 Data storage method and system
CN115391361A (en) * 2022-08-24 2022-11-25 国任财产保险股份有限公司 Real-time data processing method and device based on distributed database
CN117349384A (en) * 2023-12-04 2024-01-05 四川才子软件信息网络有限公司 Database synchronization method, system and equipment

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005284781A (en) * 2004-03-30 2005-10-13 Nomura Research Institute Ltd Mq data synchronizing system and mq data synchronizing program
CN103905503A (en) * 2012-12-27 2014-07-02 中国移动通信集团公司 Data storage method, data scheduling method, device and system
CN104572689A (en) * 2013-10-17 2015-04-29 腾讯科技（深圳）有限公司 Data synchronizing method, device and system
WO2017141229A1 (en) * 2016-02-21 2017-08-24 Geir Christian Karlsen System and method for securely exchanging data between devices
CN107783975A (en) * 2016-08-24 2018-03-09 北京京东尚科信息技术有限公司 The method and apparatus of distributed data base synchronization process
CN107844524A (en) * 2017-10-12 2018-03-27 金蝶软件（中国）有限公司 Data processing method, data processing equipment, computer equipment and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005284781A (en) * 2004-03-30 2005-10-13 Nomura Research Institute Ltd Mq data synchronizing system and mq data synchronizing program
CN103905503A (en) * 2012-12-27 2014-07-02 中国移动通信集团公司 Data storage method, data scheduling method, device and system
CN104572689A (en) * 2013-10-17 2015-04-29 腾讯科技（深圳）有限公司 Data synchronizing method, device and system
WO2017141229A1 (en) * 2016-02-21 2017-08-24 Geir Christian Karlsen System and method for securely exchanging data between devices
CN107783975A (en) * 2016-08-24 2018-03-09 北京京东尚科信息技术有限公司 The method and apparatus of distributed data base synchronization process
CN107844524A (en) * 2017-10-12 2018-03-27 金蝶软件（中国）有限公司 Data processing method, data processing equipment, computer equipment and storage medium

Cited By (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110471896A (en) * 2019-06-21 2019-11-19 深圳中琛源科技股份有限公司 A kind of data processing method, system and server
CN110471896B (en) * 2019-06-21 2023-11-28 陕西融华电子科技有限公司 Data processing method, system and server
CN110321387A (en) * 2019-07-10 2019-10-11 中国联合网络通信集团有限公司 Method of data synchronization, equipment and terminal device
CN110321387B (en) * 2019-07-10 2022-02-01 中国联合网络通信集团有限公司 Data synchronization method, equipment and terminal equipment
CN110516124B (en) * 2019-08-09 2022-04-22 济南浪潮数据技术有限公司 File analysis method and device and computer readable storage medium
CN110516124A (en) * 2019-08-09 2019-11-29 济南浪潮数据技术有限公司 A kind of document analysis method, apparatus and computer readable storage medium
CN110807067A (en) * 2019-09-29 2020-02-18 北京淇瑀信息科技有限公司 Data synchronization method, device and equipment for relational database and data warehouse
CN110807067B (en) * 2019-09-29 2023-12-22 北京淇瑀信息科技有限公司 Data synchronization method, device and equipment for relational database and data warehouse
CN110852778A (en) * 2019-09-30 2020-02-28 口口相传(北京)网络技术有限公司 Data processing method and device for business object
CN110941623A (en) * 2019-11-12 2020-03-31 北京达佳互联信息技术有限公司 Data synchronization method and device
CN111013133A (en) * 2019-11-29 2020-04-17 北京奇艺世纪科技有限公司 Data processing method and device
CN111026774A (en) * 2019-12-03 2020-04-17 深圳前海环融联易信息科技服务有限公司 Data sequence synchronization method and device, computer equipment and storage medium
CN111209126A (en) * 2020-01-03 2020-05-29 北京明略软件系统有限公司 Data transmission method and device between microservices and electronic equipment
CN111400407A (en) * 2020-04-10 2020-07-10 浙江大华技术股份有限公司 Data synchronization method and device, storage medium and electronic device
CN111400407B (en) * 2020-04-10 2023-09-26 浙江大华技术股份有限公司 Data synchronization method and device, storage medium and electronic device
CN111752910A (en) * 2020-06-24 2020-10-09 上海微盟企业发展有限公司 Data synchronization method, system and related device for heterogeneous platform
CN111831748A (en) * 2020-06-30 2020-10-27 北京小米松果电子有限公司 Data synchronization method, device and storage medium
CN111831748B (en) * 2020-06-30 2024-04-30 北京小米松果电子有限公司 Data synchronization method, device and storage medium
CN111880955A (en) * 2020-08-04 2020-11-03 北京立通智联科技有限公司 Message processing method, device, terminal and medium for Internet of things equipment
CN111813868B (en) * 2020-08-13 2023-11-10 中国工商银行股份有限公司 Data synchronization method and device
CN111813868A (en) * 2020-08-13 2020-10-23 中国工商银行股份有限公司 Data synchronization method and device
CN112333083A (en) * 2020-10-30 2021-02-05 平安付科技服务有限公司 Transaction information processing method and device, computer equipment and computer readable medium
CN112463886A (en) * 2020-11-30 2021-03-09 浙江大华技术股份有限公司 Data processing method and device, electronic equipment and storage medium
CN112612799A (en) * 2020-12-08 2021-04-06 福建天泉教育科技有限公司 Data synchronization method and terminal
CN112612583A (en) * 2020-12-16 2021-04-06 平安消费金融有限公司 Data synchronization method and device, computer equipment and readable storage medium
CN112597247B (en) * 2020-12-25 2022-05-31 杭州数梦工场科技有限公司 Data synchronization method and device
CN112597247A (en) * 2020-12-25 2021-04-02 杭州数梦工场科技有限公司 Data synchronization method and device
CN112685426A (en) * 2021-01-21 2021-04-20 浪潮云信息技术股份公司 NiFi-based Kafka consumption NewSQL CDC stream data conversion method
CN112925743A (en) * 2021-02-07 2021-06-08 中国工商银行股份有限公司 File generation method and device and storage medium
CN113064950A (en) * 2021-03-18 2021-07-02 北京沃东天骏信息技术有限公司 Data synchronization method, device, equipment and storage medium
CN113064950B (en) * 2021-03-18 2024-04-16 北京沃东天骏信息技术有限公司 Data synchronization method, device, equipment and storage medium
CN112905706A (en) * 2021-03-19 2021-06-04 平安消费金融有限公司 Database synchronization method and device, storage medium and computer equipment
CN113220791A (en) * 2021-06-03 2021-08-06 西安热工研究院有限公司 Data cascade synchronization system and method
CN113220791B (en) * 2021-06-03 2023-07-28 西安热工研究院有限公司 Data cascading synchronization system and method
CN113312192A (en) * 2021-06-07 2021-08-27 平安证券股份有限公司 Data synchronization method and device based on window, electronic equipment and storage medium
CN113407637A (en) * 2021-07-13 2021-09-17 上海浦东发展银行股份有限公司 Data synchronization method and device, electronic equipment and storage medium
CN113609199B (en) * 2021-07-27 2023-09-12 远景智能国际私人投资有限公司 Database system, server, and storage medium
CN113609199A (en) * 2021-07-27 2021-11-05 远景智能国际私人投资有限公司 Database system, server, and storage medium
CN113722390A (en) * 2021-09-03 2021-11-30 小马国炬(玉溪)科技有限公司 Data storage method and system
CN115391361A (en) * 2022-08-24 2022-11-25 国任财产保险股份有限公司 Real-time data processing method and device based on distributed database
CN117349384A (en) * 2023-12-04 2024-01-05 四川才子软件信息网络有限公司 Database synchronization method, system and equipment
CN117349384B (en) * 2023-12-04 2024-03-15 四川才子软件信息网络有限公司 Database synchronization method, system and equipment

Also Published As

Publication number Publication date
CN109739929B (en) 2021-03-16

Similar Documents

Publication Publication Date Title
CN109739929A (en) Method of data synchronization, apparatus and system
US7849227B2 (en) Stream data processing method and computer systems
CN104809201B (en) A kind of method and apparatus of database synchronization
CN104809200B (en) A kind of method and apparatus of database synchronization
US10223437B2 (en) Adaptive data repartitioning and adaptive data replication
CN112231402A (en) Real-time synchronization method, device, equipment and storage medium for heterogeneous data
CN111860667A (en) Method and device for determining equipment fault, storage medium and electronic device
CN112559475A (en) Data real-time capturing and transmitting method and system
CN107040576A (en) Information-pushing method and device, communication system
CN106844694B (en) Method and apparatus for synchrodata
CN109451078A (en) Transaction methods and device under a kind of distributed structure/architecture
CN105069029A (en) Real-time ETL (extraction-transformation-loading) system and method
CN110362456A (en) A kind of method and device obtaining server-side performance data
CN108182204A (en) The processing method and processing device of data query based on house prosperity transaction multi-dimensional data
CN107016075A (en) Company-data synchronous method and device
EP3905064A1 (en) Method and apparatus for synchronously replicating database
CN110909072B (en) Data table establishment method, device and equipment
CN116186082A (en) Data summarizing method based on distribution, first server and electronic equipment
CN116304390A (en) Time sequence data processing method and device, storage medium and electronic equipment
CN110333984A (en) Interface method for detecting abnormality, device, server and system
CN110019274A (en) A kind of Database Systems and the method and apparatus for inquiring database
CN115344633A (en) Data processing method, device, equipment and storage medium
CN115269719A (en) Data synchronization method and device
CN113010385B (en) Task state updating method, device, equipment and medium
CN109063201B (en) Impala online interactive query method based on mixed storage scheme

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant