CN107301196A

CN107301196A - Method of data synchronization, device, equipment and monitoring server

Info

Publication number: CN107301196A
Application number: CN201710305043.4A
Authority: CN
Inventors: 梁福坤
Original assignee: Beijing Xiaodu Information Technology Co Ltd
Current assignee: Beijing Xiaodu Information Technology Co Ltd
Priority date: 2017-05-03
Filing date: 2017-05-03
Publication date: 2017-10-27

Abstract

The embodiments of the invention provide a kind of method of data synchronization, device, equipment and server, it is related to technical field of data processing.Wherein, creation data is write source database system by data producer, and issues the data update notification of creation data, and monitoring server monitors the data update notification that whether there is for purpose Database Systems；The data update notification is what data producer was generated according to creation data；If there is the data update notification for the purpose Database Systems, updating the data for the purpose Database Systems request synchronization is obtained from source database system；Updated the data described in loading to the purpose Database Systems.The embodiment of the present invention improves the promptness of data renewal.

Description

Method of data synchronization, device, equipment and monitoring server

Technical field

Filled the present embodiments relate to technical field of data processing, more particularly to a kind of method of data synchronization, data syn-chronization Put, data production equipment, information storaging apparatus and monitoring server.

Background technology

Heterogeneous database system refers to the set of multiple Database Systems of correlation, in actual applications, data production one As just for one of Database Systems, namely the data of data producer production are only placed in one of Database Systems In, other Database Systems can carry out data renewal by data syn-chronization mode.

In the prior art, it is typically according to the data syn-chronization for any purpose Database Systems by a monitoring server Demand configuration querying task, and query task is performed according to the designated time, demand number is obtained from source database system with routine According to, then demand data is loaded onto in purpose Database Systems, updated with the data for realizing purpose Database Systems.

But existing this data syn-chronization mode, it is impossible to meet the promptness of data renewal.

The content of the invention

In view of this, set the embodiments of the invention provide a kind of method of data synchronization, data synchronization unit, data production Standby, information storaging apparatus and monitoring server, the poor technical problem of promptness is updated for solving data.

In a first aspect, a kind of method of data synchronization is provided in the embodiment of the present invention, including：

Monitor the data update notification that whether there is for purpose Database Systems；The data update notification is given birth to for data Production root is generated according to creation data；

If there is the data update notification for the purpose Database Systems, the mesh is obtained from source database system Database Systems request synchronous update the data；

Updated the data described in loading to the purpose Database Systems.

A kind of method of data synchronization is provided in second aspect, the embodiment of the present invention, including：

Creation data is write to source database system；

According to the creation data, data update notification is generated；

Issue the data update notification；It is logical that the data update notification determines that the data update for monitoring server Know corresponding purpose Database Systems, and the purpose Database Systems request synchronization is obtained more from the source database system New data, and updated the data described in loading to the purpose Database Systems.

A kind of method of data synchronization is provided in the third aspect, the embodiment of the present invention, including：

Monitor monitoring server and be directed to the topic of subscription that purpose Database Systems are registered；The topic of subscription is by the monitoring The synchronous data identification that server is registered according to purpose Database Systems is determined；

If the topic of subscription receives data update notification, initiate to have adjusted instruction to the monitoring server, for institute State that monitoring server response is described to have adjusted instruction, synchronization is asked more from the source database system acquisition purpose Database Systems New data；

Wherein, the data update notification is generated according to creation data by data producer and pushes to the creation data The corresponding topic of subscription.

A kind of data synchronization unit is provided in fourth aspect, the embodiment of the present invention, including：

Notify to monitor module, monitor the data update notification that whether there is for purpose Database Systems；The data are more It is new to notify what is generated for data producer according to creation data；

Data acquisition module, for if there is the data update notification for the purpose Database Systems, from source number Updating the data for the purpose Database Systems request synchronization is obtained according to storehouse system；

Data load-on module, for loading described update the data to the purpose Database Systems.

A kind of data synchronization unit is provided in 5th aspect, the embodiment of the present invention, including：

Data write. module, for writing creation data to source database system；

Generation module is notified, for according to the creation data, generating data update notification；

Release module is notified, for issuing the data update notification；The data update notification is used for monitoring server The corresponding purpose Database Systems of the data update notification are determined, and the purpose data are obtained from the source database system Storehouse system request is synchronous to be updated the data, and is updated the data described in loading to the purpose Database Systems.

A kind of data synchronization unit is provided in 6th aspect, the embodiment of the present invention, including：

Theme monitors module, and the topic of subscription that purpose Database Systems are registered is directed to for monitoring monitoring server；It is described The synchronous data identification that topic of subscription is registered by the monitoring server according to purpose Database Systems is determined；

Module has been adjusted, if receiving data update notification for the topic of subscription, has initiated to adjust to the monitoring server Instruction is played, instruction has been adjusted so that monitoring server response is described, the purpose data base set is obtained from source database system System request is synchronous to be updated the data；

7th aspect, provide in the embodiment of the present invention a kind of monitoring server including one or more memories and One or more processors；

One or more of computer instructions of memory storage one or more, one or more computer instruction For calling execution by one or more of processors；

One or more of processors are used for：

Updated the data described in loading to the purpose Database Systems.

In eighth aspect, the embodiment of the present invention provide a kind of data production equipment, including one or more memories with And one or more processors；

One or more of processors are used for：

Creation data is write to source database system；

According to the creation data, data update notification is generated；

9th aspect, provide in the embodiment of the present invention a kind of information storaging apparatus, including one or more memories with And one or more processors；

One or more of processors are used for：

In the embodiment of the present invention, data producer writes creation data after source database system, can be according to production number According to, creation data update notification, and data update notification is issued, monitoring server monitors the number for purpose Database Systems After update notification, you can to take the purpose Database Systems to ask synchronous update the data from source database system；And Loading is updated the data to purpose Database Systems, realizes the data syn-chronization of purpose Database Systems, using notice and monitoring side Formula so that can will update the data in time and be synchronized to purpose Database Systems, improves the promptness of data renewal.

The aspects of the invention or other aspects can more straightforwards in the following description.

Brief description of the drawings

In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing There is the accompanying drawing used required in technology description to be briefly described, it should be apparent that, drawings in the following description are this hairs Some bright embodiments, for those of ordinary skill in the art, on the premise of not paying creative work, can be with root Other accompanying drawings are obtained according to these accompanying drawings.

Fig. 1 is a kind of flow chart of method of data synchronization one embodiment that the present invention is provided；

Fig. 2 is a kind of flow chart for another embodiment of method of data synchronization that the present invention is provided；

Fig. 3 is a kind of flow chart for another embodiment of method of data synchronization that the present invention is provided；

Fig. 4 is a kind of signaling diagram for another embodiment of method of data synchronization that the present invention is provided；

Fig. 5 is a kind of structural representation of data synchronization unit one embodiment that the present invention is provided；

Fig. 6 is a kind of structural representation of monitoring server one embodiment that the present invention is provided；

Fig. 7 is a kind of structural representation for another embodiment of data synchronization unit that the present invention is provided；

Fig. 8 is a kind of structural representation of data production equipment one embodiment that the present invention is provided；

Fig. 9 is a kind of structural representation for another embodiment of data synchronization unit that the present invention is provided；

Figure 10 is a kind of structural representation of information storaging apparatus one embodiment that the present invention is provided.

Embodiment

In order that those skilled in the art more fully understand the present invention program, below in conjunction with the embodiment of the present invention Accompanying drawing, the technical scheme in the embodiment of the present invention is clearly and completely described.

In some flows of description in description and claims of this specification and above-mentioned accompanying drawing, contain according to Particular order occur multiple operations, but it should be clearly understood that these operation can not herein occur according to it is suitable Sequence is performed or performed parallel, and the sequence number such as 101,102 etc. of operation is only used for distinguishing each different operation, sequence number Any execution sequence is not represented for itself.In addition, these flows can include more or less operations, and these operations can To perform or perform parallel in order.It should be noted that the description such as " first ", " second " herein, is to be used to distinguish not Same message, equipment, module etc., does not represent sequencing, it is different types also not limit " first " and " second ".

The technical scheme of the embodiment of the present invention is primary to be applied in heterogeneous database system.Heterogeneous database system is being counted greatly It is widely used according to process field, particularly the arrival of electronic commerce times, is inquired about and analyzed by big data, it will be appreciated that is used Family consumption habit etc., improve service quality.The data source types of multiple Database Systems in heterogeneous database system can be with Difference, for example, can include Mysql (a kind of Relational DBMS), a hive (data bins based on Hadoop Storehouse instrument), impala (a kind of data query system), greenplum (a kind of data query engine), a kind of presto (data Query engine), kylin (distributed analysis engine software of increasing income), hbase (a kind of distributed memory system) etc., to meet Different pieces of information analyzes demand.

As described in the background art, in the prior art, by monitoring server according to for any purpose Database Systems Data syn-chronization demand configuration querying task, and according to the designated time perform query task, with routine from source database system Demand data is obtained, then demand data is loaded onto in purpose Database Systems, to realize the data of purpose Database Systems more Newly.Wherein source database system refers to that data producer stores any one Database Systems of creation data, and purpose database System refers to the Database Systems that there is data syn-chronization demand.

Wherein, the data of data producer production are typically batch data, the demand number obtained from source database system According to being also to be loaded onto in purpose database to realize batch updating in Load (loading) mode.

Inventor has found under study for action, using the technical scheme of prior art, and customary execution interval is empirically data Setting, and need to be mutually synchronized data between multiple Database Systems, disparate databases system may need synchronous same number According to the data in the system of storehouse, cause synchronous interaction process extremely complex, it is impossible to accomplish that data upgrade in time.

In order to solve the technical problem that data update promptness difference, inventor is by a series of skills for researching and proposing the present invention Art scheme, in embodiments of the present invention, data producer are carried out after data production, are generated according to creation data and are issued data Update notification, monitoring server is monitored according to the synchronisation requirement of purpose Database Systems, and purpose database is directed to if existing The data update notification of system, then updating the data for purpose Database Systems request synchronization is obtained from source database system, and Purpose Database Systems are loaded onto, the data for completing purpose Database Systems update, using notice and listening mode so that can Purpose Database Systems will be synchronized to timely will update the data, the promptness of data renewal is improved.

Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete Site preparation is described, it is clear that described embodiment is only a part of embodiment of the invention, rather than whole embodiments.It is based on Embodiment in the present invention, the every other implementation that those skilled in the art are obtained under the premise of creative work is not made Example, belongs to the scope of protection of the invention.

Fig. 1 is a kind of flow chart of method of data synchronization one embodiment provided in an embodiment of the present invention, the present embodiment Technical scheme is mainly used in monitoring server, and this method can include following steps：

101：Monitor the data update notification that whether there is for purpose Database Systems.If so, step 102 is performed, it is no Then continue to monitor.

Wherein, the data update notification is what data producer was generated according to creation data；

102：Updating the data for the purpose Database Systems request synchronization is obtained from source database system.

103：Updated the data described in loading to the purpose Database Systems.

Wherein, source database system refers to that data producer stores any one Database Systems of creation data, and it can be with It is the Database Systems of arbitrary data Source Type, such as Mysql, hive, impala, data producer is according to creation data institute The data source types of category, are stored into corresponding source database system.And purpose Database Systems are to refer to exist from source data Any database system of data syn-chronization is carried out in the system of storehouse.

In the embodiment of the present invention, while data producer stores creation data to source database system, data are generated Update notification.

The creation data mark and data more new range of creation data can be included in the data update notification.Wherein, The creation data represents that the library name of database, the table name of tables of data and/or the field name belonging to creation data can be included Claim etc..Data more new range can include time range and number range, time range define creation data it is corresponding when Between information, number range defines numerical values recited of creation data etc..

In the present embodiment, after data producer creation data update notification, if the data update notification is listened to, then from Obtained in source database system and update the data and be loaded onto in purpose Database Systems so that once there is the progress of data producer After data production, synchronous updating the data can be needed to be loaded onto purpose the purpose Database Systems in creation data in time In Database Systems, it is ensured that the promptness that data update.

Wherein, what the request of purpose Database Systems was synchronous update the data can be according to the registration requests of purpose Database Systems It is determined that, monitoring server can receive the registration request for purpose Database Systems, and the registration request can pass through request end Send.Therefore, in certain embodiments, before the topic of subscription determines step, methods described can also include：

Receive the registration request that request end is initiated.

In the registration request can include synchronous data identification, the synchronous data identification include library name, table name and/or Field name etc., thus according to synchronous data identification can determine in creation data to should synchronous data identification renewal number According to.Certainly, this, which is updated the data, not necessarily includes the corresponding total data of synchronous data identification.

The request end can be the client that user oriented is used or user circle being integrated into monitoring server Face interaction end.

Wherein, checking information of purpose Database Systems etc. can also be included in the registration request, monitoring server is to mesh Database Systems be verified after, can also be succeeded in registration message etc. to request end feedback.

Wherein, obtained from source database system after updating the data, can also be according to the data lattice of purpose Database Systems Formula requirement, by update the data switch to be adapted to purpose Database Systems purpose data format, with will can update the data successfully plus It is loaded onto in purpose Database Systems.

In addition, monitoring server can respectively be monitored for disparate databases system, known whether in time with facilitating In the presence of the corresponding data update notification of each Database Systems, in practical application, the creation data of data producer is typically batch Data are measured, purpose Database Systems are also batch updating.In order to ensure information transmit-receive promptness, " Publish-subscribe " can be used Mode realize the monitoring of data update notification.

Therefore, in certain embodiments, monitoring the data update notification that whether there is for purpose Database Systems can be with Including：

The synchronous data identification registered according to the purpose Database Systems, determines that the synchronous data identification is corresponding and orders Read theme (topic)；The synchronous data identification includes library name, table name and/or field name；

Monitor whether the topic of subscription receives data update notification；The data update notification is specially the data life Production root is generated according to creation data, and pushes to the corresponding topic of subscription of the creation data.

That is, data producer can be identified according to the creation data of creation data, determine that the creation data is corresponding Topic of subscription；The data update notification of the generation correspondence topic of subscription；And the data update notification is pushed to the subscription Theme；When listening to the topic of subscription so as to the monitoring server and receive data update notification, you can with from the source data Storehouse system obtains updating the data for the purpose Database Systems request synchronization.

Creation data mark includes the corresponding library name of creation data, table name and/or field name etc..

In addition, information transmit-receive for convenience, can enter row information using MQ (Message Queue, message queue) mode Transmission, sets message-oriented middleware, is exchanged by message-oriented middleware completion message.

Therefore, alternatively, whether the monitoring topic of subscription, which receives data update notification, to include：

The topic of subscription is monitored in part registration between in the message, to monitor whether receive data update notification；

It is described to obtain updating the data and including for the purpose Database Systems request synchronization from source database system：

Instruction is played in response to the tune of the message-oriented middleware, obtaining the purpose Database Systems from source database system please Step of seeking common ground is updated the data；It is described to have adjusted instruction to detect that the topic of subscription receives data update notification by the message-oriented middleware When send.

Namely data producer creation data update notification, can be with and after determining the corresponding topic of subscription of creation data The topic of subscription specifically by data update notification into message-oriented middleware.

Monitoring server in the message between part registration monitor the topic of subscription, if message-oriented middleware detect the topic of subscription receive During to data update notification, it can initiate to have adjusted instruction to monitoring server, to trigger monitoring server from source database system Middle acquisition is updated the data.

In addition, in order to improve data accuracy, it is ensured that meet purpose data from updating the data for source database system acquisition The data demand of storehouse system, to avoid failure of data synchronization, can be supervised to updating the data carry out data check, therefore, alternatively Control server from source database system obtain purpose Database Systems request it is synchronous update the data after, can also examine Whether described update the data meets verification requirement, if so, reloading into purpose Database Systems, otherwise can return and continue to hold Row obtains the step of updating the data of the purpose Database Systems request synchronization from source database system.

But, this mode can cause to repeat inquiry source database system, increase the inquiry pressure of source database system.

Therefore, as another embodiment, this method can also include：

The corresponding data check rule of the purpose Database Systems is registered in the message-oriented middleware.

And data producer can read data check rule from message-oriented middleware, and according to data check rule Verify the creation data；The data producer is met in the creation data after the data check rule, further according to The creation data generates data update notification.

Wherein, the creation data of data producer not only corresponds to a topic of subscription, and disparate databases system may also Same topic of subscription can be subscribed to, therefore in order to ensure that creation data disclosure satisfy that the data demand of disparate databases system, In some embodiments, data producer can read monitoring server from the message-oriented middleware to be directed to different purpose databases The data check rule of system registry；

Verify the data check the rule whether creation data is satisfied by matching with the creation data；

If the creation data is satisfied by the data check rule matched with the creation data, generates and ordered described in correspondence Read the data update notification of theme.

And if creation data is unsatisfactory for any data verification rule matched with the creation data, then can generate report Alert prompt message, or data backtracking is carried out, write data to source database system again.

Wherein, the data check rule for example can include numerical values recited requirement, day ring than or chow ring than value will Ask etc..

Data system corresponding data check rule in different purpose storehouses can each be asked for purpose storehouse data system What one or more target incremental datas of verification were set.

From being described above, a Database Systems can correspond to multiple topic of subscription, and same topic of subscription can It can correspond to multiple purpose Database Systems.

, can be in message in order to which accurate obtain in creation data updates the data with the presence or absence of purpose Database Systems are corresponding Routing rule is set in middleware.

Therefore, in certain embodiments, it is described in the message between part registration monitor the topic of subscription and can include：

The topic of subscription is monitored in part registration between in the message, and triggers the message-oriented middleware according to the synchrodata Mark, sets corresponding routing rule；

The routing rule is used to indicate that the message-oriented middleware is determined in the creation data, there is the synchrodata Mark is corresponding to be retransmited when updating the data and described has adjusted what is instructed.

Routing rule defines the corresponding relation of Data Identification and purpose Database Systems.Data update notification is according to production The creation data mark of data and the generation of data more new range.Therefore message-oriented middleware is according to data update notification, you can with Determine in creation data to update the data with the presence or absence of purpose Database Systems are corresponding.

If existed in creation data, purpose Database Systems are corresponding to be updated the data, then initiates tune to the monitoring server Play instruction.Otherwise instruction has not been adjusted to monitoring server initiation.

Wherein, in the embodiment of the present invention, the division and setting of topic of subscription can be preset, monitoring server and number Know setting rule according to producer.

Topic of subscription can be divided from database aspect so that database one topic of subscription of correspondence, be subscribed to The subject name of theme can think the library name of database；

It can certainly be divided from tables of data or field aspect so that a tables of data or field correspondence one Topic of subscription, corresponding subject name is table name or field name；

It can certainly be divided from data source types aspect so that a Database Systems correspondence one subscribes to master Topic, corresponding subject name and the title for Database Systems.

, will if for example divided according to tables of data or field if topic of subscription can increase too much safeguards pressure Cause topic of subscription too many, and if too many, can cause routing rule complicated again.

Because in practical application, data syn-chronization is typically to be carried out according to the dimension of tables of data, therefore alternatively, subscription master Topic can be divided according to data source types so that Database Systems one topic of subscription of correspondence.Such as impala, Hive, hbase, mysql etc. correspond to a topic of subscription respectively.It can be determined according to the synchronous data identification of purpose database Which Database Systems belonged to, so as to can determine correspondence topic of subscription.Data producer is according to the production of creation data Data Identification can determine which Database Systems belonged to, so that corresponding topic of subscription can be determined, so as to Data update notification is pushed into the topic of subscription.

Wherein, in the embodiment of the present invention, monitoring server obtains updating the data for request synchronization from source database system, can Obtained with being inquired about using query statement, therefore, in certain embodiments, the obtaining step that updates the data can include：

If there is the data update notification for purpose Database Systems, with reference to the synchronous data identification information and The data more new range, generates query statement；

Submit the query statement to the source database system, request is synchronous to be updated the data to obtain.

Source database system performs the query statement, you can be updated the data.

Wherein, query template can also be carried in registration request, can according to synchronous data identification and creation data mark What is updated the data with determining updates the data mark, thus will update the data mark and data more new range fill to the inquiry Query statement can be obtained in template.

So that query statement is SQL as an example, the query template can be：

Select count(id)from t_1where update_time>@START_TIME@and update_time <2017-01-01 12:25:32@END_TIME@

Wherein, the process object in the query template such as " count (id) " " t_1 " can combine update the data identify into Row updates." START_TIME " and " END_TIME " is time placeholder, represents time started and end time, can be with The time range in scope is updated with reference to data to be updated.

The data format of the data update notification for example can be：

{target：Defaultdb.fact_info_order, range：{key：update_time,from: 20170101 12:23:32, to:20170101 12:29:32}}

In the example, data update notification is json data formats, and wherein target informs the library name of creation data, table name Claim and/or field name；Range informs data more new range.During the example informs that creation data is defalutdb this database Fact_info_order this tables of data, the data area major key of this tables of data is update_time, therefore data are more New range is time range, and time range is to interval codomain：20170101 12:23:32~20170101 12:29:32。

With reference to above-mentioned SQL templates, the query statement of acquisition can be：

Select*from defaultdb.fact_info_order where update_time>‘20170101 12: 23:32’and update_time<‘20170101 12:29:32’

Fig. 2 is a kind of flow chart of another embodiment of method of data synchronization provided in an embodiment of the present invention, the present embodiment Technical scheme is applied to data producer, and this method can include following steps：

201：Creation data is write to source database system.

202：According to the creation data, data update notification is generated.

203：Issue the data update notification.

Wherein, the data update notification determines the corresponding purpose data of the data update notification for monitoring server Storehouse system, and updating the data for the purpose Database Systems request synchronization is obtained from the source database system, and load institute State and update the data to the purpose Database Systems.

In certain embodiments, institute's data update notification generation step can include：

Identified according to the creation data of the creation data, determine the corresponding topic of subscription of the creation data；

The data update notification of the generation correspondence topic of subscription；

The data update notification issuing steps can include：

The data update notification is pushed to the topic of subscription；The topic of subscription is monitored for the monitoring server When receiving data update notification, the synchronous renewal number of the purpose Database Systems request is obtained from the source database system According to.

In certain embodiments, the data update notification to the topic of subscription that pushes can include：

The topic of subscription of the data update notification into message-oriented middleware is pushed, so that the message-oriented middleware is true Determine the corresponding purpose Database Systems of the data update notification, and instruction has been adjusted to monitoring server initiation；Institute State monitoring server by respond it is described adjusted instruction, obtain purpose Database Systems request from the source database system Synchronous updates the data.

In certain embodiments, the data update notification of the generation correspondence topic of subscription can include：

The data check rule that monitoring server is registered for different purpose Database Systems are read from the message-oriented middleware Then；

In certain embodiments, if the creation data is unsatisfactory for any data verification matched with the creation data Rule, generates prompt messages.

In certain embodiments, the data update notification can include the creation data creation data mark and Data changing range；

It is described according to the creation data, generation data update notification can include：

Database, tables of data and/or field according to belonging to the creation data, determine that creation data is identified；The life Producing Data Identification includes library name, table name and/or field name；

According to creation data correspondence time range and/or number range, data changing range is determined；

With reference to creation data mark and the data changing range, data update notification is generated.

Fig. 3 is a kind of flow chart of another embodiment of method of data synchronization provided in an embodiment of the present invention, the present embodiment Technical scheme is applied in message-oriented middleware, and message-oriented middleware can be that monitoring server is entered with data producer using MQ modes The message exchange equipment set during row data transfer.Therefore, this method can include following steps：

301：Monitor monitoring server and be directed to the topic of subscription that purpose Database Systems are registered.

Wherein, the synchronous data identification that the topic of subscription is registered by the monitoring server according to purpose Database Systems It is determined that；

302：If the topic of subscription receives data update notification, initiate to have adjusted instruction to the monitoring server.

The monitoring server response is described to have adjusted instruction, you can to obtain the purpose database from source database system System request is synchronous to be updated the data；

In certain embodiments, this method can also include：

Receive the data check rule that the monitoring server is registered for the purpose Database Systems；The data school Rule is tested to read for the data producer and verify the creation data.

In certain embodiments, this method can also include：

According to the synchronous data identification of the purpose Database Systems, routing rule is set；

It is then described to have adjusted instruction step of initiating to include：

If the topic of subscription receives data update notification, and determines that the creation data is deposited according to the routing rule Update the data the synchronous data identification is corresponding, initiate to have adjusted instruction to the monitoring server.

Fig. 4 is a kind of signaling diagram of another embodiment of method of data synchronization provided in an embodiment of the present invention, and this method can To comprise the following steps：

401：Data producer writes creation data to source database system.

402：Data producer is identified according to the creation data of the creation data, is determined that the creation data is corresponding and is ordered Read theme.

403：Data producer reads monitoring server from the message-oriented middleware and noted for different purpose Database Systems The data check rule of volume.

Wherein, data check rule can be carried in the registration request that request end is sent, and existed by monitoring server Message-oriented middleware completes registration.

404：Data producer verifies the data check whether creation data is satisfied by matching with the creation data Rule, if performing step 405.

If it is not, then prompt messages can be generated or data backtracking etc. is carried out.

405：The data update notification of the generation correspondence topic of subscription.

Wherein, alternatively, the data update notification includes creation data mark and the data change of the creation data More scope；

The data update notification generation step can include：

406：Push the topic of subscription of the data update notification into message-oriented middleware.

407：Monitoring server receives the registration request that request end is initiated.

Wherein, the registration request includes the corresponding synchronous data identification of purpose Database Systems.

The synchronous data identification can include library name, table name and/or field name.

408：Monitoring server determines that the synchronous data identification is corresponding and subscribes to master according to the synchronous data identification Topic.

409：Monitoring server in the message between part registration monitor the topic of subscription.

In addition, monitoring server and can also trigger the message-oriented middleware according to the synchronous data identification, setting pair The routing rule answered.

410：Message-oriented middleware sets routing rule according to the synchronous data identification of the purpose Database Systems.

411：Message-oriented middleware monitors whether the topic of subscription receives data update notification, if it is, performing step 412, continue to monitor if not.

412：Message-oriented middleware determines to whether there is the synchrodata in the creation data according to the routing rule Mark is corresponding to be updated the data, if performing step 413, continues to monitor if not.

413：Message-oriented middleware initiates to have adjusted instruction to the monitoring server.

414：Monitoring server obtains updating the data for the purpose Database Systems request synchronization from source database system.

Wherein, alternatively, monitoring server can be with reference to the synchronous data identification information and data renewal Scope, generates query statement；The query statement is submitted to the source database system, to obtain the renewal number that request is synchronous According to.

415：Updated the data described in monitoring server loading to the purpose Database Systems.

It is loaded onto wherein it is possible to will be updated the data using Load modes in purpose Database Systems.

Further, it is also possible to the purpose data format for being converted into adaptation purpose Database Systems will be updated the data, with can be into Work(is loaded onto purpose Database Systems.

By the technical scheme of the embodiment of the present invention, realize data and upgrade in time, particularly when batch data updates, It ensure that the promptness that batch data updates.The invalid inquiry to source database system can be reduced simultaneously, mitigate source database The inquiry pressure of system.

Fig. 5 is a kind of structural representation of data synchronization unit one embodiment provided in an embodiment of the present invention, the device It can be only fitted in actual applications in monitoring server, the device can include：

Notify to monitor module 501, monitor the data update notification that whether there is for purpose Database Systems；The data Update notification is what data producer was generated according to creation data；

Data acquisition module 502, for if there is the data update notification for the purpose Database Systems, from source Database Systems obtain updating the data for the purpose Database Systems request synchronization；

Data load-on module 503, for loading described update the data to the purpose Database Systems.

In certain embodiments, updating the data for purpose Database Systems request synchronization can be according to purpose data base set The registration request of system is determined, therefore alternatively, the device can also include：

Request receiving module, the registration request for receiving request end initiation, the registration request includes being directed to the mesh Database Systems synchronous data identification.

In some embodiments, in order to ensure information transmit-receive promptness, data can be realized by the way of " Publish-subscribe " The monitoring of update notification, it is therefore, alternatively, described to notify that monitoring module include：

Theme determining unit, for the synchronous data identification registered according to the purpose Database Systems, is determined described same Step data identifies corresponding topic of subscription；The synchronous data identification includes library name, table name and/or field name；

Monitoring unit is notified, for monitoring whether the topic of subscription receives data update notification；The data update logical Know that specially described data producer is generated according to creation data, and push to the corresponding topic of subscription of the creation data.

Therefore, alternatively, the notice monitoring unit can monitor described subscribe to specifically for part registration between in the message Theme, to monitor whether receive data update notification；

The data acquisition module specifically for：

In addition, in order to improve data accuracy, it is ensured that meet purpose data from updating the data for source database system acquisition The data demand of storehouse system, to avoid failure of data synchronization, can to updating the data carry out data check, therefore, alternatively, In some embodiments, the device can also include：

Regular Registering modules, for registering the corresponding data check of the purpose Database Systems in the message-oriented middleware Rule；The data check rule is read by the data producer and verifies the creation data；The data update notification Specially described data producer is met in the creation data after the data check rule, is given birth to according to the creation data Into.

, can be in message in order to which accurate obtain in creation data updates the data with the presence or absence of purpose Database Systems are corresponding Routing rule is set in middleware.In certain embodiments, the data update notification is according to the production number of the creation data According to mark and the generation of data more new range；

It is described notice monitoring unit specifically for：

The routing rule is used to indicate that the message-oriented middleware is determined in the creation data, there is the synchrodata Mark is corresponding to be retransmited when updating the data and described has adjusted instruction.

Wherein, in the embodiment of the present invention, monitoring server obtains updating the data for request synchronization from source database system, can Obtained with being inquired about using query statement, therefore, in certain embodiments, the data update notification includes the creation data Creation data is identified and data more new range；

The data acquisition module can include：

Sentence generation unit, for if there is the data update notification for purpose Database Systems, with reference to described same Step data identification information and the data more new range, generate query statement；

Data capture unit, for submitting the query statement to the source database system, to obtain request synchronization Update the data.

In a possible design, the data synchronization unit of embodiment illustrated in fig. 5 can be implemented as monitoring server, such as Shown in Fig. 6, the monitoring server can include one or more processors 601 and one or more memories 602；

Wherein, one or more of memories 602 store one or more computer instruction, wherein, described one or A plurality of computer instruction calls execution for one or more of processors 601.

One or more of processors 601 are used for：

Updated the data described in loading to the purpose Database Systems.

Alternatively, one or more of processors are additionally operable to perform all or part of step in aforementioned approaches method step Suddenly.

Wherein, processor can be application specific integrated circuit (ASIC), digital signal processor (DSP), Digital Signal Processing Equipment (DSPD), PLD (PLD), field programmable gate array (FPGA), controller, microcontroller, microprocessor Device or other electronic components are realized, for performing the above method.

One or more memories can be by any kind of volatibility or non-volatile memory device or their group Close and realize, such as static RAM (SRAM), Electrically Erasable Read Only Memory (EEPROM) is erasable to compile Journey read-only storage (EPROM), programmable read only memory (PROM), read-only storage (ROM), magnetic memory, flash Device, disk or CD.

The embodiment of the present invention additionally provides a kind of computer program product, including computer-readable storage medium is used to store Fig. 5 Computer software instructions used in the data synchronization unit, it, which is included, is used to perform data synchronization unit institute in above-described embodiment The program being related to.

Fig. 7 is a kind of structural representation of another embodiment of data synchronization unit provided in an embodiment of the present invention, the dress Putting can configure and in data producer, the data producer can be data server, service server in actual applications Deng the server for creation data, in e-commerce field, the data producer can be recorded according to online transaction, production Server of transaction data etc..

The device can include：

Data write. module 701, for writing creation data to source database system；

Generation module 702 is notified, for according to the creation data, generating data update notification；

Release module 703 is notified, for issuing the data update notification；The data update notification is used for monitoring service Device determines the corresponding purpose Database Systems of the data update notification, and obtains the purpose number from the source database system Update the data, and updated the data described in loading to the purpose Database Systems according to storehouse system request is synchronous.

In certain embodiments, the notice generation module can include：

Theme determining unit, for being identified according to the creation data of the creation data, determines the creation data correspondence Topic of subscription；

Generation unit is notified, the data update notification for generating the correspondence topic of subscription；

It is described notice release module specifically for：The data update notification is pushed to the topic of subscription；It is described to subscribe to When theme receives data update notification for monitoring server monitoring, the purpose number is obtained from the source database system Updated the data according to storehouse system request is synchronous.

In certain embodiments, the notice release module can be specifically for：

Exist in some embodiments, the notice generation module can also include：

Data check unit, reads monitoring server from the message-oriented middleware and is registered for different purpose Database Systems Data check rule；

It is described notice generation unit specifically for：

In certain embodiments, the device can also include：

Reminding module, if verifying rule for any data that the creation data is unsatisfactory for matching with the creation data Then, prompt messages are generated.

In certain embodiments, the data update notification includes the creation data mark and data of the creation data Changing range；

It is described notice generation module specifically for：

In a possible design, the data synchronization unit of embodiment illustrated in fig. 7 can be implemented as data production equipment, In actual applications, the data production equipment is also the data producer in being described above, as shown in figure 8, the data are produced Equipment can include one or more processors 801 and one or more memories 802；

Wherein, one or more of memories 802 store one or more computer instruction, wherein, described one or A plurality of computer instruction calls execution for one or more of processors 801.

One or more of processors 801 are used for：

Creation data is write to source database system；

According to the creation data, data update notification is generated；

Alternatively, one or more of processors 801 are additionally operable to perform all or part in aforementioned approaches method step Step.

Wherein, processor can be application specific integrated circuit (ASIC), digital signal processor (DSP), data signal It is processing equipment (DSPD), PLD (PLD), field programmable gate array (FPGA), controller, microcontroller, micro- Processor or other electronic components are realized, for performing the above method.

The embodiment of the present invention additionally provides a kind of computer program product, including computer-readable storage medium is used to store Fig. 7 Computer software instructions used in the data synchronization unit, it, which is included, is used to perform data synchronization unit institute in above-described embodiment The program being related to.

Fig. 9 is a kind of structural representation of another embodiment of data synchronization unit provided in an embodiment of the present invention, the dress Putting can configure and in message-oriented middleware, message-oriented middleware can be that monitoring server is carried out with data producer using MQ modes The message exchange equipment set during data transfer.

The device can include：

Theme monitors module 901, and the topic of subscription that purpose Database Systems are registered is directed to for monitoring monitoring server；Institute The synchronous data identification that topic of subscription registered by the monitoring server according to purpose Database Systems is stated to determine；

Module 902 has been adjusted, if receiving data update notification for the topic of subscription, has been initiated to the monitoring server Instruction has been adjusted, instruction has been adjusted so that monitoring server response is described, the purpose database is obtained from source database system System request is synchronous to be updated the data；

Exist in some embodiments, the device can also include：

Regular receiving module, the data school that the purpose Database Systems are registered is directed to for receiving the monitoring server Test rule；The data check rule reads for the data producer and verifies the creation data.

In certain embodiments, the device can also include：

Setup module is route, for the synchronous data identification according to the purpose Database Systems, routing rule is set；

It is then described to have adjusted module be specifically for：

In a possible design, the data synchronization unit of embodiment illustrated in fig. 9 can be implemented as information storaging apparatus, In actual applications, the information storaging apparatus can think the message-oriented middleware in being described above, and as shown in Figure 10, the message is deposited Storage equipment can include one or more processors 1001 and one or more memories 1002；

Wherein, one or more of memories 1002 store one or more computer instruction, wherein, described one or A plurality of computer instruction calls execution for one or more of processors 1001.

One or more of processors 1001 are used for：

Wherein, processor can be one or more application specific integrated circuits (ASIC), digital signal processor (DSP), digital signal processing appts (DSPD), PLD (PLD), field programmable gate array (FPGA), control Device, microcontroller, microprocessor or other electronic components are realized, for performing the above method.

The embodiment of the present invention additionally provides a kind of computer program product, including computer-readable storage medium is used to store Fig. 9 Computer software instructions used in the data synchronization unit, it, which is included, is used to perform data synchronization unit institute in above-described embodiment The program being related to.

It is apparent to those skilled in the art that, for convenience and simplicity of description, the system of foregoing description, The specific work process of device and unit, may be referred to the corresponding process in preceding method embodiment, will not be repeated here.

Device embodiment described above is only schematical, wherein the unit illustrated as separating component can To be or may not be physically separate, the part shown as unit can be or may not be physics list Member, you can with positioned at a place, or can also be distributed on multiple NEs.It can be selected according to the actual needs In some or all of module realize the purpose of this embodiment scheme.Those of ordinary skill in the art are not paying creativeness Work in the case of, you can to understand and implement.

Through the above description of the embodiments, those skilled in the art can be understood that each embodiment can Realized by the mode of software plus required general hardware platform, naturally it is also possible to pass through hardware.Understood based on such, on The part that technical scheme substantially in other words contributes to prior art is stated to embody in the form of software product, should Computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disc, CD, including some fingers Order is to cause a computer equipment (can be personal computer, server, or network equipment etc.) to perform each implementation Method described in some parts of example or embodiment.

Finally it should be noted that：The above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although The present invention is described in detail with reference to the foregoing embodiments, it will be understood by those within the art that：It still may be used To be modified to the technical scheme described in foregoing embodiments, or equivalent substitution is carried out to which part technical characteristic； And these modification or replace, do not make appropriate technical solution essence depart from various embodiments of the present invention technical scheme spirit and Scope.

The invention discloses A1, a kind of method of data synchronization, including：

Updated the data described in loading to the purpose Database Systems.

A2, the method according to A1, the monitoring step include：

The synchronous data identification registered according to the purpose Database Systems, determines that the synchronous data identification is corresponding and orders Read theme；The synchronous data identification includes library name, table name and/or field name；

A3, the method according to A2, it is characterised in that whether the monitoring topic of subscription, which receives data, updates logical Know including：

The obtaining step that updates the data includes：

A4, the method according to A3, in addition to：

The corresponding data check rule of the purpose Database Systems is registered in the message-oriented middleware；The data check Rule is read by the data producer and verifies the creation data；The data update notification is specially the data production Side is met in the creation data after the data check rule, is generated according to the creation data.

A5, the method according to A3, the data update notification according to the creation data of the creation data identify with And the generation of data more new range；

It is described in the message between part registration monitor the topic of subscription and include：

A6, the method according to A1 or A2, the data update notification include the creation data mark of the creation data Know and data more new range；

The obtaining step that updates the data includes：

A7, the method according to A1, the topic of subscription determine before step that methods described also includes：

The registration request that request end is initiated is received, the registration request includes the synchronization for the purpose Database Systems Data Identification.

B8, a kind of method of data synchronization, including：

Creation data is write to source database system；

According to the creation data, data update notification is generated；

B9, the method according to B8, institute's data update notification generation step include：

The data update notification issuing steps include：

B10, the method according to B9, the push data update notification to the topic of subscription include：

B11, the method according to B10, the data update notification of the generation correspondence topic of subscription include：

B12, the method according to B11, in addition to：

If the creation data is unsatisfactory for any data verification rule matched with the creation data, generation alarm is carried Show information.

B13, the method according to B8, the creation data of the data update notification including the creation data identify with And data changing range；

It is described according to the creation data, generation data update notification includes：

C14, a kind of method of data synchronization, including：

C15, the method according to C14, in addition to：

C16, the method according to C14, in addition to：

It is described to have adjusted instruction step of initiating to include：

D17, a kind of data synchronization unit, including：

D18, the device according to D17, the notice, which monitors module, to be included：

D19, the device according to D18, the notice monitoring unit monitor institute specifically for part registration between in the message Topic of subscription is stated, to monitor whether receive data update notification；

The data acquisition module specifically for：

D20, the device according to D19, in addition to：

D21, the device according to D19, the data update notification are identified according to the creation data of the creation data And the generation of data more new range；

It is described notice monitoring unit specifically for：

D22, the device according to D17 or D18, the data update notification include the creation data of the creation data Mark and data more new range；

The data acquisition module includes：

D23, the device according to D17, in addition to：

E24, a kind of data synchronization unit, including：

Data write. module, for writing creation data to source database system；

E25, the device according to E24, the notice generation module include：

E26, the device according to E25, the notice release module specifically for：

E27, the device according to E26, the notice generation module also include：

It is described notice generation unit specifically for：

E28, the device according to E27, in addition to：

E29, the device according to E24, the creation data that the data update notification includes the creation data are identified And data changing range；

It is described notice generation module specifically for：

F30, a kind of data synchronization unit, including：

F31, the device according to F30, in addition to：

F32, the device according to F30, in addition to：

It is described adjusted module specifically for：

G33, a kind of monitoring server including one or more memories and one or more processors；

One or more of processors are used for：

Updated the data described in loading to the purpose Database Systems.

H34, a kind of data production equipment, including one or more memories and one or more processors；

One or more of processors are used for：

Creation data is write to source database system；

According to the creation data, data update notification is generated；

I35, a kind of information storaging apparatus, including one or more memories and one or more processors；

One or more of processors are used for：

Claims

1. a kind of method of data synchronization, it is characterised in that including：

Monitor the data update notification that whether there is for purpose Database Systems；The data update notification is data producer Generated according to creation data；

If there is the data update notification for the purpose Database Systems, the purpose number is obtained from source database system Updated the data according to storehouse system request is synchronous；

Updated the data described in loading to the purpose Database Systems.

2. according to the method described in claim 1, it is characterised in that the monitoring step includes：

The synchronous data identification registered according to the purpose Database Systems, determines that the synchronous data identification is corresponding and subscribes to master Topic；The synchronous data identification includes library name, table name and/or field name；

Monitor whether the topic of subscription receives data update notification；The data update notification is specially the data producer Generated according to creation data, and push to the corresponding topic of subscription of the creation data.

3. method according to claim 2, it is characterised in that whether the monitoring topic of subscription receives data renewal Notice includes：

The obtaining step that updates the data includes：

Instruction is played in response to the tune of the message-oriented middleware, it is same to obtain the purpose Database Systems request from source database system What is walked updates the data；It is described to send out when having adjusted instruction to detect that the topic of subscription receives data update notification by the message-oriented middleware Send.

4. a kind of method of data synchronization, it is characterised in that including：

Creation data is write to source database system；

According to the creation data, data update notification is generated；

Issue the data update notification；The data update notification determines the data update notification pair for monitoring server The purpose Database Systems answered, and obtain the synchronous renewal number of the purpose Database Systems request from the source database system According to, and updated the data described in loading to the purpose Database Systems.

5. a kind of method of data synchronization, it is characterised in that including：

Monitor monitoring server and be directed to the topic of subscription that purpose Database Systems are registered；The topic of subscription is by the monitoring service The synchronous data identification that device is registered according to purpose Database Systems is determined；

If the topic of subscription receives data update notification, initiate to have adjusted instruction to the monitoring server, for the prison Control server response is described to have adjusted instruction, and the synchronous renewal number of the purpose Database Systems request is obtained from source database system According to；

Wherein, the data update notification is generated according to creation data by data producer and pushes to the creation data correspondence The topic of subscription.

6. a kind of data synchronization unit, it is characterised in that including：

Notify to monitor module, monitor the data update notification that whether there is for purpose Database Systems；The data update logical Know what is generated for data producer according to creation data；

Data acquisition module, for if there is the data update notification for the purpose Database Systems, from source database System obtains updating the data for the purpose Database Systems request synchronization；

7. a kind of data synchronization unit, it is characterised in that including：

Data write. module, for writing creation data to source database system；

Release module is notified, for issuing the data update notification；The data update notification is determined for monitoring server The corresponding purpose Database Systems of the data update notification, and obtain the purpose data base set from the source database system System request is synchronous to be updated the data, and is updated the data described in loading to the purpose Database Systems.

8. a kind of monitoring server, it is characterised in that it includes one or more memories and one or more processors；

One or more of computer instructions of memory storage one or more, one or more computer instruction is used for Execution is called by one or more of processors；

One or more of processors are used for：

Updated the data described in loading to the purpose Database Systems.

9. a kind of data production equipment, it is characterised in that including one or more memories and one or more processors；

One or more of processors are used for：

Creation data is write to source database system；

According to the creation data, data update notification is generated；

10. a kind of information storaging apparatus, it is characterised in that including one or more memories and one or more processors；

One or more of processors are used for：