CN107040578A - Method of data synchronization, device and system - Google Patents
Method of data synchronization, device and system Download PDFInfo
- Publication number
- CN107040578A CN107040578A CN201611271367.2A CN201611271367A CN107040578A CN 107040578 A CN107040578 A CN 107040578A CN 201611271367 A CN201611271367 A CN 201611271367A CN 107040578 A CN107040578 A CN 107040578A
- Authority
- CN
- China
- Prior art keywords
- data
- chronization
- database
- syn
- processing server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 107
- 238000012545 processing Methods 0.000 claims abstract description 237
- 230000008569 process Effects 0.000 claims description 75
- 230000001360 synchronised effect Effects 0.000 claims description 45
- 238000003860 storage Methods 0.000 claims description 29
- 238000004458 analytical method Methods 0.000 claims description 22
- 238000012544 monitoring process Methods 0.000 description 13
- 238000013500 data storage Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000007726 management method Methods 0.000 description 5
- 230000002159 abnormal effect Effects 0.000 description 4
- 238000004904 shortening Methods 0.000 description 4
- 230000006399 behavior Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000014759 maintenance of location Effects 0.000 description 2
- 230000006641 stabilisation Effects 0.000 description 2
- 238000011105 stabilization Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of method of data synchronization, applied to data processing server, the method for data synchronization includes：When receiving data syn-chronization instruction, data processing server determines the network memory of the data processing server association；It is determined that the network memory in obtain default data syn-chronization rule；According to the data syn-chronization of acquisition rule, the database of pending data syn-chronization in the database server of the data processing server association is determined, and data syn-chronization operation is performed according to the database of determination.The invention also discloses a kind of data synchronization unit and system.The present invention improves the convenience and whole efficiency of data syn-chronization.
Description
Technical field
The present invention relates to data processing field, more particularly to a kind of method of data synchronization, device and system.
Background technology
When existing enterprise-level application carries out data syn-chronization, the same of data can be carried out by multiple data processing servers
Step, but existing data syn-chronization, it is necessary in data processing server data storage synchronization rules, expanding new number every time
When carrying out data syn-chronization according to processing server, it is required for storing corresponding rule, in this case, not only to data simultaneously operating
It is cumbersome, and due to needing the cost time to go data storage synchronization rules, cause the whole efficiency reduction to data syn-chronization.
The content of the invention
It is a primary object of the present invention to provide a kind of method of data synchronization, device and system, it is intended to solve existing number
According to the method for synchronization, cumbersome, and the relatively low technical problem of whole efficiency of data syn-chronization is operated.
To achieve the above object, a kind of method of data synchronization that the present invention is provided, it is described applied to data processing server
Method of data synchronization includes：
When receiving data syn-chronization instruction, data processing server determines the network of the data processing server association
Memory；
It is determined that the network memory in obtain default data syn-chronization rule；
According to the data syn-chronization of acquisition rule, in the database server for determining the data processing server association
The database of pending data syn-chronization, and data syn-chronization operation is performed according to the database of determination.
Preferably, the data syn-chronization rule according to acquisition, determines the number of the data processing server association
According to the database of pending data syn-chronization in the server of storehouse, and according to the database of determination perform data syn-chronization operate the step of wrap
Include：
It is same whether the corresponding operating system of the data processing server analysis data processing server meets data
Step condition；
When the corresponding operating system of the data processing server meets data syn-chronization condition, according to the number of acquisition
According to synchronization rules, it is determined that in the database server of association pending data syn-chronization database, and according to the database of determination
Perform data syn-chronization operation.
Preferably, the data processing server analyzes whether the corresponding operating system of the data processing server meets
The step of data syn-chronization condition, includes：
The data processing server analyzes the corresponding operation of the data processing server by preset schedule program
The resource situation of system, wherein, the resource situation includes cpu resource or internal memory shows program MEM resources；
The resource situation based on analysis, determines whether the corresponding operating system of the data processing server meets number
According to synchronous condition, wherein, when the resources left utilization rate of the cpu resource or MEM resources is more than predetermined threshold value, it is determined that described
The corresponding operating system of data processing server meets data syn-chronization condition.
Preferably, described perform performs the step of data syn-chronization is operated simultaneously according to the database of determination, performs following walk
Suddenly：
The data processing server records the data syn-chronization and operates corresponding synchronizing process；
By the database day of the synchronizing process storage of record to local text log and the database server
In will, in order to check data synchronization process.
Preferably, the method for data synchronization also includes：
If the data processing server detects other data processing services relative to the data processing server
There is data syn-chronization interruption in data synchronization process in device, it is determined that each database that data syn-chronization is interrupted；
Data syn-chronization operation is re-executed to each database of data sync break；
If it is synchronous by other data processing servers to have detected partial data, with the data of re-synchronization
Update synchronous data.
In addition, to achieve the above object, the present invention also provides a kind of data synchronization unit, applied to data processing service
Device, the data synchronization unit includes：
Determining module, for when receiving data syn-chronization instruction, determining the network of the data processing server association
Memory；
Acquisition module, for it is determined that the network memory in obtain default data syn-chronization rule；
Processing module, for the data syn-chronization rule according to acquisition, determines the data processing server association
The database of pending data syn-chronization in database server, and data syn-chronization operation is performed according to the database of determination.
Preferably, the processing module includes：
Submodule is analyzed, for analyzing whether the corresponding operating system of the data processing server meets data syn-chronization bar
Part；
Submodule is handled, for when the corresponding operating system of the data processing server meets data syn-chronization condition,
According to the data syn-chronization of acquisition rule, it is determined that in the database server of association pending data syn-chronization database, and
Data syn-chronization operation is performed according to the database of determination.
Preferably, the analysis submodule includes：
Analytic unit, the money for analyzing the corresponding operating system of the data processing server by preset schedule program
Source situation, wherein, the resource situation includes cpu resource or internal memory shows program MEM resources；
Determining unit, for the resource situation based on analysis, determines the corresponding operation of the data processing server
Whether system meets data syn-chronization condition, wherein, it is more than in the resources left utilization rate of the cpu resource or MEM resources default
During threshold value, determine that the corresponding operating system of the data processing server meets data syn-chronization condition.
Preferably, the data synchronization unit also includes：
Logging modle, for while the processing module performs data syn-chronization operation according to the database of determination, remembering
Record the data syn-chronization and operate corresponding synchronizing process；
Memory module, for the synchronizing process storage of record to be arrived into local text log and the database service
In the database journal of device, in order to check data synchronization process.
Preferably, the determining module, if being additionally operable to detect other data relative to the data processing server
There is data syn-chronization interruption in data synchronization process in processing server, it is determined that each database that data syn-chronization is interrupted；
The processing module, is additionally operable to re-execute each database of data sync break data syn-chronization operation；
The data synchronization unit also includes：
Update module, if synchronous by other data processing servers for having detected partial data, is used
The data of re-synchronization update synchronous data.
In addition, to achieve the above object, the present invention also provides a kind of data synchronous system, the data synchronous system bag
Include：Data processing server, network memory and database server, the data of the data processing server as described above
Sychronisation.
Method of data synchronization proposed by the present invention, device and system, applied to data processing server, the data processing
Server first determines the network memory of the data processing server association, Ran Hou when receiving data syn-chronization instruction
Default data syn-chronization rule is obtained in the network memory determined, finally according to the data syn-chronization of acquisition rule,
The database of pending data syn-chronization in the database server of the data processing server association is determined, and according to determination
Database performs data syn-chronization operation.When realizing synchrodata, even if to extend new data processing server carries out data
Synchronization, it is not required that the interpolation data synchronization rules in each data processing server, as long as by data syn-chronization rule add
Into network memory, as long as each follow-up data processing server obtains data syn-chronization rule just from network memory
The synchronization of data can be carried out to the database in data processing server, the synchronization to data is very convenient, due to shortening
The time of interpolation data synchronization rules, so as to improve the whole efficiency of data syn-chronization.
Brief description of the drawings
Fig. 1 is the schematic flow sheet of method of data synchronization first embodiment of the present invention；
The refinement schematic flow sheet that Fig. 2 is step S30 in Fig. 1；
The refinement schematic flow sheet that Fig. 3 is step S31 in Fig. 2；
Fig. 4 is the schematic flow sheet of method of data synchronization second embodiment of the present invention；
Fig. 5 is the preferred application schematic diagram of a scenario of data synchronization process of the present invention；
Fig. 6 is the high-level schematic functional block diagram of data synchronization unit first embodiment of the present invention；
Fig. 7 is the refinement high-level schematic functional block diagram of processing module 30 in Fig. 6；
Fig. 8 is the refinement high-level schematic functional block diagram of analysis submodule 31 in Fig. 7；
Fig. 9 is the high-level schematic functional block diagram of data synchronization unit second embodiment of the present invention.
The realization, functional characteristics and advantage of the object of the invention will be described further referring to the drawings in conjunction with the embodiments.
Embodiment
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
The present invention provides a kind of method of data synchronization, applied to data processing server.
Reference picture 1, Fig. 1 is the schematic flow sheet of method of data synchronization first embodiment of the present invention.
In the present embodiment, the method for data synchronization includes：
Step S10, when receiving data syn-chronization instruction, data processing server determines that the data processing server is closed
The network memory of connection.
In the present embodiment, before the step S10, including step：Data processing server and network memory are closed
Connection gets up, and by data syn-chronization rule storage into the network memory.It is worth noting that, described by data syn-chronization rule
Store in the webserver, be actually that the corresponding shared magnetic of the webserver is arrived into data syn-chronization rule storage
In disk, data syn-chronization rule is used for each data processing server associated will pass through shared disk technology.
In the present embodiment, the data processing server is preferably Kettle servers, data processing clothes hereinafter
Business device represents Kettle servers.It is worth noting that, the Kettle servers are really a ETL from starting of increasing income
(Extract-Transform-Load) instrument, for extracting, cleaning, change, loading data, pure java writes, can be many
Run on platform, data pick-up efficient stable.The network memory be preferably NAS (Network Attached Storage,
Network attached storage) server, similarly, network memory hereinafter also represents nas server.
In the present embodiment, a nas server can set up incidence relation with multiple Kettle servers, with specific reference to
The identification information of the identification information of Kettle servers and the nas server is associated the foundation of relation.Due to this reality
It is preferably Kettle servers to apply the data processing server in example, therefore, and the data syn-chronization rule is alternatively referred to as
Kettle program metadatas.
When the data processing server receives data syn-chronization instruction, you can according to the incidence relation set up in advance,
Directly determine the network memory of association.
Step S20, it is determined that the network memory in obtain default data syn-chronization rule.
That is, it is determined that the data processing server association network memory after, you can it is determined that institute
State and default data syn-chronization rule is obtained in network memory, institute is stored in by the synchronization rules are hereinbefore described
State in the corresponding shared disk of the webserver, therefore, it is described it is determined that the network memory in obtain default data
Synchronization rules, are actually that the data syn-chronization rule is obtained in the corresponding shared disk of the webserver.This implementation
The data syn-chronization rule in example includes but is not limited to：Each database of pending data syn-chronization, data class to be synchronized
Type, and data synchronizing frequency etc..
It should be appreciated that in the shared disk of network memory data storage, and by shared disk technology for other
Each data processing server carries out the acquisition of data, is synchronously advised without data storage is distinguished in each data processing server
Then so that follow-up data syn-chronization is more convenient, and data synchronization efficiency is also higher.
Step S30, according to the data syn-chronization of acquisition rule, determines the database of pending data syn-chronization, and according to
The database of determination performs data syn-chronization operation.
In the present embodiment, after data syn-chronization rule is got, first according to the data syn-chronization of acquisition rule, really
The database of pending data syn-chronization in the database server of the fixed data processing server association, wherein, the data
The foundation of processing server and database server incidence relation, and above with network memory to set up mode consistent, and
And, equally it is that a database server can set up incidence relation with multiple Kettle servers, does not do unnecessary repeat herein.
In the present embodiment, database to be synchronized is to have the Service Database of data syn-chronization demand, it is necessary to by the data
Processing server completes the synchronization of data, also, the quantity of database to be synchronized at least includes two.For example, pending data
Synchronous database includes the first database and the second database, then it is determined that after the first database and the second database,
The data processing server is executable data syn-chronization operation, wherein, the data syn-chronization operation includes but is not limited to：By
Data A in one database is synchronized in the second database, or the data B in the second database is synchronized into the first database
In.
In the present embodiment, in order to ensure the efficiency of data syn-chronization, reference picture 2, the step S30 includes：
Step S31, whether the data processing server analyzes the corresponding operating system of the data processing server full
Sufficient data syn-chronization condition；
Step S32, when the corresponding operating system of the data processing server meets data syn-chronization condition, according to acquisition
Data syn-chronization rule, it is determined that in the database server of association pending data syn-chronization database, and according to determination
Database perform data syn-chronization operation.
In the present embodiment, analyze whether the corresponding operating system of the data processing server meets data syn-chronization bar
Part, can refer to Fig. 3, specifically, and the step S31 includes：
Step S311, the data processing server analyzes the data processing server correspondence by preset schedule program
Operating system resource situation, wherein, the resource situation includes cpu resource or internal memory and shows program MEM resources；
Step S312, the resource situation based on analysis determines the corresponding operating system of the data processing server
Whether data syn-chronization condition is met, wherein, it is more than predetermined threshold value in the resources left utilization rate of the cpu resource or MEM resources
When, determine that the corresponding operating system of the data processing server meets data syn-chronization condition.
In the present embodiment, the data processing server first passes through the money that preset schedule program analyzes current operation system
Source situation, wherein, the preset schedule program is preferably KEA (Kettle Enterprise application, Kettle enterprises
Industry level application) scheduler program, the resource situation include CPU (Central Processing Unit, central processing unit) resource
Or MEM (internal memory display program) resource.Then, the resource situation based on analysis, determines the data processing server pair
Whether the operating system answered is met in data syn-chronization condition, the present embodiment, preferably in the cpu resource or the resource of MEM resources
When remaining utilization rate is more than predetermined threshold value, it just can determine that the corresponding operating system of the data processing server meets data syn-chronization
Condition, wherein, the predetermined threshold value is configured according to actual conditions, is not limited herein.
That is, described data processing server can first analyze whether current operation system meets data syn-chronization condition, if currently
Operating system meets data syn-chronization condition, just can determine pending data syn-chronization according to the data syn-chronization of acquisition rule
Database, and data syn-chronization operation is performed according to the database of determination.If current operation system is unsatisfactory for data syn-chronization condition,
The data processing server does not perform data synchronization process, and the synchronization of data is carried out by other data processing servers.
If it should be appreciated that just carrying out the synchronization of data when the resource situation of operating system is undesirable, it is most likely that
Occur in data synchronization process, the situation of failure of data synchronization, or synchrodata speed slow situation excessively.Therefore this reality
Apply in example, whether data syn-chronization condition is met by the operating system of analyze data processing server, is just decided whether into line number
According to synchronization, effectively prevent above-mentioned situation, the normal synchronized of data ensure that to a certain extent, while also improving data
Synchronous efficiency.
Further, whether completely the data processing server analyzes the corresponding operating system of the data processing server
After sufficient data syn-chronization condition, can also service-user be showed to check the situation of analysis, so that service-user understanding data are same
The situation of step.
Further, data syn-chronization is monitored in order to realize, described perform " performs data according to the database of determination
While simultaneously operating ", following steps are performed：
Step A, the data processing server records the data syn-chronization and operates corresponding synchronizing process；
Step B, by the database of the synchronizing process storage of record to local text log and database server
In daily record, in order to check data synchronization process.
That is, the data processing server is performing data syn-chronization behaviour according to the first database and the second database
While work, it can also record the data syn-chronization and operate corresponding synchronizing process, then by the synchronizing process storage of record to originally
In the database journal of the text log on ground and the database server, for KEA scheduler programs analysis and timing exhibition
Operation management person is now given, then operation management person can check data synchronization process, to reach the effect of data syn-chronization monitoring.This
In embodiment, the synchronizing process mainly includes configuration processor, performs parameter, performs the index such as time and implementing result.
Further, also checking for corresponding synchronizing process can be operated in the data syn-chronization for receiving operation management person's input
During instruction, the data synchronization process recorded in the database journal that the database server is first shown in display interface；
If exception occurs in the data display detected in the database journal of the database server, call local
Text log, with the corresponding synchronizing process of display data simultaneously operating.
That is, local text log and database service are arrived because data processing server stores the synchronizing process of record
In the database journal of device, therefore, the database journal storage of database server is exactly that all data processing servers enter
The synchronizing process of row data simultaneously operating, and the local text log storage current data processing server of data processing server
The synchronizing process of corresponding data syn-chronization operation.So, when checking data synchronization process, preferably first in database server
Database journal in check, when database journal is abnormal, just call local text log to carry out data synchronization process
Check, and all shown without two daily records, improve the flexibility of data syn-chronization monitoring.
To be best understood from, citing detailed description：For example, currently thering is data processing server a and data processing server b all to exist
Carry out data syn-chronization, then data processing server a and data processing server b all can data syn-chronization operation synchronizing process note
In the database journal for recording database server, but data processing server a and data processing server b only can be by certainly
In the local text log of the synchronizing process oneself of oneself data syn-chronization operation, i.e. data processing server a storages are to locally
Text log a in, data processing server b store into local text log b.Subsequently, can be in database server
Database journal can check the data synchronization process of each data processing server, if database journal is abnormal, then count
Text log a is called to carry out checking for data synchronization process according to processing server a, data processing server b calls text log b
Carry out checking for data synchronization process.
The method of data synchronization that the present embodiment is proposed, applied to data processing server, the data processing server exists
When receiving data syn-chronization instruction, first determine the network memory of data processing server association, then it is determined that institute
State and default data syn-chronization rule is obtained in network memory, finally according to the data syn-chronization of acquisition rule, it is determined that described
The database of pending data syn-chronization in the database server of data processing server association, and held according to the database of determination
Row data simultaneously operating.When realizing synchrodata, even if to extend the synchronization that new data processing server carries out data,
Need not in each data processing server interpolation data synchronization rules, as long as data syn-chronization rule is added into the network storage
In device, if each follow-up data processing server obtains that data syn-chronization is regular from network memory just can be with logarithm evidence
Database in processing server carries out the synchronization of data, and the synchronization to data is very convenient, same due to shortening interpolation data
The time of rule is walked, so as to improve the whole efficiency of data syn-chronization.
Further, the second embodiment of method of data synchronization of the present invention is proposed based on first embodiment, in the present embodiment
In, reference picture 4, the method for data synchronization also includes：
Step S40, if the data processing server detects other data relative to the data processing server
There is data syn-chronization interruption in data synchronization process in processing server, it is determined that each database that data syn-chronization is interrupted；
Step S50, data syn-chronization operation is re-executed to each database of data sync break；
Step S60, if it is synchronous by the other data processing servers to have detected partial data, with again same
The data of step update synchronous data.
It should be appreciated that the step S40 can be applied to after step S30, it can also be applied to before step S10,
Specific order is not limited.
In the present embodiment, if detecting other data processing servers relative to the data processing server in number
According to occurring data syn-chronization interruption in synchronizing process, it is determined that each database that data syn-chronization is interrupted, such as other data processings
Server is carrying out data syn-chronization to the 3rd database and the 4th database, and sync break occurs, now, the processing clothes
Business device re-executes data syn-chronization operation to two databases of data sync break, i.e., to the 3rd database and the 4th database
Data syn-chronization operation is re-executed, data syn-chronization operation now is consistent with the above, is no longer repeated herein.
When the data processing server re-executes data syn-chronization operation to each database of data sync break,
If in data syn-chronization operating process is performed, having detected partial data and synchronously having been crossed by other data processing servers,
Now, in order to ensure the accuracy of data syn-chronization, the data processing server can complete whole data synchronization process, and with weight
New synchronous data update synchronous data, that is, update out the data of other data processing servers synchronized mistake, it is ensured that
The accuracy of data syn-chronization.
It should be appreciated that existing ETL instruments Kettle servers, in C/S (Client/Server, client-server
Structure) the enterprise-level application interface function under framework is complete；It is difficult in terms of deployment, scheduling, monitoring；And platform extensibility
Difference；More very, in data syn-chronization, it is necessary to which first data storage synchronization rules internally, are subsequently advised according to the data syn-chronization of storage
Data syn-chronization operation could be performed, when needing to extend multiple Kettle servers in platform, it is necessary in each Kettle clothes
It is engaged in device distinguishing data storage synchronization rules, the integrated operation to data syn-chronization is relatively complicated.
Data syn-chronization scheme of the present invention, applied to Kettle enterprise-level applications (Kettle Enterprise
Application, abbreviation KEA) in, based on Kettle enterprise-level applications are a set of Community Editions by Kettle, batch synchronization number
According to comprehensive application system, be the data syn-chronization ecosystem for integrating storage, dispatching, monitoring, database can also be realized
Open up the convenience of appearance.
The Kettle enterprise-level applications include：
1st, KEA frameworks
Nas server stores Kettle program metadatas, and the Kettle of whole platform is supplied by shared disk memory technology
Server is used, and Kettle servers are got after Kettle program metadatas, passes through KEA scheduler program analysis operation systems
Resource situation decides whether that perform synchronizing traffic data (after the completion of business datum, accesses, and then be presented to for business application system
Service-user), then the database of the information record of synchronizing traffic data to local text log and database server
(mainly corresponding day is recorded in daily record including important indicators such as configuration processor, execution parameter, execution time and implementing results
In will storehouse, analyzed for monitoring application program and be presented to operation engineer in time, the steady of data synchronous system is safeguarded to reach
It is fixed).
2nd, KEA is stored
KEA is to store Kettle program metadatas on nas server with file forms, and same Kettle platforms are shared to be made
Use this metadata；
Main feature：Distributed platform framework, platform extension is easy.
3rd, KEA is dispatched
Data syn-chronization dispatches corresponding service synchronization journey by Kettle intelligent scheduling primary control program RunFileJob.sh
Sequence, and the result that service synchronization program is performed recorded the database day of local text log and database server respectively
In will；
Major function：
Intelligent scheduling：Configuration processor is decided whether by CPU the and MEM resources of analysis operation system；
Data syn-chronization：Dispatch kitchen.sh programs and perform service synchronization；
Log：Record the database of local text log and database server respectively by WriteLog.sh
Daily record.
4th, KEA is monitored
The monitoring interface that the database journal of database server is provided, various monitoring are customized for monitoring application system；
Major function：Local text log is mainly what is used when daily record storehouse goes wrong for system investigation, just
Usage log storehouse monitor supervision platform in the case of often.
The data synchronization process of the present invention, the enterprise-level application of cluster mode ground, realize how to be extended in platform it is multiple
Kettle instruments, realize cluster mode application, and the synchronization for the carry out data that can stablize in platform, it is ensured that data are same
The continuation of step, and the data syn-chronization dynamic for grasping platform can be realized with monitoring data synchronizing process in platform.
In technical scheme, with advantages below：
1st, the characteristics of only needing to and add new Kettle servers, data syn-chronization is that metadata is read less, data syn-chronization amount
Big the characteristics of, realize that platform shares metadata with NAS technologies, with realizing easy dilatation distributed structure/architecture；
2nd, KEA is in scheduler program, CPU the and MEM resources by the program from exploitation first to the machine are estimated, and determines
Whether and when configuration processor, ensure the stabilization of data syn-chronization function and system；
The 3rd, text, the dual daily record of database are provided, externally provides unified log interface and facilitates each application system monitoring ground
Meanwhile, it also ensure that platform independence.
Based on the specific descriptions of embodiment described in Fig. 1 to Fig. 4, the present embodiment describes this hair again with specific application scenarios
The implementation process of bright method of data synchronization.
Fig. 5 is refer to, Fig. 5 is the concrete scene schematic diagram of data syn-chronization of the present invention.
Nas server stores Kettle program metadatas, and the Kettle of whole platform is supplied by shared disk memory technology
Server is used, and Kettle servers are got after Kettle program metadatas, passes through KEA scheduler program analysis operation systems
Resource situation decides whether that perform synchronizing traffic data (after the completion of business datum, accesses, and then be presented to for business application system
Service-user), if needing to perform synchronizing traffic data, then by the data syn-chronization in the first database into the second database,
Again the data of the information record of synchronizing traffic data to local text log (database in Fig. 5) and database server
In storehouse daily record (the daily record storehouse in Fig. 5), wherein, daily record storehouse mainly includes configuration processor, performs parameter, performs time and execution
The important indicator record such as a result, is presented to for the analysis of monitoring application program and in time and runs engineer, safeguards that data are same to reach
The stabilization of step system.
The present invention further provides a kind of data synchronization unit, applied to data processing server.
Reference picture 6, Fig. 6 is the high-level schematic functional block diagram of the first embodiment of data synchronization unit 100 of the present invention.
It is emphasized that it will be apparent to those skilled in the art that functional block diagram shown in Fig. 6 is only one preferably real
The exemplary plot of example is applied, those skilled in the art can enter easily around the functional module of the data synchronization unit 100 shown in Fig. 6
The supplement of the new functional module of row；The title of each functional module is self-defined title, is only used for auxiliary and understands that the data syn-chronization is filled
Put 100 each program function block, be not used in restriction technical scheme, the core of technical solution of the present invention is, each
Define the function to be reached of functional module of title.
In the present embodiment, the data synchronization unit 100 includes：
Determining module 10, for when receiving data syn-chronization instruction, determining the net of the data processing server association
Network memory.
In the present embodiment, before the network memory that determining module 10 determines the data processing server association,
Need first to associate data processing server with network memory, and the network storage is arrived into data syn-chronization rule storage
In device.It is worth noting that, described store data syn-chronization rule into the webserver, it is actually by data syn-chronization
Rule storage is into the corresponding shared disk of the webserver, will pass through each data of shared disk technology for association
Processing server uses data syn-chronization rule.
In the present embodiment, the data processing server is preferably Kettle servers, data processing clothes hereinafter
Business device represents Kettle servers.It is worth noting that, the Kettle servers are really a ETL from starting of increasing income
(Extract-Transform-Load) instrument, for extracting, cleaning, change, loading data, pure java writes, can be many
Run on platform, data pick-up efficient stable.The network memory be preferably NAS (Network Attached Storage,
Network attached storage) server, similarly, network memory hereinafter also represents nas server.
In the present embodiment, a nas server can set up incidence relation with multiple Kettle servers, with specific reference to
The identification information of the identification information of Kettle servers and the nas server is associated the foundation of relation.Due to this reality
It is preferably Kettle servers to apply the data processing server in example, therefore, and the data syn-chronization rule is alternatively referred to as
Kettle program metadatas.
When receiving data syn-chronization instruction, the determining module 10 can be according to the incidence relation set up in advance, directly
It is determined that the network memory of association.
Acquisition module 20, for it is determined that the network memory in obtain default data syn-chronization rule；
That is, after the network memory that the determining module 10 determines the data processing server association,
Acquisition module 20 can it is determined that the network memory in obtain default data syn-chronization rule, due to having been described above
The synchronization rules are stored in the corresponding shared disk of the webserver, therefore, the acquisition module 20
It is determined that the network memory in obtain default data syn-chronization rule, be actually corresponding in the webserver
The data syn-chronization rule is obtained in shared disk.The data syn-chronization rule in the present embodiment includes but is not limited to：Treat into
Synchronizing frequency of each database of row data syn-chronization, data type to be synchronized, and data etc..
It should be appreciated that in the shared disk of network memory data storage, and by shared disk technology for other
Each data processing server carries out the acquisition of data, is synchronously advised without data storage is distinguished in each data processing server
Then so that follow-up data syn-chronization is more convenient, and data synchronization efficiency is also higher.
Processing module 30, for the data syn-chronization rule according to acquisition, determines the data processing server association
Database server in pending data syn-chronization database, and according to the database of determination perform data syn-chronization operation.
In the present embodiment, after the acquisition module 20 gets data syn-chronization rule, the first basis of processing module 30
The data syn-chronization rule obtained, determines that pending data are same in the database server of the data processing server association
The database of step, wherein, the foundation of the data processing server and database server incidence relation, and above and network
Memory to set up mode consistent, also, be equally that a database server can be set up with multiple Kettle servers and close
Connection relation, unnecessary repeat is not done herein.
In the present embodiment, database to be synchronized is to have the Service Database of data syn-chronization demand, it is necessary to by the data
Processing server completes the synchronization of data, also, the quantity of database to be synchronized at least includes two.For example, pending data
Synchronous database includes the first database and the second database, then the processing module 30 is it is determined that the first database and the
After two databases, you can perform data syn-chronization operation, wherein, the data syn-chronization operation includes but is not limited to：By the first number
It is synchronized to according to the data A in storehouse in the second database, or the data B in the second database is synchronized in the first database.
In the present embodiment, in order to ensure the efficiency of data syn-chronization, reference picture 7, the processing module 30 includes：
Submodule 31 is analyzed, for analyzing whether the corresponding operating system of the data processing server meets data syn-chronization
Condition；
Submodule 32 is handled, for meeting data syn-chronization condition in the corresponding operating system of the data processing server
When, according to the data syn-chronization of acquisition rule, it is determined that in the database server of association pending data syn-chronization database,
And data syn-chronization operation is performed according to the database of determination.
In the present embodiment, whether the corresponding operating system of the analyze data processing server meets data syn-chronization bar
Part, can refer to Fig. 8, specifically, and the analysis submodule 31 includes：
Analytic unit 311, for analyzing the corresponding operating system of the data processing server by preset schedule program
Resource situation, wherein, the resource situation includes cpu resource or internal memory and shows program MEM resources；
Determining unit 312, for the resource situation based on analysis, determines the corresponding behaviour of the data processing server
Make whether system meets data syn-chronization condition, wherein, it is more than in the resources left utilization rate of the cpu resource or MEM resources pre-
If during threshold value, determining that the corresponding operating system of the data processing server meets data syn-chronization condition.
In the present embodiment, analytic unit 311 first passes through the resource situation that preset schedule program analyzes current operation system,
Wherein, the preset schedule program is preferably that (Kettle Enterprise application, Kettle enterprise-level should by KEA
With) scheduler program, the resource situation include CPU (Central Processing Unit, central processing unit) resources or MEM
(internal memory display program) resource.Then, it is determined that the resource situation of the unit 312 based on analysis, determines the data processing clothes
Whether the corresponding operating system of business device is met in data syn-chronization condition, the present embodiment, preferably in the cpu resource or MEM resources
Resources left utilization rate be more than predetermined threshold value when, just can determine that the corresponding operating system of the data processing server meets number
According to synchronous condition, wherein, the predetermined threshold value is configured according to actual conditions, is not limited herein.
That is, whether analysis submodule 31 first analysis current operation system can meet data syn-chronization condition, if current operation system
System meets data syn-chronization condition, and processing submodule 32 just can determine pending data according to the data syn-chronization rule of acquisition
Synchronous database, and data syn-chronization operation is performed according to the database of determination.If current operation system is unsatisfactory for data syn-chronization
Condition, then do not perform data synchronization process, and the synchronization of data is carried out by other data processing servers.
If it should be appreciated that just carrying out the synchronization of data when the resource situation of operating system is undesirable, it is most likely that
Occur in data synchronization process, the situation of failure of data synchronization, or synchrodata speed slow situation excessively.Therefore this reality
Apply in example, whether data syn-chronization condition is met by the operating system of analyze data processing server, is just decided whether into line number
According to synchronization, effectively prevent above-mentioned situation, the normal synchronized of data ensure that to a certain extent, while also improving data
Synchronous efficiency.
Further, the analysis submodule 31 analyzes whether the corresponding operating system of the data processing server meets
After data syn-chronization condition, the situation of analysis can also be showed service-user to check by the data synchronization unit 100, for industry
Business user understands the situation of data syn-chronization.
Further, data syn-chronization is monitored in order to realize, the data synchronization unit also includes：
Logging modle, for while the processing module performs data syn-chronization operation according to the database of determination, remembering
Record the data syn-chronization and operate corresponding synchronizing process；
Memory module, for the synchronizing process storage of record to be arrived into local text log and the database service
In the database journal of device, in order to check data synchronization process.
That is, logging modle according to the first database and the second database while data syn-chronization operation is performed,
The data syn-chronization can also be recorded and operate corresponding operating process, then the operating process storage of record is arrived local by memory module
Text log and database server database journal in, for the KEA scheduler programs analysis and timing be presented to fortune
Keeper is sought, then operation management person can check data synchronization process, to reach the effect of data syn-chronization monitoring.The present embodiment
In, the synchronizing process mainly includes configuration processor, performs parameter, performs the index such as time and implementing result.
Further, also checking for corresponding synchronizing process can be operated in the data syn-chronization for receiving operation management person's input
During instruction, the data synchronization process recorded in the database journal that the database server is first shown in display interface；
If exception occurs in the data display detected in the database journal of the database server, described is called
The text log on ground, with the corresponding synchronizing process of display data simultaneously operating.
That is, local text log and database server are arrived because the memory module stores the synchronizing process of record
Database journal in, therefore, the storage of the database journal of database server is exactly all data processing servers progress
The synchronizing process of data syn-chronization operation, and the local text log storage current data processing server pair of data processing server
The synchronizing process for the data syn-chronization operation answered.So, when checking data synchronization process, preferably first in database server
Checked in database journal, when database journal is abnormal, just call local text log to carry out looking into for data synchronization process
See, and all shown without two daily records, improve the flexibility of data syn-chronization monitoring.
To be best understood from, citing detailed description：For example, currently thering is data processing server a and data processing server b all to exist
Carry out data syn-chronization, then data processing server a and data processing server b all can data syn-chronization operation synchronizing process note
In the database journal for recording database server, but data processing server a and data processing server b only can be by certainly
In the local text log of the synchronizing process oneself of oneself data syn-chronization operation, i.e. data processing server a storages are to locally
Text log a in, data processing server b store into local text log b.Subsequently, can be in database server
Database journal can check the data synchronization process of each data processing server, if database journal is abnormal, then count
Text log a is called to carry out checking for data synchronization process according to processing server a, data processing server b calls text log b
Carry out checking for data synchronization process.
The data synchronization unit that the present embodiment is proposed, applied to data processing server, the data processing server exists
When receiving data syn-chronization instruction, first determine the network memory of data processing server association, then it is determined that institute
State and default data syn-chronization rule is obtained in network memory, finally according to the data syn-chronization of acquisition rule, it is determined that described
The database of pending data syn-chronization in the database server of data processing server association, and held according to the database of determination
Row data simultaneously operating.When realizing synchrodata, even if to extend the synchronization that new data processing server carries out data,
Need not in each data processing server interpolation data synchronization rules, as long as data syn-chronization rule is added into the network storage
In device, if each follow-up data processing server obtains that data syn-chronization is regular from network memory just can be with logarithm evidence
Database in processing server carries out the synchronization of data, and the synchronization to data is very convenient, same due to shortening interpolation data
The time of rule is walked, so as to improve the whole efficiency of data syn-chronization.
Further, reference picture 9, the second embodiment of data synchronization unit 100 of the present invention is proposed based on first embodiment,
In the present embodiment, the determining module 10, if being additionally operable to detect other data relative to the data processing server
There is data syn-chronization interruption in data synchronization process in processing server, it is determined that each database that data syn-chronization is interrupted；
The processing module 30, is additionally operable to re-execute each database of data sync break data syn-chronization operation；
The data synchronization unit also includes：
Update module 40, if synchronous by other data processing servers for having detected partial data,
Synchronous data are updated with the data of re-synchronization.
In the present embodiment, if detecting other data processing servers relative to the data processing server in number
According to occurring data syn-chronization interruption in synchronizing process, then the determining module 10 determines each database that data syn-chronization is interrupted, example
As other data processing servers are carrying out data syn-chronization to the 3rd database and the 4th database, and there is sync break,
Now, the processing module 30 re-executes data syn-chronization operation to two databases of data sync break, i.e., counted to the 3rd
Data syn-chronization operation is re-executed according to storehouse and the 4th database, data syn-chronization operation now is consistent with the above, herein not
Repeated again.
When the data processing server re-executes data syn-chronization operation to each database of data sync break,
If in data syn-chronization operating process is performed, having detected partial data and synchronously having been crossed by other data processing servers,
Now, in order to ensure the accuracy of data syn-chronization, update module 40 can complete whole data synchronization process, and with re-synchronization
Data update synchronous data, that is, update out the data of other data processing servers synchronized mistake, it is ensured that data are same
The accuracy of step.
The present invention further provides a kind of data synchronous system, the data synchronous system includes：Data processing server,
Network memory and database server, the data synchronization unit of the data processing server as described above.
The present embodiment propose data synchronous system, the data processing server receive data syn-chronization instruction when,
First determine the network memory of data processing server association, then it is determined that the network memory in obtain default
Data syn-chronization rule, finally according to the data syn-chronization of acquisition rule, determine the number of the data processing server association
Data syn-chronization operation is performed according to the database of pending data syn-chronization in the server of storehouse, and according to the database of determination.Realize
During synchrodata, even if to extend the synchronization that new data processing server carries out data, it is not required that in each data processing
Interpolation data synchronization rules in server, as long as data syn-chronization rule is added in network memory, follow-up each number
Just can be with logarithm according to the data in processing server as long as obtaining data syn-chronization rule from network memory according to processing server
Storehouse carries out the synchronization of data, and the synchronization to data is very convenient, due to shortening the time of interpolation data synchronization rules, so as to carry
The high whole efficiency of data syn-chronization.
It should be noted that herein, term " comprising ", "comprising" or its any other variant are intended to non-row
His property is included, so that process, method, article or device including a series of key elements not only include those key elements, and
And also including the other key elements being not expressly set out, or also include for this process, method, article or device institute inherently
Key element.In the absence of more restrictions, the key element limited by sentence "including a ...", it is not excluded that including this
Also there is other identical element in process, method, article or the device of key element.
The embodiments of the present invention are for illustration only, and the quality of embodiment is not represented.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can add the mode of required general hardware platform to realize by software, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Understood based on such, technical scheme is substantially done to prior art in other words
Going out the part of contribution can be embodied in the form of software product, and the computer software product is stored in a storage medium
In (such as ROM/RAM, magnetic disc, CD), including some instructions are to cause a station terminal equipment (can be mobile phone, computer, clothes
It is engaged in device, air conditioner, or network equipment etc.) perform method described in each embodiment of the invention.
The preferred embodiments of the present invention are these are only, are not intended to limit the scope of the invention, it is every to utilize this hair
Equivalent structure or equivalent flow conversion that bright specification and accompanying drawing content are made, or directly or indirectly it is used in other related skills
Art field, is included within the scope of the present invention.
Claims (11)
1. a kind of method of data synchronization, applied to data processing server, it is characterised in that the method for data synchronization includes：
When receiving data syn-chronization instruction, data processing server determines the network storage of the data processing server association
Device；
It is determined that the network memory in obtain default data syn-chronization rule；
According to the data syn-chronization of acquisition rule, treated in the database server for determining the data processing server association into
The database of row data syn-chronization, and data syn-chronization operation is performed according to the database of determination.
2. method of data synchronization as claimed in claim 1, it is characterised in that described to be advised according to the data syn-chronization of acquisition
Then, the database of pending data syn-chronization in the database server of the data processing server association is determined, and according to true
Fixed database, which performs the step of data syn-chronization is operated, to be included：
The data processing server analyzes whether the corresponding operating system of the data processing server meets data syn-chronization bar
Part；
It is same according to the data of acquisition when the corresponding operating system of the data processing server meets data syn-chronization condition
Step rule, it is determined that association database server in pending data syn-chronization database, and according to the database of determination perform
Data syn-chronization is operated.
3. method of data synchronization as claimed in claim 2, it is characterised in that the data processing server analyzes the data
The step of whether corresponding operating system of processing server meets data syn-chronization condition includes：
The data processing server analyzes the corresponding operating system of the data processing server by preset schedule program
Resource situation, wherein, the resource situation includes cpu resource or internal memory shows program MEM resources；
The resource situation based on analysis, determines whether the corresponding operating system of the data processing server meets data same
Step condition, wherein, when the resources left utilization rate of the cpu resource or MEM resources is more than predetermined threshold value, determine the data
The corresponding operating system of processing server meets data syn-chronization condition.
4. method of data synchronization as claimed in claim 1, it is characterised in that the execution performs number according to the database of determination
According to the step of simultaneously operating simultaneously, perform following steps：
The data processing server records the data syn-chronization and operates corresponding synchronizing process；
The synchronizing process of record is stored into the database journal of local text log and the database server,
In order to check data synchronization process.
5. the method for data synchronization as described in claim any one of 1-4, it is characterised in that the method for data synchronization is also wrapped
Include：
If other data processing servers that the data processing server detects relative to the data processing server exist
Occurs data syn-chronization interruption in data synchronization process, it is determined that each database that data syn-chronization is interrupted；
Data syn-chronization operation is re-executed to each database of data sync break；
If having detected, partial data is synchronous by other data processing servers, is updated with the data of re-synchronization
Synchronous data.
6. a kind of data synchronization unit, applied to data processing server, it is characterised in that the data synchronization unit includes：
Determining module, for when receiving data syn-chronization instruction, determining the network storage of the data processing server association
Device；
Acquisition module, for it is determined that the network memory in obtain default data syn-chronization rule；
Processing module, for the data syn-chronization rule according to acquisition, determines the data of the data processing server association
The database of pending data syn-chronization in the server of storehouse, and data syn-chronization operation is performed according to the database of determination.
7. data synchronization unit as claimed in claim 6, it is characterised in that the processing module includes：
Submodule is analyzed, for analyzing whether the corresponding operating system of the data processing server meets data syn-chronization condition；
Submodule is handled, for when the corresponding operating system of the data processing server meets data syn-chronization condition, according to
Obtain the data syn-chronization rule, it is determined that association database server in pending data syn-chronization database, and according to
The database of determination performs data syn-chronization operation.
8. data synchronization unit as claimed in claim 7, it is characterised in that the analysis submodule includes：
Analytic unit, the resource feelings for analyzing the corresponding operating system of the data processing server by preset schedule program
Condition, wherein, the resource situation includes cpu resource or internal memory shows program MEM resources；
Determining unit, for the resource situation based on analysis, determines the corresponding operating system of the data processing server
Whether data syn-chronization condition is met, wherein, it is more than predetermined threshold value in the resources left utilization rate of the cpu resource or MEM resources
When, determine that the corresponding operating system of the data processing server meets data syn-chronization condition.
9. the data synchronization unit stated such as claim 6, it is characterised in that the data synchronization unit also includes：
Logging modle, for while the processing module performs data syn-chronization operation according to the database of determination, recording institute
State data syn-chronization and operate corresponding synchronizing process；
Memory module, for the synchronizing process storage of record to be arrived into local text log and the database server
In database journal, in order to check data synchronization process.
10. the data synchronization unit as described in claim any one of 6-9, it is characterised in that the determining module, if being additionally operable to
It is same that data occur in data synchronization process in the other data processing servers detected relative to the data processing server
Step is interrupted, it is determined that each database that data syn-chronization is interrupted；
The processing module, is additionally operable to re-execute each database of data sync break data syn-chronization operation；
The data synchronization unit also includes：
Update module, if synchronous by other data processing servers for having detected partial data, with again
Synchronous data update synchronous data.
11. a kind of data synchronous system, it is characterised in that the data synchronous system includes：Data processing server, network are deposited
Reservoir and database server, data synchronization unit of the data processing server as described in claim any one of 6-10.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611271367.2A CN107040578B (en) | 2016-12-30 | 2016-12-30 | Data synchronization method, device and system |
| PCT/CN2017/089484 WO2018120681A1 (en) | 2016-12-30 | 2017-06-22 | Data synchronization method, device and system, data processing server, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611271367.2A CN107040578B (en) | 2016-12-30 | 2016-12-30 | Data synchronization method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107040578A true CN107040578A (en) | 2017-08-11 |
| CN107040578B CN107040578B (en) | 2020-09-01 |
Family
ID=59531287
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611271367.2A Active CN107040578B (en) | 2016-12-30 | 2016-12-30 | Data synchronization method, device and system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN107040578B (en) |
| WO (1) | WO2018120681A1 (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108629002A (en) * | 2018-05-03 | 2018-10-09 | 山东汇贸电子口岸有限公司 | A kind of big data comparison method and device based on kettle |
| CN109086295A (en) * | 2018-06-13 | 2018-12-25 | 中国平安人寿保险股份有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN109325037A (en) * | 2018-08-17 | 2019-02-12 | 中国平安人寿保险股份有限公司 | A kind of method of data synchronization, storage medium and server for mass data |
| CN109688256A (en) * | 2018-08-27 | 2019-04-26 | 平安科技（深圳）有限公司 | Calling-out method, system, equipment and computer readable storage medium |
| CN110545301A (en) * | 2018-05-29 | 2019-12-06 | 视联动力信息技术股份有限公司 | monitoring resource synchronization method and device |
| CN110661869A (en) * | 2019-09-26 | 2020-01-07 | 苏州浪潮智能科技有限公司 | Node online method, system, device, equipment and computer storage medium |
| CN111414416A (en) * | 2020-02-28 | 2020-07-14 | 平安科技（深圳）有限公司 | Data processing method, device, equipment and storage medium |
| CN114710387A (en) * | 2022-03-25 | 2022-07-05 | 江苏云从曦和人工智能有限公司 | Data transmission processing system |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109873857A (en) * | 2018-12-26 | 2019-06-11 | 广州凡象科技有限公司 | Mobile terminal method of data synchronization and device under a kind of no network state |
| CN109871413A (en) * | 2019-01-03 | 2019-06-11 | 深圳壹账通智能科技有限公司 | It is a kind of based on kettle for counting method, system, terminal device and medium |
| CN110955722B (en) * | 2019-11-05 | 2023-02-07 | 北京天元创新科技有限公司 | Data synchronization method and device |
| CN112732828A (en) * | 2020-12-22 | 2021-04-30 | 航天信息股份有限公司 | Cross-platform data sharing method based on data warehouse tool |
| CN112948340A (en) * | 2021-02-04 | 2021-06-11 | 北京金山云网络技术有限公司 | Data synchronization method and device, electronic equipment and readable storage medium |
| CN114547193A (en) * | 2022-01-29 | 2022-05-27 | 北京声智科技有限公司 | Data processing method, device, system, electronic equipment and storage medium |
| CN114564371A (en) * | 2022-04-30 | 2022-05-31 | 四川野马科技有限公司 | Operation monitoring method and system applied to keyboard |
| CN116095096B (en) * | 2023-01-05 | 2024-05-03 | 中国联合网络通信集团有限公司 | Data synchronization method, device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020055939A1 (en) * | 2000-11-06 | 2002-05-09 | Joseph Nardone | System for a configurable open database connectivity conduit |
| CN103379129A (en) * | 2012-04-12 | 2013-10-30 | 阿里巴巴集团控股有限公司 | Data synchronization method, servers and distributed system |
| CN103873519A (en) * | 2012-12-14 | 2014-06-18 | 北京金山云网络技术有限公司 | Data synchronization method, client end, server, terminal and system |
| CN106021494A (en) * | 2016-05-19 | 2016-10-12 | 中国银联股份有限公司 | Master data synchronization method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103699580B (en) * | 2013-12-03 | 2016-08-24 | 中国铁路总公司 | Database synchronization method and device |
-
2016
- 2016-12-30 CN CN201611271367.2A patent/CN107040578B/en active Active
-
2017
- 2017-06-22 WO PCT/CN2017/089484 patent/WO2018120681A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020055939A1 (en) * | 2000-11-06 | 2002-05-09 | Joseph Nardone | System for a configurable open database connectivity conduit |
| CN103379129A (en) * | 2012-04-12 | 2013-10-30 | 阿里巴巴集团控股有限公司 | Data synchronization method, servers and distributed system |
| CN103873519A (en) * | 2012-12-14 | 2014-06-18 | 北京金山云网络技术有限公司 | Data synchronization method, client end, server, terminal and system |
| CN106021494A (en) * | 2016-05-19 | 2016-10-12 | 中国银联股份有限公司 | Master data synchronization method and device |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108629002A (en) * | 2018-05-03 | 2018-10-09 | 山东汇贸电子口岸有限公司 | A kind of big data comparison method and device based on kettle |
| CN110545301A (en) * | 2018-05-29 | 2019-12-06 | 视联动力信息技术股份有限公司 | monitoring resource synchronization method and device |
| CN110545301B (en) * | 2018-05-29 | 2022-06-17 | 视联动力信息技术股份有限公司 | Monitoring resource synchronization method and device |
| CN109086295A (en) * | 2018-06-13 | 2018-12-25 | 中国平安人寿保险股份有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN109086295B (en) * | 2018-06-13 | 2023-05-30 | 中国平安人寿保险股份有限公司 | Data synchronization method, device, computer equipment and storage medium |
| CN109325037A (en) * | 2018-08-17 | 2019-02-12 | 中国平安人寿保险股份有限公司 | A kind of method of data synchronization, storage medium and server for mass data |
| CN109688256A (en) * | 2018-08-27 | 2019-04-26 | 平安科技（深圳）有限公司 | Calling-out method, system, equipment and computer readable storage medium |
| CN110661869A (en) * | 2019-09-26 | 2020-01-07 | 苏州浪潮智能科技有限公司 | Node online method, system, device, equipment and computer storage medium |
| CN111414416A (en) * | 2020-02-28 | 2020-07-14 | 平安科技（深圳）有限公司 | Data processing method, device, equipment and storage medium |
| WO2021169268A1 (en) * | 2020-02-28 | 2021-09-02 | 平安科技（深圳）有限公司 | Data processing method, apparatus and device, and storage medium |
| CN111414416B (en) * | 2020-02-28 | 2023-05-23 | 平安科技（深圳）有限公司 | Data processing method, device, equipment and storage medium |
| CN114710387A (en) * | 2022-03-25 | 2022-07-05 | 江苏云从曦和人工智能有限公司 | Data transmission processing system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107040578B (en) | 2020-09-01 |
| WO2018120681A1 (en) | 2018-07-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107040578A (en) | Method of data synchronization, device and system | |
| CN104246767B (en) | Telemetry system for cloud synchronization system | |
| CN110213371A (en) | Information consumption method, apparatus, equipment and computer storage medium | |
| KR101600129B1 (en) | Application efficiency engine | |
| CN109597685A (en) | Method for allocating tasks, device and server | |
| CN109144699A (en) | Distributed task dispatching method, apparatus and system | |
| JP2019523952A (en) | Streaming data distributed processing method and apparatus | |
| CN107291750A (en) | A kind of data migration method and device | |
| CN110442610A (en) | The method, apparatus of load balancing calculates equipment and medium | |
| CN109408286A (en) | Data processing method, device, system, computer readable storage medium | |
| CN107291744A (en) | It is determined that and with the method and device of the relationship between application program | |
| CN110287009A (en) | A kind of working node selection method, device, storage medium and server | |
| CN109062697A (en) | It is a kind of that the method and apparatus of spatial analysis service are provided | |
| CN109800269A (en) | Data managing method, device, computer equipment and storage medium | |
| CN108268296A (en) | Loading method, the client and server of page layout | |
| CN107423942A (en) | A kind of method and device of work flow | |
| CN108959048A (en) | The method for analyzing performance of modular environment, device and can storage medium | |
| CN105069029B (en) | A kind of real-time ETL system and method | |
| CN109359109A (en) | A kind of data processing method and system calculated based on distributed stream | |
| CN108009004A (en) | The implementation method of service application availability measurement monitoring based on Docker | |
| EP3945420A1 (en) | Method and apparatus for data processing, server and storage medium | |
| CN117455568B (en) | Transaction incentive resource transmitting method, device, computer equipment and storage medium | |
| CN109165712A (en) | Distributed generation method, device and computer storage medium by stages number | |
| CN104166650B (en) | Data storage device and date storage method | |
| CN107862040A (en) | The update method of data, device and a kind of cluster in a kind of caching of application example |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |