CN105447046A - Distributed system data consistency processing method, device and system - Google Patents
Distributed system data consistency processing method, device and system Download PDFInfo
- Publication number
- CN105447046A CN105447046A CN201410443886.7A CN201410443886A CN105447046A CN 105447046 A CN105447046 A CN 105447046A CN 201410443886 A CN201410443886 A CN 201410443886A CN 105447046 A CN105447046 A CN 105447046A
- Authority
- CN
- China
- Prior art keywords
- business datum
- module
- monitored object
- comparison
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention relates to the field of data processing, and in particular to a distributed system data consistency processing device. The distributed system data consistency processing device comprises a configuration module, a database message generation module and a data consistency comparison task module, wherein the configuration module is used for pre-configuring a monitoring object and a monitoring rule corresponding to the monitoring object; the database message generation module is used for monitoring and analyzing a transaction log file corresponding to the preset monitoring object, generating a business data change message when determining that the preset monitoring object changes, and sending the business data change message to the data consistency comparison task module; the business data change message contains business data to be compared; and the data consistency comparison task module is used for receiving the business data change message sent by the database message generation module, and comparing the consistency of the business data to be compared contained in the business data change message according to the preset monitoring rule to obtain a comparison result.
Description
Technical field
The present invention relates to technical field of data processing, particularly relate to a kind of distributed system data consistency processing method, device and system.
Background technology
Database be organize according to data structure, the warehouse of store and management data.Database is widely used at data Storage and Processing.The Distributed database transaction process of current Large-Scale Interconnected net application adopts BASE model mostly, the feature that this model is different from the ACID model of relational database is the mode adopting asynchronous refresh, permission data mode can be asynchronous for some time, as long as reach the final consistent of data.That is, BASE model, by sacrificing real-time high consistance, obtains high availability and high reliability.Therefore, how the problem just becoming needs and solve is monitored and processed to inconsistent data.
In prior art, there is a kind of method of monitoring data consistency between plurality of databases, the consistance monitoring that checking request carrys out trigger data is sent by application system, according to the critical field of the service logic, database address and the association multiple database that relate to described in described checking request, concrete data are taken out in corresponding Service Database, and mate corresponding service logic verification whether meet coherence request, to obtain check results.
But, the method provided due to prior art needs the checking request that sends according to application system can the consistance monitoring of trigger data, need to carry out transforming the consistance monitoring that can realize data to old application system, and being not suitable for all application systems and database, cost is higher.In addition, due to when carrying out data consistency monitoring, need real time access, query traffic data storehouse to obtain concrete data, when relating to data volume and being larger, larger pressure will be caused to Service Database, real-time reading Service Database also can cause the performance loss of system, so produce system responses slow, the defect such as monitored results fast and accurately cannot be obtained.
Summary of the invention
For solving the problems of the technologies described above, the invention discloses a kind of distributed system data consistency processing method, device and system, available lower cost and system loss realize data consistance comparison fast and accurately, can't produce pressure to Service Database.
Technical scheme is as follows:
According to the first aspect of the embodiment of the present invention, disclose a kind of distributed system data consistency treating apparatus, described device comprises configuration module, database message generation module and comparison of data consistency task module, wherein:
Described configuration module, for the monitoring rules of pre-configured monitored object and corresponding described monitored object;
Described database message generation module, for resolving the transaction log file of corresponding described monitored object, when determining that described monitored object changes, generating business datum in real time and changing message, sending described business datum to described comparison of data consistency task module and change message; Wherein, described business datum changes the business datum to be compared that message comprises corresponding described monitored object;
Described comparison of data consistency task module, the described business datum sent for receiving described database message generation module changes message, change to described business datum the business datum described to be compared comprised in message according to the monitoring rules of the described monitored object of described correspondence to compare in real time, obtain comparison result.
Preferably, described database message generation module specifically for:
When preset described monitored object derives from disparate databases, resolve the transaction log file of the corresponding described monitored object of each database respectively, when determining that described monitored object changes in the transaction log file that arbitrary database is corresponding, real-time generation business datum changes message, sends described business datum change message to described comparison of data consistency task module; Wherein, described business datum change message comprises business datum corresponding to the transaction log file of described monitored object in each database.
Preferably, described comparison of data consistency task module specifically for:
When receiving the business datum change message that described database message generation module sends, the business datum described to be compared that described data change message comprises is stored to local data base, utilize described local data base to carry out consistance comparison according to the monitoring rules of the described monitored object of described correspondence in real time to described business datum to be compared, obtain comparison result.
Preferably, described comparison of data consistency task module also for:
Hash operation is carried out to described business datum to be compared, according to Hash operation result, described business datum to be compared is divided into groups, the business datum of difference group is assigned in different task engines to compare in real time to described business datum according to preset described monitoring rules.
Preferably, described device also comprises:
Mass storage module, for receive and store described comparison of data consistency task module send comparison result; And, receive inquiry request, return the comparison data corresponding with described inquiry request.
Preferably, described device also comprises:
Consistent state recovery module, for when described comparison result shows that business datum exists inconsistent, realizes the consistent state recovery of business datum; Wherein, described consistent state recovery module specifically for:
According to remote interface and the call parameters of the pre-configured service application of described configuration module, call described service application to realize the consistent state recovery of business datum.
Preferably, described device also comprises:
Monitoring module, is configured for the running status monitoring described comparison of data consistency task module, when described running status shows that the quantity of non-comparison data exceedes predetermined threshold value, and output alarm information.
According to the second aspect of the embodiment of the present invention, disclose a kind of distributed system, described distributed system comprises data consistency treating apparatus and Service Database, wherein:
Described Service Database is used for storage service data;
Described data consistency treating apparatus is used for the monitoring rules of pre-configured monitored object and corresponding described monitored object; Resolve the transaction log file of the Service Database of corresponding described monitored object, when determining that described monitored object changes, generating business datum in real time and changing message; Wherein, described business datum changes the business datum to be compared that message comprises corresponding described monitored object; And, according to the monitoring rules of described correspondence described monitored object, the business datum described to be compared comprised in message is changed to described business datum and compare in real time, obtain comparison result.
Preferably, described data consistency treating apparatus specifically for:
When preset described monitored object derives from disparate databases, resolve the transaction log file of the corresponding described monitored object of each database respectively, when determining that described monitored object changes in the transaction log file that arbitrary database is corresponding, real-time generation business datum changes message, sends described business datum change message to described comparison of data consistency task module; Wherein, described business datum change message comprises business datum corresponding to the transaction log file of described monitored object in each database.
According to the third aspect of the embodiment of the present invention, disclose a kind of distributed system data consistency processing method, be applied to distributed system data consistency treating apparatus, described device comprises configuration module, database message generation module and comparison of data consistency task module, then described method comprises:
The monitoring rules of the pre-configured monitored object of described configuration module and corresponding described monitored object;
Described database message generation module resolves transaction log file corresponding to corresponding described monitored object, when determining that described monitored object changes, real-time generation business datum changes message, sends described business datum change message to described comparison of data consistency task module; Wherein, described business datum changes the business datum to be compared that message comprises corresponding described monitored object;
Described comparison of data consistency task module receives the business datum change message that described database message generation module sends, change to described business datum the business datum described to be compared comprised in message according to the monitoring rules of the described monitored object of described correspondence to compare in real time, obtain comparison result.
Preferably, described database message generation module resolves transaction log file corresponding to corresponding described monitored object, when determining that described monitored object changes, real-time generation business datum changes message, sends described business datum change message comprise to described comparison of data consistency task module:
When preset described monitored object derives from disparate databases, described database message generation module resolves the transaction log file of the corresponding described monitored object of each database respectively, when determining that described monitored object changes in the transaction log file that arbitrary database is corresponding, real-time generation business datum changes message, sends described business datum change message to described comparison of data consistency task module; Wherein, described business datum change message comprises business datum corresponding to the transaction log file of described monitored object in each database.
Preferably, described comparison of data consistency task module receives the business datum change message that described database message generation module sends, change to described business datum the business datum to be compared comprised in message according to preset monitoring rules to compare in real time, obtain comparison result and comprise:
When receiving the business datum change message that described database message generation module sends, the business datum described to be compared that described data change message comprises is stored to local data base by described comparison of data consistency task module, utilize described local data base to compare in real time to described business datum to be compared according to the monitoring rules of the preset described monitored object of described correspondence, obtain comparison result.
Preferably, described method also comprises:
Described comparison of data consistency task module carries out Hash operation to the business datum described to be compared received, according to Hash operation result, described business datum to be compared is divided into groups, the business datum of difference group is assigned in different task engines to compare in real time to described business datum according to preset described monitoring rules.
Preferably, described method also comprises:
Comparison of data consistency task module sends described comparison result to mass storage module, and described mass storage module receives and stores described comparison result; And described mass storage module receives inquiry request, returns the comparison data corresponding with described inquiry request.
Preferably, described method also comprises:
When described comparison result shows that business datum exists inconsistent, consistent state recovery module realizes the consistent state recovery of business datum;
Wherein, the consistent state recovery that described consistent state recovery module realizes business datum comprises:
Consistent state recovery module, according to the remote interface of pre-configured service application and call parameters, calls described service application to realize the consistent state recovery of business datum.
Preferably, described method also comprises:
Monitor the running status of described comparison of data consistency task module, when described running status shows that the quantity of non-comparison data exceedes predetermined threshold value, output alarm information.
The beneficial effect that an aspect of the embodiment of the present invention can reach is: in embodiments of the present invention, configuration module is pre-configured monitored object and monitoring rules, the db transaction journal file corresponding to preset monitored object by database message generation module is monitored and is resolved, when finding that monitored object changes in the transaction log file of arbitrary database, then generate the business datum comprising data to be compared in real time and change message, and described message is sent to described comparison of data consistency task module, according to preset monitoring rules, the business datum to be compared comprised in message is changed to described business datum to make described comparison of data consistency task module and carry out consistance comparison in real time, obtain comparison result.In the present invention, carry out triggering can trigger consistance comparison task according to preset monitored object and monitoring rules without the need to service application, therefore without the need to transforming existing application system, namely available less cost realizes consistance comparison and the process of data.In addition, because the present invention is when performing consistance comparison task, data to be compared are obtained by resolving transaction log file, without the need to real-time query Service Database, pressure can't be brought to Service Database, and reduce the system overhead and loss that bring thus, improve the real-time comparison speed of data, data consistance comparison fast and accurately can be realized.In addition, the db transaction journal file corresponding to described monitored object according to preset monitored object due to the present invention is monitored in real time, at described monitored object once change, namely generate business datum in real time and change message, the consistance comparison of real-time trigger data, and utilize comparison of data consistency task module to treat comparison data to carry out comparison in real time, therefore, method comparing provided by the invention real-time, and improve the efficiency of comparing.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, the accompanying drawing that the following describes is only some embodiments recorded in the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
The distributed system data consistency treating apparatus schematic diagram that Fig. 1 provides for the embodiment of the present invention;
The distributed system schematic diagram that Fig. 2 provides for the embodiment of the present invention;
The distributed system data consistency processing method one embodiment schematic diagram that Fig. 3 provides for the embodiment of the present invention;
The another embodiment schematic diagram of distributed system data consistency processing method that Fig. 4 provides for the embodiment of the present invention;
The distributed system data consistency processing method embodiment schematic diagram again that Fig. 5 provides for the embodiment of the present invention;
Another embodiment schematic diagram of distributed system data consistency processing method that Fig. 6 provides for the embodiment of the present invention.
Embodiment
Inventor is realizing finding in process of the present invention, the method provided due to prior art needs the checking request that sends according to application system can the consistance monitoring of trigger data, need to carry out transforming the consistance monitoring that can realize data to old application system, and being not suitable for all application systems and database, cost is higher.In addition, due to when carrying out data consistency monitoring, need real time access, query traffic data storehouse to obtain concrete data, when relating to data volume and being larger, larger pressure will be caused to Service Database, real-time reading Service Database also can cause the performance loss of system, so produce system responses slow, the defect such as monitored results fast and accurately cannot be obtained.
In addition, the method that prior art provides is by the mode back-checking result of the result of verification by daily record, mail or note, and the Integrated query of inconvenient data.On the other hand, the method that prior art provides is not owing to relating to multitasking, and when therefore facing a large amount of comparison data, the treatment effeciency of data is not high.Again on the one hand, due to do not relate in prior art data inconsistent time date restoring problem, therefore cannot realize the consistent state recovery process of data.
Based on this, the invention discloses a kind of distributed system data consistency processing method, device and system, cost that can be lower and system loss realize data consistance comparison fast and accurately, can't produce pressure to Service Database.
Technical scheme in the present invention is understood better in order to make those skilled in the art person, below in conjunction with the accompanying drawing in the embodiment of the present invention, technical scheme in the embodiment of the present invention is clearly and completely described, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, should belong to the scope of protection of the invention.
One embodiment
See Fig. 1, it is the distributed system data consistency treating apparatus schematic diagram that the embodiment of the present invention provides.Device shown in Fig. 1 is to realize the method shown in Fig. 3-Fig. 5.
A kind of distributed system data consistency treating apparatus 100, described device comprises configuration module 101, database message generation module 102 and comparison of data consistency task module 103, wherein:
Described configuration module, for the monitoring rules of pre-configured monitored object and corresponding described monitored object.
Described database message generation module, for resolving the transaction log file of corresponding described monitored object, when determining that described monitored object changes, generating business datum in real time and changing message, sending described business datum to described comparison of data consistency task module and change message; Wherein, described business datum changes the business datum to be compared that message comprises corresponding described monitored object.
Described comparison of data consistency task module, the business datum sent for receiving described database message generation module changes message, change to described business datum the business datum described to be compared comprised in message according to the monitoring rules of the described monitored object of described correspondence to compare in real time, obtain comparison result.
Be described in detail to the distributed system data consistency treating apparatus shown in Fig. 1 below.
During specific implementation, described configuration module 101 pre-sets the monitoring rules of monitored object and corresponding described monitored object.Wherein, described monitored object comprises database information and the service data information of needs monitoring.Described service data information can comprise again in described database the table and/or field that need to monitor.Such as, arrange and need the object of monitoring to be the business datum that field C in the table 1 of database A is corresponding.Usually, in order to ensure the consistance of data, need the business datum of monitoring usually from two different Service Databases.Such as, preset monitored object can be just the business datum that the field D in the business datum and database B that the field C in the table 1 of database A is corresponding in table 2 is corresponding, has incidence relation between two business datums.This incidence relation is embodied in that the two is identical type, same alike result is for characterizing the business datum of same intension usually.Such as, in an e-commerce system, business module A produces a business, in fund module B, produce a corresponding fund business simultaneously, the data of business module A are kept in database A, and the data of fund module B are kept in database B, and at this moment monitored object can be just the data associated with business module A in database A, and, the data associated with fund module B in database B.Arranging of monitored object can set according to specific needs, and the present invention does not limit at this.It should be noted that, the monitoring rules relied on when carrying out comparing also can set as required.Different monitored object can adopt identical monitoring rules, also can adopt different monitoring rules.Described monitoring rules can be such as how to change to the business datum of separate sources the rule that message carries out consistance comparison, and for example, described monitoring rules can also comprise: before carrying out comparing, the rule that the data type of field value to be compared is changed or computation rule, can be such as how to convert separate sources, dissimilar comparison data the data of same type to, compare to facilitate.The setting of the present invention to monitoring rules does not limit.
Further, described configuration module 101, except configuration monitoring object and monitoring rules, can also configure the message format generated when each business datum changes.During specific implementation, described configuration module 101 can send monitoring configuration messages to described database message generation module 102, the message format generated when described monitoring configuration messages can comprise monitored object and business datum change and message queue title.Described configuration module 101 can also send comparison configuration messages to described comparison of data consistency task module 103, carries out the monitoring rules of comparing time institute foundation for setting data consistance comparison task module.
The monitored object that described database message generation module 102 pre-sets according to described configuration module, resolves the transaction log file of the database corresponding with described monitored object.Relational database generally adopts the retouching operation of transaction log file record data, every bar daily record wherein or the logical operation performed by record, or records front picture and the after image of Update Table.Wherein, front similarly is operate the data trnascription before performing, and after image is the data trnascription after operation performs.The present invention is then by the transaction log file of monitor database and analyzes transaction log file, extracts SQL statement and the data of up-to-date change, thus realizes catching of data change event.In embodiments of the present invention, be different from prior art the mode of database being inquired about, accesses to obtain business datum to be compared, the present invention can be monitored by the data variation of transaction log file to database, do not take database own resources, can not to database build-up of pressure.In addition, be the up-to-date log content changed due to what analyze, catch, the efficiency of data capture is very high.
During specific implementation, described database message generation module 102 is according to preset monitored object, and the db transaction journal file corresponding to described monitored object is monitored in real time, and the content of analysis and drawing out change.When changing according to transaction log file determination business datum, then according to pre-set, message format that described business datum is corresponding generates business datum in real time and changes message, described business datum changes the business datum to be compared that message comprises corresponding described monitored object.Then, described database message generation module 102 sends described business datum to described comparison of data consistency task module 103 and changes message.Further, described comparison of data consistency task module 103 can also change message according to the configuration subscribing service data of described configuration module 101.At this moment, described database message generation module 102 can send the business datum corresponding with described subscription rules according to the subscription rules pre-set to described comparison of data consistency task module 103 and change message.Wherein, described business datum change message comprises business datum to be compared.
In a kind of possible implementation, described database message generation module 102 specifically for: when preset described monitored object derives from disparate databases, monitor and resolve transaction log file corresponding to each database respectively, when determining that described monitored object produces data change daily record in the transaction log file that arbitrary database is corresponding, then generate business datum and change message, and send described business datum change message to described comparison of data consistency task module 103; Wherein, described business datum change message comprises business datum corresponding to the transaction log file of described monitored object in each database.
Wherein, described comparison of data consistency task module 103 specifically for: when receive described database message generation module 102 send business datum change message time, the business datum to be compared that described business datum change message comprises is stored to local data base, utilize described local data base to carry out consistance comparison according to preset monitoring rules in real time to described business datum to be compared, obtain comparison result.
In a kind of possible implementation, described comparison of data consistency task module 103 also for: Hash operation is carried out to business datum to be compared, according to Hash operation result, business datum to be compared is divided into groups, the business datum of difference group is assigned in different task engines to carry out consistance comparison according to preset monitoring rules to described business datum.Such as can comprise, when the data volume of single business scenario is very large, comparison of data consistency task module first can carry out Hash operation to the key assignments of data to be compared, the data of same key assignments are divided in same group, thus the business datum of difference group is assigned in different task engines to perform consistance comparison task.In this way, can realize the load balancing of comparison task, the extensibility of system is stronger.
Further, described comparison of data consistency task module 103 is after acquisition comparison result, can send comparison result to mass storage module, then described device also comprises mass storage module, for receive and store described comparison of data consistency task module 103 send comparison result; And, receive inquiry request, return the comparison data corresponding with described inquiry request.
Further, described device also comprises: consistent state recovery module, for when described comparison result shows that business datum exists inconsistent, realizes the consistent state recovery of business datum.Particularly, described consistent state recovery module, specifically for according to the remote interface of the pre-configured service application of described configuration module 101 and call parameters, calls described service application to realize the consistent state recovery of business datum.
Further, described device also comprises: monitoring module, is configured for the running status monitoring described comparison of data consistency task module, when described running status shows that the quantity of non-comparison data exceedes predetermined threshold value, and output alarm information.Described comparison of data consistency task module can timing to monitoring module send running state information, to facilitate the monitoring of monitoring module.Such as, when do not compare in monitoring module determination comparison of data consistency task module or data volume that comparison makes mistakes more than 100 time, then output alarm information.
Another embodiment
The distributed system schematic diagram that Fig. 2 provides for the embodiment of the present invention.Distributed system shown in Fig. 2 is to realize the method shown in Fig. 3-Fig. 5.
A kind of distributed system 300, described distributed system 300 comprises data consistency treating apparatus 100 and Service Database 200, wherein:
Described Service Database 200 is for storage service data.
Described data consistency treating apparatus 100 is for the monitoring rules of pre-configured monitored object and corresponding described monitored object; Resolve the transaction log file of the Service Database of corresponding described monitored object, when determining that described monitored object changes, generating business datum in real time and changing message; Wherein, described business datum changes the business datum to be compared that message comprises corresponding described monitored object; And, according to preset described monitoring rules, the business datum to be compared comprised in message is changed to described business datum and compare in real time, obtain comparison result.
Wherein, described data consistency treating apparatus specifically for:
When preset described monitored object derives from disparate databases, resolve the transaction log file of the corresponding described monitored object of each database respectively, when determining that described monitored object changes in the transaction log file that arbitrary database is corresponding, real-time generation business datum changes message, sends described business datum change message to described comparison of data consistency task module; Wherein, described business datum change message comprises business datum corresponding to the transaction log file of described monitored object in each database.
Wherein, described distributed system can comprise one or more Service Database.
It should be noted that, in this embodiment, described data consistency treating apparatus can realize with reference to the device shown in Fig. 1.Its specific implementation with reference to the structure of device in the embodiment shown in Fig. 1 and annexation, can not repeat them here.
An embodiment again
See Fig. 3, it is the distributed system data consistency processing method one embodiment schematic diagram that the embodiment of the present invention provides.In the distributed system data consistency treating apparatus that can be applied to as shown in Figure 1 embodiment illustrated in fig. 3, described device comprises configuration module 101, database message generation module 102 and comparison of data consistency task module 103.
S301, the pre-configured monitored object of configuration module and the monitoring rules corresponding with described monitored object.
In embodiments of the present invention, described configuration module pre-sets monitored object and the monitoring rules corresponding with described monitored object.Wherein, described monitored object comprises database information and the service data information of needs monitoring.Described service data information can comprise again in described database the table and/or field that need to monitor.Such as, arrange and need the object of monitoring to be the business datum that field C in the table 1 of database A is corresponding.Usually, in order to ensure the consistance of data, need the business datum of monitoring usually from two different Service Databases.Such as, preset monitored object can be just the business datum that the field D in the business datum and database B that the field C in the table 1 of database A is corresponding in table 2 is corresponding, has incidence relation between two business datums.This incidence relation is embodied in that the two is identical type, same alike result is for characterizing the business datum of same intension usually.Such as, in an e-commerce system, business module A produces a business, in fund module B, produce a corresponding fund business simultaneously, the data of business module A are kept in database A, the data of fund module B are kept in database B, and at this moment monitored object can be just the data associated with fund module B in the data and database B associated with business module A in database A.Arranging of monitored object can set according to specific needs, and the present invention does not limit at this.It should be noted that, the monitoring rules relied on when carrying out comparing also can set as required.Different monitored object can adopt identical monitoring rules, also can adopt different monitoring rules.Described monitoring rules can be such as how to change to the business datum of separate sources the rule that message carries out consistance comparison, and for example, described monitoring rules can also comprise: before carrying out comparing, the rule change the data type of field value to be compared or computation rule.Can be such as how to convert separate sources, dissimilar comparison data the data of same type to, compare to facilitate.The setting of the present invention to monitoring rules does not limit.
Further, described configuration module, except configuration monitoring object and monitoring rules, can also configure the message format and message queue title that generate when each business datum changes.During specific implementation, described configuration module can push monitoring configuration information to described database message generation module, the message format generated when described monitoring configuration information can comprise monitored object and business datum change and message queue title.Described configuration module can also push comparison configuration information to described comparison of data consistency task module, carries out the monitoring rules of comparing time institute foundation for setting data consistance comparison task module.
S302, database message generation module resolves the transaction log file of corresponding described monitored object, when determining that described monitored object changes, generating business datum in real time and changing message, sending described business datum to comparison of data consistency task module and change message; Wherein, described business datum changes the business datum to be compared that message comprises corresponding described monitored object.
The monitored object that database message generation module pre-sets according to configuration module, resolves the transaction log file of the database corresponding with described monitored object.Relational database generally adopts the retouching operation of transaction log file record data, every bar daily record wherein or the logical operation performed by record, or records front picture and the after image of Update Table.Wherein, front similarly is operate the data trnascription before performing, and after image is the data trnascription after operation performs.The present invention is then by the transaction log file of monitor database and analyzes transaction log file, extracts SQL statement and the data of up-to-date change, thus realizes catching of data change event.In embodiments of the present invention, be different from prior art the mode of database being inquired about, accesses to obtain business datum to be compared, the present invention can be monitored by the data variation of transaction log file to database, do not take database own resources, can not to database build-up of pressure.In addition, be the up-to-date log content changed due to what analyze, catch, the efficiency of data capture is very high.
During specific implementation, the db transaction journal file that database message generation module is corresponding to described monitored object according to preset monitored object is monitored in real time, and the content of analysis and drawing out change.When changing according to transaction log file determination business datum, then according to pre-set, message format that described business number is corresponding generates business datum and changes message.Then, send described business datum to comparison of data consistency task module and change message.Further, comparison of data consistency task module can also change message according to the configuration subscribing service data of configuration module.At this moment, database message generation module after generation business datum changes message, then can send the business datum corresponding with described subscribe message according to the subscribe message prestored to described comparison of data consistency task module and change message.Wherein, described business datum change message comprises business datum to be compared.
When the monitored object of configuration module configuration is from disparate databases, transaction log file corresponding to each database is monitored and resolved to database message generation module respectively, when determining that described monitored object changes in the transaction log file that arbitrary database is corresponding, then generate business datum in real time and change message, and send described business datum change message to described comparison of data consistency task module; Wherein, described business datum change message comprises business datum corresponding to the transaction log file of described monitored object in each database.
That is, in embodiments of the present invention, can carry out continuing to resolve generating comparison data according to the db transaction journal file of preset monitored object to two separate sources, the data variation of any one database all can trigger comparison task in real time.Relative to the method for prior art, initiatively initiate monitoring request without the need to application, invasive is not had to application, the monitoring to existing legacy system can be realized easily.In addition, in the embodiment of the present invention, comparison data, from transaction log file, can not operate Service Database, can not cause application to other operations of Service Database.In addition, owing to resolving transaction log file, any change all can be caught, better ageing, and comparison result is more accurate.When after date restoring, new comparison result also can be embodied.
S303, comparison of data consistency task module receives the business datum change message that described database message generation module sends, change to described business datum the business datum to be compared comprised in message according to the monitoring rules of the described monitored object of described correspondence to compare in real time, obtain comparison result.
In embodiments of the present invention, comparison of data consistency task module is after the business datum receiving the transmission of described database message generation module changes message, the business datum to be compared that described data change message comprises can be stored to local data base, the monitoring rules utilizing described local data base to configure according to configuration module is compared in real time to described business datum to be compared, judge that whether data to be compared are consistent, record comparison result.In embodiments of the present invention, after obtaining data to be compared by parsing transaction log file, data to be compared can be compared stored in local data base when carrying out consistance comparison, the Performance And Reliability of system can be provided, to meet TPS (TransactionPerSecond, the issued transaction rate per second) demand of the real-time comparison of a large number of services data.
Further, when comparison of data consistency task module receives data to be compared, can also comprise: Hash operation is carried out to the business datum to be compared received, according to Hash operation result, business datum to be compared is divided into groups, the business datum of difference group is assigned in different task engines to carry out consistance comparison according to preset monitoring rules in real time to described business datum.Such as can comprise, when the data volume of single business scenario is very large, comparison of data consistency task module first can carry out Hash operation to the key assignments of data to be compared, the data of same key assignments are divided in same group, thus the business datum of difference group is assigned in different task engines to perform consistance comparison task.In this way, can realize the load balancing of comparison task, real-time is stronger.
Further, comparison of data consistency task module is after acquisition comparison result, and can send comparison result to mass storage module, described mass storage module receives and stores described comparison result.Comparison result can be stored to mass storage data module by comparison of data consistency task module in real time, also timing batch comparison data can be stored to mass storage data module.In addition, mass storage data module can also receive the inquiry request of other modules or device, returns the comparison data corresponding with described inquiry request.
Further, when described comparison result shows that business datum exists inconsistent, consistent state recovery module realizes the consistent state recovery of business datum.During specific implementation, consistent state recovery module, according to the remote interface of pre-configured service application and call parameters create-rule, is called described service application and is carried out business retry to realize the consistent state recovery of business datum.Preferably, http protocol realization far call can be adopted, be configured the httpurl and parameter extraction rule that call by configuration module.
Further, in embodiments of the present invention, the running status of monitoring module to described comparison of data consistency task module can also be utilized to monitor, when described running status shows that the quantity of non-comparison data exceedes predetermined threshold value, output alarm information.Described comparison of data consistency task module can timing to monitoring module send running state information, to facilitate the monitoring of monitoring module.Such as, when do not compare in monitoring module determination comparison of data consistency task module or data volume that comparison makes mistakes more than 100 time, then output alarm information.
In embodiments of the present invention, carry out triggering can trigger consistance comparison task according to preset monitored object and monitoring rules without the need to service application, therefore without the need to transforming existing application system, namely consistance comparison and the process of data can be realized by less cost.In addition, because the present invention is when performing consistance comparison task, data to be compared are obtained by resolving transaction log file, without the need to real-time query Service Database, pressure can't be brought to Service Database, and reduce the system overhead and loss that bring thus, improve the real-time comparison speed of data, data consistance comparison fast and accurately can be realized.
In addition, in a kind of possible implementation, the present invention adopts distributed stream data to realize affixing one's name to comparison task at multiple stage machine upper portion with lasting computing platform in real time, realizes the maximum equilibrium of resources conservation and treatment effeciency.
In addition, comparison data is stored in mass data storage module by the present invention, conveniently can realize the memory requirement of mass data and can realize real-time query comparison result.In addition, the present invention, by needing remote interface and the parameter of the service application called in allocation data recovering process, can realize the consistent state recovery function of data.
Another embodiment
See Fig. 4, it is an application scenarios schematic diagram of distributed system data consistency processing method of the present invention.As shown in Figure 4, a kind of implementation of illustrated data consistency treating apparatus can comprise database message generation system (module), in real time computing platform (onboard data consistance comparison task module), distributed KV storage system, configuration module and monitoring module.
Below in conjunction with the application scenarios of Fig. 4, the distributed system data consistency processing method that the embodiment of the present invention illustrated in fig. 5 provides is described in detail.
S501, the pre-configured monitored object of described configuration module and the monitoring rules corresponding with described monitored object.
Wherein, described monitored object comprises database and the business datum that described database message generation module carries out monitoring parsing.Described monitoring rules comprises rule when comparison of data consistency task module carries out consistance comparison.
Further, described configuration module can also need the logic that performs and method when configuration service data carry out consistent state recovery further.Such as, remote interface during configuration certain service application of far call and call parameters.
S502, described configuration module pushes the monitoring configuration information comprising monitored object to described database message generation module.
Described monitoring configuration information comprises database and the business datum that database message generation module carries out monitoring parsing, and the title of the message format generated when each business datum changes and message queue.
S503, described configuration module pushes the comparison configuration information comprising monitored object and monitoring rules to described comparison of data consistency task module.
Wherein, described comparison configuration information can comprise monitoring rules when carrying out consistance comparison to the business datum of separate sources.
S504, described comparison of data consistency task module creates local data base automatically when program starts in task execution machine.
S505, described comparison of data consistency task module changes message according to the configuration subscribing service data of configuration module.
S506, described database message generation module receives the subscribe message of comparison of data consistency task module transmission and the monitoring configuration information of configuration module transmission.
S507, the db transaction journal file of described database message generation module to monitored object carries out monitoring and resolves.
During specific implementation, database message generation module can according to the monitoring configuration information determination monitored object of configuration module, thus the db transaction journal file corresponding to monitored object carries out monitoring resolves.Such as can comprise, application scenarios as shown in Figure 4, the transaction log file of database message generation module to database A and database B is monitored, is resolved.
S508, described database message generation module, when determining that the business datum in described db transaction journal file changes, generates the business datum comprising data to be compared and changes message.
S509, the business datum comprising data to be compared described in described database message generation module sends to described comparison of data consistency task module changes message.
S510, described comparison of data consistency task module receives business datum and changes after message, by described business change message stored in carrying out comparison of data consistency according to the monitoring rules of configuration module configuration in local data base, obtains comparison result.
S511, described Data Data consistance comparison task module by comparison result stored in described distributed KV storage system.
Further, described Data Data consistance comparison task module timing batch is in comparison data write magnanimity KV storage system.
See another embodiment schematic diagram of distributed system data consistency processing method that Fig. 6, Fig. 6 provide for the embodiment of the present invention.Below in conjunction with Fig. 6, distributed system comparing process is described.
S601, receives data to be compared.
Wherein, data to be compared generative process and Fig. 3, Fig. 4 or embodiment illustrated in fig. 5 realize identical.
S602, by described data write local data base to be compared.
S603, checks whether data to be compared are all sent to local data base; If so, step S604 is entered; If not, start timer, when the timer expires, by data to be compared and comparison result write magnanimity KV database.Comparison result only has data centering to be compared only to have data to be sent to local data base for showing, fails to data to be compared to comparing.
S604, changes according to preset rule described data to be compared, calculates, and compares to the data after process.
According to comparison result, S605, judges that whether two data to be compared are consistent; If so, step S606 is entered; If not, step S607 is entered.
S606, if unanimously, is labeled as comparison success in the local database, enters step S608.
S607, if inconsistent, be labeled as comparison failure in the local database, enter step S608.
S608, by comparison result timing batch write magnanimity KV database.
S609, terminates program.
By with a concrete application example, the method shown in Fig. 5 or Fig. 6 is described below.Suppose that the method embodiment of the present invention provided is applied to an electronic business web site, the business module A of described e-commerce website produces a business, produces a corresponding fund business in fund module B simultaneously.The data of business module A are stored in mysql database A, and the data of fund module B are stored in mysql database B.In order to monitor the consistance of data between two modules, create transaction journal parsing and message transformation supplying system (internal database message generating module) and distributed stream data in real time with lasting computing system (onboard data consistance comparison task module).Meanwhile, local stored data base is set up for storing data to be compared; Set up distributed KV storage system, for storing comparison of data consistency result.
During specific implementation, utilize configuration module to carry out system configuration, such as, can comprise monitored object, monitoring rules, consistent state recovery processing logic etc.Then, call transaction journal parsing and the transaction journal binlog file of message transformation system to database A and database B and carry out parsing and generating messages.Then, according to the configuration of configuration module, create distributed stream data in real time with the task module (i.e. comparison of data consistency task module) of lasting computing system, receive transaction journal and resolve the business datum change message generated, compare at local data base, timing writes distributed KV storage system comparison result batch.Preferably, if the key assignments that the data volume of single business scenario first can treat very greatly comparison data carries out consistent hashing calculating, message is changed to data and divides into groups, by group, task matching is performed to multiple task engine.Further, the implementation status of monitoring module monitor task program, inquiry comparison result can be utilized.Further, consistent state recovery module can be triggered to inconsistent data is manual at monitoring module.Consistent state recovery module is carried out far call by configuration and is carried out business retry, carries out consistent state recovery.
It will be understood by those skilled in the art that; above exemplary illustration is carried out to embodiment of the method; below be not considered as limitation of the present invention, those skilled in the art all belong to protection scope of the present invention not paying other implementations obtained under creative work.
It should be noted that, owing to being explained in detail embodiment of the method, comparatively simple to the description of device embodiment, it will be appreciated by persons skilled in the art that and reference method embodiment can construct device embodiment of the present invention.Those skilled in the art all belong to protection scope of the present invention not paying other implementations obtained under creative work.
It should be noted that, in this article, the such as relational terms of first and second grades and so on is only used for an entity or operation to separate with another entity or operational zone, and not necessarily requires or imply the relation that there is any this reality between these entities or operation or sequentially.And, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thus make to comprise the process of a series of key element, method, article or equipment and not only comprise those key elements, but also comprise other key elements clearly do not listed, or also comprise by the intrinsic key element of this process, method, article or equipment.When not more restrictions, the key element limited by statement " comprising ... ", and be not precluded within process, method, article or the equipment comprising described key element and also there is other identical element.The present invention can describe in the general context of computer executable instructions, such as program module.Usually, program module comprises the routine, program, object, assembly, data structure etc. that perform particular task or realize particular abstract data type.Also can put into practice the present invention in a distributed computing environment, in these distributed computing environment, be executed the task by the remote processing devices be connected by communication network.In a distributed computing environment, program module can be arranged in the local and remote computer-readable storage medium comprising memory device.
Each embodiment in this instructions all adopts the mode of going forward one by one to describe, between each embodiment identical similar part mutually see, what each embodiment stressed is the difference with other embodiments.Especially, for device embodiment, because it is substantially similar to embodiment of the method, so describe fairly simple, relevant part illustrates see the part of embodiment of the method.Device embodiment described above is only schematic, the wherein said unit illustrated as separating component or can may not be and physically separates, parts as unit display can be or may not be physical location, namely can be positioned at a place, or also can be distributed in multiple network element.Some or all of module wherein can be selected according to the actual needs to realize the object of the present embodiment scheme.Those of ordinary skill in the art, when not paying creative work, are namely appreciated that and implement.The above is only the specific embodiment of the present invention; it should be pointed out that for those skilled in the art, under the premise without departing from the principles of the invention; can also make some improvements and modifications, these improvements and modifications also should be considered as protection scope of the present invention.
Claims (16)
1. a distributed system data consistency treating apparatus, is characterized in that, described device comprises configuration module, database message generation module and comparison of data consistency task module, wherein:
Described configuration module, for the monitoring rules of pre-configured monitored object and corresponding described monitored object;
Described database message generation module, for resolving the transaction log file of corresponding described monitored object, when determining that described monitored object changes, generating business datum in real time and changing message, sending described business datum to described comparison of data consistency task module and change message; Wherein, described business datum changes the business datum to be compared that message comprises corresponding described monitored object;
Described comparison of data consistency task module, the described business datum sent for receiving described database message generation module changes message, change to described business datum the business datum described to be compared comprised in message according to the monitoring rules of the described monitored object of described correspondence to compare in real time, obtain comparison result.
2. device according to claim 1, is characterized in that, described database message generation module specifically for:
When preset described monitored object derives from disparate databases, resolve the transaction log file of the corresponding described monitored object of each database respectively, when determining that described monitored object changes in the transaction log file that arbitrary database is corresponding, real-time generation business datum changes message, sends described business datum change message to described comparison of data consistency task module; Wherein, described business datum change message comprises business datum corresponding to the transaction log file of described monitored object in each database.
3. device according to claim 1 and 2, is characterized in that, described comparison of data consistency task module specifically for:
When receiving the business datum change message that described database message generation module sends, the business datum described to be compared that described data change message comprises is stored to local data base, utilize described local data base to carry out consistance comparison according to the monitoring rules of the described monitored object of described correspondence in real time to described business datum to be compared, obtain comparison result.
4. device according to claim 1, is characterized in that, described comparison of data consistency task module also for:
Hash operation is carried out to described business datum to be compared, according to Hash operation result, described business datum to be compared is divided into groups, the business datum of difference group is assigned in different task engines to compare in real time to described business datum according to preset described monitoring rules.
5. device according to claim 1, is characterized in that, described device also comprises:
Mass storage module, for receive and store described comparison of data consistency task module send comparison result; And, receive inquiry request, return the comparison data corresponding with described inquiry request.
6. device according to claim 1, is characterized in that, described device also comprises:
Consistent state recovery module, for when described comparison result shows that business datum exists inconsistent, realizes the consistent state recovery of business datum;
Wherein, described consistent state recovery module specifically for:
According to remote interface and the call parameters of the pre-configured service application of described configuration module, call described service application to realize the consistent state recovery of business datum.
7. device according to claim 1, is characterized in that, described device also comprises:
Monitoring module, is configured for the running status monitoring described comparison of data consistency task module, when described running status shows that the quantity of non-comparison data exceedes predetermined threshold value, and output alarm information.
8. a distributed system, is characterized in that, described distributed system comprises data consistency treating apparatus and Service Database, wherein:
Described Service Database is used for storage service data;
Described data consistency treating apparatus is used for the monitoring rules of pre-configured monitored object and corresponding described monitored object; Resolve the transaction log file of the Service Database of corresponding described monitored object, when determining that described monitored object changes, generating business datum in real time and changing message; Wherein, described business datum changes the business datum to be compared that message comprises corresponding described monitored object; And, according to the monitoring rules of described correspondence described monitored object, the business datum described to be compared comprised in message is changed to described business datum and compare in real time, obtain comparison result.
9. system according to claim 8, is characterized in that, described data consistency treating apparatus specifically for:
When preset described monitored object derives from disparate databases, resolve the transaction log file of the corresponding described monitored object of each database respectively, when determining that described monitored object changes in the transaction log file that arbitrary database is corresponding, real-time generation business datum changes message, sends described business datum change message to described comparison of data consistency task module; Wherein, described business datum change message comprises business datum corresponding to the transaction log file of described monitored object in each database.
10. a distributed system data consistency processing method, it is characterized in that, be applied to distributed system data consistency treating apparatus, described device comprises configuration module, database message generation module and comparison of data consistency task module, then described method comprises:
The monitoring rules of the pre-configured monitored object of described configuration module and corresponding described monitored object;
Described database message generation module resolves transaction log file corresponding to corresponding described monitored object, when determining that described monitored object changes, real-time generation business datum changes message, sends described business datum change message to described comparison of data consistency task module; Wherein, described business datum changes the business datum to be compared that message comprises corresponding described monitored object;
Described comparison of data consistency task module receives the business datum change message that described database message generation module sends, change to described business datum the business datum described to be compared comprised in message according to the monitoring rules of the described monitored object of described correspondence to compare in real time, obtain comparison result.
11. methods according to claim 10, it is characterized in that, described database message generation module resolves transaction log file corresponding to corresponding described monitored object, when determining that described monitored object changes, real-time generation business datum changes message, sends described business datum change message comprise to described comparison of data consistency task module:
When preset described monitored object derives from disparate databases, described database message generation module resolves the transaction log file of the corresponding described monitored object of each database respectively, when determining that described monitored object changes in the transaction log file that arbitrary database is corresponding, real-time generation business datum changes message, sends described business datum change message to described comparison of data consistency task module; Wherein, described business datum change message comprises business datum corresponding to the transaction log file of described monitored object in each database.
12. methods according to claim 10 or 11, it is characterized in that, described comparison of data consistency task module receives the business datum change message that described database message generation module sends, change to described business datum the business datum to be compared comprised in message according to preset monitoring rules to compare in real time, obtain comparison result and comprise:
When receiving the business datum change message that described database message generation module sends, the business datum described to be compared that described data change message comprises is stored to local data base by described comparison of data consistency task module, utilize described local data base to compare in real time to described business datum to be compared according to the monitoring rules of the preset described monitored object of described correspondence, obtain comparison result.
13. methods according to claim 10, is characterized in that, described method also comprises:
Described comparison of data consistency task module carries out Hash operation to the business datum described to be compared received, according to Hash operation result, described business datum to be compared is divided into groups, the business datum of difference group is assigned in different task engines to compare in real time to described business datum according to preset described monitoring rules.
14. methods according to claim 10, is characterized in that, described method also comprises:
Comparison of data consistency task module sends described comparison result to mass storage module, and described mass storage module receives and stores described comparison result; And described mass storage module receives inquiry request, returns the comparison data corresponding with described inquiry request.
15. methods according to claim 10, is characterized in that, described method also comprises:
When described comparison result shows that business datum exists inconsistent, consistent state recovery module realizes the consistent state recovery of business datum;
Wherein, the consistent state recovery that described consistent state recovery module realizes business datum comprises:
Consistent state recovery module, according to the remote interface of pre-configured service application and call parameters, calls described service application to realize the consistent state recovery of business datum.
16. methods according to claim 10, is characterized in that, described method also comprises:
Monitor the running status of described comparison of data consistency task module, when described running status shows that the quantity of non-comparison data exceedes predetermined threshold value, output alarm information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410443886.7A CN105447046A (en) | 2014-09-02 | 2014-09-02 | Distributed system data consistency processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410443886.7A CN105447046A (en) | 2014-09-02 | 2014-09-02 | Distributed system data consistency processing method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105447046A true CN105447046A (en) | 2016-03-30 |
Family
ID=55557230
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410443886.7A Pending CN105447046A (en) | 2014-09-02 | 2014-09-02 | Distributed system data consistency processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105447046A (en) |
Cited By (31)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106097104A (en) * | 2016-06-07 | 2016-11-09 | 众安在线财产保险股份有限公司 | A kind of judge the method and system of internet data integrity under distributed system |
| CN106339408A (en) * | 2016-08-10 | 2017-01-18 | 深圳中兴网信科技有限公司 | Data synchronization method, data synchronization device and server |
| CN106371984A (en) * | 2016-08-31 | 2017-02-01 | 广州品唯软件有限公司 | Data monitoring method, equipment and system |
| CN106371973A (en) * | 2016-08-31 | 2017-02-01 | 广州品唯软件有限公司 | Data processing method, equipment and system |
| CN106375382A (en) * | 2016-08-26 | 2017-02-01 | 苏州蓝海彤翔系统科技有限公司 | Data distribution method and system, control server and node client |
| CN106528664A (en) * | 2016-10-21 | 2017-03-22 | 广东亿迅科技有限公司 | Distributed relation database based transaction data monitoring method and system |
| CN107247749A (en) * | 2017-05-25 | 2017-10-13 | 阿里巴巴集团控股有限公司 | A kind of database positioning determines method, consistency verification method and device |
| CN107273378A (en) * | 2016-04-07 | 2017-10-20 | 阿里巴巴集团控股有限公司 | A kind of file memory method and device |
| CN107360006A (en) * | 2017-06-12 | 2017-11-17 | 北京三快在线科技有限公司 | A kind of resource billing method and device |
| CN107633015A (en) * | 2017-08-23 | 2018-01-26 | 阿里巴巴集团控股有限公司 | A kind of data processing method, device and equipment |
| CN107644077A (en) * | 2017-09-19 | 2018-01-30 | 杭州贝购科技有限公司 | Data consistency monitoring method, computer equipment and storage medium |
| CN107995242A (en) * | 2016-10-27 | 2018-05-04 | 北京京东尚科信息技术有限公司 | A kind of method for processing business and system |
| CN108376142A (en) * | 2018-01-10 | 2018-08-07 | 北京思特奇信息技术股份有限公司 | A kind of distributed memory database method of data synchronization and system |
| CN109254998A (en) * | 2018-09-04 | 2019-01-22 | 中国联合网络通信集团有限公司 | Data managing method, internet of things equipment, database server and system |
| CN109491767A (en) * | 2018-11-13 | 2019-03-19 | 上海联寓智能科技有限公司 | The processing method and distributed system of distributed transaction |
| CN109902256A (en) * | 2019-02-26 | 2019-06-18 | 北京金堤科技有限公司 | Monitoring method, device, electronic equipment and the computer-readable medium of data |
| CN110012008A (en) * | 2019-04-02 | 2019-07-12 | 山东国子软件股份有限公司 | Heterogeneous data source distributed coordination coherence method and system based on http protocol |
| CN110059068A (en) * | 2019-04-11 | 2019-07-26 | 厦门网宿有限公司 | Data verification method and data verification system in a kind of distributed memory system |
| CN110069462A (en) * | 2019-04-11 | 2019-07-30 | 苏州浪潮智能科技有限公司 | Batch image file consistency checking method and system between a kind of different server |
| CN110347746A (en) * | 2019-06-13 | 2019-10-18 | 武汉达梦数据库有限公司 | A kind of heterogeneous database synchrodata consistency desired result method and device |
| CN110363640A (en) * | 2019-07-08 | 2019-10-22 | 中国平安人寿保险股份有限公司 | Monitoring system service operation method, apparatus and storage medium, server |
| CN110377588A (en) * | 2019-07-23 | 2019-10-25 | 中国工商银行股份有限公司 | A kind of database object access monitoring method, server and terminal |
| CN110674145A (en) * | 2019-08-15 | 2020-01-10 | 平安普惠企业管理有限公司 | Data consistency detection method and device, computer equipment and storage medium |
| CN110704273A (en) * | 2019-08-28 | 2020-01-17 | 贝壳技术有限公司 | Configuration information processing method and device, electronic equipment and storage medium |
| CN111177165A (en) * | 2019-12-23 | 2020-05-19 | 拉扎斯网络科技（上海）有限公司 | Method, device and equipment for detecting data consistency |
| CN111274255A (en) * | 2020-01-20 | 2020-06-12 | 拉扎斯网络科技（上海）有限公司 | Service data monitoring method and system, monitoring architecture, equipment and storage medium |
| WO2020119709A1 (en) * | 2018-12-11 | 2020-06-18 | 中兴通讯股份有限公司 | Data merging implementation method, device, system, and storage medium |
| CN111563088A (en) * | 2020-04-20 | 2020-08-21 | 成都库珀区块链科技有限公司 | Data consistency detection method and device |
| CN112363873A (en) * | 2020-11-27 | 2021-02-12 | 上海爱数信息技术股份有限公司 | Distributed consistent backup and recovery system and backup method thereof |
| WO2021027126A1 (en) * | 2019-08-14 | 2021-02-18 | 平安科技（深圳）有限公司 | Database monitoring method, apparatus, device and computer-readable storage medium |
| CN113051120A (en) * | 2019-12-26 | 2021-06-29 | 医渡云（北京）技术有限公司 | Data monitoring method and device, readable medium and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101094051A (en) * | 2007-06-27 | 2007-12-26 | 中国移动通信集团四川有限公司 | System and method for synchronizing comparison of data consistency |
| CN101998354A (en) * | 2009-08-24 | 2011-03-30 | 中国移动通信集团甘肃有限公司 | Method and device for checking user data consistency |
| CN102682096A (en) * | 2012-04-27 | 2012-09-19 | 北京航空航天大学 | Collaborative management device and collaborative management method for simulation resource information and model source codes |
-
2014
- 2014-09-02 CN CN201410443886.7A patent/CN105447046A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101094051A (en) * | 2007-06-27 | 2007-12-26 | 中国移动通信集团四川有限公司 | System and method for synchronizing comparison of data consistency |
| CN101998354A (en) * | 2009-08-24 | 2011-03-30 | 中国移动通信集团甘肃有限公司 | Method and device for checking user data consistency |
| CN102682096A (en) * | 2012-04-27 | 2012-09-19 | 北京航空航天大学 | Collaborative management device and collaborative management method for simulation resource information and model source codes |
Non-Patent Citations (1)
| Title |
|---|
| 王玉标等: "异构环境下数据库增量同步更新机制", 《计算机工程与设计》 * |
Cited By (43)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107273378A (en) * | 2016-04-07 | 2017-10-20 | 阿里巴巴集团控股有限公司 | A kind of file memory method and device |
| CN106097104A (en) * | 2016-06-07 | 2016-11-09 | 众安在线财产保险股份有限公司 | A kind of judge the method and system of internet data integrity under distributed system |
| CN106339408A (en) * | 2016-08-10 | 2017-01-18 | 深圳中兴网信科技有限公司 | Data synchronization method, data synchronization device and server |
| CN106339408B (en) * | 2016-08-10 | 2019-08-23 | 深圳中兴网信科技有限公司 | Method of data synchronization, data synchronization unit and server |
| CN106375382A (en) * | 2016-08-26 | 2017-02-01 | 苏州蓝海彤翔系统科技有限公司 | Data distribution method and system, control server and node client |
| CN106371984B (en) * | 2016-08-31 | 2019-03-26 | 广州品唯软件有限公司 | A kind of data monitoring method, equipment and system |
| CN106371984A (en) * | 2016-08-31 | 2017-02-01 | 广州品唯软件有限公司 | Data monitoring method, equipment and system |
| CN106371973A (en) * | 2016-08-31 | 2017-02-01 | 广州品唯软件有限公司 | Data processing method, equipment and system |
| CN106371973B (en) * | 2016-08-31 | 2019-04-09 | 广州品唯软件有限公司 | A kind of data processing method, equipment and system |
| CN106528664A (en) * | 2016-10-21 | 2017-03-22 | 广东亿迅科技有限公司 | Distributed relation database based transaction data monitoring method and system |
| CN107995242A (en) * | 2016-10-27 | 2018-05-04 | 北京京东尚科信息技术有限公司 | A kind of method for processing business and system |
| US10789243B2 (en) | 2017-05-25 | 2020-09-29 | Alibaba Group Holding Limited | Database state determining method and device, and consistency verifying method and device |
| CN107247749B (en) * | 2017-05-25 | 2020-08-25 | 创新先进技术有限公司 | Database state determination method, consistency verification method and device |
| CN107247749A (en) * | 2017-05-25 | 2017-10-13 | 阿里巴巴集团控股有限公司 | A kind of database positioning determines method, consistency verification method and device |
| CN107360006B (en) * | 2017-06-12 | 2021-01-22 | 北京三快在线科技有限公司 | Resource charging method and device |
| CN107360006A (en) * | 2017-06-12 | 2017-11-17 | 北京三快在线科技有限公司 | A kind of resource billing method and device |
| CN107633015A (en) * | 2017-08-23 | 2018-01-26 | 阿里巴巴集团控股有限公司 | A kind of data processing method, device and equipment |
| CN107644077A (en) * | 2017-09-19 | 2018-01-30 | 杭州贝购科技有限公司 | Data consistency monitoring method, computer equipment and storage medium |
| CN108376142B (en) * | 2018-01-10 | 2021-05-14 | 北京思特奇信息技术股份有限公司 | Distributed memory database data synchronization method and system |
| CN108376142A (en) * | 2018-01-10 | 2018-08-07 | 北京思特奇信息技术股份有限公司 | A kind of distributed memory database method of data synchronization and system |
| CN109254998B (en) * | 2018-09-04 | 2021-06-15 | 中国联合网络通信集团有限公司 | Data management method, Internet of things equipment, database server and system |
| CN109254998A (en) * | 2018-09-04 | 2019-01-22 | 中国联合网络通信集团有限公司 | Data managing method, internet of things equipment, database server and system |
| CN109491767A (en) * | 2018-11-13 | 2019-03-19 | 上海联寓智能科技有限公司 | The processing method and distributed system of distributed transaction |
| CN111309799A (en) * | 2018-12-11 | 2020-06-19 | 中兴通讯股份有限公司 | Method, device and system for realizing data merging and storage medium |
| WO2020119709A1 (en) * | 2018-12-11 | 2020-06-18 | 中兴通讯股份有限公司 | Data merging implementation method, device, system, and storage medium |
| CN109902256A (en) * | 2019-02-26 | 2019-06-18 | 北京金堤科技有限公司 | Monitoring method, device, electronic equipment and the computer-readable medium of data |
| CN110012008A (en) * | 2019-04-02 | 2019-07-12 | 山东国子软件股份有限公司 | Heterogeneous data source distributed coordination coherence method and system based on http protocol |
| CN110069462A (en) * | 2019-04-11 | 2019-07-30 | 苏州浪潮智能科技有限公司 | Batch image file consistency checking method and system between a kind of different server |
| CN110059068A (en) * | 2019-04-11 | 2019-07-26 | 厦门网宿有限公司 | Data verification method and data verification system in a kind of distributed memory system |
| CN110347746A (en) * | 2019-06-13 | 2019-10-18 | 武汉达梦数据库有限公司 | A kind of heterogeneous database synchrodata consistency desired result method and device |
| CN110363640B (en) * | 2019-07-08 | 2023-07-25 | 中国平安人寿保险股份有限公司 | Monitoring system service operation method and device, storage medium and server |
| CN110363640A (en) * | 2019-07-08 | 2019-10-22 | 中国平安人寿保险股份有限公司 | Monitoring system service operation method, apparatus and storage medium, server |
| CN110377588A (en) * | 2019-07-23 | 2019-10-25 | 中国工商银行股份有限公司 | A kind of database object access monitoring method, server and terminal |
| WO2021027126A1 (en) * | 2019-08-14 | 2021-02-18 | 平安科技（深圳）有限公司 | Database monitoring method, apparatus, device and computer-readable storage medium |
| CN110674145A (en) * | 2019-08-15 | 2020-01-10 | 平安普惠企业管理有限公司 | Data consistency detection method and device, computer equipment and storage medium |
| CN110674145B (en) * | 2019-08-15 | 2024-04-19 | 西藏松岩科技有限公司 | Data consistency detection method, device, computer equipment and storage medium |
| CN110704273A (en) * | 2019-08-28 | 2020-01-17 | 贝壳技术有限公司 | Configuration information processing method and device, electronic equipment and storage medium |
| CN110704273B (en) * | 2019-08-28 | 2024-01-05 | 贝壳技术有限公司 | Processing method and device of configuration information, electronic equipment and storage medium |
| CN111177165A (en) * | 2019-12-23 | 2020-05-19 | 拉扎斯网络科技（上海）有限公司 | Method, device and equipment for detecting data consistency |
| CN113051120A (en) * | 2019-12-26 | 2021-06-29 | 医渡云（北京）技术有限公司 | Data monitoring method and device, readable medium and electronic equipment |
| CN111274255A (en) * | 2020-01-20 | 2020-06-12 | 拉扎斯网络科技（上海）有限公司 | Service data monitoring method and system, monitoring architecture, equipment and storage medium |
| CN111563088A (en) * | 2020-04-20 | 2020-08-21 | 成都库珀区块链科技有限公司 | Data consistency detection method and device |
| CN112363873A (en) * | 2020-11-27 | 2021-02-12 | 上海爱数信息技术股份有限公司 | Distributed consistent backup and recovery system and backup method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105447046A (en) | Distributed system data consistency processing method, device and system | |
| CN109033123B (en) | Big data-based query method and device, computer equipment and storage medium | |
| CN109255622B (en) | Traceable anti-fake data storage system | |
| US20170279840A1 (en) | Automated event id field analysis on heterogeneous logs | |
| CN106980699B (en) | Data processing platform and system | |
| CN103034735A (en) | Big data distributed file export method | |
| WO2020253122A1 (en) | Data verification method and device, computer equipment and storage medium | |
| CN107133231B (en) | Data acquisition method and device | |
| CN105989140B (en) | A kind of data block processing method and equipment | |
| WO2018192432A1 (en) | Working information processing method and apparatus, computer device, and storage medium | |
| CN110222028A (en) | A kind of data managing method, device, equipment and storage medium | |
| CN113127307A (en) | Method for processing tracing request, related device, system and storage medium | |
| CN105491078A (en) | Data processing method and device in SOA system, and SOA system | |
| CN104639546A (en) | Multi-biometric feature comprehensive authentication method, device and system | |
| CN113672692B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN107885634B (en) | Method and device for processing abnormal information in monitoring | |
| WO2016095716A1 (en) | Fault information processing method and related device | |
| CN110661851A (en) | Data exchange method and device | |
| CN113836237A (en) | Method and device for auditing data operation of database | |
| CN111159131A (en) | Performance optimization method, device, equipment and computer readable storage medium | |
| US10855563B2 (en) | Supplementing log messages with metadata | |
| CN109389271B (en) | Application performance management method and system | |
| CN110737655A (en) | Method and device for reporting data | |
| CN105791350B (en) | Project data processing method and system | |
| CN105718485B (en) | A kind of method and device by data inputting database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160330 |
|
| RJ01 | Rejection of invention patent application after publication |