CN104252481B - The dynamic check method and apparatus of master-slave database consistency - Google Patents

The dynamic check method and apparatus of master-slave database consistency Download PDF

Info

Publication number
CN104252481B
CN104252481B CN201310263694.3A CN201310263694A CN104252481B CN 104252481 B CN104252481 B CN 104252481B CN 201310263694 A CN201310263694 A CN 201310263694A CN 104252481 B CN104252481 B CN 104252481B
Authority
CN
China
Prior art keywords
data
database
major key
base recording
primary
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201310263694.3A
Other languages
Chinese (zh)
Other versions
CN104252481A (en
Inventor
叶晖俊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Advanced New Technologies Co Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN201310263694.3A priority Critical patent/CN104252481B/en
Publication of CN104252481A publication Critical patent/CN104252481A/en
Application granted granted Critical
Publication of CN104252481B publication Critical patent/CN104252481B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/004—Error avoidance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/275—Synchronous replication

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Quality & Reliability (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

This application discloses the dynamic check method and apparatus of master-slave database consistency, wherein this method includes：Repeat following steps n times, wherein N is preset natural number：From data-base recording corresponding with the major key in difference list of primary keys is retrieved in primary database, to obtain primary database record set；From from corresponding with the major key in difference list of primary keys data-base recording is retrieved in database, to obtain from database record set；It obtains primary database record set and concentrates the major key corresponding to the data-base recording having differences from data-base recording, the major key in difference list of primary keys is updated to the major key corresponding to the data-base recording having differences；After having executed n times step, if still having major key in difference list of primary keys, judge primary database with it is inconsistent from database, otherwise, then judge that primary database is consistent with from database.Present application addresses in the related technology because of the technical problem of the result inaccuracy of consistency desired result caused by synchronization delay.

Description

The dynamic check method and apparatus of master-slave database consistency
Technical field
This application involves internet arenas, in particular to a kind of dynamic check method of master-slave database consistency And device.
Background technology
MySQL is widely used PostgreSQL database software in internet industry.Many large-scale websites are all using MySQL Database, as business scale constantly increases, a MySQL database has been difficult the huge data access amount of support.
Currently, a more effective solution scheme is exactly to establish MySQL principal and subordinate's clusters, wherein from database（Also known as The either standby library of standby database）It is primary database（Also known as master library）Complete mirror image.It is only enterprising in master library for write request It goes, on read request meeting equiblibrium mass distribution to all databases, more capacity can be provided by allowing for entire data-base cluster in this way Read-write service.If MySQL master libraries receive newly-increased either change data, just by synchronization means, by these increase newly or Person is the real time data synchronization of change to for library.
Although there are many kinds of the tools of MySQL master-slave synchronisations, due to increasingly complicated, the database failure of production environment Occasionally there is generation, MySQL Binlog are there is also some known Bug, and data are inconsistent between inevitably will appear principal and subordinate Situation.The data in principal and subordinate library are inconsistent, can substantially reduce the availability of entire data-base cluster, while being likely to result in serious Fault in production.Therefore, it checks the data consistency of principal and subordinate's data, search data inconsistent in master-slave database and it is carried out Reparation just becomes particularly important.In addition, when researching and developing types of databases master-slave synchronisation tool, data consistency school is quickly carried out It tests, searches and export inconsistent data, be indispensable in the research and development of database synchronization tool, test.
Large-scale Internet enterprises are all ceaselessly online services in 7*24 hours, for the data base machine on production line, It is required that more harsh.Also the requirement of following several respects is just proposed to the verification of master-slave database dynamic conformance：
1）Calibration tool does not allow generally in master library, data is written from library；
2）Production environment database cannot stop externally servicing, and have hundreds of thousands of data changes each second, or even not The of short duration stopping of energy externally services, i.e., cannot lock full table；
3）Production environment flow pressure is big, and the flow load of calibration tool needs to be controllable；
4）If it find that data are inconsistent, it would be desirable to be able to be modified, or the modified data of output, then turn by artificial immediately Data are repaired in operation.
The data base consistency(-tance) calibration tool often having at present mainly has：Static state is carried out using Check Sum SQL statements The mk-table-checksum of verification；Or the special method of synchronization is utilized, such as carry out the dynamic check of fast illuminated The Binlog of Statement patterns is synchronized.Using the Binlog of more usually ROW patterns in practical application, however these Which field that the data base consistency(-tance) calibration tool increased income can not position specifically which data is inconsistent, can not more complete Data repair.So that such Open-Source Tools can not be applied to actual production environment, it can not also be applied to test Error in data caused by positioning BUG synchronous with searching data library in environment.
The verification of general logarithm evidence is all that will be carried out in the primary database of same time and standby database in the prior art It compares to determine whether the data between two databases have realized synchronization, however, because the synchronization of data is usually in master data After data update occurs for library, newer data are synchronized in standby database by primary database again, result in data to synchronize in this way During can have certain delay, it is past if being compared with the data between two databases of same time point Toward can because delay problem cause should to be determined as that consistent data are judged to it is inconsistent.For example, the time delay of database synchronization is 1ms, has then just carried out consistency desired result in this 1ms, and it is same to standby database to allow for the primary database in this 1ms in this way The data of step are not detected, to will produce a wrong judging result.
As it can be seen that in existing method of calibration detection process, need to locking data table, or the entire database of locking, Even stop external some period of time in service, it is inaccurate so as to cause the result of consistency desired result.In view of the above-mentioned problems, not yet proposing Effective solution method.
Invention content
This application provides a kind of dynamic check method and apparatus of master-slave database consistency, at least to solve existing skill Because of the technical problem of the result inaccuracy of consistency desired result caused by synchronization delay in art, raising master-slave database is reached Verify the technique effect of accuracy.
According to the one side of the application, a kind of dynamic check method of master-slave database consistency is provided, including：Weight Following steps n times are executed again, wherein N is preset natural number：From retrieval and difference list of primary keys in primary database The corresponding data-base recording of major key, to obtain primary database record set；Retrieval and above-mentioned difference list of primary keys from from database In the corresponding data-base recording of major key, to obtain from database record set；Obtain above-mentioned primary database record set with it is above-mentioned from Data-base recording concentrates the major key corresponding to the data-base recording having differences, and the major key in above-mentioned difference list of primary keys is updated For the major key corresponding to the above-mentioned data-base recording having differences；After having executed n times above-mentioned steps, if above-mentioned difference primary key column Still have major key in table, then judge above-mentioned primary database with it is above-mentioned inconsistent from database, otherwise, judge above-mentioned primary database With it is above-mentioned consistent from database.
Optionally, judging above-mentioned primary database and above-mentioned after database is inconsistent, above-mentioned master-slave database one Cause property dynamic check method further include：Execute data reparation operation, wherein above-mentioned data reparation, which operates, includes：From having executed N It is searched in the above-mentioned difference list of primary keys obtained after secondary above-mentioned steps simultaneously by above-mentioned primary database and above-mentioned from database The data-base recording major key that includes, be denoted as the major key of the first major key type, the use of in above-mentioned primary database with type be above-mentioned The corresponding data-base recording covering of major key of first major key type is above-mentioned corresponding with above-mentioned first major key type from database Data-base recording；It is searched by above-mentioned primary database in the above-mentioned difference list of primary keys obtained after having executed n times above-mentioned steps In data-base recording include but the major key that does not include by the above-mentioned data-base recording from database, be denoted as the second master The major key of key type copies to data-base recording corresponding with the above-mentioned major key of second major key type in above-mentioned primary database It states from database；It is searched by above-mentioned from data in the above-mentioned difference list of primary keys obtained after having executed n times above-mentioned steps Major key that is that data-base recording in library includes but not including by the data-base recording in above-mentioned primary database, is denoted as third Major key type, and data-base recording corresponding with the above-mentioned major key of third major key type is deleted from database from above-mentioned.
Optionally, obtain above-mentioned primary database record set concentrates the database having differences to remember with above-mentioned from data-base recording Recording corresponding major key includes：If same major key in above-mentioned primary database record set corresponding data-base recording with it is above-mentioned from Data-base recording concentrates corresponding data-base recording different, then the major key is recorded as the above-mentioned data-base recording institute having differences Corresponding major key；If same major key is in above-mentioned primary database record set there are corresponding data-base recording and above-mentioned from number According to corresponding data-base recording is not present in the record set of library, then the major key is recorded as the above-mentioned data-base recording institute having differences Corresponding major key；If same major key is concentrated from data-base recording there are corresponding data-base recording and in above-mentioned main number above-mentioned According to corresponding data-base recording is not present in the record set of library, then the major key is recorded as the above-mentioned data-base recording institute having differences Corresponding major key.
Optionally, before repeating above-mentioned steps n times, the dynamic check method of above-mentioned master-slave database consistency is also Including：Generate the initial value of above-mentioned difference list of primary keys, wherein the difference list of primary keys of above-mentioned generation initialization includes：Parallel Ground executes the multiple tasks in task queue to scan above-mentioned primary database and the above-mentioned data-base recording from database, to obtain Obtain the initial value of above-mentioned difference list of primary keys, wherein each above-mentioned task includes following operation, wherein task minimum key value Initial value is the minimum value in the corresponding major key of all data-base recordings of detected data table in above-mentioned primary database：Above-mentioned It is pre- that since data-base recording corresponding with above-mentioned task minimum key value the sequential scan of major key from small to large is pressed in primary database The data-base recording for determining line number, by Current Scan in above-mentioned primary database to data-base recording be recorded as current primary database and remember Record collection obtains the maximum value in the major key corresponding to the data-base recording in above-mentioned current primary database record set as task most Big key assignments creates next task, using above-mentioned task maximum key assignments as the task minimum key value of above-mentioned next task, and will The above-mentioned next task created is put into above-mentioned task queue；In the above-mentioned scanning from database with value in range [above-mentioned Business minimum key value, above-mentioned task maximum key assignments）The corresponding data-base recording of interior major key, by the above-mentioned Current Scan from database To data-base recording be recorded as currently from database record set；If same major key is right in above-mentioned current primary database record set The data-base recording answered currently concentrates corresponding data-base recording different from above-mentioned from data-base recording, then records the major key For the initial value member of above-mentioned difference list of primary keys；If same major key is in above-mentioned current primary database record set, there are corresponding Data-base recording and it is above-mentioned currently from data-base recording concentrate be not present corresponding data-base recording, then the major key is recorded For the initial value member of above-mentioned difference list of primary keys；If same major key it is above-mentioned currently from data-base recording concentrate there are corresponding Data-base recording and in above-mentioned current primary database record set be not present corresponding data-base recording, then the major key is recorded For the initial value member of above-mentioned difference list of primary keys；The set that the initial value member of above-mentioned difference list of primary keys is constituted is stored in In one list, to constitute the initial value of above-mentioned difference list of primary keys.
Optionally, the above-mentioned multiple tasks of generation are put into task queue, it is above-mentioned more to execute in the following manner A task：Judge to whether there is pending task in above-mentioned task queue；If in the presence of judgement is currently being performed in parallel , the quantity of thread for executing above-mentioned task whether be less than scheduled maximum thread；If above-mentioned currently holding parallel The quantity of capable thread is less than scheduled maximum thread, then above-mentioned pending task is taken out from above-mentioned task queue, and An idle thread is called to execute above-mentioned pending task；If the quantity etc. of the above-mentioned current thread executed parallel In or more than above-mentioned scheduled maximum thread, then above-mentioned pending task is remained in above-mentioned task queue.
Optionally, configuration file configuration N and the adjacent interval time M executed twice between above-mentioned steps are first passed through in advance.
Optionally, an above-mentioned steps were executed every M seconds, wherein the value of M is the fixed value or random in [2,10] The value range of value, N is [1,10].
According to the another aspect of the application, a kind of dynamic corrector of master-slave database consistency is provided, including：More New unit, for repeating following steps n times, wherein N is preset natural number：It is retrieved from primary database and poor The corresponding data-base recording of major key in different list of primary keys, to obtain primary database record set；From from database retrieval with it is upper The corresponding data-base recording of major key in difference list of primary keys is stated, to obtain from database record set；Obtain above-mentioned primary database Record set and the major key corresponding to the above-mentioned data-base recording concentrated and had differences from data-base recording, by above-mentioned difference primary key column Major key in table is updated to the major key corresponding to the above-mentioned data-base recording having differences；Judging unit, for having executed n times After stating step, if still having major key in above-mentioned difference list of primary keys, judge above-mentioned primary database with it is above-mentioned from database not Unanimously, otherwise, judge above-mentioned primary database with it is above-mentioned consistent from database.
Optionally, the dynamic corrector of above-mentioned master-slave database consistency further includes：Unit is repaired, for judging Above-mentioned primary database and above-mentioned after database is inconsistent, execution data reparation operation, wherein above-mentioned data reparation operation packet It includes：It is searched in the above-mentioned difference list of primary keys obtained after having executed n times above-mentioned steps simultaneously by above-mentioned primary database and upper The major key for including from the data-base recording in database is stated, the major key of the first major key type is denoted as, using in above-mentioned primary database With type be the corresponding data-base recording covering of the above-mentioned major key of first major key type it is above-mentioned from database with above-mentioned first master The corresponding data-base recording of key type；Quilt is searched in the above-mentioned difference list of primary keys obtained after having executed n times above-mentioned steps Master that is that data-base recording in above-mentioned primary database includes but not including by the above-mentioned data-base recording from database Key is denoted as the major key of the second major key type, by data corresponding with the above-mentioned major key of second major key type in above-mentioned primary database Library record copies to above-mentioned from database；It is looked into the above-mentioned difference list of primary keys obtained after having executed n times above-mentioned steps It looks for including by the above-mentioned data-base recording from database but does not include by the data-base recording in above-mentioned primary database Major key is denoted as third major key type, and deletes number corresponding with the above-mentioned major key of third major key type from database from above-mentioned It is recorded according to library.
Optionally, the dynamic corrector of above-mentioned master-slave database consistency further includes：Initialization unit, for repeating Before executing above-mentioned steps n times, the initial value of above-mentioned difference list of primary keys is generated, wherein the difference master of above-mentioned generation initialization Key list includes：The multiple tasks in task queue are performed in parallel to scan above-mentioned primary database and above-mentioned from database Data-base recording, to obtain the initial value of above-mentioned difference list of primary keys, wherein each above-mentioned task includes following operation, wherein In the corresponding major key of all data-base recordings of the initial value of task minimum key value for detected data table in above-mentioned primary database Minimum value：Pressed since data-base recording corresponding with above-mentioned task minimum key value in above-mentioned primary database major key from it is small to The data-base recording of big sequential scan predetermined number of lines, by Current Scan in above-mentioned primary database to data-base recording be recorded as Current primary database record set, obtains in the major key corresponding to the data-base recording in above-mentioned current primary database record set most Big value is used as task maximum key assignments, creates next task, using above-mentioned task maximum key assignments appointing as above-mentioned next task Business minimum key value, and the above-mentioned next task of establishment is put into above-mentioned task queue；It is above-mentioned from database scanning with Value is in range [above-mentioned task minimum key value, above-mentioned task maximum key assignments）The corresponding data-base recording of interior major key, by it is above-mentioned from In database Current Scan to data-base recording be recorded as currently from database record set；If same major key is in above-mentioned current master Data-base recording concentrate corresponding data-base recording with it is above-mentioned currently from the corresponding data-base recording of data-base recording concentration not Together, then the major key is recorded as to the initial value member of above-mentioned difference list of primary keys；If same major key is in above-mentioned current primary database Corresponding database note is not present in above-mentioned currently concentrated from data-base recording there are corresponding data-base recording in record set The major key, then is recorded as the initial value member of above-mentioned difference list of primary keys by record；If same major key is above-mentioned currently from database There are corresponding data-base recording in record set, there is no corresponding databases to remember in above-mentioned current primary database record set The major key, then is recorded as the initial value member of above-mentioned difference list of primary keys by record；By the initial value of above-mentioned difference list of primary keys at The set that member is constituted stores in a list, to constitute the initial value of above-mentioned difference list of primary keys.
In this application, using difference list of primary keys as an intermediate quantity and tracking object, master and slave data are repeatedly compared Data in library, will be having but current restore consistent major key record in master and slave database gradually in difference list of primary keys It screens out, to efficiently solve in the related technology because of the technology of the result inaccuracy of consistency desired result caused by synchronization delay Problem has reached the technique effect for improving master-slave database verification accuracy.
In addition, in this application, by the division to the data record in database, with generate parallel multiple tasks into Row scanning, and the step of carrying out subsequent consistency desired result realize to reach the technique effect for shortening checking time Processing capacity is efficiently used, library is not locked in consistency desired result to meet, does not lock table, wanting of not stopping externally servicing It asks, realizes the effect for carrying out dynamic detection when database and tables of data are in normal read-write state.
Further, by the processing of the accurate result to master-slave database consistency desired result, technical scheme is also The effective reparation that can realize the data to being had differences in master and slave database, it is consistent to reach master-slave database dynamic Effect.
Description of the drawings
Attached drawing described herein is used for providing further understanding of the present application, constitutes part of this application, this Shen Illustrative embodiments and their description please do not constitute the improper restriction to the application for explaining the application.In the accompanying drawings：
Fig. 1 is a kind of preferred structure block diagram according to the check system of the master-slave database consistency of the embodiment of the present application；
Fig. 2 is a kind of preferred structure frame according to the dynamic corrector of the master-slave database consistency of the embodiment of the present application Figure；
Fig. 3 is another preferred structure according to the dynamic corrector of the master-slave database consistency of the embodiment of the present application Block diagram；
Fig. 4 is another preferred structure according to the dynamic corrector of the master-slave database consistency of the embodiment of the present application Block diagram；
Fig. 5 is a kind of preferred flow according to the dynamic check method of the master-slave database consistency of the embodiment of the present application Figure；
Fig. 6 is another preferred flow according to the dynamic check method of the master-slave database consistency of the embodiment of the present application Figure；
Fig. 7 is another preferred flow according to the dynamic check method of the master-slave database consistency of the embodiment of the present application Figure；
Fig. 8 is another preferred flow according to the dynamic check method of the master-slave database consistency of the embodiment of the present application Figure；
Fig. 9 is another preferred flow according to the dynamic check method of the master-slave database consistency of the embodiment of the present application Figure；
Figure 10 is another preferred stream according to the dynamic check method of the master-slave database consistency of the embodiment of the present application Cheng Tu.
Specific implementation mode
The application is described in detail below with reference to attached drawing and in conjunction with the embodiments.It should be noted that not conflicting In the case of, the features in the embodiments and the embodiments of the present application can be combined with each other.
Before the further details of description present embodiments, it will be described with reference to Figure 1 and can be used for realizing this Shen The suitable counting system structure of one of principle please.In the following description, unless otherwise specified, will refer to by one or The symbolic indication of action and operation that multiple computers execute describes present embodiments.As such, it will be understood that sometimes Be referred to as computer execution this kind of action and operation include computer processing unit to indicating data with structured form The manipulation of electric signal.It is safeguarded on this manipulation transforms data or position in the storage system of computer, this is with this The mode that the technical staff in field understands reconfigures or changes the operation of computer.Safeguard that the data structure of data is that have The physical location of the memory of particular community defined in the format of data.However, although this Shen described in above-mentioned context Please, but it does not mean restrictive, as understood by those skilled in the art, action described below and operation Various aspects can also be realized with hardware.
Attached drawing is turned to, wherein identical reference label refers to identical element, the principle of the application is illustrated as closing at one It is realized in suitable computing environment.It is described below based on the embodiments herein, and is not considered as about herein not The alternative embodiment that is expressly recited and limit the application.
Fig. 1 shows the schematic diagram for the example computer architecture that can be used for these equipment.For the mesh of description , the architecture painted is only an example of proper environment, is not proposed to the use scope of the application or function any Limitation.The computing system should not be also construed to have any dependence or demand to any component shown in FIG. 1 or combinations thereof.
The principle of the application can be operated using other general or specialized calculating or communication environment or configuration.Suitable for this Well-known computing system, environment and the example of configuration of application include but not limited to personal computer, server, many places Manage device system, the system based on microprocessor, minicomputer, mainframe computer and the distribution including any above system or equipment Formula computing environment.
The check system 100 of one of configuration most basic as its, the master-slave database consistency in Fig. 1 can be wrapped at least It includes：One server 102 and the one or more distributed processing terminals 104 being connect with the server 102.Wherein, it takes Business device 102 can include but is not limited to the processing unit of Micro-processor MCV or programmable logic device FPGA etc., for storing number According to storage device and the transmitting device that is communicated with processing terminal；Processing terminal 104 may include：Micro-processor MCV and clothes The transmitting device of business device communication, the display device interacted with user.In the present specification and claims, " system " can also It is defined as being able to carry out software, firmware or microcode to realize any hardware component of function or combination principal and subordinate's number of hardware component Can even is that according to the check system 100 of library consistency it is distributed, to realize distributed function.
Optionally, 104 running example of each processing terminal, only responsible dynamic chek a pair of database table, i.e. a main number According to the tables of data in library and a corresponding tables of data from database；According to configuration while several processing terminals can be started 104 carry out running example, both can be performed simultaneously consistency dynamic detection to multiple tables of data parallel, each processing terminal 104 is transported Row example is responsible for the dynamic conformance inspection of a pair of of database table.
As used in this application, term " module ", " component " or " unit " can refer in master-slave database consistency The software object or routine executed on check system 100.Difference component, module, unit, engine and service described herein can It is implemented as executing on the check system 100 of master-slave database consistency（For example, as individual thread）Object or into Journey.Although system and method described herein are preferably realized with software, the combination of hardware or software and hardware Realize to be also that may and be contemplated.
Embodiment 1
Fig. 2 is shown according to a kind of preferred structure of the dynamic corrector of the embodiment of the present application master-slave database consistency It is intended to, it is preferred that the system in the present embodiment can be located on server 102.
The inventive concept of this preferred embodiment may include mainly：Initialize difference list of primary keys；To difference list of primary keys Carry out the update operation of pre-determined number；After completing the update operation of above-mentioned pre-determined number, if difference list of primary keys is not still Sky then shows that the database of the corresponding master-slave database of difference list of primary keys is inconsistent.Wherein, it is protected in the difference list of primary keys What is stayed is the major key worked as corresponding to time primary database relatively determined after the completion and the data being had differences from database.
Preferably, in order to achieve the above object, the dynamic corrector of master-slave database consistency as shown in Figure 2 can be with Including：Updating unit 202 and judging unit 204, are below specifically described the structure.
1）Updating unit 202, for repeating following steps n times, wherein N is preset natural number：From main number According to data-base recording corresponding with the major key in difference list of primary keys is retrieved in library, to obtain primary database record set；From from number According to data-base recording corresponding with the major key in the difference list of primary keys is retrieved in library, to obtain from database record set；It obtains It takes primary database record set and concentrates the major key corresponding to the data-base recording having differences from data-base recording, by difference major key Major key in list is updated to the major key corresponding to the data-base recording having differences；
2）Judging unit 204 is coupled with updating unit 202, is used for after having executed n times above-mentioned steps, if difference master Still have major key in key list, then judge primary database with it is inconsistent from database, otherwise, then judge primary database with from number It is consistent according to library.
Difference list of primary keys, as one of the concept proposed in technical scheme, for indicating a kind of major key Set.Optionally, in the embodiment of the present application, the specific implementation as this major key set, form can be major key institute only The set of one corresponding key assignments is stored in a list, to execute further scanning, retrieval etc. in server 102 Read-write operation and sequence, than reciprocity arithmetic operation.
If its title is revealed, it is right that difference list of primary keys is mainly used for the master-slave database having differences record institute The major key answered.However in the embodiment of the present application, as between the multiple above-mentioned update operation performed by updating unit 202 The area of a room, difference list of primary keys is simultaneously non-static, be stayed wherein the major key stored can be renewed in primary update operation relaying, can also be by It removes, and updated difference list of primary keys can be sent into new primary update operation, can also be exported to judging unit 204 In to obtain the check results of master-slave database consistency, so as to by initial difference list of primary keys repeatedly more Newly, the erroneous judgement factor that consistency judges caused by master-slave database synchronization delay is gradually excluded, the model of judgement is then reduced It encloses, the final position for realizing the accurate data-base recording for determining and being had differences in master-slave database in primary complete check-verifying period It sets, and these positions can be exported and be reported for verification, to check or further to handle.
Optionally, in the once-through operation that updating unit 202 executes, the difference master of incipient stage can be operated according to this time First group of major key recorded in key list retrieves opposite with first group of major key respectively in primary database and from database The data-base recording answered, and indicated respectively with primary database record set and from database record set, it then can be in main number It is compared according to library record set and between database record set, finds out second group corresponding to the part wherein having differences Major key, and first group of major key is replaced with second group of major key in difference list of primary keys, then terminate this operation, to complete one The secondary update to difference list of primary keys.
It is worth noting that, the quantity of major key can be one or more in first group of major key, or zero, second group Major key may include one or more of first group of major key, can not also include it is therein any one；Second group of major key be The a subset of first group of major key.On the other hand, the determination of second group of major key can carry out in above-mentioned comparison process, can also It is carried out after above-mentioned comparison process.In addition, the above-mentioned description for replacing first group of major key with second group of major key should be understood as A kind of description for the embodiment of the present application can't constitute any restriction to the application, for example, updating unit 202 can be with For performing the following operations to realize said effect：1）Third major key group is obtained in the first major key group, wherein third major key group It is consistent with data-base recording content corresponding from database in primary database；2）By third major key group from difference list of primary keys Middle deletion.
Under above-mentioned scene, by a limited number of major keys in difference list of primary keys and its corresponding database note Record rather than the processing of entire master and slave database, it is possible to reduce the single detection cycle of consistency desired result is carried out to master-slave database Take, to which the consistency of master-slave database is carried out in the case where not stopping externally servicing for database and its place system Dynamic check provide technical foundation.On the other hand, due to the time-consuming shortening of single detection, it is consistent to improve master-slave database Property verification timeliness, the error rate that caused by time delay consistency judges is reduced, to improve check results Accuracy.
Further, the once-through operation that above-mentioned updating unit 202 executes can execute again：In new once-through operation, Major key recorded in the difference list of primary keys of incipient stage is second group of major key, thereafter, respectively in primary database and from data Data-base recording corresponding with second group of major key is retrieved in library, finds out the 4th group corresponding to the part wherein having differences Major key, and second group of major key is replaced with the 4th group of major key in difference list of primary keys, then terminate this operation, to complete again Once to the update of difference list of primary keys.
Under above-mentioned scene, i.e. based on the difference major key recorded in obtained difference list of primary keys after above-mentioned operation twice 4th major key group operates the result that obtained master-slave database consistency judges by the judgement that judging unit 204 executes and has There are higher confidence level and lower False Rate, details are as follows for cardinal principle：
Imagination server, which is made that the data-base recording corresponding to a certain major key in primary database at the first moment, to be repaiied Change, and as the synchronous repair action with the modification, server is at the second moment to corresponding to the major key from database Data-base recording is made that corresponding modification, and in the once-through operation that above-mentioned updating unit 202 executes, to difference major key i.e. the The acquisition of two groups of major keys is completed between the first moment and the second moment, then the major key corresponding with the data-base recording of change To be credited in second group of major key, even if the master-slave database record then corresponding to the second moment major key realize it is same Step, to the false judgment of consistency caused by being delayed occur.However this mistake can be by carrying out after the preset lime Above-mentioned second operation eliminate, for example, the acquisition setting of difference major key i.e. the 4th group major key is the in operating for second It is completed after two moment, then above-mentioned major key corresponding with the data-base recording of change will not be logged in the 4th group of major key, The result that master-slave database consistency to be obtained by judging unit 204 judges has higher confidence level and lower False Rate.
Further, the once-through operation that above-mentioned updating unit 202 executes can also repeat n times, to obtain higher Confidence level and lower False Rate.For example, for data delay on line time 1 second or so the case where, if acquiescence It is configured to, acquiescence progress 8 wheel verification primary to variance data duplication check every 5 seconds.Then there was only very extreme situation, just meeting The verification every time data occurred in this 8 times verifications all changed, and just master-slave synchronisation is not completed also, then these data It will be mistaken for inconsistent data.However, such case is minimum in the probability that production environment occurs, and user can basis The variance data of output reports, again anti-data looked into master-slave database by hand.Therefore, pass through the above-mentioned invention structure of the application Think in actual production practice process, check results have very high confidence level and very low False Rate.
Optionally, judging unit 204 judge it is inconsistent after also determined that it is inconsistent between master-slave database, in this way Can use certain mode to the data of master-slave database be updated with realize two databases in data synchronization, In one preferred embodiment, as shown in figure 3, the dynamic corrector of above-mentioned master-slave database consistency can also include：
Repair unit 302, couple with judging unit 204, for judge primary database and from database it is inconsistent it Afterwards, data reparation operation is executed, wherein above-mentioned data reparation, which operates, includes：From having executed the difference obtained after n times above-mentioned steps It is searched simultaneously by primary database and the major key for including from the data-base recording in database in different list of primary keys, is denoted as the first major key The major key of type, using being the covering of the major key of the first major key type corresponding data-base recording with type in primary database from data Data-base recording corresponding with the first major key type in library；From having executed the difference list of primary keys obtained after n times above-mentioned steps It is middle to search master that is including by the data-base recording in primary database but not included from the data-base recording in database Key is denoted as the major key of the second major key type, and data-base recording corresponding with the major key of the second major key type in primary database is answered It makes from database；It is searched from having executed in the difference list of primary keys that is obtained after n times above-mentioned steps by from database Major key that is that data-base recording includes but not including by the data-base recording in primary database is denoted as third major key type, and Data-base recording corresponding with the major key of third major key type is deleted from from database.
In above-mentioned preferred embodiment, using difference list of primary keys as an intermediate quantity and tracking object, repeatedly compare It, will be having in difference list of primary keys but current restore consistent master in master and slave database to the data in master and slave database Key record gradually screens out, to efficiently solve in the related technology because the result of consistency desired result is not caused by synchronization delay Accurate technical problem has reached the technique effect for improving master-slave database verification accuracy.
In a preferred embodiment, as shown in figure 4, the dynamic corrector of above-mentioned master-slave database consistency may be used also To include：
Initialization unit 402 is coupled with updating unit 202, for before repeating above-mentioned steps n times, it to be poor to generate The initial value of different list of primary keys, wherein the difference list of primary keys for generating initialization includes：It is performed in parallel more in task queue A task scans primary database and from the data-base recording in database, to obtain the initial value of difference list of primary keys, wherein Each task includes following operation, wherein the initial value of task minimum key value is all of detected data table in primary database Minimum value in the corresponding major key of data-base recording：It is opened from data-base recording corresponding with task minimum key value in primary database Beginning presses the data-base recording of the sequential scan predetermined number of lines of major key from small to large, the database that Current Scan in primary database is arrived It is recorded as current primary database record set, obtains the major key corresponding to the data-base recording in current primary database record set In maximum value as task maximum key assignments；Next task is created, using task maximum key assignments as the task of next task Minimum key value, and the next task of establishment is put into task queue；In the scanning from database and value, in range, [task is most Small key assignments, task maximum key assignments）The corresponding data-base recording of interior major key, by from database Current Scan to database remember Record is recorded as currently from database record set；If same major key in current primary database record set corresponding data-base recording with It concentrates corresponding data-base recording different from data-base recording currently, then the major key is recorded as the initial of difference list of primary keys It is worth member；If same major key is in current primary database record set there are corresponding data-base recording and currently from database Corresponding data-base recording is not present in record set, then the major key is recorded as to the initial value member of difference list of primary keys；If same One major key is not deposited from data-base recording concentration there are corresponding data-base recording in current primary database record set currently In corresponding data-base recording, then the major key is recorded as to the initial value member of difference list of primary keys；By difference list of primary keys The set that initial value member is constituted stores in a list, to constitute the initial value of difference list of primary keys.
In operation performed by above-mentioned initialization unit 402, by generation to computer task, distributes and hold Row, realizes the initialization to difference list of primary keys.It is alternatively possible to primary database is ranked up before executing the task, For example, being arranged in order by the key assignments size of corresponding major key, in turn, a major key of key assignments minimum can be therefrom obtained, It is denoted as minimum key value and minimum major key.The minimum key value can be used as and execute parameter, be inputted together with the scheduled number of scanning lines Into task, so as to be starting with minimum major key, the scanning to the primary database record in the predetermined number of lines is executed, it will be with Corresponding major key in maximum one of key assignments be denoted as maximum major key, and it is complete between minimum major key and largest key value to key assignments The corresponding master and slave data-base recording of major key institute of portion is compared, and the major key corresponding to the part wherein having differences is remembered For the initial value of difference list of primary keys.Further, the largest key value that a upper task obtains can also be sent to next As new minimum key value in task, to realize the segmentation parallel initialization for full database.
The above-mentioned realization method initialized to difference list of primary keys is described in detail below in conjunction with Tables 1 and 2, In, Tables 1 and 2 is used to indicate to be swept for the executing the initialization of difference list of primary keys before and after same master-slave database twice of the task The data-base recording retouched, and the data-base recording is ranked up by Major key from small to large.It is readily appreciated that, the data-base recording Corresponding Major key might not be numerically continuous, for example, spare key assignments can be reserved between Major key, in case The use etc. of newly-increased data-base recording, to which in the database, the Major key of data-base recording and its corresponding major key can be such as table 1 Shown in table 2.
Table 1
For the task of the acquisition difference list of primary keys initial value generated for the first time, minimum key value can be using value as master data Minimum key value in library, for example, 001.It is assumed that the corresponding primary database of major key that key assignments is 001 is recorded as the 1st row, then for this Task, by taking the number of scanning lines=5 as an example, in the execution of task, the primary database corresponding to major key by scanning with key assignments for 001 It is recorded as the 5 rows record of starting, i.e., the 1st row of primary database record is to the 5th row, and is denoted as primary database record set, such as 1 institute of table Show, the key assignments for corresponding to major key is followed successively by：001,003,005,007 and 009, wherein minimum key value 001, largest key value are 009.To, according to the embodiment of the present application, in the execution of task, will further from database retrieve key assignments [001, 009) major key in this section, i.e., 001,003,005,007, it is corresponding from data-base recording, and be denoted as from database and remember Record collection.It is possible to further the primary database record set of acquisition and from carried out between database record set line by line, word for word section Comparison, to find out the data having differences and its corresponding major key, such as the 2nd row being had differences shown in table 1 and the 4th row The major key that key assignments corresponding to data is 003 and 007, is recorded as the initial value of difference list of primary keys.
Table 2
For generating the task of next acquisition difference list of primary keys initial value again, minimum key value can be with value The largest key value that last task obtains, for example, largest key value acquired in above-mentioned generating for the first time for task, 009.Then for The task, equally by taking the number of scanning lines=5 as an example, in the execution of task, the master corresponding to major key by scanning with key assignments for 009 Data-base recording is the 5 rows record of starting, i.e., the 5th row of primary database record is to the 9th row, and is denoted as primary database record set, As shown in table 2, the key assignments for corresponding to major key is followed successively by：009,011,013,015 and 019, wherein minimum key value 009, it is maximum Key assignments is 019.To, according to the embodiment of the present application, in the execution of task, will further from database retrieve key assignments exist [009,019) major key in this section, i.e., 009,013,015,017, it is corresponding from data-base recording, and be denoted as from data Library record set.It is possible to further the primary database record set of acquisition and from carried out between database record set line by line, by The comparison of field is recorded as the initial value of difference list of primary keys to find out the data having differences and its corresponding major key. In the present embodiment, since there are the corresponding data-base recordings of major key that key assignments is 011 in primary database record set, and from database Corresponding data-base recording is not present in record set, so as to which key assignments is recorded as difference list of primary keys for 011 major key Initial value member；Further, since being concentrated from data-base recording, there are the corresponding data-base recordings of major key that key assignments is 017, and lead Data-base recording, which is concentrated, is not present corresponding data-base recording, so as to which key assignments is recorded as difference major key for 017 major key The initial value member of list；And then the set that the initial value member of difference list of primary keys is constituted can be stored in a list In, to constitute the initial value of difference list of primary keys.
Optionally, above-mentioned task can also repeatedly generate and execute repeatedly, until terminating to the entire scan of primary database. It optionally, can be after obtaining largest key value, by this in the execution of the task of above-mentioned acquisition difference list of primary keys initial value Largest key value generates next task as new minimum key value, the step for can be in the content to master and slave data-base recording It carries out before being compared, is consumed so as to shorten a series of initialization of the difference list of primary keys carried out for full database Overall time.
As an optional implementation manner, sentencing for the scan task of the end stage of the list of primary keys of primary database It is disconnected, following operation can be executed by initialization unit 402 to realize：If carrying out the scanning institute of predetermined number of lines to primary database Data volume in the data volume or primary database record set that retrieve is less than the scheduled number of scanning lines, then it is assumed that scanned The end of the list of primary keys of primary database has been arrived, has not regenerated the task of new acquisition difference list of primary keys initial value.It is more optional Ground, initialization unit 402 can be also used for executing following operation：1）It is more than master library largest key value scanning key assignments from database The corresponding data-base recording of major key；2）If there are the corresponding data-base recording of major key that key assignments is more than master library largest key value, The major key is recorded as to the initial value member of difference list of primary keys.Similarly, initialization unit 402 can be also used for executing following Operation：1）Major key corresponding data-base recording of the key assignments less than master library minimum key value is being scanned from database；2）If there are keys The major key is then recorded as the initial value of difference list of primary keys by value less than the corresponding data-base recording of major key of master library minimum key value Member.
By aforesaid operations, can will from database it is that may be present with from the corresponding key assignments of library data record [main Library minimum key value, master library largest key value] other than major key be recorded in the initial value of difference list of primary keys, with realize to master and slave The complete scan of database, to make the record of the difference list of primary keys of acquisition not omit.
In order to realize the parallel processing to data, to improve the efficiency of data processing, multi-threaded parallel execution may be used Mode data are handled.In a preferred embodiment, with calling multiple thread parallels to the master and slave database In record be scanned, to obtain the initial value of difference list of primary keys, wherein each in the multiple thread is responsible for holding One task of row, i.e., in the primary database, since task minimum major key, the data of predetermined number of lines are scanned, In the implementation procedure of the task of each first scan type, next first scan task can be generated as the case may be and is put into To etc. pending task queue.
Several task types in task are explained below：
Etc. pending task queue（Prepared Task List）：Task to be performed, the queue such as store In task be to be executed according to the sequence of first in first out；
Active task queue（Active Task List）：Being carrying out for task is stored, times in the queue Business is to have executed just removing for task.
Scheduling thread is needed to execute these tasks, if task can be realized by dispatching multiple threads simultaneously It is parallel to execute.If waiting has task in pending task list, and available free task execution thread, then just being held from waiting A task is taken out in capable task queue, is put into active task queue, is then distributed to a task execution thread；
Being handled in a manner of obtaining difference list of primary keys above by thread parallel to be：If waiting pending task There is no task in list, there is no task in active task list（I.e. one wheel, which checks, to be over）, and do not checked also Specified wheel number, and difference list of primary keys is not sky, then shows that there are discrepant major keys in two databases, thus Using current difference list of primary keys as the difference list of primary keys of initialization.
Then several new tasks are created again, and task type is to scan again, by the major key in all differences list of primary keys It is assigned in new task, newly-built task is all added in task list, carry out the corresponding principal and subordinate's data of difference list of primary keys The comparison of library data record.This batch is being established again after scan task, scheduling thread suspend mode M seconds then proceedes to scheduler task and holds Row.Optionally, dormancy time of the scheduling thread between two-wheeled task M seconds is a Configuration Values, is defaulted as 5 seconds.It is again in type The task of secondary scanning is all performed finish after, if reaching specified execution wheel number not yet, and difference list of primary keys is not Sky creates several new tasks again then such as above-mentioned step 1 sample, is put into task waiting list, scheduling thread suspend mode S again Second, a new wheel scan starts again at.
If waiting does not have task in pending task list, also without task in active task list, and Completion has checked specified wheel number（Finally it is determined with variance data）Or difference list of primary keys is sky（Final judgement indifference Data）, then simultaneously output verification summary report is generated.
In a preferred embodiment, input configuration is first passed through in advance to specify maximum scan wheel number N and adjacent execute twice Interval time M between above-mentioned steps.Optionally, the value range of M is [2,10], and the value range of N is [1,10].It is optional Ground, the dormancy time M among every two wheel scan, can be a dynamic random value, such as a random number between [2,10].
Embodiment 2
On the basis of Fig. 2-systems shown in Fig. 4, this application provides a kind of preferred master-slave database consistency Dynamic check method.As shown in figure 5, this method may include steps of：
Step S502：Repeat following steps n times, wherein N is preset natural number：It is examined from primary database Rope data-base recording corresponding with the major key in difference list of primary keys, to obtain primary database record set；It is examined from from database Rope data-base recording corresponding with the major key in the difference list of primary keys, to obtain from database record set；Obtain the master Database record set and the major key corresponding to the data-base recording concentrated and had differences from data-base recording, by the difference Major key in list of primary keys is updated to the major key corresponding to the data-base recording having differences；
Step S504：After having executed step described in n times, if recording variant major key in the difference list of primary keys, Judge the primary database with it is described inconsistent from database, otherwise, then judge the primary database with it is described from data Library is consistent.
Optionally, in the once-through operation in step S502, the difference primary key column of incipient stage can be operated according to this time First group of major key recorded in table retrieves corresponding with first group of major key respectively in primary database and from database Data-base recording, and indicate respectively with primary database record set and from database record set, it then can be in primary database It record set and is compared between database record set, finds out second group of master corresponding to the part wherein having differences Key, and first group of major key is replaced with second group of major key in difference list of primary keys, then terminate this operation, it is primary to complete Update to difference list of primary keys.
It is worth noting that, the quantity of major key can be one or more in first group of major key, or zero, second group Major key may include one or more of first group of major key, can not also include it is therein any one；Second group of major key be The subset of first group of major key.On the other hand, the determination of second group of major key can carry out in above-mentioned comparison process, can also be upper It is carried out after stating comparison process.In addition, the above-mentioned description that first group of major key is replaced with second group of major key should be understood as A kind of description of the embodiment of the present application can't constitute any restriction to the application, such as shown in Fig. 6, this result can be with It is realized via following step：
S602：Third major key group is obtained in the first major key group, wherein third major key group primary database with from database In corresponding data-base recording content it is consistent；
S604：Third major key group is deleted from difference list of primary keys.
And then in step S502, obtains primary database record set and concentrate the database having differences from data-base recording Major key in difference list of primary keys is updated to the major key corresponding to the data-base recording having differences by the corresponding major key of record The step for, it can also be not limited solely to be previously mentioned in the embodiment of the present application, for example, it is also possible to needle there are many realization method Screening is carried out one by one to the major key in diff list, to realize the update to the major key in difference list of primary keys.
Under above-mentioned scene, by a limited number of major keys in difference list of primary keys and its corresponding database note Record rather than the processing of entire master and slave database, it is possible to reduce the single detection cycle of consistency desired result is carried out to master-slave database Take, to which the consistency of master-slave database is carried out in the case where not stopping externally servicing for database and its place system Dynamic check provide technical foundation.On the other hand, due to the time-consuming shortening of single detection, it is consistent to improve master-slave database Property verification timeliness, the error rate that caused by time delay consistency judges is reduced, to improve check results Accuracy.
Further, the once-through operation in above-mentioned step S502 can execute again：In new once-through operation, start Major key recorded in the difference list of primary keys in stage is second group of major key, thereafter, respectively in primary database and from database Data-base recording corresponding with second group of major key is retrieved, the 4th group of master corresponding to the part wherein having differences is found out Key, and second group of major key is replaced with the 4th group of major key in difference list of primary keys, then terminate this operation, it is another to complete The secondary update to difference list of primary keys.
Under above-mentioned scene, according to step S504 to it is above-mentioned operate twice after record in obtained difference list of primary keys The master-slave database consistency that i.e. the 4th major key group of difference major key carries out judges there is higher confidence level and lower erroneous judgement Rate, details are as follows for cardinal principle：
Imagination server, which is made that the data-base recording corresponding to a certain major key in primary database at the first moment, to be repaiied Change, and as the synchronous repair action with the modification, server is at the second moment to corresponding to the major key from database Data-base recording is made that corresponding modification, and in the once-through operation in above-mentioned steps S502, to difference major key i.e. second group The acquisition of major key is completed between the first moment and the second moment, then the major key corresponding with the data-base recording of change will be by It charges in second group of major key, even if the master-slave database record then corresponding to the second moment major key realizes synchronization, from And there is the false judgment of consistency caused by delay.However this mistake can be above-mentioned by what is carried out after the preset lime It is operated for the second time to eliminate, for example, the acquisition of difference major key i.e. the 4th group major key was arranged at the second moment in operating for second After complete, then it is above-mentioned to be logged in the 4th group of major key with the corresponding major key of data-base recording of change, to root Have according to the step S504 results judged the consistency that difference major key i.e. the 4th group major key recorded in difference list of primary keys is made There are higher confidence level and lower False Rate.
Further, above-mentioned once-through operation can also repeat n times, to obtain higher confidence level and lower False Rate.For example, for data delay on line time 1 second or so the case where, if default configuration is every 5 seconds to difference Heteromerism is primary according to duplication check, and acquiescence carries out 8 wheel verifications.Then there was only very extreme situation, just will appear in this 8 times verifications It verifies the data every time all to change, and just master-slave synchronisation is not completed also, then these data will be mistaken for differing Cause data.However, such case is minimum in the probability that production environment occurs, and user can be according to the variance data report of output It accuses, again the manual anti-data looked into master-slave database.Therefore, it is put into practice in actual production by the above-mentioned inventive concept of the application In the process, check results have very high confidence level and very low False Rate.
On the other hand, it will be understood by those of skill in the art that in the embodiment of the present application, being opened for above-mentioned single operation The composition of the difference list of primary keys in stage beginning is defined not necessarily, for example, its composition can be last same operation Output as a result, can also be master-slave database global major key in any one section, optionally, as shown in fig. 7, can also be through By the initial value of the difference list of primary keys acquired in following steps：
S702：The multiple tasks in task queue are performed in parallel to scan primary database and from the database in database Record, to obtain the initial value of difference list of primary keys, wherein each task includes following operation, wherein task minimum key value Initial value is the minimum value in the corresponding major key of all data-base recordings of detected data table in primary database：
S704：Press since data-base recording corresponding with task minimum key value in primary database major key from small to large The data-base recording of sequential scan predetermined number of lines, by Current Scan in primary database to data-base recording be recorded as current main number According to library record set, the maximum value in the major key corresponding to the data-base recording in current primary database record set is obtained as task Largest key value creates next task, using task maximum key assignments as the task minimum key value of next task, and by establishment Next task is put into task queue；
S706：In the scanning from database and value in range [task minimum key value, task maximum key assignments）Interior major key pair The data-base recording answered, by from database Current Scan to data-base recording be recorded as currently from database record set；
S708：It is currently compared in current primary database record set and between database record set, and according to comparing Result execute following operation：
S710：If same major key in current primary database record set corresponding data-base recording with currently from database Corresponding data-base recording is different in record set, then the major key is recorded as to the initial value member of difference list of primary keys；
S712：If same major key in current primary database record set there are corresponding data-base recording and currently from Data-base recording, which is concentrated, is not present corresponding data-base recording, then the major key is recorded as the initial value of difference list of primary keys at Member；
S714：If same major key is led from data-base recording concentration there are corresponding data-base recording currently currently Data-base recording, which is concentrated, is not present corresponding data-base recording, then the major key is recorded as the initial value of difference list of primary keys at Member；
S716：The set that the initial value member of difference list of primary keys is constituted stores in a list, to constitute difference The initial value of list of primary keys.
In above-mentioned steps, by generation to computer task, distributes and execute, realize to difference list of primary keys Initialization.It is alternatively possible to the major key of primary database detected data table is ranked up before executing the task, for example, It is arranged in order by the key assignments size of corresponding major key, in turn, can therefrom obtain a major key of key assignments minimum, be denoted as most Small key assignments and minimum major key.The minimum key value can be used as and execute parameter, and task is input to together with the scheduled number of scanning lines In, so as to be starting with minimum major key, by the sequence of major key from small to large, execute to the primary database in the predetermined number of lines The scanning of record, by maximum one maximum major key for being denoted as the task of key assignments in corresponding major key, and to key assignments most The corresponding master and slave data-base recording of whole major keys institute between small major key and largest key value is compared, and will wherein be existed Major key corresponding to the part of difference is denoted as the initial value of difference list of primary keys.Further, a upper task can also be obtained To largest key value be sent in next task as new minimum key value, to realize that the segmentation for full database is initial Change.
It is described in detail below in conjunction with the Tables 1 and 2 provided in the embodiment of the present application before above-mentioned to difference major key The realization method that list is initialized, wherein Tables 1 and 2 executes twice for indicating to be directed to before and after same master-slave database The data-base recording that is scanned of task of difference list of primary keys initialization, and the data-base recording pressed Major key from small to large into Row sequence.It is readily appreciated that, the Major key corresponding to the data-base recording might not be numerically continuous, for example, in Major key Between can reserve spare key assignments, in case the use etc. of newly-increased data-base recording, in the database, data-base recording And its Major key of corresponding major key can be as shown in Table 1 and Table 2.
For the task of the acquisition difference list of primary keys initial value generated for the first time, minimum key value can be using value as master data Minimum key value in library, for example, 001.It is assumed that the corresponding primary database of major key that key assignments is 001 is recorded as the 1st row, then for this Task, by taking the number of scanning lines=5 as an example, in the execution of task, the primary database corresponding to major key by scanning with key assignments for 001 It is recorded as the 5 rows record of starting, i.e., the 1st row of primary database record is to the 5th row, and is denoted as primary database record set, such as 1 institute of table Show, the key assignments for corresponding to major key is followed successively by：001,003,005,007 and 009, wherein minimum key value 001, largest key value are 009.To, according to the embodiment of the present application, in the execution of task, will further from database retrieve key assignments [001, 009) major key in this section, i.e., 001,003,005,007, it is corresponding from data-base recording, and be denoted as from database and remember Record collection.It is possible to further the primary database record set of acquisition and from carried out between database record set line by line, word for word section Comparison, to find out the data having differences and its corresponding major key, such as the 2nd row being had differences shown in table 1 and the 4th row The major key that key assignments corresponding to data is 003 and 007, is recorded as the initial value of difference list of primary keys.
For the task of the acquisition difference list of primary keys initial value generated again, minimum key value can be the last time with value The largest key value that task obtains, for example, largest key value acquired in above-mentioned generating for the first time for task, 009.Then for this Business, equally by taking the number of scanning lines=5 as an example, in the execution of task, the master data corresponding to major key by scanning with key assignments for 009 Library is recorded as the 5 rows record of starting, i.e., the 5th row of primary database record is to the 9th row, and is denoted as primary database record set, such as table 2 Shown, the key assignments for corresponding to major key is followed successively by：009,011,013,015 and 019, wherein minimum key value 009, largest key value It is 019.To, according to the embodiment of the present application, in the execution of task, will further from database retrieve key assignments exist [009,019) major key in this section, i.e., 009,013,015,017, it is corresponding from data-base recording, and be denoted as from data Library record set.It is possible to further the primary database record set of acquisition and from carried out between database record set line by line, by The comparison of field is recorded as the initial value of difference list of primary keys to find out the data having differences and its corresponding major key. In the present embodiment, since there are the corresponding data-base recordings of major key that key assignments is 011 in primary database record set, and from database Corresponding data-base recording is not present in record set, so as to which key assignments is recorded as difference list of primary keys for 011 major key Initial value member；Further, since being concentrated from data-base recording, there are the corresponding data-base recordings of major key that key assignments is 017, and lead Data-base recording, which is concentrated, is not present corresponding data-base recording, so as to which key assignments is recorded as difference major key for 017 major key The initial value member of list；And then the set that the initial value member of difference list of primary keys is constituted can be stored in a list In, to constitute the initial value of difference list of primary keys.
Optionally, above-mentioned task can also repeatedly generate and execute repeatedly, until terminating to the entire scan of primary database. Optionally, in the execution of the task of above-mentioned acquisition difference list of primary keys initial value, can obtain the task maximum key assignments it Afterwards, this largest key value is generated into next task as new minimum key value, the step for can be to master and slave data-base recording Content be compared before carry out, so as to shorten a series of initialization institute of the difference list of primary keys carried out for full database The overall time of consumption.
Optionally, as shown in Figure 10, the multiple tasks of generation can be put into task queue, is come in the following manner Execute multiple task：
S1002：Judge to whether there is pending task in task queue；
S1004：If in the presence of whether the quantity of thread that judgement is currently being performed in parallel, for executing task Less than scheduled maximum thread；
S1006：If the quantity of the current thread executed parallel is less than scheduled maximum thread, from task queue It is middle to take out pending task, and an idle thread is called to execute pending task；
S1008：If the quantity of the current thread executed parallel is equal to or more than scheduled maximum thread, will wait for The task of execution remains in task queue.
As an alternative embodiment, the scan task for the end stage of the list of primary keys of primary database is sentenced It is disconnected, it can be realized by following operation：
S802：If data volume or primary database that the scanning for carrying out predetermined number of lines to primary database is retrieved are remembered The data volume that record is concentrated is less than the scheduled number of scanning lines, then it is assumed that the end of the scanned list of primary keys for having arrived primary database, The task of new acquisition difference list of primary keys initial value is not regenerated.
Optionally, as shown in figure 8, after step S802, largest key value, that is, master library obtained in S802 is maximum Following operation can also be performed in key assignments：
S804：Major key corresponding data-base recording of the key assignments more than master library largest key value is being scanned from database；
S806：If there are the corresponding data-base recording of major key that key assignments is more than master library largest key value, which is recorded For the initial value member of difference list of primary keys.
Optionally, as shown in figure 9, after the minimum key value in obtaining primary database, that is, master library minimum key value, it is poor to obtain The step of initial value of different list of primary keys can also include：
S902：Major key corresponding data-base recording of the key assignments less than master library minimum key value is being scanned from database；
S904：If there are the corresponding data-base recording of major key that key assignments is less than master library minimum key value, which is recorded For the initial value member of difference list of primary keys.
By aforesaid operations, can will from database it is that may be present with from the corresponding key assignments of library data record [main Library minimum key value, master library largest key value] other than major key be recorded in the initial value of difference list of primary keys, with realize to master and slave The complete scan of database, to make the record of the difference list of primary keys of acquisition not omit.
Optionally, it is directed to the scanning and search operaqtion of database in the above-described embodiments, can be realized by SQL statement, For example, pressing since data-base recording corresponding with task minimum key value in primary database major key from small to large for above-mentioned The data-base recording of sequential scan predetermined number of lines can be realized by following sentence：
SELECT*FROM TABLE_NAME WHERE PK>=${MIN_PK}ORDER BY PK ASC LIMIT${MAX_ COUNT}；
And for it is above-mentioned retrieved from database key assignments [minimum key value, largest key value) major key in this section, It can be realized by following sentence：
SELECT*FROM TABLE_NAME WHERE PK>=${MIN_PK}AND PK<${MAX_PK}ORDER BY PK ASC；
Wherein, MIN_PK is for indicating minimum key value, and MAX_PK is for indicating largest key value, and MAX_COUNT is for indicating The scheduled number of scanning lines.
Certainly, only example is used as to occur the acquisition modes of difference list of primary keys in above-described embodiment, it can't be to this Application constitutes any restriction, for example, above-mentioned key value size is ranked up not major key and its corresponding data-base recording It is necessary, can also the last figure of key value division is made to list of primary keys and its corresponding data-base recording, etc..
By the operation of the initial value of the acquisition difference list of primary keys shown in above-described embodiment, may be implemented to master and slave number According to data record inconsistent in library completely, rapidly screen, and can divide there will be the major key exhaustive of difference Batch record is in a series of difference list of primary keys, to make each difference list of primary keys scale reduce, is walked convenient for subsequent check Rapid execution, and the verification task carried out for a series of above-mentioned difference list of primary keys can be in computer or processor parallel Ground executes, to substantially shorten the overall time of the consistency desired result for master-slave database.
Optionally, after being judged the consistency of master-slave database by the above embodiment, if there is two The inconsistent situation of person then can further execute it reparation operation, it is alternatively possible to include the following steps：
It is searched simultaneously by the primary database in the difference list of primary keys obtained after having executed step described in n times The major key for including with the data-base recording from database is denoted as the major key of the first major key type, uses the master data With type be in library the major key of the first major key type corresponding data-base recording covering it is described from database with described the The corresponding data-base recording of one major key type；
It is searched by the primary database in the difference list of primary keys obtained after having executed step described in n times Major key that is that data-base recording includes but not including by the data-base recording from database, is denoted as the second major key class The major key of type, by data-base recording corresponding with the major key of the second major key type in the primary database copy to it is described from In database；
It is searched in the difference list of primary keys obtained after having executed step described in n times by described from database Major key that is that data-base recording includes but not including by the data-base recording in the primary database, is denoted as third major key class Type, and data-base recording corresponding with the major key of third major key type is deleted from database from described.
Optionally, it has also determined that inconsistent between master-slave database in above-mentioned steps S504, thus may be used Certain mode is updated to realize the consistency for repairing data in two databases, at one the data of master-slave database In preferred embodiment, the synchronization to data can be realized in the following way：It is selected from the primary database and has executed N Major key existing for primary database in the difference list of primary keys after the secondary step, to obtain the 5th major key group；From it is described from All numbers corresponding with difference major key after step described in n times in the difference list of primary keys has been executed are deleted in database According to, and the major key in the 5th major key group copied to described from database.That is, for comparing primary database from database In the data that have more carry out delete operation, the data being not present from database for having in primary database, then from main number According to replicating these data in library to from database, to realize the synchronization of data in master-slave database.
In the dynamic check method according to the consistency of the master-slave database of the embodiment of the present application, in order to realize to data Parallel processing, to improve the efficiency of data processing, may be used multi-threaded parallel execution mode data are handled. In one preferred embodiment, the major key that the minimum major key more than or equal to task is obtained from primary database includes：It calls multiple Thread parallel the major key in the primary database is scanned, to obtain the data more than or equal to the task minimum major key Record, wherein each in the multiple thread is to predetermined since the minimum major key of the task in the primary database The data record of line number is scanned.
Several task types in task are explained below：
Etc. pending task queue（Prepared Task List）：Task to be performed, the queue such as store In task be to be executed according to the sequence of first in first out；
Active task queue（Active Tastk List）：Being carrying out for task is stored, times in the queue Business is to have executed just removing for task.
Scheduling thread is needed to execute these tasks, if task can be realized by dispatching multiple threads simultaneously It is parallel to execute.If waiting has task in pending task list, and available free task execution thread, then just being held from waiting A task is taken out in capable task queue, is put into active task queue, is then distributed to a task execution thread；
Being handled in a manner of obtaining difference list of primary keys above by thread parallel to be：If waiting pending task There is no task in list, there is no task in active task list（I.e. one wheel, which checks, to be over）, and do not checked also Specified wheel number, and difference list of primary keys is not sky, then shows that there are discrepant major keys in two databases, thus Using current difference list of primary keys as the difference list of primary keys of initialization.
Then several new tasks are created again, and task type is to scan again, by the major key in all differences list of primary keys It is assigned in new task, newly-built task is all added in task list, carry out difference list of primary keys and master-slave database It compares.To execute repeatedly these class new tasks, executed once it is alternatively possible to be defaulted as every five seconds for example.
If waiting does not have task in pending task list, also without task in active task list, and Completion has checked specified wheel number（Finally it is determined with variance data）Or difference list of primary keys is sky（Final judgement indifference Data）, then simultaneously output verification summary report is generated.
In a preferred embodiment, input or configuration file are first passed through in advance to configure N and adjacent execute institute twice Step S502 is stated to the interval time M between step S504.Optionally, an above-mentioned steps S502 was executed every M seconds to step S504, wherein the value range of M is [2,10], and the value range of N is [1,10].
The dynamic check method and apparatus of the master-slave database consistency of the application have following advantage, certainly, implement this Any product of application does not necessarily require achieving all the advantages described above at the same time：
1）Start several processes by configuration, a pair of primary database table of the same name of each process scanning and from database table, Only need the minimum major key of reading primary database table, you can create first scan task, each first run scan task can be according to reading The data volume gone out judges whether to create next first run scan task；Entirety need not be known in advance during entire data scanning Data scale carry out task dismantling again；
2）Assign thread（Dispatcher Thread）After every wheel scan task, judge whether to reach configuration Whether maximum scan wheel number, global disparity data list of primary keys are empty, judge whether the scan task for establishing next round；
3）Scan task is that multi-threaded parallel executes；
4）In primary database and from database, data are all the sequences carried out by major key sequence, read same zone respectively Domain, i.e., identical minimum major key or maximum major key（The first run scans）Or identical major key（It scans again）So that data comparison becomes It is very easy and rapid to obtain；
5）When the most bull wheel number of configuration requirement is completed in scanning, when global disparity data list of primary keys is not empty, every is exported Variance data；
6）In entire scanning process, database loads are controllable, because reading data using following several SQL statements batch, More configuration it can control the batch size read and reading frequency：
SELECT*FROM TABLE_NAME WHERE PK>${MIN_PK}ORDER BY PK ASC LIMIT${MAX_ COUNT}；
SELECT*FROM TABLE_NAME WHERE PK>${MIN_PK}AND PK<=${MAX_PK}ORDER BY PK ASC；
SELECT*FROM TABLE_NAME WHERE PK in(a,b,c…)ORDER BY PK ASC；
7）Entire checking procedure is dynamic check, does not need master-slave database and stops externally service, even moment stops Only externally service, such as without lockking database, or latching tables of data.
By above description it is found that the application realizes following technique effect：
1）Using difference list of primary keys as an intermediate quantity and tracking object, the number in master and slave database is repeatedly compared According to, by it is having in difference list of primary keys but it is current restored consistent major key record in master and slave database and gradually screened out, from And efficiently solve in the related technology because of the technical problem of the result inaccuracy of consistency desired result caused by synchronization delay, it reaches The technique effect for improving master-slave database verification accuracy is arrived；
2）By the division to the data record in database, it is scanned with the multiple tasks for generating parallel, Yi Jijin The step of row subsequent consistency desired result, reduces pair on the other hand to reach the technique effect for shortening checking time The requirement of processor performance, realizes and is efficiently used to processing capacity, does not lock library, no in consistency desired result to meet The requirement for locking table realizes the effect that dynamic detection is carried out when database and tables of data are in normal read-write state；
3）By the processing of the accurate result to master-slave database consistency desired result, technical scheme can also be realized Effective reparation to the data being had differences in master and slave database, to achieve the effect that master-slave database dynamic is consistent.
Obviously, those skilled in the art should be understood that each module of above-mentioned the application or each step can be with general Computing device realize that they can be concentrated on a single computing device, or be distributed in multiple computing devices and formed Network on, optionally, they can be realized with the program code that computing device can perform, it is thus possible to which they are stored It is performed by computing device in the storage device, and in some cases, it can be with different from shown in sequence execution herein The step of going out or describing, either they are fabricated to each integrated circuit modules or by them multiple modules or Step is fabricated to single integrated circuit module to realize.It is combined in this way, the application is not limited to any specific hardware and software.
The foregoing is merely the preferred embodiments of the application, are not intended to limit this application, for the skill of this field For art personnel, the application can have various modifications and variations.Within the spirit and principles of this application, any made by repair Change, equivalent replacement, improvement etc., should be included within the protection domain of the application.

Claims (10)

1. a kind of dynamic check method of master-slave database consistency, which is characterized in that including：
Repeat following steps n times, wherein N is preset natural number：Retrieval and difference primary key column from primary database The corresponding data-base recording of major key in table, to obtain primary database record set；Retrieval and the difference master from from database The corresponding data-base recording of major key in key list, to obtain from database record set；Obtain the primary database record set with The major key concentrated from data-base recording corresponding to the data-base recording having differences, by the master in the difference list of primary keys Key is updated to the major key corresponding to the data-base recording having differences；
After having executed step described in n times, if still having major key in the difference list of primary keys, judge the primary database with It is described inconsistent from database, otherwise, judge the primary database with it is described consistent from database；
Wherein, the major key corresponding to master-slave database record that the difference list of primary keys is used to have differences.
2. according to the method described in claim 1, it is characterized in that, judge the primary database with it is described from database not After consistent, further include：Execute data reparation operation, wherein the data reparation, which operates, includes：
It is searched simultaneously by the primary database and institute in the difference list of primary keys obtained after having executed step described in n times The major key for including from the data-base recording in database is stated, the major key of the first major key type is denoted as, using in the primary database With type be the major key of the first major key type corresponding data-base recording covering it is described from database with first master The corresponding data-base recording of key type；
It is searched by the data in the primary database in the difference list of primary keys obtained after having executed step described in n times Major key that is that library record includes but not including by the data-base recording from database, is denoted as the second major key type Major key copies to data-base recording corresponding with the major key of the second major key type in the primary database described from data In library；
It is searched by the data from database in the difference list of primary keys obtained after having executed step described in n times Major key that is that library record includes but not including by the data-base recording in the primary database, is denoted as third major key type, and Data-base recording corresponding with the major key of third major key type is deleted from database from described.
3. according to the method described in claim 1, it is characterized in that, obtain the primary database record set with it is described from database The major key corresponding to data-base recording being had differences in record set includes：
If same major key corresponding data-base recording in the primary database record set is concentrated with described from data-base recording Corresponding data-base recording is different, then the major key is recorded as to the major key corresponding to the data-base recording having differences；
If same major key is in the primary database record set there are corresponding data-base recording and described from data-base recording It concentrates and corresponding data-base recording is not present, then the major key is recorded as to the master corresponding to the data-base recording having differences Key；
If same major key is recorded in described concentrated from data-base recording there are corresponding data-base recording in the primary database It concentrates and corresponding data-base recording is not present, then the major key is recorded as to the master corresponding to the data-base recording having differences Key.
4. according to the method described in claim 1, it is characterized in that, before repeating the step n times, further include：It is raw At the initial value of the difference list of primary keys, wherein it is described generate initialization difference list of primary keys include：It is performed in parallel and appoints Multiple tasks in business queue scan the primary database and the data-base recording from database, to obtain the difference The initial value of different list of primary keys, wherein each task includes following operation, wherein the initial value of task minimum key value is Minimum value in the primary database in the corresponding major key of all data-base recordings of detected data table：
Press since data-base recording corresponding with the task minimum key value in the primary database major key from small to large The data-base recording of sequential scan predetermined number of lines, by Current Scan in the primary database to data-base recording be recorded as currently Primary database record set obtains the maximum value in the major key corresponding to the data-base recording in the current primary database record set As task maximum key assignments, next task is created, most using the task maximum key assignments as the task of the next task Small key assignments, and the next task of establishment is put into the task queue；
It is described from database scanning with value range [the task minimum key value, the task maximum key assignments) in major key Corresponding data-base recording, by the Current Scan from database to data-base recording be recorded as currently from data-base recording Collection；
If same major key in the current primary database record set corresponding data-base recording with described currently from database Corresponding data-base recording is different in record set, then the major key is recorded as to the initial value member of the difference list of primary keys；
If same major key is in the current primary database record set there are corresponding data-base recording and described currently from number According to corresponding data-base recording is not present in the record set of library, then the major key is recorded as the initial value of the difference list of primary keys at Member；
If same major key is currently concentrated from data-base recording there are corresponding data-base recording and in the current main number described
According to corresponding data-base recording is not present in the record set of library, then the major key is recorded as the initial of the difference list of primary keys It is worth member；
The set that the initial value member of the difference list of primary keys is constituted stores in a list, to constitute the difference master The initial value of key list.
5. according to the method described in claim 4, it is characterized in that, the multiple task of generation is put into task queue In, execute the multiple task in the following manner：
Judge to whether there is pending task in the task queue；
If in the presence of it is pre- to judge whether the quantity of thread be currently performed in parallel, for executing the task is less than Fixed maximum thread；
If the quantity of the current thread executed parallel is less than scheduled maximum thread, from the task queue The pending task is taken out, and calls an idle thread to execute the pending task；If it is described it is current just It is equal to or more than the scheduled maximum thread in the quantity of the thread executed parallel, then continues the pending task It is retained in the task queue.
6. according to the method described in claim 1, it is characterized in that, pre- first pass through configuration file configuration N and adjacent execute twice Interval time M between the step.
7. according to the method described in claim 1, it is characterized in that, executing the primary step every M seconds, wherein the value of M Value range for the fixed value or random value in [2,10], N is [1,10].
8. a kind of dynamic corrector of master-slave database consistency, which is characterized in that including：
Updating unit, for repeating following steps n times, wherein N is preset natural number：It is examined from primary database Rope data-base recording corresponding with the major key in difference list of primary keys, to obtain primary database record set；It is examined from from database Rope data-base recording corresponding with the major key in the difference list of primary keys, to obtain from database record set；Obtain the master Database record set and the major key corresponding to the data-base recording concentrated and had differences from data-base recording, by the difference Major key in list of primary keys is updated to the major key corresponding to the data-base recording having differences；
Judging unit, if still having major key in the difference list of primary keys, is judged after having executed step described in n times The primary database with it is described inconsistent from database, otherwise, judge the primary database with it is described consistent from database；
Wherein, the major key corresponding to master-slave database record that the difference list of primary keys is used to have differences.
9. device according to claim 8, which is characterized in that further include：
Unit is repaired, for judging the primary database and described after database is inconsistent, execution data reparation behaviour Make, wherein the data reparation, which operates, includes：In the difference list of primary keys obtained after having executed step described in n times The major key for including by the primary database and the data-base recording from database simultaneously is searched, the first major key type is denoted as Major key, using being that the corresponding data-base recording of the major key of the first major key type covers institute with type in the primary database State the data-base recording corresponding with the first major key type from database；It has been obtained after having executed step described in n times In the difference list of primary keys search include by the data-base recording in the primary database but not by described from database In the data-base recording major key that includes, be denoted as the major key of the second major key type, will in the primary database with second master The corresponding data-base recording of major key of key type copies to described from database；It is obtained after having executed step described in n times The difference list of primary keys in search include by the data-base recording from database but not by the master data The major key that data-base recording in library includes is denoted as third major key type, and is deleted and the third from database from described The corresponding data-base recording of major key of major key type.
10. device according to claim 8, which is characterized in that further include：
Initialization unit, for before repeating the step n times, generating the initial value of the difference list of primary keys, In, the difference list of primary keys for generating initialization includes：It is described to scan to be performed in parallel the multiple tasks in task queue Primary database and the data-base recording from database, to obtain the initial value of the difference list of primary keys, wherein each The task includes following operation, wherein the initial value of task minimum key value is detected data table in the primary database Minimum value in the corresponding major key of all data-base recordings：From corresponding with the task minimum key value in the primary database Data-base recording starts the data-base recording of the sequential scan predetermined number of lines by major key from small to large, by the primary database when The preceding data-base recording scanned is recorded as current primary database record set, obtains the number in the current primary database record set According to the maximum value in the corresponding major key of library record as task maximum key assignments, next task is created, by the task maximum Task minimum key value of the key assignments as the next task, and the next task of establishment is put into the task queue In；It is described from database scanning with value range [the task minimum key value, the task maximum key assignments) in major key Corresponding data-base recording, by the Current Scan from database to data-base recording be recorded as currently from data-base recording Collection；If same major key corresponding data-base recording in the current primary database record set is currently remembered from database with described Record concentrates corresponding data-base recording different, then the major key is recorded as to the initial value member of the difference list of primary keys；If same One major key is in the current primary database record set there are corresponding data-base recording and described currently from data-base recording It concentrates and corresponding data-base recording is not present, then the major key is recorded as to the initial value member of the difference list of primary keys；If same One major key is recorded in described currently concentrated from data-base recording there are corresponding data-base recording in the current primary database It concentrates and corresponding data-base recording is not present, then the major key is recorded as to the initial value member of the difference list of primary keys；By institute The set for stating the initial value member composition of difference list of primary keys stores in a list, to constitute the difference list of primary keys Initial value.
CN201310263694.3A 2013-06-27 2013-06-27 The dynamic check method and apparatus of master-slave database consistency Active CN104252481B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201310263694.3A CN104252481B (en) 2013-06-27 2013-06-27 The dynamic check method and apparatus of master-slave database consistency

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201310263694.3A CN104252481B (en) 2013-06-27 2013-06-27 The dynamic check method and apparatus of master-slave database consistency

Publications (2)

Publication Number Publication Date
CN104252481A CN104252481A (en) 2014-12-31
CN104252481B true CN104252481B (en) 2018-10-19

Family

ID=52187383

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310263694.3A Active CN104252481B (en) 2013-06-27 2013-06-27 The dynamic check method and apparatus of master-slave database consistency

Country Status (1)

Country Link
CN (1) CN104252481B (en)

Families Citing this family (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104714785A (en) * 2015-03-31 2015-06-17 中芯睿智（北京）微电子科技有限公司 Task scheduling device, task scheduling method and data parallel processing device
CN106649333B (en) * 2015-10-29 2021-12-10 阿里巴巴集团控股有限公司 Method and device for detecting consistency of field sequence
CN105630635B (en) * 2015-12-29 2019-07-30 北京京东尚科信息技术有限公司 Data disaster tolerance system and method
CN107133249A (en) * 2016-02-29 2017-09-05 上海格尔软件股份有限公司 It is a kind of to having deleted the database synchronization method that data are synchronized
CN107656940B (en) * 2016-07-26 2020-06-16 北京国双科技有限公司 Database processing method and device
CN107819795A (en) * 2016-09-12 2018-03-20 北京金山云网络技术有限公司 A kind of major-minor server data synchronous method of DNS, apparatus and system
CN106649814A (en) * 2016-12-29 2017-05-10 国网江苏省电力公司南京供电公司 Method for consistency checking of distribution automation cross-domain database
CN108322492B (en) * 2017-01-16 2021-09-17 医渡云（北京）技术有限公司 Medical data synchronization method and device
CN107038414B (en) * 2017-03-08 2020-08-14 Oppo广东移动通信有限公司 Fingerprint template synchronization method and device
CN108287886B (en) * 2018-01-16 2022-04-08 创新先进技术有限公司 Method and device for synchronizing data change information
CN108319719B (en) * 2018-02-12 2022-04-01 腾讯科技（深圳）有限公司 Database data verification method and device, computer equipment and storage medium
CN108694249B (en) * 2018-05-30 2022-02-25 平安科技（深圳）有限公司 Data processing method, data processing device, computer equipment and storage medium
CN109241500B (en) * 2018-08-14 2023-02-03 中国平安人寿保险股份有限公司 File batch comparison method and device, computer equipment and storage medium
CN109284301A (en) * 2018-08-31 2019-01-29 阿里巴巴集团控股有限公司 Verification of data method and device
CN109325032B (en) * 2018-09-18 2020-10-27 厦门市美亚柏科信息股份有限公司 Index data storage and retrieval method, device and storage medium
CN109271451B (en) * 2018-10-12 2021-06-22 武汉达梦数据库有限公司 Method for extracting external links of database
CN110209521B (en) * 2019-02-22 2022-03-18 腾讯科技（深圳）有限公司 Data verification method and device, computer readable storage medium and computer equipment
CN110222027A (en) * 2019-04-24 2019-09-10 福建天泉教育科技有限公司 The quantity method of calibration and computer readable storage medium of Data Migration
CN110399376A (en) * 2019-08-08 2019-11-01 北京明略软件系统有限公司 The method and device of the former data variation of automatic identification table
CN110737642B (en) * 2019-09-05 2022-09-02 平安科技（深圳）有限公司 Database information analysis method, database information analysis device, computer device and storage medium
CN110852058A (en) * 2019-09-24 2020-02-28 贵阳朗玛信息技术股份有限公司 List comparison method and device
CN112579591B (en) * 2019-09-30 2023-06-16 重庆小雨点小额贷款有限公司 Data verification method, device, electronic equipment and computer readable storage medium
CN110990414B (en) * 2019-10-31 2023-06-16 口碑(上海)信息技术有限公司 Data processing method and device
CN110968584B (en) * 2019-12-03 2023-03-14 北京明略软件系统有限公司 Portrait generation system, method, electronic device and readable storage medium
CN111198868B (en) * 2019-12-12 2023-09-05 北京淇瑀信息科技有限公司 Intelligent database-dividing real-time data migration method and device
CN111125110B (en) * 2019-12-25 2023-08-18 深圳前海环融联易信息科技服务有限公司 Non-primary key field checking method, device, computer equipment and storage medium
CN111949728A (en) * 2020-07-14 2020-11-17 北京思特奇信息技术股份有限公司 Dynamic data difference comparison method and system
CN112905602B (en) * 2021-03-26 2022-09-30 掌阅科技股份有限公司 Data comparison method, computing device and computer storage medium
CN113177056B (en) * 2021-06-29 2021-09-21 企查查科技有限公司 Method, device and storage medium for checking consistency of service data of multiple databases
CN113760910A (en) * 2021-08-31 2021-12-07 中国银联股份有限公司 Data synchronization method and device
CN113704359B (en) * 2021-09-03 2024-04-26 优刻得科技股份有限公司 Method, system and server for synchronizing multiple data copies of time sequence database
CN115759729A (en) * 2021-11-12 2023-03-07 江西省国土空间调查规划研究院 Disaster risk early warning management system based on algorithm and prediction method
CN115905273B (en) * 2022-11-10 2023-09-26 贝壳找房(北京)科技有限公司 Data detection method in digital bin table, electronic equipment and storage medium
CN115952185B (en) * 2023-03-10 2023-06-30 布比（北京）网络技术有限公司 Data processing method and device, equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1784677A (en) * 2004-03-31 2006-06-07 微软公司 System and method for a consistency check of a database backup
CN101582920A (en) * 2009-06-09 2009-11-18 中兴通讯股份有限公司 Method and device for verifying and synchronizing data blocks in distributed file system
CN102521406A (en) * 2011-12-26 2012-06-27 中国科学院计算技术研究所 Distributed query method and system for complex task of querying massive structured data

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1784677A (en) * 2004-03-31 2006-06-07 微软公司 System and method for a consistency check of a database backup
CN101582920A (en) * 2009-06-09 2009-11-18 中兴通讯股份有限公司 Method and device for verifying and synchronizing data blocks in distributed file system
CN102521406A (en) * 2011-12-26 2012-06-27 中国科学院计算技术研究所 Distributed query method and system for complex task of querying massive structured data

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
"MySQL主从数据一致性校验";阿炯;《http://www.freeoa.net/development/perl/mysql-master-slave-consistency-check_1856.html》;20121230;第1页 *
"percona-toolkit 之 【pt-table-checksum】、【pt-table-sync】说明";jyzhou;《http://www.cnblogs.com/zhoujinyi/archive/2013/05/09/3067045.html?utm_source=tuicool&utm_medium=referral》;20130509;第1-4页 *

Also Published As

Publication number Publication date
CN104252481A (en) 2014-12-31

Similar Documents

Publication Publication Date Title
CN104252481B (en) The dynamic check method and apparatus of master-slave database consistency
CN104423960B (en) A kind of method and system of project continuous integrating
CN110309071B (en) Test code generation method and module, and test method and system
CN110795219A (en) Resource scheduling method and system suitable for multiple computing frameworks
CN108153619A (en) A kind of data proofreading method and device
CN107741903A (en) Application compatibility method of testing, device, computer equipment and storage medium
BR102012002858A2 (en) SYSTEM
CN113238924B (en) Chaotic engineering realization method and system in distributed graph database system
CN111309734B (en) Method and system for automatically generating table data
CN112799782B (en) Model generation system, method, electronic device and storage medium
CN114780138B (en) Flow field simulation software code version management method and device and storage medium
CN109213645A (en) A kind of method and apparatus of distributed memory system performance test
Tsai et al. Test algebra for combinatorial testing
CN111309581B (en) Application performance detection method and device in database upgrading scene
CN113760513A (en) Distributed task scheduling method, device, equipment and medium
CN108140022B (en) Data query method and database system
US20210263837A1 (en) Real time fault localization using combinatorial test design techniques and test case priority selection
CN110659138B (en) Data synchronization method, device, terminal and storage medium based on timing task
CN112559525B (en) Data checking system, method, device and server
CN110399309A (en) A kind of test data generating method and device
CN111382198A (en) Data recovery method, device, equipment and storage medium
CN112486957A (en) Database migration detection method, device, equipment and storage medium
CN114721952A (en) Method, device, equipment and storage medium for synchronously deploying multiple sets of test environments
Klein et al. Application-Specific Evaluation of No SQL Databases
CN112699035A (en) Multi-partition airborne application software association index testing method and device

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20191210

Address after: P.O. Box 31119, grand exhibition hall, hibiscus street, 802 West Bay Road, Grand Cayman, Cayman Islands

Patentee after: Innovative advanced technology Co., Ltd

Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands

Patentee before: Alibaba Group Holding Co., Ltd.