CN104252481A - Dynamic check method and device for consistency of main and salve databases - Google Patents

Dynamic check method and device for consistency of main and salve databases Download PDF

Info

Publication number
CN104252481A
CN104252481A CN201310263694.3A CN201310263694A CN104252481A CN 104252481 A CN104252481 A CN 104252481A CN 201310263694 A CN201310263694 A CN 201310263694A CN 104252481 A CN104252481 A CN 104252481A
Authority
CN
China
Prior art keywords
data
major key
base
database
base recording
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201310263694.3A
Other languages
Chinese (zh)
Other versions
CN104252481B (en
Inventor
叶晖俊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Advanced New Technologies Co Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN201310263694.3A priority Critical patent/CN104252481B/en
Publication of CN104252481A publication Critical patent/CN104252481A/en
Application granted granted Critical
Publication of CN104252481B publication Critical patent/CN104252481B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/004—Error avoidance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/275—Synchronous replication

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a dynamic check method and device for the consistency of main and salve databases. The method comprises the steps of repeatedly executing the following steps for N times, wherein N is a preset natural number: retrieving a database record corresponding to a main key in a difference main key list from the main database to obtain a main database record set; retrieving a database record corresponding to a main key in the difference main key list from a slave database to obtain a slave database record set; obtaining the main keys corresponding to the database records with difference in the main database record set and the slave database record set, updating the main keys in the difference main key list as the main keys corresponding to the database records with the differences; after the steps are executed for N times, if the main key still exists in the difference main key list, judging the inconsistency of the main database and the salve database, otherwise, judging the consistency of the main database and the slave database. The dynamic check method and device for the consistency of the main and salve databases solves the technical problem of inaccurate consistency check result caused by the synchronous delay in related art is solved.

Description

The conforming dynamic check method and apparatus of master-slave database
Technical field
The application relates to internet arena, in particular to the conforming dynamic check method and apparatus of a kind of master-slave database.
Background technology
MySQL is widely used PostgreSQL database software in internet industry.A lot of large-scale website is all in use MySQL database, and along with business scale constantly increases, a MySQL database has been difficult to support huge data access amount.
At present, a comparatively effective solution sets up MySQL principal and subordinate cluster exactly, wherein, is the complete mirror image of master data base (being also called master library) from database (being also called standby database or standby storehouse).Only carry out on master library for write request, read request meeting equiblibrium mass distribution, on all databases, so just makes whole data-base cluster that more jumbo read-write can be provided to serve.If MySQL master library receives newly-increased or changes data, just by synchronization means, the real time data synchronization these being increased newly or change is to standby storehouse.
Although the instrument of MySQL master-slave synchronisation has a variety of, but day by day complicated due to production environment, database failure is even generation, and MySQL Binlog also exists some known Bug, inevitably there will be the situation that between principal and subordinate, data are inconsistent.The data in principal and subordinate storehouse are inconsistent, greatly can reduce the availability of whole data-base cluster, may cause serious fault in production simultaneously.Therefore, check the data consistency of principal and subordinate data, search data inconsistent in master-slave database and it is repaired and just become particularly important.In addition, when researching and developing types of databases master-slave synchronisation instrument, carrying out consistency verification of data fast, search and export inconsistent data, is indispensable in the research and development of database synchronization instrument, test.
Large-scale Internet enterprises is all ceaselessly online service in 7*24 hour, for the data base machine on production line, requires comparatively harsh.Also just the verification of master-slave database dynamic conformance is proposed to the requirement of following several respects:
1) calibration tool does not generally allow at master library, from storehouse, writes data;
2) production environment database can not stop external service, has p.s. hundreds of thousands of data to change, even of short duration stopping externally can not serving, namely can not pin full table;
3) production environment flow pressure is large, and the flow load of calibration tool needs to be controlled;
4) if find that data are inconsistent, needs can be revised immediately, or export the data revised, then turn and repaired data by manual operation.
The data base consistency(-tance) calibration tool often had at present mainly contains: utilize Check Sum SQL statement to carry out the mk-table-checksum of static verification; Or utilizing the special method of synchronization, the Binlog such as carrying out the Statement pattern of the dynamic check of fast illuminated is synchronous.Using more in practical application is generally the Binlog of ROW pattern, but which field that these data base consistency(-tance) calibration tools of increasing income cannot locate specifically which bar data is inconsistent, more cannot complete data restore work.Thus make such Open-Source Tools cannot be applied to actual production environment, also cannot be applied in test environment the error in data located and search database synchronization BUG and cause.
In prior art, the verification of general logarithm certificate is all master data base at one time and standby database are compared to determine whether the data between two databases have realized synchronously, but, the synchronous of factor data is generally after master data base generation Data Update, master data base again by upgrade data syn-chronization in standby database, so just cause there is certain time delay in the process of data syn-chronization, if compared with the data between same time point two databases, often cause being judged to be that consistent data are judged to because of delay problem inconsistent.Such as, the time delay of database synchronization is 1ms, then just in this 1ms, has carried out consistency desired result, so just makes master data base in this 1ms not be detected to the data that standby database is synchronous, thus can produce a wrong judged result.
Visible, in existing method of calibration testing process, need locking data table, or lock whole database, even stop externally serving a period of time, thus cause the result of consistency desired result inaccurate.For the problems referred to above, effective solution is not yet proposed.
Summary of the invention
This application provides the conforming dynamic check method and apparatus of a kind of master-slave database, at least to solve the inaccurate technical matters of result of the consistency desired result caused because of synchronization delay in prior art, reach the technique effect improving master-slave database verification accuracy.
According to an aspect of the application, provide a kind of conforming dynamic check method of master-slave database, comprise: repeat following steps N time, wherein, N is the natural number preset: the data-base recording that retrieval is corresponding with the major key in difference list of primary keys from master data base, to obtain master data base record set; From the data-base recording that retrieval from database is corresponding with the major key in above-mentioned difference list of primary keys, to obtain from database record set; Obtain above-mentioned master data base record set and above-mentionedly concentrate the major key corresponding to data-base recording that there are differences from data-base recording, the major key in above-mentioned difference list of primary keys is updated to the major key corresponding to the above-mentioned data-base recording that there are differences; After executing N above-mentioned steps, if still have major key in above-mentioned difference list of primary keys, then judge above-mentioned master data base and above-mentioned inconsistent from database, otherwise, judge that above-mentioned master data base is consistent from database with above-mentioned.
Alternatively, judging above-mentioned master data base and above-mentioned from after database is inconsistent, the conforming dynamic check method of above-mentioned master-slave database also comprises: perform data restore operation, wherein, the operation of above-mentioned data restore comprises: by above-mentioned master data base and the above-mentioned major key comprised from the data-base recording database while of searching from the above-mentioned difference list of primary keys obtained after executing N above-mentioned steps, be designated as the major key of the first major key type, use in above-mentioned master data base is that the data-base recording that the major key of above-mentioned first major key type is corresponding covers above-mentioned data-base recording corresponding with above-mentioned first major key type from database with type, search from the above-mentioned difference list of primary keys obtained after executing N above-mentioned steps comprised by the data-base recording in above-mentioned master data base but not by the above-mentioned major key comprised from the data-base recording database, be designated as the major key of the second major key type, data-base recording corresponding with the major key of above-mentioned second major key type in above-mentioned master data base is copied to above-mentioned from database, search from the above-mentioned difference list of primary keys obtained after executing N above-mentioned steps by above-mentioned from the data-base recording database comprise but not by major key that the data-base recording in above-mentioned master data base comprises, be designated as the 3rd major key type, and from database, delete the data-base recording corresponding with the major key of above-mentioned 3rd major key type from above-mentioned.
Alternatively, obtaining above-mentioned master data base record set concentrates the major key corresponding to the data-base recording that there are differences to comprise from above-mentioned from data-base recording: if the data-base recording corresponding above-mentioned master data base record set of same major key concentrates corresponding data-base recording different with above-mentioned from data-base recording, then this major key is recorded as the major key corresponding to the above-mentioned data-base recording that there are differences; If there is corresponding data-base recording and at the above-mentioned data-base recording that there is not correspondence of concentrating from data-base recording, then this major key be recorded as the major key corresponding to the above-mentioned data-base recording that there are differences in same major key in above-mentioned master data base record set; If same major key exists corresponding data-base recording above-mentioned to concentrate from data-base recording and there is not corresponding data-base recording above-mentioned master data base record set, then this major key is recorded as the major key corresponding to the above-mentioned data-base recording that there are differences.
Alternatively, before repeating above-mentioned steps N time, above-mentioned master-slave database conforming dynamic check method also comprises: the initial value generating above-mentioned difference list of primary keys, wherein, above-mentioned generation initialized difference list of primary keys comprises: multiple tasks of executing the task concurrently in queue are to scan above-mentioned master data base and above-mentioned from the data-base recording database, to obtain the initial value of above-mentioned difference list of primary keys, wherein, each above-mentioned task comprises following operation, wherein, the initial value of task minimum key value is the minimum value in the major key that all data-base recordings of detected data table in above-mentioned master data base are corresponding: the data-base recording scanning predetermined number of lines in above-mentioned master data base from the data-base recording corresponding with above-mentioned task minimum key value by major key order from small to large, by Current Scan in above-mentioned master data base to data-base recording be recorded as current master data base record set, obtain maximal value in the major key corresponding to data-base recording in above-mentioned current master data base record set as task maximum key assignments, create next task, using the task minimum key value of above-mentioned task maximum key assignments as above-mentioned next task, and the above-mentioned next task created is put into above-mentioned task queue, above-mentioned from database scanning and value scope [above-mentioned task minimum key value, above-mentioned task maximum key assignments) in data-base recording corresponding to major key, by above-mentioned from database Current Scan to data-base recording be recorded as current from database record set, if the data-base recording corresponding in above-mentioned current master data base record set of same major key currently concentrates corresponding data-base recording different from data-base recording from above-mentioned, then this major key is recorded as the initial value member of above-mentioned difference list of primary keys, if there is corresponding data-base recording and at the above-mentioned current data-base recording that there is not correspondence of concentrating from data-base recording, then this major key be recorded as the initial value member of above-mentioned difference list of primary keys in same major key in above-mentioned current master data base record set, if same major key exists corresponding data-base recording above-mentioned current to concentrate from data-base recording and there is not corresponding data-base recording above-mentioned current master data base record set, then this major key is recorded as the initial value member of above-mentioned difference list of primary keys, the set initial value member of above-mentioned difference list of primary keys formed stores in a list, to form the initial value of above-mentioned difference list of primary keys.
Alternatively, the above-mentioned multiple task generated is put in task queue, performs above-mentioned multiple task in the following manner: judge whether there is pending task in above-mentioned task queue; If exist, then judge current that performing concurrently, whether be less than predetermined maximum thread for the quantity performing the thread of above-mentioned task; Currently just be less than predetermined maximum thread in the quantity of the thread of executed in parallel if above-mentioned, then from above-mentioned task queue, take out above-mentioned pending task, and call an idle thread to perform above-mentioned pending task; Currently just be equal to or greater than above-mentioned predetermined maximum thread in the quantity of the thread of executed in parallel if above-mentioned, then above-mentioned pending task continued to be retained in above-mentioned task queue.
Alternatively, in advance by configuration file configuration N and adjacent M interval time performed between above-mentioned steps for twice.
Alternatively, perform an above-mentioned steps second every M, wherein, the value of M is fixed value in [2,10] or random value, and the span of N is [1,10].
According to the another aspect of the application, provide the conforming dynamic corrector of a kind of master-slave database, comprise: updating block, for repeating following steps N time, wherein, N is the natural number preset: the data-base recording that retrieval is corresponding with the major key in difference list of primary keys from master data base, to obtain master data base record set; From the data-base recording that retrieval from database is corresponding with the major key in above-mentioned difference list of primary keys, to obtain from database record set; Obtain above-mentioned master data base record set and above-mentionedly concentrate the major key corresponding to data-base recording that there are differences from data-base recording, the major key in above-mentioned difference list of primary keys is updated to the major key corresponding to the above-mentioned data-base recording that there are differences; Judging unit, after executing N above-mentioned steps, if still have major key in above-mentioned difference list of primary keys, then judges above-mentioned master data base and above-mentioned inconsistent from database, otherwise, judge that above-mentioned master data base is consistent from database with above-mentioned.
Alternatively, the conforming dynamic corrector of above-mentioned master-slave database also comprises: repair unit, for judging above-mentioned master data base and above-mentioned from after database is inconsistent, execution data restore operates, wherein, the operation of above-mentioned data restore comprises: by above-mentioned master data base and the above-mentioned major key comprised from the data-base recording database while of searching from the above-mentioned difference list of primary keys obtained after executing N above-mentioned steps, be designated as the major key of the first major key type, use in above-mentioned master data base is that the data-base recording that the major key of above-mentioned first major key type is corresponding covers above-mentioned data-base recording corresponding with above-mentioned first major key type from database with type, search from the above-mentioned difference list of primary keys obtained after executing N above-mentioned steps comprised by the data-base recording in above-mentioned master data base but not by the above-mentioned major key comprised from the data-base recording database, be designated as the major key of the second major key type, data-base recording corresponding with the major key of above-mentioned second major key type in above-mentioned master data base is copied to above-mentioned from database, search from the above-mentioned difference list of primary keys obtained after executing N above-mentioned steps by above-mentioned from the data-base recording database comprise but not by major key that the data-base recording in above-mentioned master data base comprises, be designated as the 3rd major key type, and from database, delete the data-base recording corresponding with the major key of above-mentioned 3rd major key type from above-mentioned.
Alternatively, the conforming dynamic corrector of above-mentioned master-slave database also comprises: initialization unit, for before repeating above-mentioned steps N time, generate the initial value of above-mentioned difference list of primary keys, wherein, above-mentioned generation initialized difference list of primary keys comprises: multiple tasks of executing the task concurrently in queue are to scan above-mentioned master data base and above-mentioned from the data-base recording database, to obtain the initial value of above-mentioned difference list of primary keys, wherein, each above-mentioned task comprises following operation, wherein, the initial value of task minimum key value is the minimum value in the major key that all data-base recordings of detected data table in above-mentioned master data base are corresponding: the data-base recording scanning predetermined number of lines in above-mentioned master data base from the data-base recording corresponding with above-mentioned task minimum key value by major key order from small to large, by Current Scan in above-mentioned master data base to data-base recording be recorded as current master data base record set, obtain maximal value in the major key corresponding to data-base recording in above-mentioned current master data base record set as task maximum key assignments, create next task, using the task minimum key value of above-mentioned task maximum key assignments as above-mentioned next task, and the above-mentioned next task created is put into above-mentioned task queue, above-mentioned from database scanning and value scope [above-mentioned task minimum key value, above-mentioned task maximum key assignments) in data-base recording corresponding to major key, by above-mentioned from database Current Scan to data-base recording be recorded as current from database record set, if the data-base recording corresponding in above-mentioned current master data base record set of same major key currently concentrates corresponding data-base recording different from data-base recording from above-mentioned, then this major key is recorded as the initial value member of above-mentioned difference list of primary keys, if there is corresponding data-base recording and at the above-mentioned current data-base recording that there is not correspondence of concentrating from data-base recording, then this major key be recorded as the initial value member of above-mentioned difference list of primary keys in same major key in above-mentioned current master data base record set, if same major key exists corresponding data-base recording above-mentioned current to concentrate from data-base recording and there is not corresponding data-base recording above-mentioned current master data base record set, then this major key is recorded as the initial value member of above-mentioned difference list of primary keys, the set initial value member of above-mentioned difference list of primary keys formed stores in a list, to form the initial value of above-mentioned difference list of primary keys.
In this application, using difference list of primary keys as an intermediate quantity and tracing object, data repeatedly in the master and slave database of comparison, by have in difference list of primary keys but currently in master and slave database, recover consistent major key record successively screen out, thus the inaccurate technical matters of the result efficiently solving the consistency desired result caused because of synchronization delay in correlation technique, reach the technique effect improving master-slave database verification accuracy.
In addition, in this application, by the division to the data record in database, scan to generate parallel multiple tasks, and carry out the step of follow-up consistency desired result, thus reach the technique effect shortening checking time, achieve the effective utilization to processing power, thus meet the requirement of not locking storehouse in consistency desired result, not locking table, not stopping externally service, achieve the effect of carrying out detection of dynamic when database and tables of data are in normal read-write state.
Further, by the process of the accurate result to master-slave database consistency desired result, technical scheme can also realize the effective reparation to the data that there are differences in master and slave database, thus reaches the dynamically consistent effect of master-slave database.
Accompanying drawing explanation
Accompanying drawing described herein is used to provide further understanding of the present application, and form a application's part, the schematic description and description of the application, for explaining the application, does not form the improper restriction to the application.In the accompanying drawings:
Fig. 1 is a kind of preferred structure block diagram according to the conforming check system of the master-slave database of the embodiment of the present application;
Fig. 2 is a kind of preferred structure block diagram according to the conforming dynamic corrector of the master-slave database of the embodiment of the present application;
Fig. 3 is the another kind of preferred structure block diagram according to the conforming dynamic corrector of the master-slave database of the embodiment of the present application;
Fig. 4 is another the preferred structure block diagram according to the conforming dynamic corrector of the master-slave database of the embodiment of the present application;
Fig. 5 is a kind of preferred flow charts according to the master-slave database of the embodiment of the present application conforming dynamic check method;
Fig. 6 is the another kind of preferred flow charts according to the master-slave database of the embodiment of the present application conforming dynamic check method;
Fig. 7 is another preferred flow charts according to the conforming dynamic check method of the master-slave database of the embodiment of the present application;
Fig. 8 is another preferred flow charts according to the conforming dynamic check method of the master-slave database of the embodiment of the present application;
Fig. 9 is another preferred flow charts according to the conforming dynamic check method of the master-slave database of the embodiment of the present application;
Figure 10 is another preferred flow charts according to the conforming dynamic check method of the master-slave database of the embodiment of the present application.
Embodiment
Hereinafter also describe the application in detail with reference to accompanying drawing in conjunction with the embodiments.It should be noted that, when not conflicting, the embodiment in the application and the feature in embodiment can combine mutually.
Before the further details of each embodiment describing the application, the counting system structure that of the principle that can be used for realizing the application is suitable is described with reference to Fig. 1.In the following description, except as otherwise noted, the symbol otherwise with reference to the action performed by one or more computing machine and operation represents each embodiment describing the application.Thus, be appreciated that processing unit that this kind of action performed sometimes referred to as computing machine and operation comprise computing machine is to the manipulation of electric signal representing data with structured form.It is safeguarded in this manipulation transforms data or the position in the accumulator system of computing machine, and this reshuffles or changes the operation of computing machine in the mode that those skilled in the art understands.The data structure of service data is the physical location of the storer of the particular community that the form with data defines.But although describe the application in above-mentioned context, it does not also mean that restrictive, as understood by those skilled in the art, hereinafter described action and each side of operation also can realize with hardware.
Turn to accompanying drawing, wherein identical reference number refers to identical element, and the principle of the application is shown in a suitable computing environment and realizes.Below describe the embodiment based on described the application, and should not think about the alternative embodiment clearly do not described herein and limit the application.
Fig. 1 shows the schematic diagram of the example computer architecture that can be used for these equipment.For purposes of illustration, the architecture of painting is only an example of proper environment, not proposes any limitation to the usable range of the application or function.This computing system should be interpreted as, to the arbitrary assembly shown in Fig. 1 or its combination, there is any dependence or demand yet.
The principle of the application can use other universal or special calculating or communication environment or configuration to operate.Be applicable to the well-known computing system of the application, the example of environment and configuration includes but not limited to, personal computer, server, multicomputer system, system based on micro-process, minicomputer, mainframe computer and comprise the distributed computing environment of arbitrary said system or equipment.
As one of configuration that it is the most basic, the conforming check system of the master-slave database in Fig. 1 100 can at least comprise: a server 102 and the one or more distributed processing terminal 104 be connected with this server 102.Wherein, server 102 can include but not limited to Micro-processor MCV or programmable logic device (PLD) FPGA etc. treating apparatus, for the memory storage that stores data and the transmitting device communicated with processing terminal; Processing terminal 104 can comprise: Micro-processor MCV, with the transmitting device of server communication, display device with user interactions.In the present specification and claims, " system " also can be defined as the conforming check system of combination master-slave database 100 of executive software, firmware or the microcode any nextport hardware component NextPort or nextport hardware component NextPort that carry out practical function can being even distributed, to realize distributed function.
Alternatively, each processing terminal 104 running example, only responsible dynamic chek a pair database table, the i.e. tables of data of a master data base, and the tables of data from database that corresponding; Can start several processing terminals 104 simultaneously and carry out running example according to configuration, both can walk abreast performs consistance detection of dynamic to multiple tables of data simultaneously, and each processing terminal 104 running example is responsible for the dynamic conformance inspection of a pair database table.
As used in this application, term " module ", " assembly " or " unit " can refer to the software object that performs on the conforming check system 100 of master-slave database or routine.Different assembly described herein, module, unit, engine and service can be implemented as on the conforming check system of master-slave database 100, perform (such as, as independent thread) object or process.Although system and method described herein preferably realizes with software, the realization of the combination of hardware or software and hardware also may and conceived.
Embodiment 1
Fig. 2 is the preferred structural representation of one according to the conforming dynamic corrector of the embodiment of the present application master-slave database, and preferably, the system in the present embodiment can be positioned on server 102.
The inventive concept of this preferred embodiment mainly can comprise: initialization difference list of primary keys; Difference list of primary keys is carried out to the renewal rewards theory of pre-determined number; After the renewal rewards theory completing above-mentioned pre-determined number, if difference list of primary keys is not still empty, then show that the database of the master-slave database that this difference list of primary keys is corresponding is inconsistent.Wherein, what retain in this difference list of primary keys is when the major key corresponding to the master data base determined after secondary comparison and the data that there are differences from database.
Preferably, in order to achieve the above object, the conforming dynamic corrector of master-slave database as shown in Figure 2 can comprise: updating block 202 and judging unit 204, be specifically described this structure below.
1) updating block 202, for repeating following steps N time, wherein, N is the natural number preset: the data-base recording that retrieval is corresponding with the major key in difference list of primary keys from master data base, to obtain master data base record set; From the data-base recording that retrieval from database is corresponding with the major key in described difference list of primary keys, to obtain from database record set; Obtain master data base record set and concentrate the major key corresponding to data-base recording that there are differences from data-base recording, the major key in difference list of primary keys is updated to the major key corresponding to the data-base recording that there are differences;
2) judging unit 204, is coupled with updating block 202, for after executing N above-mentioned steps, if still have major key in difference list of primary keys, then judges that master data base is with inconsistent from database, otherwise, then judge that master data base is consistent with from database.
Difference list of primary keys, as one of concept proposed in technical scheme, for representing a kind of set of major key.Alternatively, in the embodiment of the present application, as the specific implementation of this major key set, its form can be major key the set of unique corresponding key assignments, be stored in a list, to perform the read-write operations such as further scanning, retrieval and sequence in server 102, than reciprocity arithmetic operation.
As its title disclose, difference list of primary keys is mainly used in storing the major key corresponding to master-slave database record that there are differences.But in the embodiment of the present application, as the intermediate quantity between the repeatedly above-mentioned renewal rewards theory performed by updating block 202, difference list of primary keys non-static, the major key wherein stored can renew at a renewal rewards theory relaying and stay, also can be removed, and the difference list of primary keys after upgrading can be sent in a new renewal rewards theory, also can export in judging unit 204 to draw the conforming check results of master-slave database, thus can by the repeatedly renewal to initial difference list of primary keys, progressively get rid of the erroneous judgement factor of the consistance judgement caused because of master-slave database synchronization delay, then the scope of judgement is reduced, the position of the data-base recording that there are differences in master-slave database in complete check-verifying period is accurately determined once in final realization, and these positions can be exported as verification report, to check or further to process.
Alternatively, in the single job that updating block 202 performs, can according to the first group of major key recorded in the difference list of primary keys of this operation incipient stage, respectively in master data base and from database, retrieve the data-base recording corresponding with first group of major key, and represent with master data base record set and from database record set respectively, then can contrast at master data base record set and between database record set, find out corresponding to part the second group of major key that wherein there are differences, and first group of major key is replaced with second group of major key in difference list of primary keys, then this operation is terminated, thus complete once to the renewal of difference list of primary keys.
It should be noted that the quantity of major key in first group of major key can for one or more, it is one or more also can be that zero, the second group of major key can comprise in first group of major key, also can not comprise wherein any one; Second group of major key is a subset of first group of major key.On the other hand, the determination of second group of major key can be carried out in above-mentioned comparison process, also can carry out after above-mentioned comparison process terminates.In addition, the description that first group of major key replaced by above-mentioned use second group of major key should be understood to describe for the one of the embodiment of the present application, any restriction can't be formed to the application, such as, updating block 202 can also be used for performing following operation to realize above-mentioned effect: 1) in the first major key group, obtain the 3rd major key group, wherein, the 3rd major key group is consistent with data-base recording content corresponding from database in master data base; 2) the 3rd major key group is deleted from difference list of primary keys.
Under above-mentioned scene, by to major key a limited number of in difference list of primary keys and corresponding data-base recording thereof but not the process of whole master and slave database, can reduce and the consuming time of the single sense cycle of consistency desired result is carried out to master-slave database, thus the conforming dynamic check carrying out master-slave database when not stopping externally service for database and place system thereof provides technical foundation.On the other hand, because single detects shortening consuming time, improve the ageing of master-slave database consistency desired result, reduce the error rate of the consistance judgement caused due to time delay, thus improve the accuracy of check results.
Further, the single job that above-mentioned updating block 202 performs can perform again: in new single job, the major key recorded in the difference list of primary keys of incipient stage is second group of major key, thereafter, respectively in master data base and retrieve the data-base recording corresponding with second group of major key from database, find out the corresponding to part the 4th group of major key that wherein there are differences, and second group of major key is replaced with the 4th group of major key in difference list of primary keys, then terminate this operation, thus complete the renewal again to difference list of primary keys.
Under above-mentioned scene, based on the difference major key recorded in the difference list of primary keys obtained after above-mentioned twice operation i.e. the 4th major key group, the result that the judgement performed by judging unit 204 operates the master-slave database consistance judgement obtained has higher confidence level and lower False Rate, and details are as follows for its cardinal principle:
Imagination server has made amendment in the first moment to the data-base recording corresponding to a certain major key in master data base, and as the synchronous repair action with this amendment, server has made corresponding amendment in the second moment to the data-base recording from database corresponding to this major key, and in the single job of above-mentioned updating block 202 execution, the difference major key i.e. acquisition of second group of major key is completed between the first moment and the second moment, this then corresponding with the data-base recording of change major key will be credited in second group of major key, even if the master-slave database record subsequently corresponding to this major key of the second moment achieves synchronously, thus there is the conforming false judgment that time delay causes.But this mistake can be eliminated by the above-mentioned second time operation carried out after the preset lime, such as, complete after in second time operation, the difference major key i.e. acquisition of the 4th group of major key was arranged on for the second moment, then the above-mentioned major key corresponding with the data-base recording of change can not be logged in the 4th group of major key, thus have higher confidence level and lower False Rate by the result that the master-slave database consistance that judging unit 204 obtains judges.
Further, the single job that above-mentioned updating block 202 performs can also repeat N time, to obtain higher confidence level and lower False Rate.Such as, for time of data delay on line the situation of about 1 second, if default configuration is every 5 seconds to variance data duplication check once, acquiescence is carried out 8 and is taken turns verification.Then only have very extreme situation, these data of each verification that just there will be in these 8 times verifications all changed, and just master-slave synchronisation does not also complete, and so these data will be mistaken for inconsistent data.But the probability that this situation occurs in production environment is minimum, and user can according to the variance data report exported, the manual anti-data looked in master-slave database again.Therefore, by the above-mentioned inventive concept of the application in actual production practice process, check results has very high confidence level and very low False Rate.
Alternatively, judging unit 204 judge inconsistent after also just to determine between master-slave database inconsistent, the data of certain mode to master-slave database so just can be adopted to upgrade to realize the synchronous of data in two databases, in a preferred embodiment, as shown in Figure 3, the conforming dynamic corrector of above-mentioned master-slave database can also comprise:
Repair unit 302, be coupled with judging unit 204, for judge master data base with from after database is inconsistent, execution data restore operates, wherein, the operation of above-mentioned data restore comprises: by master data base and the major key that comprises from the data-base recording database while of searching from the difference list of primary keys obtained after executing N above-mentioned steps, be designated as the major key of the first major key type, using in master data base is that the data-base recording that the major key of the first major key type is corresponding covers data-base recording corresponding with the first major key type from database with type; Search from the difference list of primary keys obtained after executing N above-mentioned steps comprised by the data-base recording in master data base but not by the major key comprised from the data-base recording in database, be designated as the major key of the second major key type, data-base recording corresponding with the major key of the second major key type in master data base is copied to from database; Search from the difference list of primary keys obtained after executing N above-mentioned steps by comprise from the data-base recording in database but not by major key that the data-base recording in master data base comprises, be designated as the 3rd major key type, and delete the data-base recording corresponding with the major key of the 3rd major key type from from database.
In above-mentioned preferred implementation, using difference list of primary keys as an intermediate quantity and tracing object, data repeatedly in the master and slave database of comparison, by have in difference list of primary keys but currently in master and slave database, recover consistent major key record successively screen out, thus the inaccurate technical matters of the result efficiently solving the consistency desired result caused because of synchronization delay in correlation technique, reach the technique effect improving master-slave database verification accuracy.
In a preferred embodiment, as shown in Figure 4, the conforming dynamic corrector of above-mentioned master-slave database can also comprise:
Initialization unit 402, be coupled with updating block 202, for before repeating above-mentioned steps N time, generate the initial value of difference list of primary keys, wherein, generate initialized difference list of primary keys to comprise: multiple tasks of executing the task concurrently in queue are to scan master data base and from the data-base recording database, to obtain the initial value of difference list of primary keys, wherein, each task comprises following operation, wherein, the initial value of task minimum key value is the minimum value in the major key that all data-base recordings of detected data table in master data base are corresponding: the data-base recording scanning predetermined number of lines in master data base from the data-base recording corresponding with task minimum key value by major key order from small to large, by Current Scan in master data base to data-base recording be recorded as current master data base record set, obtain maximal value in the major key corresponding to data-base recording in current master data base record set as task maximum key assignments, create next task, using the task minimum key value of task maximum key assignments as next task, and the next task of establishment is put into task queue, scanning from database and value scope [task minimum key value, task maximum key assignments) in data-base recording corresponding to major key, by Current Scan from database to data-base recording be recorded as current from database record set, if the data-base recording that same major key is corresponding in current master data base record set concentrates corresponding data-base recording different from current from data-base recording, then this major key is recorded as the initial value member of difference list of primary keys, if there is corresponding data-base recording and at the current data-base recording that there is not correspondence of concentrating from data-base recording, then this major key be recorded as the initial value member of difference list of primary keys in same major key in current master data base record set, if same major key exists corresponding data-base recording current to concentrate from data-base recording and there is not corresponding data-base recording current master data base record set, then this major key is recorded as the initial value member of difference list of primary keys, the set initial value member of difference list of primary keys formed stores in a list, to form the initial value of difference list of primary keys.
In the operation performed by above-mentioned initialization unit 402, by the generation to computer task, distribute and perform, achieve the initialization to difference list of primary keys.Alternatively, can sort to master data base before this task of execution, such as, be arranged in order by the key assignments size of major key corresponding with it, and then, therefrom can obtain the major key that key assignments is minimum, be designated as minimum key value and minimum major key.This minimum key value can as execution parameter, be input in the lump in task with the predetermined number of scanning lines, thus can be initial with minimum major key, perform the scanning to the master data base record in this predetermined number of lines, maximum major key is designated as by maximum for key assignments in major key corresponding with it one, and to the whole major keys of key assignments between minimum major key and largest key value master and slave data-base recording corresponding respectively contrast, the major key corresponding to the part that wherein there are differences is designated as the initial value of difference list of primary keys.Further, can also the largest key value that a upper task obtains be sent to as new minimum key value in next task, to realize the segmentation parallel initialization for full database.
Describe in detail below in conjunction with table 1 and table 2 and above-mentioned initialized implementation is carried out to difference list of primary keys, wherein, table 1 and table 2 are for representing the data-base recording for performing the initialized task of difference list of primary keys and scan for twice before and after same master-slave database, and this data-base recording sorts from small to large by Major key.Easy understand, Major key corresponding to this data-base recording might not be numerically continuous, such as, key assignments for subsequent use can be reserved between Major key, in order to the use etc. of newly-increased data-base recording, thus in a database, the Major key of data-base recording and corresponding major key thereof can be as shown in Table 1 and Table 2.
Table 1
For the task of the acquisition difference list of primary keys initial value generated first, its minimum key value can value be minimum key value in master data base, such as, and 001.Assuming that master data base corresponding to major key that key assignments is 001 is recorded as the 1st row, then for this task, for the number of scanning lines=5, in the execution of task, scanning is recorded as 5 initial line items with the master data base of key assignments corresponding to the major key of 001, namely the 1st of master data base record the walks to the 5th row, and be designated as master data base record set, as shown in table 1, the key assignments of its corresponding major key is followed successively by: 001,003,005,007 and 009, wherein, minimum key value is 001, and largest key value is 009.Thus, according to the embodiment of the present application, in the execution of task, key assignments will retrieved [001 from database further, 009) major key in this interval, namely 001,003,005,007, corresponding from data-base recording, and be designated as from database record set.Further, can at the master data base record set obtained and from carrying out line by line between database record set, the contrast of word for word section, thus find out the major key of data and the correspondence thereof that there are differences, the 2nd row that such as there are differences shown in table 1 and the key assignments corresponding to the 4th row data are the major key of 003 and 007, are recorded as the initial value of difference list of primary keys.
Table 2
For the task of again generating Next acquisition difference list of primary keys initial value, its minimum key value can value be the largest key value that last task obtains, such as, and the largest key value obtained in above-mentioned generating first of task, 009.Then for this task, equally for the number of scanning lines=5, in the execution of task, scanning is recorded as 5 initial line items with the master data base of key assignments corresponding to the major key of 009, namely the 5th of master data base record the walks to the 9th row, and be designated as master data base record set, as shown in table 2, the key assignments of its corresponding major key is followed successively by: 009, and 011,013,015 and 019, wherein, minimum key value is 009, and largest key value is 019.Thus, according to the embodiment of the present application, in the execution of task, key assignments will retrieved [009 from database further, 019) major key in this interval, namely 009,013,015,017, corresponding from data-base recording, and be designated as from database record set.Further, can at the master data base record set obtained and from carrying out line by line between database record set, the contrast of word for word section, thus find out the major key of data and the correspondence thereof that there are differences, be recorded as the initial value of difference list of primary keys.In the present embodiment, owing to there is data-base recording corresponding to major key that key assignments is 011 in master data base record set, and concentrate from data-base recording and there is not corresponding data-base recording, thus can be by key assignments 011 major key be recorded as the initial value member of difference list of primary keys; In addition, there is owing to concentrating from data-base recording data-base recording corresponding to major key that key assignments is 017, and in master data base record set, there is not corresponding data-base recording, thus can be by key assignments 017 major key be recorded as the initial value member of difference list of primary keys; And then the set that the initial value member of difference list of primary keys can be formed stores in a list, to form the initial value of difference list of primary keys.
Alternatively, above-mentioned task can also repeat to generate and perform repeatedly, until terminate the entire scan of master data base.Alternatively, in the execution of the task of above-mentioned acquisition difference list of primary keys initial value, can after acquisition largest key value, using this largest key value as new minimum key value to generate next task, this step can be carried out before comparing the content of master and slave data-base recording, thus the overall time that the initialization shortening a series of difference list of primary keys of carrying out for full database consumes.
As one embodiment alternatively, for the judgement of the scan task of the end stage of the list of primary keys of master data base, following operation can be performed realize by initialization unit 402: if the data volume that the scanning carrying out predetermined number of lines to master data base retrieves, or the data volume in master data base record set is less than the predetermined number of scanning lines, then think the end of the list of primary keys having scanned master data base, the not task of the acquisition difference list of primary keys initial value that regeneration is new.More alternatively, initialization unit 402 can also be used for performing following operation: 1) be greater than data-base recording corresponding to the major key of master library largest key value scanning key assignments from database; 2) if there is key assignments to be greater than data-base recording corresponding to the major key of master library largest key value, then this major key is recorded as the initial value member of difference list of primary keys.Similarly, initialization unit 402 can also be used for performing following operation: 1) be less than data-base recording corresponding to the major key of master library minimum key value scanning key assignments from database; 2) if there is key assignments to be less than data-base recording corresponding to the major key of master library minimum key value, then this major key is recorded as the initial value member of difference list of primary keys.
Pass through aforesaid operations, can by may exist from database with from key assignments corresponding to database data record in [master library minimum key value, master library largest key value] beyond major key be recorded in the initial value of difference list of primary keys, to realize the complete scan to master and slave database, thus the record of the difference list of primary keys of acquisition is not omitted.
In order to realize the parallel processing to data, to improve the efficiency of data processing, the mode that multi-threaded parallel can be adopted to perform processes data.In a preferred embodiment, record in described master and slave database is scanned with calling multiple thread parallel, to obtain the initial value of difference list of primary keys, wherein, each in described multiple thread is responsible for execution task, namely in described master data base, from the minimum major key of task, the data of predetermined number of lines scan, in the implementation of the task of each scan type first, next scan task first can be generated as the case may be and the pending task queue such as to put into.
Below several task types in task are explained:
Task queue (Prepared Task List) etc. pending: the task to be performed such as to deposit, the task in this queue performs according to the order of first in first out;
Active task queue (Active Task List): deposit performing of task, task in the queue executes just removing of task.
Scheduling thread is needed to perform for these tasks, if dispatch the executed in parallel that multiple thread just can realize task simultaneously.If etc. there being task in pending task list, and available free tasks carrying thread, so just from etc. take out a task pending task queue, put into active task queue, be then distributed to a tasks carrying thread;
Above by thread parallel process to obtain the mode of difference list of primary keys can be: if etc. there is no task in pending task list, task (namely one take turns inspection terminate) is not had in active task list, and do not checked the wheel number of specifying, and difference list of primary keys is not empty, then show to there is discrepant major key in two databases, so just using current difference list of primary keys as initialized difference list of primary keys.
And then create several new tasks, task type is for again to scan, major key in all differences list of primary keys is assigned in new task, newly-built task is all joined in task list, carry out the comparison of master-slave database data record corresponding to difference list of primary keys.Foundation this batch again after scan task, scheduling thread dormancy M second, then continuation scheduler task execution.Alternatively, the dormancy time M second of scheduling thread between two-wheeled task is a Configuration Values, is defaulted as 5 seconds.Type be again scanning of task be all performed complete after, if also do not reach the execution wheel number of specifying, and difference list of primary keys is not empty, then as above-mentioned steps, again create several new tasks, put into task waiting list, scheduling thread dormancy S second again, a new wheel scan starts again.
If etc. there is no task in pending task list, task is not had in active task list yet, and complete the wheel number (being finally determined with variance data) that checked and specified or difference list of primary keys for empty (final judge indifference data), then generated and output verification summary report.
In a preferred embodiment, maximum scan wheel number N and adjacent M interval time performed between above-mentioned steps for twice is specified by input configuration in advance.Alternatively, the span of M is [2,10], and the span of N is [1,10].Alternatively, the dormancy time M in the middle of every two wheel scans can be a dynamic random value, as a random number between [2,10].
Embodiment 2
On the basis of the system shown in Fig. 2-Fig. 4, this application provides a kind of preferred conforming dynamic check method of master-slave database.As shown in Figure 5, the method can comprise the steps:
Step S502: repeat following steps N time, wherein, N is the natural number preset: the data-base recording that retrieval is corresponding with the major key in difference list of primary keys from master data base, to obtain master data base record set; From the data-base recording that retrieval from database is corresponding with the major key in described difference list of primary keys, to obtain from database record set; Obtain described master data base record set and describedly concentrate the major key corresponding to data-base recording that there are differences from data-base recording, the major key corresponding to data-base recording that there are differences described in being updated to by the major key in described difference list of primary keys;
Step S504: after executing N described step, if record difference major key in described difference list of primary keys, then judges described master data base and described inconsistent from database, otherwise, then judge that described master data base is consistent from database with described.
Alternatively, in single job in step S502, can according to the first group of major key recorded in the difference list of primary keys of this operation incipient stage, respectively in master data base and from database, retrieve the data-base recording corresponding with first group of major key, and represent with master data base record set and from database record set respectively, then can contrast at master data base record set and between database record set, find out corresponding to part the second group of major key that wherein there are differences, and first group of major key is replaced with second group of major key in difference list of primary keys, then this operation is terminated, thus complete once to the renewal of difference list of primary keys.
It should be noted that the quantity of major key in first group of major key can for one or more, it is one or more also can be that zero, the second group of major key can comprise in first group of major key, also can not comprise wherein any one; Second group of major key is the subset of first group of major key.On the other hand, the determination of second group of major key can be carried out in above-mentioned comparison process, also can carry out after above-mentioned comparison process terminates.In addition, the description that first group of major key replaced by above-mentioned use second group of major key should be understood to describe for the one of the embodiment of the present application, can't form any restriction to the application, and such as, shown in Fig. 6, this result can also realize via following step:
S602: obtain the 3rd major key group in the first major key group, wherein, the 3rd major key group is consistent with data-base recording content corresponding from database in master data base;
S604: the 3rd major key group is deleted from difference list of primary keys.
And then in step S502, obtain master data base record set and the major key concentrated from data-base recording corresponding to the data-base recording that there are differences, the major key difference list of primary keys be updated to this step of major key corresponding to data-base recording that there are differences, multiple implementation can also be had, and be not limited only to mention in the embodiment of the present application, such as, examination can also be carried out one by one, to realize the renewal to the major key in difference list of primary keys for the major key in diff list.
Under above-mentioned scene, by to major key a limited number of in difference list of primary keys and corresponding data-base recording thereof but not the process of whole master and slave database, can reduce and the consuming time of the single sense cycle of consistency desired result is carried out to master-slave database, thus the conforming dynamic check carrying out master-slave database when not stopping externally service for database and place system thereof provides technical foundation.On the other hand, because single detects shortening consuming time, improve the ageing of master-slave database consistency desired result, reduce the error rate of the consistance judgement caused due to time delay, thus improve the accuracy of check results.
Further, single job in above-mentioned step S502 can perform again: in new single job, the major key recorded in the difference list of primary keys of incipient stage is second group of major key, thereafter, respectively in master data base and retrieve the data-base recording corresponding with second group of major key from database, find out the corresponding to part the 4th group of major key that wherein there are differences, and second group of major key is replaced with the 4th group of major key in difference list of primary keys, then terminate this operation, thus complete the renewal again to difference list of primary keys.
Under above-mentioned scene, judge to have higher confidence level and lower False Rate according to step S504 to the master-slave database consistance that the difference major key recorded in the difference list of primary keys obtained after above-mentioned twice operation i.e. the 4th major key group is carried out, details are as follows for its cardinal principle:
Imagination server has made amendment in the first moment to the data-base recording corresponding to a certain major key in master data base, and as the synchronous repair action with this amendment, server has made corresponding amendment in the second moment to the data-base recording from database corresponding to this major key, and in single job in above-mentioned steps S502, the difference major key i.e. acquisition of second group of major key is completed between the first moment and the second moment, this then corresponding with the data-base recording of change major key will be credited in second group of major key, even if the master-slave database record subsequently corresponding to this major key of the second moment achieves synchronously, thus there is the conforming false judgment that time delay causes.But this mistake can be eliminated by the above-mentioned second time operation carried out after the preset lime, such as, complete after in second time operation, the difference major key i.e. acquisition of the 4th group of major key was arranged on for the second moment, then the above-mentioned major key corresponding with the data-base recording of change can not be logged in the 4th group of major key, thus have higher confidence level and lower False Rate according to step S504 to the difference major key recorded in the difference list of primary keys i.e. result of the 4th group of consistance that major key is made judgement.
Further, above-mentioned single job can also repeat N time, to obtain higher confidence level and lower False Rate.Such as, for time of data delay on line the situation of about 1 second, if default configuration is every 5 seconds to variance data duplication check once, acquiescence is carried out 8 and is taken turns verification.Then only have very extreme situation, these data of each verification that just there will be in these 8 times verifications all changed, and just master-slave synchronisation does not also complete, and so these data will be mistaken for inconsistent data.But the probability that this situation occurs in production environment is minimum, and user can according to the variance data report exported, the manual anti-data looked in master-slave database again.Therefore, by the above-mentioned inventive concept of the application in actual production practice process, check results has very high confidence level and very low False Rate.
On the other hand, those skilled in the art is to be understood that, in the embodiment of the present application, it is non-essential that composition for the difference list of primary keys of above-mentioned single operation incipient stage carries out restriction, and such as, its composition can be the Output rusults of last same operation, also can be any a section in the overall major key of master-slave database, alternatively, as shown in Figure 7, can also be the initial value of the difference list of primary keys obtained via following steps:
S702: multiple tasks of executing the task concurrently in queue are to scan master data base and from the data-base recording database, to obtain the initial value of difference list of primary keys, wherein, each task comprises following operation, wherein, the initial value of task minimum key value is the minimum value in the major key that all data-base recordings of detected data table in master data base are corresponding:
S704: press the data-base recording that major key order from small to large scans predetermined number of lines in master data base from the data-base recording corresponding with task minimum key value, by Current Scan in master data base to data-base recording be recorded as current master data base record set, obtain maximal value in the major key corresponding to data-base recording in current master data base record set as task maximum key assignments, create next task, using the task minimum key value of task maximum key assignments as next task, and the next task of establishment is put into task queue;
S706: scanning from database and value scope [task minimum key value, task maximum key assignments) in data-base recording corresponding to major key, by Current Scan from database to data-base recording be recorded as current from database record set;
S708: at current master data base record set and currently to compare between database record set, and perform following operation according to the result compared:
S710: if the data-base recording of same major key correspondence in current master data base record set concentrates corresponding data-base recording different from current from data-base recording, then this major key is recorded as the initial value member of difference list of primary keys;
S712: if same major key exists corresponding data-base recording and at the current data-base recording that there is not correspondence of concentrating from data-base recording, then this major key is recorded as the initial value member of difference list of primary keys in current master data base record set;
S714: if same major key exists corresponding data-base recording current to concentrate from data-base recording and there is not corresponding data-base recording current master data base record set, then this major key is recorded as the initial value member of difference list of primary keys;
S716: the set initial value member of difference list of primary keys formed stores in a list, to form the initial value of difference list of primary keys.
In above-mentioned steps, by the generation to computer task, distribute and perform, achieve the initialization to difference list of primary keys.Alternatively, can sort to the major key of master data base detected data table before this task of execution, such as, be arranged in order by the key assignments size of major key corresponding with it, and then, therefrom can obtain the major key that key assignments is minimum, be designated as minimum key value and minimum major key.This minimum key value can as execution parameter, be input in the lump in task with the predetermined number of scanning lines, thus can be initial with minimum major key, press major key order from small to large, perform the scanning to the master data base record in this predetermined number of lines, maximum for key assignments in major key corresponding with it one is designated as the maximum major key of this task, and to the whole major keys of key assignments between minimum major key and largest key value master and slave data-base recording corresponding respectively contrast, the major key corresponding to the part that wherein there are differences is designated as the initial value of difference list of primary keys.Further, can also the largest key value that a upper task obtains be sent to as new minimum key value in next task, to realize the segmentation initialization for full database.
Describe in detail below in conjunction with the table 1 provided in the embodiment of the present application before and table 2 and above-mentioned initialized implementation is carried out to difference list of primary keys, wherein, table 1 and table 2 are for representing the data-base recording for performing the initialized task of difference list of primary keys and scan for twice before and after same master-slave database, and this data-base recording sorts from small to large by Major key.Easy understand, Major key corresponding to this data-base recording might not be numerically continuous, such as, key assignments for subsequent use can be reserved between Major key, in order to the use etc. of newly-increased data-base recording, thus in a database, the Major key of data-base recording and corresponding major key thereof can be as shown in Table 1 and Table 2.
For the task of the acquisition difference list of primary keys initial value generated first, its minimum key value can value be minimum key value in master data base, such as, and 001.Assuming that master data base corresponding to major key that key assignments is 001 is recorded as the 1st row, then for this task, for the number of scanning lines=5, in the execution of task, scanning is recorded as 5 initial line items with the master data base of key assignments corresponding to the major key of 001, namely the 1st of master data base record the walks to the 5th row, and be designated as master data base record set, as shown in table 1, the key assignments of its corresponding major key is followed successively by: 001,003,005,007 and 009, wherein, minimum key value is 001, and largest key value is 009.Thus, according to the embodiment of the present application, in the execution of task, key assignments will retrieved [001 from database further, 009) major key in this interval, namely 001,003,005,007, corresponding from data-base recording, and be designated as from database record set.Further, can at the master data base record set obtained and from carrying out line by line between database record set, the contrast of word for word section, thus find out the major key of data and the correspondence thereof that there are differences, the 2nd row that such as there are differences shown in table 1 and the key assignments corresponding to the 4th row data are the major key of 003 and 007, are recorded as the initial value of difference list of primary keys.
For the task of the acquisition difference list of primary keys initial value again generated, its minimum key value can value be the largest key value that last task obtains, such as, and the largest key value obtained in above-mentioned generating first of task, 009.Then for this task, equally for the number of scanning lines=5, in the execution of task, scanning is recorded as 5 initial line items with the master data base of key assignments corresponding to the major key of 009, namely the 5th of master data base record the walks to the 9th row, and be designated as master data base record set, as shown in table 2, the key assignments of its corresponding major key is followed successively by: 009, and 011,013,015 and 019, wherein, minimum key value is 009, and largest key value is 019.Thus, according to the embodiment of the present application, in the execution of task, key assignments will retrieved [009 from database further, 019) major key in this interval, namely 009,013,015,017, corresponding from data-base recording, and be designated as from database record set.Further, can at the master data base record set obtained and from carrying out line by line between database record set, the contrast of word for word section, thus find out the major key of data and the correspondence thereof that there are differences, be recorded as the initial value of difference list of primary keys.In the present embodiment, owing to there is data-base recording corresponding to major key that key assignments is 011 in master data base record set, and concentrate from data-base recording and there is not corresponding data-base recording, thus can be by key assignments 011 major key be recorded as the initial value member of difference list of primary keys; In addition, there is owing to concentrating from data-base recording data-base recording corresponding to major key that key assignments is 017, and in master data base record set, there is not corresponding data-base recording, thus can be by key assignments 017 major key be recorded as the initial value member of difference list of primary keys; And then the set that the initial value member of difference list of primary keys can be formed stores in a list, to form the initial value of difference list of primary keys.
Alternatively, above-mentioned task can also repeat to generate and perform repeatedly, until terminate the entire scan of master data base.Alternatively, in the execution of the task of above-mentioned acquisition difference list of primary keys initial value, can after this task maximum key assignments of acquisition, using this largest key value as new minimum key value to generate next task, this step can be carried out before comparing the content of master and slave data-base recording, thus the overall time that the initialization shortening a series of difference list of primary keys of carrying out for full database consumes.
Alternatively, as shown in Figure 10, can the multiple tasks generated be put in task queue, perform the plurality of task in the following manner:
S1002: judge whether there is pending task in task queue;
S1004: if exist, then judge current that performing concurrently, whether be less than predetermined maximum thread for the quantity of the thread of executing the task;
S1006: be just less than predetermined maximum thread in the quantity of the thread of executed in parallel if current, then take out pending task from task queue, and call an idle thread to perform pending task;
S1008: be just equal to or greater than predetermined maximum thread in the quantity of the thread of executed in parallel if current, then pending task continued to be retained in task queue.
As the optional embodiment of one, for the judgement of the scan task of the end stage of the list of primary keys of master data base, can be realized by following operation:
S802: if the data volume that the scanning carrying out predetermined number of lines to master data base retrieves, or the data volume in master data base record set is less than the predetermined number of scanning lines, then think the end of the list of primary keys having scanned master data base, the not task of the acquisition difference list of primary keys initial value that regeneration is new.
Alternatively, as shown in Figure 8, after step S802, according to the largest key value obtained in S802 and master library largest key value, following operation can also be performed:
S804: be greater than data-base recording corresponding to the major key of master library largest key value scanning key assignments from database;
S806: if there is key assignments to be greater than data-base recording corresponding to the major key of master library largest key value, then this major key is recorded as the initial value member of difference list of primary keys.
Alternatively, as shown in Figure 9, after the minimum key value obtained in master data base and master library minimum key value, the step obtaining the initial value of difference list of primary keys can also comprise:
S902: be less than data-base recording corresponding to the major key of master library minimum key value scanning key assignments from database;
S904: if there is key assignments to be less than data-base recording corresponding to the major key of master library minimum key value, then this major key is recorded as the initial value member of difference list of primary keys.
Pass through aforesaid operations, can by may exist from database with from key assignments corresponding to database data record in [master library minimum key value, master library largest key value] beyond major key be recorded in the initial value of difference list of primary keys, to realize the complete scan to master and slave database, thus the record of the difference list of primary keys of acquisition is not omitted.
Alternatively, in the above-described embodiments for scanning and the search operaqtion of database, can be realized by SQL statement, such as, can be realized by following statement for the above-mentioned data-base recording scanning predetermined number of lines by major key order from small to large in master data base from the data-base recording corresponding with task minimum key value:
SELECT*FROM?TABLE_NAME?WHERE?PK>=${MIN_PK}ORDER?BY?PK?ASC?LIMIT${MAX_COUNT}；
And for above-mentioned retrieve from database key assignments [minimum key value, largest key value) major key in this interval, can be realized by following statement:
SELECT*FROM?TABLE_NAME?WHERE?PK>=${MIN_PK}AND?PK<${MAX_PK}ORDER?BY?PK?ASC；
Wherein, MIN_PK is for representing minimum key value, and MAX_PK is for representing largest key value, and MAX_COUNT is for representing the predetermined number of scanning lines.
Certainly, in above-described embodiment, the obtain manner of difference list of primary keys is only exemplarily occurred, any restriction can't be formed to the application, such as, it is non-essential that the data-base recording of above-mentioned key value size to major key and correspondence thereof carries out sequence, also the data-base recording of last figure to list of primary keys and correspondence thereof of key value division can be made, etc.
By the operation of the initial value of the acquisition difference list of primary keys shown in above-described embodiment, can realize master, data record inconsistent from database carries out complete, screen rapidly, and can the major key that there are differences be recorded in a series of difference list of primary keys without omission ground in batches, thus each difference list of primary keys scale is reduced, be convenient to the execution of subsequent check step, and can perform concurrently in computing machine or processor for the verification task that above-mentioned a series of difference list of primary keys is carried out, thus significantly shorten the overall time of the consistency desired result for master-slave database.
Alternatively, after being judged by the consistance of above-mentioned embodiment to master-slave database, if there is the situation that the two is inconsistent, then can perform it further and repair operation, alternatively, can comprise the steps:
Search simultaneously by described master data base and the described major key comprised from the data-base recording database from the described difference list of primary keys obtained after executing N described step, be designated as the major key of the first major key type, using in described master data base is that the data-base recording that the major key of described first major key type is corresponding covers described corresponding with described first major key type from database data-base recording with type;
Search from the described difference list of primary keys obtained after executing N described step comprised by the data-base recording in described master data base but not by the described major key comprised from the data-base recording database, be designated as the major key of the second major key type, data-base recording corresponding with the major key of described second major key type in described master data base copied to described from database;
Search from the described difference list of primary keys obtained after executing N described step by described from the data-base recording database comprise but not by major key that the data-base recording in described master data base comprises, be designated as the 3rd major key type, and from database, delete the data-base recording corresponding with the major key of described 3rd major key type from described.
Alternatively, what also just determine between master-slave database in above-mentioned steps S504 is inconsistent, the data of certain mode to master-slave database so just can be adopted to upgrade to realize the consistance of data in reparation two databases, in a preferred embodiment, what can realize data in such a way is synchronous: select from described master data base with execute N described step after master data base exists in described difference list of primary keys major key, to obtain the 5th major key group; From described delete from database with execute N described step after all data corresponding to difference major key in described difference list of primary keys, and from database described in the major key in described 5th major key group is copied to.Namely, deletion action is carried out than the data had more in master data base for from database, for have in master data base and from database non-existent data, then from master data base, copy these data to from database, thus realize the synchronous of data in master-slave database.
In the conforming dynamic check method of the master-slave database according to the embodiment of the present application, in order to realize the parallel processing to data, to improve the efficiency of data processing, the mode that multi-threaded parallel can be adopted to perform processes data.In a preferred embodiment, from master data base, obtain the major key being more than or equal to the minimum major key of task comprise: scan the major key in described master data base with calling multiple thread parallel, to obtain the data record being more than or equal to the minimum major key of described task, wherein, each in described multiple thread scans the data record of the predetermined number of lines in described master data base from the minimum major key of this task.
Below several task types in task are explained:
Task queue (Prepared Task List) etc. pending: the task to be performed such as to deposit, the task in this queue performs according to the order of first in first out;
Active task queue (Active Tastk List): deposit performing of task, task in the queue executes just removing of task.
Scheduling thread is needed to perform for these tasks, if dispatch the executed in parallel that multiple thread just can realize task simultaneously.If etc. there being task in pending task list, and available free tasks carrying thread, so just from etc. take out a task pending task queue, put into active task queue, be then distributed to a tasks carrying thread;
Above by thread parallel process to obtain the mode of difference list of primary keys can be: if etc. there is no task in pending task list, task (namely one take turns inspection terminate) is not had in active task list, and do not checked the wheel number of specifying, and difference list of primary keys is not empty, then show to there is discrepant major key in two databases, so just using current difference list of primary keys as initialized difference list of primary keys.
And then creating several new tasks, the major key in all differences list of primary keys, for again to scan, is assigned in new task by task type, and newly-built task is all joined in task list, carries out the comparison of difference list of primary keys and master-slave database.Thus perform repeatedly these class new tasks, alternatively, every 5 seconds can be defaulted as and perform once.
If etc. there is no task in pending task list, task is not had in active task list yet, and complete the wheel number (being finally determined with variance data) that checked and specified or difference list of primary keys for empty (final judge indifference data), then generated and output verification summary report.
In a preferred embodiment, M interval time between N and adjacent twice execution described step S502 to step S504 is configured by input or configuration file in advance.Alternatively, perform an above-mentioned steps S502 to step S504 second every M, wherein, the span of M is [2,10], and the span of N is [1,10].
The master-slave database conforming dynamic check method and apparatus of the application possesses following advantage, and certainly, the arbitrary product implementing the application might not need to reach above-described all advantages simultaneously:
1) several processes are started by configuration, each process scans a pair master data base table of the same name and from database table, only need the minimum major key reading master data base table, first scan task can be created, each first run scan task, can judge whether to create next first run scan task according to the data volume read; Do not need in whole data scanning process to know that overall data scale carries out task again and disassembles in advance;
2) assigning thread (Dispatcher Thread) is after every wheel scan task terminates, and judge whether the maximum scan wheel number reaching configuration, whether global disparity data list of primary keys is empty, judges whether the scan task setting up next round;
3) scan task is that multi-threaded parallel performs;
4) master data base and from database data be all by the sequence carried out of major key order, read same area respectively, namely identical minimum major key or maximum major key (first run scanning), or identical major key (again scan), make Data Comparison become very easily and rapidly;
5), when having scanned the most bull wheel number of configuration requirement, when global disparity data list of primary keys is not empty, every bar variance data is exported;
6) in whole scanning process, database loads is controlled, because adopting following several SQL statement batch to read data, can have more batch size and reading frequency that configuration controls reading:
SELECT*FROM?TABLE_NAME?WHERE?PK>${MIN_PK}ORDER?BY?PK?ASC?LIMIT${MAX_COUNT}；
SELECT*FROM?TABLE_NAME?WHERE?PK>${MIN_PK}AND?PK<=${MAX_PK}ORDER?BY?PK?ASC；
SELECT*FROM?TABLE_NAME?WHERE?PK?in(a,b,c…)ORDER?BY?PK?ASC；
7) whole checking procedure is dynamic check, not needing master-slave database to stop externally service, even the stopping of moment is externally served, such as, without the need to pinning database, or pinning tables of data.
Known by above description, the application achieves following technique effect:
1) using difference list of primary keys as an intermediate quantity and tracing object, data repeatedly in the master and slave database of comparison, by have in difference list of primary keys but currently in master and slave database, recovered consistent major key record successively screened out, thus the inaccurate technical matters of the result efficiently solving the consistency desired result caused because of synchronization delay in correlation technique, reach the technique effect improving master-slave database verification accuracy;
2) by the division to the data record in database, scan to generate parallel multiple tasks, and carry out the step of follow-up consistency desired result, thus reach the technique effect shortening checking time, on the other hand, reduce the requirement to processor performance, achieve the effective utilization to processing power, thus meet the requirement of not lock storehouse in consistency desired result, not locking table, achieve the effect of carrying out detection of dynamic when database and tables of data are in normal read-write state;
3) by the process to the accurate result of master-slave database consistency desired result, technical scheme can also realize the effective reparation to the data that there are differences in master and slave database, thus reaches the dynamically consistent effect of master-slave database.
Obviously, those skilled in the art should be understood that, each module of above-mentioned the application or each step can realize with general calculation element, they can concentrate on single calculation element, or be distributed on network that multiple calculation element forms, alternatively, they can realize with the executable program code of calculation element, thus, they can be stored and be performed by calculation element in the storage device, and in some cases, step shown or described by can performing with the order be different from herein, or they are made into each integrated circuit modules respectively, or the multiple module in them or step are made into single integrated circuit module to realize.Like this, the application is not restricted to any specific hardware and software combination.
The foregoing is only the preferred embodiment of the application, be not limited to the application, for a person skilled in the art, the application can have various modifications and variations.Within all spirit in the application and principle, any amendment done, equivalent replacement, improvement etc., within the protection domain that all should be included in the application.

Claims (10)

1. the conforming dynamic check method of master-slave database, is characterized in that, comprising:
Repeat following steps N time, wherein, N is the natural number preset: the data-base recording that retrieval is corresponding with the major key in difference list of primary keys from master data base, to obtain master data base record set; From the data-base recording that retrieval from database is corresponding with the major key in described difference list of primary keys, to obtain from database record set; Obtain described master data base record set and describedly concentrate the major key corresponding to data-base recording that there are differences from data-base recording, the major key corresponding to data-base recording that there are differences described in being updated to by the major key in described difference list of primary keys;
After executing N described step, if still have major key in described difference list of primary keys, then judge described master data base and described inconsistent from database, otherwise, judge that described master data base is consistent from database with described.
2. method according to claim 1, is characterized in that, judging described master data base and described from after database is inconsistent, also comprise: perform data restore operation, wherein, described data restore operation comprises:
Search simultaneously by described master data base and the described major key comprised from the data-base recording database from the described difference list of primary keys obtained after executing N described step, be designated as the major key of the first major key type, using in described master data base is that the data-base recording that the major key of described first major key type is corresponding covers described corresponding with described first major key type from database data-base recording with type;
Search from the described difference list of primary keys obtained after executing N described step comprised by the data-base recording in described master data base but not by the described major key comprised from the data-base recording database, be designated as the major key of the second major key type, data-base recording corresponding with the major key of described second major key type in described master data base copied to described from database;
Search from the described difference list of primary keys obtained after executing N described step by described from the data-base recording database comprise but not by major key that the data-base recording in described master data base comprises, be designated as the 3rd major key type, and from database, delete the data-base recording corresponding with the major key of described 3rd major key type from described.
3. method according to claim 1, is characterized in that, obtains described master data base record set and the described major key corresponding to data-base recording that there are differences of concentrating from data-base recording comprises:
If the data-base recording that same major key is corresponding in described master data base record set concentrates corresponding data-base recording different from described from data-base recording, then the major key corresponding to data-base recording that there are differences described in being recorded as by this major key;
If there is corresponding data-base recording and there is not corresponding data-base recording, then the major key corresponding to data-base recording that there are differences described in being recorded as by this major key described to concentrate from data-base recording in same major key in described master data base record set;
If there is corresponding data-base recording and there is not corresponding data-base recording described master data base record set, then the major key corresponding to data-base recording that there are differences described in being recorded as by this major key in same major key described to concentrate from data-base recording.
4. method according to claim 1, it is characterized in that, before repeating described step N time, also comprise: the initial value generating described difference list of primary keys, wherein, described generation initialized difference list of primary keys comprises: multiple tasks of executing the task concurrently in queue are to scan described master data base and described from the data-base recording database, to obtain the initial value of described difference list of primary keys, wherein, each described task comprises following operation, wherein, the initial value of task minimum key value is the minimum value in the major key that all data-base recordings of detected data table in described master data base are corresponding:
From the data-base recording corresponding with described task minimum key value, the data-base recording that major key order from small to large scans predetermined number of lines is pressed in described master data base, by Current Scan in described master data base to data-base recording be recorded as current master data base record set, obtain maximal value in the major key corresponding to data-base recording in described current master data base record set as task maximum key assignments, create next task, using the task minimum key value of described task maximum key assignments as described next task, and the described next task created is put into described task queue,
Described from database scanning and value scope [described task minimum key value, described task maximum key assignments) in data-base recording corresponding to major key, by described from database Current Scan to data-base recording be recorded as current from database record set;
If the data-base recording corresponding in described current master data base record set of same major key currently concentrates corresponding data-base recording different from data-base recording from described, then this major key is recorded as the initial value member of described difference list of primary keys;
If there is corresponding data-base recording and at the described current data-base recording that there is not correspondence of concentrating from data-base recording, then this major key be recorded as the initial value member of described difference list of primary keys in same major key in described current master data base record set;
If same major key exists corresponding data-base recording described current to concentrate from data-base recording and there is not corresponding data-base recording described current master data base record set, then this major key is recorded as the initial value member of described difference list of primary keys;
The set initial value member of described difference list of primary keys formed stores in a list, to form the initial value of described difference list of primary keys.
5. method according to claim 4, is characterized in that, puts in task queue, perform described multiple task in the following manner by the described multiple task generated:
Judge whether there is pending task in described task queue;
If exist, then judge current that performing concurrently, whether be less than predetermined maximum thread for the quantity performing the thread of described task;
Currently just be less than predetermined maximum thread in the quantity of the thread of executed in parallel if described, then from described task queue, take out described pending task, and call an idle thread to perform described pending task; Currently just be equal to or greater than described predetermined maximum thread in the quantity of the thread of executed in parallel if described, then described pending task continued to be retained in described task queue.
6. method according to claim 1, is characterized in that, in advance by M interval time between configuration file configuration N and the described step of adjacent twice execution.
7. method according to claim 1, is characterized in that, performs once described step second every M, and wherein, the value of M is fixed value in [2,10] or random value, and the span of N is [1,10].
8. the conforming dynamic corrector of master-slave database, is characterized in that, comprising:
Updating block, for repeating following steps N time, wherein, N is the natural number preset: the data-base recording that retrieval is corresponding with the major key in difference list of primary keys from master data base, to obtain master data base record set; From the data-base recording that retrieval from database is corresponding with the major key in described difference list of primary keys, to obtain from database record set; Obtain described master data base record set and describedly concentrate the major key corresponding to data-base recording that there are differences from data-base recording, the major key corresponding to data-base recording that there are differences described in being updated to by the major key in described difference list of primary keys;
Judging unit, after executing N described step, if still have major key in described difference list of primary keys, then judges described master data base and described inconsistent from database, otherwise, judge that described master data base is consistent from database with described.
9. device according to claim 8, is characterized in that, also comprises:
Repair unit, for judging described master data base and described from after database is inconsistent, execution data restore operates, wherein, the operation of described data restore comprises: by described master data base and the described major key comprised from the data-base recording database while of searching from the described difference list of primary keys obtained after executing N described step, be designated as the major key of the first major key type, use in described master data base is that the data-base recording that the major key of described first major key type is corresponding covers described corresponding with described first major key type from database data-base recording with type, search from the described difference list of primary keys obtained after executing N described step comprised by the data-base recording in described master data base but not by the described major key comprised from the data-base recording database, be designated as the major key of the second major key type, data-base recording corresponding with the major key of described second major key type in described master data base copied to described from database, search from the described difference list of primary keys obtained after executing N described step by described from the data-base recording database comprise but not by major key that the data-base recording in described master data base comprises, be designated as the 3rd major key type, and from database, delete the data-base recording corresponding with the major key of described 3rd major key type from described.
10. device according to claim 8, is characterized in that, also comprises:
Initialization unit, for before repeating described step N time, generate the initial value of described difference list of primary keys, wherein, described generation initialized difference list of primary keys comprises: multiple tasks of executing the task concurrently in queue are to scan described master data base and described from the data-base recording database, to obtain the initial value of described difference list of primary keys, wherein, each described task comprises following operation, wherein, the initial value of task minimum key value is the minimum value in the major key that all data-base recordings of detected data table in described master data base are corresponding: the data-base recording scanning predetermined number of lines in described master data base from the data-base recording corresponding with described task minimum key value by major key order from small to large, by Current Scan in described master data base to data-base recording be recorded as current master data base record set, obtain maximal value in the major key corresponding to data-base recording in described current master data base record set as task maximum key assignments, create next task, using the task minimum key value of described task maximum key assignments as described next task, and the described next task created is put into described task queue, described from database scanning and value scope [described task minimum key value, described task maximum key assignments) in data-base recording corresponding to major key, by described from database Current Scan to data-base recording be recorded as current from database record set, if the data-base recording corresponding in described current master data base record set of same major key currently concentrates corresponding data-base recording different from data-base recording from described, then this major key is recorded as the initial value member of described difference list of primary keys, if there is corresponding data-base recording and at the described current data-base recording that there is not correspondence of concentrating from data-base recording, then this major key be recorded as the initial value member of described difference list of primary keys in same major key in described current master data base record set, if same major key exists corresponding data-base recording described current to concentrate from data-base recording and there is not corresponding data-base recording described current master data base record set, then this major key is recorded as the initial value member of described difference list of primary keys, the set initial value member of described difference list of primary keys formed stores in a list, to form the initial value of described difference list of primary keys.
CN201310263694.3A 2013-06-27 2013-06-27 The dynamic check method and apparatus of master-slave database consistency Active CN104252481B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201310263694.3A CN104252481B (en) 2013-06-27 2013-06-27 The dynamic check method and apparatus of master-slave database consistency

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201310263694.3A CN104252481B (en) 2013-06-27 2013-06-27 The dynamic check method and apparatus of master-slave database consistency

Publications (2)

Publication Number Publication Date
CN104252481A true CN104252481A (en) 2014-12-31
CN104252481B CN104252481B (en) 2018-10-19

Family

ID=52187383

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310263694.3A Active CN104252481B (en) 2013-06-27 2013-06-27 The dynamic check method and apparatus of master-slave database consistency

Country Status (1)

Country Link
CN (1) CN104252481B (en)

Cited By (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104714785A (en) * 2015-03-31 2015-06-17 中芯睿智（北京）微电子科技有限公司 Task scheduling device, task scheduling method and data parallel processing device
CN106649814A (en) * 2016-12-29 2017-05-10 国网江苏省电力公司南京供电公司 Method for consistency checking of distribution automation cross-domain database
CN106649333A (en) * 2015-10-29 2017-05-10 阿里巴巴集团控股有限公司 Method and device for consistency testing of field sequence
CN107038414A (en) * 2017-03-08 2017-08-11 广东欧珀移动通信有限公司 A kind of fingerprint template synchronous method and equipment
CN107133249A (en) * 2016-02-29 2017-09-05 上海格尔软件股份有限公司 It is a kind of to having deleted the database synchronization method that data are synchronized
CN107656940A (en) * 2016-07-26 2018-02-02 北京国双科技有限公司 The treating method and apparatus of database
CN107819795A (en) * 2016-09-12 2018-03-20 北京金山云网络技术有限公司 A kind of major-minor server data synchronous method of DNS, apparatus and system
CN108287886A (en) * 2018-01-16 2018-07-17 阿里巴巴集团控股有限公司 The method and device of synchrodata modification information
CN108319719A (en) * 2018-02-12 2018-07-24 腾讯科技（深圳）有限公司 Database data method of calibration, device, computer equipment and storage medium
CN108322492A (en) * 2017-01-16 2018-07-24 医渡云（北京）技术有限公司 Medical data synchronous method and device
CN108694249A (en) * 2018-05-30 2018-10-23 平安科技（深圳）有限公司 Data processing method, device, computer equipment and storage medium
CN109241500A (en) * 2018-08-14 2019-01-18 中国平安人寿保险股份有限公司 Files in batch comparison method, device, computer equipment and storage medium
CN109271451A (en) * 2018-10-12 2019-01-25 武汉达梦数据库有限公司 A kind of extracting method of database external linkage
CN109284301A (en) * 2018-08-31 2019-01-29 阿里巴巴集团控股有限公司 Verification of data method and device
CN105630635B (en) * 2015-12-29 2019-07-30 北京京东尚科信息技术有限公司 Data disaster tolerance system and method
CN110209521A (en) * 2019-02-22 2019-09-06 腾讯科技（深圳）有限公司 Data verification method, device, computer readable storage medium and computer equipment
CN110222027A (en) * 2019-04-24 2019-09-10 福建天泉教育科技有限公司 The quantity method of calibration and computer readable storage medium of Data Migration
CN110399376A (en) * 2019-08-08 2019-11-01 北京明略软件系统有限公司 The method and device of the former data variation of automatic identification table
CN110737642A (en) * 2019-09-05 2020-01-31 平安科技（深圳）有限公司 Database information analysis method, database information analysis device, computer device and storage medium
CN110852058A (en) * 2019-09-24 2020-02-28 贵阳朗玛信息技术股份有限公司 List comparison method and device
WO2020057272A1 (en) * 2018-09-18 2020-03-26 厦门市美亚柏科信息股份有限公司 Index data storage and retrieval methods and apparatuses, and storage medium
CN110968584A (en) * 2019-12-03 2020-04-07 北京明略软件系统有限公司 Portrait generating system, method, electronic device and readable storage medium
CN110990414A (en) * 2019-10-31 2020-04-10 口碑(上海)信息技术有限公司 Data processing method and device
CN111125110A (en) * 2019-12-25 2020-05-08 深圳前海环融联易信息科技服务有限公司 Method and device for checking non-primary key field, computer equipment and storage medium
CN111198868A (en) * 2019-12-12 2020-05-26 北京淇瑀信息科技有限公司 Intelligent sub-database real-time data migration method and device
CN111339200A (en) * 2020-02-28 2020-06-26 中国平安人寿保险股份有限公司 Data verification method and data verification system based on database synchronization
CN111949728A (en) * 2020-07-14 2020-11-17 北京思特奇信息技术股份有限公司 Dynamic data difference comparison method and system
CN112579591A (en) * 2019-09-30 2021-03-30 重庆小雨点小额贷款有限公司 Data verification method and device, electronic equipment and computer readable storage medium
CN112905602A (en) * 2021-03-26 2021-06-04 掌阅科技股份有限公司 Data comparison method, computing device and computer storage medium
CN113177056A (en) * 2021-06-29 2021-07-27 企查查科技有限公司 Method, device and storage medium for checking consistency of service data of multiple databases
CN113297330A (en) * 2021-06-24 2021-08-24 维沃移动通信有限公司 Data synchronization detection method and device
CN113704359A (en) * 2021-09-03 2021-11-26 优刻得科技股份有限公司 Synchronization method, system and server for multiple data copies of time sequence database
CN113760910A (en) * 2021-08-31 2021-12-07 中国银联股份有限公司 Data synchronization method and device
CN113780902A (en) * 2021-11-12 2021-12-10 江西省国土空间调查规划研究院 Disaster risk early warning management system based on cloud computing
CN115905273A (en) * 2022-11-10 2023-04-04 贝壳找房(北京)科技有限公司 Data detection method in digital warehouse table, electronic equipment and storage medium
CN115952185A (en) * 2023-03-10 2023-04-11 布比（北京）网络技术有限公司 Data processing method and device, equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050223043A1 (en) * 2004-03-31 2005-10-06 Randal Paul S System and method for a consistency check of a database backup
CN101582920A (en) * 2009-06-09 2009-11-18 中兴通讯股份有限公司 Method and device for verifying and synchronizing data blocks in distributed file system
CN102521406A (en) * 2011-12-26 2012-06-27 中国科学院计算技术研究所 Distributed query method and system for complex task of querying massive structured data

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050223043A1 (en) * 2004-03-31 2005-10-06 Randal Paul S System and method for a consistency check of a database backup
CN1784677A (en) * 2004-03-31 2006-06-07 微软公司 System and method for a consistency check of a database backup
CN101582920A (en) * 2009-06-09 2009-11-18 中兴通讯股份有限公司 Method and device for verifying and synchronizing data blocks in distributed file system
CN102521406A (en) * 2011-12-26 2012-06-27 中国科学院计算技术研究所 Distributed query method and system for complex task of querying massive structured data

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
JYZHOU: ""percona-toolkit 之 【pt-table-checksum】、【pt-table-sync】说明"", 《HTTP://WWW.CNBLOGS.COM/ZHOUJINYI/ARCHIVE/2013/05/09/3067045.HTML?UTM_SOURCE=TUICOOL&UTM_MEDIUM=REFERRAL》 *
阿炯: ""MySQL主从数据一致性校验"", 《HTTP://WWW.FREEOA.NET/DEVELOPMENT/PERL/MYSQL-MASTER-SLAVE-CONSISTENCY-CHECK_1856.HTML》 *

Cited By (51)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104714785A (en) * 2015-03-31 2015-06-17 中芯睿智（北京）微电子科技有限公司 Task scheduling device, task scheduling method and data parallel processing device
CN106649333A (en) * 2015-10-29 2017-05-10 阿里巴巴集团控股有限公司 Method and device for consistency testing of field sequence
CN106649333B (en) * 2015-10-29 2021-12-10 阿里巴巴集团控股有限公司 Method and device for detecting consistency of field sequence
CN105630635B (en) * 2015-12-29 2019-07-30 北京京东尚科信息技术有限公司 Data disaster tolerance system and method
CN107133249A (en) * 2016-02-29 2017-09-05 上海格尔软件股份有限公司 It is a kind of to having deleted the database synchronization method that data are synchronized
CN107656940A (en) * 2016-07-26 2018-02-02 北京国双科技有限公司 The treating method and apparatus of database
CN107819795A (en) * 2016-09-12 2018-03-20 北京金山云网络技术有限公司 A kind of major-minor server data synchronous method of DNS, apparatus and system
CN106649814A (en) * 2016-12-29 2017-05-10 国网江苏省电力公司南京供电公司 Method for consistency checking of distribution automation cross-domain database
CN108322492A (en) * 2017-01-16 2018-07-24 医渡云（北京）技术有限公司 Medical data synchronous method and device
CN108322492B (en) * 2017-01-16 2021-09-17 医渡云（北京）技术有限公司 Medical data synchronization method and device
CN107038414A (en) * 2017-03-08 2017-08-11 广东欧珀移动通信有限公司 A kind of fingerprint template synchronous method and equipment
CN108287886A (en) * 2018-01-16 2018-07-17 阿里巴巴集团控股有限公司 The method and device of synchrodata modification information
CN108287886B (en) * 2018-01-16 2022-04-08 创新先进技术有限公司 Method and device for synchronizing data change information
CN108319719A (en) * 2018-02-12 2018-07-24 腾讯科技（深圳）有限公司 Database data method of calibration, device, computer equipment and storage medium
CN108319719B (en) * 2018-02-12 2022-04-01 腾讯科技（深圳）有限公司 Database data verification method and device, computer equipment and storage medium
CN108694249B (en) * 2018-05-30 2022-02-25 平安科技（深圳）有限公司 Data processing method, data processing device, computer equipment and storage medium
CN108694249A (en) * 2018-05-30 2018-10-23 平安科技（深圳）有限公司 Data processing method, device, computer equipment and storage medium
CN109241500B (en) * 2018-08-14 2023-02-03 中国平安人寿保险股份有限公司 File batch comparison method and device, computer equipment and storage medium
CN109241500A (en) * 2018-08-14 2019-01-18 中国平安人寿保险股份有限公司 Files in batch comparison method, device, computer equipment and storage medium
CN109284301A (en) * 2018-08-31 2019-01-29 阿里巴巴集团控股有限公司 Verification of data method and device
WO2020057272A1 (en) * 2018-09-18 2020-03-26 厦门市美亚柏科信息股份有限公司 Index data storage and retrieval methods and apparatuses, and storage medium
CN109271451A (en) * 2018-10-12 2019-01-25 武汉达梦数据库有限公司 A kind of extracting method of database external linkage
CN110209521A (en) * 2019-02-22 2019-09-06 腾讯科技（深圳）有限公司 Data verification method, device, computer readable storage medium and computer equipment
CN110222027A (en) * 2019-04-24 2019-09-10 福建天泉教育科技有限公司 The quantity method of calibration and computer readable storage medium of Data Migration
CN110399376A (en) * 2019-08-08 2019-11-01 北京明略软件系统有限公司 The method and device of the former data variation of automatic identification table
CN110737642B (en) * 2019-09-05 2022-09-02 平安科技（深圳）有限公司 Database information analysis method, database information analysis device, computer device and storage medium
CN110737642A (en) * 2019-09-05 2020-01-31 平安科技（深圳）有限公司 Database information analysis method, database information analysis device, computer device and storage medium
CN110852058A (en) * 2019-09-24 2020-02-28 贵阳朗玛信息技术股份有限公司 List comparison method and device
CN112579591A (en) * 2019-09-30 2021-03-30 重庆小雨点小额贷款有限公司 Data verification method and device, electronic equipment and computer readable storage medium
CN112579591B (en) * 2019-09-30 2023-06-16 重庆小雨点小额贷款有限公司 Data verification method, device, electronic equipment and computer readable storage medium
CN110990414A (en) * 2019-10-31 2020-04-10 口碑(上海)信息技术有限公司 Data processing method and device
CN110990414B (en) * 2019-10-31 2023-06-16 口碑(上海)信息技术有限公司 Data processing method and device
CN110968584B (en) * 2019-12-03 2023-03-14 北京明略软件系统有限公司 Portrait generation system, method, electronic device and readable storage medium
CN110968584A (en) * 2019-12-03 2020-04-07 北京明略软件系统有限公司 Portrait generating system, method, electronic device and readable storage medium
CN111198868A (en) * 2019-12-12 2020-05-26 北京淇瑀信息科技有限公司 Intelligent sub-database real-time data migration method and device
CN111198868B (en) * 2019-12-12 2023-09-05 北京淇瑀信息科技有限公司 Intelligent database-dividing real-time data migration method and device
CN111125110B (en) * 2019-12-25 2023-08-18 深圳前海环融联易信息科技服务有限公司 Non-primary key field checking method, device, computer equipment and storage medium
CN111125110A (en) * 2019-12-25 2020-05-08 深圳前海环融联易信息科技服务有限公司 Method and device for checking non-primary key field, computer equipment and storage medium
CN111339200A (en) * 2020-02-28 2020-06-26 中国平安人寿保险股份有限公司 Data verification method and data verification system based on database synchronization
CN111949728A (en) * 2020-07-14 2020-11-17 北京思特奇信息技术股份有限公司 Dynamic data difference comparison method and system
CN112905602A (en) * 2021-03-26 2021-06-04 掌阅科技股份有限公司 Data comparison method, computing device and computer storage medium
CN113297330A (en) * 2021-06-24 2021-08-24 维沃移动通信有限公司 Data synchronization detection method and device
CN113177056A (en) * 2021-06-29 2021-07-27 企查查科技有限公司 Method, device and storage medium for checking consistency of service data of multiple databases
CN113177056B (en) * 2021-06-29 2021-09-21 企查查科技有限公司 Method, device and storage medium for checking consistency of service data of multiple databases
CN113760910A (en) * 2021-08-31 2021-12-07 中国银联股份有限公司 Data synchronization method and device
CN113704359A (en) * 2021-09-03 2021-11-26 优刻得科技股份有限公司 Synchronization method, system and server for multiple data copies of time sequence database
CN113704359B (en) * 2021-09-03 2024-04-26 优刻得科技股份有限公司 Method, system and server for synchronizing multiple data copies of time sequence database
CN113780902A (en) * 2021-11-12 2021-12-10 江西省国土空间调查规划研究院 Disaster risk early warning management system based on cloud computing
CN115905273A (en) * 2022-11-10 2023-04-04 贝壳找房(北京)科技有限公司 Data detection method in digital warehouse table, electronic equipment and storage medium
CN115905273B (en) * 2022-11-10 2023-09-26 贝壳找房(北京)科技有限公司 Data detection method in digital bin table, electronic equipment and storage medium
CN115952185A (en) * 2023-03-10 2023-04-11 布比（北京）网络技术有限公司 Data processing method and device, equipment and storage medium

Also Published As

Publication number Publication date
CN104252481B (en) 2018-10-19

Similar Documents

Publication Publication Date Title
CN104252481A (en) Dynamic check method and device for consistency of main and salve databases
CN104423960B (en) A kind of method and system of project continuous integrating
CN113238924B (en) Chaotic engineering realization method and system in distributed graph database system
CN109063005B (en) Data migration method and system, storage medium and electronic device
CN108874678B (en) Automatic testing method and device for intelligent program
CN111309734B (en) Method and system for automatically generating table data
CN111382198B (en) Data recovery method, device, equipment and storage medium
CN110570646B (en) Four-remote signal acceptance method and system based on historical data
CN112306529A (en) System upgrading method, device, equipment and storage medium
CN116561003A (en) Test data generation method, device, computer equipment and storage medium
CN112346994B (en) Test information association method, device, computer equipment and storage medium
CN112486957B (en) Database migration detection method, device, equipment and storage medium
CN114356454A (en) Account checking data processing method, account checking data processing device, account checking data storage medium and program product
CN110177006B (en) Node testing method and device based on interface prediction model
CN112181825A (en) Test case library construction method and device, electronic equipment and medium
CN115250231B (en) Application configuration method and device
CN114692382B (en) Management method and device for nuclear power simulation model development data and computer equipment
CN117271367A (en) Configuration of application system test flow, application system test method and device
CN110175116B (en) Test date switching method, device and system
US10055297B1 (en) System, method, and computer program for smart database inflation
Pritz et al. Performance Impact of Parallel Access of Time Series in the Context of Relational, NoSQL and NewSQL Database Management Systems
CN114884799A (en) Method, system, equipment and storage medium for testing cluster alarm function
CN115017054A (en) Data synchronization test method and device, electronic equipment and storage medium
CN114201400A (en) Code debugging method and device, computer readable storage medium and server
CN117609054A (en) Automatic test method, device, equipment and storage medium

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20191210

Address after: P.O. Box 31119, grand exhibition hall, hibiscus street, 802 West Bay Road, Grand Cayman, Cayman Islands

Patentee after: Innovative advanced technology Co., Ltd

Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands

Patentee before: Alibaba Group Holding Co., Ltd.

TR01 Transfer of patent right