CN110209521A

CN110209521A - Data verification method, device, computer readable storage medium and computer equipment

Info

Publication number: CN110209521A
Application number: CN201910134147.2A
Authority: CN
Inventors: 吴双桥; 王珏; 杨繁
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Cloud Computing Beijing Co Ltd
Priority date: 2019-02-22
Filing date: 2019-02-22
Publication date: 2019-09-06
Anticipated expiration: 2039-02-22
Also published as: CN110209521B

Abstract

This application involves a kind of data verification method, device, computer readable storage medium and computer equipments, which comprises carries out verification to the first tables of data in primary database and the first checklist is written in check results；Hook program is executed, the second tables of data for corresponding to first tables of data from database is verified according to the binary log of the primary database, and the second checklist is written into check results；Compare first checklist and second checklist.Scheme provided by the present application has a wide range of application.

Description

Data verification method, device, computer readable storage medium and computer equipment

Technical field

This application involves field of computer technology, more particularly to a kind of data verification method, device, computer-readable deposit Storage media and computer equipment.

Background technique

With the development of computer technology and being skyrocketed through for data volume, database using more and more common.User exists During using database, the scene for needing to compare master-slave database consistency can be encountered often, for example repairing principal and subordinate Whether consistent need to confirm data after stationary problem, or whether verify after Data Migration migrating data correct etc..

However, current data check mode is usually to execute identical database manipulation language on master-slave database Then sentence compares the operation result respectively obtained on master-slave database, to verify master-slave database consistency.This mode is limited The principle of identical action statement is needed to be implemented in principal and subordinate library, the problem for causing the data check mode application range narrow.

Summary of the invention

Based on this, it is necessary to for the narrow technical problem of current data check mode application range, provide a kind of data Method of calibration, device, computer readable storage medium and computer equipment.

A kind of data verification method, comprising:

Verification is carried out to the first tables of data in primary database and the first checklist is written into check results；

Hook program is executed, according to the binary log of the primary database to corresponding to first data from database Second tables of data of table is verified, and the second checklist is written in check results；

Compare first checklist and second checklist.

A kind of data calibration device, comprising:

First correction verification module, for carrying out verification to the first tables of data in primary database and check results being written first Checklist；

Second correction verification module, for executing hook program, according to the binary log of the primary database to from database Second tables of data of middle correspondence first tables of data is verified, and the second checklist is written in check results；

Contrast module, for comparing first checklist and second checklist.

A kind of computer readable storage medium is stored with computer program, when the computer program is executed by processor, So that the processor executes the step of above-mentioned data verification method.

A kind of computer equipment, including memory and processor, the memory are stored with computer program, the calculating When machine program is executed by the processor, so that the step of processor executes above-mentioned data verification method.

Above-mentioned data verification method, device, computer readable storage medium and computer equipment, automatically in primary database The first tables of data verified, and by check results be written the first check table.Wherein, the binary log record of primary database Relevant information of the primary database when carrying out tables of data verification, in this way, hook program can be according to the binary system of primary database Log verifies the second tables of data of the first tables of data that primary database is corresponded to from database, and check results are written Second checklist, so that the check results of master library data consistency, pole can be obtained in the first check table of comparison and the second check table The earth has widened application range.

Detailed description of the invention

Fig. 1 is the applied environment figure of data verification method in one embodiment；

Fig. 2 is the flow diagram of data verification method in one embodiment；

Fig. 3 is the schematic illustration of consistency verification of data in one embodiment；

Fig. 4 is the schematic diagram for carrying out consistency verification of data in one embodiment during real-time synchronization；

Fig. 5 is the schematic diagram of respective data blocks in master-slave database in one embodiment；

Fig. 6 is the structural block diagram of data calibration device in one embodiment；

Fig. 7 is the structural block diagram of data calibration device in another embodiment；

Fig. 8 is the structural block diagram of computer equipment in one embodiment.

Specific embodiment

It is with reference to the accompanying drawings and embodiments, right in order to which the objects, technical solutions and advantages of the application are more clearly understood The application is further elaborated.It should be appreciated that specific embodiment described herein is only used to explain the application, and It is not used in restriction the application.

Fig. 1 is the applied environment figure of data verification method in one embodiment.Referring to Fig.1, the data verification method application In data verification system.The data verification system includes server 110 where primary database, from server 120 where database With verification server 130.Primary database place server 110 leads to from server 120 where database and verification server 120 Cross network connection.Wherein, server can be with the server cluster of the either multiple server compositions of independent server come real It is existing.It should be noted that server 110 where primary database can be the same service with from server 120 where database Device is also possible to mutually independent server；Verification server 130 can be server 110 where primary database or from number According to one of them of server 120 where library, it is also possible to independently of server 110 where primary database and where database Other servers of server 120.Verification server 130 is for executing the data verification method.

Specifically, verification server 130 can run data check tool, and verification server 130 can then pass through the data Calibration tool executes the data verification method.Wherein, data check tool can specifically be realized by calculation procedure, the computer Program, which is performed, can be performed the data verification method.

As shown in Fig. 2, in one embodiment, providing a kind of data verification method.The present embodiment is mainly in this way It is illustrated applied to the verification server 130 in above-mentioned Fig. 1.Referring to Fig. 2, which specifically includes following step It is rapid:

S202 carries out verification to the first tables of data in primary database and the first checklist is written in check results.

It is appreciated that primary database here and slave database hereinafter, is that there are Data Migration relationship or data Two mutually independent databases of synchronized relation.Under normal conditions, be by primary database by Data Migration or be synchronized to from In database.Database is the warehouse of storing data, and essence is a file system, and data will be deposited according to specific format Storage is got up.User such as can increase, modify, delete and inquire to the data in database at the operation.It is with table in database Organization unit's storing data, i.e. tables of data.Checklist is also tables of data, is recorded for the check results to data.

In a specific embodiment, primary database and from database be relevant database.

Specifically, verification server, which can be determined first, stays in primary database and from carrying out consistency verification of data between database Tables of data.Here consistency verification of data, refer to verification primary database two tables of data corresponding with from database it Between data consistency.

It is appreciated that after having repaired master-slave synchronisation problem or having carried out Data Migration, it usually needs verification principal and subordinate's number According to the data consistency in library, to guarantee between principal and subordinate normal synchronized or Data Migration successfully etc..So, master data is stayed in here Library and between database carry out consistency verification of data tables of data, can be the tables of data of master-slave synchronisation, be also possible to count According to the tables of data after migration.Stay in primary database and can from the quantity of tables of data for carrying out consistency verification of data between database Think one or more.

Further, verification server can be to the tables of data of consistency verification of data pending in primary database, hereinafter referred to as Make the first tables of data, verified, and the first checklist is written into obtained check results.Here the first tables of data is carried out Verification, different from consistency verification of data above-mentioned, refers to and verifies according to preset verification mode to the data in tables of data. Here there are many verification modes, such as cyclic redundancy check (Cyclic Redundancy Check, CRC) or exclusive or school Test (Block Check Character, BCC) etc..

For example, for example, primary database migrates tables of data 1,2 and 3 to after database, tables of data 1,2 and 3 can be with To stay in primary database and from the tables of data for carrying out data check between database, the tables of data 1,2 and 3 in primary database is First tables of data.For another example, primary database and there are synchronized relation, tables of data 4,5 from the tables of data 4,5 and 6 between database And 6 can be to stay in primary database and the tables of data 4,5 from the tables of data for carrying out data check between database, in primary database With 6 i.e. the first tables of data.

Certainly, it stays in primary database and can be from the tables of data for carrying out consistency verification of data between database and all move The tables of data of shifting is also possible to the tables of data of part migration；Stay in primary database and from carrying out data consistency between database The tables of data of verification can be the tables of data of whole master-slave synchronisations, be also possible to the database of part master-slave synchronisation, and user can be with Customized selection.

It should be noted that in the embodiment of the present application, the process of consistency verification of data between carrying out master-slave database In the tables of data in master-slave database is not locked.That is, the consistency verification of data between carrying out master-slave database Process, can have that data are synchronous, and user can also be written and read primary database between master-slave database in real time.

In a further embodiment, verification server can also indicate that the server where primary database to carry out verification behaviour Make, the first tables of data is verified as the server where primary database, and obtained check results write-in first is verified Table.

S204 executes hook program, according to the binary log of primary database to the first tables of data corresponding from database The second tables of data verified, and by check results be written the second checklist.

Wherein, hook program is the program segment of processing event.Whenever triggering the particular event that hook program is hooked, hook Subprogram is just initiated to execute corresponding operation.In the present embodiment, the particular event that hook program is linked up with is from number According in library reset primary database binary log in log recording relevant to the first tables of data verification operation.Hook program is opened Corresponding checking command is then generated according to the log recording of the binary log of primary database after dynamic, and according to the checking command pair The second tables of data of corresponding first tables of data is verified from database, and the second checklist is written in check results.

It is appreciated that carrying out the process of data synchronization i.e. leader follower replication between master-slave database.In simple terms, it is exactly Guarantee primary database (Master) and is consistent from the data of database (Slave).After data change occurs for Master, Slave can be synchronized from Master the data of change automatically and (be might have certain delay), to come in this way Guarantee the consistency of data, here it is leader follower replications.

There are many modes of leader follower replication, and one of which is the mode based on binary log.Specifically, Master will be counted It is recorded in binary log according to change, Slave reads the binary log of Master by I/O thread, by binary log The data change of middle record is performed locally one by one, is completed the change of local data, is changed the data of primary database to realize Become reflection to from database, that is, binary log is reset.

Wherein, the binary log of primary database, for recording the related letter that can generate the operation of modification to primary database Breath.Generally, the operation of database will not be modified similar to Select or Show etc., binary log not will record this then The relevant information of operation；The operation of database can be modified similar to Insert or delete etc., binary log then records should The relevant information of operation.It, can be real between master-slave database due to, the process of consistency verification of data between carrying out master-slave database When there are data are synchronous, user can also be written and read primary database.So, in the binary log of primary database not only It further include carrying out what other were operated to tables of data in primary database including log recording relevant to the first tables of data verification operation Log recording.So from database reset primary database binary log when, only with the first tables of data verification operation This relevant partial log is recorded in playback time, and hook program can be just activated.

The binary log of primary database can be there are many different modes, and the binary log under different mode is to modification The recording mode of the operation of database is different.For example, specifically can record the data modified to database in binary log It is complete also to can record execution SQL statement for library action statement (SQL, Structured Query Language) and contextual information The modification result obtained afterwards.Here modification result can be understood as the modification record of data, for example which data is modified, What kind of data etc. be modified as.

Wherein, based on STATEMENT mode (duplication based on SQL statement, Statement-Based Replication, SBR binary log), record is the action statement modified to data, i.e. SQL statement and contextual information.For example, Operation: update A set test=' test ' is performed to database, if using STATEMENT mode, this Update sentence will be recorded in binary log.Based on ROW mode (based on capable duplication, Row-Based Replication, RBR) binary log, record be then the data line that is influenced of the sentence that modification is made on database with And the modification of these rows, that is, data modification result.For example, for example, performing operation: update Aset to database Test=' test ', if using ROW mode, the modification of data line and these rows that this update sentence is influenced It will be recorded in binary log.

It should be noted that existing carried out in consistency verification of data by binary log playback, only in master The binary log of database is direct record SQL statement and context letter to the recording mode of the operation of modification database When ceasing (such as binary log based on STATEMENT mode), consistency verification of data could be successfully carried out.Its principle is SQL statement is executed to tables of data (block) in primary database, the check results of tables of data (block), will execute in Lai Shengcheng primary database SQL statement be directly recorded into the binary log based on STATEMENT mode, then reset by binary log identical SQL statement is transmitted to from database and executes, and in the check results for calculating corresponding data table (block) from database, finally compares Thus the check results of respective data blocks on master-slave database judge that the data of the tables of data (block) verified in master-slave database are It is no consistent.But this for primary database binary log to modification database operation recording mode, be not directly to remember Recording for the master-slave synchronisation of data base manipulation statement and contextual information (such as binary log based on ROW mode) is one A limitation.It is not direct record data if the binary log of primary database is to the recording mode of the operation of modification database Library action statement and contextual information (such as binary log based on ROW mode), although not complete in binary log Whole SQL statement, but also relevant information of the respective record with these SQL statements, can be by executing hook program according to two These relevant informations recorded in system log, to the tables of data (block) of the phase phase from database with identical verification mode come into Row verification, and check results are write in checklist.

Specifically, verification server the binary log of primary database can reset from database, when reset to When the relevant log recording of the first tables of data verification operation, start hook program, by the hook program according to the log of playback Record determines tables of data corresponding with the tables of data that the log recording of current playback is originated from from database, and determines master data The tables of data that library is originated from the log recording of current playback is verified passed through verification mode, is pressed to determining tables of data Determining verification mode is verified, and will be in check results write-in the second verification tables of data.Wherein, hook program can be pre- First compiled code segment.The hook program can be determined according to the log recording in the binary log of primary database from number According to tables of data corresponding with the tables of data that the log recording is originated from library, and determine the number that primary database is originated from the log recording Passed through verification mode is verified according to table, then determining tables of data is verified by determining verification mode.

It illustrates, it is assumed that the log recording 1 in the binary log of primary database, the log recording 1 are recorded It is the relevant information of the modification operation of tables of data 1 in primary database, then hook program can be determined according to log recording 1 from number According to tables of data 1 ' corresponding with tables of data 1 in library, determine that primary database is verified passed through verification mode A to tables of data 1, Tables of data 1 ' is verified by verification mode A again.

S206 compares first checklist and second checklist.

Wherein, the first checklist and the second checklist are all checklists, are the tables of data for recording check results.Specifically Ground, the first checklist specifically can be the checklist in primary database, verify for recording to tables of data in primary database Check results；Second checklist specifically can be the checklist from database, for recording to the tables of data from database The check results verified.In this way, the first checklist of comparison and the second checklist can be obtained to primary database and from data The result of library progress consistency verification of data.

Specifically, verification server can compare the check results recorded in the first checklist and the second checklist, and output has The check results of which tables of data are inconsistent.If the check results of all tables of data are consistent, illustrate that principal and subordinate's data are one It causes.

In one embodiment, the first checklist and the second checklist can be the same checklist, that is to say, that the school It tests table both and can recorde the check results verified to tables of data in primary database, also can recorde to the data from database The check results that table is verified.Certainly, when the first checklist and the second checklist are different checklist, two tables of data Can not only record the check results verified to tables of data in primary database, but record to from database tables of data verify Check results.In this way, compare in any checklist two parts check results (such as two column, one be classified as primary database verification knot It is fruit, another to be classified as from database check results), can be obtained to primary database with from the knot of database data consistency desired result Fruit.

Above-mentioned data verification method automatically verifies the first tables of data in primary database, and check results is write Enter the first check table.Wherein, the binary log of primary database has recorded correlation of the primary database when carrying out tables of data verification Information, in this way, hook program can be according to the binary log of primary database to corresponding to the first of primary database from database Second tables of data of tables of data is verified, and the second checklist is written in check results, thus the first check table of comparison and the The check results of master library data consistency can be obtained in two check tables, have greatly widened application range.

In one embodiment, data verification method further include: executed in primary database and build table statement creation first Checklist；Table statement will be built to record into the binary log of primary database；It is reset in binary log from database It builds table statement and creates the second checklist.

It is appreciated that SQL statement include DML (Data Manipulation Language, data base manipulation statement) and DDL (Data Definition Language, database definition statement).These two types of sentences are all that can modify to database Sentence will record the relevant information of these sentences in binary log when database executes these sentences.Wherein, DML language Sentence such as INSERT sentence, UPDATE statement or DELETE sentence etc..DDL sentence such as CREATE sentence etc..

Based on the binary log of different mode when recording the relevant information of DML statement, recording mode may be different.Than Binary log such as based on STATEMENT mode is believed to the related of DML statement of the binary log record based on ROW mode Breath is different.But based on the binary log of different mode when recording the relevant information of DDL sentence, recording mode is identical, i.e., Directly record the DDL sentence.

Specifically, primary database belongs to DDL language since this builds table statement after table statement the first checklist of creation is built in execution Sentence, then no matter what kind of format the binary log of primary database is, builds table statement and writes direct binary log.This Sample, this is built table statement and is resetting from database, i.e., this is directly built table statement and be transferred to from database and execute, from database It is middle to establish the second checklist identical with the first checklist.

For example, the primary support of primary database can be used after being connected to primary database in verification server The binary log (i.e. Binlog) of Replication agreement adapter tube primary database, then executed in primary database and build table statement Create the checklist of storing data comparison ephemeral data (check results verified to tables of data) cdb.checksums；Table statement will be built again to reset from database by Binlog, from database, i.e. executing this builds table statement To also create checklist cdb.checksums from database.

Fig. 3 shows the schematic illustration of consistency verification of data in one embodiment.With reference to Fig. 3, in the present embodiment, It is first executed in primary database and builds table statement, establish checklist cdb.checksums, this is then built by table statement by Binlog It is being reset from database, is establishing identical cdb.checksums.

It is appreciated that establishing the table handling of building of the first checklist in primary database, generation is establishing second from database Checklist is built before table handling.Primary database occurs to count from database pair with first to the verification operation of the first tables of data Before verification operation according to corresponding second tables of data of table.But that the second checklist is established from database builds table handling and master Database does not have specific ordinal relation to the verification operation of the first tables of data.

In the present embodiment, it after in primary database by building table statement creation checklist, i.e., is being reset from database This builds table statement, to create checklist identical with primary database from database.In this way, subsequent verify to data When obtaining check results, check results can be documented in checklist, data can be carried out by comparison checklist in this way Consistency desired result.

In one embodiment, when carrying out primary database with from the consistency verification of data of database, also allow to exist It primary database and is operated from the real-time synchronization between database, and to the read-write operation of primary database.So, in primary database Binary log in, not there is only log recording relevant to the first tables of data verification operation, there is also grasp with real-time synchronization Work and/or the relevant log recording of read-write operation.And both log recordings can be staggered-sequence.That is, meeting In the presence of real-time synchronization operation several times has been carried out, the first tables of data verification operation several times is then carried out, then has carried out read-write behaviour several times The scene of work.

Specifically, verification server is remembered when resetting the binary log of primary database from database when resetting log When record is real-time synchronization operation and/or read-write operation relevant log recording, then synchronization is executed to from the tables of data in database Operation；When resetting log recording is log recording relevant to the first tables of data verification operation, then held by hook program Row verification operation.

For example, the log recording recorded in binary log are as follows: by the data modification of the 3rd row of Table A be XXX.So, from It when resetting the log recording in database, i.e., is XXX by the data modification of the 3rd row of Table A from library, to realize primary database Data change be synchronized to from database.

For another example, the log recording recorded in binary log are as follows: joined data in checklist, which is Table A Check results；So, when resetting the log recording from database, i.e., data are also added in the checklist from library, and The data of addition are the check results of the Table A from library, and the data change of primary database is synchronized to from database to realize.

Fig. 4 shows the schematic diagram for carrying out consistency verification of data in one embodiment during real-time synchronization.With reference to Fig. 4 includes tables of data A1, A2, A3 ... in primary database, includes tables of data B1, B2, B3 ... from database, and A1 and B1 is phase The tables of data answered, A2 are corresponding tables of data with B2.In the present embodiment, consistency verification of data process is to verify A1 and B1 Data consistency and A2 and B2 data consistency.Assuming that being carried out sometime putting to the data block ai in data Table A 1 Verification, obtains the first check results Ji and is recorded in the checklist of primary database, while producing in the binary log of primary database A raw corresponding log recording Ck.Before verifying to the data block ai+1 in data Table A 1, pass through the behaviour of other users Make, once-through operation carried out to data Table A 2, while generating corresponding log recording Ck+1 in the binary log of primary database, Data block ai+1 in data Table A 1 is verified, a corresponding log note is generated in the binary log of primary database Record Ck+2.When resetting the log recording of binary log of primary database from database, when playback to log recording Ck, root It is verified according to data block bi of the log recording Ck to tables of data B1, bi is corresponding several piece with ai；When replaying to log Ck+1 is recorded, tables of data B2 is synchronized according to log recording Ck+1；When replaying to log recording Ck+2, according to log Record Ck+2 verifies the data block bi+1 of tables of data B1, and bi+1 is corresponding several piece with ai+1, and replays log Record.

In the present embodiment, the data difference having between two databases of master-slave relationship can be compared online, even if There are still real-time data synchronizations between principal and subordinate, or there is the read-write operation to primary database.

In one embodiment, S202 includes: that the first tables of data in primary database is divided into multiple first data blocks； Successively each first data block is verified, and the first checklist is written into check results.S204 includes: to execute hook journey Sequence is successively counted to from database with each described first according to the log recording of the binary log of primary database sequence It is verified according to corresponding second data block of block, and the second checklist is written into check results.

It is understood that, it is generally the case that the data volume of a tables of data is larger in database.It is once biggish to data volume When entire tables of data is verified, calculate time-consuming usually longer.In the present embodiment, it is drawn by carrying out data block to tables of data Point, a tables of data is divided into multiple data blocks, then verify respectively to different data blocks, can reduce counting in this way The time-consuming counted in improves verification efficiency.

Specifically, the first tables of data in primary database can be divided into multiple first data blocks by verification server, then according to It is secondary that each first data block is verified, and the first checklist is written into check results.In this way, the binary log of primary database In the log recording chronologically recorded, then be able to reflect multiple first data blocks in primary database verification sequence.From number Execute hook program according to library, then according to the log recording of the binary log of primary database sequence, successively to from database with Corresponding second data block of each first data block is verified, and the second checklist is written in check results.For example, main number It is ordered as M according to verification of the first data block N in library in this multiple first data block, then, it is counted from database with first According to the corresponding second data block N ' of block N, the verification sequence in the second data block corresponding with each first data block is also M.

For example, Fig. 5 shows the schematic diagram of respective data blocks in master-slave database in one embodiment.With reference to Fig. 5, In the present embodiment, the tables of data A1 in primary database with from the tables of data B1 in database be corresponding tables of data (i.e. A1 with B1 is synchronous tables of data), data block an in tables of data A1 is corresponding data block with the data block bn of tables of data B1.Work as school Server where testing server or primary database is verified to obtain the first check results and the school of primary database is written to an When testing table, a log recording m can be accordingly generated in the database journal of primary database, when log recording m is from database weight When putting, the data block for verifying server or first determining that log recording m is originated from from server where database is an, from database In data block corresponding with an be bn, then verified to obtain the write-in of the second check results from the checklist of database to bn.

It is appreciated that the verification of data block is successively to carry out in primary database, having time sequence；So number of primary database It is also to be sequentially generated according to the log recording in the log of library, having time sequence.In the database for resetting primary database from database Log recording in log is also sequentially to reset.The operating result that is operated in this way to checklist is being reset from database When, and successively the operating result of playback is handled.

In a specific embodiment, the field of checklist may include database name (db), data table name (tbl), number According to block name (chunk), data block left margin (lower_boundary), data block right margin (upper_boundary), main number According to the line number (master_ of the data block verified in the check results (master_crc) of data block in library, primary database Count), from the line number of check results (this_crc) and the data block verified from database of data block in database (this_count) etc..In this way, when resetting log recording, server is verified or from server where database, it can be fast Speed determines the data block that log recording is originated from according to the data in the write in table recorded in log recording.

With continued reference to Fig. 3, server where verifying server or primary database to first chunk of data Table A into When row verification, it may be determined that the left margin of the chuck is denoted as X (for example the left margin of 1-1000 data block is 1), and judges whether There are the left margins of next chunk.When there are the left margin of next chunk, illustrate current chunk non-data Table A The last one chunk, cyclic redundancy check is carried out to current chunk and obtains the master_crc of current chunk, and is current The line number master_count of chunk.When the left margin of next chunk is not present, illustrate that current chunk is tables of data The last one chunk of A is carried out by the right margin Y of the maximum value of current chunk chunk the most current, and to current chunk Cyclic redundancy check obtains the master_crc of current chunk, and the line number master_count of current chunk.

In above-described embodiment, tables of data to be verified is divided into data block, is calculated, is avoided as unit of data block The entirety of tables of data is locked, reduces the influence for normally reading and writing business to primary database.Moreover, with each in primary database The corresponding second Data block check sequence of one data block, the verification sequence consensus with each first data block ensure that primary database It is verified in the same order with from the corresponding data block of database, avoid checking procedure is influenced to cause by other database manipulations The problem of verification error.

In one embodiment, successively each first data block is verified, and the first checklist is written into check results, It include: that will successively divide each of to obtain the first data block as current data block；School is carried out to all column of current data block Test every trade number statistics of going forward side by side；The first checklist is written into check results and line number, and using next first data block as current Data block carries out verification and line number is counted to the last one the first data block.

Wherein, the first data block may include multirow data.Specifically, verification server can successively will divide obtain it is every A first data block verifies all column of current data block as current data block, and goes to current data block The first checklist is written, then using next first data block as current data in obtained check results and line number by number statistics Block executes aforementioned operation to current data block until the last one first data block.

In one embodiment, each first data block is identical as the line number of corresponding second data block.Hook program is executed, According to the log recording of the binary log of primary database sequence, successively respectively corresponded to from database with each first data block The second data block verified, and by check results be written the second checklist, comprising: execute hook program, according to master data The log recording sequence of the binary log in library successively determines the second number corresponding with each first data block from database According to block；Verify every trade number statistics of going forward side by side to all column of each second data block successively determined；By check results and line number The second checklist is written.

It is appreciated that verification server is executing hook program from database, what is determined is corresponding with the first data block Second data block is identical with the line number of first data block.If line number is not identical, that just illustrates the number of master-slave database According to having differences.And in existing consistency verification of data mode, under line number unanimous circumstances, to from the data in database When block is verified, only the data in the data block with the respective column of primary database respective data blocks are verified.So, exist When the data column of corresponding data table are inconsistent in master-slave database, existing consistency verification of data mode can generate wrong comparison As a result the problem of.In the present embodiment, when by hook program to being verified from the data block in database, then to the data The data of all column are verified in block, efficiently avoid the generation of the mistake.

It should be noted that the process successively verified to data block can be continuously, that is to say, that current After data block has been handled, then next data block is handled immediately.When primary database or from the pressure of database it is larger When, may exist time interval between different data block processing, such as it is every handle one data block and waits for 100ms reprocess Next data block etc..

In addition, when the verification of tables of data interruption needs to re-start verification, or again to currently processed data block When processing, primary database can delete corresponding history check results in the first checklist.And delete operation can be through binary system day Will is synchronized to from database, from the history check results that also will be deleted respective data blocks in the second checklist on database.

It in the above-described embodiments, is that all data of the data block are arranged when being verified to the data block in master library Data are verified；And when according to hook program to be verified to data block corresponding from library, and to from library The data of all data column of the corresponding data block are all verified, and are efficiently avoided in principal and subordinate library in associated databases Not the problem of column not will appear check errors simultaneously.

In one embodiment, the first tables of data in primary database is divided into multiple first data blocks, comprising: obtain The unique index of first tables of data of primary database；The key assignments according to included by unique index, by the first tables of data divided by row For multiple first data blocks.

Wherein, the unique index of a tables of data is the set of the key assignments of each row in the tables of data.Included by unique index Key assignments it is different.A unique index may be present in the tables of data in database under normal conditions.

Specifically, verification server can obtain the tables of data only after clearly requiring the tables of data for carrying out data check One index, the key assignments according to included by unique index are ranked up the multirow data for including in tables of data, by primary database Tables of data key value is divided into multiple data blocks, that is, divided by row is multiple data blocks, and each data block includes multirow number According to.

For example, a wherein table t1 for the tables of data for pending data check, finds unique rope of this table Draw, the data of the table are divided into small data block (chuck) one by one, such as 1- according to key assignments included by unique index 1000,1001-1999 ... data etc..

It is appreciated that the multirow data for including in a data block can be adjacent data line, it is also possible to non-conterminous Data line.This is because serial number of the key assignments of line number of the row data in tables of data and row data in unique index is not necessarily It is identical.For example, the third line data in tables of data A, the row in tables of data A is ordered as 3, but the key assignments of the row data is only Sequence in one index is not 3.And since there may be the operations of the additions and deletions to tables of data, so line number is unable to unique identification a line Data, but by the key assignments of row data come unique identification the row data.

When verification server does not find the unique index of the tables of data, the displaying of prompting message can trigger.The prompt The content of message includes asking the user whether to add unique index for the tables of data, and do not add the risk of unique index, with The approach independently selected is provided for user.Here unique index is not added, will increase the time-consuming of data check, it is difficult to ensure that master It is not influenced from synchronizing, but not influences the correctness of result.

Referring again to Fig. 3, in the present embodiment, the tables of data of pending data check includes tables of data A, then then obtaining number According to the unique index of Table A, tables of data A divided by row is multiple data blocks by the key assignments according to included by unique index (chunk)。

In the present embodiment, it is block by row cutting by table to be compared according to unique index, calculates, avoid in blocks The entirety of tables of data is locked, it can be with less time-consuming；And can reduce in checking procedure, to primary database and from database Different from the influence of other read-write operations of the verification operation.

In one embodiment, successively each first data block is verified, and the first checklist is written into check results, It include: that will successively divide each of to obtain the first data block as current data block；When being verified to current data block, open Dynamic transaction operation carries out operation to current data block according to preset verification mode by transaction operation and obtains corresponding first school Test result；After the first checklist is written in the first check results by transaction operation, end transaction operates and by next the One data block carries out verification until the last one first data block as current data block.

It should be noted that including the different types of operation such as normal operations and transaction operation to the operation of database.It is general Logical operation includes the single operations such as increase, deletion, lookup or modification to database, and transaction operation is to database batch The operations such as increase, deletion, lookup, modification.It is atomic operation under affairs, only includes two kinds of success or failure as a result, will not go out Existing intermediate result.

Specifically, server where verifying server or primary database can will successively divide obtained each data block and make Start transaction operation, by transaction operation according to preset verification when verifying to current data block for current data block Mode carries out operation to current data block and obtains corresponding first check results, and continues through transaction operation for current data block The first check results record to the first checklist, then terminate the transaction operation, then using next data block as current number According to block, the above-mentioned operation carried out to current data block is executed, until the last one data block.It is appreciated that for each A data block carries out the verification of the data block by a transaction operation and check results records.

In one embodiment, server where verifying server or primary database is determining pending data check When tables of data, whether engine corresponding to these tables of data supports transaction operation in detectable primary database.When in primary database When engine corresponding to these tables of data supports transaction operation, server where verifying server or primary database then continues Execute subsequent step；When there are the engines corresponding to tables of data not to support transaction operation in these tables of data in primary database When, server where verifying server or primary database, which then continues to execute subsequent step, can trigger reminder message.The prompting Message updates in user for reminding user to modify engine corresponding to the tables of data for not supporting transaction operation and does not support affairs Engine corresponding to the tables of data of operation continues to execute subsequent step to after supporting transaction operation.Wherein, transaction operation is supported Engine such as MySQL the InnoDB engine or TokuDB engine etc. supported.

In one embodiment, when transaction operation is activated, the isolation level of primary database is attached most importance to re-reading rank.Start thing Business operation carries out operation to current data block according to preset verification mode by transaction operation and obtains corresponding first verification knot Fruit, comprising: starting transaction operation；Database snapshot when transaction operation is activated is generated based on con current control；It is grasped by affairs Make, operation is carried out to the current data block in database snapshot according to preset verification mode and obtains corresponding first verification knot Fruit.

It is appreciated that primary database supports other use simultaneously when verification primary database and from the tables of data between database When the read-write operation at family, if the read-write operation of other users and verification operation are directed to same tables of data, verification will affect As a result, verification is caused to malfunction.

Primary database in the embodiment of the present application support MVCC (Multi-Version Concurrency Control, it is more Version con current control).It should be noted that under normal conditions, while reading data from database if there is user, having in addition User be written data, it is possible to the user for reading data can be appreciated that " partly writing " or inconsistent data.It is asked to solve this Topic, simplest method is by making the user job of the data to be written such as all users of reading data complete to data library locking At, but efficiency in this way can be very poor.MVCC has then used a kind of different means, is each connected to the use of the reading data of database Family, in some moment it is seen that a snapshot of database, writes variation caused by the write operation of the user of data in write operation It is sightless for the user that other read data before completion (or before db transaction is submitted).

Under normal conditions, MVCC only REPEATABLE READ (repeat read rank) and READ COMMITED (read-only The data of submission) it works under two isolation levels.In the present embodiment, database is verified when by a transaction operation When, the isolation level of primary database is attached most importance to re-reading rank, and under this isolation level, behaviour is read in any repetition in the transaction operation The data for making to obtain are just as, and whether the tables of data no matter read at this time has update to operate, and ensure that the correct of check results Property.

In the present embodiment, by the setting of con current control and database isolation level, so that carrying out database verification Transaction operation in the obtained data of any repetition read operation be just as, whether the tables of data no matter read at this time has more New operation, ensure that the correctness of check results.

With continued reference to Fig. 3, when server where verifying server or primary database verifies current data block, In one transaction operation, the cyclic redundancy check result master_crc and line number master_count of current chunk are calculated, is write Enter in the checksums table of primary database.The operating result of the operation can record into the database journal of primary database, when From database when the operating result of replay data library log, starts Hook Function, current chunk phase is calculated by Hook Function The cyclic redundancy check result this_crc and line number this_count of chunk are answered, the checksums table from database is written In.Primary database can obtain the this_crc and this_count obtained from database, and the checksums table of primary database is written In；The master_crc and master_count that primary database obtains can also be obtained from database, is written from database In checksums table.Server where verification server or primary database obtains next chunk again, and circulation is executed to working as The operation that preceding chunk is executed.Until current chunk be the last one chunk to be verified when, chunk be (- ∞, X) or (Y ,+∞) at this time terminates the verification of data Table A, can the checklist according to primary database or the checklist from database, Obtain check results.

It is appreciated that when being verified to data Table A, it is also possible to the operation for increase to the tables of data A row, because This, is set as (- ∞ ,+∞) for the range of the unique index of tables of data A.

In above-described embodiment, the data check an of data block and the note of check results are completed by a transaction operation Record, ensure that the correctness of check results.

Generally speaking, it must be based on that existing consistency verification of data, which requires the database journal of primary database, The binary log of STATEMENT mode, and for the binary log of other modes (such as be based on ROW pattern binary day Will) then it is restricted.And the master-slave synchronisation of current industry, since STATEMENT mode has more known problem, such as can It can seriously cause the inconsistent of principal and subordinate's data, so will not be led substantially using the binary log of STATEMENT mode From synchronization.For the master-slave synchronisation of the binary log of non-STATEMENT mode, need database connection session's first Binary log schema modification is STATEMENT, which needs user to provide the account of SUPER permission.In general, user It is more sensitive for the database account for providing high permission.Moreover, under certain special screnes, existing consistency verification of data The data difference of master-slave database can not be contrasted.Such as the tables of data in primary database and the corresponding data from database The definition of several fields is the same before table, but from the tables of data of database than primary database tables of data more than one or The case where multiple fields, that is to say, that for two tables of data in addition to last one or more columns per page, other data in table are the same 's.In this case, existing consistency verification of data, which compares two tables, can obtain the conclusion of mistake, so as to cause calamity The consequence of difficulty.

In conjunction with embodiment above-mentioned, it is to be understood that the embodiment of the present application, which does not need SUPER permission, forces database The schema modification of log is STATEMENT mode.Moreover, the unique index based on tables of data to be verified in the embodiment of the present application And the hook program (also referred to as hook operation) executed when resetting log recording, so that being disobeyed during consistency verification of data Lai Yu, which is reset, completes consistency verification of data task based on the SQL statement recorded in the binary log of STATEMENT mode. Meanwhile execution hook program can be equal by all data of data block corresponding with master data from database in the embodiment of the present application CRC check is participated in, it in this way can be to avoid the case where there may be erroneous judgements in the more one or more columns per pages of tables of data from database.

Moreover, between the normal database of master-slave synchronisation, if at master-slave database same snapshot time point, arbitrary number It is all consistent according to the data of block, then the data of master-slave database are consistent certainly.Due to the binary system day of primary database Will be it is strictly linear orderly on time dimension, it is fast from the data of database when being ready to carry out hook program from database It is consistent when executing corresponding data block check with primary database according to time point.This inference is easy to prove by reduction to absurdity, Because being all one in the data of master-slave database same snapshot time point arbitrary data block between the normal database of master-slave synchronisation It causes, if the data of master-slave database are inconsistent, before in comparison, principal and subordinate's data can not be in same snapshot point It is consistent.

It should be understood that although each step in the flow chart of the various embodiments described above is successively shown according to the instruction of arrow Show, but these steps are not that the inevitable sequence according to arrow instruction successively executes.Unless expressly state otherwise herein, this There is no stringent sequences to limit for the execution of a little steps, these steps can execute in other order.Moreover, above-mentioned each implementation At least part step in example may include that perhaps these sub-steps of multiple stages or stage be not necessarily for multiple sub-steps It is to execute completion in synchronization, but can execute at different times, the execution sequence in these sub-steps or stage It is not necessarily and successively carries out, but can be at least part wheel of the sub-step or stage of other steps or other steps Stream alternately executes.

As shown in fig. 6, in one embodiment, providing a kind of data calibration device 600.Referring to Fig. 6, the data check Device 600 includes: the first correction verification module 601, the second correction verification module 602 and contrast module 603.

First correction verification module 601, for verify and check results are written to the first tables of data in primary database First checklist.

Second correction verification module 602, for executing hook program, according to the binary log of primary database to from database Second tables of data of corresponding first tables of data is verified, and the second checklist is written in check results.

Contrast module 603, for comparing the first checklist and the second checklist.

Above-mentioned data calibration device 600 automatically verifies the first tables of data in primary database, and by check results The first check table is written.Wherein, the binary log of primary database has recorded phase of the primary database when carrying out tables of data verification Information is closed, in this way, hook program can be according to the binary log of primary database to corresponding to the of primary database from database Second tables of data of one tables of data is verified, and by check results be written the second checklist, thus comparison the first check table with The check results of master library data consistency can be obtained in second check table, have greatly widened application range.

In one embodiment, the first correction verification module 601 is also used to for the first tables of data in primary database being divided into more A first data block；Successively each first data block is verified, and the first checklist is written into check results.Second calibration mode Block 602 is also used to execute hook program, according to the log recording of the binary log of primary database sequence, successively to from database In the second data block corresponding with each first data block verified, and by check results be written the second checklist.

In one embodiment, the first correction verification module 601, which is also used to successively divide, each of obtains the first data block and makees For current data block；Verify every trade number statistics of going forward side by side to all column of current data block；Check results and line number are written First checklist, and using next first data block as current data block carry out verification and line number statistics until the last one the One data block.

In one embodiment, each first data block is identical as the line number of corresponding second data block.Second correction verification module 602 are also used to execute hook program, according to the log recording of the binary log of primary database sequence, successively determine from database In the second data block corresponding with each first data block；All column of each second data block successively determined are verified Every trade of going forward side by side number statistics；The second checklist is written into check results and line number.

In one embodiment, the first correction verification module 601 is also used to obtain unique rope of the first tables of data of primary database Draw；First tables of data divided by row is multiple first data blocks by the key assignments according to included by unique index.

In one embodiment, the first correction verification module 601, which is also used to successively divide, each of obtains the first data block and makees For current data block；When verifying to current data block, start transaction operation, by transaction operation according to preset verification Mode carries out operation to current data block and obtains corresponding first check results；The first check results are being write by transaction operation After entering the first checklist, end transaction operation simultaneously carries out verification using next first data block as current data block until last One the first data block.

In one embodiment, when transaction operation is activated, the isolation level of primary database is attached most importance to re-reading rank.First school Module 601 is tested to be also used to start transaction operation；Database snapshot when transaction operation is activated is generated based on con current control；Pass through Transaction operation carries out operation to the current data block in database snapshot according to preset verification mode and obtains corresponding first school Test result.

As shown in fig. 7, in one embodiment, data calibration device 600 further include: first builds table module 604 and second Build table module 605.

First builds table module 604, builds table statement the first checklist of creation for executing in primary database；Table is built by described Sentence is recorded into the binary log of the primary database.

Second builds table module 605, for building table statement wound described in reset in the binary log in database Build the second checklist.

In one embodiment, the binary log for belonging to primary database is the binary log based on row replication mode.

Fig. 8 shows the internal structure chart of computer equipment in one embodiment.The computer equipment specifically can be Fig. 1 In verification server 130.As shown in figure 8, it includes being connected by system bus that the computer equipment, which includes the computer equipment, Processor, memory and network interface.Wherein, memory includes non-volatile memory medium and built-in storage.The computer The non-volatile memory medium of equipment is stored with operating system, can also be stored with computer program, and the computer program is processed When device executes, processor may make to realize data verification method.Computer program can also be stored in the built-in storage, the calculating When machine program is executed by processor, processor may make to execute data verification method.It will be understood by those skilled in the art that Fig. 8 Shown in structure, only the block diagram of part-structure relevant to application scheme, does not constitute and is answered application scheme With the restriction of computer equipment thereon, specific computer equipment may include than more or fewer portions as shown in the figure Part perhaps combines certain components or with different component layouts.

In one embodiment, data calibration device provided by the present application can be implemented as a kind of shape of computer program Formula, computer program can be run in computer equipment as shown in Figure 8.Composition can be stored in the memory of computer equipment should Each program module of data calibration device, for example, the first correction verification module 601 shown in fig. 6, the second correction verification module 602 and right Than module 603.It is each that the computer program that each program module is constituted makes processor execute the application described in this specification Step in the data verification method of a embodiment.

For example, computer equipment shown in Fig. 8 can pass through the first school in data calibration device 600 as shown in FIG. 6 Module 601 is tested to carry out verification to the first tables of data in primary database and the first checklist is written in check results.Pass through second Correction verification module 602 executes hook program, according to the binary log of primary database to the first tables of data corresponding from database Second tables of data is verified, and the second checklist is written in check results.By contrast module 603 compare the first checklist with Second checklist.

In one embodiment, a kind of computer equipment, including memory and processor are provided, memory is stored with meter Calculation machine program, when computer program is executed by processor, so that the step of processor executes above-mentioned data verification method.It counts herein It can be the step in the data verification method of above-mentioned each embodiment according to the step of method of calibration.

In one embodiment, a kind of computer readable storage medium is provided, computer program, computer journey are stored with When sequence is executed by processor, so that the step of processor executes above-mentioned data verification method.The step of data verification method herein It can be the step in the data verification method of above-mentioned each embodiment.

Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with Relevant hardware is instructed to complete by computer program, the program can be stored in a non-volatile computer and can be read In storage medium, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, provided herein Each embodiment used in any reference to memory, storage, database or other media, may each comprise non-volatile And/or volatile memory.Nonvolatile memory may include that read-only memory (ROM), programming ROM (PROM), electricity can be compiled Journey ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms, such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) directly RAM (RDRAM), straight Connect memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..

Each technical characteristic of above embodiments can be combined arbitrarily, for simplicity of description, not to above-described embodiment In each technical characteristic it is all possible combination be all described, as long as however, the combination of these technical characteristics be not present lance Shield all should be considered as described in this specification.

The several embodiments of the application above described embodiment only expresses, the description thereof is more specific and detailed, but simultaneously The limitation to the application the scope of the patents therefore cannot be interpreted as.It should be pointed out that for those of ordinary skill in the art For, without departing from the concept of this application, various modifications and improvements can be made, these belong to the guarantor of the application Protect range.Therefore, the scope of protection shall be subject to the appended claims for the application patent.

Claims

1. a kind of data verification method, comprising:

Hook program is executed, according to the binary log of the primary database to corresponding to first tables of data from database Second tables of data is verified, and the second checklist is written in check results；

Compare first checklist and second checklist.

2. the method according to claim 1, wherein first tables of data in primary database verifies And the first checklist is written into check results, comprising:

First tables of data in the primary database is divided into multiple first data blocks；

Successively each first data block is verified, and first checklist is written into check results；

The execution hook program, according to the binary log of the primary database to corresponding to first data from database Second tables of data of table is verified, and the second checklist is written in check results, comprising:

The hook program is executed, according to the log recording of the binary log of primary database sequence, successively to institute It states the second data block corresponding with each first data block from database to be verified, and institute is written into check results State the second checklist.

3. according to the method described in claim 2, it is characterized in that, described successively verify each first data block, And first checklist is written into check results, comprising:

It will successively divide and each of obtain first data block as current data block；

Verify every trade number statistics of going forward side by side to all column of the current data block；

First checklist is written into check results and line number, and is carried out next first data block as current data block Verification and line number statistics are until the last one first data block.

4. according to the method described in claim 2, it is characterized in that, each first data block and corresponding second data block Line number is identical；It is described to execute the hook program, sequentially according to the log recording of the binary log of the primary database, Successively to described, the second data block corresponding with each first data block is verified from database, and will verification knot Second checklist is written in fruit, comprising:

The hook program is executed, according to the log recording of the binary log of primary database sequence, is successively determined Second data block corresponding with each first data block from database；

Verify every trade number statistics of going forward side by side to all column of each second data block successively determined；

Second checklist is written into check results and line number.

5. according to the method described in claim 2, it is characterized in that, first tables of data by the primary database It is divided into multiple first data blocks, comprising:

Obtain the unique index of first tables of data of the primary database；

It is multiple first data blocks by the first tables of data divided by row according to key assignments included by the unique index.

6. according to the method described in claim 2, it is characterized in that, described successively verify each first data block, And first checklist is written into check results, comprising:

When verifying to the current data block, start transaction operation, by the transaction operation according to preset verification Mode carries out operation to the current data block and obtains corresponding first check results；

After first checklist is written in first check results by the transaction operation, terminate the affairs behaviour Make, and carries out verification until the last one first data block using next first data block as current data block.

7. according to the method described in claim 6, it is characterized in that, when the transaction operation is activated, the primary database Isolation level is attached most importance to re-reading rank；The starting transaction operation, by the transaction operation according to preset verification mode to institute It states current data block progress operation and obtains corresponding first check results, comprising:

Start the transaction operation；

The database snapshot when transaction operation is activated is generated based on con current control；

By the transaction operation, the current data block in the database snapshot is carried out according to preset verification mode Operation obtains corresponding first check results.

8. the method according to claim 1, wherein the method also includes:

It is executed in the primary database and builds table statement creation first checklist；

The table statement of building is recorded into the binary log of the primary database；

Table statement is built described in reset in the binary log in database described, creates second checklist.

9. method according to claim 1 to 8, which is characterized in that the binary system day for belonging to primary database Will is the binary log based on row replication mode.

10. a kind of data calibration device, comprising:

First correction verification module, for carrying out verification to the first tables of data in primary database and verifying check results write-in first Table；

Second correction verification module, for executing hook program, according to the binary log of the primary database to right from database It answers the second tables of data of first tables of data to be verified, and the second checklist is written into check results；

Contrast module, for comparing first checklist and second checklist.

11. device according to claim 10, which is characterized in that first correction verification module is also used to the master data First tables of data in library is divided into multiple first data blocks；Successively each first data block is verified, and will First checklist is written in check results；

Second correction verification module is also used to execute the hook program, according to the binary log of the primary database Log recording sequence, successively to described, the second data block corresponding with each first data block carries out school from database It tests, and second checklist is written into check results.

12. device according to claim 10, which is characterized in that described device further include:

First builds table module, builds table statement creation first checklist for executing in the primary database；It is built described Table statement is recorded into the binary log of the primary database；

Second builds table module, for building table statement creation institute described in reset in the binary log in database described State the second checklist.

13. device according to claim 10, which is characterized in that the binary log for belonging to primary database be based on The binary log of row replication mode.

14. a kind of computer readable storage medium is stored with computer program, when the computer program is executed by processor, So that the processor is executed such as the step of any one of claims 1 to 9 the method.

15. a kind of computer equipment, including memory and processor, the memory is stored with computer program, the calculating When machine program is executed by the processor, so that the processor executes the step such as any one of claims 1 to 9 the method Suddenly.