CN101464890A - Mixed log generation method and resolution method - Google Patents
Mixed log generation method and resolution method Download PDFInfo
- Publication number
- CN101464890A CN101464890A CNA2008102418990A CN200810241899A CN101464890A CN 101464890 A CN101464890 A CN 101464890A CN A2008102418990 A CNA2008102418990 A CN A2008102418990A CN 200810241899 A CN200810241899 A CN 200810241899A CN 101464890 A CN101464890 A CN 101464890A
- Authority
- CN
- China
- Prior art keywords
- daily record
- mixing
- log
- redo
- transaction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention discloses a method for producing hybrid log and an analytical method thereof, wherein, the method for producing the hybrid log comprises the following steps: opening or creating a hybrid log file; receiving user commands and judging the types of the user commands; and concluding the type of the user command according to the judgment and registering the corresponding command type log. The analytical method of the hybrid log comprises the following steps: A, applying for a handle of a log parser and opening a hybrid log file; B, judging whether a closed REDO/ UNDO affair block node exists or not; if yes, executing step C, and if not, executing step D; D, analyzing log lines in the hybrid log file sequentially, executing the operation for producing REDO/UNDO transaction block, and then executing step B. The hybrid log combines all information of transaction log and synchronous log, so that redundant information is eliminated.
Description
Technical field
The present invention relates to the memory database field, relate in particular to a kind of mixing daily record generation method and analytic method of memory database.
Background technology
Modern communications is more and more stronger to the computer technology dependence, in the continuous development of telecom operators, set up the various information system, comprise Operation Network system, comprehensive accounting system, charge system, office automation system or the like, these systems have accumulated the valuable data information of magnanimity, by these data are carried out data mining, query analysis and on-line analysis, can discern the market opportunity, formulate market strategy.But these data informations are too huge, and the record strip number is easily in hundred million, and as note system, per second need be stored thousands of user profile.For huge data volume like this, traditional data base management system (DBMS) based on disk more and more is difficult to deal with, so industry has proposed memory database technology and main memory DBM system (MainMemory Database Management System is hereinafter to be referred as MMDBMS).
Memory database is about to the database that data are placed on direct control in the internal memory.The main memory DBM system manages the part of entire database or database in internal memory, so direct visit data and need not visit disk.Thereby memory database has just had high performance transaction capabilities.
Yet because internal memory itself has volatibility, in case i.e. power down or break down, data will all be lost, and cause the commercial system irretrievable loss.Therefore, MMDBMS can interiorly at certain time intervals backup to the dirty data in the internal memory on the permanent storage.
MMDBMS supports concurrent operations, and the process of backup may be accompanied by the renewal process of data.What obtain after the backup may be the database image file that is in the inconsistency state.If use such image file to come restore database, will make database be absorbed in the inconsistency state.
In order to obtain an image file that is in coherency state, at present many systems adopt the mode of static backup, and promptly waiting does not have affairs to back up when operation in the system.Do not allow during the backup database is upgraded.This backup method greatly reduces the availability of system.And backup operation can not be too frequent, otherwise the I/O of system can not bear.In addition, the time interval of backup is long more, in case system crash, the data volume of losing is also many more.At present general way is to adopt registration daily record (Logging) to ensure that data do not lose, lose less.
In addition, MMDBMS also can be absorbed in the inconsistency state in operational process.For example the system crash accident stops affairs, and user's active termination affairs etc. all may cause database inconsistent.Present most of system all adopts the method for registration transaction journal to come database recovery to up-to-date coherency state.
A lot of application systems, as above-mentioned note system, often the node by a lot of band databases constitutes a distributed system, often needs synchrodata between the database of these nodes.Data synchronization process is carried out sometimes in real time, only needs regularly sometimes/an irregular synchronous secondary data.Often do not need to generate sync data file in real time synchronously.And for regularly/irregularly synchronously, what industry employing at present was maximum is the way that generates sync data file.
But, the transaction journal file adds sync data file, and the data volume of its generation is very huge.The researchist is by to the research of the two memory contents, finds that the two has deposited many identical or identical information, and redundance is quite big.This will cause the serious waste of storage space, also expend the computational resource when generating these redundant informations.
In addition, when carrying out system failure recovery, the common way of industry is to set up two formations at present: REDO formation and UNDO formation.Repeat REDO and UNDO operation during system failure recovery respectively, way is more loaded down with trivial details, and the multiple scanning journal file, and efficient is lower.
Summary of the invention
For existing simultaneously, the transaction journal file that solves above-mentioned Database Systems and sync data file cause waste of storage space, the inefficient problem of system failure recovery, be necessary to provide a kind of storage space less, the mixing daily record generation method that system failure recovery efficient is higher.
In addition, also be necessary to provide a kind of mixing daily record analytic method of above-mentioned mixing journal format.
The present invention's mixing daily record generation method may further comprise the steps: open or create the mixing journal file; Receive user command, and judge the type of described user command; The type of the user command that draws according to described judgement is registered corresponding command type daily record.
The corresponding command type daily record of described registration may further comprise the steps: receive daily record registration request, judge whether described log buffer district has enough spaces to hold described daily record, if there are not enough spaces, then the data of described buffer area write with a brush dipped in Chinese ink described mixing journal file, and empty buffer area; Check the size that aims at described day side-play amount and described mixing journal file in the described buffer area, the two addition is obtained aiming at described day real offset in the described mixing journal file; Described daily record is write described buffer area, and return success byte number and the described real offset that writes.
The type that described judgement draws user command is the checkpoint order, then registers described checkpoint command log, and follows the execution of described checkpoint order, and the execution of described checkpoint order and transaction concurrency are carried out.
The log lines beginning structure of described mixing daily record is lsn:len:sync:op_code, described lsn represents the log lines sequence number, be used for illustrating the order that the log lines order begins to carry out, described len represents except except the shared length of described lsn and len, the length of described log lines, described sync represents whether described log lines is that needs produce synchronization log, and described op_code represents described log lines command code, be used to discern the type of described log lines, described colon: be separator.
Described lsn is unique in the database overall situation, and described sync is the asynchronous daily record of 0 expression, is non-0 expression synchronization log, and the coding span of described op_code is 1 to 255.
The form of described mixing daily record comprises DML statement bracket journal format, DML basic operation journal format, affairs control journal format and checkpoint journal format, the op_code of described DML statement bracket journal format, op is the DML order, code is BEGIN, END or ABORT, the op_code of described DML basic operation journal format is DELETE_VFIELD, INSERT_VFIELD, UPDATE_FIELD, DELETE_RECORD, INSERT_RECORD or FREE_PAGE, the lsn of described affairs control journal format is 0, sync is 1, op is TRANSACTION, and code is BEGIN, COMMIT or ROLLBACK, the lsn of described checkpoint journal format is 0, sync is 1, and op is CHECK_POINT, and code is BEGIN or END.
The present invention's mixing daily record analytic method may further comprise the steps:
Steps A, application date book resolver handle are opened the mixing journal file;
Step B, judge whether to exist closed REDO/UNDO transaction block node, if, execution in step C, otherwise, execution in step D, described REDO/UNDO transaction block is represented the set of all daily records that affairs comprise, and described closed REDO/UNDO transaction block is initial by TRANSACTION_BEGIN, with TRANSACTION_COMMIT or TRANSACTION_ROLLBACK ending;
Step C, described REDO/UNDO transaction block node pointer is composed to output variable, returned and obtain REDO/UNDO transaction block success status value;
Step D, resolve to mix the log lines of journal file successively, carry out the operation that generates described REDO/UNDO transaction block, again execution in step B.
Carrying out the operation that generates described REDO/UNDO transaction block among the described step D may further comprise the steps: it is TRANSACTION_BEGIN that parsing runs into log lines, generate REDO/UNDO transaction block node, judge the affiliated command type of log lines more successively, running into log lines is TRANSACTION_END, and puts described REDO/UNDO transaction block node and be closed sign.
Described REDO/UNDO transaction block comprises statement block, described statement block begins with op_BEGIN, closed statement block begins with op_BEGIN, the op_END ending, op is the DML order, described step D resolves the log lines of mixing journal file successively and comprises: it is op_BEGIN that parsing runs into described log lines, generated statement agllutination point, judge command type under the log lines more successively, running into log lines is op_END, and puts described REDO/UNDO transaction block node and be closed sign.
Described step C replaces with: obtain current statement block, generate the synchronous SQL statement of next bar according to current statement block, internal pointer is displaced to next bar be untreated log lines or next statement block node, and judge whether to generate synchronous SQL statement, if, return and obtain synchronous SQL statement success status value, otherwise, concrete error status value returned.
Compared with prior art, the present invention's mixing daily record generation method and analytic method adopt the mixing daily record that mixes journal format to combine all information of transaction journal and synchronization log, thereby guaranteed that the mixing daily record both can serve as transaction journal, can serve as the data sync daily record again, a lot of redundant informations have been eliminated, make whole daily record scale descend greatly, not only saved file storage, and reduced memory cache space and the cpu resource that consumes when generating these daily records.
And the present invention's mixing daily record generation method and analytic method are supported the concurrent execution of affairs, improve the availability and the handling capacity of system greatly.And, only need be aided with and mix journal file and carry out the coherency state that affairs REDO and UNDO guarantee data.
Moreover the present invention's mixing daily record analytic method adopts scan edge journal file limit to generate the method for REDO/UNDO transaction block, daily record is carried out one time scan parsing and the processing that just can finish whole journal file, has improved resolution speed, has simplified treatment scheme.And, the construction process of REDO formation and UNDO formation is integrated, when REDO/UNDO transaction block closure, just carry out the REDO/UNDO operation or generate synchronous SQL statement, disposing discharges the shared memory source of this REDO/UNDO transaction block immediately.
After the detailed description of reading embodiment of the present invention in conjunction with the accompanying drawings, it is clearer that characteristics of the present invention and advantage will become.
Description of drawings
Fig. 1 is the schematic flow sheet of the present invention's mixing daily record generation method.
Fig. 2 is the idiographic flow synoptic diagram of Fig. 1.
Fig. 3 is the specific implementation process flow diagram of daily record registration among Fig. 1.
Fig. 4 is the process flow diagram that the present invention is used for the mixing daily record analytic method of system failure recovery.
Fig. 5 is the process flow diagram that the present invention is used for the synchronous mixing daily record analytic method of non-real-time data.
Embodiment
Below in conjunction with accompanying drawing the present invention's mixing daily record generation method and analytic method are described.
The present invention's mixing journal format is described with expansion BNF grammer metasymbol.
BNF grammer metasymbol is as follows:
[、]、|、*、+、::
Wherein, [with] be used to define the scope of syntactic element or represent optional part.| be illustrated in optional one of its right and left.* represent 0 or a plurality of.At least 1 of+expression.Square bracket represent that to independent appearance the syntactic element that brackets is optional part.Square bracket to * or+use together, then respectively the syntactic element that brackets of expression occur 0 time to n time with at least once more than.:: the expression definition.BNF grammer metasymbol just is used for describing the instrument of journal format, is not the ingredient of described log lines.
Such as, field is used for describing a field value, and the BNF normal form of definition field is as follows:
field::name_len，name，type，prec，data_len，data
Wherein, name_len represents the string length of name, does not comprise NULL.Name represents field name.Type represents field type.Prec represents the precision of DECIMAL type.Data_len represents the byte stream length of data.Data represents the data word throttling.
For another example, field_desc is used for describing the definition of a field, except do not comprise the data and field just the same.The BNF normal form of definition field_desc is as follows:
field_desc::name_len，name，type，prec，data_len
Mix daily record and be divided into 4 types from function, it comprises: DML (Data ManipulationLanguage) statement bracket daily record, DML basic operation daily record, affairs control daily record and checkpoint daily record.Every type of corresponding respectively corresponding journal format of daily record, i.e. DML statement bracket daily record, DML basic operation daily record, affairs control daily record and the corresponding DML statement bracket of checkpoint daily record journal format, DML basic operation journal format, affairs control journal format and checkpoint journal format.
Every kind of daily record beginning all has following structure:
lsn:len:sync:op_code
Wherein, lsn represents log serial number, and is unique in the database overall situation.Lsn is used for illustrating the order that this daily record order begins to carry out.Len represents except except the shared length of lsn and len, the length of this daily record.Sync represents whether this daily record is that needs produce synchronization log, the asynchronous daily record of 0 expression, non-0 expression synchronization log.Op_code represents the daily record command code, is used to discern the type of log lines.For the ease of reading, op_code uses the underlined memonic symbol form of capitalization English letter to represent, replace to different separately numerical value by the implementor during specific implementation and get final product, as TRANSACTION_BEGIN, DELETE_BEGIN, INSERT_VFIELD, CHECK_POINT_END etc.And TRANSACTION_BEGIN, DELETE_BEGIN, INSERT_VFIELD, CHECK_POINT_END are memonic symbols, are not character strings, as the macro definition of C language.The coding span of op_code is 1～255.": " symbol that occurs for any position in the journal format all not necessarily, whether it only plays the effect of separating each field, and when resolving daily record, help verification log lines form correct.When the daily record specific implementation, can remove ": " symbol, also can keep ": " symbol, perhaps replace with other separators.
The DML statement refers to sql commands such as UPDATE, INSERT, DELETE, TRUNCATE_TABLE, DROP_TABLE.For every sql command, the mixing daily record of its correspondence comprises beginning and end mark at least, and these begin to be called the DML statement bracket with end mark.The mixing daily record of every sql command generation begins with an op_BEGIN daily record, and some basic operation daily records are being followed in the centre, finishes with an op_END or op_ABORT at last.Op replaces with the sql command name, as INSERT_BEGIN.
DML statement bracket journal format is as follows:
lsn:len:sync:op_BEGIN
lsn:len:sync:op_END
lsn:len:sync:op_ABORT
In DML statement bracket journal format, op can substitute with UPDATE, INSERT, DELETE, TRUNCATE_TABLE or DROP_TABLE.
The DML statement bracket can provide the information about relation table.Part also provides the keyword message of related record.As: lsn:len:sync:INSERT_BEGIN:tid:tn_len:table_name:other_in fo
Wherein, tn_len represents the length of table_name, is the byte type; The title that table_name then represents.Other_info represents the keyword message that writes down, as DESC:count:[(field_desc)]+, it is exactly the keyword message of the related record of this daily record order of INSERT_BEGIN.
The DML basic operation is meant the operation of finishing the required concrete enforcement change data of DML order.The DML order comprises SELECT, UPDATE, INSERT, DELETE.The daily record of DML basic operation correspondence mainly contains following 6 kinds:
1) DELETE_VFIELD, variable-length field of expression deletion;
2) INSERT_VFIELD, a variable-length field is inserted in expression;
3) UPDATE_FIELD, a fixed-length field is upgraded in expression;
4) DELETE_RECORD, record row of expression deletion;
5) INSERT_RECORD, a record row is inserted in expression;
6) FREE_PAGE, expression discharges several pages.
DML basic operation journal format:
lsn:len:sync:DELETE_VFIELD：(pno，rno)，(field)]*
lsn:len:sync:INSERT_VFIELD:(pno，rno)，(field)[，KEY:count:[(field)]*]
lsn:len:sync:UPDATE_FIELD:(pno，rno，offset)，OLD:(field)，NEW:(field)，KEY:co
unt:[(field)]*
lsn:len:sync:DELETE_RECORD:(pno，rno)，(data_len，data)，KEY:count:[(field)]*
lsn:len:sync:INSERT_RECORD:(pno，rno)，(data_len，data)
lsn:len:sync:FREE_PAGE:count:[(pno)]*
The DML statement bracket wraps up all DML basic operation daily records of this sql command, and the final execution result of indicating this sql command is successfully or fails.All DML basic operations all must be included within certain DML statement bracket daily record.So all daily records of a sql command generation are all defined by before and after the daily record of DML statement bracket, the centre comprises some DML basic operation daily records, and they are not necessarily deposited physically continuously, but logically are an integral body.All daily records that sql command is produced all be defined as sql command daily record, the perhaps daily record of DML sql command.
The journal format and the logical organization syntactic description thereof of DELETE order are as follows:
lsn:len:sync:DELETE_BEGIN:tid:tn_len:table_name
[
[lsn:len:sync:DELETE_VFIELD:(pno，rno)，(field)]*
lsn:len:sync:DELETE_RECORD:(pno，rno)，(data_len，data)，KEY:count:[(field)]*
]+
lsn:len:sync:DELETE_END|lsn:len:sync:DELETE_ABORT
By the journal format of above-mentioned sql command and logical organization syntactic description thereof as can be known, the SQL statement daily record comprises to get over is engaged in recovering and the required full detail of data sync.
Affairs control daily record refers to the daily record of an affairs initiation command and the finish command correspondence.A complete transaction journal begins with a TRANSACTION_BEGIN daily record, the centre comprises some SQL statement daily records, end-state according to affairs are carried out ends up with a TRANSACTION_COMMIT daily record or TRANSACTION_ROLLBACK daily record at last.Each affairs control daily record has all intactly comprised all sql command daily records that belong to it.An affairs control daily record the inside can comprise 0 to a sql command daily record arbitrarily.Each affairs all has an affairs ID (Identity).Affairs ID is by affairs controller order assignment, affairs of unique expression in certain time period, the time sequencing of also having represented affairs to start basically.But,, reflect the sequencing that affairs are carried out fully so can not definitely guarantee affairs ID owing to allow transaction concurrency to carry out.But, because affairs have the ACID attribute, be atomicity (Atomicity), consistance (Consistency), isolation (Isolation), persistence (Durability), perhaps exactly, because affairs have isolation, make that the effect of independent execution of affairs and concurrent execution is duplicate.
Affairs control journal format:
0:len:1:TRANSACTION_BEGIN:tid
0:len:1:TRANSACTION_COMMIT:tid
0:len:1:TRANSACTION_ROLLBACK:tid
In affairs control journal format, lsn is set to 0, and sync is set to 1.
Checkpoint daily record index knowledge checkpoint begins the daily record with the finish time.Registration CHECK_POINT_BEGIN daily record was registered the CHECK_POINT_END daily record after finishing when the checkpoint began.
The checkpoint journal format:
0:len:1:CHECK_POINT_BEGIN:cpn
0:len:1:CHECK_POINT_END:cpn
In the journal format of checkpoint, lsn is set to 0, and sync is set to 1.
Similar with the daily record of DML statement bracket, affairs control daily record and checkpoint daily record all are the daily record brackets, promptly all comprise a beginning daily record and a corresponding end log.These three kinds aim at writing down the op_BEGIN daily record before practical operation begins day, after practical operation is finished according to the daily record of execution result end of record (EOR), as INSERT_END, INSERT_ABORT, TRANSACTION_COMMIT, TRANSACTION_ROLLBACK, CHECK_POINT_END etc.
In addition, the start-stop that the checkpoint daily record only indicates checkpointed does not constantly have in logic relation of inclusion with affairs control daily record or sql command daily record.The checkpoint daily record is mainly used to do system failure recovery.Affairs control daily record is used for supporting transaction failure to recover, and participation system fault recovery and media failure recover.So affairs control daily record needs when not only system moves, and all needs when fault recovery or system restart.
See also Fig. 1 and Fig. 2, wherein, Fig. 1 is the schematic flow sheet of the present invention's mixing daily record generation method, and Fig. 2 is the idiographic flow synoptic diagram of Fig. 1.
Mixing daily record generation method may further comprise the steps:
S1, open or create the mixing journal file;
S2, receive user command, and judge the type of described user command;
The type of S3, the user command that draws according to described judgement is registered corresponding command type daily record.
The type that the user sends order comprises DML sql command, transaction commands or checkpoint order, its corresponding DML sql command daily record respectively, affairs control daily record and checkpoint daily record.
The idiographic flow that mixes daily record generation method may further comprise the steps:
101, open or create one and mix journal file;
102, receive user command;
103, judge whether it is the DML sql command, if, carry out 104, otherwise, carry out 109;
104, register the op_BEGIN daily record of this order;
105, registration DML basic operation daily record;
106, carry out the DML basic operation;
107, judge whether next bar DML basic operation in addition, if, carry out 105, otherwise, carry out 108;
108, register the op_END daily record of this order, carry out 115 again;
109, judge whether it is transaction commands, if, carry out 110, otherwise, carry out 111;
110, registration correspondent transaction command log carries out 115 again;
111, judge whether it is the checkpoint order, if carry out 112, otherwise carry out 115;
112, registration CHECK_POINT_BEGIN daily record;
113, carry out the checkpoint process;
114, registration CHECK_POINT_END daily record;
115, judge whether next command in addition, if carry out 103, otherwise carry out 116;
116, wait for user input commands.
By above step as can be known, for the DML sql command, registration op_BEGIN daily record earlier before command execution, in the process of implementation, during DML basic operation of every beginning, its DML basic operation daily record of registration earlier executes after all DML basic operations, according to the net result of carrying out, if failure or abnormal ending, registration op_ABORT daily record, otherwise success or normal termination, registration op_END daily record; Except the querying command of SELECT and so on is not required, the execution of all the other DML orders all will produce the sql command daily record.
For transaction commands, only need registration corresponding daily record affairs initiation command TRANSACTION_BEGIN, affairs submiting command TRANSACTION_COMMIT or transaction retraction order TRANSACTION_ABORT before carrying out corresponding order.If create when mixing journal file, the log buffer of distribution district is arranged, then after finishing registration TRANSACTION_COMMIT or TRANSACTION_ABORT, log buffer district data are brushed into the mixing journal file.
For the checkpoint order, the checkpoint process is then carried out in registration CHECK_POINT_BEGIN daily record earlier, registers the CHECK_POINT_END daily record at last.
Described daily record generation method is followed and is write formula daily record (Write-Ahead Logging is called for short WAL) principle in advance.
In the memory database system operational process, system can produce the mixing journal file that a sequence numbers in order, and manually the automatic checkpointed of initiating of checkpointed of initiating or system all will produce or updating memory database image file.Before each checkpointed, all do not finish the journal file side-play amount offset of affairs minimum and the journal file log_file_no of current use to write down current time.Wherein, offset is meant when article one affairs control daily record TRANSACTION_BEGIN that all ongoing offices produce writes journal file that with respect to the minimum value in the side-play amount of file header, it calculates with byte.By the time after checkpointed finishes, these two values are write respectively in the header structure of image file, simultaneously the header structure of image file is flushed in the permanent storage, perhaps remember in the affairs control documents.
Adopt the generation method of mixing daily record of the present invention, just do not carry out when checkpointed does not need issued transaction to remain static, it can be carried out with transaction concurrency.This mode belongs to the dynamic backup mode, and the image file of its generation not necessarily is in coherency state, so need assist with journal file, promptly is aided with and mixes journal file and carry out that affairs are reformed and rollback, guarantees the coherency state of data with this.
See also Fig. 3, it is the specific implementation process flow diagram of daily record registration among Fig. 1.
The daily record registration is finished by daily record registration function.Daily record registration function comprises that at least two are gone into ginseng, i.e. daily record length and daily record volume data.If the log buffer of distribution district is arranged, the specific implementation of daily record registration may further comprise the steps:
201, receive daily record registration request;
202, whether the audit log buffer area has enough spaces to hold this daily record, if, carry out 204, otherwise, carry out 203;
203, the buffer area data are all outputed to the mixing journal file, empty buffer area, carry out 204;
204, check the size that aims at side-play amount and current journal file in the buffer area this day, the two addition is obtained the real offset that this daily record begins to locate with respect to file in the future in mixing journal file;
205, write this daily record to the log buffer district;
206, return success the byte number that writes and aim at mixing real offset in the journal file this day to caller.
If do not distribute the log buffer district, the specific implementation of daily record registration only need be checked the side-play amount that this day aims at mixing in the journal file and the size of current journal file, and the two addition is obtained aiming at mixing in the journal file with respect to mixing the real offset that journal file begins to locate in described day; Write described daily record to mixing journal file; Return success byte number that writes and the actual shifts that aimed at mixing in the journal file in described day to caller.
Because transaction concurrency is carried out,,, deposit but intersect so in journal file, these daily records can't be deposited continuously according to its logical organization so sql command daily record and other daily records all may produce registration at any time at any time.But every daily record is complete depositing, and can therefrom not got involved by other daily records and interrupt.
See also Fig. 4, it is the process flow diagram that the present invention is used for the mixing daily record analytic method of system failure recovery.
The present invention adopts scan edge journal file limit to generate the method for REDO/UNDO transaction block, only need scan the parsing that just can finish whole journal file one time.The set of all daily records that affairs are comprised calls a REDO/UNDO transaction block (REDO/UNDO Transaction Block).The set of the sql command statement that affairs are comprised calls the sql command statement block, is called for short statement block.Bi He REDO/UNDO transaction block is initial by TRANSACTION BEGIN, and with TRANSACTION COMMIT or TRANSACTION ROLLBACK ending, the centre comprises statement blocks such as several DELETE, UPDATE or INSERT.
In resolving, run into TRANSACTION COMMIT or TRANSACTIONROLLBACK, there is closed REDO/UNDO transaction block node in expression.
Be used for the mixing daily record analytic method of system failure recovery, it may further comprise the steps:
After a REDO/UNDO transaction block forms, then carry out the REDO/UNDO operation immediately.
301, daily record resolver handle hp of application, read-only mode is opened the mixing journal file.Earlier image file is written into internal memory, find the deviation post of the daily record that first mixing journal file and article one will read then according to the journal file of aforesaid minimum skew offset and journal file log_file_no value, read-only mode is opened the mixing journal file;
302, judge whether to exist closed REDO/UNDO transaction block node, if, carry out 303, otherwise, carry out 304;
303, REDO/UNDO transaction block node pointer is composed to output variable, returned and obtain REDO/UNDO transaction block success status value;
304, read the log lines data, and judge whether to run into and read document lending or end-of-file mark, if, then carry out 305, otherwise, carry out 306;
305, judge whether it is to run into end-of-file mark, if, return the state value that expression runs into end-of-file mark, otherwise, concrete error status value returned;
306, resolve this log lines, and judge whether to resolve successfully, if carry out 307, otherwise return concrete error status value;
307, judge whether this log lines is TRANSACTION_BEGIN, if, carry out 308, otherwise, carry out 309;
308, generate a REDO/UNDO transaction block node, add it to daily record resolver handle hp, and carry out 304;
309, judge whether this log lines is op_BEGIN, if, carry out 310, otherwise, carry out 311;
310, generate a statement block node, be inserted in the affiliated REDO/UNDO transaction block node, and carry out 304;
311, judge whether this log lines is TRANSACTION_END, if, carry out 312, otherwise, carry out 313;
312, find the REDO/UNDO transaction block node of coupling, this log lines is inserted, and put its state and be closed sign, and carry out 303;
313, judge whether to find the statement block node of coupling, if, carry out 315, otherwise, carry out 314;
314, return concrete error status value;
315, this log lines is inserted, and judged whether this log lines is op_END, if, carry out 316, otherwise, carry out 304;
316, this statement block is changed to closure state, and carries out 304.
By above step as can be known, in the resolving, before returning the sign that runs into the end of file, be accompanied by the process of reading journal file and resolving log lines, up to returning a REDO/UNDO transaction block success status value.These REDO/UNDO transaction block constitute a formation, the REDO formation when being exactly system failure recovery.Described REDO formation may comprise the REDO/UNDO transaction block with the ROLLBACK ending, need handle according to UNDO, and all the other are all handled according to REDO.After all journal file dissection process finish, may remain some among the daily record resolver hp in addition and not have closed REDO/UNDO transaction block, use this moment and force taking-up mode backward to obtain each REDO/UNDO transaction block, UNDO formation when having constituted system failure recovery thus, and handle according to UNDO.In the system failure recovery process, handle the REDO formation, handle the UNDO formation again, this moment, database recovery was to up-to-date coherency state.
See also Fig. 5, it is the process flow diagram that the present invention is used for the synchronous mixing daily record analytic method of non-real-time data.
Non-real-time data is synchronous, promptly regular or irregular data sync.The mixing daily record analytic method that non-real-time data is synchronous, much the same with the analytic method of system failure recovery, its difference only is that the target of the mixing daily record analytic method that non-real-time data is synchronous is the synchronous SQL statement that obtains according to the statement block generation.The step difference is: step 403 is obtained current statement block, generates the synchronous SQL statement of next bar according to current statement block, and internal pointer is displaced to next bar be untreated log lines or next statement block node.Judge whether to generate synchronous SQL statement again, if, return and obtain synchronous SQL statement success status value, otherwise, concrete error status value returned.
For fault recovery, only need return and obtain REDO/UNDO transaction block success status value and get final product; And for data sync, at first obtain next bar log lines pointer of current statement block node, generate the synchronous SQL statement of next bar, simultaneously internal pointer is displaced to the untreated log lines of next bar, if run into op_END or op_ABORT log lines, then be displaced to next statement block node, return the SQL statement text string of generation at last.
For the multinode database application system, the database of different nodes may need to carry out regular or irregular data sync, and these Database Systems are isomorphism not necessarily, may be Oracle or the such traditional database of Sybase such as goal systems.Use the mixing journal file to resolve and generate the synchronous SQL statement text string that meets goal systems SQL grammer.
Compared with prior art, the present invention's mixing daily record generation method and analytic method adopt the mixing daily record that mixes journal format to combine all information of transaction journal and synchronization log, each SQL statement daily record comprises to get over is engaged in recovering and the required full detail of data sync, thereby guaranteed that the mixing daily record both can serve as transaction journal, can serve as the data sync daily record again, a lot of redundant informations have been eliminated, make whole daily record scale descend greatly, not only save file storage, and reduced memory cache space and the cpu resource that consumes when generating these daily records.
And the present invention's mixing daily record generation method is supported the concurrent execution of affairs, and the concurrent execution of affairs and checkpoint process, improves the availability and the handling capacity of system greatly.And, only need be aided with and mix that journal file carries out that affairs are reformed and rollback guarantees the coherency state of data.
Moreover the present invention's mixing daily record analytic method adopts scan edge journal file limit to generate the method for REDO/UNDO transaction block, daily record is carried out one time scan parsing and the processing that just can finish whole journal file, has improved resolution speed, has simplified treatment scheme.And, unified the construction process of REDO formation and UNDO formation, when REDO/UNDO transaction block closure, just carry out the REDO/UNDO operation or generate synchronous SQL statement, disposing discharges the shared memory source of this REDO/UNDO transaction block immediately.
Only be preferred case study on implementation of the present invention below, be not limited to the present invention, for a person skilled in the art, the present invention can have various changes and variation.Within the spirit and principles in the present invention all, any modification of being done, etc. Q with replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1, a kind of mixing daily record generation method is characterized in that, may further comprise the steps:
Open or create the mixing journal file;
Receive user command, and judge the type of described user command;
The type of the user command that draws according to described judgement is registered corresponding command type daily record.
2, mixing daily record generation method according to claim 1 is characterized in that, the corresponding command type daily record of described registration may further comprise the steps:
Receive daily record registration request, judge whether described log buffer district has enough spaces to hold described daily record,, then the data of described buffer area are write with a brush dipped in Chinese ink described mixing journal file, and empty buffer area if there are not enough spaces;
Check the size that aims at described day side-play amount and described mixing journal file in the described buffer area, the two addition is obtained aiming at described day real offset in the described mixing journal file;
Described daily record is write described buffer area, and return success byte number and the described real offset that writes.
3, mixing daily record generation method according to claim 1, it is characterized in that the type that described judgement draws user command is the checkpoint order, then registers described checkpoint command log, and follow the execution of described checkpoint order, the execution of described checkpoint order and transaction concurrency are carried out.
4, according to claim 1,2 or 3 described mixing daily record generation methods, it is characterized in that, the log lines beginning structure of described mixing daily record is lsn:len:sync:op_code, described lsn represents the log lines sequence number, be used for illustrating the order that the log lines order begins to carry out, described len represents except except the shared length of described lsn and len, the length of described log lines, described sync represents whether described log lines is that needs produce synchronization log, described op_code represents described log lines command code, is used to discern the type of described log lines, described colon: be separator.
5, mixing daily record generation method according to claim 4 is characterized in that described lsn is unique in the database overall situation, and described sync is the asynchronous daily record of 0 expression, is non-0 expression synchronization log, and the coding span of described op_code is 1 to 255.
6, according to claim 1,2 or 3 described mixing daily record generation methods, it is characterized in that, the form of described mixing daily record comprises DML statement bracket journal format, DML basic operation journal format, affairs control journal format and checkpoint journal format, the op_code of described DML statement bracket journal format, op are the DML order, and code is BEGIN, END or ABORT, the op_code of described DML basic operation journal format is DELETE_VFIELD, INSERT_VFIELD, UPDATE_FIELD, DELETE_RECORD, INSERT_RECORD or FREE_PAGE, the lsn of described affairs control journal format is 0, and sync is 1, and op is TRANSACTION, code is BEGIN, COMMIT or ROLLBACK, the lsn of described checkpoint journal format is 0, sync is 1, op is CHECK_POINT, and code is BEGIN or END.
7, a kind of mixing daily record analytic method is characterized in that, may further comprise the steps:
Steps A, application date book resolver handle are opened the mixing journal file;
Step B, judge whether to exist closed REDO/UNDO transaction block node, if, execution in step C, otherwise, execution in step D, described REDO/UNDO transaction block is represented the set of all daily records that affairs comprise, and described closed REDO/UNDO transaction block is initial by TRANSACTION_BEGIN, with TRANSACTION_COMMIT or TRANSACTION_ROLLBACK ending;
Step C, described REDO/UNDO transaction block node pointer is composed to output variable, returned and obtain REDO/UNDO transaction block success status value;
Step D, resolve to mix the log lines of journal file successively, carry out the operation that generates described REDO/UNDO transaction block, again execution in step B.
8, mixing daily record analytic method according to claim 7, it is characterized in that, carrying out the operation that generates described REDO/UNDO transaction block among the described step D may further comprise the steps: it is TRANSACTION_BEGIN that parsing runs into log lines, generate REDO/UNDO transaction block node, judge the affiliated command type of log lines more successively, running into log lines is TRANSACTION_END, and puts described REDO/UNDO transaction block node and be closed sign.
9, according to claim 7 or 8 described mixing daily record analytic methods, it is characterized in that, described REDO/UNDO transaction block comprises statement block, described statement block begins with op_BEGIN, closed statement block begins with op_BEGIN, the op_END ending, op is the DML order, described step D resolves the log lines of mixing journal file successively and comprises: it is op_BEGIN that parsing runs into described log lines, generated statement agllutination point, judge command type under the log lines more successively, running into log lines is op_END, and puts described REDO/UNDO transaction block node and be closed sign.
10, mixing daily record analytic method according to claim 9, it is characterized in that, described step C replaces with: obtain current statement block, generate the synchronous SQL statement of next bar according to current statement block, internal pointer is displaced to next bar be untreated log lines or next statement block node, and judges whether to generate synchronous SQL statement, if, return and obtain synchronous SQL statement success status value, otherwise, concrete error status value returned.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2008102418990A CN101464890B (en) | 2008-12-30 | 2008-12-30 | Mixed log generation method and resolution method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2008102418990A CN101464890B (en) | 2008-12-30 | 2008-12-30 | Mixed log generation method and resolution method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101464890A true CN101464890A (en) | 2009-06-24 |
| CN101464890B CN101464890B (en) | 2012-05-23 |
Family
ID=40805465
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2008102418990A Expired - Fee Related CN101464890B (en) | 2008-12-30 | 2008-12-30 | Mixed log generation method and resolution method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101464890B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102385624A (en) * | 2011-10-25 | 2012-03-21 | 曙光信息产业(北京)有限公司 | DFS (distributed file system)-oriented log data organization method |
| CN102970373A (en) * | 2012-12-03 | 2013-03-13 | 亚信联创科技（中国）有限公司 | Data synchronization method, device and system |
| CN106528607A (en) * | 2016-09-26 | 2017-03-22 | 武汉开目信息技术股份有限公司 | A method for guaranteeing consistency after separation of attribute data and file data in a PDM system |
| CN107679195A (en) * | 2017-10-09 | 2018-02-09 | 杭州安恒信息技术有限公司 | Half master-slave database synchronous method and system based on the transmission of HTTPS agreements |
| CN107766490A (en) * | 2017-10-17 | 2018-03-06 | 链家网（北京）科技有限公司 | Automatic operation method, system, equipment and the storage medium of SQL files |
| CN108073500A (en) * | 2016-11-14 | 2018-05-25 | 深圳联友科技有限公司 | A kind of log recording method and system |
| CN108647112A (en) * | 2018-03-16 | 2018-10-12 | 阿里巴巴集团控股有限公司 | The method, apparatus and distributing real time system system of data backup |
| CN110019062A (en) * | 2017-08-14 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Method of data synchronization and system |
| CN110121694A (en) * | 2017-12-05 | 2019-08-13 | 华为技术有限公司 | A kind of blog management method, server and Database Systems |
| CN110231909A (en) * | 2019-05-15 | 2019-09-13 | 广州视源电子科技股份有限公司 | Write the treating method and apparatus of operation |
| CN110633258A (en) * | 2019-08-07 | 2019-12-31 | 平安科技（深圳）有限公司 | Log insertion method, device, computer device and storage medium |
| CN110780857A (en) * | 2019-10-23 | 2020-02-11 | 杭州涂鸦信息技术有限公司 | Unified log component |
| CN112256656A (en) * | 2019-07-22 | 2021-01-22 | 中兴通讯股份有限公司 | Transaction rollback method and device, database, system and computer storage medium |
-
2008
- 2008-12-30 CN CN2008102418990A patent/CN101464890B/en not_active Expired - Fee Related
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102385624A (en) * | 2011-10-25 | 2012-03-21 | 曙光信息产业(北京)有限公司 | DFS (distributed file system)-oriented log data organization method |
| CN102970373A (en) * | 2012-12-03 | 2013-03-13 | 亚信联创科技（中国）有限公司 | Data synchronization method, device and system |
| CN102970373B (en) * | 2012-12-03 | 2015-11-11 | 亚信科技（中国）有限公司 | A kind of method of data synchronization, Apparatus and system |
| CN106528607A (en) * | 2016-09-26 | 2017-03-22 | 武汉开目信息技术股份有限公司 | A method for guaranteeing consistency after separation of attribute data and file data in a PDM system |
| CN106528607B (en) * | 2016-09-26 | 2019-10-29 | 武汉开目信息技术股份有限公司 | Attribute data separates the ensuring method of rear consistency with file data in PDM system |
| CN108073500A (en) * | 2016-11-14 | 2018-05-25 | 深圳联友科技有限公司 | A kind of log recording method and system |
| CN110019062A (en) * | 2017-08-14 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Method of data synchronization and system |
| CN107679195A (en) * | 2017-10-09 | 2018-02-09 | 杭州安恒信息技术有限公司 | Half master-slave database synchronous method and system based on the transmission of HTTPS agreements |
| CN107766490A (en) * | 2017-10-17 | 2018-03-06 | 链家网（北京）科技有限公司 | Automatic operation method, system, equipment and the storage medium of SQL files |
| CN110121694A (en) * | 2017-12-05 | 2019-08-13 | 华为技术有限公司 | A kind of blog management method, server and Database Systems |
| CN110121694B (en) * | 2017-12-05 | 2020-08-07 | 华为技术有限公司 | Log management method, server and database system |
| US11748215B2 (en) | 2017-12-05 | 2023-09-05 | Huawei Technologies Co., Ltd. | Log management method, server, and database system |
| CN108647112A (en) * | 2018-03-16 | 2018-10-12 | 阿里巴巴集团控股有限公司 | The method, apparatus and distributing real time system system of data backup |
| CN110231909A (en) * | 2019-05-15 | 2019-09-13 | 广州视源电子科技股份有限公司 | Write the treating method and apparatus of operation |
| CN112256656A (en) * | 2019-07-22 | 2021-01-22 | 中兴通讯股份有限公司 | Transaction rollback method and device, database, system and computer storage medium |
| CN110633258A (en) * | 2019-08-07 | 2019-12-31 | 平安科技（深圳）有限公司 | Log insertion method, device, computer device and storage medium |
| CN110633258B (en) * | 2019-08-07 | 2022-08-12 | 平安科技（深圳）有限公司 | Log insertion method, device, computer device and storage medium |
| CN110780857A (en) * | 2019-10-23 | 2020-02-11 | 杭州涂鸦信息技术有限公司 | Unified log component |
| CN110780857B (en) * | 2019-10-23 | 2024-01-30 | 杭州涂鸦信息技术有限公司 | Unified log component |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101464890B (en) | 2012-05-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101464890B (en) | Mixed log generation method and resolution method | |
| US5878410A (en) | File system sort order indexes | |
| Graefe | Sorting And Indexing With Partitioned B-Trees. | |
| US6411964B1 (en) | Methods for in-place online reorganization of a database | |
| DeWitt et al. | Implementation techniques for main memory database systems | |
| US6263338B1 (en) | Method relating to databases | |
| US7117229B2 (en) | Method and system for online reorganization of databases | |
| CN101639848B (en) | Spatial data engine and method applying management spatial data thereof | |
| Mohan et al. | Algorithms for creating indexes for very large tables without quiescing updates | |
| US8560500B2 (en) | Method and system for removing rows from directory tables | |
| Graefe | A survey of B-tree logging and recovery techniques | |
| CN101876996A (en) | Method and system for synchronizing data from main memory data bank to file data bank | |
| WO2007083371A1 (en) | Data integration device, method, and recording medium containing program | |
| CN101853288A (en) | Configurable full-text retrieval service system based on document real-time monitoring | |
| CN101136016A (en) | Indexes on-line updating method of full text retrieval system | |
| JP4575762B2 (en) | Data processing method and apparatus, storage apparatus and processing program therefor | |
| CN102012849A (en) | Flash memory-based database restoring method | |
| Ghanem et al. | Bulk operations for space-partitioning trees | |
| CN100520781C (en) | Main memory data centre data processing method | |
| CN112463447B (en) | Optimization method for realizing physical backup based on distributed database | |
| Liu et al. | Incorporating transaction semantics to reduce reprocessing overhead in replicated mobile data applications | |
| CN103685350B (en) | The synchronous method of storage system and relevant equipment | |
| CN109284276A (en) | A kind of database accelerated method based on big data framework | |
| Pollari-Malmi et al. | Concurrency control for B-trees with differential indices | |
| Graefe et al. | Logical recovery from single-page failures |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20120523 Termination date: 20171230 |