WO2018184376A1 - Relational database log extraction method and device, and storage medium - Google Patents

Relational database log extraction method and device, and storage medium Download PDF

Info

Publication number
WO2018184376A1
WO2018184376A1 PCT/CN2017/107110 CN2017107110W WO2018184376A1 WO 2018184376 A1 WO2018184376 A1 WO 2018184376A1 CN 2017107110 W CN2017107110 W CN 2017107110W WO 2018184376 A1 WO2018184376 A1 WO 2018184376A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
read
data block
log record
log
Prior art date
Application number
PCT/CN2017/107110
Other languages
French (fr)
Chinese (zh)
Inventor
刘赛
聂庆节
刘军
刘嘉华
张磊
马悦皎
周忠冉
Original Assignee
南京南瑞集团公司
南京南瑞信息通信科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 南京南瑞集团公司, 南京南瑞信息通信科技有限公司 filed Critical 南京南瑞集团公司
Publication of WO2018184376A1 publication Critical patent/WO2018184376A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems

Definitions

  • the invention relates to a relation database log extraction method, device and storage medium, and belongs to the technical field of database data replication.
  • Database log extraction technology is widely used in the field of information system integration and interaction. It mainly uses logic-level log extraction technology. Most of these data extraction technologies rely on the plug-ins provided by the database to realize change data capture. The data extraction delay is large. The extraction process will have a certain impact on the performance of the database itself. At present, the traditional logic-based log extraction technology can not solve the problem of database log extraction efficiency, and has a great impact on the production library.
  • the embodiment of the present invention provides a relational database log extraction method, device and storage medium, which can accurately extract change log data, reduce the impact of database log extraction on the performance of the database itself, and improve the security of the database operation. And stability.
  • an embodiment of the present invention provides a method for extracting a relational database log, including:
  • log record information When the log record information is incomplete, the log record information is read from the next data block until it is assembled into complete log record information;
  • the valid change data in all the change operation data is added to the data result set.
  • the log file storage format in the log file information includes a database file system format, a raw device format, and an ASM format.
  • S3 s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
  • the reading the log record information from the current data block includes:
  • Read the log record in the data block obtain the log record header identifier bit; determine whether the header identifier bit is 0, if yes, read the next data block; if not, obtain the log record valid information, determine the header
  • the number of bits in the bit is identified, and the header bit is parsed according to the number of bits, the length of the log record information is calculated, and the log record information is read according to the length of the log record information.
  • adding the valid change data in all the change operation data to the data result set includes:
  • the obtaining the change operation data in the log record information includes:
  • the binary data in the change operation is read according to the operation code, which is the change operation data.
  • the redo operation OP_UNDO_UNDO corresponds to the operation code 0x0501
  • the start new transaction operation OP_UNDO_BEGINTRANS corresponds to the operation code 0x0502
  • the transaction commit operation OP_UNDO_COMMIT corresponds to the operation code 0x0504
  • the new session operation OP_UNDO_NEWSESSION corresponds to the operation code 0x0513
  • the switching session operation OP_UNDO_SWITCHSESSION corresponds to the operation code 0x0514
  • Insert row operation OP_ROW_INSERT corresponding to opcode 0x0b02
  • delete row operation OP_ROW_DELETE corresponding opcode 0x0b03
  • lock row operation OP_ROW_LOCK corresponding opcode 0x0b04
  • row link operation OP_ROW_CHAIN corresponding opcode 0x0b06
  • the embodiment of the invention further provides a relational database log extraction device, comprising:
  • a log file obtaining unit configured to obtain database log file information
  • a reading unit configured to read a data block and read log record information from the current data block
  • the assembling unit is configured to read the log record information from the next data block until the log record information is incomplete, until the complete log record information is assembled;
  • a data obtaining unit configured to acquire all changed operation data in the log record information based on the complete log record information
  • Add a unit configured to add valid change data from all change operation data to the data result set.
  • the reading unit is specifically configured to determine the relationship between the SEQ numbers before and after the data block, and set the SEQ number of the current data block to s1, and the SEQ number of the next data block to s2.
  • the SEQ number of the database is s3;
  • S3 s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
  • the reading unit is specifically configured to read a log record in the data block, obtain a log record header identifier bit, determine whether the header identifier bit is 0, and if so, read the next data block; If not, the valid information of the log record is obtained, the number of bits of the header identifier is determined, and the header identifier is parsed according to the number of bits, the length of the log record information is calculated, and the log record information is read according to the length of the log record information.
  • the adding unit is specifically configured to check whether the change operation data is valid change data. If the data is not valid change, the change operation data is discarded, and the next change operation data is read; if it is valid change data, Then add it to the data result set.
  • the data acquiring unit is configured to read a change operation in the log record information, and obtain a change operation header identifier bit;
  • the binary data in the change operation is read according to the operation code, which is the change operation data.
  • the embodiment of the invention further provides a storage medium storing an executable program, and when the executable program is executed by the processor, the foregoing relational database log extraction method is implemented.
  • the beneficial effects achieved by the embodiments of the present invention are: by analyzing and locating the database log, extracting the log change data at the first time of the database log change, and generating a log data file in a customized format;
  • the change log data can be accurately extracted, and the performance impact of the database log extraction on the database itself can be reduced, which has important practical significance for ensuring the safe and stable operation of the database.
  • FIG. 1 is a schematic flowchart of a basic processing procedure of a relational database log extraction method according to an embodiment of the present invention
  • 2A is a schematic flowchart of a detailed processing process of a relational database log extraction method according to an embodiment of the present invention
  • 2B is a schematic flowchart of a detailed processing process of a relational database log extraction method according to an embodiment of the present invention
  • 2C is a schematic flowchart of a detailed processing process of a relational database log extraction method according to an embodiment of the present invention
  • FIG. 3 is an architectural diagram corresponding to a method for extracting a relational database log according to an embodiment of the present invention
  • FIG. 4 is a schematic structural diagram of a structure of a relational database log extracting apparatus according to an embodiment of the present invention.
  • the A node in FIG. 2A is connected to the A node in FIG. 2B.
  • the B-node in FIG. 2B is connected to the Node B in FIG. 2C.
  • the processing flow of the relational database log extraction method provided by the embodiment of the present invention includes the following steps:
  • Step 1 Obtain database log file information.
  • the database log file information includes: a log file path and a log file storage format.
  • the Redo Read process (RedoReader), the database connection, and the log file path and the log file storage format are initialized;
  • the log file storage format includes a database file system format, a raw device format, and an ASM format.
  • Step 2 reading a data block
  • the log file is opened, the database is read, and the size and number of the data blocks are obtained to determine the database version information.
  • S3 s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
  • Step 3 Read the log record information from the current data block.
  • the initialization log record resolution process (RecordParse)
  • the change log resolution process (ChangeParse);
  • the log record header (RecordHead); determine whether the header flag is 0 (that is, whether there is a waste), and if so, read the next block; if not Then, the valid information (Valid) of the log record is obtained, the number of bits of the header identifier bit is determined, and the header identifier bit is parsed according to the number of bits, the length of the log record information is calculated, and the log record information is read according to the length of the log record information.
  • the valid information (Valid) of the log record is obtained
  • the number of bits of the header identifier bit is determined, and the header identifier bit is parsed according to the number of bits, the length of the log record information is calculated, and the log record information is read according to the length of the log record information.
  • the part identification bit is 68 bits, otherwise the part identification bit is 24 bits.
  • Log record length log record length - log record header bit length.
  • Step 4 When the log record information is incomplete, the log record information is read from the next data block until the complete log record information is assembled.
  • Step 5 Obtain all change operation data in the log record information based on the complete log record information.
  • the complete log record information is parsed, the operation code is obtained, and all the change operation data in the log record information are sequentially read according to the operation code.
  • reading a change operation in the log record information obtaining a change operation header flag (Change Head); obtaining an operation code from the change operation header flag bit, and reading Change the length of each segment in the operation; read the binary data in the change operation according to the operation code, the binary data is the change operation data, and put it into the record value (Record Value); check the change operation in the log record information Have you read it, if not, continue reading until you read all the change operation data.
  • the operation code is as follows: redo operation OP_UNDO_UNDO corresponding operation code 0x0501, start new transaction operation OP_UNDO_BEGINTRANS corresponding operation code 0x0502, transaction commit operation OP_UNDO_COMMIT corresponding operation code 0x0504, new session operation OP_UNDO_NEWSESSION corresponding operation code 0x0513, switching session operation OP_UNDO_SWITCHSESSION corresponding operation code 0x0514 Insert row operation OP_ROW_INSERT corresponding to opcode 0x0b02, delete row operation OP_ROW_DELETE corresponding opcode 0x0b03, lock row operation OP_ROW_LOCK corresponding opcode 0x0b04, update row operation OP_ROW_UPDATE corresponding opcode 0x0b05, row link operation OP_ROW_CHAIN corresponding opcode 0x0b06, program operation OP_ROW_MFC Operation code 0x0
  • step 6 the valid change data in all the change operation data is added to the data result set.
  • the change operation data is valid change data. If it is not the valid change data, the change operation data is discarded, and the next change operation data is read; if it is valid change data, it is added to Data result set; after checking all change operation data, one data extraction ends.
  • the above method uses a multi-process method to extract log data from the online log, and each extracted data is stored in the data result set.
  • the online log extraction call relationship to the database table is as shown in Table 1. Show.
  • the log change data is extracted at a time, and the change log data set is generated, which provides a basis for subsequent data format conversion and log storage.
  • This method can not only accurately extract the change data, but also reduce the performance impact of database log extraction on the production library, which has important practical significance for ensuring the safe and stable operation of the information disaster recovery system.
  • the embodiment of the invention further provides a schematic structural diagram of a relational database log extraction device, as shown in FIG. 4, which includes:
  • the log file obtaining unit 10 is configured to obtain database log file information.
  • the reading unit 20 is configured to read the data block and read the log record information from the current data block;
  • the assembling unit 30 is configured to read the log record information from the next data block until the log record information is incomplete, until the complete log record information is assembled;
  • the data obtaining unit 40 is configured to acquire all the changed operation data in the log record information based on the complete log record information;
  • the adding unit 50 is configured to add valid change data in all the change operation data to the data result set.
  • the reading unit 20 is specifically configured to determine the relationship between the SEQ numbers before and after the data block, and set the SEQ number of the current data block to s1, the SEQ number of the next data block to s2, and the SEQ number of the database is S3;
  • S3 s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
  • the reading unit 20 is configured to read a log record in the data block, obtain a log record header identifier bit, determine whether the header identifier bit is 0, and if yes, read the next data block; If not, the valid information of the log record is obtained, the number of bits of the header identifier is determined, and the header identifier is parsed according to the number of bits, the length of the log record information is calculated, and the log record information is read according to the length of the log record information.
  • the adding unit 50 is specifically configured to check whether the change operation data is The data is effectively changed. If it is not valid change data, the change operation data is discarded, and the next change operation data is read; if it is valid change data, it is added to the data result set.
  • the data obtaining unit 40 is configured to read a change operation in the log record information, and obtain a change operation header identifier bit;
  • the binary data in the change operation is read according to the operation code, which is the change operation data.
  • the functions of the log file obtaining unit 10, the reading unit 20, the assembling unit 30, the data obtaining unit 40, and the adding unit 50 may be a general-purpose processor and a digital signal processor (DSP, in the relational database log extracting device. Digital Signal Processor), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • DSP digital signal processor
  • relational database log extraction apparatus provided by the foregoing embodiment is only illustrated by the division of each of the foregoing program modules when performing relational database log extraction. In actual applications, the foregoing processing may be assigned differently according to requirements.
  • the program module is completed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above.
  • the embodiment of the relational database log extraction device and the relational database log extraction method provided by the foregoing embodiments are in the same concept, and the specific implementation process is described in detail in the method embodiment, and details are not described herein again.
  • the embodiment of the invention further provides a storage medium storing an executable program, when the executable program is executed by the processor, executing:
  • log record information When the log record information is incomplete, the log record information is read from the next data block until it is assembled into complete log record information;
  • S3 s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
  • Read the log record in the data block obtain the log record header identifier bit; determine whether the header identifier bit is 0, if yes, read the next data block; if not, obtain the log record valid information, determine the header
  • the number of bits in the bit is identified, and the header bit is parsed according to the number of bits, the length of the log record information is calculated, and the log record information is read according to the length of the log record information.
  • the binary data in the change operation is read according to the operation code, which is the change operation data.
  • an embodiment of the present invention further provides a relational database log extracting apparatus, including: a processor and a memory for storing a computer program capable of running on the processor,
  • processor configured to execute when the computer program is executed:
  • log record information When the log record information is incomplete, the log record information is read from the next data block until it is assembled into complete log record information;
  • the non-volatile memory may be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), or an Erasable Programmable Read (EPROM). Only Memory), Electrically Erasable Programmable Read-Only Memory (EEPROM), Magnetic Random Access Memory (FRAM, Ferromagnetic Random Access Memory), Flash Memory, Magnetic Surface Memory, Optical Disk, or Compact Disc Read-Only Memory (CD-ROM); magnetic surface memory can be disk storage or tape storage .
  • the volatile memory can be a random access memory (RAM) that acts as an external cache.
  • RAM Random Access Memory
  • SRAM Static Random Access Memory
  • SSRAM Synchronous Static Random Access Memory
  • SSRAM Dynamic Random Access
  • DRAM Dynamic Random Access Memory
  • SDRAM Synchronous Dynamic Random Access Memory
  • DDRSDRAM Double Data Rate Synchronous Dynamic Random Access Memory
  • ESDRAM enhancement Enhanced Synchronous Dynamic Random Access Memory
  • SLDRAM Synchronous Dynamic Random Access Memory
  • DRRAM Direct Memory Bus Random Access Memory
  • the database log file information is acquired; the data block is read, and the log record information is read from the current data block; when the log record information is incomplete, the next data is obtained.
  • the log record information is read in the block until it is assembled into complete log record information; the log record information is obtained based on the complete log record information All change operation data in ; add valid change data from all change operation data to the data result set.
  • the log change data is extracted at the first time of the database log change, and a log data file of a custom format is generated; the method can not only accurately extract the change log data, but also reduce the database.
  • Log extraction affects the performance of the database itself, improving the security and stability of the database operation.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

Disclosed in the present invention is a relational database log extraction method, comprising: acquiring database log file information; reading a data block, and reading log record information from the current data block; when the log record information is not complete, reading log record information from the next data block, until complete log record information is obtained; on the basis of the complete log record information, acquiring all change operation data of the log record information; and adding effective change data among all of the change operation data into a data result set. Further disclosed in the present invention are a relational database log extraction device and a storage medium.

Description

一种关系数据库日志抽取方法、装置及存储介质Relational database log extraction method, device and storage medium
相关申请的交叉引用Cross-reference to related applications
本申请基于申请号为201710224035.7、申请日为2017年04月07日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请作为参考。The present application is filed on the basis of the Chinese Patent Application No. PCT Application No.
技术领域Technical field
本发明涉及一种关系数据库日志抽取方法、装置及存储介质，属于数据库数据复制技术领域。The invention relates to a relation database log extraction method, device and storage medium, and belongs to the technical field of database data replication.
背景技术Background technique
随着信息技术的深入发展，应用系统的不断升级，系统之间的集成贯通越来越多的需要数据交互，数据抽取已经逐步成为了信息系统融合的关键部分。然而，所有针对数据的抽取与交互都包含着一个基本的假设——数据的可用性和可靠性。数据库抽取技术作为各个领域的主流数据处理技术，其数据的可用性和与可靠性自然备受关注。数据库日志挖掘技术正是在这种要求下产生的。With the in-depth development of information technology, the application system is constantly upgraded, and the integration between systems requires more and more data interaction. Data extraction has gradually become a key part of information system integration. However, all extraction and interaction of data contains a basic assumption - the availability and reliability of the data. Database extraction technology is the mainstream data processing technology in various fields, and its data availability and reliability naturally attract attention. Database log mining technology is produced under this requirement.
国内外关系型数据库日志挖掘技术分为两种类型，一种是基于逻辑级日志抽取技术、一种是基于物理级重做日志Redo抽取技术。数据库日志抽取技术在信息系统集成交互领域应用广泛，主要使用的是基于逻辑级日志抽取技术，此类数据抽取技术多数需要依靠数据库自身提供的插件实现变化数据捕获，数据抽取延时较大，同时抽取过程中会对数据库自身的性能产生一定的影响，目前传统的基于逻辑级的日志抽取技术，不能较好的解决数据库日志抽取效率问题，并且对生产库影响较大。 There are two types of relational database log mining technologies at home and abroad, one is based on logical level log extraction technology, and the other is based on physical level redo log Redo extraction technology. Database log extraction technology is widely used in the field of information system integration and interaction. It mainly uses logic-level log extraction technology. Most of these data extraction technologies rely on the plug-ins provided by the database to realize change data capture. The data extraction delay is large. The extraction process will have a certain impact on the performance of the database itself. At present, the traditional logic-based log extraction technology can not solve the problem of database log extraction efficiency, and has a great impact on the production library.
发明内容Summary of the invention
为了解决上述技术问题，本发明实施例提供了一种关系数据库日志抽取方法、装置及存储介质，能够精确抽取出变化日志数据，降低数据库日志抽取对数据库自身的性能影响，提高数据库运行的安全性和稳定性。In order to solve the above technical problem, the embodiment of the present invention provides a relational database log extraction method, device and storage medium, which can accurately extract change log data, reduce the impact of database log extraction on the performance of the database itself, and improve the security of the database operation. And stability.
为了达到上述目的，本发明实施例提供一种关系数据库日志抽取方法，包括：In order to achieve the above object, an embodiment of the present invention provides a method for extracting a relational database log, including:
获取数据库日志文件信息；Obtain database log file information;
读取数据块，从当前数据块中读取日志记录信息；Reading a data block and reading log record information from the current data block;
在所述日志记录信息不完整时，从下一个数据块中读取日志记录信息，直到拼装成完整的日志记录信息；When the log record information is incomplete, the log record information is read from the next data block until it is assembled into complete log record information;
基于所述完整的日志记录信息获取日志记录信息中的所有变化操作数据；Obtaining all change operation data in the log record information based on the complete log record information;
间所有变化操作数据中的有效变化数据添加到数据结果集。The valid change data in all the change operation data is added to the data result set.
上述方案中，所述日志文件信息中的日志文件存储格式包括数据库文件系统格式、裸设备格式以及ASM格式。In the above solution, the log file storage format in the log file information includes a database file system format, a raw device format, and an ASM format.
上述方案中，读取数据块时，判断数据块前后序列号(SEQ)号之间的关系，设当前数据块的SEQ号为s1，下一个数据块的SEQ号为s2，数据库的SEQ号为s3，通过比较确定下一步处理方法：In the above solution, when the data block is read, the relationship between the sequence number (SEQ) of the data block is determined, and the SEQ number of the current data block is s1, the SEQ number of the next data block is s2, and the SEQ number of the database is S3, through comparison to determine the next processing method:
当s1＝s2时：When s1=s2:
s2＝s3，Redo正常读写，正常读取下一个数据块；S2=s3, Redo reads and writes normally, and reads the next data block normally;
s3>s2，Redo已被切换，当前Redo未完成，正常读取下一个数据块；S3>s2, Redo has been switched, the current Redo is not completed, and the next data block is read normally;
s3<s2，此情况不存在；S3<s2, this situation does not exist;
当s1<s2时：When s1<s2:
s3＝s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块； S3=s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
s3>s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块；S3>s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
s3<s1，此情况不存在；S3<s1, this situation does not exist;
当s2<＝s3、s1<＝s3、s1>s2时：When s2<=s3, s1<=s3, s1>s2:
s3＝s1，Redo下一个数据块暂时无数据写入，等待数据写入；S3=s1, Redo next data block has no data to write temporarily, waiting for data to be written;
s3＝s1+1，s1已读完，已切换到新的Redo，查询s1加1后对应的Redo，读取下一个数据块；S3=s1+1, s1 has been read, has been switched to the new Redo, query s1 plus 1 corresponding Redo, read the next data block;
s3>s1+1，s1已读完，至少两次切换到新的Redo，查询s1加1后对应归档，读取下一个数据块；S3>s1+1, s1 has been read, at least twice switched to the new Redo, query s1 plus 1 corresponding to the archive, read the next data block;
s3<s2，此情况不存在。S3<s2, this situation does not exist.
上述方案中，所述从当前数据块中读取日志记录信息，包括：In the above solution, the reading the log record information from the current data block includes:
读取数据块中的日志记录，获取日志记录头部标识位；判断头部标识位是否为0，如果是则读取下一个数据块；如果不是，则获取日志记录的有效信息，确定头部标识位的位数，并根据位数解析头部标识位，计算日志记录信息长度，根据日志记录信息长度读取日志记录信息。Read the log record in the data block, obtain the log record header identifier bit; determine whether the header identifier bit is 0, if yes, read the next data block; if not, obtain the log record valid information, determine the header The number of bits in the bit is identified, and the header bit is parsed according to the number of bits, the length of the log record information is calculated, and the log record information is read according to the length of the log record information.
上述方案中，所述将所有变化操作数据中的有效变化数据添加到数据结果集，包括：In the above solution, adding the valid change data in all the change operation data to the data result set includes:
检查变化操作数据是否为有效变化数据，如果不是有效变化数据，则丢掉所述变化操作数据，读取下一个变化操作数据；如果是有效变化数据，则将其添加到数据结果集中。Check whether the change operation data is valid change data. If it is not valid change data, the change operation data is discarded, and the next change operation data is read; if it is valid change data, it is added to the data result set.
上述方案中，所述获取日志记录信息中的所有变化操作数据，包括：In the above solution, the obtaining the change operation data in the log record information includes:
读取日志记录信息中的变化操作，获取变化操作头部标识位；Reading the change operation in the log record information, and obtaining the change operation header identifier bit;
从变化操作头部标识位中获取操作码，读取变化操作中各段长度；Obtain an operation code from the change operation header flag, and read the length of each segment in the change operation;
根据操作码读取变化操作中的二进制数据，该二进制数据即为变化操作数据。 The binary data in the change operation is read according to the operation code, which is the change operation data.
上述方案中，重做操作OP_UNDO_UNDO对应操作码0x0501、开始新事务操作OP_UNDO_BEGINTRANS对应操作码0x0502、事务提交操作OP_UNDO_COMMIT对应操作码0x0504、新会话操作OP_UNDO_NEWSESSION对应操作码0x0513、切换会话操作OP_UNDO_SWITCHSESSION对应操作码0x0514、插入行操作OP_ROW_INSERT对应操作码0x0b02、删除行操作OP_ROW_DELETE对应操作码0x0b03、锁定行操作OP_ROW_LOCK对应操作码0x0b04、更新行操作OP_ROW_UPDATE对应操作码0x0b05、行链接操作OP_ROW_CHAIN对应操作码0x0b06、程序操作OP_ROW_MFC对应操作码0x0b07、程序操作OP_ROW_CFA对应操作码0x0b08、程序操作OP_ROW_KCI对应操作码0x0b09、程序操作OP_ROW_SKL对应操作码0x0b0a、插入多行操作OP_ROW_MINSERT对应操作码0x0b0b、删除多行操作OP_ROW_MDELETE对应操作码0x0b0c、程序操作OP_ROW_LMN对应操作码0x0b10、程序操作OP_ROW_COM对应操作码0x1801。In the above solution, the redo operation OP_UNDO_UNDO corresponds to the operation code 0x0501, the start new transaction operation OP_UNDO_BEGINTRANS corresponds to the operation code 0x0502, the transaction commit operation OP_UNDO_COMMIT corresponds to the operation code 0x0504, the new session operation OP_UNDO_NEWSESSION corresponds to the operation code 0x0513, the switching session operation OP_UNDO_SWITCHSESSION corresponds to the operation code 0x0514, Insert row operation OP_ROW_INSERT corresponding to opcode 0x0b02, delete row operation OP_ROW_DELETE corresponding opcode 0x0b03, lock row operation OP_ROW_LOCK corresponding opcode 0x0b04, update row operation OP_ROW_UPDATE corresponding opcode 0x0b05, row link operation OP_ROW_CHAIN corresponding opcode 0x0b06, program operation OP_ROW_MFC corresponding operation Code 0x0b07, program operation OP_ROW_CFA corresponds to opcode 0x0b08, program operation OP_ROW_KCI corresponds to opcode 0x0b09, program operation OP_ROW_SKL corresponds to opcode 0x0b0a, insert multi-row operation OP_ROW_MINSERT corresponding opcode 0x0b0b, delete multi-row operation OP_ROW_MDELETE corresponding opcode 0x0b0c, program operation OP_ROW_LMN Corresponding to the operation code 0x0b10, the program operation OP_ROW_COM corresponds to the operation code 0x1801.
本发明实施例还提供一种关系数据库日志抽取装置，包括：The embodiment of the invention further provides a relational database log extraction device, comprising:
日志文件获取单元，配置为获取数据库日志文件信息；a log file obtaining unit configured to obtain database log file information;
读取单元，配置为读取数据块，从当前数据块中读取日志记录信息；a reading unit configured to read a data block and read log record information from the current data block;
拼装单元，配置为在所述日志记录信息不完整时，从下一个数据块中读取日志记录信息，直到拼装成完整的日志记录信息；The assembling unit is configured to read the log record information from the next data block until the log record information is incomplete, until the complete log record information is assembled;
数据获取单元，配置为基于所述完整的日志记录信息获取日志记录信息中的所有变化操作数据；a data obtaining unit, configured to acquire all changed operation data in the log record information based on the complete log record information;
添加单元，配置为将所有变化操作数据中的有效变化数据添加到数据结果集。Add a unit configured to add valid change data from all change operation data to the data result set.
上述方案中，所述读取单元，具体配置为判断数据块前后SEQ号之间的关系，设当前数据块的SEQ号为s1，下一个数据块的SEQ号为s2， 数据库的SEQ号为s3；In the above solution, the reading unit is specifically configured to determine the relationship between the SEQ numbers before and after the data block, and set the SEQ number of the current data block to s1, and the SEQ number of the next data block to s2. The SEQ number of the database is s3;
当s1＝s2时：When s1=s2:
s2＝s3，Redo正常读写，正常读取下一个数据块；S2=s3, Redo reads and writes normally, and reads the next data block normally;
s3>s2，Redo已被切换，当前Redo未完成，正常读取下一个数据块；S3>s2, Redo has been switched, the current Redo is not completed, and the next data block is read normally;
当s1<s2时：When s1<s2:
s3＝s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块；S3=s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
s3>s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块；S3>s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
当s2<＝s3、s1<＝s3、s1>s2时：When s2<=s3, s1<=s3, s1>s2:
s3＝s1，Redo下一个数据块暂时无数据写入，等待数据写入；S3=s1, Redo next data block has no data to write temporarily, waiting for data to be written;
s3＝s1+1，s1已读完，已切换到新的Redo，查询s1加1后对应的Redo，读取下一个数据块；S3=s1+1, s1 has been read, has been switched to the new Redo, query s1 plus 1 corresponding Redo, read the next data block;
s3>s1+1，s1已读完，至少两次切换到新的Redo，查询s1加1后对应归档，读取下一个数据块。S3>s1+1, s1 has been read, at least twice switched to the new Redo, query s1 plus 1 corresponding to the archive, read the next data block.
上述方案中，所述读取单元，具体配置为读取数据块中的日志记录，获取日志记录头部标识位；判断头部标识位是否为0，如果是则读取下一个数据块；如果不是，则获取日志记录的有效信息，确定头部标识位的位数，并根据位数解析头部标识位，计算日志记录信息长度，根据日志记录信息长度读取日志记录信息。In the above solution, the reading unit is specifically configured to read a log record in the data block, obtain a log record header identifier bit, determine whether the header identifier bit is 0, and if so, read the next data block; If not, the valid information of the log record is obtained, the number of bits of the header identifier is determined, and the header identifier is parsed according to the number of bits, the length of the log record information is calculated, and the log record information is read according to the length of the log record information.
上述方案中，所述添加单元，具体配置为检查变化操作数据是否为有效变化数据，如果不是有效变化数据，则丢掉所述变化操作数据，读取下一个变化操作数据；如果是有效变化数据，则将其添加到数据结果集中。In the above solution, the adding unit is specifically configured to check whether the change operation data is valid change data. If the data is not valid change, the change operation data is discarded, and the next change operation data is read; if it is valid change data, Then add it to the data result set.
上述方案中，所述数据获取单元，具体配置为读取日志记录信息中的变化操作，获取变化操作头部标识位； In the above solution, the data acquiring unit is configured to read a change operation in the log record information, and obtain a change operation header identifier bit;
从变化操作头部标识位中获取操作码，读取变化操作中各段长度；Obtain an operation code from the change operation header flag, and read the length of each segment in the change operation;
根据操作码读取变化操作中的二进制数据，该二进制数据即为变化操作数据。The binary data in the change operation is read according to the operation code, which is the change operation data.
本发明实施例还提供一种存储介质，存储有可执行程序，所述可执行程序被处理器执行时，实现上述的关系数据库日志抽取方法。The embodiment of the invention further provides a storage medium storing an executable program, and when the executable program is executed by the processor, the foregoing relational database log extraction method is implemented.
本发明实施例所达到的有益效果：本发明实施例通过对数据库日志的分析与定位，在数据库日志变化的第一时间抽取出日志变化数据，生成自定义格式的日志数据文件；通过该方法不但可以精确抽取出变化日志数据，还能降低数据库日志抽取对数据库自身的性能影响，对确保数据库的安全稳定的运行，具有重要的实际意义。The beneficial effects achieved by the embodiments of the present invention are: by analyzing and locating the database log, extracting the log change data at the first time of the database log change, and generating a log data file in a customized format; The change log data can be accurately extracted, and the performance impact of the database log extraction on the database itself can be reduced, which has important practical significance for ensuring the safe and stable operation of the database.
附图说明DRAWINGS
图1为本发明实施例关系数据库日志抽取方法的基本处理流程示意图；1 is a schematic flowchart of a basic processing procedure of a relational database log extraction method according to an embodiment of the present invention;
图2A为本发明实施例关系数据库日志抽取方法的详细处理流程示意图；2A is a schematic flowchart of a detailed processing process of a relational database log extraction method according to an embodiment of the present invention;
图2B为本发明实施例关系数据库日志抽取方法的详细处理流程示意图；2B is a schematic flowchart of a detailed processing process of a relational database log extraction method according to an embodiment of the present invention;
图2C为本发明实施例关系数据库日志抽取方法的详细处理流程示意图；2C is a schematic flowchart of a detailed processing process of a relational database log extraction method according to an embodiment of the present invention;
图3为本发明实施例关系数据库日志抽取方法对应的架构图；3 is an architectural diagram corresponding to a method for extracting a relational database log according to an embodiment of the present invention;
图4为本发明实施例关系数据库日志抽取装置的组成结构示意图。FIG. 4 is a schematic structural diagram of a structure of a relational database log extracting apparatus according to an embodiment of the present invention.
具体实施方式detailed description
下面结合附图对本发明作进一步描述。以下实施例仅用于更加清楚地说明本发明的技术方案，而不能以此来限制本发明的保护范围。The invention is further described below in conjunction with the drawings. The following examples are only intended to more clearly illustrate the technical solutions of the present invention, and are not intended to limit the scope of the present invention.
结合图1及图2A至图2C，图2A中的A节点连接图2B中的A节点， 图2B中的B节点连接图2C中的B节点，本发明实施例提供的关系数据库日志抽取方法的处理流程，包括以下步骤：Referring to FIG. 1 and FIG. 2A to FIG. 2C, the A node in FIG. 2A is connected to the A node in FIG. 2B. The B-node in FIG. 2B is connected to the Node B in FIG. 2C. The processing flow of the relational database log extraction method provided by the embodiment of the present invention includes the following steps:
步骤1，获取数据库日志文件信息；Step 1: Obtain database log file information.
其中，所述数据库日志文件信息包括：日志文件路径、日志文件存储格式。The database log file information includes: a log file path and a log file storage format.
在一可选实施方式中，初始化Redo日志读取进程(RedoReader)、数据库连接，获取日志文件路径和日志文件存储格式；日志文件存储格式包括数据库文件系统格式、裸设备格式以及ASM格式。In an optional implementation manner, the Redo Read process (RedoReader), the database connection, and the log file path and the log file storage format are initialized; the log file storage format includes a database file system format, a raw device format, and an ASM format.
步骤2，读取数据块； Step 2, reading a data block;
在一可选实施方式中，打开日志文件，读取数据库，并获取数据块大小和数量，确定数据库版本信息。In an alternative embodiment, the log file is opened, the database is read, and the size and number of the data blocks are obtained to determine the database version information.
读取数据块时，判断数据块前后SEQ号之间的关系，设当前数据块的SEQ号为s1，下一个数据块的SEQ号为s2，数据库的SEQ号为s3，通过比较确定下一步处理方法：When reading a data block, determine the relationship between the SEQ numbers before and after the data block, set the SEQ number of the current data block to s1, the SEQ number of the next data block to s2, and the SEQ number of the database to s3, and determine the next processing by comparison. method:
当s1＝s2时：When s1=s2:
s2＝s3，Redo(重做日志)正常读写，正常读取下一个数据块；S2=s3, Redo (redo log) reads and writes normally, and reads the next data block normally;
s3>s2，Redo已被切换，当前Redo未完成，正常读取下一个数据块；S3>s2, Redo has been switched, the current Redo is not completed, and the next data block is read normally;
s3<s2，此情况不存在；S3<s2, this situation does not exist;
当s1<s2时：When s1<s2:
s3＝s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块；S3=s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
s3>s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块；S3>s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
s3<s1，此情况不存在；S3<s1, this situation does not exist;
当s2<＝s3、s1<＝s3、s1>s2时： When s2<=s3, s1<=s3, s1>s2:
s3＝s1，Redo下一个数据块暂时无数据写入，等待数据写入；S3=s1, Redo next data block has no data to write temporarily, waiting for data to be written;
s3＝s1+1，s1已读完，已切换到新的Redo，查询s1加1后对应的Redo，读取下一个数据块；S3=s1+1, s1 has been read, has been switched to the new Redo, query s1 plus 1 corresponding Redo, read the next data block;
s3>s1+1，s1已读完，至少两次切换到新的Redo，查询s1加1后对应归档，读取下一个数据块；S3>s1+1, s1 has been read, at least twice switched to the new Redo, query s1 plus 1 corresponding to the archive, read the next data block;
s3<s2，此情况不存在。S3<s2, this situation does not exist.
步骤3，从当前数据块中读取日志记录信息。 Step 3. Read the log record information from the current data block.
在一可选实施方式中，初始化日志记录解析进程(RecordParse)、变化日志解析进程(ChangeParse)；In an optional implementation manner, the initialization log record resolution process (RecordParse), the change log resolution process (ChangeParse);
读取数据块中的日志记录(Record)，获取日志记录头部标识位(RecordHead)；判断头部标识位是否为0(即是否存在waste)，如果是则读取下一个数据块；如果不是，则获取日志记录的有效信息(Valid)，确定头部标识位的位数，并根据位数解析头部标识位，计算日志记录信息长度，根据日志记录信息长度读取日志记录信息。Read the log record in the data block, obtain the log record header (RecordHead); determine whether the header flag is 0 (that is, whether there is a waste), and if so, read the next block; if not Then, the valid information (Valid) of the log record is obtained, the number of bits of the header identifier bit is determined, and the header identifier bit is parsed according to the number of bits, the length of the log record information is calculated, and the log record information is read according to the length of the log record information.
如果valid值为0x04、0x05、0x06、0x09、0x0d，表示部标识位为68位，否则部标识位为24位。If the valid value is 0x04, 0x05, 0x06, 0x09, 0x0d, the part identification bit is 68 bits, otherwise the part identification bit is 24 bits.
日志记录信息长度＝日志记录长度-日志记录头部标识位长度。Log record length = log record length - log record header bit length.
步骤4，在所述日志记录信息不完整时，从下一个数据块中读取日志记录信息，直到拼装成完整的日志记录信息。Step 4: When the log record information is incomplete, the log record information is read from the next data block until the complete log record information is assembled.
步骤5，基于所述完整的日志记录信息获取日志记录信息中的所有变化操作数据。Step 5: Obtain all change operation data in the log record information based on the complete log record information.
在一具体实施方式中，解析完整的日志记录信息，获取操作码，并根据操作码依次读取日志记录信息中的所有变化操作数据。In a specific embodiment, the complete log record information is parsed, the operation code is obtained, and all the change operation data in the log record information are sequentially read according to the operation code.
举例来说，读取日志记录信息中的变化操作(Change)，获取变化操作头部标识位(Change Head)；从变化操作头部标识位中获取操作码，读取 变化操作中各段长度；根据操作码读取变化操作中的二进制数据，该二进制数据即为变化操作数据，并将其放到日志记录值(Record Value)中；检查日志记录信息中的变化操作是否都已读完，如果没有则继续读取，直至读取全部变化操作数据。For example, reading a change operation in the log record information, obtaining a change operation header flag (Change Head); obtaining an operation code from the change operation header flag bit, and reading Change the length of each segment in the operation; read the binary data in the change operation according to the operation code, the binary data is the change operation data, and put it into the record value (Record Value); check the change operation in the log record information Have you read it, if not, continue reading until you read all the change operation data.
操作码具体如下：重做操作OP_UNDO_UNDO对应操作码0x0501、开始新事务操作OP_UNDO_BEGINTRANS对应操作码0x0502、事务提交操作OP_UNDO_COMMIT对应操作码0x0504、新会话操作OP_UNDO_NEWSESSION对应操作码0x0513、切换会话操作OP_UNDO_SWITCHSESSION对应操作码0x0514、插入行操作OP_ROW_INSERT对应操作码0x0b02、删除行操作OP_ROW_DELETE对应操作码0x0b03、锁定行操作OP_ROW_LOCK对应操作码0x0b04、更新行操作OP_ROW_UPDATE对应操作码0x0b05、行链接操作OP_ROW_CHAIN对应操作码0x0b06、程序操作OP_ROW_MFC对应操作码0x0b07、程序操作OP_ROW_CFA对应操作码0x0b08、程序操作OP_ROW_KCI对应操作码0x0b09、程序操作OP_ROW_SKL对应操作码0x0b0a、插入多行操作OP_ROW_MINSERT对应操作码0x0b0b、删除多行操作OP_ROW_MDELETE对应操作码0x0b0c、程序操作OP_ROW_LMN对应操作码0x0b10、程序操作OP_ROW_COM对应操作码0x1801。The operation code is as follows: redo operation OP_UNDO_UNDO corresponding operation code 0x0501, start new transaction operation OP_UNDO_BEGINTRANS corresponding operation code 0x0502, transaction commit operation OP_UNDO_COMMIT corresponding operation code 0x0504, new session operation OP_UNDO_NEWSESSION corresponding operation code 0x0513, switching session operation OP_UNDO_SWITCHSESSION corresponding operation code 0x0514 Insert row operation OP_ROW_INSERT corresponding to opcode 0x0b02, delete row operation OP_ROW_DELETE corresponding opcode 0x0b03, lock row operation OP_ROW_LOCK corresponding opcode 0x0b04, update row operation OP_ROW_UPDATE corresponding opcode 0x0b05, row link operation OP_ROW_CHAIN corresponding opcode 0x0b06, program operation OP_ROW_MFC Operation code 0x0b07, program operation OP_ROW_CFA corresponding to operation code 0x0b08, program operation OP_ROW_KCI corresponding operation code 0x0b09, program operation OP_ROW_SKL corresponding operation code 0x0b0a, insert multi-line operation OP_ROW_MINSERT corresponding operation code 0x0b0b, delete multi-line operation OP_ROW_MDELETE corresponding operation code 0x0b0c, program operation OP_ROW_LMN corresponds to opcode 0x0b10, and program operation OP_ROW_COM corresponds to opcode 0x1801.
步骤6，将所有变化操作数据中的有效变化数据添加到数据结果集。In step 6, the valid change data in all the change operation data is added to the data result set.
在一具体实施方式中，检查变化操作数据是否为有效变化数据，如果不是有效变化数据，则丢掉所述变化操作数据，读取下一个变化操作数据；如果是有效变化数据，则将其添加到数据结果集中；检查完所有变化操作数据，则一次数据抽取结束。In a specific embodiment, it is checked whether the change operation data is valid change data. If it is not the valid change data, the change operation data is discarded, and the next change operation data is read; if it is valid change data, it is added to Data result set; after checking all change operation data, one data extraction ends.
上述方法采用多进程的方式对在线日志进行日志数据抽取，每次抽取的数据存入数据结果集中。在线日志抽取对数据库表的调用关系如表1所 示。The above method uses a multi-process method to extract log data from the online log, and each extracted data is stored in the data result set. The online log extraction call relationship to the database table is as shown in Table 1. Show.
表1在线日志抽取对数据库表的调用关系Table 1 Online log extraction call relationship to the database table
Figure PCTCN2017107110-appb-000001
Figure PCTCN2017107110-appb-000001
上述方法对应的框架图，如图3所示，通过对数据库在文件系统格式下、裸设备格式下以及对基于数据库特殊卷管理器格式下的Redo日志的分析与定位，在数据库日志变化的第一时间抽取日志变化数据，生成变化日志数据集合，为后续数据格式转换、日志入库提供基础。通过该方法不但可以精确的抽取出变化数据，还能降低数据库日志抽取对生产库的性能影响，对于确保信息灾备系统安全稳定运行具有重要的实际意义。The framework diagram corresponding to the above method, as shown in FIG. 3, by analyzing and locating the database in the file system format, the raw device format, and the Redo log based on the database special volume manager format, in the database log change The log change data is extracted at a time, and the change log data set is generated, which provides a basis for subsequent data format conversion and log storage. This method can not only accurately extract the change data, but also reduce the performance impact of database log extraction on the production library, which has important practical significance for ensuring the safe and stable operation of the information disaster recovery system.
本发明实施例还提供一种关系数据库日志抽取装置的组成结构示意图，如图4所示，包括：The embodiment of the invention further provides a schematic structural diagram of a relational database log extraction device, as shown in FIG. 4, which includes:
日志文件获取单元10，配置为获取数据库日志文件信息；The log file obtaining unit 10 is configured to obtain database log file information.
读取单元20，配置为读取数据块，从当前数据块中读取日志记录信息；The reading unit 20 is configured to read the data block and read the log record information from the current data block;
拼装单元30，配置为在所述日志记录信息不完整时，从下一个数据块中读取日志记录信息，直到拼装成完整的日志记录信息；The assembling unit 30 is configured to read the log record information from the next data block until the log record information is incomplete, until the complete log record information is assembled;
数据获取单元40，配置为基于所述完整的日志记录信息获取日志记录信息中的所有变化操作数据； The data obtaining unit 40 is configured to acquire all the changed operation data in the log record information based on the complete log record information;
添加单元50，配置为将所有变化操作数据中的有效变化数据添加到数据结果集。The adding unit 50 is configured to add valid change data in all the change operation data to the data result set.
上述方案中，所述读取单元20，具体配置为判断数据块前后SEQ号之间的关系，设当前数据块的SEQ号为s1，下一个数据块的SEQ号为s2，数据库的SEQ号为s3；In the above solution, the reading unit 20 is specifically configured to determine the relationship between the SEQ numbers before and after the data block, and set the SEQ number of the current data block to s1, the SEQ number of the next data block to s2, and the SEQ number of the database is S3;
当s1＝s2时：When s1=s2:
s2＝s3，Redo正常读写，正常读取下一个数据块；S2=s3, Redo reads and writes normally, and reads the next data block normally;
s3>s2，Redo已被切换，当前Redo未完成，正常读取下一个数据块；S3>s2, Redo has been switched, the current Redo is not completed, and the next data block is read normally;
当s1<s2时：When s1<s2:
s3＝s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块；S3=s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
s3>s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块；S3>s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
当s2<＝s3、s1<＝s3、s1>s2时：When s2<=s3, s1<=s3, s1>s2:
s3＝s1，Redo下一个数据块暂时无数据写入，等待数据写入；S3=s1, Redo next data block has no data to write temporarily, waiting for data to be written;
s3＝s1+1，s1已读完，已切换到新的Redo，查询s1加1后对应的Redo，读取下一个数据块；S3=s1+1, s1 has been read, has been switched to the new Redo, query s1 plus 1 corresponding Redo, read the next data block;
s3>s1+1，s1已读完，至少两次切换到新的Redo，查询s1加1后对应归档，读取下一个数据块。S3>s1+1, s1 has been read, at least twice switched to the new Redo, query s1 plus 1 corresponding to the archive, read the next data block.
上述方案中，所述读取单元20，具体配置为读取数据块中的日志记录，获取日志记录头部标识位；判断头部标识位是否为0，如果是则读取下一个数据块；如果不是，则获取日志记录的有效信息，确定头部标识位的位数，并根据位数解析头部标识位，计算日志记录信息长度，根据日志记录信息长度读取日志记录信息。In the above solution, the reading unit 20 is configured to read a log record in the data block, obtain a log record header identifier bit, determine whether the header identifier bit is 0, and if yes, read the next data block; If not, the valid information of the log record is obtained, the number of bits of the header identifier is determined, and the header identifier is parsed according to the number of bits, the length of the log record information is calculated, and the log record information is read according to the length of the log record information.
上述方案中，所述添加单元50，具体配置为检查变化操作数据是否为 有效变化数据，如果不是有效变化数据，则丢掉所述变化操作数据，读取下一个变化操作数据；如果是有效变化数据，则将其添加到数据结果集中。In the above solution, the adding unit 50 is specifically configured to check whether the change operation data is The data is effectively changed. If it is not valid change data, the change operation data is discarded, and the next change operation data is read; if it is valid change data, it is added to the data result set.
上述方案中，所述数据获取单元40，具体配置为读取日志记录信息中的变化操作，获取变化操作头部标识位；In the above solution, the data obtaining unit 40 is configured to read a change operation in the log record information, and obtain a change operation header identifier bit;
从变化操作头部标识位中获取操作码，读取变化操作中各段长度；Obtain an operation code from the change operation header flag, and read the length of each segment in the change operation;
根据操作码读取变化操作中的二进制数据，该二进制数据即为变化操作数据。The binary data in the change operation is read according to the operation code, which is the change operation data.
需要说明的是，日志文件获取单元10、读取单元20、拼装单元30、数据获取单元40和添加单元50的功能可以由关系数据库日志抽取装置中的通用处理器、数字信号处理器(DSP，Digital Signal Processor)，或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等实现。It should be noted that the functions of the log file obtaining unit 10, the reading unit 20, the assembling unit 30, the data obtaining unit 40, and the adding unit 50 may be a general-purpose processor and a digital signal processor (DSP, in the relational database log extracting device. Digital Signal Processor), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
需要说明的是：上述实施例提供的关系数据库日志抽取装置在进行关系数据库日志抽取时，仅以上述各程序模块的划分进行举例说明，实际应用中，可以根据需要而将上述处理分配由不同的程序模块完成，即将装置的内部结构划分成不同的程序模块，以完成以上描述的全部或者部分处理。另外，上述实施例提供的关系数据库日志抽取装置与关系数据库日志抽取方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。It should be noted that the relational database log extraction apparatus provided by the foregoing embodiment is only illustrated by the division of each of the foregoing program modules when performing relational database log extraction. In actual applications, the foregoing processing may be assigned differently according to requirements. The program module is completed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the embodiment of the relational database log extraction device and the relational database log extraction method provided by the foregoing embodiments are in the same concept, and the specific implementation process is described in detail in the method embodiment, and details are not described herein again.
本发明实施例还提供一种存储介质，存储有可执行程序，所述可执行程序被处理器运行时，执行：The embodiment of the invention further provides a storage medium storing an executable program, when the executable program is executed by the processor, executing:
获取数据库日志文件信息；Obtain database log file information;
读取数据块，从当前数据块中读取日志记录信息；Reading a data block and reading log record information from the current data block;
在所述日志记录信息不完整时，从下一个数据块中读取日志记录信息，直到拼装成完整的日志记录信息；When the log record information is incomplete, the log record information is read from the next data block until it is assembled into complete log record information;
基于所述完整的日志记录信息获取日志记录信息中的所有变化操作 数据；Obtaining all change operations in the log record information based on the complete log record information data;
将所有变化操作数据中的有效变化数据添加到数据结果集。Add valid change data from all change action data to the data result set.
所述可执行程序被处理器运行时，还执行：When the executable program is executed by the processor, it also executes:
判断数据块前后SEQ号之间的关系，设当前数据块的SEQ号为s1，下一个数据块的SEQ号为s2，数据库的SEQ号为s3，通过比较确定下一步处理方法：Determine the relationship between the SEQ numbers before and after the data block, set the SEQ number of the current data block to s1, the SEQ number of the next data block to s2, and the SEQ number of the database to s3, and determine the next processing method by comparison:
当s1＝s2时：When s1=s2:
s2＝s3，Redo正常读写，正常读取下一个数据块；S2=s3, Redo reads and writes normally, and reads the next data block normally;
s3>s2，Redo已被切换，当前Redo未完成，正常读取下一个数据块；S3>s2, Redo has been switched, the current Redo is not completed, and the next data block is read normally;
当s1<s2时：When s1<s2:
s3＝s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块；S3=s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
s3>s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块；S3>s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
当s2<＝s3、s1<＝s3、s1>s2时：When s2<=s3, s1<=s3, s1>s2:
s3＝s1，Redo下一个数据块暂时无数据写入，等待数据写入；S3=s1, Redo next data block has no data to write temporarily, waiting for data to be written;
s3＝s1+1，s1已读完，已切换到新的Redo，查询s1加1后对应的Redo，读取下一个数据块；S3=s1+1, s1 has been read, has been switched to the new Redo, query s1 plus 1 corresponding Redo, read the next data block;
s3>s1+1，s1已读完，至少两次切换到新的Redo，查询s1加1后对应归档，读取下一个数据块。S3>s1+1, s1 has been read, at least twice switched to the new Redo, query s1 plus 1 corresponding to the archive, read the next data block.
所述可执行程序被处理器运行时，还执行：When the executable program is executed by the processor, it also executes:
读取数据块中的日志记录，获取日志记录头部标识位；判断头部标识位是否为0，如果是则读取下一个数据块；如果不是，则获取日志记录的有效信息，确定头部标识位的位数，并根据位数解析头部标识位，计算日志记录信息长度，根据日志记录信息长度读取日志记录信息。 Read the log record in the data block, obtain the log record header identifier bit; determine whether the header identifier bit is 0, if yes, read the next data block; if not, obtain the log record valid information, determine the header The number of bits in the bit is identified, and the header bit is parsed according to the number of bits, the length of the log record information is calculated, and the log record information is read according to the length of the log record information.
所述可执行程序被处理器运行时，还执行：When the executable program is executed by the processor, it also executes:
检查变化操作数据是否为有效变化数据，如果不是有效变化数据，则丢掉所述变化操作数据，读取下一个变化操作数据；如果是有效变化数据，则将其添加到数据结果集中。Check whether the change operation data is valid change data. If it is not valid change data, the change operation data is discarded, and the next change operation data is read; if it is valid change data, it is added to the data result set.
所述可执行程序被处理器运行时，还执行：When the executable program is executed by the processor, it also executes:
读取日志记录信息中的变化操作，获取变化操作头部标识位；Reading the change operation in the log record information, and obtaining the change operation header identifier bit;
从变化操作头部标识位中获取操作码，读取变化操作中各段长度；Obtain an operation code from the change operation header flag, and read the length of each segment in the change operation;
根据操作码读取变化操作中的二进制数据，该二进制数据即为变化操作数据。The binary data in the change operation is read according to the operation code, which is the change operation data.
相应的，本发明实施例还提供一种关系数据库日志抽取装置，包括：处理器和用于存储能够在处理器上运行的计算机程序的存储器，Correspondingly, an embodiment of the present invention further provides a relational database log extracting apparatus, including: a processor and a memory for storing a computer program capable of running on the processor,
其中，所述处理器用于运行所述计算机程序时，执行：Wherein the processor is configured to execute when the computer program is executed:
获取数据库日志文件信息；Obtain database log file information;
读取数据块，从当前数据块中读取日志记录信息；Reading a data block and reading log record information from the current data block;
在所述日志记录信息不完整时，从下一个数据块中读取日志记录信息，直到拼装成完整的日志记录信息；When the log record information is incomplete, the log record information is read from the next data block until it is assembled into complete log record information;
基于所述完整的日志记录信息获取日志记录信息中的所有变化操作数据；Obtaining all change operation data in the log record information based on the complete log record information;
将所有变化操作数据中的有效变化数据添加到数据结果集。Add valid change data from all change action data to the data result set.
需要说明的是，存储器以由任何类型的易失性或非易失性存储设备、或者它们的组合来实现。其中，非易失性存储器可以是只读存储器(ROM，Read Only Memory)、可编程只读存储器(PROM，Programmable Read-Only Memory)、可擦除可编程只读存储器(EPROM，Erasable Programmable Read-Only Memory)、电可擦除可编程只读存储器(EEPROM，Electrically Erasable Programmable Read-Only Memory)、磁性随机存取存储器 (FRAM，Ferromagnetic Random Access Memory)、快闪存储器(Flash Memory)、磁表面存储器、光盘、或只读光盘(CD-ROM，Compact Disc Read-Only Memory)；磁表面存储器可以是磁盘存储器或磁带存储器。易失性存储器可以是随机存取存储器(RAM，Random Access Memory)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(SRAM，Static Random Access Memory)、同步静态随机存取存储器(SSRAM，Synchronous Static Random Access Memory)、动态随机存取存储器(DRAM，Dynamic Random Access Memory)、同步动态随机存取存储器(SDRAM，Synchronous Dynamic Random Access Memory)、双倍数据速率同步动态随机存取存储器(DDRSDRAM，Double Data Rate Synchronous Dynamic Random Access Memory)、增强型同步动态随机存取存储器(ESDRAM，Enhanced Synchronous Dynamic Random Access Memory)、同步连接动态随机存取存储器(SLDRAM，SyncLink Dynamic Random Access Memory)、直接内存总线随机存取存储器(DRRAM，Direct Rambus Random Access Memory)。本发明实施例描述的存储器804旨在包括但不限于这些和任意其它适合类型的存储器。It should be noted that the memory is implemented by any type of volatile or non-volatile storage device, or a combination thereof. The non-volatile memory may be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), or an Erasable Programmable Read (EPROM). Only Memory), Electrically Erasable Programmable Read-Only Memory (EEPROM), Magnetic Random Access Memory (FRAM, Ferromagnetic Random Access Memory), Flash Memory, Magnetic Surface Memory, Optical Disk, or Compact Disc Read-Only Memory (CD-ROM); magnetic surface memory can be disk storage or tape storage . The volatile memory can be a random access memory (RAM) that acts as an external cache. By way of example and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access (SSRAM). DRAM (Dynamic Random Access Memory), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), enhancement Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchronous Dynamic Random Access Memory (SLDRAM), Direct Memory Bus Random Access Memory (DRRAM) ). The memory 804 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.
以上所述仅是本发明的优选实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本发明技术原理的前提下，还可以做出若干改进和变形，这些改进和变形也应视为本发明的保护范围。The above is only a preferred embodiment of the present invention, and it should be noted that those skilled in the art can make several improvements and modifications without departing from the technical principles of the present invention. It should also be considered as the scope of protection of the present invention.
工业实用性Industrial applicability
本发明实施例中，在对关系数据库日志抽取时，获取数据库日志文件信息；读取数据块，从当前数据块中读取日志记录信息；在所述日志记录信息不完整时，从下一个数据块中读取日志记录信息，直到拼装成完整的日志记录信息；基于所述完整的日志记录信息获取日志记录信息 中的所有变化操作数据；将所有变化操作数据中的有效变化数据添加到数据结果集。如此，通过对数据库日志的分析与定位，在数据库日志变化的第一时间抽取出日志变化数据，生成自定义格式的日志数据文件；通过该方法不但可以精确抽取出变化日志数据，还能降低数据库日志抽取对数据库自身的性能影响，提高数据库运行的安全性和稳定性。 In the embodiment of the present invention, when the relational database log is extracted, the database log file information is acquired; the data block is read, and the log record information is read from the current data block; when the log record information is incomplete, the next data is obtained. The log record information is read in the block until it is assembled into complete log record information; the log record information is obtained based on the complete log record information All change operation data in ; add valid change data from all change operation data to the data result set. In this way, by analyzing and locating the database log, the log change data is extracted at the first time of the database log change, and a log data file of a custom format is generated; the method can not only accurately extract the change log data, but also reduce the database. Log extraction affects the performance of the database itself, improving the security and stability of the database operation.

Claims (13)

  1. 一种关系数据库日志抽取方法，包括：A relational database log extraction method, including:
    获取数据库日志文件信息；Obtain database log file information;
    读取数据块，从当前数据块中读取日志记录信息；Reading a data block and reading log record information from the current data block;
    在所述日志记录信息不完整时，从下一个数据块中读取日志记录信息，直到拼装成完整的日志记录信息；When the log record information is incomplete, the log record information is read from the next data block until it is assembled into complete log record information;
    基于所述完整的日志记录信息获取日志记录信息中的所有变化操作数据；Obtaining all change operation data in the log record information based on the complete log record information;
    将所有变化操作数据中的有效变化数据添加到数据结果集。Add valid change data from all change action data to the data result set.
  2. 根据权利要求1所述的一种关系数据库日志抽取方法，其中，A relational database log extraction method according to claim 1, wherein
    所述日志文件信息中的日志文件存储格式包括：数据库文件系统格式、裸设备格式以及ASM格式。The log file storage format in the log file information includes: a database file system format, a raw device format, and an ASM format.
  3. 根据权利要求1所述的一种关系数据库日志抽取方法，其中，所述读取数据块，包括：The method of extracting a relational database log according to claim 1, wherein the reading the data block comprises:
    判断数据块前后序列SEQ号之间的关系，设当前数据块的SEQ号为s1，下一个数据块的SEQ号为s2，数据库的SEQ号为s3，通过比较确定下一步处理方法：Determine the relationship between the SEQ numbers of the data block before and after the data block, set the SEQ number of the current data block to s1, the SEQ number of the next data block to s2, and the SEQ number of the database to s3, and determine the next processing method by comparison:
    当s1＝s2时：When s1=s2:
    s2＝s3，重做日志Redo正常读写，正常读取下一个数据块；S2=s3, redo log Redo reads and writes normally, and reads the next data block normally;
    s3>s2，Redo已被切换，当前Redo未完成，正常读取下一个数据块；S3>s2, Redo has been switched, the current Redo is not completed, and the next data block is read normally;
    当s1<s2时：When s1<s2:
    s3＝s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块；S3=s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
    s3>s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块； S3>s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
    当s2<＝s3、s1<＝s3、s1>s2时：When s2<=s3, s1<=s3, s1>s2:
    s3＝s1，Redo下一个数据块暂时无数据写入，等待数据写入；S3=s1, Redo next data block has no data to write temporarily, waiting for data to be written;
    s3＝s1+1，s1已读完，已切换到新的Redo，查询s1加1后对应的Redo，读取下一个数据块；S3=s1+1, s1 has been read, has been switched to the new Redo, query s1 plus 1 corresponding Redo, read the next data block;
    s3>s1+1，s1已读完，至少两次切换到新的Redo，查询s1加1后对应归档，读取下一个数据块。S3>s1+1, s1 has been read, at least twice switched to the new Redo, query s1 plus 1 corresponding to the archive, read the next data block.
  4. 根据权利要求1所述的一种关系数据库日志抽取方法，其中，所述从当前数据块中读取日志记录信息，包括：The relational database log extraction method according to claim 1, wherein the reading the log record information from the current data block comprises:
    读取数据块中的日志记录，获取日志记录头部标识位；判断头部标识位是否为0，如果是则读取下一个数据块；如果不是，则获取日志记录的有效信息，确定头部标识位的位数，并根据位数解析头部标识位，计算日志记录信息长度，根据日志记录信息长度读取日志记录信息。Read the log record in the data block, obtain the log record header identifier bit; determine whether the header identifier bit is 0, if yes, read the next data block; if not, obtain the log record valid information, determine the header The number of bits in the bit is identified, and the header bit is parsed according to the number of bits, the length of the log record information is calculated, and the log record information is read according to the length of the log record information.
  5. 根据权利要求1所述的一种关系数据库日志抽取方法，其中，所述将所有变化操作数据中的有效变化数据添加到数据结果集，包括：The relational database log extraction method according to claim 1, wherein the adding the effective change data in all the change operation data to the data result set comprises:
    检查变化操作数据是否为有效变化数据，如果不是有效变化数据，则丢掉所述变化操作数据，读取下一个变化操作数据；如果是有效变化数据，则将其添加到数据结果集中。Check whether the change operation data is valid change data. If it is not valid change data, the change operation data is discarded, and the next change operation data is read; if it is valid change data, it is added to the data result set.
  6. 根据权利要求1所述的一种关系数据库日志抽取方法，其中，所述获取日志记录信息中的所有变化操作数据，包括：The relational database log extraction method according to claim 1, wherein the obtaining all the change operation data in the log record information comprises:
    读取日志记录信息中的变化操作，获取变化操作头部标识位；Reading the change operation in the log record information, and obtaining the change operation header identifier bit;
    从变化操作头部标识位中获取操作码，读取变化操作中各段长度；Obtain an operation code from the change operation header flag, and read the length of each segment in the change operation;
    根据操作码读取变化操作中的二进制数据，该二进制数据即为变化操作数据。The binary data in the change operation is read according to the operation code, which is the change operation data.
  7. 根据权利要求6所述的一种关系数据库日志抽取方法，其中，重做操作OP_UNDO_UNDO对应操作码0x0501、开始新事务操作 OP_UNDO_BEGINTRANS对应操作码0x0502、事务提交操作OP_UNDO_COMMIT对应操作码0x0504、新会话操作OP_UNDO_NEWSESSION对应操作码0x0513、切换会话操作OP_UNDO_SWITCHSESSION对应操作码0x0514、插入行操作OP_ROW_INSERT对应操作码0x0b02、删除行操作OP_ROW_DELETE对应操作码0x0b03、锁定行操作OP_ROW_LOCK对应操作码0x0b04、更新行操作OP_ROW_UPDATE对应操作码0x0b05、行链接操作OP_ROW_CHAIN对应操作码0x0b06、程序操作OP_ROW_MFC对应操作码0x0b07、程序操作OP_ROW_CFA对应操作码0x0b08、程序操作OP_ROW_KCI对应操作码0x0b09、程序操作OP_ROW_SKL对应操作码0x0b0a、插入多行操作OP_ROW_MINSERT对应操作码0x0b0b、删除多行操作OP_ROW_MDELETE对应操作码0x0b0c、程序操作OP_ROW_LMN对应操作码0x0b10、程序操作OP_ROW_COM对应操作码0x1801。The relational database log extraction method according to claim 6, wherein the redo operation OP_UNDO_UNDO corresponds to the operation code 0x0501, and starts a new transaction operation. OP_UNDO_BEGINTRANS corresponding to opcode 0x0502, transaction commit operation OP_UNDO_COMMIT corresponding opcode 0x0504, new session operation OP_UNDO_NEWSESSION corresponding opcode 0x0513, handover session operation OP_UNDO_SWITCHSESSION corresponding opcode 0x0514, insert row operation OP_ROW_INSERT corresponding opcode 0x0b02, delete row operation OP_ROW_DELETE corresponding opcode 0x0b03 Lock row operation OP_ROW_LOCK corresponding to opcode 0x0b04, update row operation OP_ROW_UPDATE corresponding opcode 0x0b05, row link operation OP_ROW_CHAIN corresponding opcode 0x0b06, program operation OP_ROW_MFC corresponding opcode 0x0b07, program operation OP_ROW_CFA corresponding opcode 0x0b08, program operation OP_ROW_KCI corresponding opcode 0x0b09, program operation OP_ROW_SKL corresponds to operation code 0x0b0a, insert multi-row operation OP_ROW_MINSERT corresponding operation code 0x0b0b, delete multi-line operation OP_ROW_MDELETE corresponding operation code 0x0b0c, program operation OP_ROW_LMN corresponding operation code 0x0b10, program operation OP_ROW_COM corresponding operation code 0x1801.
  8. 一种关系数据库日志抽取装置，包括：A relational database log extraction device, comprising:
    日志文件获取单元，配置为获取数据库日志文件信息；a log file obtaining unit configured to obtain database log file information;
    读取单元，配置为读取数据块，从当前数据块中读取日志记录信息；a reading unit configured to read a data block and read log record information from the current data block;
    拼装单元，配置为在所述日志记录信息不完整时，从下一个数据块中读取日志记录信息，直到拼装成完整的日志记录信息；The assembling unit is configured to read the log record information from the next data block until the log record information is incomplete, until the complete log record information is assembled;
    数据获取单元，配置为基于所述完整的日志记录信息获取日志记录信息中的所有变化操作数据；a data obtaining unit, configured to acquire all changed operation data in the log record information based on the complete log record information;
    添加单元，配置为将所有变化操作数据中的有效变化数据添加到数据结果集。Add a unit configured to add valid change data from all change operation data to the data result set.
  9. 根据权利要求8所述的装置，其中，The device according to claim 8, wherein
    所述读取单元，具体配置为判断数据块前后序列号SEQ号之间的关 系，设当前数据块的SEQ号为s1，下一个数据块的SEQ号为s2，数据库的SEQ号为s3；The reading unit is specifically configured to determine the relationship between the sequence number and the SEQ number of the data block before and after Department, the SEQ number of the current data block is s1, the SEQ number of the next data block is s2, and the SEQ number of the database is s3;
    当s1＝s2时：When s1=s2:
    s2＝s3，重做日志Redo正常读写，正常读取下一个数据块；S2=s3, redo log Redo reads and writes normally, and reads the next data block normally;
    s3>s2，Redo已被切换，当前Redo未完成，正常读取下一个数据块；S3>s2, Redo has been switched, the current Redo is not completed, and the next data block is read normally;
    当s1<s2时：When s1<s2:
    s3＝s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块；S3=s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
    s3>s2，Redo已被切换，s1可能有未读完数据，下一个数据块被新数据覆盖，从s1对应的归档中读取下一个数据块；S3>s2, Redo has been switched, s1 may have unread data, the next data block is overwritten by new data, and the next data block is read from the archive corresponding to s1;
    当s2<＝s3、s1<＝s3、s1>s2时：When s2<=s3, s1<=s3, s1>s2:
    s3＝s1，Redo下一个数据块暂时无数据写入，等待数据写入；S3=s1, Redo next data block has no data to write temporarily, waiting for data to be written;
    s3＝s1+1，s1已读完，已切换到新的Redo，查询s1加1后对应的Redo，读取下一个数据块；S3=s1+1, s1 has been read, has been switched to the new Redo, query s1 plus 1 corresponding Redo, read the next data block;
    s3>s1+1，s1已读完，至少两次切换到新的Redo，查询s1加1后对应归档，读取下一个数据块。S3>s1+1, s1 has been read, at least twice switched to the new Redo, query s1 plus 1 corresponding to the archive, read the next data block.
  10. 根据权利要求8所述的装置，其中，The device according to claim 8, wherein
    所述读取单元，具体配置为读取数据块中的日志记录，获取日志记录头部标识位；判断头部标识位是否为0，如果是则读取下一个数据块；如果不是，则获取日志记录的有效信息，确定头部标识位的位数，并根据位数解析头部标识位，计算日志记录信息长度，根据日志记录信息长度读取日志记录信息。The reading unit is configured to read a log record in the data block, obtain a log record header identifier bit, determine whether the header identifier bit is 0, and if yes, read the next data block; if not, obtain the The effective information of the log record determines the number of bits in the header identifier, and resolves the header identifier according to the number of bits, calculates the length of the log record information, and reads the log record information according to the length of the log record information.
  11. 根据权利要求8所述的装置，其中，The device according to claim 8, wherein
    所述添加单元，具体配置为检查变化操作数据是否为有效变化数据，如果不是有效变化数据，则丢掉所述变化操作数据，读取下一个变化操作 数据；如果是有效变化数据，则将其添加到数据结果集中。The adding unit is specifically configured to check whether the changed operation data is valid change data. If the data is not valid change, the change operation data is discarded, and the next change operation is read. Data; if it is a valid change data, it is added to the data result set.
  12. 根据权利要求8所述的装置，其中，The device according to claim 8, wherein
    所述数据获取单元，具体配置为读取日志记录信息中的变化操作，获取变化操作头部标识位；The data acquiring unit is configured to read a change operation in the log record information, and obtain a change operation header identifier bit;
    从变化操作头部标识位中获取操作码，读取变化操作中各段长度；Obtain an operation code from the change operation header flag, and read the length of each segment in the change operation;
    根据操作码读取变化操作中的二进制数据，该二进制数据即为变化操作数据。The binary data in the change operation is read according to the operation code, which is the change operation data.
  13. 一种存储介质，存储有可执行程序，所述可执行程序被处理器执行时，实现权利要求1至7任一项所述的关系数据库日志抽取方法。 A storage medium storing an executable program, the executable program being executed by a processor, implementing the relational database log extraction method according to any one of claims 1 to 7.
PCT/CN2017/107110 2017-04-07 2017-10-20 Relational database log extraction method and device, and storage medium WO2018184376A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710224035.7 2017-04-07
CN201710224035.7A CN107122424B (en) 2017-04-07 2017-04-07 A kind of relational database log abstracting method

Publications (1)

Publication Number Publication Date
WO2018184376A1 true WO2018184376A1 (en) 2018-10-11

Family

ID=59726261

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/107110 WO2018184376A1 (en) 2017-04-07 2017-10-20 Relational database log extraction method and device, and storage medium

Country Status (2)

Country Link
CN (1) CN107122424B (en)
WO (1) WO2018184376A1 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107122424B (en) * 2017-04-07 2019-11-05 南京南瑞集团公司 A kind of relational database log abstracting method
CN110083811B (en) * 2019-04-14 2022-11-22 杭州拓深科技有限公司 Method for replaying equipment operation site
CN112650725A (en) * 2020-12-30 2021-04-13 四川效率源信息安全技术股份有限公司 Method for recovering DM8 database log file
CN113312194B (en) * 2021-06-10 2024-01-23 中国民航信息网络股份有限公司 Service data acquisition method and device

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090037422A1 (en) * 2007-07-31 2009-02-05 Lik Wong Combining capture and apply in a distributed information sharing system
CN103412924A (en) * 2013-08-12 2013-11-27 东软集团股份有限公司 Log multi-language query method and system
CN105677876A (en) * 2016-01-12 2016-06-15 国家电网公司 Method for log mining based on physical level database
CN107122424A (en) * 2017-04-07 2017-09-01 南京南瑞集团公司 A kind of relational database daily record abstracting method

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8510270B2 (en) * 2010-07-27 2013-08-13 Oracle International Corporation MYSQL database heterogeneous log based replication
CN104331435B (en) * 2014-10-22 2017-11-21 国家电网公司 A kind of efficient mass data abstracting method of low influence based on Hadoop big data platforms
CN105488187A (en) * 2015-12-02 2016-04-13 北京四达时代软件技术股份有限公司 Method and device for extracting multi-source heterogeneous data increment

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090037422A1 (en) * 2007-07-31 2009-02-05 Lik Wong Combining capture and apply in a distributed information sharing system
CN103412924A (en) * 2013-08-12 2013-11-27 东软集团股份有限公司 Log multi-language query method and system
CN105677876A (en) * 2016-01-12 2016-06-15 国家电网公司 Method for log mining based on physical level database
CN107122424A (en) * 2017-04-07 2017-09-01 南京南瑞集团公司 A kind of relational database daily record abstracting method

Also Published As

Publication number Publication date
CN107122424B (en) 2019-11-05
CN107122424A (en) 2017-09-01

Similar Documents

Publication Publication Date Title
WO2018184376A1 (en) Relational database log extraction method and device, and storage medium
CN110879813B (en) Binary log analysis-based MySQL database increment synchronization implementation method
WO2017190580A1 (en) Method and device for accessing database
US10671586B2 (en) Optimal sort key compression and index rebuilding
CN109558452B (en) Synchronization method for query table building operation
CN102609337A (en) Rapid data recovery method for memory database
WO2016095672A1 (en) Stack-based exception detection method and device
CN109508346B (en) Cascade synchronous control method and system for DDL operation
WO2021169496A1 (en) Data reading method, apparatus, and device, and storage medium
CN105677876A (en) Method for log mining based on physical level database
US20160321144A1 (en) Database rollback using wal
CN110990365A (en) Data synchronization method, device, server and storage medium
WO2022048358A1 (en) Data processing method and device, and storage medium
WO2016107306A1 (en) Message subscription method, processing node device and message bus
CN103440265A (en) MapReduce-based CDC (Change Data Capture) method of MYSQL database
US20160275134A1 (en) Nosql database data validation
US11182375B2 (en) Metadata validation tool
WO2017067397A1 (en) Data recovery method and device
WO2015055062A1 (en) Data file writing method and system, and data file reading method and system
US11132351B2 (en) Executing transactions based on success or failure of the transactions
CN111026736A (en) Data blood margin management method and device and data blood margin analysis method and device
CN112612647B (en) Log parallel replay method, device, equipment and storage medium
US11604805B2 (en) Terminal, storage medium, and database synchronization method thereof
CN111767168B (en) Block chain-based storage system facilitating data backtracking
CN111026764B (en) Data storage method and device, electronic product and storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17904790

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17904790

Country of ref document: EP

Kind code of ref document: A1