WO2023272895A1 - 一种数据和日志一体化的值日志实现方法、装置、设备及存储介质 - Google Patents

一种数据和日志一体化的值日志实现方法、装置、设备及存储介质 Download PDF

Info

Publication number
WO2023272895A1
WO2023272895A1 PCT/CN2021/112761 CN2021112761W WO2023272895A1 WO 2023272895 A1 WO2023272895 A1 WO 2023272895A1 CN 2021112761 W CN2021112761 W CN 2021112761W WO 2023272895 A1 WO2023272895 A1 WO 2023272895A1
Authority
WO
WIPO (PCT)
Prior art keywords
value log
data
disk
operations
value
Prior art date
Application number
PCT/CN2021/112761
Other languages
English (en)
French (fr)
Inventor
付新
王学海
姜久文
许雄凌
张静修
李庆春
葛玉梅
郭继红
乔阳
张付美
Original Assignee
达梦数据技术(江苏)有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 达梦数据技术(江苏)有限公司 filed Critical 达梦数据技术(江苏)有限公司
Publication of WO2023272895A1 publication Critical patent/WO2023272895A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466—Performance evaluation by tracing or monitoring
    • G06F11/3476—Data logging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases

Definitions

  • the invention relates to the technical field of databases, in particular to a method, device, equipment and storage medium for implementing a value log integrating data and logs.
  • Logs are an important part of data processing systems such as databases and various Key-Value (referred to as KV).
  • Data processing systems such as databases/KV only need to persist the modified content in the data page to the log, and then only the The copy of the data in memory is modified without writing the data to disk for each modification.
  • the advantage of this is that log writing is sequential I/O in a small area, while data writing in data processing systems such as database/KV is usually random I/O. Through the persistence of logs, the reliability of data storage is guaranteed. performance, and improve the efficiency of data writing.
  • the technical problem to be solved by the present invention is to provide a value log implementation method, device, equipment and storage medium that integrates data and logs. Only sequential writing is required, and the performance is improved by 10 times or even more than traditional random writing. Input performance, only one copy of data needs to be written, avoiding the situation where traditional databases need to write 3-4 copies of data.
  • the present invention provides a method for implementing a value log integrating data and logs, including the following steps:
  • the working thread or process in the data processing system parses the received operation request to determine the operation type
  • the worker thread or process performs an insert operation in the memory and generates a physical record in the same format as the final write to the disk storage device in the memory, and then directly writes the physical record in the memory to the value log file, or first Write to the value log buffer, wait for the disk writing thread or process to flush the disk, and send the operation result to the client after the disk is flushed;
  • the disk writing thread or process continuously writes the physical records of the value log buffer to the end of the value log file in sequence, and notifies the worker thread or process to complete the task of flushing the disk. Since the entire data update history is in the value log file, this file can simultaneously perform the functions of data, online log, rollback segment, and archive log, which is called the value log system.
  • the data update operation includes insertion, deletion and modification.
  • the worker thread or process converts the delete and modify operations into insert operations specifically: convert the delete operation into inserting a record containing a key and a delete mark, and convert the modify operation into inserting a record containing a key and Deleting the marked record and inserting a new modified record makes all data update operations in the entire system become insert operations.
  • a value log realization device integrating data and logs, including:
  • the operation parsing module is responsible for parsing the received operation request and determining the operation type
  • An operation conversion module configured to convert data update operations
  • Execute the module complete the data insertion operation, and generate physical records in memory with the same format as the final value log file; then choose to write the physical records directly to the end of the value log file, or first write to the value log buffer, and then
  • the special value log writing module is responsible for writing the physical records in the value log buffer to the value log file, and after the disk is wiped, the operation result is sent to the client;
  • the value log buffer module is used to cache the physical records generated by the data update operation; after the execution module generates the physical records in the memory, it can choose to write the physical records into the value log buffer.
  • the disk writing thread continuously writes the physical records of the value log buffer to the end of the value log file in sequence, and notifies the worker thread/process that the task of flushing the disk is completed.
  • the operation conversion module is used to convert the data update operation specifically as follows: when the operation type is a data update operation, the operation conversion module converts both deletion and modification operations into insertion operations, and converts the deletion operation into an insertion containing a key and Delete marked record, the modification operation is transformed into inserting a record with key and delete mark and inserting a new modified record.
  • the data update operation includes insertion, deletion and modification.
  • a value log implementation device integrating data and logs, including: one or more processors;
  • a storage device for storing one or more programs and user data
  • the one or more processors When the one or more programs are executed by one or more processors, the one or more processors are made to implement the method for implementing value logs integrating data and logs as described in any one of the embodiments of the present invention.
  • a value log that integrates data and logs realizes a storage medium, on which a computer program is stored, and when the program is executed by a processor, a kind of data and log integration as described in any one of the embodiments of the present invention is realized.
  • the value of the log implementation method is realized.
  • the beneficial effects of the present invention are: the entire system only has sequential writing, and the performance is improved by 10 times or even more than the traditional random writing; only one copy of data is needed to avoid the need to write 3-4 copies of data in traditional databases Condition.
  • FIG. 1 is a schematic flowchart of an implementation method provided by Embodiment 1 of the present invention.
  • FIG. 2 is a schematic flowchart of an implementation method provided by Embodiment 2 of the present invention.
  • FIG. 3 is a schematic flowchart of an implementation method provided by Embodiment 3 of the present invention.
  • the embodiments of the present invention realize the integration of user data, online logs, rollback files, and archived log files by converting data update operations into sequential write operations at the end of files in a data processing system such as a database or KV.
  • the working thread/process of the data processing system receives the operation request and judges the operation type.
  • the operation type is a data update (including insert, delete, modify) operation
  • the worker thread/process converts the delete and modify operations into insert operations, such as converting the delete operation into inserting a record containing a key and a delete mark (indicating that it contains The record corresponding to the key has been deleted), and the modification operation is transformed into inserting a record containing the key and the deletion mark and inserting a new modified record.
  • the worker thread/process inserts into the memory and generates a physical record, and then directly writes the physical record to the end of the value log file, or writes it to the value log buffer first, and then waits for the disk write thread/process to flush the disk .
  • the disk writing thread/process continuously writes the physical records of the value log buffer to the end of the value log file in sequence, and notifies the worker thread/process that the task of flushing the disk is completed. In this way, all data update operations in the entire system become sequential write operations. Finally the worker thread/process notifies the client that the operation is complete.
  • FIG. 1 is a flow chart of a method for implementing a value log provided in Embodiment 1 of the present invention.
  • This embodiment can be used to realize the value log system, and the method can be executed by the value log device in the example of the present invention, and the device can be implemented in the form of software and/hardware, as shown in Figure 1, the method specifically includes the following steps:
  • a data processing system such as a database/KV receives a processing request and determines an operation type.
  • the worker thread/process converts the delete and modify operations into insert operations, such as converting the delete operation into inserting a record containing a key and a delete mark ( Indicates that the record containing the corresponding key has been deleted, and the deletion mark can have various representations in the specific implementation), and the modification operation is transformed into inserting a record containing the key and the deletion mark and inserting a new modified record. This makes all data update operations in the entire system into insert operations.
  • LSN has the same meaning as the common log sequence number in the database system. It is a self-increasing integer.
  • the primary key index contains LSN as the last column of the index, and the LSN is sorted in descending order, so that subsequent queries can be guaranteed The most recent records are queried first.
  • the deletion mark indicates that it has been deleted, and an empty result set is returned to the user directly.
  • marking techniques can be used to indicate that a certain record has been deleted.
  • LSN can be implemented using other similar schemes, and even the LSN can be arranged in ascending order. In this case, it is necessary to scan the records of the same KEY until the last visible result. For the above use cases, if the LSN is in ascending order Arrangement, the same KEY scans to the last record, you can find a deletion mark, and return an empty result set.
  • the modification operation can be directly converted into inserting a new modified record instead of generating a record with a deletion mark records to further improve performance.
  • the worker thread/process performs an insert operation in the memory and generates a physical record in the same format in the memory as it is finally written to a storage device such as a disk, and then writes the physical record in the memory to the value log buffer, and waits
  • the disk writing thread/process flashes the disk, and after the disk is flushed, the operation result is sent to the client.
  • the value log buffer is similar to the log buffer of a traditional database and can be recycled. As long as the corresponding buffer content has been written to disk, the corresponding buffer space can be reused.
  • the thread/process of disk brushing continuously writes the physical records of the value log buffer to the end of the value log file in sequence, and notifies the worker thread/process to complete the task of disk brushing.
  • FIG. 2 is a flow chart of a method for implementing a value log provided by Embodiment 2 of the present invention.
  • This embodiment can be used to realize the value log system, and the method can be executed by the value log device in the example of the present invention, and the device can be implemented in the form of software and/hardware, as shown in Figure 2, the method specifically includes the following steps:
  • a data processing system such as a database/KV receives a processing request and determines an operation type.
  • the worker thread/process converts the delete and modify operations into insert operations, such as converting the delete operation into inserting a record containing a key and a delete mark ( Indicates that the record containing the corresponding key has been deleted, and the deletion mark can have various representations in the specific implementation), and the modification operation is transformed into inserting a record containing the key and the deletion mark and inserting a new modified record. This makes all data update operations in the entire system into insert operations.
  • LSN has the same meaning as the common log sequence number in the database system. It is a self-increasing integer.
  • the primary key index contains LSN as the last column of the index, and the LSN is sorted in descending order, so that subsequent queries can be guaranteed The most recent records are queried first.
  • the deletion mark indicates that it has been deleted, and an empty result set is returned to the user directly.
  • marking techniques can be used to indicate that a certain record has been deleted.
  • LSN can be implemented using other similar schemes, and even the LSN can be arranged in ascending order. In this case, it is necessary to scan the records of the same KEY until the last visible result. For the above use cases, if the LSN is in ascending order Arrangement, the same KEY scans to the last record, you can find a deletion mark, and return an empty result set.
  • the modification operation can be directly converted into inserting a new modified record instead of generating a record with a deletion mark records to further improve performance.
  • the worker thread/process performs an insert operation in the memory and generates a physical record in the same format in the memory as it is finally written to a storage device such as a disk, and then directly writes the physical record to the end of the value log file, and waits for the disk to be flushed after completion , to send the result of the operation to the client.
  • FIG. 3 is a flow chart of a method for implementing a value log provided by Embodiment 2 of the present invention.
  • This embodiment can be used to realize the value log system, and the method can be executed by the value log device in the example of the present invention, and the device can be realized by software and/hardware, as shown in FIG. 3 , the device includes: an operation analysis module 301 , an operation conversion module 302 and an execution module 303 .
  • Operation parsing module This module is responsible for parsing the received operation request and determining the operation type.
  • Operation conversion module This module is used to convert data update operations.
  • the operation type is a data update (including insert, delete, modify) operation
  • the operation conversion module converts both the delete and modify operations into insert operations.
  • Execution module This module completes data insertion (insert, update, and delete have been converted into insert operations), and generates physical records in memory with the same format as in the final value log file. Then write the physical records in the memory directly to the end of the value log file, and then send the operation results to the client, or write the physical records to the value log buffer, and wait for the disk writing thread/process to flush the disk. After the disk flushing is completed , and then send the operation result to the client.
  • the value log buffer module is used to cache physical records generated by data update operations. After the execution module generates the physical records in the memory, it will write them into the value log buffer, and then wait for the disk writing thread/process to flush the disk;
  • the value log writing module continuously writes the physical records of the value log buffer to the end of the value log file sequentially, and notifies the worker thread/process to complete the task of flushing the disk after writing.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Debugging And Monitoring (AREA)

Abstract

一种数据和日志一体化的值日志实现方法、装置、设备及存储介质，其中方法包括：数据库或者KV等数据处理系统的工作线程/进程接收操作请求，判断操作类型；当操作类型是数据更新操作时，工作线程/进程将删除、修改操作都转化为插入操作；接下来工作线程/进程在内存中进行插入操作并生成物理记录，之后可以直接将物理记录写入到值日志文件尾部，或者先写入到值日志缓冲区，然后等待写盘线程/进程刷盘，写盘线程/进程不断将值日志缓冲区的物理记录顺序写入到值日志文件末尾，并通知工作线程/进程刷盘任务完成。这样，整个装置只有顺序写，性能比传统存在随机写的情况提升10倍甚至更多的写入性能；只用写1份数据，避免传统数据库需要写3-4份数据的情况。

Description

一种数据和日志一体化的值日志实现方法、装置、设备及存储介质 技术领域
本发明涉及数据库技术领域，尤其是一种数据和日志一体化的值日志实现方法、装置、设备及存储介质。
背景技术
日志是数据库和各种Key-Value(简称为KV)等数据处理系统的重要组成部分，数据库/KV等数据处理系统只需要将数据页中修改的内容持久化到日志中，便可以只对该数据在内存中的拷贝进行修改，而不需要每次修改都将数据写到磁盘。这样的好处是日志写入是一小块区域的顺序I/O，而通常数据库/KV等数据处理系统数据的写入是随机I/O，通过日志的持久化，既保障了数据存储的可靠性，又提高了数据写入的效率。
在实际应用中，为了保证系统可以恢复到任意时间点，通常现有数据库和各种基于KV的数据处理的系统都还会开启归档日志。另外，为了满足多版本并发控制的需要，大多数数据库如Oracle等在更新数据时，还会把历史版本的数据写入到回滚段以便其他事务可以访问修改前的数据。因此，传统数据库在实际应用中相当于需要写四份数据：第一份是用户实际的数据，第二份是循环重写的联机重做日志，第三份是数据更新时写入的回滚段数据，最后一份是归档日志。传统数据库在实际生产环境需要写三到四份数据，导致整个过程处理复杂，性能低下。
发明内容
本发明所要解决的技术问题在于，提供一种数据和日志一体化的值日志实现方法、装置、设备及存储介质，只有顺序写，性能比传统存在随机写的情况提升10倍甚至更多的写入性能，只用写1份数据，避免传统数据库需要写3-4份数据的情况。
为解决上述技术问题，本发明提供一种数据和日志一体化的值日志实现方法，包括如下步骤：
(1)数据处理系统中的工作线程或进程解析接收到的操作请求，确定操作类型；
(2)当操作类型是数据更新操作时，工作线程或进程将删除、修改操作都转化为插入操作；
(3)工作线程或进程在内存中执行插入操作并在内存中生成和最终写入到磁盘存储设备的相同格式的物理记录，随后将内存中的物理记录直接写入到值日志文件，或先写入到值日志缓冲区，等待写盘线程或进程刷盘，待刷盘完成后，把操作结果发送给客 户端；
(4)启用值日志缓冲区时，写盘线程或进程不断将值日志缓冲区的物理记录顺序写入到值日志文件末尾，并通知工作线程或进程完成刷盘任务。由于整个数据更新的历史都在值日志文件中，这个文件可以同时起到数据、联机日志、回滚段、归档日志的功能，称之为值日志系统。
优选的，步骤(2)中，数据更新操作包括插入、删除和修改。
优选的，步骤(2)中，工作线程或进程将删除、修改操作都转化为插入操作具体为：将删除操作转化为插入一条含有键和删除标记的记录，修改操作转化为插入一条含有键和删除标记的记录和插入一条新的修改后的记录，使得整个系统中所有的数据更新操作都变成了插入操作。
相应的，一种数据和日志一体化的值日志实现装置，包括：
操作解析模块，负责解析接收到的操作请求，确定操作类型；
操作转化模块，用于对数据更新操作进行转化；
执行模块，完成数据插入操作，并在内存中生成和最终值日志文件中格式一致的物理记录；之后选择将物理记录直接写入到值日志文件尾部，或者先写入到值日志缓冲区，然后由专门的值日志写入模块负责把值日志缓冲区中的物理记录写入到值日志文件，完成刷盘后，把操作结果发送给客户端；
值日志缓冲区模块，用于缓存数据更新操作生成的物理记录；执行模块生成内存中的物理记录后，可以选择把物理记录写入到值日志缓冲区。
值日志写入模块，写盘线程不断将值日志缓冲区的物理记录顺序写入到值日志文件末尾，并通知工作线程/进程刷盘任务完成。
优选的，操作转化模块用于对数据更新操作进行转化具体为：当操作类型是数据更新操作时，操作转化模块将删除、修改操作都转化为插入操作，将删除操作转化为插入一条含有键和删除标记的记录，修改操作转化为插入一条含有键和删除标记的记录和插入一条新的修改后的记录。
优选的，数据更新操作包括插入、删除和修改。
相应的，一种数据和日志一体化的值日志实现设备，包括：一个或多个处理器；
存储装置，用于存储一个或多个程序、用户数据；
当所述一个或多个程序被一个或多个处理器执行，使得所述一个或多个处理器实现 如本发明实施例中任一所述的数据和日志一体化的值日志实现方法。
相应的，一种数据和日志一体化的值日志实现存储介质，其上存储有计算机程序，该程序被处理器执行时实现如本发明实施例中任一所述的一种数据和日志一体化的值日志实现方法。
本发明的有益效果为：整个系统只有顺序写，性能比传统存在随机写的情况提升10倍甚至更多的写入性能；只用写1份数据，避免传统数据库需要写3-4份数据的情况。
附图说明
图1为本发明实施例1提供的实现方法流程示意图。
图2为本发明实施例2提供的实现方法流程示意图。
图3为本发明实施例3提供的实现方法流程示意图。
具体实施方式
本发明实施例通过在数据库或者KV等数据处理系统中将数据更新操作转化为在文件尾部的顺序写入操作实现用户数据、联机日志、回滚文件、归档日志文件的一体化。
数据库或者KV等数据处理系统的工作线程/进程接收操作请求，判断操作类型。当操作类型是数据更新(包括插入、删除、修改)操作时，工作线程/进程将删除、修改操作都转化为插入操作，如将删除操作转化为插入一条含有键和删除标记的记录(表示包含对应键的记录已经删除)，修改操作转化为插入一条含有键和删除标记的记录和插入一条新的修改后的记录。接下来工作线程/进程在内存中进行插入操作并生成物理记录，之后可以直接将物理记录写入到值日志文件尾部，或者先写入到值日志缓冲区，然后等待写盘线程/进程刷盘。写盘线程/进程不断将值日志缓冲区的物理记录顺序写入到值日志文件末尾，并通知工作线程/进程刷盘任务完成。这样使得整个系统中所有的数据更新操作都变成了顺序写操作。最后工作线程/进程通知客户端操作完成。
实施例一：
图1为本发明实施例一提供的一种值日志实现方法流程图。本实施例可用于实现值日志系统，该方法可以由本发明实例中的值日志装置来执行，该装置可以采用软件和/硬件的方式实现，如图1所示，该方法具体包括如下步骤：
S101，数据库/KV等数据处理系统接收处理请求，并确定操作类型。
S102，当操作类型是数据更新(包括插入、删除、修改)操作时，工作线程/进程将删除、修改操作都转化为插入操作，如将删除操作转化为插入一条含有键和删除标记 的记录(表示包含对应键的记录已经删除，删除标记在具体实现时可以有各种不同表示方法)，修改操作转化为插入一条含有键和删除标记的记录和插入一条新的修改后的记录。这样使得整个系统中所有的数据更新操作都变成了插入操作。
以数据库中删除语句为例，假定存在表T1，PK是主键列，执行如下语句：
DELETE FROM T1 WHERE PK＝1；
在处理时，转化为类似如下伪语句
INSERT INTO T1(PK,LSN,FLAG)VALUES(1,当前LSN,DELETE_FLAG)；
其中LSN与数据库系统中常见的日志顺序号的含义相同，是一个自增的整数，执行该语句时，主键索引中包含了LSN作为索引的最后一列，且LSN按降序排列，这样可以确保后续查询先查询到最近的记录。
之后用于如果执行下面的查询
SELECT*FROM T1 WHERE PK＝1；
由于LSN按降序排列，就会最先查找到带删除标记的记录，而不是删除操作前的记录，带删除标记表示已经被删除，直接返回空结果集给用户。
可选的，可以采用不同的标志技术来表示某个记录已经删除。
可选的，LSN可以采用其他类似的方案来实现，甚至可以让LSN按升序排列，这样的话，需要把相同KEY的记录一直扫描到最后一个可见的结果为止，对于上述用例，如果LSN是按升序排列，则相同KEY扫描到最后一条记录，可以发现有删除标记，从而返回空结果集。
可选的，对于修改操作，如果存在类似数据库中主键或者KV系统中KEY来保证唯一性的话，可以直接将修改操作转化为插入一条新的修改后的记录，而不用先生成插入一个带删除标记的记录，以进一步提升性能。
S103，工作线程/进程在内存中执行插入操作并在内存中生成和最终写入到磁盘等存储设备的相同格式的物理记录，随后可以将内存中的物理记录写入到值日志缓冲区，等待写盘线程/进程刷盘，待刷盘完成后，把操作结果发送给客户端。
具体的，值日志缓冲区和传统数据库的日志缓冲区类似，可以循环使用，只要对应的缓冲区内容已经写入磁盘，就可以重用对应的缓冲区空间。
S104，由刷盘线程/进程不断将值日志缓冲区的物理记录顺序写入到值日志文件末尾，并通知工作线程/进程完成刷盘任务。
实施例二：
图2为本发明实施例二提供的一种值日志实现方法流程图。本实施例可用于实现值日志系统，该方法可以由本发明实例中的值日志装置来执行，该装置可以采用软件和/硬件的方式实现，如图2所示，该方法具体包括如下步骤：
S201，数据库/KV等数据处理系统接收处理请求，并确定操作类型。
S202，当操作类型是数据更新(包括插入、删除、修改)操作时，工作线程/进程将删除、修改操作都转化为插入操作，如将删除操作转化为插入一条含有键和删除标记的记录(表示包含对应键的记录已经删除，删除标记在具体实现时可以有各种不同表示方法)，修改操作转化为插入一条含有键和删除标记的记录和插入一条新的修改后的记录。这样使得整个系统中所有的数据更新操作都变成了插入操作。
以数据库中删除语句为例，假定存在表T1，PK是主键列，执行如下语句：
DELETE FROM T1 WHERE PK＝1；
在处理时，转化为类似如下伪语句
INSERT INTO T1(PK,LSN,FLAG)VALUES(1,当前LSN,DELETE_FLAG)；
其中LSN与数据库系统中常见的日志顺序号的含义相同，是一个自增的整数，执行该语句时，主键索引中包含了LSN作为索引的最后一列，且LSN按降序排列，这样可以确保后续查询先查询到最近的记录。
之后用于如果执行下面的查询
SELECT*FROM T1 WHERE PK＝1；
由于LSN按降序排列，就会最先查找到带删除标记的记录，而不是删除操作前的记录，带删除标记表示已经被删除，直接返回空结果集给用户。
可选的，可以采用不同的标志技术来表示某个记录已经删除。
可选的，LSN可以采用其他类似的方案来实现，甚至可以让LSN按升序排列，这样的话，需要把相同KEY的记录一直扫描到最后一个可见的结果为止，对于上述用例，如果LSN是按升序排列，则相同KEY扫描到最后一条记录，可以发现有删除标记，从而返回空结果集。
可选的，对于修改操作，如果存在类似数据库中主键或者KV系统中KEY来保证唯一性的话，可以直接将修改操作转化为插入一条新的修改后的记录，而不用先生成插入一个带删除标记的记录，以进一步提升性能。
S203，工作线程/进程在内存中执行插入操作并在内存中生成和最终写入到磁盘等存储设备的相同格式的物理记录，随后直接把物理记录写入值日志文件尾部，待刷盘完成后，把操作结果发送给客户端。
实施例三：
图3为本发明实施例二提供的一种值日志实现方法流程图。本实施例可用于实现值日志系统，该方法可以由本发明实例中的值日志装置来执行，该装置可以采用软件和/硬件的方式实现，如图3所示，该装置包括：操作解析模块301、操作转化模块302、执行模块303。
操作解析模块：该模块负责解析接收到的操作请求，确定操作类型。
操作转化模块：该模块用于对数据更新操作进行转化。当操作类型是数据更新(包括插入、删除、修改)操作时，操作转化模块将删除、修改操作都转化为插入操作。
执行模块：该模块完成数据插入(插入、更新和删除都已经转化为插入操作)操作，并在内存中生成和最终值日志文件中格式一致的物理记录。随后将内存中的物理记录直接写入值日志文件尾部，再把操作结果发送给客户端，或者把物理记录写入到值日志缓冲区，等待写盘线程/进程刷盘，待刷盘完成后，再把操作结果发送给客户端。
可选的，值日志缓冲区模块用于缓存数据更新操作生成的物理记录。执行模块生成内存中的物理记录后，会写入到值日志缓冲区，然后等待写盘线程/进程刷盘；
可选的，值日志写入模块不断将值日志缓冲区的物理记录顺序写入到值日志文件末尾，并在写完后通知工作线程/进程完成刷盘任务。

Claims (8)

  1. 一种数据和日志一体化的值日志实现方法，其特征在于，包括如下步骤：
    (1)数据处理系统中的工作线程或进程解析接收到的操作请求，确定操作类型；
    (2)当操作类型是数据更新操作时，工作线程或进程将删除、修改操作都转化为插入操作；
    (3)工作线程或进程在内存中执行插入操作并在内存中生成和最终写入到磁盘存储设备的相同格式的物理记录，随后将内存中的物理记录直接写入到值日志文件，或先写入到值日志缓冲区，等待写盘线程或进程刷盘，待刷盘完成后，把操作结果发送给客户端；
    (4)启用值日志缓冲区时，写盘线程或进程不断将值日志缓冲区的物理记录顺序写入到值日志文件末尾，并通知工作线程或进程完成刷盘任务。
  2. 如权利要求1所述的数据和日志一体化的值日志实现方法，其特征在于，步骤(2)中，数据更新操作包括插入、删除和修改。
  3. 如权利要求1所述的数据和日志一体化的值日志实现方法，其特征在于，步骤(2)中，工作线程或进程将删除、修改操作都转化为插入操作具体为：将删除操作转化为插入一条含有键和删除标记的记录，修改操作转化为插入一条含有键和删除标记的记录和插入一条新的修改后的记录，使得整个系统中所有的数据更新操作都变成了插入操作。
  4. 一种数据和日志一体化的值日志实现装置，其特征在于，包括：
    操作解析模块，负责解析接收到的操作请求，确定操作类型；
    操作转化模块，用于对数据更新操作进行转化；
    执行模块，完成数据插入操作，并在内存中生成和最终值日志文件中格式一致的物理记录；之后选择将物理记录直接写入到值日志文件尾部，或者先写入到值日志缓冲区，然后由专门的值日志写入模块负责把值日志缓冲区中的物理记录写入到值日志文件，完成刷盘后，把操作结果发送给客户端；
    值日志缓冲区模块，用于缓存数据更新操作生成的物理记录；执行模块生成内存中的物理记录后，可以选择把物理记录写入到值日志缓冲区。
    值日志写入模块，写盘线程不断将值日志缓冲区的物理记录顺序写入到值日志文件末尾，并通知工作线程/进程刷盘任务完成。
  5. 如权利要求4所述的数据和日志一体化的值日志实现装置，其特征在于，操作 转化模块用于对数据更新操作进行转化具体为：当操作类型是数据更新操作时，操作转化模块将删除、修改操作都转化为插入操作，将删除操作转化为插入一条含有键和删除标记的记录，修改操作转化为插入一条含有键和删除标记的记录和插入一条新的修改后的记录。
  6. 如权利要求4所述的数据和日志一体化的值日志实现装置，其特征在于，数据更新操作包括插入、删除和修改。
  7. 一种数据和日志一体化的值日志实现设备，其特征在于，包括：一个或多个处理器；
    存储装置，用于存储一个或多个程序、用户数据；
    当所述一个或多个程序被一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求1至3任一所述的数据和日志一体化的值日志实现方法。
  8. 一种数据和日志一体化的值日志实现存储介质，其特征在于，其上存储有计算机程序，该程序被处理器执行时实现如权利要求1至3中任一所述的一种数据和日志一体化的值日志实现方法。
PCT/CN2021/112761 2021-06-29 2021-08-16 一种数据和日志一体化的值日志实现方法、装置、设备及存储介质 WO2023272895A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110728025.3A CN113419937A (zh) 2021-06-29 2021-06-29 一种数据和日志一体化的值日志实现方法、装置、设备及存储介质
CN202110728025.3 2021-06-29

Publications (1)

Publication Number Publication Date
WO2023272895A1 true WO2023272895A1 (zh) 2023-01-05

Family

ID=77717112

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/112761 WO2023272895A1 (zh) 2021-06-29 2021-08-16 一种数据和日志一体化的值日志实现方法、装置、设备及存储介质

Country Status (2)

Country Link
CN (1) CN113419937A (zh)
WO (1) WO2023272895A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117348586A (zh) * 2023-10-11 2024-01-05 江苏云涌电子科技股份有限公司 一种基于储能ems系统的事件顺序记录soe实现方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120221528A1 (en) * 2011-01-14 2012-08-30 Sap Ag Logging scheme for column-oriented in-memory databases
US20170351731A1 (en) * 2016-06-03 2017-12-07 Dell Products L.P. Maintaining i/o transaction metadata in log-with-index structure
CN110109927A (zh) * 2019-04-25 2019-08-09 上海新炬网络技术有限公司 基于LSM树的Oracle数据库数据处理方法
CN110609813A (zh) * 2019-08-14 2019-12-24 北京华电天仁电力控制技术有限公司 一种数据存储系统及方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5244332B2 (ja) * 2006-10-30 2013-07-24 株式会社日立製作所 情報システム、データ転送方法及びデータ保護方法
CN110209642A (zh) * 2018-02-05 2019-09-06 北京智明星通科技股份有限公司 信息处理的方法、装置、服务器及计算机可读介质
JP7354631B2 (ja) * 2018-08-10 2023-10-03 株式会社デンソー 電子制御装置、車両用電子制御システム、差分データの整合性判定方法及び差分データの整合性判定プログラム

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120221528A1 (en) * 2011-01-14 2012-08-30 Sap Ag Logging scheme for column-oriented in-memory databases
US20170351731A1 (en) * 2016-06-03 2017-12-07 Dell Products L.P. Maintaining i/o transaction metadata in log-with-index structure
CN110109927A (zh) * 2019-04-25 2019-08-09 上海新炬网络技术有限公司 基于LSM树的Oracle数据库数据处理方法
CN110609813A (zh) * 2019-08-14 2019-12-24 北京华电天仁电力控制技术有限公司 一种数据存储系统及方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117348586A (zh) * 2023-10-11 2024-01-05 江苏云涌电子科技股份有限公司 一种基于储能ems系统的事件顺序记录soe实现方法
CN117348586B (zh) * 2023-10-11 2024-02-27 江苏云涌电子科技股份有限公司 一种基于储能ems系统的事件顺序记录soe实现方法

Also Published As

Publication number Publication date
CN113419937A (zh) 2021-09-21

Similar Documents

Publication Publication Date Title
US11176140B2 (en) Updating a table using incremental and batch updates
US10754875B2 (en) Copying data changes to a target database
US9922077B2 (en) Reducing the cost of update, delete, and append-only insert operations in a database
CN107835983B (zh) 使用一致的数据库快照在分布式数据库中进行备份和还原
US11899641B2 (en) Trie-based indices for databases
CN109952564B (zh) 数据库系统中测试数据的形成与操纵
EP3495961B1 (en) System and methods for migrating database data by using an image copy
US9639542B2 (en) Dynamic mapping of extensible datasets to relational database schemas
US11347701B2 (en) Live zero downtime migration of databases with disparate schemata
US5873101A (en) Database backup/restore and bulk data transfer
US9471622B2 (en) SCM-conscious transactional key-value store
US20070078909A1 (en) Database System
US10289709B2 (en) Interleaved storage of dictionary blocks in a page chain
US7941451B1 (en) Dynamic preconditioning of a B+ tree
US8595190B2 (en) Methods and apparatus related to completion of large objects within a DB2 database environment
US10083192B2 (en) Deleted database record reuse
US7765247B2 (en) System and method for removing rows from directory tables
WO2023272895A1 (zh) 一种数据和日志一体化的值日志实现方法、装置、设备及存储介质
KR20220091444A (ko) 분산 데이터베이스의 글로벌 보조 인덱스 방법 및 이의 장치
CN106155837B (zh) 一种主备库数据修复的方法以及装置
US11609909B2 (en) Zero copy optimization for select * queries
US20090276603A1 (en) Techniques for efficient dataloads into partitioned tables
WO2024108640A1 (zh) 一种支持行级并发控制的纯列式更新方法及装置
US10360145B2 (en) Handling large writes to distributed logs
US10997164B2 (en) Unified table delta dictionary lazy materialization

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21947833

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE