WO2017041638A1 - 日志数据处理方法及装置 - Google Patents

日志数据处理方法及装置 Download PDF

Info

Publication number
WO2017041638A1
WO2017041638A1 PCT/CN2016/097267 CN2016097267W WO2017041638A1 WO 2017041638 A1 WO2017041638 A1 WO 2017041638A1 CN 2016097267 W CN2016097267 W CN 2016097267W WO 2017041638 A1 WO2017041638 A1 WO 2017041638A1
Authority
WO
WIPO (PCT)
Prior art keywords
log
queue
transaction
read
log queue
Prior art date
Application number
PCT/CN2016/097267
Other languages
English (en)
French (fr)
Inventor
王晶昱
Original Assignee
阿里巴巴集团控股有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 阿里巴巴集团控股有限公司 filed Critical 阿里巴巴集团控股有限公司
Publication of WO2017041638A1 publication Critical patent/WO2017041638A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems

Definitions

  • the present application relates to the field of database technologies, and in particular, to a log data processing method and apparatus.
  • the transaction log is one of the important parts of the database, mainly to record the transaction behavior of the transaction on its corresponding database and the log file modified to the database.
  • the database utilizes transaction logs to ensure features such as Durability and transaction rollback.
  • the transaction log is managed by a single machine queue to ensure the serial execution of the transaction through the advanced first of the transaction log.
  • each application server needs to be queued for transaction log writes, which is limited in scalability and processing performance.
  • the distributed database system in order to ensure that transactions are serially executed in the order of submission, the distributed database system has to adopt a single-machine queue, facing a single-machine queue bottleneck.
  • aspects of the present application provide a log data processing method and apparatus for solving a stand-alone queue bottleneck faced by a distributed database system in transaction log management, and improving scalability and performance of a distributed database system in transaction logs.
  • An aspect of the present application provides a log data processing method, including:
  • the selecting a target log queue from the at least two log queues includes:
  • a log queue is randomly selected from the at least two log queues as the target log queue.
  • the previous object in the target log queue is a logical time label periodically inserted at a specified time interval of the log server where the target log queue is located, or is in the same The transaction log written at the last moment in the time interval.
  • Another aspect of the present application provides a log data processing method, including:
  • the reading sequence of the specified log queue and the storage time of the transaction log are sequentially read into each of the at least two log queues in each time period.
  • Transaction log including:
  • Transaction logs corresponding to the same logical time label are sequentially read from each of the at least two log queues according to the log queue reading order and the insertion order of the logical time labels in the log queue; each log queue The logical time stamp in the log server is periodically inserted into each log queue at the same time interval.
  • the reading in the log queue reading order and the logical time label in the log queue is sequentially read from each of the at least two log queues.
  • the transaction log for the same logical time label including:
  • the first inserted logical time label after the current logical time label is re-determined as the current logical time label, and according to the log queue reading order, the first log queue is read corresponding to the Determining the transaction log of the current logical time label of the re-determination, and continuing to perform an operation of determining whether the currently read log queue is the last log queue that should be read according to the read order of the log queue;
  • the transaction log corresponding to the current logical time label in the next log queue is continuously read according to the log queue reading order, and the execution continues to determine whether the currently read log queue is in accordance with the log queue. The operation of the last log queue that the read order should read.
  • a log data processing apparatus including:
  • a receiving module configured to receive a transaction commit request of the application server
  • a write module configured to write a transaction log corresponding to the transaction commit request to the target log queue.
  • the selecting module is specifically configured to:
  • a log queue is randomly selected from the at least two log queues as the target log queue.
  • the transaction log is in a previous one of the target log queues.
  • the object is a logical time label periodically inserted at a specified time interval by the log server where the target log queue is located, or a transaction log written at a previous time in the same time interval.
  • a log data processing apparatus including:
  • the reading module is configured to sequentially read the transaction log in each of the at least two log queues in each time period according to the specified log queue reading order and the transaction log storage time;
  • An execution module is configured to perform a database operation according to the transaction log.
  • the reading module is specifically configured to:
  • Transaction logs corresponding to the same logical time label are sequentially read from each of the at least two log queues according to the log queue reading order and the insertion order of the logical time labels in the log queue; each log queue The logical time stamp in the log server is periodically inserted into each log queue at the same time interval.
  • the reading module is specifically configured to:
  • the first inserted logical time label after the current logical time label is re-determined as the current logical time label, and according to the log queue reading order, the first log queue is read corresponding to the Determining the transaction log of the current logical time label of the re-determination, and continuing to perform an operation of determining whether the currently read log queue is the last log queue that should be read according to the read order of the log queue;
  • the transaction log corresponding to the current logical time label in the next log queue is continuously read according to the log queue reading order, and the execution continues to determine whether the currently read log queue is in accordance with the log queue. The operation of the last log queue that the read order should read.
  • a multi-log queue is used to store the transaction log, allowing parallel writing of the transaction log.
  • the reading is sequentially performed in each
  • the transaction log in each log queue of at least two log queues is stored in the time period to ensure the order and certainty of the transaction log when it is dequeued, to meet the basic requirements of the database for the transaction log, and to solve the distributed database.
  • the single-machine queue bottleneck faced by the system in transaction log management improves the scalability and processing performance of the distributed database system in transaction log.
  • FIG. 1 is a schematic flowchart of a method for processing log data according to an embodiment of the present application
  • FIG. 2 is a schematic diagram of an application server committing a transaction in a distributed database system according to an embodiment of the present disclosure
  • FIG. 3 is a schematic structural diagram of a log server cluster according to an embodiment of the present application.
  • FIG. 3b is a schematic diagram of a state in which a log server stores a transaction log according to an embodiment of the present disclosure
  • FIG. 4 is a schematic flowchart of a method for processing log data according to another embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of a log data processing apparatus according to an embodiment of the present disclosure.
  • FIG. 6 is a schematic structural diagram of a log data processing apparatus according to another embodiment of the present disclosure.
  • both the stand-alone database and the distributed database system can only use the single machine queue to manage the transaction log, so as to ensure the transaction according to the commit order through the first-in-first-out feature of the transaction log.
  • Serial execution Managing transaction logs in a single machine queue, because it does not support parallel write transaction logs, becomes the core bottleneck of distributed database systems, resulting in scalability and processing performance degradation.
  • the inventors of the present application reinterpreted the transaction queue from a different perspective from the conventional concept. Specifically, the inventor finds that the order in which the transaction logs are submitted is not necessarily the order in which the transactions are actually generated. For example, due to the network or the like, the transactions that are generated first may be submitted instead of being committed. Based on this finding, the inventor believes that the enqueue operation of the transaction log is actually an out-of-order write queue operation. From this perspective, it means that there is no need to guarantee the enqueue order of the transaction log. However, there is still a need to ensure the order and certainty of the transaction dequeuing to meet the basic needs of the execution of the transaction sequence.
  • the present application proposes a scheme for managing transaction logs by multiple queues.
  • An extension to the single-queue scheme In the multi-queue scheme, there is no need to guarantee the enqueue sequence of the transaction log, allowing the transaction log to be written in parallel, and the transaction log is dequeued in a certain way to ensure its order and certainty.
  • the multi-queue scheme is especially suitable for distributed database systems.
  • the queue can guarantee its order and certainty, can meet its basic requirements for transaction logs, solve the single-machine queue bottleneck faced by distributed database systems, and improve the scalability and processing performance of distributed database systems.
  • FIG. 1 is a schematic flowchart diagram of a method for processing log data according to an embodiment of the present application. As shown in Figure 1, the method includes:
  • the log data processing method provided by this embodiment can be executed by the log data processing device, and mainly describes the process of the transaction log entering the queue.
  • the transaction log of this embodiment may be a WAL (Write-Ahead Logging) log, but is not limited thereto.
  • a transaction log is generated, and the transaction log mainly records information such as transaction behavior and modification of the database on the database corresponding to the transaction log.
  • the application server needs to issue a transaction commit request to the log data processing device.
  • the log data processing device may receive a transaction commit request sent by the application server, and according to the transaction commit request, the application server needs to write a transaction log to the log queue, so select a target log queue from at least two log queues.
  • the transaction log corresponding to the transaction commit request is then written to the target log queue.
  • the transaction commit request carries information related to the application server and the current transaction, such as a transaction id, an application server id, and the like.
  • the log data processing apparatus may randomly select one of the at least two log queues as the target log queue, which is simpler and more efficient.
  • a pre-set log queue usage policy such as a recurring policy, can also be used. Based on this, the log data processing device can select a corresponding log queue (for example, a log queue to be recycled) as a target log queue according to a preset log queue usage policy.
  • the log data processing apparatus may allocate transaction logs of different application servers to at least two log queues without receiving multiple transaction commit requests, instead of using Queued waiting to be written to the same log queue as in the prior art. For example, if there are 3 log queues in total and 2 application servers issue transaction commit requests, you can directly write the transaction logs of the two application servers to any of the 2 log queues. You do not need to wait in line, you can write 2 transaction logs concurrently.
  • the transaction logs of the 4 application servers can be allocated to the 3 log queues, and 2 of the transaction logs need to be allocated.
  • the same log queue only one of the two transaction logs assigned to the same log queue needs to wait for another write to be written, and the other two transaction logs do not need to wait, and can simultaneously write three transactions simultaneously.
  • Log. Therefore, in this embodiment, by using multiple log queues, multiple transaction logs can be written concurrently, thereby reducing or avoiding the waiting time during the transaction log writing to the log queue, which is beneficial to improving the scalability and processing performance in the transaction log. .
  • At least two log queues in this embodiment may be separately deployed on different servers, that is, one log queue is separately deployed on one server; or, may be a server.
  • the number of log queues is less than the number of log queues.
  • At least two log queues are allocated to these servers.
  • all log queues can be deployed on one server at the same time.
  • the server where the log queue is located may periodically insert a logical time label into the log queue at a specified time interval, which may be recorded as epoch.
  • the servers on all the log queues periodically insert logical time labels into the log queue at the same interval, so that the same logical time labels in all log queues correspond to the same time period. This logical time stamp allows you to mark the transaction logs inserted in the same time period in each log queue.
  • the log object processing device writes the previous object of the transaction log in the target log queue to be the transaction log written at the previous time in the same time interval, or the log server where the target log queue is located according to the specified A logical time stamp that is periodically inserted at intervals.
  • the distributed database system includes four application servers, which are a first application server, a second application server, a third application server, and a fourth application server.
  • the first application server generates two transactions, which are respectively Transaction 0 and transaction 1;
  • the second application server generates 2 transactions, respectively transaction 2 And transaction 3;
  • the third application server generates 2 transactions, namely transaction 4 and transaction 5;
  • the fourth application server generates 2 transactions, which are transaction 6 and transaction 7.
  • At least two log queues are specifically configured to form a log queue cluster, and each log queue in the cluster is respectively deployed on the first log server, the second log server, and the third log server.
  • Each log server inserts a logical time stamp into the log queue every n ms, and the serial number of the logical time label increases from 0.
  • the log data processing device receives the transaction commit request issued by the application server in FIG. 2, and randomly writes the transaction log of the corresponding transaction into the log queue in the log server shown in FIG. 3a according to the request, and the final result is shown in FIG. 3b.
  • the process of writing the transaction log to the log queue there is no need to guarantee the write order, and parallel writes are implemented through multiple log queues, making the transaction log write operation more flexible and convenient.
  • FIG. 4 is a schematic flowchart diagram of a method for processing log data according to another embodiment of the present application. As shown in FIG. 4, the method includes:
  • the log data processing method provided in this embodiment can be executed by the log data processing device, and mainly describes the process of the transaction log out queue.
  • the transaction log of this embodiment may be a WAL log, but is not limited thereto.
  • the log data processing device needs to read the transaction log from the log queue and perform corresponding database operations according to the read transaction log.
  • the corresponding database operation may be an insert operation, a delete operation, a modification operation, or a query operation.
  • the execution of the database operation according to the transaction log belongs to the existing process, and thus is not described in detail in this embodiment. This embodiment focuses on how to read transaction logs from at least two log queues while ensuring order and certainty.
  • the reading order of the log queues is specified in advance, thereby ensuring the order and certainty of the reading from the dimensions of the log queues.
  • the time in which the transaction log is stored in the log queue it is also necessary to ensure the order and certainty of the reading from the time dimension.
  • the log data processing device sequentially reads the transaction log in each of the at least two log queues in each time period according to the specified log queue reading order and the transaction log storage time.
  • the foregoing time period may be specified by using a log server where the log queue is located.
  • the time interval is periodically inserted into the logical time label in the log queue to distinguish, and a logical time label represents a time period.
  • the logical time label in each log queue is periodically inserted into each log queue at the same interval according to the log server where each log queue is located.
  • the log data processing device may specifically read the order of the log queue and the insertion order of the logical time tags in the log queue, for example, generally, from the first to the last, sequentially read from each of the at least two log queues. Take the transaction log corresponding to the same logical time label.
  • the read order of the log queue is from left to right, that is, the log queue on the first log server, the log queue on the second log server, and the log queue on the third log server.
  • the logical time tag is read in the order of epoch0, epoch1.
  • the order of reading the transaction log can be uniquely determined as follows: the transaction log corresponding to epoch0 in the log queue on the first log server, and the corresponding epoch0 in the log queue on the second log server Transaction log, the transaction log corresponding to epoch0 in the log queue on the third log server, the transaction log corresponding to epoch1 in the log queue on the first log server, the transaction log corresponding to epoch1 in the log queue on the second log server, and the third The transaction log corresponding to epoch1 in the log queue on the log server.
  • the transaction logs read based on the uniquely determined read order are: transaction log 4, transaction log 7, transaction log 3, transaction log 5, transaction log 6, transaction log 0, transaction log 2, transaction log 1.
  • the first inserted logical time label after the current logical time label is re-determined as the current logical time label, and according to the log queue reading order, the first log queue is read corresponding to the Determining the transaction log of the current logical time label of the re-determination, and continuing to perform an operation of determining whether the currently read log queue is the last log queue that should be read according to the log queue reading order;
  • the transaction log corresponding to the current logical time label in the next log queue is continuously read according to the log queue reading order, and the execution continues to determine whether the currently read log queue is in the log queue reading order. The operation of the last log queue read.
  • the log queue on the third log server is currently read, and the log queue on the third log server is the last log queue, so the epoch1 after the currently read logical time label epoch0 Re-as the current logical time label; after that, re-read the transaction log corresponding to epoch1 in the log queue on the first log server, that is, transaction log 0, transaction log 2; then, determine whether the log queue on the first log server is the last A log queue that should be read, if the judgment result is no, according to the log queue reading order, continue to read the transaction log corresponding to epoch1 in the log queue on the second log server, that is, transaction log 2; Whether the log queue on the second log server is the last log queue that should be read, and if the judgment result is no, the transaction log corresponding to epoch1 in the log queue on the third log server is continuously read according to the log queue reading order. , and then proceed.
  • the method provided in this embodiment ensures the order and certainty of reading the transaction log, and satisfies the basic requirements of the database for the transaction log.
  • a new transaction log can be added arbitrarily, thereby realizing the horizontal expansion capability of the log system.
  • FIG. 5 is a schematic structural diagram of a log data processing apparatus according to an embodiment of the present disclosure. As shown in FIG. 5, the apparatus includes a receiving module 51, a selecting module 52, and a writing module 53.
  • the receiving module 51 is configured to receive a transaction commit request of the application server.
  • the selecting module 52 is configured to select a target log queue from at least two log queues.
  • the writing module 53 is configured to write the transaction log corresponding to the transaction commit request to the target log queue.
  • the transaction log of this embodiment may be a WAL log, but is not limited thereto.
  • the selecting module 52 is specifically configured to: randomly select one of the at least two log queues as the target log queue.
  • the previous object of the transaction log in the target log queue is the target date.
  • the logical time stamp periodically inserted by the log server where the log queue is located, or the transaction log written at the last time in the same time interval.
  • the log data processing apparatus adopts a multi-log queue, and in the process of writing a transaction log to the log queue, the write sequence is not required, and parallel write is implemented through multiple log queues, so that the transaction log is written.
  • the operation is more flexible and convenient, reducing or avoiding the waiting time during the transaction log writing to the log queue, which is beneficial to improve the scalability and processing performance in the transaction log.
  • FIG. 6 is a schematic structural diagram of a log data processing apparatus according to another embodiment of the present disclosure. As shown in FIG. 6, the apparatus includes a reading module 61 and an execution module 62.
  • the reading module 61 is configured to sequentially read the transaction logs in each of the at least two log queues in each time period according to the specified log queue reading order and the transaction log storage time.
  • the executing module 62 is configured to perform a database operation according to the transaction log read by the reading module 61.
  • the transaction log of this embodiment may be a WAL log, but is not limited thereto.
  • the reading module 61 is specifically configured to:
  • Transaction logs corresponding to the same logical time label are sequentially read from each of the at least two log queues according to the order in which the log queue is read and the logical time label in the log queue is inserted; the logical time in each log queue
  • the tag is the log server where each log queue is located and is periodically inserted into each log queue at the same time interval.
  • the reading module 61 is specifically configured to:
  • the first inserted logical time label after the current logical time label is re-determined as the current logical time label, and according to the log queue reading order, the first log queue is read corresponding to the re Determining the transaction log of the current logical time label, and continuing to perform an operation of determining whether the currently read log queue is the last log queue that should be read according to the read order of the log queue;
  • the transaction log corresponding to the current logical time label in the next log queue is continuously read according to the log queue reading order, and the execution continues to determine whether the currently read log queue is read according to the log queue. The order should be read by the operation of the last log queue.
  • the log data processing apparatus provided in this embodiment cooperates with the log data processing apparatus provided in the foregoing embodiment.
  • the read sequence of the log queue is combined with the storage time of the transaction log, and the read time is sequentially read.
  • the transaction log in each log queue in at least two log queues is used in the time period. In order to ensure the order and certainty of the transaction log when it is dequeued, it satisfies the basic requirements of the database for the transaction log, and solves the single-machine queue bottleneck faced by the distributed database system in transaction log management, and improves the distributed database system in the transaction. Log scalability and processing performance.
  • the disclosed system, apparatus, and method may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.
  • the above-described integrated unit implemented in the form of a software functional unit can be stored in a computer readable storage medium.
  • the software functional unit described above is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to perform the methods described in various embodiments of the present application. Part of the steps.
  • the foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like, which can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Debugging And Monitoring (AREA)

Abstract

一种日志数据处理方法及装置。在方法中，多日志队列来存储事务日志，允许事务日志的并行写入，在读事务日志时，通过指定日志队列的读取顺序，结合事务日志的存入时间，依次读取在每个时间段内存入至少两个日志队列中每个日志队列中的事务日志，用以保证事务日志在出队时的顺序性和确定性，满足数据库对事务日志的基本需求，同时解决了分布式数据库系统在事务日志管理方面面临的单机队列瓶颈，提高了分布式数据库系统在事务日志方面的扩展性和处理性能。

Description

日志数据处理方法及装置
本申请要求2015年09月08日递交的申请号为201510566685.0、发明名称为“日志数据处理方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及数据库技术领域，尤其涉及一种日志数据处理方法及装置。
背景技术
事务日志是数据库的重要部分之一，主要是记录与其对应数据库上的事务行为和对数据库修改的日志文件。数据库利用事务日志可以确保持久性(Durability)和事务回滚(Rollback)等特性。在现有技术中，为了保证事务能够按照提交顺序串行执行，采用单机队列对事务日志进行管理，以通过事务日志的先进先出来保证事务串行执行。
对于分布式数据库系统来说，由于单机队列不支持并行写入事务日志，因此各应用服务器需要排队等待事务日志的写入，在扩展性和处理性能上会受限制。但是，就目前技术来说，为了保证事务按照提交顺序串行执行，分布式数据库系统也不得不采用单机队列，面临单机队列瓶颈。
发明内容
本申请的多个方面提供一种日志数据处理方法及装置，用以解决分布式数据库系统在事务日志管理方面面临的单机队列瓶颈，提高分布式数据库系统在事务日志方面的扩展性和和性能。
本申请的一方面，提供一种日志数据处理方法，包括：
接收应用服务器的事务提交请求；
从至少两个日志队列中选择目标日志队列；
将所述事务提交请求对应的事务日志写入所述目标日志队列。
在本申请的一可选实施方式中，所述从至少两个日志队列中选择目标日志队列，包括：
随机从所述至少两个日志队列中选择一个日志队列作为所述目标日志队列。
在本申请的一可选实施方式中，所述事务日志在所述目标日志队列中的前一个对象为所述目标日志队列所在日志服务器按照指定时间间隔定期插入的逻辑时间标签，或者为在同一所述时间间隔内上一时刻写入的事务日志。
本申请的另一方面，提供一种日志数据处理方法，包括：
按照指定的日志队列读取顺序和事务日志的存入时间，依次读取在每个时间段内存入至少两个日志队列中每个日志队列中的事务日志；
根据所述事务日志，执行数据库操作。
在本申请的一可选实施方式中，所述按照指定的日志队列读取顺序和事务日志的存入时间，依次读取在每个时间段内存入至少两个日志队列中每个日志队列中的事务日志，包括：
按照所述日志队列读取顺序和日志队列中逻辑时间标签的插入顺序，依次从所述至少两个日志队列中每个日志队列中读取对应于同一逻辑时间标签的事务日志；每个日志队列中的逻辑时间标签是每个日志队列所在日志服务器按照相同时间间隔定期插入每个日志队列中的。
在本申请的一可选实施方式中，所述按照所述日志队列读取顺序和日志队列中逻辑时间标签的插入顺序，依次从所述至少两个日志队列中每个日志队列中读取对应于同一逻辑时间标签的事务日志，包括：
判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列；
若判断结果为是，将在当前逻辑时间标签之后第一个插入的逻辑时间标签重新确定为当前逻辑时间标签，并按照所述日志队列读取顺序，读取第一个日志队列中对应于所述重新确定的当前逻辑时间标签的事务日志，并继续执行判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列的操作；
若判断结果为否，按照所述日志队列读取顺序继续读取下一个日志队列中对应于当前逻辑时间标签的事务日志，并返回继续执行判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列的操作。
本申请的又一方面，提供一种日志数据处理装置，包括：
接收模块，用于接收应用服务器的事务提交请求；
选择模块，用于从至少两个日志队列中选择目标日志队列；
写入模块，用于将所述事务提交请求对应的事务日志写入所述目标日志队列。
在本申请的一可选实施方式中，所述选择模块具体用于：
随机从所述至少两个日志队列中选择一个日志队列作为所述目标日志队列。
在本申请的一可选实施方式中，所述事务日志在所述目标日志队列中的前一个 对象为所述目标日志队列所在日志服务器按照指定时间间隔定期插入的逻辑时间标签，或者为在同一所述时间间隔内上一时刻写入的事务日志。
本申请的又一方面，提供一种日志数据处理装置，包括：
读取模块，用于按照指定的日志队列读取顺序和事务日志的存入时间，依次读取在每个时间段内存入至少两个日志队列中每个日志队列中的事务日志；
执行模块，用于根据所述事务日志，执行数据库操作。
在本申请的一可选实施方式中，所述读取模块具体用于：
按照所述日志队列读取顺序和日志队列中逻辑时间标签的插入顺序，依次从所述至少两个日志队列中每个日志队列中读取对应于同一逻辑时间标签的事务日志；每个日志队列中的逻辑时间标签是每个日志队列所在日志服务器按照相同时间间隔定期插入每个日志队列中的。
在本申请的一可选实施方式中，所述读取模块具体用于：
判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列；
若判断结果为是，将在当前逻辑时间标签之后第一个插入的逻辑时间标签重新确定为当前逻辑时间标签，并按照所述日志队列读取顺序，读取第一个日志队列中对应于所述重新确定的当前逻辑时间标签的事务日志，并继续执行判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列的操作；
若判断结果为否，按照所述日志队列读取顺序继续读取下一个日志队列中对应于当前逻辑时间标签的事务日志，并返回继续执行判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列的操作。
在本申请中，用多日志队列来存储事务日志，允许事务日志的并行写入，在读事务日志时，通过指定日志队列的读取顺序，结合事务日志的存入时间，依次读取在每个时间段内存入至少两个日志队列中每个日志队列中的事务日志，用以保证事务日志在出队时的顺序性和确定性，满足数据库对事务日志的基本需求，同时解决了分布式数据库系统在事务日志管理方面面临的单机队列瓶颈，提高了分布式数据库系统在事务日志方面的扩展性和处理性能。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例或现有技术描 述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本申请一实施例提供的日志数据处理方法的流程示意图；
图2为本申请一实施例提供的分布式数据库系统中应用服务器提交事务的示意图；
图3a为本申请一实施例提供的日志服务器集群的结构示意图；
图3b为本申请一实施例提供的日志服务器存储事务日志的状态示意图；
图4为本申请另一实施例提供的日志数据处理方法的流程示意图；
图5为本申请一实施例提供的日志数据处理装置的结构示意图；
图6为本申请另一实施例提供的日志数据处理装置的结构示意图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
在现有技术中，为了保证事务按照提交顺序串行执行，所以无论是单机数据库还是分布式数据库系统只能采用单机队列来管理事务日志，以通过事务日志的先进先出特性保证事务按照提交顺序串行执行。以单机队列来管理事务日志，由于不支持并行写入事务日志，因此成为分布式数据库系统的核心瓶颈，导致扩展性和处理性能下降。
针对上述问题，本申请发明人从不同于传统观念的角度重新对事务排队进行理解。具体的，发明人发现事务日志的提交顺序不一定是事务实际产生的先后顺序，例如由于网络等原因可能导致先产生的事务反而后提交而后产生的事务反而先提交。基于这种发现，发明人认为事务日志的入队操作实际上是一个乱序的写队列操作。如果从这个角度来看，意味着无需保证事务日志的入队顺序。但是，仍需保证事务出队的顺序性和确定性，以满足事务顺序执行的基本需求。
正是基于上述发现，本申请提出了以多队列对事务日志进行管理的方案，实现 对单队列方案的扩展。在多队列方案中，无需保证事务日志的入队顺序，允许并行写入事务日志，而在事务日志出队列时采用一定方式可以保证其顺序性和确定性。多队列方案尤其适用于分布式数据库系统，在事务日志入队时，允许其并行写入，一定程度上可以解决应用服务器等待事务日志入队的问题，发挥并发处理的优势，而在事务日志出队列是又可以保证其顺序性和确定性，可以满足其对事务日志的基本需求，可以解决分布式数据库系统面临的单机队列瓶颈，提高分布式数据库系统的扩展性和处理性能。
本申请以下方法实施例将分别从事务日志入队列的角度和事务日志出队列的角度详细说明本申请技术方案的实施流程。
图1为本申请一实施例提供的日志数据处理方法的流程示意图。如图1所示，该方法包括：
101、接收应用服务器的事务提交请求。
102、从至少两个日志队列中选择目标日志队列。
103、将事务提交请求对应的事务日志写入目标日志队列。
本实施例提供的日志数据处理方法可由日志数据处理装置来执行，主要描述事务日志入队列的过程。本实施例的事务日志可以是WAL(Write-Ahead Logging)日志，但不限于此。
具体的，应用服务器(或者俗称为事务端)需要以事务形式对数据库进行操作时，会产生事务日志，该事务日志主要记录与该事务日志对应数据库上的事务行为和对数据库的修改等信息。此时，应用服务器需要向日志数据处理装置发出事务提交请求。对日志数据处理装置来说，可以接收应用服务器发出的事务提交请求，根据该事务提交请求获知该应用服务器需要向日志队列中写入事务日志，于是从至少两个日志队列中选择目标日志队列，之后将事务提交请求对应的事务日志写入目标日志队列中。其中，事务提交请求会携带与应用服务器以及当前事务有关的信息，例如事务id、应用服务器id等。
在一可选实施方式中，日志数据处理装置可以随机从至少两个日志队列中选择一个日志队列作为目标日志队列，这种实现方式更加简单，效率更高。或者，也可以预先设定的日志队列使用策略，例如循环使用策略。基于此，日志数据处理装置可以按照预设的日志队列使用策略，选择相应的日志队列(例如被循环到的日志队列)作为目标日志队列。
对于分布式数据库系统来说，一般会有多个应用服务器。在同一时间，可能会有多个应用服务器同时发出事务提交请求。在本实施例中，由于采用至少两个日志队列，所以日志数据处理装置在接收到多个事务提交请求的情况下，可以将不同应用服务器的事务日志分配到至少两个日志队列中，而不用像现有技术中那样需要排队等待写入同一日志队列中。举例说明，假设一共有3个日志队列，同时有2个应用服务器发出事务提交请求，则可以直接将这2个应用服务器的事务日志写入其中任意2个日志队列中，这2个事务日志都不需要排队等待，可以同时并发写入2个事务日志。又例如，假设共有3个日志队列，同时有4个应用服务器发出事务提交请求，则可以将这4个应用服务器的事务日志分配到这3个日志队列中，其中有2个事务日志需要分配到相同日志队列中，则只有这2个分配到同一日志队列的事务日志中的一个需要等待另一个写入之后再行写入，另外2个事务日志不需要等待，可以同时并发写入3个事务日志。由此可见，本实施例通过采用多日志队列，可以并发写入多个事务日志，减少或避免事务日志写入日志队列过程中的等待时间，有利于提高在事务日志方面的扩展性和处理性能。
值得说明的是，在部署实现上，本实施例的至少两个日志队列可以分别部署于不同的服务器上实现，也就是说，一个日志队列单独部署于一台服务器上；或者，也可以是服务器的数量少于日志队列的数量，至少两个日志队列均分到这几台服务器上；或者，也可以是所有日志队列同时部署于一台服务器上。
在一可选实施方式中，为了给后续出队列操作提供便利条件，日志队列所在服务器可以按照指定时间间隔定期向日志队列中插入逻辑时间标签，可记为epoch。所有日志队列所在服务器均按照相同时间间隔定期向日志队列插入逻辑时间标签，这样所有日志队列中相同逻辑时间标签对应相同时间段。通过该逻辑时间标签可以标记各日志队列中同一时间段内插入的事务日志。
基于上述逻辑时间标签，则日志数据处理装置写入目标日志队列中的事务日志的前一个对象可以是在同一时间间隔内上一时刻写入的事务日志，或者是目标日志队列所在日志服务器按照指定时间间隔定期插入的逻辑时间标签。
下面结合具体应用场景说明向至少两个日志队列写入事务日志的过程。
如图2所示，假设分布式数据库系统包括4个应用服务器，分别是第一应用服务器、第二应用服务器、第三应用服务器和第四应用服务器；第一应用服务器产生2个事务，分别是事务0和事务1；第二应用服务器产生2个事务，分别是事务2 和事务3；第三应用服务器产生2个事务，分别是事务4和事务5；第四应用服务器产生2个事务，分别是事务6和事务7。
如图3a所示，本实施例至少两个日志队列具体为3个，形成一日志队列集群，集群中的每个日志队列分别部署于第一日志服务器、第二日志服务器和第三日志服务器上。每台日志服务器会每隔n个ms向日志队列中插入一个逻辑时间标签，逻辑时间标签的序号从0开始自增。
日志数据处理装置接收图2中应用服务器发出的事务提交请求，根据请求将相应事务的事务日志随机写入图3a所示日志服务器中的日志队列中，最终结果如图3b所示。在事务日志写入日志队列的过程中，不需要保证写入顺序，通过多个日志队列实现并行写入，使得事务日志的写入操作更加灵活方便。
图4为本申请另一实施例提供的日志数据处理方法的流程示意图。如图4所示，该方法包括：
401、按照指定的日志队列读取顺序和事务日志的存入时间，依次读取在每个时间段内存入至少两个日志队列中每个日志队列中的事务日志。
402、根据上述事务日志，执行数据库操作。
本实施例提供的日志数据处理方法可由日志数据处理装置来执行，主要描述事务日志出队列的过程。本实施例的事务日志可以是WAL日志，但不限于此。
日志数据处理装置需要从日志队列中读取事务日志，根据所读取的事务日志执行相应的数据库操作。所述相应的数据库操作可以是插入操作、删除操作、修改操作或查询操作等。其中，根据事务日志执行数据库操作属于现有流程，故本实施例不做详细描述。本实施例重点描述如何在保证顺序性和确定性的情况下从至少两个日志队列中读取事务日志。
具体的，考虑到本实施例具有至少两个日志队列，因此预先指定日志队列的读取顺序，从而从日志队列的维度保证读取的顺序性和确定性。另外，考虑到事务日志存入日志队列的时间有先后，因此还需要从时间维度保证读取的顺序性和确定性。本实施例按照事务日志的存入时间由先到后的顺序，依次读取各时间段内存入的事务日志，从而保证时间维度读取的顺序性和确定性。于是，日志数据处理装置按照指定的日志队列读取顺序和事务日志的存入时间，依次读取在每个时间段内存入至少两个日志队列中每个日志队列中的事务日志。
在一可选实施方式中，上述时间段可以通过日志队列所在日志服务器按照指定 时间间隔定期插入日志队列中的逻辑时间标签来区分，一个逻辑时间标签代表一个时间段。其中，每个日志队列中的逻辑时间标签是每个日志队列所在日志服务器按照相同时间间隔定期插入每个日志队列中的。
基于上述，日志数据处理装置具体可以按照日志队列读取顺序和日志队列中逻辑时间标签的插入顺序，例如一般是由先到后的顺序，依次从至少两个日志队列中每个日志队列中读取对应于同一逻辑时间标签的事务日志。
结合上述图3b所示场景，假设日志队列的读取顺序为从左到右，即依次是第一日志服务器上的日志队列、第二日志服务器上的日志队列、第三日志服务器上的日志队列；逻辑时间标签的读取顺序依次是epoch0、epoch1。将两个维度的读取顺序进行组合，则可以唯一确定事务日志的读取顺序为：第一日志服务器上的日志队列中epoch0对应的事务日志，第二日志服务器上的日志队列中epoch0对应的事务日志，第三日志服务器上的日志队列中epoch0对应的事务日志，第一日志服务器上的日志队列中epoch1对应的事务日志，第二日志服务器上的日志队列中epoch1对应的事务日志，第三日志服务器上的日志队列中epoch1对应的事务日志。基于该唯一确定的读取顺序所读取的事务日志依次为：事务日志4，事务日志7，事务日志3，事务日志5，事务日志6，事务日志0，事务日志2，事务日志1。
其中，上述按照日志队列读取顺序和日志队列中逻辑时间标签的插入顺序，依次从至少两个日志队列中每个日志队列中读取对应于同一逻辑时间标签的事务日志的过程具体可以是：
判断当前读取的日志队列是否是按照日志队列读取顺序应该读取的最后一个日志队列；
若判断结果为是，将在当前逻辑时间标签之后第一个插入的逻辑时间标签重新确定为当前逻辑时间标签，并按照所述日志队列读取顺序，读取第一个日志队列中对应于所述重新确定的当前逻辑时间标签的事务日志，并继续执行判断当前读取的日志队列是否是按照日志队列读取顺序应该读取的最后一个日志队列的操作；
若判断结果为否，按照日志队列读取顺序继续读取下一个日志队列中对应于当前逻辑时间标签的事务日志，并返回继续执行判断当前读取的日志队列是否是按照日志队列读取顺序应该读取的最后一个日志队列的操作。
例如，假设当前读取到第三日志服务器上的日志队列，第三日志服务器上的日志队列是最后一个日志队列，于是将当前读取的逻辑时间标签epoch0之后的epoch1 重新作为当前逻辑时间标签；之后，重新读取第一日志服务器上日志队列中对应于epoch1的事务日志，即事务日志0，事务日志2；然后，判断第一日志服务器上的日志队列是否是最后一个应该读取到的日志队列，判断结果为否，则按照日志队列读取顺序，继续读取第二日志服务器上的日志队列中对应于epoch1的事务日志，即事务日志2；然后，判断第二日志服务器上的日志队列是否是最后一个应该读取到的日志队列，判断结果为否，则按照日志队列读取顺序，继续读取第三日志服务器上的日志队列中对应于epoch1的事务日志，依次执行下去。
虽然上述从日志队列中读取事务日志的顺序与应用服务器提交的事务编号不同，但因为提交事务过程中有网络延迟等不确定因素，所以到达日志队列的顺序本身就是不确定的，而数据库真正需要的是在处理事务日志时要求处理顺序的确定性，所以本实施例提供的方法保证了读取事务日志的顺序性和确定性，满足了数据库对事务日志的基本需求。
基于本申请提供的多日志队列方案，可以随意的增加新的事务日志，从而实现了日志系统的水平扩展能力。
需要说明的是，对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本申请并不受所描述的动作顺序的限制，因为依据本申请，某些步骤可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本申请所必须的。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。
图5为本申请一实施例提供的日志数据处理装置的结构示意图。如图5所示，该装置包括：接收模块51、选择模块52和写入模块53。
接收模块51，用于接收应用服务器的事务提交请求。
选择模块52，用于从至少两个日志队列中选择目标日志队列。
写入模块53，用于将事务提交请求对应的事务日志写入目标日志队列。
本实施例的事务日志可以是WAL日志，但不限于此。
在一可选实施方式中，选择模块52具体可用于：随机从上述至少两个日志队列中选择一个日志队列作为目标日志队列。
在一可选实施方式中，上述事务日志在目标日志队列中的前一个对象为目标日 志队列所在日志服务器按照指定时间间隔定期插入的逻辑时间标签，或者为在同一时间间隔内上一时刻写入的事务日志。
本实施例提供的日志数据处理装置，通过采用多日志队列，在事务日志写入日志队列的过程中，不需要保证写入顺序，通过多个日志队列实现并行写入，使得事务日志的写入操作更加灵活方便，减少或避免事务日志写入日志队列过程中的等待时间，有利于提高在事务日志方面的扩展性和处理性能。
图6为本申请另一实施例提供的日志数据处理装置的结构示意图。如图6所示，该装置包括：读取模块61和执行模块62。
读取模块61，用于按照指定的日志队列读取顺序和事务日志的存入时间，依次读取在每个时间段内存入至少两个日志队列中每个日志队列中的事务日志。
执行模块62，用于根据读取模块61读取的事务日志，执行数据库操作。
本实施例的事务日志可以是WAL日志，但不限于此。
在一可选实施方式中，读取模块61具体用于：
按照日志队列读取顺序和日志队列中逻辑时间标签的插入顺序，依次从至少两个日志队列中每个日志队列中读取对应于同一逻辑时间标签的事务日志；每个日志队列中的逻辑时间标签是每个日志队列所在日志服务器按照相同时间间隔定期插入每个日志队列中的。
在一可选实施方式中，读取模块61具体用于：
判断当前读取的日志队列是否是按照日志队列读取顺序应该读取的最后一个日志队列；
若判断结果为是，将在当前逻辑时间标签之后第一个插入的逻辑时间标签重新确定为当前逻辑时间标签，并按照日志队列读取顺序，读取第一个日志队列中对应于所述重新确定的当前逻辑时间标签的事务日志，并继续执行判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列的操作；
若判断结果为否，按照日志队列读取顺序继续读取下一个日志队列中对应于当前逻辑时间标签的事务日志，并返回继续执行判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列的操作。
本实施例提供的日志数据处理装置，与上述实施例提供的日志数据处理装置相配合，在读事务日志时，通过指定日志队列的读取顺序，结合事务日志的存入时间，依次读取在每个时间段内存入至少两个日志队列中每个日志队列中的事务日志，用 以保证事务日志在出队时的顺序性和确定性，满足数据库对事务日志的基本需求，同时解决了分布式数据库系统在事务日志管理方面面临的单机队列瓶颈，提高了分布式数据库系统在事务日志方面的扩展性和处理性能。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
上述以软件功能单元的形式实现的集成的单元，可以存储在一个计算机可读取存储介质中。上述软件功能单元存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本申请各个实施例所述方法的部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
最后应说明的是：以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术 特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。

Claims (12)

  1. 一种日志数据处理方法，其特征在于，包括：
    接收应用服务器的事务提交请求；
    从至少两个日志队列中选择目标日志队列；
    将所述事务提交请求对应的事务日志写入所述目标日志队列。
  2. 根据权利要求1所述的方法，其特征在于，所述从至少两个日志队列中选择目标日志队列，包括：
    随机从所述至少两个日志队列中选择一个日志队列作为所述目标日志队列。
  3. 根据权利要求1或2所述的方法，其特征在于，所述事务日志在所述目标日志队列中的前一个对象为所述目标日志队列所在日志服务器按照指定时间间隔定期插入的逻辑时间标签，或者为同一所述时间间隔内上一时刻写入的事务日志。
  4. 一种日志数据处理方法，其特征在于，包括：
    按照指定的日志队列读取顺序和事务日志的存入时间，依次读取在每个时间段内存入至少两个日志队列中每个日志队列中的事务日志；
    根据所述事务日志，执行数据库操作。
  5. 根据权利要求4所述的方法，其特征在于，所述按照指定的日志队列读取顺序和事务日志的存入时间，依次读取在每个时间段内存入至少两个日志队列中每个日志队列中的事务日志，包括：
    按照所述日志队列读取顺序和日志队列中逻辑时间标签的插入顺序，依次从所述至少两个日志队列中每个日志队列中读取对应于同一逻辑时间标签的事务日志；每个日志队列中的逻辑时间标签是每个日志队列所在日志服务器按照相同时间间隔定期插入每个日志队列中的。
  6. 根据权利要求5所述的方法，其特征在于，所述按照所述日志队列读取顺序和日志队列中逻辑时间标签的插入顺序，依次从所述至少两个日志队列中每个日志队列中读取对应于同一逻辑时间标签的事务日志，包括：
    判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列；
    若判断结果为是，将在当前逻辑时间标签之后第一个插入的逻辑时间标签重新确定为当前逻辑时间标签，并按照所述日志队列读取顺序，读取第一个日志队列中对应于所述重新确定的当前逻辑时间标签的事务日志，并继续执行判断当前读取的日志队列是否 是按照所述日志队列读取顺序应该读取的最后一个日志队列的操作；
    若判断结果为否，按照所述日志队列读取顺序继续读取下一个日志队列中对应于当前逻辑时间标签的事务日志，并返回继续执行判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列的操作。
  7. 一种日志数据处理装置，其特征在于，包括：
    接收模块，用于接收应用服务器的事务提交请求；
    选择模块，用于从至少两个日志队列中选择目标日志队列；
    写入模块，用于将所述事务提交请求对应的事务日志写入所述目标日志队列。
  8. 根据权利要求7所述的装置，其特征在于，所述选择模块具体用于：
    随机从所述至少两个日志队列中选择一个日志队列作为所述目标日志队列。
  9. 根据权利要求7或8所述的装置，其特征在于，所述事务日志在所述目标日志队列中的前一个对象为所述目标日志队列所在日志服务器按照指定时间间隔定期插入的逻辑时间标签，或者为在同一所述时间间隔内上一时刻写入的事务日志。
  10. 一种日志数据处理装置，其特征在于，包括：
    读取模块，用于按照指定的日志队列读取顺序和事务日志的存入时间，依次读取在每个时间段内存入至少两个日志队列中每个日志队列中的事务日志；
    执行模块，用于根据所述事务日志，执行数据库操作。
  11. 根据权利要求10所述的装置，其特征在于，所述读取模块具体用于：
    按照所述日志队列读取顺序和日志队列中逻辑时间标签的插入顺序，依次从所述至少两个日志队列中每个日志队列中读取对应于同一逻辑时间标签的事务日志；每个日志队列中的逻辑时间标签是每个日志队列所在日志服务器按照相同时间间隔定期插入每个日志队列中的。
  12. 根据权利要求11所述的装置，其特征在于，所述读取模块具体用于：
    判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列；
    若判断结果为是，将在当前逻辑时间标签之后第一个插入的逻辑时间标签重新确定为当前逻辑时间标签，并按照所述日志队列读取顺序，读取第一个日志队列中对应于所述重新确定的当前逻辑时间标签的事务日志，并继续执行判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列的操作；
    若判断结果为否，按照所述日志队列读取顺序继续读取下一个日志队列中对应于当 前逻辑时间标签的事务日志，并返回继续执行判断当前读取的日志队列是否是按照所述日志队列读取顺序应该读取的最后一个日志队列的操作。
PCT/CN2016/097267 2015-09-08 2016-08-30 日志数据处理方法及装置 WO2017041638A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510566685.0 2015-09-08
CN201510566685.0A CN106503020B (zh) 2015-09-08 2015-09-08 日志数据处理方法及装置

Publications (1)

Publication Number Publication Date
WO2017041638A1 true WO2017041638A1 (zh) 2017-03-16

Family

ID=58240651

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/097267 WO2017041638A1 (zh) 2015-09-08 2016-08-30 日志数据处理方法及装置

Country Status (2)

Country Link
CN (1) CN106503020B (zh)
WO (1) WO2017041638A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111538711A (zh) * 2020-04-14 2020-08-14 中国人民财产保险股份有限公司 日志存储方法及装置
WO2023280053A1 (zh) * 2021-07-07 2023-01-12 阿里云计算有限公司 数据处理方法、系统、电子设备及存储介质

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110445667B (zh) * 2019-05-29 2021-10-26 北京大米科技有限公司 一种链路延迟检测方法、装置、存储介质以及终端
CN111158876B (zh) * 2019-12-26 2023-06-06 杭州安恒信息技术股份有限公司 一种日志处理方法、装置、设备及计算机可读存储介质
CN112181778A (zh) * 2020-09-30 2021-01-05 银盛支付服务股份有限公司 一种日志戳跨系统传递的方法
CN112231290A (zh) * 2020-11-03 2021-01-15 上海智臻智能网络科技股份有限公司 一种本地日志的处理方法、装置、设备及存储介质
CN112559476B (zh) * 2020-12-16 2023-07-21 中国平安人寿保险股份有限公司 一种用于提高目标系统性能的日志存储方法及其相关设备
CN115185787B (zh) * 2022-09-06 2022-12-30 北京奥星贝斯科技有限公司 处理事务日志的方法及装置
CN115905402B (zh) * 2022-09-26 2023-07-18 北京奥星贝斯科技有限公司 处理事务日志的方法及装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102368210A (zh) * 2011-10-25 2012-03-07 曙光信息产业(北京)有限公司 一种分布式系统中基于流水线的元数据日志的方法
CN103092903A (zh) * 2011-11-07 2013-05-08 Sap股份公司 数据库日志并行化
CN103729442A (zh) * 2013-12-30 2014-04-16 华为技术有限公司 记录事务日志的方法和数据库引擎
US20140279855A1 (en) * 2013-03-15 2014-09-18 International Business Machines Corporation Differentiated secondary index maintenance in log structured nosql data stores

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6208345B1 (en) * 1998-04-15 2001-03-27 Adc Telecommunications, Inc. Visual data integration system and method
US7028301B2 (en) * 2000-12-08 2006-04-11 Bmc Software, Inc. System and method for automatic workload characterization
US20070299928A1 (en) * 2006-06-22 2007-12-27 Pooja Kohli Maintaining sessions using cookie updation
CN101459557B (zh) * 2008-11-29 2011-02-02 成都市华为赛门铁克科技有限公司 一种安全日志集中存储方法及装置
CN102831156B (zh) * 2012-06-29 2014-12-31 浙江大学 一种云计算平台上的分布式事务处理方法
CN103259737B (zh) * 2013-04-18 2016-01-13 西安交通大学 一种并行存储高速网络流量的快速定位方法
CN104516959B (zh) * 2014-12-18 2018-01-02 杭州华为数字技术有限公司 一种管理数据库日志的方法及装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102368210A (zh) * 2011-10-25 2012-03-07 曙光信息产业(北京)有限公司 一种分布式系统中基于流水线的元数据日志的方法
CN103092903A (zh) * 2011-11-07 2013-05-08 Sap股份公司 数据库日志并行化
US20140279855A1 (en) * 2013-03-15 2014-09-18 International Business Machines Corporation Differentiated secondary index maintenance in log structured nosql data stores
CN103729442A (zh) * 2013-12-30 2014-04-16 华为技术有限公司 记录事务日志的方法和数据库引擎

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111538711A (zh) * 2020-04-14 2020-08-14 中国人民财产保险股份有限公司 日志存储方法及装置
WO2023280053A1 (zh) * 2021-07-07 2023-01-12 阿里云计算有限公司 数据处理方法、系统、电子设备及存储介质

Also Published As

Publication number Publication date
CN106503020A (zh) 2017-03-15
CN106503020B (zh) 2021-09-21

Similar Documents

Publication Publication Date Title
WO2017041638A1 (zh) 日志数据处理方法及装置
US9632826B2 (en) Prioritizing deferred tasks in pending task queue based on creation timestamp
US8468313B2 (en) Asynchronous replication with write concurrency grouping
US11210229B2 (en) Method, device and computer program product for data writing
CN103092903B (zh) 数据库日志并行化
EP2738695B1 (en) Slave side transaction ID buffering for efficient distributed transaction management
US8694647B2 (en) Read-only operations processing in a paxos replication system
CN105224444B (zh) 日志生成方法及装置
US20170109203A1 (en) Task scheduling
CN107368362B (zh) 一种对于磁盘读写数据的多线程/多进程无锁处理方法及系统
CN103885895A (zh) 容错集群存储系统中的写入性能
CN109960686A (zh) 数据库的日志处理方法和装置
US9229970B2 (en) Methods to minimize communication in a cluster database system
US11210228B2 (en) Method, device and computer program product for cache management
US9135262B2 (en) Systems and methods for parallel batch processing of write transactions
JP2005259136A (ja) 非ディスク永続メモリを利用したトランザクション処理システムおよび方法
JPWO2013018808A1 (ja) 分散ストレージシステムおよび方法
KR101574451B1 (ko) 트랜잭션 메모리 시스템 내구성 부여
US9984139B1 (en) Publish session framework for datastore operation records
CN105446893A (zh) 数据存储方法及设备
US20140215127A1 (en) Apparatus, system, and method for adaptive intent logging
US9817753B2 (en) Flash memory management
JP2012510094A5 (zh)
US20190384743A1 (en) Method, device and computer readable storage medium for deleting snapshot data
CN111158939A (zh) 数据处理方法、装置、存储介质及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16843576

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16843576

Country of ref document: EP

Kind code of ref document: A1