WO2020238737A1

WO2020238737A1 - Database task processing method and apparatus, electronic device, and readable medium

Info

Publication number: WO2020238737A1
Application number: PCT/CN2020/091467
Authority: WO
Inventors: 王剑英; 黄贵; 王鲁俊
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2019-05-30
Filing date: 2020-05-21
Publication date: 2020-12-03
Also published as: CN112015713A; CN112015713B

Abstract

A database task processing method and apparatus, an electronic device, and a readable medium, relating to the technical field of Internet. The method comprises: determining, according to a write request to a database, information of a database write task corresponding to the write request (S101); decomposing the database write task to obtain multiple subtasks corresponding to the database write task (S102); and concurrently writing, according to a locking detection result of a resource occupied by each of multiple task queues, the multiple subtasks into the multiple task queues (S103). The present invention can reduce consumption of computing resources of a database system, thereby effectively improving the task throughput of the database system.

Description

Database task processing method, device, electronic equipment and readable medium

This application claims the priority of a Chinese patent application filed on May 30, 2019 with the application number 201910462682.0 and the invention title "Database task processing method, device, electronic equipment and readable medium", the entire content of which is incorporated by reference In this application.

Technical field

The embodiments of this application relate to the field of Internet technology, and in particular to a method, device, electronic device, and computer-readable medium for processing database tasks.

Background technique

In the scheduling of concurrent transactions in traditional database systems, for a large number of tasks that require serial access in the database system, mutual exclusive access is generally avoided by queuing execution threads. However, it will cause frequent scheduling of the sleep state and wake state of the execution thread, thereby reducing the task throughput of the database system. Too much serial execution will bring about the performance of the database system.

Moreover, in the execution of the multi-stage execution logic, although most threads do not participate in concurrent execution, they need to wait on the semaphores of different stages to decide the next step is to wake up one or more threads to perform tasks. The switching of the working status of the threads brought about by this process will cause a large loss of computing resources of the database system when tasks are high in concurrency, thereby greatly reducing the task throughput of the database system.

Summary of the invention

The purpose of this application is to propose a database task processing method, device, electronic equipment and computer readable medium, which are used to solve the problem of low task throughput of the database system caused by the loss of computing resources of the database system in the prior art .

According to the first aspect of the embodiments of the present application, a method for processing database tasks is provided. The method includes: determining information of a database writing task corresponding to the writing request according to a writing request to the database; decomposing the database writing task to obtain multiple subtasks corresponding to the database writing task ; According to the lock detection results of the resources occupied by each task queue in the multiple task queues, the multiple subtasks are concurrently written into the multiple task queues.

According to a second aspect of the embodiments of the present application, a device for processing database tasks is provided. The device includes: a determining module, which is used to determine information of a database write task corresponding to the write request according to a write request to the database; a decomposition module, which is used to decompose the database write task to obtain all The multiple subtasks corresponding to the database write task; the concurrent write module is used to write the multiple subtasks concurrently to the multiple task queues according to the lock detection results of the resources occupied by each task queue in the multiple task queues Task queue.

According to a third aspect of the embodiments of the present application, there is provided an electronic device, including: one or more processors; a computer-readable medium configured to store one or more programs, when the one or more programs are The one or more processors execute, so that the one or more processors implement the method for processing database tasks as described in the first aspect of the foregoing embodiment.

According to a fourth aspect of the embodiments of the present application, there is provided a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the method for processing database tasks as described in the first aspect of the above-mentioned embodiment is implemented .

According to the technical solution provided by the embodiments of the present application, according to the write request to the database, the information of the database write task corresponding to the write request is determined, and the database write task is decomposed to obtain multiple subtasks corresponding to the database write task. , And then based on the lock detection results of the resources occupied by each task queue in the multiple task queues, multiple subtasks are concurrently written into multiple task queues. Compared with other existing methods, the computing resources of the database system can be reduced. Loss, thereby effectively improving the task throughput of the database system.

Description of the drawings

By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present application will become more apparent:

FIG. 1 is a flowchart of the steps of a method for processing database tasks in Embodiment 1 of the application;

2 is a flowchart of the steps of the method for processing database tasks in the second embodiment of this application;

3 is a schematic structural diagram of a processing device for database tasks in Embodiment 3 of this application;

4 is a schematic structural diagram of a processing device for database tasks in Embodiment 4 of this application;

5 is a schematic structural diagram of a processing device for database tasks in Embodiment 5 of this application;

6 is a schematic diagram of the structure of the electronic device in the sixth embodiment of the application;

FIG. 7 is the hardware structure of the electronic device in the seventh embodiment of the application.

Detailed ways

The application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described here are only configured to explain the related invention, but not to limit the invention. In addition, it should be noted that, for ease of description, only the parts related to the relevant invention are shown in the drawings.

It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other if there is no conflict. Hereinafter, the present application will be described in detail with reference to the drawings and in conjunction with embodiments.

1, it shows a flowchart of the steps of a method for processing a database task in Embodiment 1 of the present application.

Specifically, the method for processing database tasks in this embodiment includes the following steps:

In step S101, the information of the database write task corresponding to the write request is determined according to the write request to the database.

The solutions provided by the embodiments of the present invention can be applied to databases based on the LSM structure, including but not limited to X-DB databases, LevelDB databases, RocksDB databases, and the like.

In this embodiment, the write request may include a database transaction write request, or a write request for database operations. The information of the database write task may include the information of the write task corresponding to the database transaction, such as the write log information, the flushing information of the log data of the database transaction, the write memory information of the database transaction, or the commit information of the database transaction, etc. . It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In some optional embodiments, when the information of the database write task corresponding to the write request is determined, the write request may be parsed to obtain the information of the database write task corresponding to the write request . It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In step S102, the database writing task is decomposed to obtain multiple subtasks corresponding to the database writing task.

In some optional embodiments, the decomposition of the database writing task may be implemented in any suitable manner to obtain multiple subtasks corresponding to the database writing task. Generally speaking, database writing tasks can be divided into multiple different writing logics, and each writing logic can correspond to multiple sub-logics. Based on this, database writing tasks can be decomposed to obtain different writing Logic and/or subtasks corresponding to different sub-logics. For example, for WAL writing tasks, it may include writing active memory table tasks, writing frozen memory table tasks, persistent storage tasks, and so on. For another example, if a certain write task may write a batch of data at a time, the write task can be divided into multiple subtasks, and each subtask writes part of the data, and so on. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In step S103, the multiple subtasks are concurrently written into the multiple task queues according to the lock detection result on the resources occupied by each task queue in the multiple task queues.

The execution of each task requires a certain amount of resource support. In this embodiment, subtasks are grouped to form a task queue, and resources are allocated in units of task queues. Each group of task queues can accommodate multiple (two or more) ) Subtasks. When the subtask needs to be written into the task queue, the subtask can be written into the unlocked task queue according to the lock status of each task queue.

In this embodiment, multiple sets of slot slots are used, and each set of slots corresponds to a task queue. The multiple subtasks may try to organize the grouping of subtasks on the task queue corresponding to any group Slot. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In some optional embodiments, when the multiple subtasks are concurrently written to the multiple task queues according to the lock detection result on the resources occupied by each task queue in the multiple task queues, Select a task queue in the task queue to determine whether the selected task queue has been locked; if it has been locked, select another task queue from the remaining task queues, and add the subtasks to be added to the selected task Queue; if it is not locked, add the subtask to be added to the selected task queue. Specifically, through the lock flag bit of the selected task queue, it is determined whether the selected task queue has been locked. If it is determined that the selected queue has been locked, it is immediately determined whether the task queues corresponding to other group slots have been locked. . Accordingly, by locking the resources occupied by the task queue, the execution efficiency of the subtasks added to the task queue can be guaranteed. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In a specific example, the trylock mechanism can be used to lock the resources occupied by the task queue. Specifically, the execution thread of the task queue first tries to lock once in a very lightweight manner, and executes normally if the lock is successful. If the lock fails, the thread will not block, but immediately exit the kernel to do other tasks. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In some optional embodiments, the method further includes: for each task queue, allocating an execution thread for the first subtask to be added to the current task queue, wherein the execution thread is used to execute the tasks in the current task queue All subtasks. In this way, it is not necessary to allocate execution threads to all subtasks in the current task queue, which reduces the number of concurrent threads of the database system, and thereby reduces the scheduling overhead of the database system for the working status of the execution threads. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In a specific example, an execution thread is allocated to the first subtask that enters the task queue corresponding to the slot, and the execution thread is used to execute all the subtasks in the task queue. In other words, the execution thread is in the "leader" position in the task queue. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In some optional embodiments, the method further includes: if all the subtasks in the current task queue are executed, unlocking the resources of the current task queue; according to the idleness of the current task queue after unlocking The waiting time and/or the subtask information received again is processed on the execution thread corresponding to the current task queue. Wherein, the idle waiting time may include an idle waiting time period, and the information of the subtasks received again may include the number of subtasks received again. In this way, while ensuring task execution, the computing resources of the database system can be effectively saved. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In a specific example, processing the execution thread corresponding to the current task queue according to the idle waiting time of the current task queue after unlocking or the information of the subtask received again includes: If no new subtask is received again within the waiting time period, the corresponding relationship between the execution thread and the current task queue is released; if a new subtask is received again within the idle waiting time period, And the number of the new subtasks reaches the preset threshold, the execution thread is awakened, and the new subtasks are executed through the execution thread. Wherein, the time period of the idle waiting time can be set by those skilled in the art according to actual needs, which is not limited in the embodiment of the present application. The preset threshold can be set by those skilled in the art according to actual needs, and the embodiment of the present application does not make any limitation on this. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In a specific example, if a new subtask is not received again within the period of the idle waiting time, the execution thread automatically switches from the sleep state to the wake state, and then releases the execution thread and all the subtasks. Describe the corresponding relationship of the current task queue. If a new subtask is received again within the time period of the idle waiting time, and the number of the new subtask reaches the preset threshold, the new subtask is passed as the last one to enter the current task queue. The auxiliary thread assigned by the task triggers the wake-up of the execution thread. If a new subtask is received again within the time period of the idle waiting time, and the number of the new subtask does not reach the preset threshold, the new subtask is passed as the last one to enter the current task queue. The auxiliary thread allocated by the subtask triggers the awakening of the execution thread, and the new subtask is executed through the execution thread. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

For example, for each Slot, the first write request to enter the Slot will become the leader, and the execution thread will be allocated to it. When a leader already exists in a Slot, other incoming write requests will become followers, which will be assigned auxiliary Thread, follower will deliver its own write request to the leader for completion. After the leader in the current Slot releases the lock of the Slot, it will cond_wait for a period of time (this time can be configured by those skilled in the art according to actual needs). If the number of followers collected by the leader exceeds max_group_size (the maximum capacity of a task queue within the waiting time) ), the leader is awakened (triggered by the last follower). Otherwise, when the timeout period is reached, the leader wakes up by itself and switches the leader of the slot where it is empty (that is, the relationship between the execution thread corresponding to the leader and the slot is released).

In some optional embodiments, the method further includes: for each task queue, allocating an auxiliary thread for the subtask that is not the first to join the current task queue, wherein the auxiliary thread is used to transfer the non-first subtask The task is handed over to the execution thread for processing; it is determined whether the auxiliary thread has successfully handed over the non-first subtask to the execution thread, and if so, the auxiliary thread is allocated to obtain a new subtask. This method can be implemented through an asynchronous API. After the follower gives its subtask to the leader, it can immediately return to handle the new subtask. After the leader completes the task of the follower, it can call a callback function to complete the cleaning of the current task queue. In this way, it is not necessary to allocate execution threads to all subtasks in the current task queue, which reduces the number of concurrent threads of the database system, and thereby reduces the scheduling overhead of the database system for the working status of the execution threads. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto. It should be noted that if in the synchronous mode, after the follower handed over its subtask to the leader, it needs to wait for the leader to complete the work before returning to process the new subtask.

In the above manner, after determining that the auxiliary thread has successfully handed over the non-first subtask to the execution thread, an asynchronous API can be used to allocate the auxiliary thread to obtain a new subtask. By unbinding task execution threads and task execution processes, fewer threads are used to serve more tasks, and the loss of effective computing resources caused by thread scheduling of the database system is reduced. After the execution thread completes the processing of all the subtasks in the current task queue, the callback function is called to complete the cleaning work. In the synchronous mode, the auxiliary thread will wait for the execution thread to complete the processing of all the subtasks in the current task queue before returning to obtain a new subtask. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

Through the database task processing method provided by the embodiments of the present application, according to the write request to the database, the information of the database write task corresponding to the write request is determined, and the database write task is decomposed to obtain the database write task corresponding Multiple subtasks, and then based on the lock detection results of the resources occupied by each task queue in the multiple task queues, write multiple subtasks concurrently into multiple task queues. Compared with other existing methods, it can reduce the database system The loss of computing resources effectively improves the task throughput of the database system.

The method for processing database tasks in this embodiment can be executed by any appropriate device with data processing capabilities, including but not limited to: cameras, terminals, mobile terminals, PCs, servers, in-vehicle devices, entertainment devices, advertising devices, and personal digital devices. Assistant (PDA), tablet computer, notebook computer, handheld game console, smart glasses, smart watch, wearable device, virtual display device or display enhancement device (such as Google Glass, Oculus Rift, Hololens, Gear VR), etc.

Referring to FIG. 2, there is shown a flowchart of the steps of a method for processing a database task in the second embodiment of the present application.

In step S201, according to the write request to the database, the information of the database write task corresponding to the write request is determined.

Since step S201 is similar to the above step S101, it will not be repeated here.

In step S202, the database writing task is decomposed to obtain multiple subtasks corresponding to the database writing task.

Since step S202 is similar to the above step S102, it will not be repeated here.

In step S203, the multiple subtasks are concurrently written into the multiple task queues according to the lock detection result on the resources occupied by each task queue in the multiple task queues.

Since step S203 is similar to the above step S103, it will not be repeated here.

In step S204, if all the subtasks in the current task queue have been executed, the database transaction log data corresponding to all the subtasks are collected, and the corresponding log write task is generated.

In this embodiment, the execution thread of the current task queue collects the database transaction log data of all subtasks in the current task queue into the local cache of the task, and calculates all the subtasks in the current task queue The crc32 checksum data of the database transaction log data. Then, generate the corresponding log writing task. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In step S205, the database transaction log data is written into the log copy queue through the log writing task.

In some optional embodiments, when the database transaction log data is written into the log copy queue through the log writing task, the method further includes: allocating a log transaction identifier for the log writing task, and The log transaction identifier determines whether all log writing tasks are completed. In this way, it can be judged whether all log writing tasks are completed through the log transaction identifier. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In a specific example, the log transaction identifier is recorded and stored in the corresponding variable as a basis for whether the log writing task is completed. Specifically, the log transaction identifier may be a local serial number. When the current thread is the last thread to enter, the prerequisite for exiting the kernel is that the global sequence number is greater than or equal to the local sequence number. This ensures that all log writing tasks have been completed before the last thread exits the kernel. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In step S206, the log data in the log copy queue is written into the cache for storing log data through the first working thread.

In this embodiment, the first working thread extracts database transaction log data from the log copy queue, and writes the database transaction log data into a cache for storing log data. The first working thread may be a foreground thread or a background thread. When the first working thread can be a background thread, finer scheduling and task allocation can be performed. For example, the fixed allocation of the number of log writing threads is reasonably set according to the number of physical CPUs of the machine. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In some optional embodiments, the method further includes: sending the database transaction log data in the cache to the log flushing queue through a flushing task, so as to persistently store the database transaction log data; Two working threads, which persistently store the database transaction log data in the log flushing queue. By this, it can ensure the persistent storage of the database transaction log data in the log flushing queue. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In a specific example, the second working thread may be a foreground thread or a background thread. When the second working thread can be a background thread, finer scheduling and task allocation can be performed. After the database transaction log data is written into the cache for storing log data, the corresponding disk flushing task is generated. Then, the database transaction log data in the cache is persistently stored through the flashing task. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In some optional embodiments, the sending the log data in the cache to the log flushing queue through the flushing task includes: determining whether the current flushing task has preempted the flushing lock, and if so, passing the current The flushing task sends the corresponding log data to the log flushing queue. In this way, preemptive execution of the task of brushing can be realized. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In a specific example, the flashing task can only be executed in series by one thread, so an atomic flag is set to implement preemptive execution of the task. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In some optional embodiments, while the log data in the cache is sent to the log flushing queue through the flushing task, the method further includes: recording the number corresponding to the log data, and according to the Specify the number and send the corresponding log data to the write memory table queue. With this, it can be ensured that the log data that has been flushed is sent to the write memory table queue. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In a specific example, the second worker thread will try to flush the database transaction log data in the cache. When the log sequence number of the database transaction log data advances, it will extract the current transactions that meet the conditions from the log flush queue. Database transaction log data. Specifically, when the log sequence number of the database transaction log data currently being flushed is greater than the log sequence number of the database transaction log data in the log flushing queue, the corresponding database transaction log data is extracted from the log flushing queue. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In some optional embodiments, the method further includes: concurrently writing the log data in the memory table writing queue to the memory table through multiple memory table writing tasks. With this, the database transaction log data can be written into the memory table concurrently. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In a specific example, after persistently storing the database transaction log data in the log flushing queue through the second working thread, multiple corresponding memory table writing tasks are generated. Then, the database transaction log data in the memory table write queue is concurrently written into the memory table through multiple tasks of writing the memory table. The multiple tasks of writing memory tables can be executed concurrently, and most of the work threads are concentrated in this place, and concurrently obtain database transaction log data from the writing memory tables, and perform the operation of writing the memory tables. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In some optional embodiments, the concurrently writing the log data in the memory table writing queue to the memory table through multiple memory table writing tasks includes: obtaining the task number of the submitted memory table writing task; The largest task number among the consecutive task numbers is determined as the current task number; according to the current task number, it is determined whether the multiple tasks of writing memory tables are all completed. In this way, the task number can ensure the completion of multiple tasks of writing memory tables. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In a specific example, the memory table writing tasks corresponding to task numbers 1, 2, 3, and 6 have all been executed, but the memory table writing tasks corresponding to task numbers 4 and 5 have not been executed yet, and the current task number is updated at this time Is 3. After the task of writing memory table corresponding to task numbers 4 and 5 is executed, the current task number is updated to 6. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In some optional embodiments, the method further includes: performing a cleanup operation on multiple submitted tasks for writing memory tables through multiple submission tasks. With this, it is possible to clean up multiple submitted tasks for writing memory tables. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

In a specific example, when the task number of the memory table writing task advances, the memory table task that meets the submission condition will be pushed to the execution queue. Specifically, when the current task number is the task number of the memory table task to be submitted, the memory table task to be submitted will be pushed to the execution queue for cleaning. The cleaning operations of the memory table tasks in the execution queue can be executed concurrently. Specifically, locks are released on the computing resources occupied by the memory table task, and the memory occupied by the memory table task is reclaimed. More specifically, call the callback function to complete the cleanup work. In the synchronous mode, after the task is submitted, it is necessary to wake up the thread waiting for the task to be submitted and return the package. In asynchronous mode, use the callback function to complete the task lock and memory release. When the stand-alone storage engine X-Engine is used as an engine of the MySQL database, the callback function here also needs to complete the task of returning the package to the client. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation thereto.

It can be seen that, by splitting the task execution phase of a smaller granularity, the work of the IO device and the work of the CPU computing device are balanced without waiting for each other, so that the database system achieves the highest task throughput.

Below, a specific example is used to illustrate the above process. For example, after a group (task queue) is dequeued from the slot, the leader of the slot will collect the logs of all the followers of the slot into a large write batch, and calculate the crc32 check of the WAL log; then, the log The write task is pushed to the log copy queue. When pushed to the log copy queue, a transaction ID will be assigned to the log write task. This transaction ID will be recorded and stored in the local_expected_sequence_number as the basis for whether the transaction can exit the pipeline. When a thread is the last entry thread in the pipeline, the precondition for its exit is that the global global_version_sequence_number>=local_exptec_sequence_number, so as to ensure that all task queues are emptied before the last thread exits. The first worker thread extracts the log content written by the log writing task from the copy log queue, writes the log content into the buffer of the WAL log module, records the lsn (the number corresponding to the log data), and then writes the contents of the memory table Push the task to the log flushing queue. This part of the task can only be executed serially by one thread, so an atomic flag is set to achieve preemptive execution of the task. As for the log flushing queue, the log content of the tasks in the queue has been written to the log buffer, but there is no flushing. The worker thread will try to flush the buffer in the log module. When the lsn of the log file is advanced, it will extract the (flush_lsn>append_lsn) task that meets the conditions from the log flush queue and push it to the write memtable queue. The tasks in the writememtable queue have completed the log flushing and are waiting for the task of writing the memory table. This part of the task can be executed concurrently. Most of the work threads are concentrated in this place, and concurrently obtain tasks from write_memtable_queue and execute the operation of writing the memory table. . Because of the data written to the memory table, the read operation can be read. In order to ensure consistency, a sliding window needs to be maintained to update the global read version number. For example, transactions 1, 2, 3, and 6 have been executed. But transactions 4 and 5 have not been executed yet, and the global version number is updated to 3. When 4 and 5 are submitted, the global version number is updated to 6. When the global version number advances, tasks that meet the submission conditions will be pushed to commit_queue. The tasks in commit_queue need to complete submission, unlocking and memory recovery, etc. This part of the task can be executed concurrently. In the synchronous mode, after the transaction is committed, the thread waiting for the transaction to be submitted needs to be awakened and returned to the package. In asynchronous mode, a callback function is used to complete the transaction lock and memory release. When used as an engine of MySQL, the callback function here also needs to complete the task of returning the package to the client.

On the basis of the first embodiment of this application, if all subtasks in the current task queue have been executed, the database transaction log data corresponding to all subtasks are collected, and the corresponding log writing task is generated, and then the log writing The task writes database transaction log data into the log copy queue, and then through the first working thread, writes the database transaction log data in the log copy queue into the cache for storing log data. Compared with other existing methods, it can The database transaction log data corresponding to the subtasks that have been executed are written into the cache for storing the log data, thereby ensuring the consistency of the execution of the database tasks.

Referring to FIG. 3, there is shown a schematic structural diagram of an apparatus for processing database tasks in Embodiment 3 of the present application.

The database task processing device of this embodiment includes: a determining module 301, configured to determine the information of the database write task corresponding to the write request according to a write request to the database; and a decomposition module 302, configured to check the database The writing task is decomposed to obtain multiple subtasks corresponding to the database writing task; the concurrent writing module 303 is configured to detect the multiple task queues according to the lock detection result of the resources occupied by each task queue in the multiple task queues. The subtasks are concurrently written into the multiple task queues.

The database task processing apparatus of this embodiment is used to implement the corresponding database task processing methods in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

Referring to FIG. 4, there is shown a schematic structural diagram of a database task processing device in the fourth embodiment of the present application.

The database task processing device of this embodiment includes: a determining module 401, configured to determine the database write task information corresponding to the write request according to a write request to the database; The writing task is decomposed to obtain multiple subtasks corresponding to the database writing task; the concurrent writing module 403 is configured to detect the multiple task queues according to the lock detection result of the resources occupied by each task queue in the multiple task queues. The subtasks are concurrently written into the multiple task queues.

Optionally, the concurrent writing module 403 is specifically configured to: select a task queue from the multiple task queues, and determine whether the selected task queue has been locked; if it has been locked, select a task queue from the remaining Another task queue is selected from the task queue, and the subtask to be added is added to the selected task queue; if it is not locked, the subtask to be added is added to the selected task queue.

Optionally, the device further includes: a first allocation module 404, configured to allocate an execution thread for the first subtask added to the current task queue for each task queue, wherein the execution thread is used to execute the current task queue. All subtasks in the task queue.

Optionally, the device further includes: a resource unlocking module 405, configured to unlock the resources of the current task queue if all subtasks in the current task queue are executed; a processing module 406, configured to unlock resources according to the unlock The subsequent idle waiting time of the current task queue and/or the information of the subtask received again is processed on the execution thread corresponding to the current task queue.

Optionally, the processing module 406 is specifically configured to: if a new subtask is not received again within the idle waiting time period, release the corresponding relationship between the execution thread and the current task queue; If a new subtask is received again within the period of the idle waiting time, and the number of the new subtask reaches the preset threshold, the execution thread is awakened, and the new execution thread is executed Of subtasks.

Optionally, the device further includes: a second allocation module 407, configured to allocate auxiliary threads to the subtasks that are not the first to join the current task queue for each task queue, wherein the auxiliary thread is used for The non-first subtask is handed over to the execution thread for processing; the third allocation module 408 is used to determine whether the auxiliary thread has successfully handed over the non-first subtask to the execution thread, and if so, allocate the The auxiliary thread acquires a new subtask.

Referring to FIG. 5, there is shown a schematic structural diagram of a database task processing device in Embodiment 5 of the present application.

The database task processing device of this embodiment includes: a determining module 501, configured to determine the information of the database write task corresponding to the write request according to a write request to the database; and a decomposition module 502, configured to check the database The writing task is decomposed to obtain multiple subtasks corresponding to the database writing task; the concurrent writing module 503 is configured to detect the multiple task queues according to the lock detection result of the resources occupied by each task queue in the multiple task queues. The subtasks are concurrently written into the multiple task queues.

Optionally, the device further includes: a collection module 504, configured to collect database transaction log data corresponding to all subtasks if all subtasks in the current task queue have been executed, and generate corresponding log writes Into the task; the first writing module 505, used to write the log data into the log copy queue through the log writing task; the second writing module 506, used to copy the log through the first working thread The log data in the queue is written into the buffer for storing the log data.

Optionally, the device further includes: a fourth allocation module 507, configured to allocate a log transaction identifier for the log writing task; a judging module 508, configured to determine whether all log writing tasks are all based on the log transaction identifier carry out.

Optionally, the device further includes: a sending module 509, configured to send the log data in the cache to the log flushing queue through a flushing task, so as to perform persistent storage on the log data; a persistent storage module 510, configured to perform persistent storage on the log data in the log flushing queue through a second working thread.

Optionally, the sending module 509 is specifically configured to: determine whether the current flashing task has seized the flashing lock, and if so, sending the corresponding log data to the log flashing through the current flashing task queue.

Optionally, the device further includes: a recording module 511, configured to record a serial number corresponding to the log data, and according to the serial number, send the corresponding log data to the write memory table queue.

Optionally, the device further includes: a third writing module 512, configured to concurrently write the log data in the memory table writing queue into the memory table through multiple memory table writing tasks.

Optionally, the third writing module 512 is specifically configured to: obtain the task number of the task of writing the memory table that has been submitted; determine the largest task number among multiple consecutive task numbers as the current task number; according to the current task Numbering determines whether the multiple tasks of writing memory tables are all completed.

Optionally, the device further includes: a cleaning module 513, configured to perform cleaning operations on multiple submitted tasks for writing memory tables through multiple submission tasks.

FIG. 6 is a schematic structural diagram of an electronic device in Embodiment 6 of this application; the electronic device may include:

One or more processors 601;

The computer-readable medium 602 can be configured to store one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the method for processing database tasks as described in the first embodiment or the second embodiment.

FIG. 7 is the hardware structure of the electronic device in the seventh embodiment of the application; as shown in FIG. 7, the hardware structure of the electronic device may include: a processor 701, a communication interface 702, a computer-readable medium 703 and a communication bus 704;

The processor 701, the communication interface 702, and the computer-readable medium 703 communicate with each other through the communication bus 704;

Optionally, the communication interface 702 may be an interface of a communication module, such as an interface of a GSM module;

The processor 701 may be specifically configured to: according to a write request to the database, determine the information of the database write task corresponding to the write request; decompose the database write task to obtain the database write task Corresponding multiple subtasks; according to the lock detection result of the resources occupied by each task queue in the multiple task queues, the multiple subtasks are concurrently written into the multiple task queues.

The processor 701 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), etc. ), ready-made programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.

The computer-readable medium 703 may be, but is not limited to, a random access storage medium (Random Access Memory, RAM), a read-only storage medium (Read Only Memory, ROM), and a programmable read-only storage medium (Programmable Read-Only Memory, PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code configured to execute the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication part, and/or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access storage media (RAM), read-only storage media (ROM), erasable Type programmable read-only storage medium (EPROM or flash memory), optical fiber, portable compact disk read-only storage medium (CD-ROM), optical storage medium, magnetic storage medium, or any suitable combination of the above. In this application, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and a computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium. The computer-readable medium may send, propagate, or transmit a program configured to be used by or in combination with the instruction execution system, apparatus, or device . The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

The computer program code configured to perform the operations of the present application can be written in one or more programming languages or a combination thereof. The programming languages include object-oriented programming languages—such as Java, Smalltalk, C++, and also conventional Procedural programming language-such as "C" language or similar programming language. The program code can be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network: including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect to the Internet connection).

The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation of the system architecture, functions, and operations of the system, method, and computer program product according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, program segment, or part of code, and the module, program segment, or part of code contains one or more configurations to achieve the specified logical function Executable instructions. In the foregoing specific embodiments, there are specific sequence relationships, but these sequence relationships are only exemplary. In specific implementation, these steps may be fewer, more, or the execution order may be adjusted. That is, in some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two blocks shown in succession can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations Or it can be realized by a combination of dedicated hardware and computer instructions.

The modules involved in the embodiments described in the present application can be implemented in software or hardware. The described module may also be provided in the processor, for example, it may be described as: a processor includes a determination module, a decomposition module, and a concurrent writing module. Among them, the names of these modules do not constitute a limitation on the module itself under certain circumstances. For example, the determining module can also be described as "based on a write request to the database, determine the database write corresponding to the write request Module of task information".

As another aspect, the present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method for processing database tasks as described in the first embodiment or the second embodiment is realized.

As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above-mentioned embodiments; or it may exist alone without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the device, the device is caused to determine the database write task corresponding to the write request according to the write request to the database Decompose the database write task to obtain multiple subtasks corresponding to the database write task; according to the lock detection results of the resources occupied by each task queue in the multiple task queues, the multiple subtasks Tasks are written into the multiple task queues concurrently.

The expressions "first", "second", "the first" or "the second" used in various embodiments of the present disclosure may modify various components regardless of order and/or importance , But these expressions do not limit the corresponding components. The above expressions are only configured for the purpose of distinguishing elements from other elements. For example, the first user equipment and the second user equipment represent different user equipment, although both are user equipment. For example, without departing from the scope of the present disclosure, the first element may be referred to as the second element, and similarly, the second element may be referred to as the first element.

When an element (for example, a first element) is referred to as being "(operably or communicatively) coupled" or "(operably or communicatively) coupled to" another element (for example, a second element) When an element (e.g., a second element) or "connected to" another element (e.g., a second element), it should be understood that the one element is directly connected to the other element or the one element passes through another element (e.g., The third element) is indirectly connected to the other element. On the contrary, it can be understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (a second element), no element (e.g., a third element) is inserted between the two elements. Between those.

The above description is only a preferred embodiment of the application and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solution formed by the specific combination of the above technical features, and should also cover the above technical features or technical solutions without departing from the above inventive concept. Other technical solutions formed by any combination of its equivalent features. For example, the above-mentioned features and the technical features disclosed in this application (but not limited to) with similar functions are mutually replaced to form a technical solution.

Claims

A method for processing database tasks, characterized in that the method includes:

Determine the information of the database write task corresponding to the write request according to the write request to the database;

Decompose the database writing task to obtain multiple subtasks corresponding to the database writing task;

The multiple subtasks are concurrently written into the multiple task queues according to the lock detection result on the resources occupied by each task queue in the multiple task queues.
The method according to claim 1, wherein the multiple subtasks are concurrently written into the multiple task queues according to the lock detection result of the resources occupied by each task queue in the multiple task queues, include:

Select a task queue from the multiple task queues, and determine whether the selected task queue has been locked;

If it has been locked, select another task queue from the remaining task queues, and add the subtasks to be added to the selected task queue;

If it is not locked, the subtask to be added is added to the selected task queue.
The method according to claim 1 or 2, wherein the method further comprises:

For each task queue, an execution thread is allocated to the first subtask that joins the current task queue, where the execution thread is used to execute all the subtasks in the current task queue.
The method according to claim 3, wherein the method further comprises:

If all subtasks in the current task queue are executed, unlock the resources of the current task queue;

The execution thread corresponding to the current task queue is processed according to the unlocked idle waiting time of the current task queue and/or the information of the subtasks received again.
The method according to claim 4, wherein the processing is performed on the execution thread corresponding to the current task queue according to the unlocked idle waiting time of the current task queue or the information of the subtask received again, include:

If no new subtask is received again within the time period of the idle waiting time, releasing the corresponding relationship between the execution thread and the current task queue;

If a new subtask is received again within the period of the idle waiting time, and the number of the new subtask reaches the preset threshold, the execution thread is awakened, and the new execution thread is executed Subtasks.
The method according to claim 3, wherein the method further comprises:

For each task queue, an auxiliary thread is allocated to the subtask that is not the first to join the current task queue, where the auxiliary thread is used to hand the non-first subtask to the execution thread for processing;

It is determined whether the auxiliary thread has successfully handed over the non-first subtask to the execution thread, and if so, the auxiliary thread is allocated to obtain a new subtask.
The method of claim 1, wherein the method further comprises:

If all subtasks in the current task queue have been executed, collect the database transaction log data corresponding to all subtasks, and generate corresponding log write tasks;

Writing the log data into the log copy queue through the log writing task;

Through the first working thread, the log data in the log copy queue is written into the cache for storing the log data.
8. The method according to claim 7, wherein when the log data is written to the log copy queue through the log writing task, the method further comprises:

Allocating a log transaction identifier for the log writing task;

According to the log transaction identifier, it is determined whether all log writing tasks are completed.
The method according to claim 7, wherein the method further comprises:

Sending the log data in the cache to the log flushing queue through a flushing task, so as to perform persistent storage on the log data;

Through the second working thread, the log data in the log flushing queue is persistently stored.
The method according to claim 9, wherein the sending the log data in the cache to the log flushing queue through a flushing task comprises:

It is determined whether the current flashing task has seized the flashing lock, and if so, the corresponding log data is sent to the log flashing queue through the current flashing task.
The method according to claim 9, wherein while the log data in the cache is sent to the log flushing queue through the flushing task, the method further comprises:

Record the number corresponding to the log data, and send the corresponding log data to the write memory table queue according to the number.
The method of claim 11, wherein the method further comprises:

The log data in the memory table writing queue is concurrently written into the memory table through multiple tasks of writing the memory table.
The method according to claim 12, wherein the concurrently writing the log data in the memory table writing queue to the memory table through multiple tasks of writing the memory table includes:

Get the task number of the submitted task of writing memory table;

Determine the largest task number among consecutive task numbers as the current task number;

According to the current task number, it is determined whether the multiple tasks of writing memory tables are all completed.
The method of claim 13, wherein the method further comprises:

Multiple submitted tasks are used to clean up multiple submitted tasks for writing memory tables.
A processing device for database tasks, characterized in that the device comprises:

The determining module is used to determine the information of the database write task corresponding to the write request according to the write request to the database;

The decomposition module is used to decompose the database writing task to obtain multiple subtasks corresponding to the database writing task;

The concurrent writing module is configured to concurrently write the multiple subtasks into the multiple task queues according to the lock detection result of the resources occupied by each task queue in the multiple task queues.
The apparatus according to claim 15, wherein the concurrent writing module is specifically configured to:

Select a task queue from the multiple task queues, and determine whether the selected task queue has been locked;

If it has been locked, select another task queue from the remaining task queues, and add the subtasks to be added to the selected task queue;

If it is not locked, the subtask to be added is added to the selected task queue.
The device according to claim 15 or 16, wherein the device further comprises:

The first allocation module is configured to allocate execution threads to the first subtasks added to the current task queue for each task queue, wherein the execution threads are used to execute all subtasks in the current task queue.
The device according to claim 17, wherein the device further comprises:

The resource unlocking module is configured to unlock the resources of the current task queue if all the subtasks in the current task queue are executed;

The processing module is configured to process the execution thread corresponding to the current task queue according to the unlocked idle waiting time of the current task queue and/or the information of the subtasks received again.
The device according to claim 18, wherein the processing module is specifically configured to:

If no new subtask is received again within the time period of the idle waiting time, releasing the corresponding relationship between the execution thread and the current task queue;

If a new subtask is received again within the period of the idle waiting time, and the number of the new subtask reaches the preset threshold, the execution thread is awakened, and the new execution thread is executed Subtasks.
The device according to claim 17, wherein the device further comprises:

The second allocation module is configured to allocate auxiliary threads for each task queue that is not the first subtask to join the current task queue, wherein the auxiliary thread is used to hand over the non-first subtask to the execution thread To process

The third allocation module is used to determine whether the auxiliary thread has successfully handed over the non-first subtask to the execution thread, and if so, allocate the auxiliary thread to obtain a new subtask.
An electronic device, comprising: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus;

The memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute the database task processing method according to any one of claims 1-14.
A computer storage medium with a computer program stored thereon, and when the program is executed by a processor, the method for processing database tasks according to any one of claims 1-14 is realized.