WO2011137672A1

WO2011137672A1 - Method and device for task execution based on database

Info

Publication number: WO2011137672A1
Application number: PCT/CN2011/070456
Authority: WO
Inventors: 马天俊
Original assignee: 中兴通讯股份有限公司
Priority date: 2010-05-05
Filing date: 2011-01-21
Publication date: 2011-11-10
Also published as: CN101853182A

Abstract

The present invention discloses a method and device for task execution based on database. The method includes: adding a database connection; obtaining a task to be executed from a task group by the database connection; and judging whether a foreground task of the obtained task is executed successfully. The obtained task is executed by the database connection, if the foreground task of the obtained task is executed successfully. The task to be executed is continuously obtained from the task group by the database connection, if the foreground task of the obtained task is not executed successfully. The present invention allows dynamic control over concurrency of data processing.

Description

The present invention relates to the field of communications, and in particular to a database-based task execution method and apparatus. BACKGROUND OF THE INVENTION In application software development processes, relational databases are often used to store and process data. In the related art, each data processing task is usually performed serially in a database connection of a database. This processing method performs task inefficiently, and when the task processing error occurs, it needs to be completely re-executed, and cannot be from the occurrence of an error. Resume execution. To this end, a related method for parallel execution of multiple database tasks is proposed in the related art, which divides the tasks of data processing into different groups, and tasks belonging to different groups are executed in parallel, and tasks in each group are independent of each other. This reduces the dependency between tasks and shortens the time required to perform tasks. However, this method has the following disadvantages: In this method, parallel execution is implemented between groups and groups, but sub-tasks in the group are still serially executed, and execution of tasks with complex execution dependencies cannot be implemented, for example, if The predecessor task of task C is two tasks A and B in parallel, and the method cannot implement the data processing of the C task after the parallel execution of the A task and the B task; in addition, the method is grouped in advance. The execution order of each group of execution tasks is fixed, and the number of groups is fixed. That is to say, the degree of concurrency of data processing is fixed and cannot be dynamically adjusted with hardware resources. SUMMARY OF THE INVENTION A primary object of the present invention is to provide a database-based task execution method and apparatus to solve at least the above problems. According to an aspect of the present invention, a database-based task execution method is provided, including: joining a database connection; obtaining a task to be executed from a task group; and determining whether the pre-task of the acquired task is all successfully executed, If so, the database connection performs the acquired task. If not, the database connection continues to obtain the task to be executed from the task group. Preferably, the database connection obtains the task to be executed from the task group, including: the database connection locks the record row of the query operation type of the task group; the database connection acquires the task that meets the predetermined condition, wherein the predetermined condition includes: the acquired task The execution result status is initialized or reopened; the database connection sets the execution result status of the acquired task to selected; the database connection unlocks the record line of the query operation type. Preferably, after determining that the pre-task of the acquired task is not all successfully executed, the method further includes: the database connection sets the execution result status of the acquired task to be re-opened. Preferably, when the execution result status of the acquired task is set to be re-opened, the acquired task is added to the pre-task temporary table, and the predetermined condition further includes that the acquired task is not in the pre-task temporary table. Preferably, the performing the obtaining task comprises: the database connection setting the execution result status of the acquired task to run, and executing the running instruction of the acquired task. Preferably, in the case that the running instruction of the executed task is successful, the method further includes: the database connection locks the record line of the deletion operation type of the task group; the database connection sets the execution result status of the acquired task to be successful, and deletes The task in the pre-task temporary table; the database connection unlocks the record line of the delete operation type of the task group. Preferably, in the case that the running instruction of the executed task fails, the method further includes: the database connection sets the execution result status of the acquired task to a failure, and sets the execution result status of the acquired task's post task to a dependency. failure. Preferably, after the database connection obtains the task to be executed from the current task group, or after performing the acquired task, the method further includes: determining whether the sum of the number of successfully executed tasks and the number of failed tasks is equal to the task in the task group. The total number, or, determines whether the user issues a stop command, and if so, stops execution, and if not, returns the step of obtaining the task to be executed from the task group. Preferably, the database connection obtaining the task to be executed from the current task group fails includes: the database connection receiving the notification for indicating the lock waiting timeout; or the database connection receiving the task for indicating that there is no pending task in the task group announcement of. Preferably, determining whether the pre-task of the acquired task is all successfully executed comprises: determining a pre-task of the acquired task according to a dependency relationship between the tasks in the pre-set task group; determining whether the pre-task task is all successfully executed. According to another aspect of the present invention, a database-based task execution apparatus is provided, including: an adding module, configured to join a database connection; an obtaining module, configured to obtain a task to be executed from the task group; and a determining module, being set to Determining whether the pre-task of the acquired task is all successfully executed; the scheduling module is configured to: when the judgment result of the judging module is yes, scheduling the execution module, and if the judgment result of the judging module is no, scheduling the acquisition module; Execution module, set to perform the acquired task. According to the present invention, the task to be executed is obtained from the task group by using the added database connection, and it is determined whether the pre-task of the task is all executed successfully. If yes, the acquired task is executed, and if not, the task group is continued. Obtaining the task to be executed solves the problem that the data processing concurrency is fixed in the related art, resulting in low processing flexibility, and can dynamically control the concurrency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are set to illustrate,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, In the drawings: FIG. 1 is a flowchart of a database-based task execution method according to an embodiment of the present invention; FIG. 2 is a structural block diagram of a database-based task execution apparatus according to an embodiment of the present invention; A detailed structural block diagram of an apparatus for implementing a database-based task execution method of Example 1; FIG. 4 is a flowchart of an execution process of each sub-module in an execution module according to Embodiment 2 of the present invention; A schematic diagram of the composition of a task having a network dependency relationship of Embodiment 4. BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the embodiments in the present application and the features in the embodiments may be combined with each other without conflict. FIG. 1 is a flowchart of a database-based task execution method according to an embodiment of the present invention. As shown in FIG. 1, the method includes the following steps: Step S102, adding a database connection; step S104, the database connection acquires a task to be executed from the task group; and step S106, determining whether the pre-task of the acquired task is all executed successfully, and if so, the database connection is executed. The acquired task, if not, the database connection continues to get the task to be executed from the task group. In the related art, the tasks to be executed are grouped in advance, the tasks in the group are performed in a serial manner in the group, and the tasks in different groups are executed in parallel, thereby realizing concurrent processing of data, but the method The number of concurrent connections needs to be pre-defined, flexible configuration is not possible, and the execution of tasks for mixed relationships cannot be supported. In the method provided by the embodiment of the present invention, the newly added database connection first obtains the task to be executed, and then determines the subsequent processing according to whether the pre-task is successfully executed. In this method, the database connection can be added at any time, that is, in the process of data processing, the thread for processing the task can be added at any time, and the dynamic increase or decrease of the database connection is supported, thereby dynamically adjusting the concurrency of the data processing. In addition, by judging whether the pre-tasks are all executed successfully, the execution logic of the hybrid relationship task is ensured, and the data calculation of the mesh dependency relationship can be realized. Preferably, the step S104 may include: Step S1042, the database connection locks the record row of the query operation type of the task group; Step S1044, the database connection acquires a task that satisfies the predetermined condition, wherein the predetermined condition includes: The execution result status of the task is initialized or reopened; in step S1046, the database connection sets the execution result status of the acquired task to selected; in step S1048, the database connection unlocks the record line of the query operation type. With this method, it is ensured that a task to be executed is acquired by only one database connection at the same time, preventing execution confusion caused by different database connections to obtain the same task, and ensuring the robustness of the system. Preferably, after determining that the pre-task of the acquired task is not all successfully executed, the method further includes: the database connection sets the execution result status of the acquired task to be re-opened. With this method, it is possible to ensure that subsequent database connections (possibly for the database connection itself) get the task. Preferably, when the execution result status of the acquired task is set to be re-opened, the acquired task is added to the pre-task temporary table, and the predetermined condition further includes that the acquired task is not in the pre-task temporary table. By setting the method, the database connection can be prevented from repeatedly capturing tasks that are not successfully executed by the same predecessor task, and finally the database connection enters a meaningless loop state for a period of time, thereby improving the database connection. Processing efficiency. Preferably, the process of executing the acquired task includes: the database connection sets the execution result state of the acquired task to run, and executes the running instruction of the acquired task. By setting the execution result status of the task to run, it can be guaranteed that the task will not be captured by other database connections during the running process, which ensures the robustness of the system. Preferably, in the case that the running instruction of the executed task is successful, the method further includes: the database connection locks the record line of the deletion operation type of the task group; the database connection sets the execution result status of the acquired task to be successful, and deletes The task in the pre-task temporary table; the database connection unlocks the record line of the delete operation type of the task group. By deleting the tasks in the pre-task temporary table in the case of successful execution, it is guaranteed that the database connection can continue to try to grab the previously grabbed and released tasks (the pre-task of the task was not fully executed at the time), improve The operating efficiency of the system, and the mechanism of deleting the lock, prevents multiple database connections from simultaneously performing the delete operation, and ensures the robustness of the system. Preferably, in the case that the running instruction of the executed task fails, the method further includes: the database connection sets the execution result status of the acquired task to a failure, and sets the execution result status of the acquired task's post task to a dependency. failure. By setting the currently acquired task and the execution status of its post task, it is possible to prevent the above task from being crawled again, to prevent wasting system resources if the task cannot be successfully executed, and at the same time, to identify the failed task, so that Subsequent execution from the failed task improves the efficiency of the system. Preferably, after the database connection obtains the task to be executed from the current task group, or after performing the acquired task, the method further includes: determining whether the sum of the number of successfully executed tasks and the number of failed tasks is equal to the task in the task group. The total number, or, determines whether the user issues a stop command, and if so, stops execution, and if not, returns the step of obtaining the task to be executed from the task group. The method can automatically determine whether to stop execution, or determine whether to stop execution according to the user's instruction, thereby improving the controllability of the system. Preferably, the failure of the database connection to obtain the task to be executed from the current task group comprises: the database connection receiving the notification for indicating the lock waiting timeout; or the database connection is received for Indicates that there are no notifications for the task to be executed in the task group. When the database connection lock wait timeout or there is no task to be executed in the task group, it is determined whether it is necessary to stop execution, which can prevent waste of system resources. Preferably, determining whether the pre-task of the acquired task is all successfully executed comprises: determining a pre-task of the acquired task according to a dependency relationship between the tasks in the pre-set task group; determining whether the pre-task task is all successfully executed. By this method, tasks are executed in accordance with the relationship between tasks, and the execution of tasks of serial relationships, parallel relationships, and serial and parallel mixed relationships can be realized. 2 is a structural block diagram of a database-based task execution apparatus according to an embodiment of the present invention. The apparatus includes: an adding module 21 configured to join a database connection; an obtaining module 22 configured to obtain a task to be executed from the task group; The module 23 is configured to determine whether the pre-task of the acquired task is all successfully executed. The scheduling module 24 is configured to, when the judgment result of the judging module 23 is YES, the scheduling execution module 25, and the judgment result of the judging module 23 is In the case of no, the scheduling acquisition module 22; the execution module 25 is set to perform the acquired task. In the related art, it is necessary to pre-specify the number of concurrent connections, which cannot implement flexible configuration, and cannot support the execution of tasks of mixed relationships. In the device provided by the embodiment of the present invention, the newly added database connection first executes the obtaining module 22, obtains the task to be executed, and determines the subsequent processing according to whether the pre-task task performed by the determining module 23 is all successfully executed. The database connection can be added at any time, that is, in the process of data processing, the thread for processing the task can be added at any time, and the dynamic increase or decrease of the database connection is supported, thereby dynamically adjusting the concurrency of the data processing. In addition, the judging module 23 judges whether all the pre-tasks are successfully executed, ensures the execution logic of the hybrid relationship task, and can implement data calculation of the mesh dependency relationship. The embodiments 1-4 described below combine the technical solutions of the above-described plurality of preferred embodiments. Embodiment 1 FIG. 3 is a detailed structural block diagram of an apparatus for implementing a database-based task execution method according to Embodiment 1 of the present invention. As shown in FIG. 3, the apparatus includes: an input module, an execution module, and an output module. among them:

(1) The input module is set to accept the input of the execution task data into the task table, the input of the task dependency data into the task relationship table, and the input of the lock usage data into the task lock table. Preferably, the input module is further configured to generate a task group, specify a task group label, and use the task group label as the group identifier to implement data isolation of the hybrid execution. Specifically, the execution task data includes a sequence number, a task group, a task name, a task execution instruction, an execution result status, and the like; the task dependency data includes a sequence number, a pre-task name, a subsequent task name, information data of the task group, and the like; the task lock data includes Serial number, lock name, lock type, information data of task group, etc. The input module generates four rows of records for each task group to add, delete, modify, and query operation types. It should be noted that, in the case of a task containing a mixed relationship type, the task dependency data may have multiple predecessor names.

(2) Execution module, set to perform multiple database connections according to task dependency data, and perform multiple tasks. Specifically, FIG. 4 is a flowchart of an execution process of each submodule in an execution module according to Embodiment 2 of the present invention. As shown in FIG. 4, the module includes four submodules, namely, a task acquisition submodule and a dependency control subroutine. Module, Execute Task Submodule, Exit Control Submodule. In the execution module, a database connection first executes the task to obtain the sub-module to obtain an execution task, and then executes the dependency control sub-module, analyzes and obtains the execution dependency of the task, analyzes whether the task can be executed, and if executed, executes the execution task sub-module. , obtain the task running instruction, execute the task running instruction; finally execute the exit control sub-module, analyze whether a database connection can exit the execution module, and no longer perform execution.

(3) Output module, set to output task execution result and task node presentation, node layout data, and so on. In the database-based task execution apparatus, an input module is first executed to input task data and task dependency data; and then multiple database connections are started; each database is connected to a function of the execution module to complete execution of each task, so as to Dependencies implement hybrid execution; finally execute the output module, output task execution results and task node presentation, node layout data, and so on. Embodiment 2 The following describes a database-based task execution method in conjunction with the structure of the database-based task execution apparatus provided in Embodiment 1, the method comprising the following steps: Step 1: A database connection execution input module, the input module generates a task group , accept the input of the task, the task relationship data, and generate a four-line record for each task group. Step 2, a database connection execution module, specifically, as shown in FIG. 4, each sub-module in the execution module is executed in the following order: 1) A database connection locks the entire row of the record row of the query operation type of the task group;

2) A database connection obtains an unexecuted task from the task group's execution task data, the task is to satisfy the running error code equal to -1, the error number is less than 5 times (configurable), and the task execution result status is initialized and re A task that is open, and not in the predecessor task temporary table; when no task is executed, returns a task of "NOTASK (ie, no task)" tag, when the lock waits for a timeout, returns a "TIMEOUT (ie, timeout), , the task of the label;

3) The database connection unlocks the record line of the query operation tag of the task group;

4) If the database connection in step 2) gets an execution task, go directly to step 5) Execute the dependency control sub-module; if the database connection in step 2) gets the task of the "NOTASK" tag or the task of the "TIMEOUT" tag , then go directly to step 9) to execute the exit control sub-module;

5) Dependent control sub-module function According to the task relationship table, whether the task acquired in step 2) has a pre-task waiting, specifically, whether all the pre-task tasks are executed successfully, and if the judgment result is true, directly enter Step 6) Execute the execution task sub-module. If the judgment result is false, release the task, set the execution result status of the task to re-open, and insert data into the pre-task temporary table to prevent the database connection and other database connections. Re-crawl the task, go directly to step 9) Execute the exit control sub-module;

6) A database connection updates the execution result status of the acquired task to run, and executes the running instruction of the task;

7) If the task is successfully executed, the database connection locks the entire row of the record row of the delete operation tag of the task group, and after obtaining the lock, the execution result of the task is updated to be successful, and the pre-task temporary table is added. In the middle of this connection data deletion, all subsequent tasks of the other connections are deleted; if the task fails to run, the number of errors is less than 5 times, the status of the execution result of updating the task is reopened, the number of errors is increased by 1, and the number of errors is equal to 5 times. The execution result status of the subsequent tasks of the task is set to the dependency failure, the running times are equal to 100, the execution result status of the task is set to failure, and the running times are equal to 5;

8) The database connection unlocks the record line of the delete operation tag of the task group.

9) A database connection analyzes the task table data. When the number of successful tasks is increased, the number of failed execution tasks is equal to the total number of executed tasks, or the user pre-posts a stop command, If the polling condition is false, exit the execution module; if not, continue polling execution if the polling condition is true, and execute the next polling execution, that is, return to step 1) to continue execution. Step 3, a database connection execution output module, output task execution result, task node presentation, node layout data, and the like. Embodiment 3 This embodiment describes a process in which a user implements a hybrid relationship operation by using the database-based task execution device in a specific implementation process: Step 1, using a database client tool or an application, connecting to a database, and executing an input module, The input task data is input, the task relationship data is input, or the task data and the task relationship data have entered the temporary table in advance, and only the contents of the temporary table are updated to the task table and the task relationship table. Step 2: Connect to the database with the database client tool or application, execute the interface function sp_set_loop of the execution module, the parameter is the task group name, and start an execution database connection. Step 3: In the same way as step 2, start other database connections to implement multi-task hybrid execution. It should be noted that in the above method, the task group is used to isolate data. Under the premise of database resource permission, multiple task groups can work at the same time, and the working groups work independently to apply to data processing of different projects. Embodiment 4 FIG. 5 is a schematic structural diagram of a task having a network dependency according to Embodiment 4 of the present invention. As shown in FIG. 5, the task group includes nine tasks, and the embodiment is combined with Embodiment 1 above. -3 provides a detailed process for processing a task group having the task relationship as shown in FIG. 5, including the following steps: Step 1, when starting the calculation, by the second step in Embodiment 3 The method of starting the database connection starts a database connection (recorded as process a) to process the task; step 2, according to the hierarchical value of the task, successively execute, E, G tasks (A, E, G tasks have the same level, successively The order can be determined randomly, or determined according to other pre-set rules;); Step 3: The process a obtains the B task that meets the rules described in Embodiment 2, and determines that the pre-task A task is successfully executed, and then executes the B task; Step 4, the startup database connection described in step 2 of Embodiment 3 is performed. The way, another database connection is added (denoted as process b); Step 5, process b acquires the task, according to the inter-task dependency relationship, the task that can satisfy the condition that the pre-task execution is successful only has F or H task. It is assumed that the process b sends the acquired F task to the dependency control subunit, and the dependency control subunit determines that the F task is executable, and the process b executes the F task; while the process b executes the F task, the process a executes the B task, and the process a Obtaining the H task, according to the inter-task dependency, the H task satisfies the execution condition, then the process a executes the H task; Step 6, the process a has completed the H task, but the F task executed by the process b has not been completed, and the process a performs the task After the process is performed, it is judged that the captured task does not satisfy the condition that the pre-task has been successfully executed, and the captured task cannot be executed, and then waits, preferably, Add the captured task to the pre-task temporary table; Step 7, after the process b executes the F task, the process a selects to execute after the C task, and the process b selects to execute after the K task; Step 8, the process a completes the C task After that, the D task is captured, but the pre-task K of the D task is still being executed, and the process a has no tasks to do, waiting, preferably, the task D can be added to the pre-task temporary table; Step 9, process b After the K task is completed, the process b selects the D task, and the process a waits; Step 10, after the process b completes the D task, the exit control subunit determines that the process b should exit the calculation, then the process b session ends, and at the same time, the control subunit is exited. It is determined that process a should exit the calculation and process a session ends. In summary, the solution provided by the embodiment of the present invention can add a thread for processing a task at any time during the data processing process, and support dynamic increase or decrease of the database connection, thereby dynamically adjusting the concurrency of the data processing, and , can achieve data calculation of mesh dependencies. In addition, the method only needs a relational database, which can be implemented without special application service support, and is simple to use. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in an execution machine system such as a set of executable instructions, and although the logical sequence is shown in the flowchart, in some cases The steps shown or described may be performed in a different order than here. It will be apparent to those skilled in the art that the various modules or steps of the present invention described above can be implemented by a general-purpose execution device, which can be centralized on a single execution device or distributed over a network of multiple execution devices. Alternatively, they may be implemented by program code executable by the execution device, so that they may be stored in the storage device by the execution device, or they may be separately fabricated into individual integrated circuit modules, or they may be Multiple modules or steps are made into a single integrated circuit module. Thus, the invention is not limited to any specific combination of hardware and software. The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes can be made to the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the scope of the present invention are intended to be included within the scope of the present invention.

Claims

Claim

1. A database-based task execution method, including:

Join the database connection;

The database connection obtains a task to be executed from the task group; and determines whether the pre-task of the acquired task is all successfully executed, and if so, the database connection performs the acquired task, and if not, the database The connection continues to obtain the task to be executed from the task group.

2. The method according to claim 1, wherein the obtaining, by the database connection, the task to be executed from the task group comprises:

The database connection locks the record row of the query operation type of the task group; the database connection acquires a task that satisfies a predetermined condition, wherein the predetermined condition includes: the execution result state of the acquired task is initialization Or reopening; the database connection sets the execution result status of the acquired task to be selected; and the database connection unlocks the record line of the query operation type.

The method according to claim 2, wherein, after determining that the pre-task of the acquired task is not all successfully executed, the method further comprises: the database connection setting the execution result status of the acquired task to being re-opened .

The method according to claim 3, wherein, when the execution result status of the acquired task is set to be reopened, the acquired task is further added to the pre-task temporary table, and the predetermined condition further includes The acquired task is not in the pre-task temporary table.

The method according to claim 4, wherein the performing the acquiring task comprises: the database connection setting an execution result state of the acquired task to run, and executing a running instruction of the acquired task.

The method according to claim 5, wherein, in the case that the running instruction of the acquired task is successful, the method further includes:

The database connection locks a record row of a deletion operation type of the task group; The database connection sets the execution result status of the acquired task to be successful, and deletes the task in the pre-task temporary table;

The database connection unlocks a record line of a deletion operation type of the task group.

The method according to claim 5, wherein, in the case that the running instruction of the acquired task fails, the method further includes:

The database connection sets the execution result status of the acquired task to a failure, and sets the execution result status of the post task of the acquired task to a dependency failure.

8. The method according to claim 2, wherein, after the database connection obtains a task to be executed from the current task group, or after performing the acquired task, the method further includes:

Determining whether the sum of the number of successfully executed tasks and the number of failed tasks is equal to the total number of tasks in the task group, or determining whether the user issues a stop instruction, and if so, stopping execution, if not, returning from the task group Get the steps of the task to be performed.

9. The method according to claim 8, wherein the obtaining, by the database connection, the task to be executed from the current task group fails comprises:

The database connection receives a notification for indicating a lock wait timeout; or the database connection receives a notification indicating that there is no task to be executed in the task group.

10. The method according to claim 1, wherein determining whether the pre-task of the acquired task is all successfully executed comprises:

Determining a pre-task of the acquired task according to a dependency relationship between tasks in the task group set in advance;

It is determined whether all of the pre-tasks are successfully executed.

11. A database-based task execution device, comprising:

Add a module to join the database connection;

An obtaining module, configured to obtain a task to be executed from the task group, and a determining module, configured to determine whether all the pre-tasks of the acquired task are successfully executed; a scheduling module, configured to: when the determination result of the determining module is yes, scheduling an execution module, if the determination result of the determining module is negative, scheduling the acquiring module;

The execution module is configured to perform the acquired task.