WO2018126964A1

WO2018126964A1 - Task execution method and apparatus and server

Info

Publication number: WO2018126964A1
Application number: PCT/CN2017/118957
Authority: WO
Inventors: 单立明; 钟陈练; 匡林林
Original assignee: 中兴通讯股份有限公司
Priority date: 2017-01-04
Filing date: 2017-12-27
Publication date: 2018-07-12
Also published as: CN108280023B; CN108280023A

Abstract

Disclosed are a task execution method and apparatus and a server. The method comprises : reading a task description file corresponding to a task, wherein a path of a database public packet for executing the task, a path of a database script file for representing an entity of the task, and a first parameter corresponding to the task are recorded in the task description file, with the first parameter being used for replacing a variable in the database script file (110); pushing the database script file to a computing node, with the computing node being provided with the database public packet (120); and invoking the database public packet on the computing node according to the path of the database public packet, acquiring, via the database public packet and according to the path of the database script file, the database script file, and replacing the variable in the database script file with the first parameter, and executing the database script file so as to obtain an execution result of the task (130).

Description

Task execution method, device and server

Technical field

The present disclosure relates to the field of computer technology, for example, to a task execution method, apparatus, and server.

Background technique

Spark Spark is a big data processing framework built around speed, ease of use and complex analysis. It promotes MapReduce (a programming model) to a lower cost Suffle (data cleaning algorithm) in the data processing process. Higher level. Spark takes advantage of in-memory data storage and near real-time processing power, which is many times faster than other big data processing technologies.

In the related art, the data cleaning task often needs to develop a large amount of code to support. When there are many cleaning tasks, the engineering package will become bloated, which brings various drawbacks. For example: 1. The code has a high repetition rate and serious human waste. When the project is very large, there will be a considerable part of the repetitive function code between different developers. The complexity of the merged code is high, which will greatly waste human resources. 2. The maintenance cost of the code is too high. When the project finds that there is a loophole in the task that needs to be adjusted, it is necessary to stop the entire project. This will cause most good tasks to wait for a problem task, resulting in serious waste of resources. 3. The code is not flexible enough. When there is a need to deliver, if there is a problem with the code of several tasks, the whole project will not be delivered. 4. The debugging, running and transfer of the code are inconvenient. Larger projects due to more tasks can make the debugging and running time of the code longer and make code transfer difficult.

Summary of the invention

The present disclosure provides a task execution method, apparatus, and server to solve at least one of the problems of code duplication, large amount of engineering, and inconvenient task management in a data cleaning task.

The present disclosure provides a task execution method, including:

Reading a task description file of the task, wherein the task description file records a path of a database common package for executing the task, a path of a database script file for representing an entity of the task, and a corresponding One parameter

Pushing the database script file to a computing node, where the computing node has the database common package;

Calling the database common package according to the path of the database common package on the computing node, obtaining the database script file by using the database common package according to the path of the database script file, and replacing the first parameter with the first parameter a variable in the database script file, and executing the database script file to obtain an execution result of the task.

Optionally, a path of the task configuration file is also recorded in the task description file, where the task configuration file records a second parameter used to replace a threshold value in the database script file. The method further includes:

Pushing the task configuration file to the computing node;

And obtaining, by the database common package, the second parameter in the task configuration file to replace a threshold value in the database script file according to a path of the task configuration file.

Optionally, acquiring the database script file and replacing the variables in the database script file with the first parameter, including:

Obtaining the database script file and generating a data exchange file according to a preset data exchange language, and extracting the first parameter from the data exchange file to replace a variable in the database script file.

Optionally, before the task description file of the task is read, the method further includes:

And according to the priority of the task, when the data required by the task has been acquired, adding the task to a task list recorded with at least one task arranged in order;

The tasks are read from the task list according to the order of the tasks in the task list.

Optionally, the task description file includes information about an input table corresponding to the task and information of an output table;

The executing the database script file includes:

The input data of the database script is obtained from the input table and the database script file is executed, and the obtained result is added to the output table.

The task execution device of the present invention comprises:

a reading module, configured to read a task description file corresponding to the task, where the task description file records a path of a database common package for executing the task, a path of a database script file for an entity representing the task, And a first parameter corresponding to the task, where the first parameter is used to replace a variable in the database script file;

a push module configured to push the database script file to a compute node, the compute node having the database common package;

An execution module, configured to invoke the database common package according to the path of the database common package on the computing node, obtain the database script file by using the database common package according to the path of the database script file, and use the database script file The first parameter replaces a variable in the database script file, and executes the database script file to obtain an execution result of the task.

Optionally, a path of the task configuration file is recorded in the task description file, where the task configuration file records a second parameter used to replace a threshold value in the database script file.

The push module is further configured to push the task configuration file to the computing node;

The execution module is further configured to acquire the second parameter in the task configuration file to replace a threshold value in the database script file by using the database common package according to the path of the task configuration file.

Optionally, the execution module is configured to invoke, according to the path of the database common package, the database common package on the computing node, and obtain, according to the path of the database script file, the database common package. And the database script file generates a corresponding data exchange file according to a preset data exchange language, and extracts the first parameter from the data exchange file to replace a variable in the database script file, and Executing the database script file to obtain an execution result of the task.

Optionally, the device further includes:

a task list module, configured to add, before the task description file of the read task, the task to the record with at least the priority of the task, when the data required by the task has been acquired The task list of a task; the task is read from the task list according to the order of the tasks in the task list.

The execution module is configured to invoke the database common package according to the path of the database common package on the computing node, obtain a database script file by using the database common package according to the path of the database script file, and use the database script file The first parameter replaces a variable in the database script file, and obtains input data of the database script from the input table and executes the database script file, and adds the obtained result to the output table.

The present disclosure also provides a server, comprising: the task execution device according to any of the above.

The present disclosure also provides a computer readable storage medium storing computer executable instructions for performing any of the methods described above.

The present disclosure also provides a server including one or more processors, a memory, and one or more programs, the one or more programs being stored in a memory, when executed by one or more processors, executing The above method.

The present disclosure also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, Having the computer perform any of the methods described above.

The task execution method, device and server provided by the present disclosure perform tasks based on multiple files compared with related technologies, and multiple files are independent of each other, which can effectively improve code flexibility, make code maintenance simple and convenient, and display database text. The separation solves the drawback that the main program is huge and difficult to maintain; because the use of the database common package reduces the code repetition rate, the task code becomes simpler and saves manpower.

DRAWINGS

FIG. 1 is a flow chart of a task execution method according to an embodiment.

2 is a flow chart of another task execution method of an embodiment.

3 is a block diagram of a task execution apparatus of an embodiment.

4 is a block diagram of another task execution apparatus of an embodiment.

FIG. 5 is a file structure diagram of a task execution method according to an embodiment.

FIG. 6 is a schematic diagram of a task execution method according to an embodiment.

FIG. 7 is a flow chart showing the operation of a task execution method according to an embodiment.

FIG. 8 is a flow chart showing the operation of another task execution method according to an embodiment.

FIG. 9 is a schematic diagram showing the hardware structure of a server according to an embodiment.

detailed description

Embodiment 1

As shown in FIG. 1 , an embodiment of the present application provides a task execution method, including:

Step 110: The task description file corresponding to the task is read, and the path of the database common package for executing the task, the path of the database script file for indicating the entity of the task, and the first parameter corresponding to the task are recorded in the task description file. A parameter is used to replace variables in the database script file. In this embodiment, the task description file may be an Extensible Markup Language (xml) format file, and the database script file may be an sql format file, and the tasks performed include, but are not limited to, a data cleaning task, and the database public package Can be a public jar package.

In step 120, the database script file is pushed to the computing node, and the computing node has a database common package.

Step 130: Call the database common package according to the path of the database common package on the computing node, obtain the database script file by using the database common package according to the path of the database script file, and replace the variable in the database script file with the first parameter, and execute Database script file to get the execution result of the task. According to the technical solution of the embodiment, each data cleaning task can be written as a separate algorithm, and each data cleaning task is submitted by using the application deployment tool spark-submit. The data cleaning task with the data source as the database table structure is extracted and made into a common public jar package file (for example, a database public package). The developer can describe the input and output of the task by configuring the xml description file corresponding to the task. The task execution method provided in this embodiment can solve the problems of code duplication, excessive engineering, inconvenient task management, and the like, and can also reduce the development cost of the data cleaning task with the data source as the database table structure. Maintenance costs.

Optionally, a path of the task configuration file is recorded in the task description file, where the task configuration file records a second parameter used to replace a threshold value in the database script file. The method further includes: The task configuration file is pushed to the computing node; the second parameter in the task configuration file is obtained by the database common package according to the path of the task configuration file to replace the gate in the database script file. Limit.

Optionally, acquiring the database script file and replacing the variable in the database script file with the first parameter comprises: acquiring the database script file and using the task description file according to a preset data exchange language. Generating a data exchange file, extracting the first parameter from the data exchange file to replace a variable in the database script file.

Optionally, before the task description file of the task is read, the method further includes: adding the task to the record in order according to the priority of the task, when the data required by the task has been acquired The task list of one or more tasks; the tasks are read from the task list according to the order of the tasks in the task list.

Optionally, the task description file includes information of an input table corresponding to the task and information of an output table, and the executing the database script file includes: acquiring input data of the database script from the input table. And executing the database script file, and adding the obtained result to the output table.

Embodiment 2

As shown in FIG. 2, an embodiment of the present application provides a task execution method, including:

Step 210: The task description file corresponding to the task is read, and the path of the database common package for executing the task, the path of the database script file for indicating the entity of the task, and the first parameter corresponding to the task are recorded in the task description file. One parameter is used to replace the variable in the database script file, the task description file also records the path of the task configuration file, the task configuration file records the second parameter for replacing the threshold value in the database script file, and the task description file includes the task. The information of the corresponding input table and the information of the output table.

In this embodiment, an original task can be composed of three parts: a task.xml file (ie, an xml format file), a task .sql file (ie, a file in sql format), and a task .conf file (ie, a file in a conf format). If there is no threshold, it can be omitted). In the task.xml file, save the task type (sql task or RDD task), input the table (the database, table name, type, partition, etc. information/file), and output the table (the database, table name, type, partition) Such information/file), execution time (timed task), jar package path for task execution, task .conf file, task.sql file path, etc. The task .sql file holds the task entity, that is, the sql statement with variables. Constant information such as task thresholds is saved in the task .conf file.

Step 220: Push the database script file to the computing node, push the task configuration file to the computing node, and have a database common package at the computing node. Based on the technical solution of the embodiment, a main program can be implemented. After starting the main program, the main program reads the information in the task .xml file into the memory task metadata list, and pushes the task .sql file and the task .conf file to at least one spark computing node.

Step 230, according to the priority of the task, whether the data required by the task has been acquired, adding the task to the task list recorded with at least one task arranged in order; reading from the task list according to the order of the tasks in the task list After the task, according to the preset data exchange language, the task description file generates a corresponding data exchange file, extracts the first parameter from the data exchange file to replace the variable in the database script file, and invokes the database public according to the path of the database common package. The package executes the database script file to get the execution result of the task. In this embodiment, the main program may add a task to the task scheduling list, that is, the task list, according to the task metadata description (data driven/time driven), data arrival status, task priority status, and the like.

In this embodiment, the main program can use the spark-submit to submit tasks to the spark cluster and perform the above tasks on the spark cluster computing node. The main program monitors the use of the spark resource. When the resource allows, the main program generates the task .json file, the sql public jar package, the parameters in the task.json file, and the task in the .conf file according to the required parameters in the task metadata. Threshold, and replace the variables in the task .sql file to generate a complete task sql statement (task entity). Obtain the corresponding database connection information (spark/gbase/mysql/other) in the json file according to the input table type, thereby connecting to the corresponding database and executing the sql statement in the task entity. According to the output type, save the task execution result to the corresponding database or file.

Step 240: Replace the threshold value in the database script file by obtaining the second parameter in the task configuration file by using the database common package, and obtain the input data of the database script from the input table and execute the database script file, and add the obtained result to the output. In the table. In this embodiment, the public jar package, that is, the database public package, can return the result of executing the task to the main program, and the main program judges the final execution result of the task.

According to the technical solution of the embodiment, comparing the multiple tasks can effectively improve the code flexibility and make the code maintenance simple and convenient. Separating the main program from the algorithm solves the drawback that the main program is huge and difficult to maintain. In addition, the extraction of the common jar package reduces the duplication of code, making the data cleaning task easier and saving manpower.

Embodiment 3

As shown in FIG. 3, an embodiment of the present application provides a task execution apparatus, including: a reading module 310, a pushing module 320, and an execution module 330;

The reading module 310 is configured to read a task description file corresponding to the task, where the task description file records the path of the database common package for executing the task, the path of the database script file for the entity representing the task, and the corresponding task A parameter, the first parameter is used to replace variables in the database script file. In this embodiment, the task description file may adopt an xml format file, and the database script file may be a sql format file, and the tasks performed include, but are not limited to, a data cleaning task, and the database public package may be a public jar package.

The push module 320 is configured to push the database script file to the computing node, where the computing node has a database common package.

The executing module 330 is configured to invoke the database common package according to the path of the database common package on the computing node, obtain the database script file by using the database common package according to the path of the database script file, and replace the variable in the database script file with the first parameter. And execute the database script file to get the execution result of the task.

According to the technical solution of the embodiment, each data cleaning task can be written into a separate algorithm, and the task is submitted by using the spark-submit, and the data cleaning task with the data source as the database table structure is extracted to be a common public jar package file. The developer can describe the input and output of the task by configuring an XML description file corresponding to the task. The task execution device provided by the embodiment can solve the problems of duplication of the cleaning task code, excessive engineering, inconvenient task management, and the development cost and maintenance cost of the data cleaning task with the data source as the database table structure.

Optionally, a path of the task configuration file is recorded in the task description file, where the task configuration file records a second parameter used to replace a threshold value in the database script file; the push module 320 further sets In order to push the task configuration file to the computing node, the execution module is further configured to acquire the second parameter in the task configuration file according to the path of the task configuration file by using the database common package. To replace the threshold in the database script file.

Optionally, the execution module 330 is configured to invoke the database common package according to the path of the database common package on the computing node, and obtain the location by using the database common package according to the path of the database script file. Descripting a database script file and generating a corresponding data exchange file according to a preset data exchange language, and extracting the first parameter from the data exchange file to replace a variable in the database script file, And executing the database script file to obtain an execution result of the task.

Optionally, the apparatus described with reference to FIG. 4 further includes: a task list module 430, configured to set, before the task description file of the task, according to the priority of the task, when the data required by the task has been acquired And adding the task to a task list recorded with one or more tasks arranged in order; reading the task from the task list according to an order of the tasks in the task list.

The execution module 330 is configured to invoke the database common package according to the path of the database common package on the computing node, obtain a database script file by using the database common package according to the path of the database script file, and use The first parameter replaces a variable in the database script file, and obtains input data of the database script from the input table and executes the database script file, and adds the obtained result to the output table.

Embodiment 4

As shown in FIG. 4, in another embodiment of the present application, another task execution apparatus is provided, including:

The reading module 410 is configured to read a task description file corresponding to the task, where the task description file records the path of the database common package for executing the task, the path of the database script file for representing the entity of the task, and the corresponding task a parameter, the first parameter is used to replace the variable in the database script file, the task description file also records the path of the task configuration file, and the task configuration file records the second parameter for replacing the threshold value of the database script file, and the task description file The information of the input table corresponding to the task and the information of the output table are included.

In this embodiment, an original task can be composed of three parts: a task.xml file, a task.sql file, and a task.conf file (if no threshold can be omitted). In the task.xml file, save the task type (sql task / RDD task), input the table (the database, table name, type, partition, etc. information / file), output table (the database, table name, type, partition) Such information/file), execution time (timed task), jar package path for task execution, task .conf file, task.sql file path, etc. The task .sql file holds the task entity, that is, the sql statement with variables. Constant information such as task thresholds is saved in the task .conf file.

The push module 420 is configured to push the database script file to the computing node, push the task configuration file to the computing node, and have a database common package at the computing node. Based on the technical solution of the embodiment, a main program can be implemented. After starting the main program, the main program reads the information in the task .xml file into the memory task metadata list, and pushes the task .sql file and the task .conf file to at least one spark computing node.

The task list module 430 is configured to add the task to the task list recorded with at least one task arranged in order according to the priority of the task, when the data required by the task has been acquired; according to the order of the tasks in the task list, After reading the task in the task list, according to the preset data exchange language, the task description file generates a corresponding data exchange file, and the first parameter is extracted from the data exchange file to replace the variable in the database script file, and the database public package is The path calls the database common package to execute the database script file to get the execution result of the task. In this embodiment, the main program may add a task to the task scheduling list, that is, the task list, according to the task metadata description (data driven/time driven), data arrival status, task priority status, and the like.

In this embodiment, the main program can use the spark-submit to submit tasks to the spark cluster and perform the above tasks on the spark cluster computing node. The main program monitors the use of the spark resource. When the resource allows, the main program generates the task .json file, the sql public jar package, the parameters in the task.json file, and the task in the .conf file according to the required parameters in the task metadata. Threshold, and replace the variables in the task .sql file to generate a complete task sql statement (task entity). Obtain the corresponding database connection information (spark/gbase/mysql/other) in the .json file according to the input table type, thereby connecting to the corresponding database and executing the sql statement in the task entity. According to the output type, save the task execution result to the corresponding database or file.

The executing module 440 is configured to replace the threshold value in the database script file by obtaining the second parameter in the task configuration file by using the database common package, and obtain the input data of the database script from the input table and execute the database script file, and the obtained The result is added to the output table. In this embodiment, the public jar package returns the result of executing the task to the main program, and the main program determines the final execution result of the task.

According to the technical solution of the embodiment, the task independence can effectively improve the code flexibility and make the code maintenance simple and convenient compared with the related art. Separating the main program from the algorithm solves the drawback that the main program is huge and difficult to maintain. In addition, the extraction of the common jar package reduces the duplication of code, making the data cleaning task easier and saving manpower.

Embodiment 5

In one embodiment of the present application, a server is provided, which includes any one of the task execution devices described in Embodiment 3 or Embodiment 4. It should be understood by those skilled in the art that the above-mentioned task execution apparatus may be included on the server, that is, each function module of the above-mentioned task execution apparatus may be implemented by server-based software and/or hardware, and the above embodiment may be implemented by the server of the embodiment. The technical effect of the task execution device.

An application example of an embodiment of the present application is as follows:

1) Write the original task description file, which is the three files shown in Figure 5.

For example, a task named lte_subject_poorquality_cell_day first edits the task lte_subject_poorquality_cell_day.xml file according to the template. The xml file records the following contents: task name; execution granularity, which is executed every day; indication information, indicating that the task exists sql statement; sql public jar package entry class and entry function (can be omitted, there is a default value in the main program); The path of the task entity sql file on the spark computing node; the path of the task configuration file on the spark computing node; the following is the description of the dependency table information in the task; the information description of the input table; the output table information description; And memory; delete stale information patterns.

Edit the lte_subject_poorquality_cell_day.sql file. Among them, the variable replacement rule can be a name surrounded by two ‘$’ symbols. The threshold replacement rule can be a name surrounded by two ‘#’ symbols. The method may include: an output table alias, an output table partition value, and a threshold value, wherein the output table alias corresponds to an alias in the task .xml file, and is replaced with an actual name of the output table when the algorithm public jar package is executed; and the output table partition value corresponds to the task. The alias in the xml file is replaced with the actual value when the algorithm public jar package is executed; the threshold value corresponds to the value in the task.conf file, and is replaced with the actual value when the algorithm public jar package is executed.

According to whether the above sql file has a threshold value to determine whether to write the lte_subject_poorquality_cell_day.conf file, in this example, the presence threshold value is taken as an example, and the threshold value is extracted to a separate configuration file to facilitate subsequent modification of the threshold value.

2) Import the original task file into the main program. The main program scans all the tasks in the contract directory at startup, and the tasks added and modified separately can also be added in the form of patches by special interfaces. The main program loads the original task file as shown in Figure 6. The main program reads the task .xml file into memory and adds the task metadata list for use after generating the task .json file. The task .conf and the task .sql file are used. Pushed to the corresponding compute node (such as the spark Spark task node in the figure) for use by the sql public jar package when the task is executed.

3) Wait for the task to execute. A task with data is a task that can be performed. There are sparks in which computing resources are available to actually perform the task. When all the preparation conditions are met, the task will be submitted.

4) sql public jar package execution. As shown in Figure 7, step 710, parsing the task .json file. When the task is submitted, the sql public jar package reads the task .json file generated by the main program. Step 720: Generate a parameter information replacement list. Step 730: Read the sql file, and replace the item in the sql file containing the condition information replacement list. Get the task .sql file path and the task .conf file path to read the sql statement in the sql file. The sql statement of each task can be multiple. The time conditions, input tables, output tables, and thresholds in these statements are variables according to the framework constraint format, and need to be replaced in the sql public jar package. The replacement value is obtained by reading the relevant parameters in the task .json file. Step 740: According to the input type, establish a corresponding driver driver, and execute a complete sql statement. After the replacement is completed, the sql statement is a sql statement that can be run directly. At this time, according to the database type corresponding to the input table, a link is established with the database, and the sql statement is executed. Step 750: Whether the input table and the output table are tables of the same database. If the output is the same as the input under the database, the task ends (insert statement). If the output is another database table, go to step 760. Step 760: Store the execution result of the sql file, and save the execution result into a corresponding file. The execution result will generate the corresponding file, execute the stored procedure corresponding to the database and store the execution result in the library. Step 770: Determine whether the output is a file format. If the output is in file format, the corresponding end. If the output is not in the file format, go to step 780. Step 780: Load the file of the execution result into the corresponding database. The overall flow of this example is shown in Figure 8, where the task is controlled in order by the task list.

Step 810: Load a task description file to generate task metadata.

Step 820: Generate a task .json file and execute the task according to the task scheduling list.

In step 830, whether it is a Sql task, if yes, step 840 is performed, and if not, step 850 is performed.

Step 840: The sql public jar package runs the sql task, and the number of task lists is decreased by 1, and step 860 is performed.

Step 850: Corresponding to the operation of the RDD public jar package, the number of task lists is decreased by 1, and step 860 is performed.

In step 860, the task list is empty. If it is empty, the process ends. If not, step 820 is performed.

5) The sql public jar package passes the final execution result to the main program.

The embodiment further provides a computer readable storage medium storing computer executable instructions for performing the above method.

FIG. 9 is a schematic diagram showing the hardware structure of a server according to an embodiment. As shown in FIG. 9, the server includes: one or more processors 910 and a memory 920. One processor 910 is taken as an example in FIG.

The server may also include an input device 930 and an output device 940.

The processor 910, the memory 920, the input device 930, and the output device 940 in the server may be connected by a bus or other means, and the bus connection is taken as an example in FIG.

Input device 930 can receive input numeric or character information, and output device 940 can include a display device such as a display screen.

The memory 920 is a computer readable storage medium that can be used to store software programs, computer executable programs, and modules. The processor 910 performs various functional applications and data processing by executing software programs, instructions, and modules stored in the memory 920 to implement any of the above-described embodiments.

The memory 920 may include a storage program area and an storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to usage of the server, and the like. In addition, the memory may include volatile memory such as random access memory (RAM), and may also include non-volatile memory such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device.

Memory 920 can be a non-transitory computer storage medium or a transitory computer storage medium. The non-transitory computer storage medium, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 920 can optionally include memory remotely located relative to processor 910, which can be connected to the server over a network. Examples of the above networks may include the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

Input device 930 can be used to receive input numeric or character information and to generate key signal inputs related to user settings and function control of the server. Output device 940 can include a display device such as a display screen.

The server of this embodiment may also include communication means 950 for transmitting and/or receiving information over a communication network.

A person skilled in the art can understand that all or part of the process of implementing the above embodiment method can be completed by executing related hardware by a computer program, and the program can be stored in a non-transitory computer readable storage medium. The program, when executed, may include the flow of an embodiment of the method as described above, wherein the non-transitory computer readable storage medium may be a magnetic disk, an optical disk, a read only memory (ROM), or a random access memory (RAM). Wait.

Industrial applicability

The task execution method, device and server provided by the disclosure can solve the problems of code duplication, excessive engineering, inconvenient task management, etc. in the data cleaning task, and reduce development cost and maintenance cost.

Claims

A task execution method, including:

Reading a task description file of the task, wherein the task description file records a path of a database common package for executing the task, a path of a database script file for representing an entity of the task, and a corresponding One parameter

Pushing the database script file to a computing node, where the computing node has the database common package;

Calling the database common package according to the path of the database common package on the computing node, obtaining the database script file by using the database common package according to the path of the database script file, and replacing the first parameter with the first parameter a variable in the database script file, and executing the database script file to obtain an execution result of the task.
The method according to claim 1, wherein a path of a task configuration file is recorded in the task description file, and the task configuration file records a second parameter for replacing a threshold value in the database script file; The method further includes:

Pushing the task configuration file to the computing node;

And obtaining, by the database common package, the second parameter in the task configuration file to replace a threshold value in the database script file according to a path of the task configuration file.
The method of claim 1, wherein the obtaining the database script file and replacing the variables in the database script file with the first parameter comprises:

Obtaining the database script file and generating a data exchange file according to a preset data exchange language, and extracting the first parameter from the data exchange file to replace a variable in the database script file.
The method of claim 1, wherein before the task description file of the task is read, the method further comprises:

And according to the priority of the task, when the data required by the task has been acquired, adding the task to a task list recorded with at least one task arranged in order;

The tasks are read from the task list according to the order of the tasks in the task list.
The method according to claim 1, wherein the task description file includes information of an input table corresponding to the task and information of an output table;

The executing the database script file includes:

The input data of the database script is obtained from the input table and the database script file is executed, and the obtained result is added to the output table.
A task execution device comprising:

a reading module, configured to read a task description file corresponding to the task, where the task description file records a path of a database common package for executing the task, a path of a database script file for an entity representing the task, And a first parameter corresponding to the task, where the first parameter is used to replace a variable in the database script file;

a push module configured to push the database script file to a compute node, the compute node having the database common package;

An execution module, configured to invoke the database common package according to the path of the database common package on the computing node, obtain the database script file by using the database common package according to the path of the database script file, and use the database script file The first parameter replaces a variable in the database script file, and executes the database script file to obtain an execution result of the task.
The apparatus according to claim 6, wherein a path of a task configuration file is recorded in the task description file, and the task configuration file records a second parameter for replacing a threshold value in the database script file;

The push module is further configured to push the task configuration file to the computing node;

The execution module is further configured to acquire the second parameter in the task configuration file to replace a threshold value in the database script file by using the database common package according to the path of the task configuration file.
The apparatus of claim 6, wherein the execution module is configured to invoke the database common package on the computing node according to a path of the database common package, and press the database script through the database common package a path of the file, acquiring the database script file, and generating a corresponding data exchange file according to a preset data exchange language, and extracting the first parameter from the data exchange file to replace the database A variable in the script file, and executing the database script file to obtain an execution result of the task.
The apparatus of claim 6 wherein said apparatus further comprises:

a task list module, configured to add, before the task description file of the read task, the task to the record with at least the priority of the task, when the data required by the task has been acquired The task list of a task; the task is read from the task list according to the order of the tasks in the task list.
The apparatus according to claim 6, wherein the task description file includes information of an input table corresponding to the task and information of an output table;

The execution module is configured to invoke the database common package according to the path of the database common package on the computing node, obtain a database script file by using the database common package according to the path of the database script file, and use the database script file The first parameter replaces a variable in the database script file, and obtains input data of the database script from the input table and executes the database script file, and adds the obtained result to the output table.
A server that includes:

A task execution device according to any one of claims 6 to 10.
A computer readable storage medium storing computer executable instructions for performing the method of any of claims 1-5.