WO2018126964A1 - Procédé et appareil d'exécution de tâches, et serveur - Google Patents

Procédé et appareil d'exécution de tâches, et serveur Download PDF

Info

Publication number
WO2018126964A1
WO2018126964A1 PCT/CN2017/118957 CN2017118957W WO2018126964A1 WO 2018126964 A1 WO2018126964 A1 WO 2018126964A1 CN 2017118957 W CN2017118957 W CN 2017118957W WO 2018126964 A1 WO2018126964 A1 WO 2018126964A1
Authority
WO
WIPO (PCT)
Prior art keywords
task
file
database
script file
database script
Prior art date
Application number
PCT/CN2017/118957
Other languages
English (en)
Chinese (zh)
Inventor
单立明
钟陈练
匡林林
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2018126964A1 publication Critical patent/WO2018126964A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/362—Software debugging
    • G06F11/3628—Software debugging of optimised code
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/70—Software maintenance or management
    • G06F8/73—Program documentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs

Definitions

  • the present disclosure relates to the field of computer technology, for example, to a task execution method, apparatus, and server.
  • Spark Spark is a big data processing framework built around speed, ease of use and complex analysis. It promotes MapReduce (a programming model) to a lower cost Suffle (data cleaning algorithm) in the data processing process. Higher level. Spark takes advantage of in-memory data storage and near real-time processing power, which is many times faster than other big data processing technologies.
  • the data cleaning task often needs to develop a large amount of code to support.
  • the engineering package will become bloated, which brings various drawbacks.
  • the code has a high repetition rate and serious human waste.
  • the complexity of the merged code is high, which will greatly waste human resources.
  • the maintenance cost of the code is too high.
  • the project finds that there is a loophole in the task that needs to be adjusted, it is necessary to stop the entire project. This will cause most good tasks to wait for a problem task, resulting in serious waste of resources.
  • the code is not flexible enough.
  • the present disclosure provides a task execution method, apparatus, and server to solve at least one of the problems of code duplication, large amount of engineering, and inconvenient task management in a data cleaning task.
  • the present disclosure provides a task execution method, including:
  • Reading a task description file of the task wherein the task description file records a path of a database common package for executing the task, a path of a database script file for representing an entity of the task, and a corresponding One parameter
  • a path of the task configuration file is also recorded in the task description file, where the task configuration file records a second parameter used to replace a threshold value in the database script file.
  • the method further includes:
  • acquiring the database script file and replacing the variables in the database script file with the first parameter including:
  • the method further includes:
  • the tasks are read from the task list according to the order of the tasks in the task list.
  • the task description file includes information about an input table corresponding to the task and information of an output table;
  • the executing the database script file includes:
  • the input data of the database script is obtained from the input table and the database script file is executed, and the obtained result is added to the output table.
  • the task execution device of the present invention comprises:
  • a reading module configured to read a task description file corresponding to the task, where the task description file records a path of a database common package for executing the task, a path of a database script file for an entity representing the task, And a first parameter corresponding to the task, where the first parameter is used to replace a variable in the database script file;
  • a push module configured to push the database script file to a compute node, the compute node having the database common package
  • An execution module configured to invoke the database common package according to the path of the database common package on the computing node, obtain the database script file by using the database common package according to the path of the database script file, and use the database script file
  • the first parameter replaces a variable in the database script file, and executes the database script file to obtain an execution result of the task.
  • a path of the task configuration file is recorded in the task description file, where the task configuration file records a second parameter used to replace a threshold value in the database script file.
  • the push module is further configured to push the task configuration file to the computing node
  • the execution module is further configured to acquire the second parameter in the task configuration file to replace a threshold value in the database script file by using the database common package according to the path of the task configuration file.
  • the execution module is configured to invoke, according to the path of the database common package, the database common package on the computing node, and obtain, according to the path of the database script file, the database common package.
  • the database script file generates a corresponding data exchange file according to a preset data exchange language, and extracts the first parameter from the data exchange file to replace a variable in the database script file, and Executing the database script file to obtain an execution result of the task.
  • the device further includes:
  • a task list module configured to add, before the task description file of the read task, the task to the record with at least the priority of the task, when the data required by the task has been acquired
  • the task list of a task the task is read from the task list according to the order of the tasks in the task list.
  • the task description file includes information about an input table corresponding to the task and information of an output table;
  • the execution module is configured to invoke the database common package according to the path of the database common package on the computing node, obtain a database script file by using the database common package according to the path of the database script file, and use the database script file
  • the first parameter replaces a variable in the database script file, and obtains input data of the database script from the input table and executes the database script file, and adds the obtained result to the output table.
  • the present disclosure also provides a server, comprising: the task execution device according to any of the above.
  • the present disclosure also provides a computer readable storage medium storing computer executable instructions for performing any of the methods described above.
  • the present disclosure also provides a server including one or more processors, a memory, and one or more programs, the one or more programs being stored in a memory, when executed by one or more processors, executing The above method.
  • the present disclosure also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, Having the computer perform any of the methods described above.
  • the task execution method, device and server provided by the present disclosure perform tasks based on multiple files compared with related technologies, and multiple files are independent of each other, which can effectively improve code flexibility, make code maintenance simple and convenient, and display database text.
  • the separation solves the drawback that the main program is huge and difficult to maintain; because the use of the database common package reduces the code repetition rate, the task code becomes simpler and saves manpower.
  • FIG. 1 is a flow chart of a task execution method according to an embodiment.
  • FIG. 2 is a flow chart of another task execution method of an embodiment.
  • FIG. 3 is a block diagram of a task execution apparatus of an embodiment.
  • FIG. 4 is a block diagram of another task execution apparatus of an embodiment.
  • FIG. 5 is a file structure diagram of a task execution method according to an embodiment.
  • FIG. 6 is a schematic diagram of a task execution method according to an embodiment.
  • FIG. 7 is a flow chart showing the operation of a task execution method according to an embodiment.
  • FIG. 8 is a flow chart showing the operation of another task execution method according to an embodiment.
  • FIG. 9 is a schematic diagram showing the hardware structure of a server according to an embodiment.
  • an embodiment of the present application provides a task execution method, including:
  • Step 110 The task description file corresponding to the task is read, and the path of the database common package for executing the task, the path of the database script file for indicating the entity of the task, and the first parameter corresponding to the task are recorded in the task description file.
  • a parameter is used to replace variables in the database script file.
  • the task description file may be an Extensible Markup Language (xml) format file
  • the database script file may be an sql format file
  • the tasks performed include, but are not limited to, a data cleaning task
  • the database public package Can be a public jar package.
  • step 120 the database script file is pushed to the computing node, and the computing node has a database common package.
  • Step 130 Call the database common package according to the path of the database common package on the computing node, obtain the database script file by using the database common package according to the path of the database script file, and replace the variable in the database script file with the first parameter, and execute Database script file to get the execution result of the task.
  • each data cleaning task can be written as a separate algorithm, and each data cleaning task is submitted by using the application deployment tool spark-submit.
  • the data cleaning task with the data source as the database table structure is extracted and made into a common public jar package file (for example, a database public package).
  • the developer can describe the input and output of the task by configuring the xml description file corresponding to the task.
  • the task execution method provided in this embodiment can solve the problems of code duplication, excessive engineering, inconvenient task management, and the like, and can also reduce the development cost of the data cleaning task with the data source as the database table structure. Maintenance costs.
  • a path of the task configuration file is recorded in the task description file, where the task configuration file records a second parameter used to replace a threshold value in the database script file.
  • the method further includes: The task configuration file is pushed to the computing node; the second parameter in the task configuration file is obtained by the database common package according to the path of the task configuration file to replace the gate in the database script file. Limit.
  • acquiring the database script file and replacing the variable in the database script file with the first parameter comprises: acquiring the database script file and using the task description file according to a preset data exchange language. Generating a data exchange file, extracting the first parameter from the data exchange file to replace a variable in the database script file.
  • the method further includes: adding the task to the record in order according to the priority of the task, when the data required by the task has been acquired
  • the task list of one or more tasks the tasks are read from the task list according to the order of the tasks in the task list.
  • the task description file includes information of an input table corresponding to the task and information of an output table
  • the executing the database script file includes: acquiring input data of the database script from the input table. And executing the database script file, and adding the obtained result to the output table.
  • an embodiment of the present application provides a task execution method, including:
  • Step 210 The task description file corresponding to the task is read, and the path of the database common package for executing the task, the path of the database script file for indicating the entity of the task, and the first parameter corresponding to the task are recorded in the task description file.
  • One parameter is used to replace the variable in the database script file
  • the task description file also records the path of the task configuration file
  • the task configuration file records the second parameter for replacing the threshold value in the database script file
  • the task description file includes the task.
  • an original task can be composed of three parts: a task.xml file (ie, an xml format file), a task .sql file (ie, a file in sql format), and a task .conf file (ie, a file in a conf format). If there is no threshold, it can be omitted).
  • a task.xml file save the task type (sql task or RDD task), input the table (the database, table name, type, partition, etc. information/file), and output the table (the database, table name, type, partition)
  • execution time timed task
  • jar package path for task execution task .conf file
  • task.sql file path etc.
  • the task .sql file holds the task entity, that is, the sql statement with variables. Constant information such as task thresholds is saved in the task .conf file.
  • Step 220 Push the database script file to the computing node, push the task configuration file to the computing node, and have a database common package at the computing node.
  • a main program can be implemented. After starting the main program, the main program reads the information in the task .xml file into the memory task metadata list, and pushes the task .sql file and the task .conf file to at least one spark computing node.
  • Step 230 according to the priority of the task, whether the data required by the task has been acquired, adding the task to the task list recorded with at least one task arranged in order; reading from the task list according to the order of the tasks in the task list
  • the task description file After the task, according to the preset data exchange language, the task description file generates a corresponding data exchange file, extracts the first parameter from the data exchange file to replace the variable in the database script file, and invokes the database public according to the path of the database common package.
  • the package executes the database script file to get the execution result of the task.
  • the main program may add a task to the task scheduling list, that is, the task list, according to the task metadata description (data driven/time driven), data arrival status, task priority status, and the like.
  • the main program can use the spark-submit to submit tasks to the spark cluster and perform the above tasks on the spark cluster computing node.
  • the main program monitors the use of the spark resource.
  • the main program When the resource allows, the main program generates the task .json file, the sql public jar package, the parameters in the task.json file, and the task in the .conf file according to the required parameters in the task metadata. Threshold, and replace the variables in the task .sql file to generate a complete task sql statement (task entity).
  • the output type save the task execution result to the corresponding database or file.
  • Step 240 Replace the threshold value in the database script file by obtaining the second parameter in the task configuration file by using the database common package, and obtain the input data of the database script from the input table and execute the database script file, and add the obtained result to the output.
  • the public jar package that is, the database public package, can return the result of executing the task to the main program, and the main program judges the final execution result of the task.
  • comparing the multiple tasks can effectively improve the code flexibility and make the code maintenance simple and convenient. Separating the main program from the algorithm solves the drawback that the main program is huge and difficult to maintain. In addition, the extraction of the common jar package reduces the duplication of code, making the data cleaning task easier and saving manpower.
  • an embodiment of the present application provides a task execution apparatus, including: a reading module 310, a pushing module 320, and an execution module 330;
  • the reading module 310 is configured to read a task description file corresponding to the task, where the task description file records the path of the database common package for executing the task, the path of the database script file for the entity representing the task, and the corresponding task A parameter, the first parameter is used to replace variables in the database script file.
  • the task description file may adopt an xml format file
  • the database script file may be a sql format file
  • the tasks performed include, but are not limited to, a data cleaning task
  • the database public package may be a public jar package.
  • the push module 320 is configured to push the database script file to the computing node, where the computing node has a database common package.
  • the executing module 330 is configured to invoke the database common package according to the path of the database common package on the computing node, obtain the database script file by using the database common package according to the path of the database script file, and replace the variable in the database script file with the first parameter. And execute the database script file to get the execution result of the task.
  • each data cleaning task can be written into a separate algorithm, and the task is submitted by using the spark-submit, and the data cleaning task with the data source as the database table structure is extracted to be a common public jar package file.
  • the developer can describe the input and output of the task by configuring an XML description file corresponding to the task.
  • the task execution device provided by the embodiment can solve the problems of duplication of the cleaning task code, excessive engineering, inconvenient task management, and the development cost and maintenance cost of the data cleaning task with the data source as the database table structure.
  • a path of the task configuration file is recorded in the task description file, where the task configuration file records a second parameter used to replace a threshold value in the database script file; the push module 320 further sets In order to push the task configuration file to the computing node, the execution module is further configured to acquire the second parameter in the task configuration file according to the path of the task configuration file by using the database common package. To replace the threshold in the database script file.
  • the execution module 330 is configured to invoke the database common package according to the path of the database common package on the computing node, and obtain the location by using the database common package according to the path of the database script file. Descripting a database script file and generating a corresponding data exchange file according to a preset data exchange language, and extracting the first parameter from the data exchange file to replace a variable in the database script file, And executing the database script file to obtain an execution result of the task.
  • the apparatus described with reference to FIG. 4 further includes: a task list module 430, configured to set, before the task description file of the task, according to the priority of the task, when the data required by the task has been acquired And adding the task to a task list recorded with one or more tasks arranged in order; reading the task from the task list according to an order of the tasks in the task list.
  • a task list module 430 configured to set, before the task description file of the task, according to the priority of the task, when the data required by the task has been acquired And adding the task to a task list recorded with one or more tasks arranged in order; reading the task from the task list according to an order of the tasks in the task list.
  • the task description file includes information about an input table corresponding to the task and information of an output table;
  • the execution module 330 is configured to invoke the database common package according to the path of the database common package on the computing node, obtain a database script file by using the database common package according to the path of the database script file, and use The first parameter replaces a variable in the database script file, and obtains input data of the database script from the input table and executes the database script file, and adds the obtained result to the output table.
  • another task execution apparatus including:
  • the reading module 410 is configured to read a task description file corresponding to the task, where the task description file records the path of the database common package for executing the task, the path of the database script file for representing the entity of the task, and the corresponding task a parameter, the first parameter is used to replace the variable in the database script file, the task description file also records the path of the task configuration file, and the task configuration file records the second parameter for replacing the threshold value of the database script file, and the task description file
  • the task description file records the path of the database common package for executing the task, the path of the database script file for representing the entity of the task, and the corresponding task a parameter
  • the first parameter is used to replace the variable in the database script file
  • the task description file also records the path of the task configuration file
  • the task configuration file records the second parameter for replacing the threshold value of the database script file
  • the information of the input table corresponding to the task and the information of the output table are included.
  • an original task can be composed of three parts: a task.xml file, a task.sql file, and a task.conf file (if no threshold can be omitted).
  • the task.xml file save the task type (sql task / RDD task), input the table (the database, table name, type, partition, etc. information / file), output table (the database, table name, type, partition) Such information/file), execution time (timed task), jar package path for task execution, task .conf file, task.sql file path, etc.
  • the task .sql file holds the task entity, that is, the sql statement with variables. Constant information such as task thresholds is saved in the task .conf file.
  • the push module 420 is configured to push the database script file to the computing node, push the task configuration file to the computing node, and have a database common package at the computing node.
  • a main program can be implemented. After starting the main program, the main program reads the information in the task .xml file into the memory task metadata list, and pushes the task .sql file and the task .conf file to at least one spark computing node.
  • the task list module 430 is configured to add the task to the task list recorded with at least one task arranged in order according to the priority of the task, when the data required by the task has been acquired; according to the order of the tasks in the task list, After reading the task in the task list, according to the preset data exchange language, the task description file generates a corresponding data exchange file, and the first parameter is extracted from the data exchange file to replace the variable in the database script file, and the database public package is The path calls the database common package to execute the database script file to get the execution result of the task.
  • the main program may add a task to the task scheduling list, that is, the task list, according to the task metadata description (data driven/time driven), data arrival status, task priority status, and the like.
  • the main program can use the spark-submit to submit tasks to the spark cluster and perform the above tasks on the spark cluster computing node.
  • the main program monitors the use of the spark resource.
  • the main program When the resource allows, the main program generates the task .json file, the sql public jar package, the parameters in the task.json file, and the task in the .conf file according to the required parameters in the task metadata. Threshold, and replace the variables in the task .sql file to generate a complete task sql statement (task entity).
  • the output type save the task execution result to the corresponding database or file.
  • the executing module 440 is configured to replace the threshold value in the database script file by obtaining the second parameter in the task configuration file by using the database common package, and obtain the input data of the database script from the input table and execute the database script file, and the obtained The result is added to the output table.
  • the public jar package returns the result of executing the task to the main program, and the main program determines the final execution result of the task.
  • the task independence can effectively improve the code flexibility and make the code maintenance simple and convenient compared with the related art.
  • Separating the main program from the algorithm solves the drawback that the main program is huge and difficult to maintain.
  • the extraction of the common jar package reduces the duplication of code, making the data cleaning task easier and saving manpower.
  • a server which includes any one of the task execution devices described in Embodiment 3 or Embodiment 4. It should be understood by those skilled in the art that the above-mentioned task execution apparatus may be included on the server, that is, each function module of the above-mentioned task execution apparatus may be implemented by server-based software and/or hardware, and the above embodiment may be implemented by the server of the embodiment. The technical effect of the task execution device.
  • a task named lte_subject_poorquality_cell_day first edits the task lte_subject_poorquality_cell_day.xml file according to the template.
  • the xml file records the following contents: task name; execution granularity, which is executed every day; indication information, indicating that the task exists sql statement; sql public jar package entry class and entry function (can be omitted, there is a default value in the main program);
  • the variable replacement rule can be a name surrounded by two ‘$’ symbols.
  • the threshold replacement rule can be a name surrounded by two ‘#’ symbols.
  • the method may include: an output table alias, an output table partition value, and a threshold value, wherein the output table alias corresponds to an alias in the task .xml file, and is replaced with an actual name of the output table when the algorithm public jar package is executed; and the output table partition value corresponds to the task.
  • the alias in the xml file is replaced with the actual value when the algorithm public jar package is executed; the threshold value corresponds to the value in the task.conf file, and is replaced with the actual value when the algorithm public jar package is executed.
  • the presence threshold value is taken as an example, and the threshold value is extracted to a separate configuration file to facilitate subsequent modification of the threshold value.
  • the main program scans all the tasks in the contract directory at startup, and the tasks added and modified separately can also be added in the form of patches by special interfaces.
  • the main program loads the original task file as shown in Figure 6.
  • the main program reads the task .xml file into memory and adds the task metadata list for use after generating the task .json file.
  • the task .conf and the task .sql file are used.
  • Pushed to the corresponding compute node (such as the spark Spark task node in the figure) for use by the sql public jar package when the task is executed.
  • a task with data is a task that can be performed. There are sparks in which computing resources are available to actually perform the task. When all the preparation conditions are met, the task will be submitted.
  • step 710 parsing the task .json file.
  • the sql public jar package reads the task .json file generated by the main program.
  • Step 720 Generate a parameter information replacement list.
  • Step 730 Read the sql file, and replace the item in the sql file containing the condition information replacement list. Get the task .sql file path and the task .conf file path to read the sql statement in the sql file.
  • the sql statement of each task can be multiple.
  • the time conditions, input tables, output tables, and thresholds in these statements are variables according to the framework constraint format, and need to be replaced in the sql public jar package.
  • the replacement value is obtained by reading the relevant parameters in the task .json file.
  • Step 740 According to the input type, establish a corresponding driver driver, and execute a complete sql statement. After the replacement is completed, the sql statement is a sql statement that can be run directly. At this time, according to the database type corresponding to the input table, a link is established with the database, and the sql statement is executed.
  • Step 750 Whether the input table and the output table are tables of the same database. If the output is the same as the input under the database, the task ends (insert statement). If the output is another database table, go to step 760.
  • Step 760 Store the execution result of the sql file, and save the execution result into a corresponding file.
  • Step 770 Determine whether the output is a file format. If the output is in file format, the corresponding end. If the output is not in the file format, go to step 780.
  • Step 780 Load the file of the execution result into the corresponding database. The overall flow of this example is shown in Figure 8, where the task is controlled in order by the task list.
  • Step 810 Load a task description file to generate task metadata.
  • Step 820 Generate a task .json file and execute the task according to the task scheduling list.
  • step 830 whether it is a Sql task, if yes, step 840 is performed, and if not, step 850 is performed.
  • Step 840 The sql public jar package runs the sql task, and the number of task lists is decreased by 1, and step 860 is performed.
  • Step 850 Corresponding to the operation of the RDD public jar package, the number of task lists is decreased by 1, and step 860 is performed.
  • step 860 the task list is empty. If it is empty, the process ends. If not, step 820 is performed.
  • the embodiment further provides a computer readable storage medium storing computer executable instructions for performing the above method.
  • FIG. 9 is a schematic diagram showing the hardware structure of a server according to an embodiment. As shown in FIG. 9, the server includes: one or more processors 910 and a memory 920. One processor 910 is taken as an example in FIG.
  • the server may also include an input device 930 and an output device 940.
  • the processor 910, the memory 920, the input device 930, and the output device 940 in the server may be connected by a bus or other means, and the bus connection is taken as an example in FIG.
  • Input device 930 can receive input numeric or character information
  • output device 940 can include a display device such as a display screen.
  • the memory 920 is a computer readable storage medium that can be used to store software programs, computer executable programs, and modules.
  • the processor 910 performs various functional applications and data processing by executing software programs, instructions, and modules stored in the memory 920 to implement any of the above-described embodiments.
  • the memory 920 may include a storage program area and an storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to usage of the server, and the like.
  • the memory may include volatile memory such as random access memory (RAM), and may also include non-volatile memory such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device.
  • Memory 920 can be a non-transitory computer storage medium or a transitory computer storage medium.
  • the non-transitory computer storage medium such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device.
  • memory 920 can optionally include memory remotely located relative to processor 910, which can be connected to the server over a network. Examples of the above networks may include the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
  • Input device 930 can be used to receive input numeric or character information and to generate key signal inputs related to user settings and function control of the server.
  • Output device 940 can include a display device such as a display screen.
  • the server of this embodiment may also include communication means 950 for transmitting and/or receiving information over a communication network.
  • a person skilled in the art can understand that all or part of the process of implementing the above embodiment method can be completed by executing related hardware by a computer program, and the program can be stored in a non-transitory computer readable storage medium.
  • the program when executed, may include the flow of an embodiment of the method as described above, wherein the non-transitory computer readable storage medium may be a magnetic disk, an optical disk, a read only memory (ROM), or a random access memory (RAM). Wait.
  • the task execution method, device and server provided by the disclosure can solve the problems of code duplication, excessive engineering, inconvenient task management, etc. in the data cleaning task, and reduce development cost and maintenance cost.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Library & Information Science (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Stored Programmes (AREA)

Abstract

L'invention concerne un procédé et un appareil d'exécution de tâches, et un serveur. Le procédé comporte les étapes consistant à: lire un fichier de description de tâche correspondant à une tâche, un chemin d'un paquet public de base de données destiné à exécuter la tâche, un chemin d'un fichier de script de base de données destiné à représenter une entité de la tâche, et un premier paramètre correspondant à la tâche étant enregistrés dans le fichier de description de tâche, le premier paramètre étant utilisé pour remplacer une variable dans le fichier de script de base de données (110); distribuer sélectivement le fichier de script de base de données à un nœud de calcul, le nœud de calcul étant muni du paquet public de base de données (120); et invoquer le paquet public de base de données sur le nœud de calcul selon le chemin du paquet public de base de données, acquérir, via le paquet public de base de données et selon le chemin du fichier de script de base de données, le fichier de script de base de données, et remplacer la variable dans le fichier de script de base de données par le premier paramètre, et exécuter le fichier de script de base de données de façon à obtenir un résultat d'exécution de la tâche (130).
PCT/CN2017/118957 2017-01-04 2017-12-27 Procédé et appareil d'exécution de tâches, et serveur WO2018126964A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710003855.3A CN108280023B (zh) 2017-01-04 2017-01-04 任务执行方法、装置和服务器
CN201710003855.3 2017-01-04

Publications (1)

Publication Number Publication Date
WO2018126964A1 true WO2018126964A1 (fr) 2018-07-12

Family

ID=62789189

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/118957 WO2018126964A1 (fr) 2017-01-04 2017-12-27 Procédé et appareil d'exécution de tâches, et serveur

Country Status (2)

Country Link
CN (1) CN108280023B (fr)
WO (1) WO2018126964A1 (fr)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110222122A (zh) * 2019-07-26 2019-09-10 深圳市元征科技股份有限公司 一种MongoDB的数据同步方法及相关设备
CN110704210A (zh) * 2019-09-20 2020-01-17 天翼电子商务有限公司 脚本任务调用方法、系统、介质及装置
CN110990669A (zh) * 2019-10-16 2020-04-10 广州丰石科技有限公司 一种基于规则生成的dpi解析方法和系统
CN113239005A (zh) * 2021-06-02 2021-08-10 上海许继电气有限公司 一种电力监控系统i、iv区数据同步方法及装置
CN113721824A (zh) * 2021-08-10 2021-11-30 深圳市一博科技股份有限公司 一种cr5000平台一键设置库路径的方法
CN113760489A (zh) * 2020-09-21 2021-12-07 北京沃东天骏信息技术有限公司 一种资源配置方法和装置
CN115061785A (zh) * 2022-04-15 2022-09-16 支付宝(杭州)信息技术有限公司 信息下发方法、装置、存储介质及服务器
CN117609102A (zh) * 2024-01-23 2024-02-27 云筑信息科技(成都)有限公司 一种建筑产业互联网造数平台系统测试方法

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112540858B (zh) * 2019-09-23 2023-10-27 华为云计算技术有限公司 任务处理方法、服务器、客户端及系统
CN111027196B (zh) * 2019-12-03 2023-04-28 南方电网科学研究院有限责任公司 一种电力设备的仿真分析任务处理方法、装置及存储介质
CN114489995B (zh) * 2022-02-15 2022-09-30 北京永信至诚科技股份有限公司 一种分布式调度处理方法及系统
CN115061741A (zh) * 2022-05-31 2022-09-16 北京奇艺世纪科技有限公司 一种目标插件运行方法、装置、电子设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102880546A (zh) * 2012-09-03 2013-01-16 上海方正数字出版技术有限公司 一种基于xml数据库的软件集成测试方法及系统
CN103678098A (zh) * 2012-09-06 2014-03-26 百度在线网络技术（北京）有限公司 一种hadoop程序测试的方法和系统
EP2977899A2 (fr) * 2014-06-27 2016-01-27 General Electric Company Intégration d'exécution de calcul analytique dans un environnement de traitement mapreduce
CN105487943A (zh) * 2015-12-09 2016-04-13 浪潮电子信息产业股份有限公司 一种自动向集群服务器各节点拷贝文件的方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009295013A (ja) * 2008-06-06 2009-12-17 Hitachi Ltd データベース管理方法、データベース管理装置およびプログラム
CN105224348A (zh) * 2014-06-11 2016-01-06 中兴通讯股份有限公司 一种MySQL数据库的安装方法及装置
CN104317928A (zh) * 2014-10-31 2015-01-28 北京思特奇信息技术股份有限公司 一种基于分布式数据库的业务etl方法及系统
CN105808776A (zh) * 2016-03-29 2016-07-27 中国建设银行股份有限公司 一种分布式数据库的数据管理系统及方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102880546A (zh) * 2012-09-03 2013-01-16 上海方正数字出版技术有限公司 一种基于xml数据库的软件集成测试方法及系统
CN103678098A (zh) * 2012-09-06 2014-03-26 百度在线网络技术（北京）有限公司 一种hadoop程序测试的方法和系统
EP2977899A2 (fr) * 2014-06-27 2016-01-27 General Electric Company Intégration d'exécution de calcul analytique dans un environnement de traitement mapreduce
CN105487943A (zh) * 2015-12-09 2016-04-13 浪潮电子信息产业股份有限公司 一种自动向集群服务器各节点拷贝文件的方法

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110222122A (zh) * 2019-07-26 2019-09-10 深圳市元征科技股份有限公司 一种MongoDB的数据同步方法及相关设备
CN110704210A (zh) * 2019-09-20 2020-01-17 天翼电子商务有限公司 脚本任务调用方法、系统、介质及装置
CN110704210B (zh) * 2019-09-20 2023-10-10 天翼电子商务有限公司 脚本任务调用方法、系统、介质及装置
CN110990669A (zh) * 2019-10-16 2020-04-10 广州丰石科技有限公司 一种基于规则生成的dpi解析方法和系统
CN113760489A (zh) * 2020-09-21 2021-12-07 北京沃东天骏信息技术有限公司 一种资源配置方法和装置
CN113760489B (zh) * 2020-09-21 2024-05-17 北京沃东天骏信息技术有限公司 一种资源配置方法和装置
CN113239005A (zh) * 2021-06-02 2021-08-10 上海许继电气有限公司 一种电力监控系统i、iv区数据同步方法及装置
CN113239005B (zh) * 2021-06-02 2022-12-02 上海许继电气有限公司 一种电力监控系统i、iv区数据同步方法及装置
CN113721824B (zh) * 2021-08-10 2024-05-03 深圳市一博科技股份有限公司 一种cr5000平台一键设置库路径的方法
CN113721824A (zh) * 2021-08-10 2021-11-30 深圳市一博科技股份有限公司 一种cr5000平台一键设置库路径的方法
CN115061785A (zh) * 2022-04-15 2022-09-16 支付宝(杭州)信息技术有限公司 信息下发方法、装置、存储介质及服务器
CN117609102A (zh) * 2024-01-23 2024-02-27 云筑信息科技(成都)有限公司 一种建筑产业互联网造数平台系统测试方法
CN117609102B (zh) * 2024-01-23 2024-05-28 云筑信息科技(成都)有限公司 一种建筑产业互联网造数平台系统测试方法

Also Published As

Publication number Publication date
CN108280023A (zh) 2018-07-13
CN108280023B (zh) 2022-11-01

Similar Documents

Publication Publication Date Title
WO2018126964A1 (fr) Procédé et appareil d'exécution de tâches, et serveur
US10810110B1 (en) Methods, systems, and articles of manufacture for testing web services using a behavior-driven development domain specific language framework
CN110069572B (zh) 基于大数据平台的hive任务调度方法、装置、设备及存储介质
US10061858B2 (en) Method and apparatus for processing exploding data stream
US9672140B1 (en) Processing special requests at dedicated application containers
CN111124906A (zh) 基于动态埋点的跟踪方法、编译方法、装置和电子设备
CN104484216A (zh) 服务接口文档和在线测试工具生成方法、装置
CN108984155B (zh) 数据处理流程设定方法和装置
CN106557470B (zh) 数据提取方法和装置
WO2020238597A1 (fr) Procédé, dispositif, système et support de mise à jour de données basés sur hadoop
AU2017254506B2 (en) Method, apparatus, computing device and storage medium for data analyzing and processing
CN110727429B (zh) 一种前端页面的生成方法、装置及设备
CN113656503A (zh) 数据同步方法、装置、系统及计算机可读存储介质
CN110764894A (zh) 一种定时任务管理方法、装置、设备及存储介质
CN114036183A (zh) 一种数据etl处理方法、装置、设备及介质
CN108399095B (zh) 支持动态管理定时任务的方法、系统、设备和存储介质
CN117032668A (zh) 可视化规则引擎的处理方法、装置、系统及平台
CN110188308B (zh) 客户端自动打点上报方法、存储介质、设备及系统
CN113806429A (zh) 基于大数据流处理框架的画布式日志分析方法
CN113656001A (zh) 平台组件开发方法、装置、计算机设备及存储介质
CN108804088B (zh) 协议处理方法和装置
CN112883088A (zh) 一种数据处理方法、装置、设备及存储介质
CN111143310A (zh) 日志记录方法及装置、可读存储介质
WO2021036987A1 (fr) Procédé et dispositif de réalisation d'exploitation et de surveillance de maintenance
CN115378996B (zh) 一种系统间数据传输的方法、装置、设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17890421

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17890421

Country of ref document: EP

Kind code of ref document: A1