CN114780174A - Configuration method and device for task operation in workflow, computer equipment and medium - Google Patents
Configuration method and device for task operation in workflow, computer equipment and medium Download PDFInfo
- Publication number
- CN114780174A CN114780174A CN202210441111.0A CN202210441111A CN114780174A CN 114780174 A CN114780174 A CN 114780174A CN 202210441111 A CN202210441111 A CN 202210441111A CN 114780174 A CN114780174 A CN 114780174A
- Authority
- CN
- China
- Prior art keywords
- task
- running
- workflow
- file
- log information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present application relates to the field of task data processing, and in particular, to a method and an apparatus for configuring task operation in a workflow, a computer device, and a storage medium, where the method includes: acquiring a first task file of a first workflow; analyzing the first task file to obtain the dependency relationship of the first task in the first task file; establishing an associated operation relation between the first task and the second task according to the dependency relation; the second task is a task in a second workflow; acquiring running log information of the first task, and determining the running time of the first task according to the running log information; generating the optimized running time of the second task according to the running time of the first task and the associated running relation; and modifying the configuration information of the second task in the second workflow according to the optimized running time. The method and the device can reduce the whole waiting time of the system task, reduce the occupation of queue resources and improve the processing efficiency of the task.
Description
Technical Field
The present application relates to the field of task data processing, and in particular, to a method and an apparatus for configuring task operation in a workflow, a computer device, and a storage medium.
Background
A complete data system is generally composed of a large number of task units, each task unit can generate corresponding workflow, the starting time, the ending time and the ending time of each work task can be clearly monitored in the same workflow which can be generated by the same task unit through the preset dependency relationship among the work tasks, so that the running state of the work tasks is monitored, in a complex data system, hierarchical calculation is often needed, at the moment, the data system can generate a large number of workflows, and the large number of workflows can have cross-workflow dependency. For such a situation, at present, the time used for actual logic processing of each task in the workflow cannot be accurately distinguished, so that the accuracy of the starting time of the tasks in the cross-workflow is low, and the overall task processing efficiency of the system is low.
Disclosure of Invention
The application mainly aims to provide a configuration method and device for task operation in a workflow, a computer device and a storage medium, and aims to solve the problem that the overall task processing efficiency of a system is low due to low accuracy of starting time of tasks in cross-workflow.
In order to achieve the above object, the present application provides a method for configuring task execution in a workflow, where the method includes:
acquiring a first task file of a first workflow;
analyzing the first task file to obtain the dependency relationship of the first task in the first task file;
establishing an associated operation relation between the first task and the second task according to the dependency relation; the second task is a task in a second workflow;
acquiring running log information of the first task, and determining the running time of the first task according to the running log information;
generating the optimized running time of the second task according to the running time of the first task and the associated running relation;
and modifying the configuration information of the second task in the second workflow according to the optimized running time.
Further, the analyzing the first task file to obtain the dependency relationship of the first task in the first task file includes:
analyzing the first task file to obtain a directed acyclic graph in the first task file;
and identifying the connection relation of each first task in the directed acyclic graph, and determining the dependency relation of the first task according to the connection relation.
Further, the establishing of the associated operation relationship between the first task and the second task according to the dependency relationship includes:
acquiring a second task file of a second workflow;
analyzing the second task file to obtain a second task which has a dependency relationship with the first task in the second task file;
and establishing the association operation relationship between the first task and the second task according to the dependency relationship.
Further, the analyzing the second task file to obtain a second task in the second task file, where the second task has a dependency relationship with the first task, includes:
analyzing the second task file to obtain a second task in the second task file and a pre-task of the second task;
and if the preposed task is a first task in a first workflow, determining that the second task has a dependency relationship with the second task so as to obtain a second task which has a dependency relationship with the first task in the second task file.
Further, the generating the optimized running time of the second task according to the running time of the first task and the associated running relationship includes:
acquiring the running time of a plurality of first tasks according to the association running relationship;
comparing the running time of the plurality of first tasks, and determining the maximum waiting time of the second task;
acquiring the starting time of the second task;
and subtracting the maximum waiting time according to the starting time to obtain the optimized running time of the second task.
Further, the obtaining operation log information of the first task and determining the operation time of the first task according to the operation log information include:
acquiring running log information of the first task;
identifying the running log information, and determining a plurality of items of information of the running log information;
identifying the project information, and determining a first task corresponding to the project information;
and identifying the running data of the first task, and acquiring the running time of the first task.
Further, the identifying the running log information and determining a plurality of items of information of the running log information includes:
identifying the running log information according to a preset log template, and segmenting the running log information to obtain a plurality of area information;
and identifying the area information according to a preset keyword, and determining a plurality of item information of the running log information.
The present application further provides a device for configuring task execution in a workflow, the device including:
the file acquisition module is used for acquiring a first task file of a first workflow;
the relationship analysis module is used for analyzing the first task file and acquiring the dependency relationship of the first task in the first task file;
the relationship determination module is used for establishing the associated operation relationship between the first task and the second task according to the dependency relationship; the second task is a task in a second workflow;
the log analysis module is used for acquiring running log information of the first task and determining the running time of the first task according to the running log information;
the time optimization module is used for generating the optimized running time of the second task according to the running time of the first task and the associated running relation;
and the time configuration module is used for modifying the configuration information of the second task in the second workflow according to the optimized running time.
The application also provides a computer device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the configuration method for running the tasks in the workflow when executing the computer program.
The application also provides a method for configuring the running of tasks in the cross-workflow, which is used for reconfiguring the running time of each task in the cross-workflow, and comprises the steps of firstly obtaining a first task file of a first workflow, analyzing the first task file in order to distinguish a naming mode of different workflows, obtaining the dependency relationship of the first task in the first task file, and establishing the associated running relationship of the first task and a second task according to the dependency relationship, wherein the first workflow does not refer to a workflow formed by a task unit and does not refer to one workflow; the second task is a task in a second workflow, after the dependency relationship in one workflow is obtained through analysis, the second task is compared with the dependency relationship in the other workflow, so that the dependency relationship between the first task and the second task is established according to the dependency relationship, the dependency relationship between different tasks across workflows is defined as an associated running relationship, running log information of the first task is obtained, the running time of the first task is determined according to the running log information, because the running of the second task is influenced by the running result of the first task, the optimized running time of the second task is generated according to the running time of the first task and the associated running relationship, the configuration information of the second task in the second workflow is modified according to the optimized running time, and the dependency relationship of different tasks across workflows is analyzed, and then determining the optimized running time of each work task according to the dependency relationship and the running time of the work tasks, thereby reducing the invalid waiting time of a large number of tasks in a cross-workflow and reducing the whole waiting time of the whole data system, thereby reducing the occupation of queue resources and improving the processing efficiency of the whole tasks of the data system.
Drawings
Fig. 1 is a schematic flowchart of an embodiment of a configuration method for task execution in a workflow of the present application;
fig. 2 is a schematic flowchart of an embodiment of obtaining a dependency relationship of a first task in the first task file according to the present application;
fig. 3 is a schematic flowchart of an embodiment of establishing an associated operation relationship between the first task and the second task according to the dependency relationship in the present application;
fig. 4 is a schematic flowchart of an embodiment of obtaining a second task having a dependency relationship with the first task in the second task file according to the present application;
FIG. 5 is a flowchart illustrating an embodiment of generating an optimized runtime of a second task according to the present application;
FIG. 6 is a flowchart illustrating an embodiment of determining a runtime of a first task;
fig. 7 is a schematic flowchart of an embodiment of identifying the operation log information and determining a plurality of items of information of the operation log information according to the present application;
FIG. 8 is a schematic structural diagram illustrating an embodiment of a configuration device for task execution in a workflow of the present application;
FIG. 9 is a block diagram illustrating a computer device according to an embodiment of the present disclosure.
The implementation, functional features and advantages of the object of the present application will be further explained with reference to the embodiments, and with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.
Referring to fig. 1, an embodiment of the present application provides a method for configuring task operations in a workflow, where the method for configuring task operations in a workflow includes steps S101 to S106, and details of each step of the method for configuring task operations in the workflow are described as follows.
S101, acquiring a first task file of a first workflow.
The embodiment is applied to a workflow scheduling system scene, including a workflow scheduling scene applied to a data system, in a complex data system, layered calculation is often required, and the workflow scheduling system comprises an ODS (oxide Dispersion strengthened) overlay layer, a DWD (discrete wavelet decomposition) data extraction layer, a DWM (discrete wavelet transform) data basic model layer and a DMM (distributed data market) data market model layer. Specifically, a first task file of a first workflow is first obtained, where the first workflow does not refer to a workflow formed by one task unit, does not refer to one workflow, and is a naming mode for distinguishing different workflows.
S102, analyzing the first task file, and obtaining the dependency relationship of the first task in the first task file.
In this embodiment, after a first task file of a first workflow is obtained, the first task file is parsed first, and a dependency relationship of a first task in the first task file is obtained, where the first task file includes a plurality of first tasks, such as job1, job2, and job3, and the first task file further records a dependency relationship between the first tasks.
S103, establishing a correlation operation relation between the first task and the second task according to the dependency relation; the second task is a task in a second workflow.
In this embodiment, after the first task file is analyzed and the dependency relationship of the first task in the first task file is obtained, the associated operation relationship between the first task and the second task is established according to the dependency relationship; the second task is a task in a second workflow, when the data processing system is complex, that is, when the data processing system has a plurality of data processing units or multiple data processing layers, often, a phenomenon that the workflow generated by each data processing unit or data processing layer crosses the workflow, that is, the first workflow crosses the second workflow, is specifically characterized in that the processing of the second task of the second workflow is affected by the processing result of the first task in the first workflow, and therefore, after the dependency relationship in one workflow is obtained through analysis, the dependency relationship is compared with the dependency relationship in another workflow, so that the dependency relationship between the first task and the second task is established according to the dependency relationship, and the dependency relationship between different tasks of the cross-workflow is defined as an associated operation relationship.
S104, obtaining running log information of the first task, and determining the running time of the first task according to the running log information.
In this embodiment, after the associated operation relationship between the first task and the second task is established according to the dependency relationship, since the operation of the second task of the second workflow is affected by the first task in the first workflow, the operation log information of the first task is obtained, where the operation log information records data of different operations of the first task, including start time, end time, operation start value, and operation end value.
And S105, generating the optimized running time of the second task according to the running time of the first task and the associated running relation.
In this embodiment, after obtaining the operation log information of the first task, determining the operation time of the first task according to the operation log information, that is, determining the start time and the end time of the operation of the first task, and then calculating the operation time of the first task, because the first task and the second task have an associated operation relationship, that is, the operation of the second task is affected by the operation result of the first task, the optimized operation time of the second task is generated according to the operation time of the first task and the associated operation relationship.
And S106, modifying the configuration information of the second task in the second workflow according to the optimized running time.
In this embodiment, after generating the optimized runtime of the second task according to the runtime of the first task and the associated running relationship, modifying the configuration information of the second task in the second workflow according to the optimized runtime, that is, after determining the optimized runtime of the second task, in order to reduce the waiting time of the second task, modifying the configuration information of the second task in the second workflow according to the optimized runtime, so as to modify the start time of the second task to the optimized runtime, which can reduce the invalid waiting time of the second task, by analyzing the dependency relationship across different work tasks in the workflow, and then determining the optimized runtime of each work task according to the dependency relationship and the runtime of the work task, thereby reducing the invalid waiting time across a large number of tasks in the workflow, the whole waiting time of the whole data system is reduced, so that the occupation of queue resources is reduced, and the processing efficiency of the whole task of the data system is improved.
The embodiment provides a method for configuring task running in a cross-workflow, which is used for reconfiguring the running time of each task in the cross-workflow, and comprises the steps of firstly obtaining a first task file of a first workflow, analyzing the first task file in order to distinguish a naming mode of different workflows, obtaining a dependency relationship of a first task in the first task file, and establishing an associated running relationship between the first task and a second task according to the dependency relationship, wherein the first workflow does not refer to a workflow formed by a task unit and does not refer to one workflow; the second task is a task in a second workflow, after the dependency relationship in one workflow is obtained through analysis, the second task is compared with the dependency relationship in the other workflow, so that the dependency relationship between the first task and the second task is established according to the dependency relationship, the dependency relationship between different tasks across workflows is defined as an associated running relationship, running log information of the first task is obtained, the running time of the first task is determined according to the running log information, because the running of the second task is influenced by the running result of the first task, the optimized running time of the second task is generated according to the running time of the first task and the associated running relationship, the configuration information of the second task in the second workflow is modified according to the optimized running time, and the dependency relationship of different tasks across workflows is analyzed, and then determining the optimized running time of each work task according to the dependency relationship and the running time of the work tasks, thereby reducing the invalid waiting time of a large number of tasks in a cross-workflow and reducing the whole waiting time of the whole data system, thereby reducing the occupation of queue resources and improving the processing efficiency of the whole tasks of the data system.
In an embodiment, as shown in fig. 2, the parsing the first task file to obtain the dependency relationship of the first task in the first task file further includes steps S201 to S202:
s201, analyzing the first task file to obtain a directed acyclic graph in the first task file;
s202, identifying the connection relation of each first task in the directed acyclic graph, and determining the dependency relation of the first task according to the connection relation.
In this embodiment, in the process of parsing the first task file to obtain the dependency relationship of the first task in the first task file, the first task file is first parsed to obtain the directed acyclic graph in the first task file, in one embodiment, the first task file records the operation relationship of each first task in the form of a directed acyclic graph, so the first task file is first parsed to obtain the directed acyclic graph in the first task file, then the connection relationship of each first task in the directed acyclic graph is identified, the dependency relationship of the first task is determined according to the connection relationship, specifically, the preceding task of each first task is identified, when the first task job2 exists in the preceding first task job1, the first task job1 is determined as the dependency of the first task 2, so as to determine the dependency relationship between all first tasks in the first task file, therefore, the accuracy of extracting the dependency relationship among the tasks is improved.
In one embodiment, as shown in fig. 3, the establishing an associated operation relationship between the first task and the second task according to the dependency relationship includes steps S301 to S303:
s301, acquiring a second task file of a second workflow;
s302, analyzing the second task file, and acquiring a second task which has a dependency relationship with the first task in the second task file;
s303, establishing the correlation operation relationship between the first task and the second task according to the dependency relationship.
In this embodiment, in the process of establishing the associated operation relationship between the first task and the second task according to the dependency relationship, first, the second task file of the second workflow is obtained, then, the second task file is analyzed, the second task in the second task file, which has a dependency relationship with the first task, is obtained, that is, the second task is the same as the first task file in processing manner, then, the associated operation relationship between the first task and the second task is established according to the dependency relationship, the same analysis operation is performed through the first task file and the second task file, and the dependency relationship in the first task file and the second task file is extracted, so that the accuracy of the dependency relationship of the tasks across workflows is improved.
In an embodiment, as shown in fig. 4, the parsing the second task file to obtain a second task in the second task file that has a dependency relationship with the first task further includes steps S401 to S402:
s401, analyzing the second task file, and acquiring a second task in the second task file and a pre-task of the second task;
s402, if the preposed task is a first task in a first workflow, determining that the second task has a dependency relationship with the second task, so as to obtain a second task which has a dependency relationship with the first task in the second task file.
In this embodiment, in the process of analyzing the second task file to obtain a second task in the second task file, where the second task has a dependency relationship with the first task, the second task file is first analyzed to obtain a second task in the second task file and a pre-task of the second task, and if the pre-task is a first task in a first workflow, it is determined that the second task has a dependency relationship with the second task to obtain a second task in the second task file, where the first task has a dependency relationship with the first task, and whether the pre-task of the second task is a first task is determined by comparing the pre-tasks of the second task one by one, so as to determine the dependency relationship between the first task and the second task to establish an associated operation relationship between the first task and the second task, thereby improving accuracy of task operation configuration across workflows.
In one embodiment, as shown in fig. 5, the generating the optimized runtime of the second task according to the runtime of the first task and the associated running relationship further includes steps S501-S504:
s501, acquiring the running time of a plurality of first tasks according to the associated running relation;
s502, comparing the running time of the plurality of first tasks, and determining the maximum waiting time of the second task;
s503, acquiring the starting time of the second task;
s504, the maximum waiting time is subtracted from the starting time to obtain the optimized running time of the second task.
In this embodiment, in the process of generating the optimized running time of the second task according to the running time of the first task and the associated running relationship, firstly, the running times of a plurality of first tasks are obtained according to the associated running relationship, the method comprises the steps of acquiring the running time of a plurality of first tasks which have associated running relation with a second task, comparing the running time of the plurality of first tasks, determining the maximum waiting time of the second task, the maximum waiting time is the ending time of the first task which is the last operation ending in the plurality of first tasks, then the starting time of the second task is obtained, obtaining the optimized running time of the second task according to the starting time minus the maximum waiting time, therefore, the optimal starting running time of the second task is accurately determined, and the accuracy of the task running configuration of the cross workflow is improved.
In one embodiment, as shown in fig. 6, the obtaining operation log information of the first task, and determining the operation time of the first task according to the operation log information, further includes steps S601-S604:
s601, acquiring running log information of the first task;
s602, identifying the running log information, and determining a plurality of items of the running log information;
s603, identifying the project information, and determining a first task corresponding to the project information;
s604, identifying the running data of the first task, and obtaining the running time of the first task.
In this embodiment, in the process of obtaining running log information of the first task and determining the running time of the first task according to the running log information, the running log information of the first task is obtained, the running log information is identified, a plurality of items of the running log information are determined, since the content distribution of the running log information has a certain specification, a template of the running log information is obtained by collecting the running log information with a plurality of different content distribution specifications and extracting common features of the running log information, then the running log information is identified according to the preset template of the running log information, a plurality of items of the running log information are determined, that is, the information content range of different first tasks in the running log information is determined, then the items of the running log information are identified, and the first task corresponding to the items of the running log information is determined, and identifying the running data of the first task, and acquiring the running time of the first task, so that the running time of each first task is accurately extracted from the running log information, and the accuracy of identifying the running log information is improved.
In one embodiment, as shown in fig. 7, the identifying the execution log information and determining a plurality of items of information of the execution log information further includes steps S701 to S702:
s701, identifying the running log information according to a preset log template, and dividing the running log information to obtain a plurality of pieces of regional information;
s702, identifying the area information according to a preset keyword, and determining a plurality of item information of the running log information.
In this embodiment, in the process of identifying the operation log information and determining the plurality of items of the operation log information, since the content distribution of the operation log information has a certain specification, by collecting the operation log information with various different content distribution specifications and extracting common features of the operation log information, a template of the operation log information is obtained, the operation log information is identified according to a preset template of the operation log information, the operation log information is divided to obtain a plurality of pieces of area information, the area information is identified according to preset keywords, a plurality of items of the operation log information are determined, and the item information in each area information is determined, that is, the operation data corresponding to each first task is represented, so that the accuracy of identifying the operation information of the first task is improved.
Referring to fig. 8, the present application further provides a configuration apparatus for task execution in a workflow, including:
a file obtaining module 101, configured to obtain a first task file of a first workflow;
the relationship analysis module 102 is configured to analyze the first task file to obtain a dependency relationship of a first task in the first task file;
the relationship determination module 103 is configured to establish an associated operation relationship between the first task and the second task according to the dependency relationship; the second task is a task in a second workflow;
the log analysis module 104 is configured to obtain running log information of the first task, and determine a running time of the first task according to the running log information;
the time optimization module 105 is configured to generate an optimized running time of the second task according to the running time of the first task and the associated running relationship;
a time configuration module 106, configured to modify configuration information of a second task in the second workflow according to the optimized runtime.
As described above, it can be understood that each component of the configuration device for task operation in the workflow proposed in the present application may implement the function of any one of the configuration methods for task operation in the workflow described above.
In one embodiment, the parsing the first task file to obtain the dependency relationship of the first task in the first task file includes:
analyzing the first task file to obtain a directed acyclic graph in the first task file;
and identifying the connection relation of each first task in the directed acyclic graph, and determining the dependency relation of the first task according to the connection relation.
In one embodiment, the establishing an associated operation relationship between the first task and the second task according to the dependency relationship includes:
acquiring a second task file of a second workflow;
analyzing the second task file to obtain a second task which has a dependency relationship with the first task in the second task file;
and establishing the association operation relationship between the first task and the second task according to the dependency relationship.
In an embodiment, the parsing the second task file to obtain a second task in the second task file, where the second task has a dependency relationship with the first task, includes:
analyzing the second task file to obtain a second task in the second task file and a pre-task of the second task;
and if the preposed task is a first task in a first workflow, determining that the second task has a dependency relationship with the second task so as to obtain the second task which has a dependency relationship with the first task in the second task file.
In one embodiment, the generating the optimized runtime of the second task according to the runtime of the first task and the associated running relationship includes:
acquiring the running time of a plurality of first tasks according to the association running relationship;
comparing the running time of the plurality of first tasks, and determining the maximum waiting time of the second task;
acquiring the starting time of the second task;
and subtracting the maximum waiting time according to the starting time to obtain the optimized running time of the second task.
In one embodiment, the obtaining running log information of the first task and determining the running time of the first task according to the running log information include:
acquiring running log information of the first task;
identifying the running log information, and determining a plurality of items of information of the running log information;
identifying the project information, and determining a first task corresponding to the project information;
and identifying the running data of the first task, and acquiring the running time of the first task.
In one embodiment, the identifying the running log information and determining a plurality of items of information of the running log information includes:
identifying the running log information according to a preset log template, and segmenting the running log information to obtain a plurality of area information;
and identifying the area information according to preset keywords, and determining a plurality of item information of the running log information.
Referring to fig. 9, an embodiment of the present application further provides a computer device, where the computer device may be a mobile terminal, and an internal structure of the computer device may be as shown in fig. 9. The computer equipment comprises a processor, a memory, a network interface, a display device and an input device which are connected through a system bus. Wherein, the network interface of the computer equipment is used for communicating with an external terminal through network connection. The display device of the computer device is used for displaying the offline application. The input device of the computer device is used for receiving the input of the user in offline application. The computer designed processor is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media. The non-volatile storage medium stores an operating system, a computer program, and a database. The database of the computer device is used for storing the original data. The computer program is executed by a processor to implement a method of configuring the operation of tasks in a workflow.
The processor executes a configuration method for task operation in the workflow, and the method comprises the following steps: acquiring a first task file of a first workflow; analyzing the first task file to obtain the dependency relationship of the first task in the first task file; establishing a correlation operation relation between the first task and the second task according to the dependency relation; the second task is a task in a second workflow; acquiring running log information of the first task, and determining the running time of the first task according to the running log information; generating the optimized running time of the second task according to the running time of the first task and the associated running relation; and modifying the configuration information of the second task in the second workflow according to the optimized running time.
The computer equipment provides a method for configuring the running of tasks in cross-workflow, which is used for reconfiguring the running time of each task in the cross-workflow, firstly, a first task file of a first workflow is obtained, the first workflow does not refer to a workflow formed by a task unit and does not refer to one workflow, the first task file is analyzed in order to distinguish a naming mode of different workflows, the dependency relationship of the first task in the first task file is obtained, and the associated running relationship of the first task and a second task is established according to the dependency relationship; the second task is a task in a second workflow, after the dependency relationship in one workflow is obtained through analysis, the second task is compared with the dependency relationship in the other workflow, so that the dependency relationship between the first task and the second task is established according to the dependency relationship, the dependency relationship between different tasks across workflows is defined as an associated running relationship, running log information of the first task is obtained, the running time of the first task is determined according to the running log information, because the running of the second task is influenced by the running result of the first task, the optimized running time of the second task is generated according to the running time of the first task and the associated running relationship, the configuration information of the second task in the second workflow is modified according to the optimized running time, and the dependency relationship of different tasks across workflows is analyzed, and then determining the optimized running time of each work task according to the dependency relationship and the running time of the work tasks, thereby reducing the invalid waiting time of a large number of tasks in a cross-workflow, reducing the whole waiting time of the whole data system, reducing the occupation of queue resources and improving the processing efficiency of the whole tasks of the data system.
An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by the processor, implements a method for configuring task execution in a workflow, including the steps of: acquiring a first task file of a first workflow; analyzing the first task file to obtain the dependency relationship of the first task in the first task file; establishing an associated operation relation between the first task and the second task according to the dependency relation; the second task is a task in a second workflow; acquiring running log information of the first task, and determining the running time of the first task according to the running log information; generating the optimized running time of the second task according to the running time of the first task and the associated running relation; and modifying the configuration information of the second task in the second workflow according to the optimized running time.
The computer readable storage medium provides a method for configuring task running in cross-workflow, which is used for reconfiguring the running time of each task in the cross-workflow, and comprises the steps of firstly obtaining a first task file of a first workflow, analyzing the first task file in order to distinguish a naming mode of different workflows, obtaining a dependency relationship of a first task in the first task file, and establishing an associated running relationship between the first task and a second task according to the dependency relationship, wherein the first workflow does not refer to a workflow formed by a task unit and does not refer to one workflow; the second task is a task in a second workflow, after the dependency relationship in one workflow is obtained through analysis, the second task is compared with the dependency relationship in the other workflow, so that the dependency relationship between the first task and the second task is established according to the dependency relationship, the dependency relationship between different tasks across workflows is defined as an associated operation relationship, the operation log information of the first task is obtained, the operation time of the first task is determined according to the operation log information, because the operation of the second task is influenced by the operation result of the first task, the optimized operation time of the second task is generated according to the operation time of the first task and the associated operation relationship, the configuration information of the second task in the second workflow is modified according to the optimized operation time, and the dependency relationship between different workflows across workflows is analyzed, and then determining the optimized running time of each work task according to the dependency relationship and the running time of the work tasks, thereby reducing the invalid waiting time of a large number of tasks in a cross-workflow and reducing the whole waiting time of the whole data system, thereby reducing the occupation of queue resources and improving the processing efficiency of the whole tasks of the data system.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above may be implemented by hardware instructions of a computer program, which may be stored in a non-volatile computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium provided herein and used in the examples may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double-rate SDRAM (SSRSDRAM), Enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all the equivalent structures or equivalent processes that can be directly or indirectly applied to other related technical fields by using the contents of the specification and the drawings of the present application are also included in the scope of the present application.
Claims (10)
1. A method for configuring task execution in a workflow, the method comprising:
acquiring a first task file of a first workflow;
analyzing the first task file to obtain the dependency relationship of the first task in the first task file;
establishing an associated operation relation between the first task and the second task according to the dependency relation; the second task is a task in a second workflow;
acquiring running log information of the first task, and determining the running time of the first task according to the running log information;
generating the optimized running time of the second task according to the running time of the first task and the associated running relation;
and modifying the configuration information of the second task in the second workflow according to the optimized running time.
2. The method according to claim 1, wherein the parsing the first task file to obtain the dependency relationship of the first task in the first task file comprises:
analyzing the first task file to obtain a directed acyclic graph in the first task file;
and identifying the connection relation of each first task in the directed acyclic graph, and determining the dependency relation of the first task according to the connection relation.
3. The method for configuring task execution in workflow according to claim 1, wherein said establishing an associated execution relationship between the first task and the second task according to the dependency relationship comprises:
acquiring a second task file of a second workflow;
analyzing the second task file to obtain a second task which has a dependency relationship with the first task in the second task file;
and establishing the association operation relationship between the first task and the second task according to the dependency relationship.
4. The method according to claim 3, wherein the parsing the second task file to obtain a second task in the second task file that has a dependency relationship with the first task includes:
analyzing the second task file to obtain a second task in the second task file and a preposed task of the second task;
and if the preposed task is a first task in a first workflow, determining that the second task has a dependency relationship with the second task so as to obtain the second task which has a dependency relationship with the first task in the second task file.
5. The method for configuring task running in workflow according to claim 1, wherein the generating an optimized running time of the second task according to the running time of the first task and the associated running relationship comprises:
acquiring the running time of a plurality of first tasks according to the association running relationship;
comparing the running time of the plurality of first tasks, and determining the maximum waiting time of the second task;
acquiring the starting time of the second task;
and subtracting the maximum waiting time according to the starting time to obtain the optimized running time of the second task.
6. The method for configuring task running in workflow according to claim 1, wherein the obtaining running log information of the first task and determining the running time of the first task according to the running log information comprises:
acquiring running log information of the first task;
identifying the running log information, and determining a plurality of items of information of the running log information;
identifying the project information, and determining a first task corresponding to the project information;
and identifying the running data of the first task, and acquiring the running time of the first task.
7. The method for configuring task execution in workflow according to claim 6, wherein said identifying said execution log information and determining a number of items of said execution log information comprises:
identifying the running log information according to a preset log template, and segmenting the running log information to obtain a plurality of pieces of area information;
and identifying the area information according to preset keywords, and determining a plurality of item information of the running log information.
8. An apparatus for configuring task execution in a workflow, the apparatus comprising:
the file acquisition module is used for acquiring a first task file of a first workflow;
the relationship analysis module is used for analyzing the first task file and acquiring the dependency relationship of the first task in the first task file;
the relationship determination module is used for establishing the associated operation relationship between the first task and the second task according to the dependency relationship; the second task is a task in a second workflow;
the log analysis module is used for acquiring running log information of the first task and determining the running time of the first task according to the running log information;
the time optimization module is used for generating the optimized running time of the second task according to the running time of the first task and the associated running relation;
and the time configuration module is used for modifying the configuration information of the second task in the second workflow according to the optimized running time.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor when executing the computer program implements a method of configuring the operation of a task in a workflow according to any one of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a method for configuring the operation of tasks in a workflow according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210441111.0A CN114780174A (en) | 2022-04-25 | 2022-04-25 | Configuration method and device for task operation in workflow, computer equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210441111.0A CN114780174A (en) | 2022-04-25 | 2022-04-25 | Configuration method and device for task operation in workflow, computer equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114780174A true CN114780174A (en) | 2022-07-22 |
Family
ID=82433671
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210441111.0A Pending CN114780174A (en) | 2022-04-25 | 2022-04-25 | Configuration method and device for task operation in workflow, computer equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114780174A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130132940A1 (en) * | 2011-11-17 | 2013-05-23 | Andreas Meier | Adaptation of data flow of one or multiple systems within a system landscape |
| CN103824162A (en) * | 2014-02-28 | 2014-05-28 | 北京航空航天大学 | Reliability and performance integrated flexible workflow implementing method based on instruction chain |
| US20150134386A1 (en) * | 2013-11-08 | 2015-05-14 | Prachi Prasad Jogalekar | Collectively optimizing group schedules to minimize project completion time and cost |
| CN107688488A (en) * | 2016-08-03 | 2018-02-13 | 中国移动通信集团湖北有限公司 | A kind of optimization method and device of the task scheduling based on metadata |
| CN108628665A (en) * | 2018-05-16 | 2018-10-09 | 天津科技大学 | Task scheduling based on data-intensive scientific workflow and virtual machine integration method |
| CN112905307A (en) * | 2021-04-22 | 2021-06-04 | 北京理工大学 | Multi-target cloud workflow scheduling method based on lean and rich optimization algorithm |
| US20210390486A1 (en) * | 2020-06-03 | 2021-12-16 | Citrix Systems, Inc. | Prioritizing sequential application tasks |
| CN113811852A (en) * | 2019-05-14 | 2021-12-17 | 微软技术许可有限责任公司 | Automatic resolution of dependency version conflicts |
| CN114356530A (en) * | 2022-01-12 | 2022-04-15 | 平安普惠企业管理有限公司 | Task scheduling method and device, electronic equipment and storage medium |
-
2022
- 2022-04-25 CN CN202210441111.0A patent/CN114780174A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130132940A1 (en) * | 2011-11-17 | 2013-05-23 | Andreas Meier | Adaptation of data flow of one or multiple systems within a system landscape |
| US20150134386A1 (en) * | 2013-11-08 | 2015-05-14 | Prachi Prasad Jogalekar | Collectively optimizing group schedules to minimize project completion time and cost |
| CN103824162A (en) * | 2014-02-28 | 2014-05-28 | 北京航空航天大学 | Reliability and performance integrated flexible workflow implementing method based on instruction chain |
| CN107688488A (en) * | 2016-08-03 | 2018-02-13 | 中国移动通信集团湖北有限公司 | A kind of optimization method and device of the task scheduling based on metadata |
| CN108628665A (en) * | 2018-05-16 | 2018-10-09 | 天津科技大学 | Task scheduling based on data-intensive scientific workflow and virtual machine integration method |
| CN113811852A (en) * | 2019-05-14 | 2021-12-17 | 微软技术许可有限责任公司 | Automatic resolution of dependency version conflicts |
| US20210390486A1 (en) * | 2020-06-03 | 2021-12-16 | Citrix Systems, Inc. | Prioritizing sequential application tasks |
| CN112905307A (en) * | 2021-04-22 | 2021-06-04 | 北京理工大学 | Multi-target cloud workflow scheduling method based on lean and rich optimization algorithm |
| CN114356530A (en) * | 2022-01-12 | 2022-04-15 | 平安普惠企业管理有限公司 | Task scheduling method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109543925B (en) | Risk prediction method and device based on machine learning, computer equipment and storage medium | |
| CN110297656B (en) | Method and device for evaluating codes based on configuration model and computer equipment | |
| CN111008188A (en) | Data migration method and device, computer equipment and storage medium | |
| CN113961332A (en) | Method and device for realizing workflow engine, electronic equipment and storage medium | |
| CN114091688B (en) | Computing resource obtaining method and device, electronic equipment and storage medium | |
| CN115098600A (en) | Directed acyclic graph construction method and device for data warehouse and computer equipment | |
| CN112699040A (en) | Pressure testing method, device, equipment and computer readable storage medium | |
| CN112948504B (en) | Data acquisition method and device, computer equipment and storage medium | |
| CN114780174A (en) | Configuration method and device for task operation in workflow, computer equipment and medium | |
| CN113255742A (en) | Policy matching degree calculation method and system, computer equipment and storage medium | |
| CN110084476B (en) | Case adjustment method, device, computer equipment and storage medium | |
| CN110727436A (en) | Operation interface script execution method and device, terminal equipment and storage medium | |
| CN111061637A (en) | Interface test method, interface test device and storage medium | |
| CN115640278A (en) | Method and system for intelligently optimizing database performance | |
| CN113946363A (en) | Method and device for executing and configuring service data, computer equipment and storage medium | |
| CN110378481B (en) | Decision processing method and device based on rough set, computer and storage medium | |
| CN111984253A (en) | Method and device for adding programming role based on graphical programming tool | |
| CN112732819A (en) | ETL-based data processing method, device, equipment and storage medium | |
| CN111027319A (en) | Method and device for analyzing natural language time words and computer equipment | |
| CN111291889A (en) | Knowledge base construction method and device | |
| CN112925997B (en) | Task dependency graph display method suitable for Azkaban and related equipment | |
| CN118093346A (en) | Task mining method, device, equipment, storage medium and product | |
| CN113641911B (en) | Advertisement interception rule base establishing method, device, equipment and storage medium | |
| CN117827316A (en) | Method and device for generating extensible markup language configuration file and electronic equipment | |
| CN117369857A (en) | Multi-task association query method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |