CN113377348B - Task adjusting method applied to task engine, related device and storage medium - Google Patents
Task adjusting method applied to task engine, related device and storage medium Download PDFInfo
- Publication number
- CN113377348B CN113377348B CN202110651201.8A CN202110651201A CN113377348B CN 113377348 B CN113377348 B CN 113377348B CN 202110651201 A CN202110651201 A CN 202110651201A CN 113377348 B CN113377348 B CN 113377348B
- Authority
- CN
- China
- Prior art keywords
- task
- operator
- directed acyclic
- acyclic graph
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 238000012545 processing Methods 0.000 claims abstract description 115
- 238000004590 computer program Methods 0.000 claims description 20
- 230000006378 damage Effects 0.000 claims description 17
- 230000001419 dependent effect Effects 0.000 claims description 17
- 230000006870 function Effects 0.000 claims description 10
- 238000012217 deletion Methods 0.000 claims description 9
- 230000037430 deletion Effects 0.000 claims description 9
- 238000011161 development Methods 0.000 abstract description 7
- 230000008569 process Effects 0.000 description 21
- 238000010586 diagram Methods 0.000 description 7
- 238000012544 monitoring process Methods 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011423 initialization method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
- G06F8/24—Object-oriented
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The application is applicable to the technical field of blockchain/information, and particularly relates to a task adjusting method, a related device and a storage medium applied to a task engine. The task adjusting method comprises the following steps: receiving an input task adding request, wherein the task adding request comprises an indication of the following information: processing objects and processing logic of tasks to be created; determining the dependency relationship between the task to be created and each task associated with the current directed acyclic graph based on the processing object; determining a target operator based on the processing logic, wherein the target operator has a function of executing the processing logic; if the target operator is successfully determined, loading a dynamic library of the target operator and instantiating the target operator; the current directed acyclic graph is updated based on the instantiated target operator and the dependency relationship creation such that the task engine performs corresponding task processing based on the updated directed acyclic graph. The technical scheme provided by the application can effectively relieve the development task amount of the developer.
Description
Technical Field
The application belongs to the technical field of blockchain/information, and particularly relates to a task adjusting method, a related device and a storage medium applied to a task engine.
Background
With the development of information technology, digital services are promoted and applied. The digitalized business is mainly realized by abstract business flow design, and different business flows usually correspond to different business flow logics.
In the prior art, corresponding task engines are designed and developed based on tasks corresponding to the business process logic and dependency relations among the tasks so as to realize digital business process processing based on the task engines. When the business flow logic changes each time, a developer is required to re-develop a new task engine aiming at the new business flow logic, and the development task amount is large.
Disclosure of Invention
The embodiment of the application provides a task adjusting method, a related device and a storage medium applied to a task engine, which can relieve the development task amount of developers.
The first aspect of the present application provides a task adjustment method applied to a task engine, where the task engine is configured based on a directed acyclic graph, each node in the directed acyclic graph is associated with a task, processing logic of each task is implemented by a corresponding operator, and each edge in the directed acyclic graph is used to describe a dependency relationship between related tasks, where the task adjustment method includes: comprising the following steps:
receiving an input task adding request, wherein the task adding request comprises an instruction for the following information: processing objects and processing logic of tasks to be created;
determining the dependency relationship between the task to be created and each task associated with the current directed acyclic graph based on the processing object;
determining a target operator based on the processing logic, wherein the target operator has a function of executing the processing logic;
when the target operator is successfully determined, loading a dynamic library of the target operator and instantiating the target operator;
Updating the current directed acyclic graph based on the instantiated target operator and the dependency relationship so that the task engine performs corresponding task processing based on the updated directed acyclic graph.
Based on the first aspect, in a first possible implementation manner, the determining the target operator based on the processing logic specifically includes: based on the processing logic, an operator with the least resource occupancy rate is determined as a target operator from a plurality of operators executing the processing logic.
In a second possible implementation manner, according to the first possible implementation manner of the first aspect, the determining, as the target operator, the operator with the minimum resource occupancy rate from a plurality of operators that are configured to execute the processing logic includes:
determining the resource availability of the equipment where the task engine is located;
if the operator with the resource occupancy rate not greater than the resource availability rate exists in the operators with the execution of the processing logic, determining the operator with the minimum resource occupancy rate from the operators with the execution of the processing logic as a target operator;
And if no operator with the resource occupancy rate not greater than the resource availability rate exists in the plurality of operators with the execution logic, outputting alarm information, wherein the alarm information is used for indicating that the target operator cannot be successfully determined currently.
Based on the first aspect, the first possible implementation manner of the first aspect, or the second possible implementation manner of the first aspect, in a third possible implementation manner, the created node is provided with a corresponding destruction condition;
the creating node updates the directed acyclic graph, and then further comprises:
After the task corresponding to the created node is executed, judging whether the created node meets the destruction condition or not;
and if the current created node meets the destruction condition, deleting the created node in the current directed acyclic graph and releasing the resources occupied by the target operator.
Based on the first possible implementation manner of the first aspect, or the second possible implementation manner of the first aspect, in a fourth possible implementation manner, the task adjusting method further includes:
Receiving an input task deletion request, wherein the task deletion request comprises indication information of a task to be deleted;
And deleting the nodes associated with the tasks to be deleted in the current directed acyclic graph based on the indication information, and releasing the resources occupied by operators corresponding to the tasks to be deleted.
Based on the first aspect, the first possible implementation manner of the first aspect, or the second possible implementation manner of the first aspect, in a fifth possible implementation manner, the directed acyclic graph includes: an output node associated with a multi-output task, the output node depending on more than two nodes and having no dependent nodes, the multi-output task configured to output a processing result of the dependent nodes;
The task adding request further includes: output indication information, the output indication information is used for indicating whether the task to be created is dependent on the multi-output task;
The determining the dependency relationship between the task to be created and each task associated with the directed acyclic graph based on the processing object is as follows: and determining the dependency relationship between the task to be created and each task associated with the directed acyclic graph based on the processing object and the output indication information.
Based on the first aspect, the first possible implementation manner of the first aspect, or the second possible implementation manner of the first aspect, in a sixth possible implementation manner, the creating the updated current directed acyclic graph based on the instantiated target operator and the dependency further includes: uploading the updated directed acyclic graph to the blockchain.
A second aspect of the present application provides a task adjusting device applied to a task engine, where the task engine is configured based on a directed acyclic graph, each node in the directed acyclic graph is associated with a task, processing logic of each task is implemented by a corresponding operator, and each edge in the directed acyclic graph is used to describe a dependency relationship between related tasks, the task adjusting device includes:
the receiving module is used for receiving an input task adding request, wherein the task adding request comprises an instruction for the following information: processing objects and processing logic of tasks to be created;
The relation determining module is used for determining the dependency relation between the task to be created and each task associated with the current directed acyclic graph based on the processing object;
An operator determining module, configured to determine a target operator based on the processing logic, where the target operator has a function of executing the processing logic;
an operator processing module, configured to load a dynamic library of the target operator and instantiate the target operator when the operator determining module successfully determines the target operator;
And the updating module is used for updating the current directed acyclic graph based on the instantiated target operator and the dependency relationship so that the task engine can execute corresponding task processing based on the updated directed acyclic graph.
In a first possible implementation manner, the operator determining module is specifically configured to: based on the processing logic, an operator with the least resource occupancy rate is determined as a target operator from a plurality of operators executing the processing logic.
The third aspect of the present application provides a task adjusting device applied to a task engine, where the task engine is configured based on a directed acyclic graph, each node in the directed acyclic graph is associated with a task, processing logic of each task is implemented by a corresponding operator, and each edge in the directed acyclic graph is used to describe a dependency relationship between related tasks; the task adjusting device includes: a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the task adjustment method according to any one of the above first aspects when executing the computer program.
A fourth aspect of the present application provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the task tuning method as described above in the first aspect.
A fifth aspect of the application provides a computer program product for, when run on an electronic device, causing the electronic device to perform the task adjustment method of any of the first aspects described above.
It will be appreciated that the advantages of the second to fifth aspects may be found in the relevant description of the first aspect, and are not described here again.
Compared with the prior art, the application has the beneficial effects that: in a task engine configured based on the directed acyclic graph, the creation of a task and the update of the directed acyclic graph can be triggered by a task newly-added request, on one hand, when processing logic needs to be newly added in original business process logic, the task in the task engine can be adjusted by inputting the task newly-added request, a developer is not required to re-develop the task engine aiming at the changed business process logic, and the development task quantity of the developer can be relieved to a certain extent; on the other hand, based on the scheme of the application, the light weight of the task engine is realized, the light weight task engine can be developed in the early stage, and the task engine is subjected to task adjustment after the follow-up requirement.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a DAG in a scenario provided by an embodiment of the present application;
FIG. 2 is a schematic diagram of a task engine according to an embodiment of the present application;
FIG. 3 is a flow chart of a task adjustment method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a task processing flow based on a DAG according to an embodiment of the present application;
FIG. 5 is a flowchart illustrating another task adjustment method according to an embodiment of the present application;
FIG. 6 is a flowchart illustrating a task adjustment method according to an embodiment of the present application;
Fig. 7 is a schematic structural diagram of a task adjusting device according to an embodiment of the present application;
Fig. 8 is a schematic structural diagram of another task adjusting device according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.
The embodiment provides a task adjusting method, a related device and a storage medium applied to a task engine, wherein the task engine is configured based on a directed acyclic graph (DAG, directed Acyclic Graph), each node in the directed acyclic graph is associated with a task, processing logic of each task is realized by a corresponding operator, and each edge in the directed acyclic graph is used for describing a dependency relationship between related tasks.
As described in more detail below, a DAG is a loop-free directed graph that may be used to implement dependency management, such as: before a certain task (e.g., task 1) is performed, another task (e.g., task 2) needs to be completed, because the input to task 1 is the output of task 2. When there are multiple similar dependencies, the need for dependency management arises. A schematic diagram of the DAG may be shown with reference to fig. 1, where each task may be described as a node in the DAG, such as node 1, node 2, node 3, node 4, and node 5 in fig. 1 are respectively associated with different tasks, and processing logic of each task is implemented by a corresponding operator, and in fig. 1, a dependency relationship between different tasks may be described as an edge in the DAG, such as edge a in fig. 1, to describe a dependency of a task associated with node 2 on a task associated with node 1, that is, node 2 depends on node 1, and node 1 depends on node 2. In this embodiment, the DAG may be configured by using file configuration information, and specifically, the file configuration information may include: a runtime environment (e.g., linux, windows or ARM), input information, output information, operator information, and the like. Wherein the operator information may include: operator name, operator type, operator id, operator initialization information, run dependencies between operators, etc.
The task engine mentioned in this embodiment is configured based on the DAG to enable scheduling and execution of tasks by the DAG. Specifically, in an application scenario, the framework of the task engine may be as shown in fig. 2, and includes: the task graph parser and task graph run engine.
The task graph analyzer is configured to read and analyze the DAG (e.g., read and analyze the file configuration information) to obtain an operator list related to the DAG and a dependency relationship between nodes, dynamically load an operator dynamic library according to the operator list, instantiate a corresponding operator, and generate an input and output parameter and operation configuration information of the operator based on the instantiated operator and the dependency relationship configuration between the nodes.
The task graph execution engine is configured to schedule and execute tasks based on the DAG, and specifically includes: operator households and task households, the operator households to manage instantiated operators, the task households to implement scheduling of tasks, the operator households are configured with operator interfaces to be compatible with different types of operators, as shown in fig. 2, the operator interfaces can be compatible with PYtorch operators (i.e., operators implemented based on pytorch framework), TF operators (i.e., operators implemented based on tensorflow framework), caffe operators (i.e., operators implemented based on Caffe framework), ONNX operators (i.e., operators implemented based on ONNX framework), and other general operators.
The task engine described above may be implemented in the pure c++ language to facilitate minimizing code size. Further, the task manager may perform task scheduling based on the ring buffer.
In the following, a task adjustment method applied to a task engine is described in an embodiment, and in particular, a framework of the task engine may refer to the foregoing description, as shown in fig. 3, the task adjustment method includes:
step 301, receiving an input task adding request;
in this embodiment, the task addition request includes an instruction for: processing objects and processing logic for tasks to be created.
In practical application, when a task needs to be newly added on the basis of the existing task engine, the task adding request can be input through a cmd instruction or a pre-configured visual interface to trigger the execution of the flow shown in fig. 3. The processing object of the task to be created may indicate an input source of the task to be created, and the processing logic refers to a specific operation algorithm. For example, if on the basis of fig. 1, a node is to be created to perform anomaly monitoring on the output condition of the node 2, the processing object of the task to be created may be configured as the output data of the node 2, and the processing logic is a corresponding anomaly monitoring algorithm.
Step 302, determining the dependency relationship between the task to be created and each task associated with the current directed acyclic graph based on the processing object;
Since the processing object may indicate input information of the task to be created, in step 302, a dependency relationship between the task to be created and each task associated with the current directed acyclic graph may be determined based on the processing object.
Further to the foregoing example, when the processing object of the task to be created is the output data of the node 2, it may be determined that the task to be created depends on the task associated with the node 2.
Optionally, the task engine of the present embodiment also supports multiple outputs. Specifically, the DAG may include: and an output node associated with a multi-output task, the output node being dependent on two or more nodes and there being no dependent node, the multi-output task being configured to output a processing result of the dependent node. The task add request mentioned in step 301 further includes: and outputting indication information, wherein the output indication information is used for indicating whether the task to be created is depended on by the multi-output task. In step 302, a dependency relationship between the task to be created and each task associated with the directed acyclic graph may be determined based on the processing object and the output instruction information.
Step 303, determining a target operator based on the processing logic;
Wherein the target operator has a function of executing the processing logic. In step 303, knowing the processing logic of the task to be created, the matching operator can be looked up in a local dictionary or in a remote operator library as the target operator. Specifically, operators matched with the processing logic can be preferentially searched in the global dictionary, and when the matched operators cannot be searched, operators matched with the processing logic are searched in a remote operator library, so that a target operator is finally determined.
It should be appreciated that in step 303, there are cases where the target operator can be successfully determined and the target operator cannot be successfully determined, and when the target operator is successfully determined, step 304 and the following steps may be performed. When the target algorithm cannot be successfully targeted, step 304 and subsequent steps are not performed.
Since the resources occupied by the operators of different types may be different, in order to further improve the resource utilization, optionally, in step 303, when there are a plurality of operators with executing the processing logic, an operator with the minimum resource occupancy may be determined as a target operator from the plurality of operators with executing the processing logic. That is, step 303 may be specifically expressed as: based on the processing logic, an operator with the least resource occupancy rate is determined as a target operator from a plurality of operators executing the processing logic. Specifically, the resource occupancy rate of the operator may be calculated based on the resources required by the operator (e.g., the number of gpus, the number of cpus, the memory/video memory size, etc. required).
Furthermore, in order to improve the reliability of the system and avoid downtime caused by excessive consumption of resources, the method can also combine the resource availability of the equipment where the task engine is located to determine the target operator before determining the target operator. Specifically, the determining, as the target operator, the operator having the smallest resource occupancy rate from among the plurality of operators having the processing logic includes:
determining the resource availability of the equipment where the task engine is located;
if the operator with the resource occupancy rate not greater than the resource availability rate exists in the operators with the execution of the processing logic, determining the operator with the minimum resource occupancy rate from the operators with the execution of the processing logic as a target operator;
And if no operator with the resource occupancy rate not greater than the resource availability rate exists in the plurality of operators with the execution logic, outputting alarm information, wherein the alarm information is used for indicating that the target operator cannot be successfully determined currently.
Step 304, when the target operator is successfully determined, loading a dynamic library of the target operator and instantiating the target operator;
In this embodiment, when step 303 successfully determines the target operator, a dynamic library of the target operator may be loaded and the target operator may be instantiated.
Specifically, in this embodiment, a key may be maintained as an operator id through an operator list, a value creates a dictionary of functions for the operator, and a dynamic library implementation of each operator needs to be registered on the dictionary. Similar to REGISTER (NS, CLASS, OPID, OPVERSION), where the first parameter NS represents the operator name, the second parameter CLASS represents the operator type, the third parameter represents the operator id, and the fourth parameter OPVERSION represents the operator version. Each operator has a corresponding configuration parameter which can be set by a developer in advance according to the data processing requirement.
In this embodiment, when the above target operator is determined, a dynamic library corresponding to the target operator may be acquired and loaded, and then the target operator is instantiated to obtain a corresponding operator object. The specific process of instantiating the operator can be implemented with reference to the prior art, and will not be described in detail herein.
Step 305, updating the current directed acyclic graph based on the instantiated target operator and the dependency relationship;
In step 305, a corresponding task may be generated based on the instantiated target operator and the dependency relationship, and associated nodes are created, and the current directed acyclic graph is updated based on the created nodes, such that the task engine performs a corresponding task process based on the updated directed acyclic graph.
Specifically, the process of executing the corresponding task processing by the task engine based on the updated directed acyclic graph may refer to the process shown in fig. 4, including:
step 401, reading file configuration to obtain a DAG basic data structure;
step 402, loading a dynamic library according to the operator list;
step 403, instantiating an operator according to the operator list, and automatically calling an operator initialization method according to an initialization parameter of the operator;
step 404, creating a DAG according to the dependency relationship and setting operation parameters to start to operate the task engine;
step 405, obtaining a corresponding output result based on the task engine.
Specifically, the specific process from step 401 to step 404 may refer to the description of the foregoing embodiments, and will not be repeated here.
From the above, in the task engine configured based on the directed acyclic graph, the creation of the task and the update of the directed acyclic graph can be triggered by the task adding request, on one hand, when the processing logic needs to be added in the original business process logic, the task in the task engine can be adjusted by inputting the task adding request, the task engine does not need to be re-developed by a developer for the changed business process logic, and the development task amount of the developer can be relieved to a certain extent; on the other hand, based on the scheme of the application, the light weight of the task engine is realized, the light weight task engine can be developed in the early stage, and the task engine is subjected to task adjustment after the follow-up requirement.
Further, on the basis of the embodiment shown in fig. 3, the user may destroy unnecessary nodes to release resources and improve the resource utilization rate. Taking a specific application scenario as an example, for example, in order to capture the output situation of a certain task (for example, task a), the user triggers the process shown in fig. 3 to create a task (for example, task B) for monitoring the output situation of task a through the task adding request. When a problem is found according to the captured output condition, task B is no longer needed, and task B can be deleted at this time, so that resources occupied by the corresponding operator of task B are released (i.e., recycled), and overall performance is improved. Specifically, on the basis of the embodiment shown in fig. 3, as shown in fig. 5, the task adjustment method provided in another embodiment of the present application includes:
steps 501 to 505, specifically, steps 501 to 505 may refer to descriptions of steps 301 to 305 in the embodiment shown in fig. 3, and are not repeated herein;
step 506, after the task corresponding to the created node is executed, judging whether the created node meets the destruction condition;
In this embodiment, each created node may set a corresponding destruction condition, where the destruction condition may be, for example, that the number of operations of the corresponding node satisfies a threshold, that the operation result satisfies a condition, and so on.
In step 506, it may be determined whether the current created node satisfies the corresponding destruction condition after the task corresponding to the created node is executed, if the current created node satisfies the corresponding destruction condition, step 507 is entered, otherwise, step 506 may be returned, and the corresponding determination is performed after waiting for the task to be executed again.
Step 507, deleting the created node in the current directed acyclic graph and releasing the resources occupied by the target operator;
In step 507, the nodes satisfying the destruction condition (for convenience of reference, the deleted nodes will be described later) are deleted, and the resources occupied by the corresponding operators are released, so as to improve the overall performance. Further, if there is a node in the current DAG that depends on the deleted node, then the dependency between the two (i.e., the deleted node and the node that depends on the deleted node) is deleted. Further, if a node that depends on the deleted node depends only on the deleted node before, the node can be synchronously deleted and the resources occupied by the corresponding operator can be released.
The embodiment shown in fig. 5 is automatic destruction when the newly created node meets the destruction condition, and in other embodiments, the user may actively request to delete a certain task in the task engine, so that the task engine has higher flexibility. Specifically, as shown in fig. 6, the task adjustment method further includes:
Step 601, receiving an input task deletion request;
The task deleting request comprises indicating information of a task to be deleted.
In practical application, when a task needs to be deleted on the basis of an existing task engine, the task deletion request can be input through a cmd instruction or a pre-configured visual interface to trigger the execution of the flow shown in fig. 6.
Step 602, deleting the node associated with the task to be deleted in the current directed acyclic graph based on the indication information, and releasing the resource occupied by the operator corresponding to the deleted task;
In step 602, the node to be deleted is determined based on the indication information, the node associated with the task to be deleted is deleted in the current directed acyclic graph, and the resource occupied by the operator corresponding to the deleted task is released. Further, if there is a task that depends on the task to be deleted, the dependency relationship between the task to be deleted and the task that depends on the task to be deleted is deleted. Further, if the task depending on the task to be deleted depends only on the task to be deleted before, the node associated with the task can be deleted synchronously and the resources occupied by the corresponding operator can be released.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present application. The steps with the same content are not repeated to describe the working process.
Furthermore, the present embodiment further provides a task adjusting device applied to a task engine, where the task engine is configured based on a directed acyclic graph, each node in the directed acyclic graph is associated with a task, processing logic of each task is implemented by a corresponding operator, and each edge in the directed acyclic graph is used to describe a dependency relationship between related tasks, and specifically, the task adjusting device may be composed of software and/or hardware, and may be integrated in the task engine or may be a device independent of the task engine. As shown in fig. 7, the task adjusting device includes:
A receiving module 701, configured to receive an input task adding request, where the task adding request includes an instruction for: processing objects and processing logic of tasks to be created;
a relationship determining module 702, configured to determine, based on the processing object, a dependency relationship between the task to be created and each task associated with the current directed acyclic graph;
An operator determining module 703, configured to determine a target operator based on the processing logic, where the target operator has a function of executing the processing logic;
an operator processing module 704, configured to load a dynamic library of the target operator and instantiate the target operator when the operator determining module successfully determines the target operator;
and the updating module 705 is configured to update the current directed acyclic graph based on the instantiated target operator and the dependency relationship, so that the task engine performs corresponding task processing based on the updated directed acyclic graph.
Optionally, the operator determining module 704 is specifically configured to: based on the processing logic, an operator with the least resource occupancy rate is determined as a target operator from a plurality of operators executing the processing logic.
Further, the operator determining module 704 is specifically configured to: determining the resource availability of the equipment where the task engine is located; if the operator with the resource occupancy rate not greater than the resource availability rate exists in the operators with the execution of the processing logic, determining the operator with the minimum resource occupancy rate from the operators with the execution of the processing logic as a target operator; and if no operator with the resource occupancy rate not greater than the resource availability rate exists in the plurality of operators with the execution logic, outputting alarm information, wherein the alarm information is used for indicating that the target operator cannot be successfully determined currently.
Optionally, the created node is provided with a corresponding destruction condition. The update module 705 may also be used to: after the task corresponding to the created node is executed, judging whether the created node meets the destruction condition or not; and if the current created node meets the destruction condition, deleting the created node in the current directed acyclic graph and releasing the resources occupied by the target operator.
Optionally, the receiving module 701 may be further configured to receive an input task deletion request, where the task deletion request includes indication information of a task to be deleted. The update module 705 may also be used to: when the receiving module 701 receives the task deletion request, deleting the node associated with the task to be deleted in the current directed acyclic graph based on the indication information, and releasing the resource occupied by the operator corresponding to the deleted task.
Optionally, the directed acyclic graph includes: and an output node associated with a multi-output task, the output node being dependent on two or more nodes and there being no dependent node, the multi-output task being configured to output a processing result of the dependent node. The task adding request further includes: and outputting indication information, wherein the output indication information is used for indicating whether the task to be created is depended on by the multi-output task. The relationship determining module 702 is specifically configured to determine, based on the processing object and the output instruction information, a dependency relationship between the task to be created and each task associated with the directed acyclic graph.
From the above, in the task engine configured based on the directed acyclic graph, the creation of the task and the update of the directed acyclic graph can be triggered by the task adding request, on one hand, when the processing logic needs to be added in the original business process logic, the task in the task engine can be adjusted by inputting the task adding request, the task engine does not need to be re-developed by a developer for the changed business process logic, and the development task amount of the developer can be relieved to a certain extent; on the other hand, based on the scheme of the application, the light weight of the task engine is realized, the light weight task engine can be developed in the early stage, and the task engine is subjected to task adjustment after the follow-up requirement.
It should be noted that, in the specific execution process of the above modules, information interaction between modules, and the like, because the specific functions and the technical effects brought by the specific execution process and the information interaction between modules are based on the same concept as those of the foregoing method embodiment, specific reference may be made to the method embodiment section, and details are not repeated herein.
Fig. 8 is a schematic structural diagram of a task adjusting device according to another embodiment of the present application. As shown in fig. 8, this embodiment further provides a task adjusting device, including: the system comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor realizes the task adjustment method according to the embodiment of the method when executing the computer program. Wherein the memory and the processor are interconnected by a bus.
The Processor may be a central processing unit (Central Processing Unit, CPU), other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may be an internal storage unit of the task orchestration device, such as a hard disk or memory, in some embodiments, or an external storage device of the task orchestration device, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD), etc. Further, the memory may also include both internal storage units and external storage devices. The memory is used to store an operating system, application programs, boot loader (BootLoader), data, and other programs, etc., such as program code for a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
Embodiments of the present application also provide a computer readable storage medium storing a computer program which, when executed by a processor, implements steps for implementing the various method embodiments described above.
Embodiments of the present application provide a computer program product that when run on an electronic device (e.g., a cell phone, tablet computer, notebook, etc.), causes the electronic device to perform steps that may be performed in the various method embodiments described above.
In all embodiments of the present application, the updated DAG may be further uploaded into the blockchain, which may ensure its security and fair transparency to the user. The user device may download the DAG from the blockchain to verify that the DAG has been tampered with. The blockchain referred to in this example is a novel mode of application for computer technology such as distributed data storage, point-to-point transmission, consensus mechanisms, encryption algorithms, and the like. The blockchain (Blockchain), essentially a de-centralized database, is a string of data blocks that are generated in association using cryptographic methods, each of which contains information from a batch of network transactions for verifying the validity (anti-counterfeit) of its information and generating the next block. The blockchain may include a blockchain underlying platform, a platform product services layer, an application services layer, and the like.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiments, and may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing device/terminal apparatus, recording medium, computer Memory, read-Only Memory (ROM), random access Memory (RAM, random Access Memory), electrical carrier signals, telecommunications signals, and software distribution media. Such as a U-disk, removable hard disk, magnetic or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.
Claims (7)
1. A task adjustment method applied to a task engine, wherein the task engine is configured based on a directed acyclic graph, each node in the directed acyclic graph is associated with a task, processing logic of each task is realized by a corresponding operator, each edge in the directed acyclic graph is used for describing a dependency relationship between related tasks, the task engine comprises a task graph parser and a task graph running engine, the task graph parser is configured to read and parse the directed acyclic graph, and the task graph running engine is configured to schedule and execute the tasks based on the directed acyclic graph; the task adjustment method comprises the following steps:
Receiving an input task adding request, wherein the task adding request comprises an indication of the following information: processing objects and processing logic of tasks to be created;
determining the dependency relationship between the task to be created and each task associated with the current directed acyclic graph based on the processing object;
determining a target operator based on the processing logic, wherein the target operator has a function of executing the processing logic;
when the target operator is successfully determined, loading a dynamic library of the target operator and instantiating the target operator;
Generating corresponding tasks based on the instantiated target operators and the dependency relationships, creating associated nodes, and updating the current directed acyclic graph based on the created nodes so that the task engine executes corresponding task processing based on the updated directed acyclic graph;
The determining a target operator based on the processing logic specifically comprises: determining an operator with the minimum resource occupancy rate from a plurality of operators with the execution of the processing logic as a target operator based on the processing logic;
the determining, from a plurality of operators having the processing logic, an operator with a minimum resource occupancy as a target operator includes:
determining the resource availability of the equipment where the task engine is located;
If the operator with the resource occupancy rate not larger than the resource availability rate exists in the operators with the execution of the processing logic, determining the operator with the minimum resource occupancy rate from the operators with the execution of the processing logic as a target operator;
If no operator with the resource occupancy rate not larger than the resource availability rate exists in the operators for executing the processing logic, outputting alarm information, wherein the alarm information is used for indicating that a target operator cannot be successfully determined at present;
The directed acyclic graph includes: an output node associated with a multi-output task, the output node being dependent on more than two nodes and there being no dependent node, the multi-output task to output a processing result of the dependent node;
The task adding request further includes: outputting indication information, wherein the output indication information is used for indicating whether the task to be created is dependent by the multi-output task;
The determining, based on the processing object, the dependency relationship between the task to be created and each task associated with the directed acyclic graph is: and determining the dependency relationship between the task to be created and each task associated with the directed acyclic graph based on the processing object and the output indication information.
2. The task tuning method as claimed in claim 1, wherein the created node is provided with corresponding destruction conditions;
the updating of the current directed acyclic graph based on the created node further comprises:
After the task corresponding to the created node is executed, judging whether the created node meets the destruction condition currently;
And if the current created node meets the destruction condition, deleting the created node in the current directed acyclic graph and releasing the resources occupied by the target operator.
3. The task adjustment method according to claim 1, characterized in that the task adjustment method further comprises:
Receiving an input task deletion request, wherein the task deletion request comprises indication information of a task to be deleted;
And deleting the node associated with the task to be deleted in the current directed acyclic graph based on the indication information, and releasing the resources occupied by the operators corresponding to the deleted task.
4. The task tuning method of claim 1, wherein the generating a corresponding task based on the instantiated target operator and the dependency relationship and creating an associated node, updating the current directed acyclic graph based on the created node further comprises:
uploading the updated directed acyclic graph to the blockchain.
5. A task adjustment device applied to a task engine, wherein the task engine is configured based on a directed acyclic graph, each node in the directed acyclic graph is associated with a task, processing logic of each task is realized by a corresponding operator, each edge in the directed acyclic graph is used for describing a dependency relationship between related tasks, the task engine comprises a task graph parser and a task graph running engine, the task graph parser is configured to read and parse the directed acyclic graph, and the task graph running engine is configured to schedule and execute the tasks based on the directed acyclic graph; the task adjusting device includes:
the receiving module is used for receiving an input task adding request, wherein the task adding request comprises an indication of the following information: processing objects and processing logic of tasks to be created;
The relation determining module is used for determining the dependency relation between the task to be created and each task associated with the current directed acyclic graph based on the processing object;
An operator determining module, configured to determine a target operator based on the processing logic, where the target operator has a function of executing the processing logic;
An operator processing module for loading a dynamic library of the target operator and instantiating the target operator when the operator determining module successfully determines the target operator;
the updating module is used for generating corresponding tasks based on the instantiated target operators and the dependency relationships, creating associated nodes, updating the current directed acyclic graph based on the created nodes, and enabling the task engine to execute corresponding task processing based on the updated directed acyclic graph;
The operator determining module is specifically configured to: determining the resource availability of the equipment where the task engine is located based on the processing logic; if the operator with the resource occupancy rate not larger than the resource availability rate exists in the operators with the execution of the processing logic, determining the operator with the minimum resource occupancy rate from the operators with the execution of the processing logic as a target operator; if no operator with the resource occupancy rate not larger than the resource availability rate exists in the operators for executing the processing logic, outputting alarm information, wherein the alarm information is used for indicating that a target operator cannot be successfully determined at present;
The directed acyclic graph includes: an output node associated with a multi-output task, the output node being dependent on more than two nodes and there being no dependent node, the multi-output task to output a processing result of the dependent node; the task adding request further includes: outputting indication information, wherein the output indication information is used for indicating whether the task to be created is dependent by the multi-output task; the relation determining module is specifically configured to: and determining the dependency relationship between the task to be created and each task associated with the directed acyclic graph based on the processing object and the output indication information.
6. A task adjustment device applied to a task engine, wherein the task engine is configured based on a directed acyclic graph, each node in the directed acyclic graph is associated with a task, processing logic of each task is realized by a corresponding operator, each edge in the directed acyclic graph is used for describing a dependency relationship between related tasks, the task engine comprises a task graph parser and a task graph running engine, the task graph parser is configured to read and parse the directed acyclic graph, and the task graph running engine is configured to schedule and execute the tasks based on the directed acyclic graph;
the task adjusting device includes: memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 4 when the computer program is executed.
7. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110651201.8A CN113377348B (en) | 2021-06-10 | 2021-06-10 | Task adjusting method applied to task engine, related device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110651201.8A CN113377348B (en) | 2021-06-10 | 2021-06-10 | Task adjusting method applied to task engine, related device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113377348A CN113377348A (en) | 2021-09-10 |
| CN113377348B true CN113377348B (en) | 2024-06-28 |
Family
ID=77573974
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110651201.8A Active CN113377348B (en) | 2021-06-10 | 2021-06-10 | Task adjusting method applied to task engine, related device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113377348B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114637492B (en) * | 2022-03-17 | 2024-09-03 | 湖南快乐阳光互动娱乐传媒有限公司 | Processing method and device of object-oriented business logic |
| CN114817241A (en) * | 2022-03-25 | 2022-07-29 | 北京邮电大学 | Task evaluation method and device based on dynamic expansion of calculation node of Flink engine |
| CN115378999B (en) * | 2022-10-26 | 2023-03-24 | 小米汽车科技有限公司 | Service capacity adjusting method and device |
| CN115987751B (en) * | 2023-03-20 | 2023-06-13 | 阿里云计算有限公司 | Cloud network architecture and method for maintaining entity objects with dependencies |
| CN116088968A (en) * | 2023-04-10 | 2023-05-09 | 北京渲光科技有限公司 | Graphic engine management method, device, terminal and storage medium |
| CN116107665B (en) * | 2023-04-10 | 2023-10-27 | 北京渲光科技有限公司 | Project configuration method, device, terminal and storage medium |
| CN116721007B (en) * | 2023-08-02 | 2023-10-27 | 摩尔线程智能科技(北京)有限责任公司 | Task control method, system and device, electronic equipment and storage medium |
| CN117539630A (en) * | 2023-11-20 | 2024-02-09 | 上海直真君智科技有限公司 | Cloud heterogeneous joint calculation method and system based on DAG extension mechanism |
| CN117610320B (en) * | 2024-01-23 | 2024-04-02 | 中国人民解放军国防科技大学 | Directed acyclic graph workflow engine cyclic scheduling method, device and equipment |
| CN117950764B (en) * | 2024-01-31 | 2024-09-17 | 中国兵器工业计算机应用技术研究所 | Task breakpoint re-running method and system based on intermediate data |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110888720A (en) * | 2019-10-08 | 2020-03-17 | 北京百度网讯科技有限公司 | Task processing method and device, computer equipment and storage medium |
| CN111831424A (en) * | 2019-04-17 | 2020-10-27 | 杭州海康威视数字技术股份有限公司 | Task processing method, system and device |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103257900B (en) * | 2013-05-24 | 2016-05-18 | 杭州电子科技大学 | Real-time task collection method for obligating resource on the multiprocessor that minimizing CPU takies |
| CN104239135A (en) * | 2014-05-19 | 2014-12-24 | 江苏科技大学 | Hybrid task scheduling method of directed acyclic graph (DGA) based reconfigurable system |
| US10437600B1 (en) * | 2017-05-02 | 2019-10-08 | Ambarella, Inc. | Memory hierarchy to transfer vector data for operators of a directed acyclic graph |
| CN112069204B (en) * | 2020-09-30 | 2024-10-29 | 北京百度网讯科技有限公司 | Processing method and device for operator service, intelligent workstation and electronic equipment |
| CN112363819B (en) * | 2020-12-02 | 2024-08-09 | 深圳市房多多网络科技有限公司 | Big data task dynamic arrangement scheduling method and device and computing equipment |
| CN112685154A (en) * | 2020-12-25 | 2021-04-20 | 北京有竹居网络技术有限公司 | Data processing method of task flow engine, device and medium |
-
2021
- 2021-06-10 CN CN202110651201.8A patent/CN113377348B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111831424A (en) * | 2019-04-17 | 2020-10-27 | 杭州海康威视数字技术股份有限公司 | Task processing method, system and device |
| CN110888720A (en) * | 2019-10-08 | 2020-03-17 | 北京百度网讯科技有限公司 | Task processing method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113377348A (en) | 2021-09-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113377348B (en) | Task adjusting method applied to task engine, related device and storage medium | |
| US9135045B2 (en) | Generating user-requested virtual machine templates from super virtual machine templates and cacheable patches | |
| WO2021022713A1 (en) | Distributed module update method, device, and storage medium | |
| CN111427596B (en) | Software upgrading method and device and terminal equipment | |
| CN109472540B (en) | Service processing method and device | |
| CN113835713B (en) | Source code packet downloading method, device, computer equipment and storage medium | |
| US20170286095A1 (en) | Software discovery using exclusion | |
| CN112529711B (en) | Transaction processing method and device based on block chain virtual machine multiplexing | |
| CN111988429A (en) | Algorithm scheduling method and system | |
| CN109683992B (en) | Method and apparatus for managing objects | |
| CN115357663A (en) | Data synchronization method, system and device based on incremental data synchronization component | |
| CN111338716A (en) | Data processing method and device based on rule engine and terminal equipment | |
| US20240086176A1 (en) | Continuous deployment and orchestration of feature processing units in a managed cloud and a provider network for consistent data processing | |
| CN114281653A (en) | Application program monitoring method and device and computing equipment | |
| CN112199200B (en) | Resource scheduling method and device, computer equipment and storage medium | |
| CN112947907B (en) | Method for creating code branches | |
| US11347533B2 (en) | Enhanced virtual machine image management system | |
| CN110968339A (en) | Method and device for front-end construction tool and electronic equipment | |
| CN108089895B (en) | Activity registration method and device in plug-in, electronic equipment and storage medium | |
| CN112068895B (en) | Code configuration method, device, video playing equipment and storage medium | |
| CN114816478A (en) | Offline resource packet difference method, device, equipment and medium | |
| CN115277857A (en) | Method and device for interface verification, electronic equipment and storage medium | |
| EP4194067A1 (en) | Game engine resource processing method and apparatus, and electronic device and computer-readable storage medium | |
| CN113934453B (en) | Risk detection method, risk detection device and storage medium | |
| CN114942801B (en) | FSM-based application release task processing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |