CN113094164A

CN113094164A - Task processing device and method, computer equipment and storage medium

Info

Publication number: CN113094164A
Application number: CN202110457261.6A
Authority: CN
Inventors: 朱雁博; 张行程; 顾乡
Original assignee: Shanghai Sensetime Technology Development Co Ltd
Current assignee: Shanghai Sensetime Technology Development Co Ltd
Priority date: 2021-04-27
Filing date: 2021-04-27
Publication date: 2021-07-09
Also published as: WO2022227447A1

Abstract

The present disclosure provides a task processing device, a method, a computer apparatus, and a storage medium, wherein the task processing device includes: a scheduler, and a target executor; the scheduler is used for analyzing the task flow to be executed to obtain task information of task nodes in the task flow and determining a target actuator corresponding to the task nodes; at least one task node; and the target executor is used for acquiring the target component of the corresponding task node based on the task information of the corresponding task node and executing the task of the corresponding task node based on the target component and the task information. The task processing device provided by the embodiment of the disclosure avoids the task of switching execution environments when different task loops are executed, and improves the efficiency of task processing.

Description

Task processing device and method, computer equipment and storage medium

Technical Field

The present disclosure relates to the technical field of computer science, and in particular, to a task processing apparatus, a task processing method, a computer device, and a storage medium.

Background

At present, Deep Neural Networks (DNNs) are widely used as basic models in the field of machine learning in many fields such as computer vision and natural language processing. The deep neural network is applied in a certain field, and needs to be deployed to a corresponding execution environment through a plurality of task links; the current process of deploying neural networks has the problem of being inefficient.

Disclosure of Invention

The embodiment of the disclosure at least provides a task processing device, a task processing method, computer equipment and a storage medium.

In a first aspect, an embodiment of the present disclosure provides a task processing device, including: a scheduler, and a target executor; the scheduler is used for analyzing a task flow to be executed to obtain task information of task nodes in the task flow and determining a target actuator corresponding to the task nodes; the number of the task nodes is at least one; the target executor is used for acquiring a target component of a corresponding task node based on task information of the corresponding task node and executing a task of the corresponding task node based on the target component and the task information.

Therefore, the corresponding target executors are flexibly determined for the task nodes through the scheduling of the scheduler, the target executors corresponding to the task nodes can automatically call the target assemblies required by the corresponding task nodes to process, the tasks corresponding to the task nodes are executed, and the deployment environment of the executors can be flexibly set according to the actual application requirements, so that the work of executing environment switching during the execution of different task loops is avoided, and the task processing efficiency is improved.

In an optional embodiment, the task information of the corresponding task node includes at least one of: the component information, parameters, execution entries, and concurrency of the corresponding task nodes.

In an optional embodiment, when determining the target executor corresponding to the task node, the scheduler is configured to: and determining a target executor corresponding to the task node based on the topological structure of the task flow and the task information of the task node.

In this way, when the target executor is determined by using the topology of the task flow and the task information of the task node, the execution order of the plurality of executors can be determined at an early stage, and the time for searching the executor corresponding to the task node in the actual task processing can be reduced, thereby improving the efficiency of the task processing.

In an optional embodiment, the scheduler is further configured to: and determining a task execution sequence based on the topological structure of the task flow and/or the task information of the task node, and issuing the task information of the task node to the target executor according to the task execution sequence.

In an optional implementation manner, when issuing the task information of the task node to the target executor, the scheduler is further configured to: issuing result data output by a first task node to the target actuator; the first task node is a task node which a second task node depends on; the result data output by the first task node is used for the target executor to execute the task of the second task node; the result data is reported to the scheduler after a target executor corresponding to the first task node executes a task corresponding to the first task node; the task nodes include the first task node and the second task node.

In this way, the target executor can accurately transfer the result data output by the first task node to the second task node dependent on the first task node, so as to ensure that the task flow is accurately and orderly performed.

In an optional implementation manner, when the scheduler issues the task information of the task node to the target executor, the scheduler is configured to: issuing task information corresponding to the task node to a task queue corresponding to the target actuator; the target executor is further configured to obtain the task information from the corresponding task queue.

Therefore, the task information corresponding to the task node can be issued to the task queue corresponding to the target actuator, and the target actuator does not need to wait for the task currently being executed to be processed completely, so that the efficiency is higher. In addition, the target executor can directly acquire the task information from the corresponding task queue, so that different executors can be isolated when acquiring the task information, the executors can be prevented from processing tasks required to be processed by other executors by mistake, and the safety is higher.

In an optional embodiment, when determining the target executor corresponding to the task node, the scheduler is configured to: and determining a corresponding target actuator for each task node based on the state of each actuator in the plurality of actuators and/or the execution environment of each actuator deployment.

Therefore, the method is more suitable for processing the condition of synchronously performing task processing on the task flow under various environments, and can improve the safety and the accuracy of the task processing.

In an optional implementation, the scheduler is further configured to: after task information corresponding to task nodes contained in the task stream is obtained, determining whether components corresponding to the task nodes exist or not based on the task information of the task nodes; and determining a target executor for the task node under the condition that the component corresponding to the task node exists.

In an optional implementation manner, the scheduler is further configured to determine, after obtaining task information corresponding to a task node included in the task flow, whether a user initiating the task flow has permission to use a target component corresponding to the task node based on the task information of the task node; and under the condition that the user is determined to have the right to use the corresponding target component of the task node, determining a target executor for the task node.

In an optional embodiment, the method further comprises: a right management unit; the scheduler, when determining whether the user initiating the task flow has permission to use the target component corresponding to the task node, is configured to: sending the user identification of the user and the component identification corresponding to the target component to the authority management unit; receiving a permission verification result fed back by the permission management unit based on the user identification and the component identification; the authority management unit is used for determining a target component based on the component identification; and determining whether the user has the right to use the target component or not based on the owner identification of the target component, the sharing state of the target component and the user identification.

Therefore, different users can be effectively managed to maintain safe and normal use of the target assembly.

In an optional embodiment, the system further comprises the component management unit; the target executor is configured to, when acquiring the target component of the corresponding task node based on the task information of the corresponding task node: sending the component identification in the task information to a component management unit; receiving a target component corresponding to the component identifier and issued by the component management unit; and the component management unit is used for sending the target component corresponding to the component identifier to the target executor after receiving the component identifier sent by the target executor.

In an optional embodiment, when sending the target component corresponding to the component identifier to the target executor, the component management unit is configured to determine whether a user initiating the task flow has permission to use the target component corresponding to the component identifier; and sending the target component corresponding to the component identification to the target executor under the condition that the user is determined to be authorized to use the target component corresponding to the component identification.

Therefore, the user can upload or establish the corresponding component by himself, and flexibility is stronger.

In an optional embodiment, the task corresponding to the task node includes at least one of the following: the method comprises a model training task, a data preprocessing task, a model mode testing task and a model code packaging task.

In a second aspect, an embodiment of the present disclosure further provides a task processing method, which is applied to a task processing device, where the task processing device includes: a scheduler, and a target executor; the task processing method comprises the following steps: the scheduler analyzes a task flow to be executed to obtain task information of task nodes in the task flow and determine a target actuator corresponding to the task nodes; the number of the task nodes is at least one; the target executor acquires a target component of a corresponding task node based on task information of the corresponding task node, and executes a task of the corresponding task node based on the target component and the task information.

In an optional embodiment, the determining, by the scheduler, a target executor corresponding to the task node includes: and determining a target executor corresponding to the task node based on the topological structure of the task flow and the task information of the task node.

In an optional embodiment, the method further comprises: and the scheduler determines a task execution sequence based on the topological structure of the task flow and/or the task information of the task nodes, and issues the task information of the task nodes to the target executor according to the task execution sequence.

In an optional implementation manner, the issuing, by the scheduler, the task information of the task node to the target executor includes: the scheduler issues result data output by a first task node to the target executor; the first task node is a task node which a second task node depends on; the result data output by the first task node is used for the target executor to execute the task of the second task node; the result data is reported to the scheduler after a target executor corresponding to the first task node executes a task corresponding to the first task node; the task nodes include the first task node and the second task node.

In an optional implementation manner, the issuing, by the scheduler, the task information of the task node to the target executor includes: the scheduler sends task information corresponding to the task node to a task queue corresponding to the target actuator; and the target executor acquires the task information from the corresponding task queue.

In an optional embodiment, the determining, by the scheduler, a target executor corresponding to the task node includes: and determining a corresponding target actuator for each task node based on the state of each actuator in the plurality of actuators and/or the execution environment of each actuator deployment.

In an optional embodiment, the method further comprises: the scheduler determines whether an assembly corresponding to the task node exists or not based on the task information of the task node after obtaining the task information corresponding to the task node contained in the task flow; and determining a target executor for the task node under the condition that the component corresponding to the task node exists.

In an optional embodiment, the method further comprises: after the task information corresponding to the task node contained in the task flow is obtained, the scheduler determines whether a user initiating the task flow has the authority to use a target assembly corresponding to the task node or not based on the task information of the task node; and under the condition that the user is determined to have the right to use the corresponding target component of the task node, determining a target executor for the task node.

In an optional embodiment, the method further comprises: the scheduler determines whether a user initiating the task flow has permission to use a target component corresponding to the task node, including: the dispatcher sends the user identification of the user and the component identification corresponding to the target component to the authority management unit; receiving a permission verification result fed back by the permission management unit based on the user identification and the component identification; the authority management unit determines a target component based on the component identifier; and determining whether the user has the right to use the target component or not based on the owner identification of the target component, the sharing state of the target component and the user identification.

In an optional implementation manner, the obtaining, by the target executor, a target component of a corresponding task node based on task information of the corresponding task node includes: the target executor sends the component identifier in the task information to a component management unit; receiving a target component corresponding to the component identifier and issued by the component management unit; and after receiving the component identifier sent by the target executor, the component management unit sends the target component corresponding to the component identifier to the target executor.

In an optional embodiment, the sending, by the component management unit, the target component corresponding to the component identifier to the target executor includes: the component management unit determines whether a user initiating the task flow has the right to use a target component corresponding to the component identification; and sending the target component corresponding to the component identification to the target executor under the condition that the user is determined to be authorized to use the target component corresponding to the component identification.

In a third aspect, this disclosure also provides a computer device, a processor, and a memory, where the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions are executed by the processor to perform the steps of the second aspect or any one of the possible implementations of the second aspect.

In a fourth aspect, alternative implementations of the present disclosure also provide a computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to perform the steps of the second aspect described above, or any one of the possible implementations of the second aspect.

For the description of the effect of the task processing method, the computer device, and the computer-readable storage medium, reference is made to the description of the task processing apparatus, which is not repeated herein.

In order to make the aforementioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with figures are described in detail below.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly described below, and the drawings herein incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the technical solutions of the present disclosure. It is appreciated that the following drawings depict only certain embodiments of the disclosure and are therefore not to be considered limiting of its scope, for those skilled in the art will be able to derive additional related drawings therefrom without the benefit of the inventive faculty.

Fig. 1 is a schematic diagram illustrating a task processing device provided by an embodiment of the present disclosure;

FIG. 2 illustrates a schematic diagram of two task flows provided by an embodiment of the present disclosure;

FIG. 3 is a block diagram illustrating a task processing device according to an embodiment of the present disclosure;

FIG. 4 is a schematic diagram illustrating a task flow provided by an embodiment of the present disclosure;

fig. 5 shows a flowchart of a task processing method provided by an embodiment of the present disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, not all of the embodiments. The components of embodiments of the present disclosure, as generally described and illustrated herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the disclosure, as claimed, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the disclosure without making creative efforts, shall fall within the protection scope of the disclosure.

Research shows that many application links are needed to realize the deployment of the deep neural network; the task links are different, and the execution environments deployed during the execution of the required task are different, so that the execution environments to be deployed need to be switched when the tasks corresponding to the different task links are executed, and the switching work of the deployed execution environments causes the deep neural network to need very complicated and time-consuming repeated work from model training to landing, such as the migration of codes under different execution environments, and the like, thereby causing low efficiency.

Based on the above research, the present disclosure provides a task processing apparatus, which flexibly determines a corresponding target executor for each task node through scheduling of a scheduler, where the target executor corresponding to each task node can automatically invoke a target component required by the corresponding task node to process, and execute a task corresponding to the task node, and an execution environment for deployment of the executor can be flexibly set according to actual application requirements, thereby avoiding a task of switching the execution environment when executing different task loops, and improving task processing efficiency.

The switching of the execution environment using deployment is not needed; and for the task flow, the scheduler can determine the target component by using the task node included in the task flow and complete the task by using the target component, so that the task processing device can manage all tasks in the task flow, and the efficiency is high.

The above-mentioned drawbacks are the results of the inventor after practical and careful study, and therefore, the discovery process of the above-mentioned problems and the solutions proposed by the present disclosure to the above-mentioned problems should be the contribution of the inventor in the process of the present disclosure.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

A task processing device provided in an embodiment of the present disclosure is described below.

Referring to fig. 1, a schematic diagram of a task processing device provided in an embodiment of the present disclosure is shown, where the task processing device includes: a scheduler 10 and a target executor 20;

the scheduler 10 is configured to analyze a task stream to be executed, obtain task information of task nodes in the task stream, and determine a target actuator corresponding to the task node; at least one task node;

and the target executor 20 is configured to obtain a target component of the corresponding task node based on the task information of the corresponding task node, and execute the task of the corresponding task node based on the target component and the task information.

According to the method and the device, the task flow to be executed is analyzed by the scheduler, the task information of each task node and the corresponding target actuator are determined, and the target actuator acquires the target assembly of the corresponding task node according to the task information, so that the task of the corresponding task node is executed. Therefore, the corresponding target executors are flexibly determined for the task nodes through the scheduling of the scheduler, the target executors corresponding to the task nodes can automatically call the target assemblies required by the corresponding task nodes to process, the tasks corresponding to the task nodes are executed, and the deployment environment of the executors can be flexibly set according to the actual application requirements, so that the work of executing environment switching during the execution of different task loops is avoided, and the task processing efficiency is improved.

The task stream to be executed may include a plurality of task nodes, and each task node corresponds to one task to be processed. For example, for a process from training to applying the deep neural network to an actual application scenario, the corresponding plurality of tasks to be processed may include at least one of a data preprocessing task, a model training task, a model pattern testing task, and a model code packing task. A plurality of tasks to be processed may form a task stream to be executed.

Referring to fig. 2, a schematic diagram of two task flows provided by the embodiment of the present disclosure is shown; the schematic diagram of a task flow shown in fig. 2 (a) includes a task node a corresponding to a data preprocessing task, a task node B corresponding to a model training task, a task node C corresponding to a model pattern testing task, and a task node D corresponding to a model code packing task. In the schematic diagram of the task flow shown in fig. 2, since a plurality of tasks are processed in series, there is only one corresponding flow direction. In another possible embodiment, there may be multiple task nodes operating in parallel in one task flow, for example, in a multi-model training task, corresponding to the task node B corresponding to the model training task shown in fig. 2 (a), there may be multiple task nodes corresponding to model training tasks respectively, for example, in the case of including 3 models, the task nodes corresponding to the model training tasks may include B1, B2, and B3; in this case, as shown in fig. 2 (b), for example, the task flow of the multi-model training may have a plurality of corresponding flow directions when the task flow is processed because there is a part of the parallel task processing. Alternatively, the task of the model training task corresponding to each of the 3 models is corresponding to one task node, and the corresponding task flow can be directly represented in the form shown in fig. 2 (a).

In addition, since each task node in the task flow is sequentially executed according to the sequence from the data preprocessing task to the model code packing task, the task node in the task flow has an input task node, for example, a task node a corresponding to the data preprocessing task shown in (a) of fig. 2; and there is an output task node, for example, a task node D corresponding to the model code packaging task shown in (a) of fig. 2.

In the task processing device provided by the embodiment of the present disclosure, the task flow may constitute a Directed Acyclic Graph (DAG). The task processing device may provide the user with the functionality to customize the task flow, i.e., the task flow may be autonomously determined by the user. Illustratively, a user may determine a task flow in two different ways: the browser interface visualizes drag and drop, or emphasizes, text in the Yet Another Markup Language (YAML) format centered on the data. When a user determines a task flow, the dependency relationship between task nodes, that is, the execution sequence of the task flow, can be defined in the task flow by determining task nodes corresponding to different tasks.

For each task node in the task flow, the task information of the task node is corresponding to the task node. The task information corresponding to each task node is related to its corresponding component.

Illustratively, the components may include different types, such as two types, distributed version control system Git and application container engine Docker. A Git-type component, which mainly comprises an executable script or a file; a component of the Docker type, which is primarily a mirror image packaged in Docker format.

Task information (also referred to as attribute information) corresponding to the component may include at least one of the following: component information, parameters, execution entries, and concurrency for the corresponding task nodes.

The component information of the corresponding task node may include, for example, a component identification of the corresponding target component. Taking the example that the task node includes a task node corresponding to the model code packing task, the corresponding target component may include a packing component, and then the component information of the task node correspondingly includes a component identifier corresponding to the packing component. And under the condition that a plurality of components corresponding to the task nodes exist, the component information corresponding to the task nodes comprises component identifications respectively corresponding to the plurality of components. The parameters of the corresponding task node may include, for example, parameters of the component runtime, including input (input) parameters and output (output) parameters. The execution entry for the corresponding node may include, for example, an entry for the target component execution for the corresponding node. The concurrency of the corresponding task node may include, for example, determining based on the number of task nodes that can be processed synchronously with the corresponding task node in the task stream. Taking a task flow shown in (B) of fig. 2 as an example, after the first task node, 3 task nodes including B1, B2, and B3 are included, and therefore the concurrency information of the 3 task nodes may be set to 3. Specifically, the task information of the task node may be determined according to an actual situation, and is not limited herein.

After the scheduler parses the task stream to be executed, the target executor 20 corresponding to the task node may be determined.

In one possible embodiment, the target executor 20 corresponding to the task node may be determined based on the topology of the task flow and the task information of the task node, for example.

Specifically, when the scheduler 10 determines the target actuators 20 corresponding to the task nodes according to the topology of the task flow, for example, the execution order of the plurality of actuators may be determined in advance according to the task flow, and then when determining the corresponding target actuators 20 for the task nodes, the actuators corresponding to the positions of the plurality of actuators may be determined as the target actuators 20 corresponding to the task nodes according to the positions of the task nodes in the task flow.

Take a task flow shown in fig. 2 (a) as an example. When the executor corresponding to the task node a includes an executor a, the executor corresponding to the task node B includes an executor B, the executor corresponding to the task node C includes an executor C, and the executor corresponding to the task node D includes an executor D, an execution sequence of the plurality of executors, that is, the executor a, the executor B, the executor C, and the executor D, may be determined by using a corresponding relationship between the task nodes and the executors and a topology structure of the task flow. When the scheduler 10 determines the corresponding target executor 20 for the task node C, since the task node C is located at the third position in the order in the topological graph of the task flow, it may be determined that the executor corresponding to the third position in the execution order of the executor, that is, the executor C, is the target executor 20 corresponding to the task node C. In this way, the execution sequence of the multiple executors can be determined in the early stage, and the time for searching the executors of the corresponding task nodes can be reduced in the actual task processing, so that the task processing efficiency is improved.

When the scheduler determines the target executor 20 corresponding to the task node according to the task information of the task node, since different executors can process different tasks, an executor capable of executing the processing task corresponding to the task node exists for any task node in the task stream.

Also taking a task flow shown in fig. 2 (a) as an example, for the task node a, which corresponds to the data preprocessing task, at least one executor exists among the executors that can execute the data preprocessing task. Since other task nodes have such a correspondence relationship, when the target executor 20 is specified for a task node, it is possible to search for an executor capable of completing the task from among a plurality of executors in the task processing device according to the task corresponding to the task node, and to use the executor as the target executor 20. In this way, when only one executor capable of completing the task corresponding to the task node is included in any task node in the task stream, the scheduler 10 can quickly search for the target executor 20 according to the function of the executor.

In addition, the scheduler 10 may also determine a task execution order based on the topology of the task flow and/or the task information of the task nodes, and issue the task information of the task nodes to the target executor 20 according to the task execution order.

When the scheduler 10 issues the task information of the task node to the target executor, the scheduler is configured to issue the task information corresponding to the task node to the task queue corresponding to the target executor 20; the target executor 20 is also configured to obtain task information from the corresponding task queue.

Since the task information of the task node issued by the scheduler 10 to the target executor 20 may include a plurality of task information, and the computing power of the target executor 20 is limited, there may be task information issued to the target executor 20 but not executed by the target executor 20, and a corresponding task queue may be set for the target executor 20. When the scheduler 10 issues the task information of the task node to the target executor 20 according to the task node, the task information corresponding to the task node may be issued to the task queue corresponding to the target executor 20 without waiting for the target executor 20 to finish processing the currently executed task, so that the efficiency is higher; in addition, the target executor 20 can directly acquire the task information from the corresponding task queue, so that different executors can be isolated when acquiring the task information, thereby preventing the executors from processing the tasks required to be processed by other executors by mistake, and having higher safety.

When the scheduler 10 issues the task information of the task node to the target executor 20, the result data output by the first task node may be issued to the target executor 20. The first task node is a task node which the second task node depends on; the result data is reported to the scheduler 10 after the target executor corresponding to the first task node executes the task corresponding to the first task node; the task nodes include a first task node and a second task node. And the result data output by the first task node is used for the target executor to execute the task of the second task node.

For example, for the task flow shown in fig. 2 (a), there is a dependency relationship between the task node a corresponding to the data preprocessing task and the task node B corresponding to the model training, that is, after the task of the task node a is processed by the corresponding target executor 20, the task node a continues to be processed by the target executor 20 corresponding to the task node B. At this time, when the task node a is taken as the first task node, the task node dependent on the first task node corresponds to the task node B, that is, the task node B is taken as the second task node.

After the target executor 20 corresponding to the first task node a finishes processing the corresponding task, that is, after the first task node a finishes processing the task information in the task queue corresponding to the first task node a, the result data obtained after the processing may be output and reported to the scheduler 10; after receiving the result data reported by the first task node, the scheduler 10 issues the result data output by the first task node to the target executor 20 corresponding to the second task node B.

For the remaining task nodes having dependency relationships in the task stream, the way in which the scheduler 10 issues the task information of the task node to the target executor 20 corresponding to the first task node and the second task node is similar to the process of the task node a and the task node B in the above example, and is not described again here.

In another possible embodiment, the scheduler 10 may further determine a corresponding target executor 20 for each task node based on the state of each executor of the plurality of executors and/or the execution environment in which each executor is deployed.

Illustratively, the executors may correspond to three different execution environments, for example, including Slurm execution, kubernets execution (K8s), and SSH execution. In the Slurm cluster, an executor mainly executes tasks such as a data preprocessing task, a model training task and the like; in a Kubernetes cluster, an actuator mainly executes tasks such as a data preprocessing task and a model training task; in the SSH execution cluster, an Executor mainly executes tasks such as remote embedded equipment to the network and then executes user tasks, and is mainly used for executing tasks such as model mode test tasks.

Since executors that can execute the same task may correspond to different execution environments, task flows in different execution environments may have similar task nodes and correspond to executors that execute the same task. In this case, only determining the target executor 20 according to the task corresponding to the task node may have a problem that the determined execution environment corresponding to the target executor 20 does not support, for example, if the task node in the task flow corresponds to a data preprocessing task under the churm cluster, the determined executor under the kubernets cluster cannot correctly process the data preprocessing task, and therefore the scheduler 10 may further determine the target executor 20 that can complete the task processing under the corresponding execution environment according to the execution environment deployed by each executor. The method is more suitable for processing the situation of synchronously processing the tasks of the task flow under various execution environments, and can improve the safety and the accuracy of the task processing.

In addition, in the corresponding execution environment, if a plurality of executors that can be used as the target executors 20 can be determined in the executors, the scheduler 10 may further search for a free executor from the plurality of executors as the target executor 20 according to the execution environment in which each executor is deployed; this method of determining the target actuator 20 is more flexible.

In a specific implementation, since the number of components is limited, and there may be a case where a component corresponding to a task node does not exist, the scheduler 10 is further configured to determine whether a component corresponding to a task node exists based on task information of the task node after obtaining task information corresponding to a task node included in a task stream, and determine the target executor 20 for the task node if the component corresponding to the task node exists.

In case the scheduler 10 determines that there is no component corresponding to a task node, it may stop processing the task stream and report corresponding error information to the user, for example including "error: component information not queried to task node a ".

In addition, the components may be uploaded by the user themselves or built up through the task processing device. For example, when a user generates a DAG, the user may establish components corresponding to at least part of task nodes in the DAG, store the components in a component database, and incorporate the components into the management of the component management unit. The management of the components by the component management unit may refer to the following embodiments, which are not described herein again.

In addition, since the task processing device can be oriented to different usage groups, and the different usage groups have different functions required when using the task processing device, it is possible to determine that the rights to be used are different for the different usage groups.

Illustratively, the usage group may include an owner (owner), a developer (developer), a user (user), and a viewer (guest). For an owner, its corresponding rights may include, for example, use, delete, modify, and share. For the sharing function in the owner authority, under the condition that the owner starts the sharing function, other user groups can obtain the corresponding authority; in the case where the owner turns off the sharing function, only the components in the task processing device can be used by the owner. For a developer, its corresponding rights may include, for example, usage and modification. For a user, its corresponding rights may include, for example, usage. For a viewer, its corresponding rights may include, for example, browsing.

Therefore, the scheduler 10 may also be configured to determine, based on task information of the task node, whether a user initiating the task flow has permission to use a target component corresponding to the task node after obtaining task information corresponding to the task node included in the task flow; and under the condition that the user is determined to have the right to use the target component corresponding to the task node, determining a target executor 20 for the task node.

For example, the permissions of different usage groups may be determined for a plurality of task nodes included in a task flow, respectively. For example, for the task node a, the right corresponding to the browser includes browsing, so when the browser initiates the task node a, the scheduler 10 determines that the browser initiating the task flow does not have the right to use the task information of the task node a, and the browser is not authorized to use the target component corresponding to the task node a. Since the authority corresponding to the user includes usage, when the user initiates the task node a, the scheduler 10 determines that the user initiating the task flow has authority to use the task information of the task node a, and authorizes the user to use the target component corresponding to the task node a.

Specifically, in the task processing device, a right management unit is further included. When the scheduler 10 determines whether the user initiating the task flow has the right to use the target component corresponding to the task node, the user identifier of the user and the component identifier corresponding to the target component may be sent to the right management unit; and receiving the authority verification result fed back by the authority management unit based on the user identification and the component identification.

For example, the above usage groups, i.e., the owner, the developer, the user, and the viewer, may each correspond to a logo, such as the english abbreviation O, D, U, G corresponding thereto. The users of different usage groups can determine the corresponding user identifications according to the corresponding usage groups. For example, for an owner, an "O" is included in its corresponding user identifier; for a developer, a "D" is included in its corresponding user identification.

After the scheduler 10 sends the user identifier of the user and the component identifier corresponding to the target component to the rights management unit, the rights management unit may determine the target component based on the component identifier, and determine whether the user has a right to use the target component based on the owner identifier of the target component, the sharing state of the target component, and the user identifier.

For example, the dispatcher 10 may send the component identifier of the target component to the rights management unit, and the rights management unit determines the target component according to the component identifier and queries the rights of the user group corresponding to the user identifier to the target component. Firstly, because the owner has the sharing authority, if the owner does not open the sharing authority, users of other user groups can utilize the target component to operate, and therefore the owner identification of the target component is judged firstly. After the owner is determined, if the owner does not open the sharing right, the user does not have the right to use the target component under the condition that the user identifier does not include "O", that is, the user does not include the owner, and only the user as the owner can have the right to call the target component. Under the condition that the owner opens the sharing authority, if the user identifier comprises 'D', namely the user is a developer, the user can use and modify the target component according to the corresponding authority of the developer, such as use and modification.

By the method, different users can be effectively managed to maintain safe and normal use of the target assembly.

In addition, the task processing apparatus further includes a component management unit, and the target executor 20 is configured to send the component identifier in the task information to the component management unit when acquiring the target component corresponding to the task node based on the task information corresponding to the task node. After receiving the component identifier sent by the target executor 20, the component management unit sends the target component corresponding to the component identifier to the target executor 20.

When sending the target component corresponding to the component identifier to the target executor 20, the component management unit sends the target component corresponding to the component identifier to the target executor 20 under the condition that whether the user initiating the task flow has the right to use the target component corresponding to the component identifier is determined. The manner of determining the user authority is not repeated herein.

For the component management unit, the component management unit may manage all components, and for example, may store component identifiers corresponding to different components, and after receiving a component identifier in the task information sent by the target executor 20, according to a corresponding relationship between the determined component identifier and the component, determine a component corresponding to the received component identifier from among the managed multiple components, and take the component as the target component. The component management unit then sends the determined target component to target executor 20 so that target executor 20 can complete the corresponding task with the target component.

In addition, the component management unit also provides management of user-defined tasks, and for the component management unit, a component corresponding to each task is an executable file, a script or a packaged mirror image. The user can upload or establish the corresponding component by himself.

The embodiment of the disclosure also provides a specific embodiment of processing the task flow by using the task processing device. Referring to fig. 3, a block diagram of a task processing device according to an embodiment of the present disclosure is shown in a specific embodiment.

A user customizes the task flow DAG _1 by using an entry of the task processing device, and defines each task node in the task flow, including a task node a, a task node B, and a task node C, a schematic diagram corresponding to the task flow DAG _1 is shown in fig. 4, and the three task nodes are sequentially executed in series.

When the user defines the task node, the task node also comprises task information of the corresponding task node, including component information, parameters, an execution entry and concurrency of the corresponding node. In addition, when determining the task flow DAG _1, the execution order of the task flow DAG _1, that is, the dependency relationship among the task node a, the task node B, and the task node C is also determined.

When the scheduler 10 parses the task stream DAG _1, according to the multiple task nodes and the topology obtained by parsing, a target executor is determined as an executor a for the task node a, a target executor is determined as an executor B for the task node B, and a target executor is determined as an executor C for the task node C. And the component management unit 30 may determine the component corresponding to each task node according to the component identifier corresponding to the task node.

When any user initiates a task flow, the scheduler 10 confirms to the permission management unit 40 whether the user initiating the task flow has permission to use the target component corresponding to the task node. In a specific embodiment, the user is an owner and has corresponding usage rights.

The scheduler 10 firstly issues task information to a task queue corresponding to the task node a to the actuator a, and after the actuator a finishes processing the tasks in the corresponding task queue, the scheduler outputs the result data obtained after processing, and reports the result data to the scheduler 10. The scheduler 10 issues the result data output by the task node a to the target executor corresponding to the second task node B.

Then, the scheduler 10 issues task information to the task queue corresponding to the task node B, and issues result data output by the task node a to the task node B, and after the executor B corresponding to the task node B finishes processing the task in the task queue corresponding to the executor B, the result data obtained after processing is output and reported to the scheduler 10. The scheduler 10 issues the result data output by the task node B to the target executor corresponding to the third task node C.

Then, the scheduler 10 issues task information to the task queue corresponding to the task node C, issues result data output by the task node B to the task node C, and after the actuator C corresponding to the task node C finishes processing the task in the task queue corresponding to the task node C, outputs the result data obtained after processing, and reports the result data to the scheduler 10.

At this time, after the data result data of the task node C, the execution of all tasks of the task stream DAG _1 is completed.

In addition, in the task processing apparatus, logs (logs) and accessories (Artifacts) are also included. The log 50 is used for collecting and storing the log when the executor executes the task, and then the user can query the log record executed at the current time anytime and anywhere, so that the task execution progress is conveniently tracked. The attachment 60 is used for collecting task return values in the form of files, storing the task return values in a unified manner, facilitating downloading in a browser by a user, mainly managing model files obtained by training, generating reports and the like.

It will be understood by those skilled in the art that the order of writing of steps in an apparatus of a particular embodiment does not imply a strict order of execution and any limitations on the implementation, and that the particular order of execution of the steps should be determined by their function and possibly their inherent logic.

Based on the same inventive concept, a task processing method corresponding to the task processing device is also provided in the embodiments of the present disclosure, and since the principle of solving the problem of the method in the embodiments of the present disclosure is similar to that of the task processing device in the embodiments of the present disclosure, the implementation of the method can refer to the implementation of the method, and repeated details are not described again.

Referring to fig. 5, a flowchart of a task processing method provided in an embodiment of the present disclosure is a task processing method, where the task processing method is applied to a task processing device, and includes steps S501 to S502, where:

s501: the method comprises the steps that a scheduler analyzes a task flow to be executed to obtain task information of task nodes in the task flow and determine a target actuator corresponding to the task nodes; at least one task node;

s502: the target executor acquires a target component of the corresponding task node based on the task information of the corresponding task node, and executes the task of the corresponding task node based on the target component and the task information.

An embodiment of the present disclosure further provides a computer device, including: instruction memory and the task processing device that this disclosed embodiment provided.

The task processing device provided by the embodiment of the disclosure may include a chip, an AI chip, and the like. The computer device provided by the embodiment of the present disclosure may include an intelligent terminal such as a mobile phone, or may also be other devices, servers, and the like that may be used for performing task processing, and is not limited herein.

The embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, performs the steps of the task processing method described in the above method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

The embodiments of the present disclosure also provide a computer program product, where the computer program product carries a program code, and instructions included in the program code may be used to execute steps of the task processing method in the foregoing method embodiments, which may be referred to specifically in the foregoing method embodiments, and are not described herein again.

The computer program product may be implemented by hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the system and the apparatus described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

Finally, it should be noted that: the above-mentioned embodiments are merely specific embodiments of the present disclosure, which are used for illustrating the technical solutions of the present disclosure and not for limiting the same, and the scope of the present disclosure is not limited thereto, and although the present disclosure is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive of the technical solutions described in the foregoing embodiments or equivalent technical features thereof within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present disclosure, and should be construed as being included therein. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

1. A task processing apparatus, comprising: a scheduler, and a target executor;

the scheduler is used for analyzing a task flow to be executed to obtain task information of task nodes in the task flow and determining a target actuator corresponding to the task nodes; the number of the task nodes is at least one;

the target executor is used for acquiring a target component of a corresponding task node based on task information of the corresponding task node and executing a task of the corresponding task node based on the target component and the task information.

2. The task processing apparatus according to claim 1, wherein the task information of the corresponding task node includes at least one of: the component information, parameters, execution entries, and concurrency of the corresponding task nodes.

3. The task processing apparatus according to claim 1, wherein the scheduler, when determining the target executor corresponding to the task node, is configured to:

and determining a target executor corresponding to the task node based on the topological structure of the task flow and the task information of the task node.

4. A task processing device according to any one of claims 1 to 3, wherein the scheduler is further configured to: and determining a task execution sequence based on the topological structure of the task flow and/or the task information of the task node, and issuing the task information of the task node to the target executor according to the task execution sequence.

5. The task processing apparatus according to claim 4, wherein the scheduler, when issuing the task information of the task node to the target executor, is further configured to:

issuing result data output by a first task node to the target actuator; the first task node is a task node which a second task node depends on; the result data output by the first task node is used for the target executor to execute the task of the second task node;

the result data is reported to the scheduler after a target executor corresponding to the first task node executes a task corresponding to the first task node;

the task nodes include the first task node and the second task node.

6. The task processing apparatus according to claim 4 or 5, wherein the scheduler, when issuing the task information of the task node to the target executor, is configured to:

issuing task information corresponding to the task node to a task queue corresponding to the target actuator;

the target executor is further configured to obtain the task information from the corresponding task queue.

7. The task processing apparatus according to any one of claims 1 to 6, wherein the scheduler, when determining the target executor corresponding to the task node, is configured to: and determining a corresponding target actuator for each task node based on the state of each actuator in the plurality of actuators and/or the execution environment of each actuator deployment.

8. Task processing device according to any of claims 1 to 7, wherein the scheduler is further configured to: after task information corresponding to task nodes contained in the task stream is obtained, determining whether components corresponding to the task nodes exist or not based on the task information of the task nodes;

and determining a target executor for the task node under the condition that the component corresponding to the task node exists.

9. The task processing device according to any one of claims 1 to 8, wherein the scheduler is further configured to determine, after obtaining task information corresponding to a task node included in the task stream, whether a user initiating the task stream has permission to use a target component corresponding to the task node based on the task information of the task node;

and under the condition that the user is determined to have the right to use the corresponding target component of the task node, determining a target executor for the task node.

10. The task processing apparatus according to claim 9, further comprising: a right management unit;

the scheduler, when determining whether the user initiating the task flow has permission to use the target component corresponding to the task node, is configured to: sending the user identification of the user and the component identification corresponding to the target component to the authority management unit; receiving a permission verification result fed back by the permission management unit based on the user identification and the component identification;

the authority management unit is used for determining a target component based on the component identification; and determining whether the user has the right to use the target component or not based on the owner identification of the target component, the sharing state of the target component and the user identification.

11. The task processing device according to any one of claims 1 to 10, further comprising the component management unit;

the target executor is configured to, when acquiring the target component of the corresponding task node based on the task information of the corresponding task node:

sending the component identification in the task information to a component management unit; receiving a target component corresponding to the component identifier and issued by the component management unit;

and the component management unit is used for sending the target component corresponding to the component identifier to the target executor after receiving the component identifier sent by the target executor.

12. The task processing apparatus according to claim 11, wherein the component management unit, when sending the target component corresponding to the component identifier to the target executor, is configured to determine whether a user initiating the task flow has a right to use the target component corresponding to the component identifier;

and sending the target component corresponding to the component identification to the target executor under the condition that the user is determined to be authorized to use the target component corresponding to the component identification.

13. A task processing apparatus according to any one of claims 1 to 12, wherein the task corresponding to the task node includes at least one of:

the method comprises a model training task, a data preprocessing task, a model mode testing task and a model code packaging task.

14. A task processing method is applied to a task processing device, and the task processing device comprises the following steps: a scheduler, and a target executor; the task processing method comprises the following steps:

the scheduler analyzes a task flow to be executed to obtain task information of task nodes in the task flow and determine a target actuator corresponding to the task nodes; the number of the task nodes is at least one;

the target executor acquires a target component of a corresponding task node based on task information of the corresponding task node, and executes a task of the corresponding task node based on the target component and the task information.

15. The task processing method according to claim 14, wherein the task information of the corresponding task node includes at least one of: the component information, parameters, execution entries, and concurrency of the corresponding task nodes.

16. The task processing method according to claim 14, wherein the scheduler determines a target executor corresponding to the task node, and comprises:

17. The task processing method according to any one of claims 14 to 16, further comprising: and the scheduler determines a task execution sequence based on the topological structure of the task flow and/or the task information of the task nodes, and issues the task information of the task nodes to the target executor according to the task execution sequence.

18. The task processing method according to claim 17, wherein the scheduler issues the task information of the task node to the target executor, and the task information includes:

the scheduler issues result data output by a first task node to the target executor; the first task node is a task node which a second task node depends on; the result data output by the first task node is used for the target executor to execute the task of the second task node;

the task nodes include the first task node and the second task node.

19. The task processing method according to claim 17 or 18, wherein the scheduler issues the task information of the task node to the target executor, and the task information includes:

the scheduler sends task information corresponding to the task node to a task queue corresponding to the target actuator;

and the target executor acquires the task information from the corresponding task queue.

20. A task processing method according to any one of claims 14 to 19, wherein the scheduler determines a target executor corresponding to the task node, and comprises: and determining a corresponding target actuator for each task node based on the state of each actuator in the plurality of actuators and/or the execution environment of each actuator deployment.

21. A task processing method according to any one of claims 14 to 20, further comprising: the scheduler determines whether an assembly corresponding to the task node exists or not based on the task information of the task node after obtaining the task information corresponding to the task node contained in the task flow;

22. A task processing method according to any one of claims 14 to 21, further comprising: after the task information corresponding to the task node contained in the task flow is obtained, the scheduler determines whether a user initiating the task flow has the authority to use a target assembly corresponding to the task node or not based on the task information of the task node;

23. The task processing method according to claim 22, further comprising: the scheduler determines whether a user initiating the task flow has permission to use a target component corresponding to the task node, including: the dispatcher sends the user identification of the user and the component identification corresponding to the target component to the authority management unit; receiving a permission verification result fed back by the permission management unit based on the user identification and the component identification;

the authority management unit determines a target component based on the component identifier; and determining whether the user has the right to use the target component or not based on the owner identification of the target component, the sharing state of the target component and the user identification.

24. The task processing method according to any one of claims 14 to 23, wherein the acquiring, by the target executor, the target component of the corresponding task node based on the task information of the corresponding task node includes:

the target executor sends the component identifier in the task information to a component management unit; receiving a target component corresponding to the component identifier and issued by the component management unit;

and after receiving the component identifier sent by the target executor, the component management unit sends the target component corresponding to the component identifier to the target executor.

25. A task processing method according to claim 24, wherein the sending, by the component management unit, the target component corresponding to the component identifier to the target executor comprises: the component management unit determines whether a user initiating the task flow has the right to use a target component corresponding to the component identification;

26. A task processing method according to any one of claims 14 to 25, wherein the task corresponding to the task node includes at least one of:

27. A computer device, comprising: instruction memory and a task processing device according to any of claims 1-13.

28. A computer-readable storage medium, characterized in that a computer program is stored thereon, which, when being executed by a task processing device, performs the steps of the task processing method according to any one of claims 14 to 26.