CN114185660A - Task management system, method, device, equipment and storage medium - Google Patents
Task management system, method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114185660A CN114185660A CN202111506788.XA CN202111506788A CN114185660A CN 114185660 A CN114185660 A CN 114185660A CN 202111506788 A CN202111506788 A CN 202111506788A CN 114185660 A CN114185660 A CN 114185660A
- Authority
- CN
- China
- Prior art keywords
- task
- executed
- interface
- execution
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
Abstract
The application relates to a task management system, a method, a device, equipment and a storage medium. The system comprises: the interface method layer is used for providing an interface connected with the human-computer interaction interface; the analysis layer is used for providing a workflow description language analyzer and providing a task scheduling interface and a task storage interface; and the execution layer is used for providing computing resources and storage resources, providing a task execution shell program to download and execute the data of the task to be executed, and uploading an execution result after the execution is finished. According to the task scheduling and storing method and device, tasks can be scheduled and stored through independent interfaces, various cloud computing platforms and cloud storage platforms can be randomly docked, the same tasks can be called and executed in a mixed environment, the granularity of task management is increased to a container level through the design of shell programs, the difference of service tools of different cloud manufacturers in the mixed cloud environment is shielded, the containerization technology is adapted to the outer layer of a computing program, cloud computing resources can be effectively utilized, computing cost is reduced, and computing amount is reduced.
Description
Technical Field
The present application relates to the field of computer system technologies, and in particular, to a task management system, a task management method, an electronic device, and a storage medium.
Background
With the practical popularization and increasing computing power of computers, the needed computing resources are continuously increasing, solutions for computing management systems are endless, and computing clouds and even up-mixed clouds become a main trend due to the requirement of massive computing resources.
In the related technical scheme, a very simple Linux resource management tool or an Airflow Platform of the Apache foundation is generally adopted, but the computing tasks of the general workflow management systems need to be bound to an execution environment, the execution tasks need to use local resources, and the storage and calling need to be written and executed by a user according to resource types, so that the tasks are difficult to use in a hybrid cloud environment, or the tasks need to be adapted to the hybrid cloud environment, so that a large amount of secondary development work is required, and the workload is increased sharply and needs to be solved.
Disclosure of Invention
In order to solve or partially solve the problems in the related art, the application provides a task management system, a task management method, an electronic device and a storage medium, which can call tasks in a hybrid cloud environment and enhance the universality and expansibility of the task management system.
A first aspect of the present application provides a task management system, including:
the interface method layer is used for providing an interface connected with a human-computer interaction interface, receiving task management operation input by a user through the human-computer interaction interface based on the interface and determining a task to be executed;
the analysis layer is used for determining a workflow description language of the task to be executed, analyzing the workflow description language to obtain an execution sequence of the task to be executed, providing a task scheduling interface for scheduling the task to be executed, and providing a task storage interface for storing the task to be executed;
and the execution layer is used for providing computing resources for executing the tasks to be executed and storage resources for storing the tasks to be executed, providing a task execution shell program for downloading and executing the data of the tasks to be executed, and uploading an execution result after the execution is finished.
As a possible implementation manner of the present application, in this implementation manner, the tasks include a primary task, a secondary task, and a tertiary task, where each of the primary tasks includes one or more of the secondary tasks, each of the secondary tasks includes one or more of the tertiary tasks, and the interface method layer includes:
the first-level task interface module is used for providing management query service of the first-level task;
the second-level task interface module is used for providing management query service of the second-level task;
and the third-level task interface module is used for providing management query services of the third-level tasks, wherein the management query services comprise task submission services, task state query services, task rerun services and task stop services.
As a possible embodiment of the present application, in this embodiment, when determining a workflow description language of the task to be executed, and analyzing the workflow description language to obtain an execution sequence of the task to be executed, the parsing layer may be configured to:
for a primary task to be executed, analyzing a workflow description language of the primary task to be executed, and determining an execution sequence of each secondary task to be executed in the primary task to be executed and a dependency relationship of each secondary task to be executed; and/or
And for one to-be-executed secondary task, determining the execution sequence of each to-be-executed tertiary task in the to-be-executed secondary task and the dependency relationship among the to-be-executed tertiary tasks.
As one possible embodiment of the present application, in this embodiment, the analysis layer includes:
the workflow parser is used for parsing the workflow description language of the tasks to be executed and determining the sequence of the tasks to be executed, the dependency relationship among the tasks, a parameter serialization method and a parameter deserialization method;
the task scheduling interface is encapsulated with a minimum general requirement for task scheduling execution and is used for scheduling the task to be executed, and the minimum general requirement for task scheduling execution is used for standardizing the functional requirements of each computing platform for the task management system;
the task storage interface is packaged with a minimum general requirement for task storage and execution and used for storing the tasks to be executed, and the minimum general requirement for task storage and execution is used for standardizing the functional requirements of each storage space for accessing the task management system.
As a possible embodiment of the present application, in this embodiment, when executing the task execution shell program, the execution layer is further configured to:
updating the task state of the task to be executed based on the execution result of the task to be executed;
detecting whether the task to be executed is executed completely, and when the task to be executed is not executed completely, continuing to execute the task to be executed; and after the task to be executed is executed, detecting whether a subsequent task exists after the task to be executed, if so, calling the subsequent task, and continuing to execute the subsequent task.
As a possible embodiment of the present application, in this embodiment, the interface method layer may further be configured to:
receiving a task state query request input by a user based on a human-computer interaction interface;
and sending the state of the corresponding task to the human-computer interaction interface for displaying based on the task state query request.
A second aspect of the present application provides a task management method, including:
receiving task management operation input by a user based on a human-computer interaction interface;
determining a task to be executed based on the task management operation, calling a task storage interface to store the task to be executed to a cloud end, and calling a task calling interface to submit the task to be executed to a cloud computing platform;
allocating container space for the task to be executed, loading a task shell program to be executed, analyzing a workflow description language of the task to be executed, and acquiring task data of the task to be executed;
and executing the task data, and uploading an execution result to a cloud end for storage.
As a possible implementation manner of the present application, in this implementation manner, after uploading the execution result to the cloud storage, the method further includes:
detecting whether the task to be executed is executed completely, and when the task to be executed is not executed completely, continuing to execute the task to be executed; and after the task to be executed is executed, detecting whether a subsequent task exists after the task to be executed, if so, calling the subsequent task, and continuing to execute the subsequent task.
A third aspect of the present application provides a task management apparatus including:
the human-computer interaction module is used for receiving task management operation input by a user based on a human-computer interaction interface;
the task calling module is used for determining a task to be executed based on the task management operation, calling a task storage interface to store the task to be executed to a cloud end, and calling the task calling interface to submit the task to be executed to a cloud computing platform;
the analysis module is used for allocating container space for the task to be executed, loading a task shell program to be executed, analyzing the workflow description language of the task to be executed and acquiring task data of the task to be executed;
and the task execution module is used for executing the task data and uploading an execution result to the cloud storage.
A fourth aspect of the present application provides an electronic device, comprising:
a processor; and
a memory having executable code stored thereon, which when executed by the processor, causes the processor to perform the task management method as described above.
A fifth aspect of the present application provides a storage medium having stored thereon executable code, which, when executed by a processor of an electronic device, causes the processor to perform a task management method as described above.
According to the embodiment of the application, the scheduling and the storage of the tasks are decoupled, the tasks are scheduled and stored through the independent interfaces, various cloud computing platforms and cloud storage platforms can be connected in an optional mode, the calling and the execution of the same tasks in a mixed environment are achieved, the universality and the expansibility of a task management system are enhanced, the granularity of task management is increased to a container level through the design of a shell program, all preparation/ending work different from the cloud environment is processed by the shell program, the difference of service tools of different cloud manufacturers in the mixed cloud environment is shielded, the containerization technology is adapted to the outer layer of the computing program, cloud computing resources can be effectively utilized, the computing cost is reduced, and the computing amount is reduced.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The foregoing and other objects, features and advantages of the application will be apparent from the following more particular descriptions of exemplary embodiments of the application as illustrated in the accompanying drawings wherein like reference numbers generally represent like parts throughout the exemplary embodiments of the application.
FIG. 1 is a schematic diagram of an architecture of a task management system according to an embodiment of the present application;
FIG. 2 is a directed acyclic graph according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating a task status updating method according to an embodiment of the present application;
FIG. 4 is a flow chart illustrating the execution of a shell program according to an embodiment of the present application;
FIG. 5 is a flowchart illustrating a task status query method according to an embodiment of the present application;
FIG. 6 is a flowchart illustrating a task management method according to an embodiment of the present application;
FIG. 7 is a schematic structural diagram of a task management device according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device shown in an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While embodiments of the present application are illustrated in the accompanying drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms "first," "second," "third," etc. may be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present application, "a plurality" means two or more unless specifically limited otherwise.
With the practical popularization and increasing computing power of computers, the needed computing resources are continuously increasing, solutions for computing management systems are endless, and computing clouds and even up-mixed clouds become a main trend due to the requirement of massive computing resources. In the related technical solutions, a very simple Linux resource management tool or an Airflow Platform of the Apache foundation is generally adopted, but the computing tasks of these general workflow management systems need to be bound to an execution environment, the execution tasks need to use local resources, and storage and invocation all need to be written and executed by a user according to resource types, and are difficult to use in a hybrid cloud environment, or a large amount of secondary development work is needed to adapt the workflow management system to the hybrid cloud environment, which leads to a rapid increase in workload and needs to solve the problem that the workload needs to be increased
In order to solve the above problems, embodiments of the present application provide a task management system, a task management method, an electronic device, and a storage medium, which can call a task in a hybrid cloud environment, and enhance the universality and expansibility of the task management system.
The technical solutions of the embodiments of the present application are described in detail below with reference to the accompanying drawings.
Fig. 1 is a schematic structural diagram of a task management system according to an embodiment of the present application.
Referring to fig. 1, in an embodiment of the present application, a task management system is provided that includes:
and the interface method layer 101 is used for providing an interface connected with a human-computer interaction interface, receiving task management operation input by a user through the human-computer interaction interface based on the interface, and determining a task to be executed.
In the embodiment of the application, a system architecture of a task management system is provided, where the task management system includes an interface method layer, an analysis layer, and an execution layer, where the interface method layer is configured to provide an interface connected to a human-computer interaction interface where a user is located, and receive, through the interface, a task management operation input by the user through the human-computer interaction interface, where the task management operation may be a task state query operation, a task submission operation, a task execution operation, a task rerun operation, a task stop operation, and the like. In the embodiment of the present application, the interface provided may be an SDK (Software Development Kit) or an HTTP (Hyper Text Transfer Protocol), the number of the interfaces may be one or more, and the interfaces may be different for different tasks. In the embodiment of the application, the task to be executed refers to a task which is determined by a user based on a human-computer interaction interface and needs to be executed, and the operation to be executed of the task can be task submission, state query, rerun, stop and the like.
And the analysis layer 102 is configured to determine a workflow description language of the task to be executed, analyze the workflow description language to obtain an execution sequence of the task to be executed, provide a task scheduling interface for scheduling the task to be executed, and provide a task storage interface for storing the task to be executed.
In the embodiment of the application, the analysis layer is an intermediate layer between the interface method layer and the execution layer, and is mainly used for analyzing the tasks to be executed, determining the execution sequence, the dependency relationship, the parameter serialization method, the parameter deserialization method and the like of the tasks to be executed, and providing a corresponding interface for scheduling and storing the tasks to be executed. In the embodiment of the application, after the task to be executed is determined, the workflow description language of the task to be executed is analyzed, the directed acyclic graph of the task to be executed is determined, the execution sequence of each step in the task to be executed and the dependency relationship among the steps are determined based on the directed acyclic graph, and a parameter serialization method and a parameter deserialization method in an algorithm are determined. In the embodiment of the application, the analysis layer further provides a task scheduling interface for scheduling the to-be-executed task, and provides a task storage interface for storing the to-be-executed task, wherein the task scheduling interface schedules the to-be-executed task and uploads the to-be-executed task to the cloud container, so that the to-be-executed task can be executed in the container conveniently; the task storage interface is used for uploading the tasks to the cloud storage space and storing the tasks to be executed.
And the execution layer 103 is used for providing a computing resource for executing the task to be executed and a storage resource for storing the task to be executed, providing a task execution shell program for downloading and executing the data of the task to be executed, and uploading an execution result after the execution is finished.
In the embodiment of the application, the execution layer is responsible for providing computing resources for executing the task to be executed and storage resources for storing the task to be executed, where the computing resources and the storage resources may be computing services and storage services provided by cloud vendors, such as an object storage service of amazon cloud, a batch computing service of amazon cloud, and the like. In the embodiment of the application, the execution layer is further used for the task execution shell program to download and execute the data of the task to be executed, and upload the execution result after the execution is finished. As a possible implementation manner of the present application, in this implementation manner, a task execution shell is wrapped around an algorithm script or a computing software submitted by a user, when the task execution shell is executed, a workflow of a task may be analyzed, a task to be executed is scheduled by using a task scheduling interface provided by an analysis layer, the task to be executed is uploaded to a container of a cloud computing center, the task is executed in the container, after the execution is completed, task data that is completed is downloaded from a cloud, and the task data that is completed is stored in a cloud storage space by using a task storage interface provided by the analysis layer.
According to the embodiment of the application, the scheduling and the storage of the tasks are decoupled, the tasks are scheduled and stored through the independent interfaces, various cloud computing platforms and cloud storage platforms can be connected in an optional mode, the calling and the execution of the same tasks in a mixed environment are achieved, the universality and the expansibility of a task management system are enhanced, the granularity of task management is increased to a container level through the design of a shell program, all preparation/ending work different from the cloud environment is processed by the shell program, the difference of service tools of different cloud manufacturers in the mixed cloud environment is shielded, the containerization technology is adapted to the outer layer of the computing program, cloud computing resources can be effectively utilized, the computing cost is reduced, and the computing amount is reduced.
As a possible implementation manner of the present application, in this implementation manner, the tasks include a primary task, a secondary task, and a tertiary task, where each of the primary tasks includes one or more of the secondary tasks, each of the secondary tasks includes one or more of the tertiary tasks, and the interface method layer includes:
the first-level task interface module is used for providing management query service of the first-level task;
the second-level task interface module is used for providing management query service of the second-level task;
and the third-level task interface module is used for providing management query services of the third-level tasks, wherein the management query services comprise task submission services, task state query services, task rerun services and task stop services.
In the embodiment of the application, the task can be divided into three levels based on the task concept, wherein the three levels comprise a primary task, a secondary task and a tertiary task, each primary task comprises one or more secondary tasks, each secondary task comprises one or more tertiary tasks, and different task interfaces are provided for the tasks of different levels. In the embodiment of the present application, the classification of the task is not limited to the classification of the task into three levels, and the task may be classified into one level, two levels, three levels, or more levels according to actual situations. As a possible implementation manner of the present application, for convenience of description, a specific embodiment is taken as an example, a first-level task is named as task, a second-level task is named as step, and a third-level task is named as jobb, where task is a top-level concept of tasks, one task may include one or more steps, step is a middle-level concept of tasks, one step may include one or more jobs, jobs is a bottom-level concept of tasks, different interfaces are provided for different levels of tasks, where a task interface is an upper-level control of an entire workflow task, and provides submission, monitoring, and management capabilities for the entire workflow task as a top-level granularity unit. The step interface controls a step node in a workflow task and provides submitting, monitoring and managing capabilities for the specific step node as a fine-grained unit. The job interface controls specific subtasks in the steps of the workflow task and serves as a finest granularity unit to provide submitting, monitoring and managing capabilities for the specific subtasks in the specific job node.
In the embodiment of the application, different task interfaces are provided for tasks of different levels, so that supervision operations such as submission, state query, rerun, stop and the like can be conveniently performed on different tasks, a task management system is more clear in labor division, and operation by a user is facilitated.
As a possible embodiment of the present application, in this embodiment, when determining a workflow description language of the task to be executed, and analyzing the workflow description language to obtain an execution sequence of the task to be executed, the parsing layer may be configured to:
for a primary task to be executed, analyzing a workflow description language of the primary task to be executed, and determining an execution sequence of each secondary task to be executed in the primary task to be executed and a dependency relationship of each secondary task to be executed; and/or
And for one to-be-executed secondary task, determining the execution sequence of each to-be-executed tertiary task in the to-be-executed secondary task and the dependency relationship among the to-be-executed tertiary tasks.
In this embodiment of the application, when determining the execution sequence of each task in the tasks to be executed, the determination may be performed according to the level of the task to be executed, for example, when the task to be executed is a primary task, a workflow description language of the task to be executed is analyzed, a secondary task in the task to be executed is determined, the execution sequence between the secondary tasks and a dependency relationship between the secondary tasks are determined, for each secondary task, a tertiary task in the secondary task is determined, the execution sequence between the tertiary tasks and the dependency relationship between the tertiary tasks are determined, and optionally, the execution sequence and the dependency relationship between the tasks may be determined in a form of a directed acyclic graph. As a possible implementation manner of the present application, in the implementation manner, when the task to be executed is a secondary task, the workflow description language of the secondary task is analyzed, a tertiary task in the secondary task is determined, and an execution sequence and a dependency relationship between the tertiary tasks are determined.
As a possible implementation manner of the present application, for convenience of explanation, taking a specific embodiment as an example, the determined task to be executed is a primary task1, when parsing a workflow description language of the task1, three secondary tasks, namely step1, step2, and step3, are obtained in the task1, and an execution sequence between the three secondary tasks is determined to be step1-step2-step3, parsing the three secondary tasks determines that two tertiary tasks are included in the step1, namely joba 1, joba 2, one tertiary task joba 3 is included in the step2, three tertiary tasks joba 4, joba 5, and joba 6 are included in the step3, where the joba 3 needs to start execution after both the step2 and the step3 are completed, and for convenience of understanding, an acyclic graph of the embodiment shown in fig. 2 is shown.
According to the embodiment of the application, the number of the tasks at each level in the tasks to be executed and the execution sequence and the dependency relationship among the tasks are determined by analyzing the workflow description language, so that the tasks are ensured to be normally executed.
As one possible embodiment of the present application, in this embodiment, the analysis layer 102 includes:
the workflow parser 121 is configured to parse the workflow description language of the to-be-executed task, and determine an order of the to-be-executed task, a dependency relationship between the tasks, a parameter serialization method, and a parameter deserialization method.
In this embodiment of the application, the parsing layer 102 includes a workflow parser 121, where the workflow parser 121 is configured to parse a standardized workflow description language in a task to be executed, perform description conversion on scientifically computed software and algorithms, and obtain an execution order of the task to be executed, a dependency relationship between the tasks, a parameter serialization method, a parameter deserialization method, and the like.
And the task scheduling interface 122 is packaged with a minimum general requirement for task scheduling execution and is used for scheduling the task to be executed, and the minimum general requirement for task scheduling execution is used for standardizing the functional requirements of each computing platform for the task management system.
In this embodiment of the present application, the parsing layer 102 includes a task scheduling interface 122, where a minimum general requirement of the entire task management system for task scheduling execution is encapsulated in the task scheduling interface 122, and the minimum general requirement can be used to standardize functional requirements when different cloud computing platforms, supercomputers, or local computer rooms are accessed, a group of minimum general methods is declared, and after any computing platform implements this interface by writing simple codes, the task management system provided in this embodiment of the present application can use computing resources of the computing platform.
And the task storage interface 123 is packaged with a minimum general requirement for task storage and execution and used for storing the task to be executed, and the minimum general requirement for task storage and execution is used for standardizing the functional requirement of each storage space for accessing the task management system.
In this embodiment of the present application, the parsing layer 102 includes a task storage interface 123, where the task storage interface 123 encapsulates minimum general requirements of the whole task management System for data processing during task execution, and can be used to specify functional requirements when different cloud storage, NFS (Network File System), or local hard disk accesses, and a set of minimum general methods is stated, and after any storage platform implements this interface by writing simple codes, the task management System provided in this embodiment of the present application can use storage resources of the storage platform.
In the task management system provided by the embodiment of the application, the parsing layer comprises a workflow parser, a task scheduling interface and a task storage interface, the workflow parser is used for parsing a standardized workflow description language and performing description conversion on scientific computing software or algorithms to obtain specific execution sequence, dependency relationship, parameter serialization method and parameter deserialization method of tasks, and the task scheduling interface and the task storage interface can guarantee normal calling of cloud computing resources and cloud storage resources.
As a possible embodiment of the present application, in this embodiment, as shown in fig. 3, when the execution layer executes the task execution shell program, the execution layer is further configured to:
step S301, based on the execution result of the task to be executed, updating the task state of the task to be executed.
In the embodiment of the application, the execution layer provides a task execution shell program except for the computing code during task execution, wherein the task execution shell program is wrapped around the algorithm script or the computing software submitted by the user for execution, and when the shell program is executed, the shell program can be further used for updating the task state of the task to be executed according to the execution result of the task to be executed. For convenience of description, for a specific example, the tasks to be executed include joba 1 and joba 2, and when the task data of joba 1 is calculated in the container, the task execution shell program is executed according to the task state of joba 1.
Step S302, detecting whether the task to be executed is executed completely, and when the task to be executed is not executed completely, continuing to execute the task to be executed; and after the task to be executed is executed, detecting whether a subsequent task exists after the task to be executed, if so, calling the subsequent task, and continuing to execute the subsequent task.
In the embodiment of the application, when it is detected that the execution of the task to be executed is completed, the shell program may be further configured to call the subsequent task and continue to execute the subsequent task. For illustrative purposes, as an example, after detecting that the execution of job1 is finished, the shell program will dispatch job2 to the container for execution if it detects that job2 has not been executed. For convenience of description, as shown in fig. 4, a task shell program flowchart provided in an embodiment of the present application is shown, after entering a shell program, the task shell program flowchart starts parsing a workflow language of a task to be executed, determines the number of tasks in the task to be executed, an execution sequence of the tasks, dependency relationships among the tasks, and the like, downloads task data, deserializes the task data into a memory object, calls a user program, uploads the task data to a container for calculation to obtain output data, serializes the output data into a file, uploads the file as a task result (i.e., an execution result), and then determines whether all current nodes are correctly ended, exits the shell program when it is determined that all the current nodes are not correctly ended, prompts the program to make an error, determines whether a subsequent task exists in a current workflow when all the current nodes are correctly ended, and when the subsequent task exists, and submitting the subsequent task, waiting for execution, and otherwise exiting the shell program. In the above possible implementation manner provided by the embodiment of the present application, the function of the shell program can be set according to the user requirement.
In the embodiment of the application, the task execution shell program can increase the granularity of task management to a container level, all preparation and ending work which are different from the cloud environment are processed by the shell program, the difference between the mixed cloud environment and a cloud manufacturer service tool is shielded, and a containerization technology is adapted to the outer layer of a computing program.
As a possible embodiment of the present application, in this embodiment, as shown in fig. 5, the interface method layer may further be configured to:
step S501, receiving a task state query request input by a user based on a human-computer interaction interface.
In the embodiment of the application, the task management system is connected with the human-computer interaction interface through the interface, and a user can manage each task in the task management system based on the human-computer interaction interface, wherein the task management system comprises task execution, task state query, task running again, task stopping and the like. The unique identification refers to an identification which can be specifically corresponding to a unique specific task; the general identifier refers to an identifier common to a specific task, such as the execution time of a task executed within a certain time period, and may be used to query the states of multiple tasks executed within the time period; the generic identification can be used for fuzzy queries and precise queries.
Step S502, the state of the corresponding task is sent to the human-computer interaction interface for displaying based on the task state query request.
In the embodiment of the application, after the task management system receives a task state query request input by a user based on a human-computer interaction interface through an interface, the state of the corresponding task is sent to the human-computer interaction interface for displaying based on the unique identifier or the universal identifier of the task in the request, so that the user can know the task execution progress conveniently.
According to the embodiment of the application, the user can check the states of all tasks in the task management system in real time through the human-computer interaction interface, and the user can know the task progress in time conveniently.
According to the embodiment of the application, the scheduling and the storage of the tasks are decoupled, the tasks are scheduled and stored through the independent interfaces, various cloud computing platforms and cloud storage platforms can be connected in an optional mode, the calling and the execution of the same tasks in a mixed environment are achieved, the universality and the expansibility of a task management system are enhanced, the granularity of task management is increased to a container level through the design of a shell program, all preparation/ending work different from the cloud environment is processed by the shell program, the difference of service tools of different cloud manufacturers in the mixed cloud environment is shielded, the containerization technology is adapted to the outer layer of the computing program, cloud computing resources can be effectively utilized, the computing cost is reduced, and the computing amount is reduced.
In an embodiment of the present application, as shown in fig. 6, a task management method is provided, where the task management method includes:
step S601, receiving task management operation input by a user based on a human-computer interaction interface.
In an embodiment of the present application, the provided task management method is applied to a task management system provided in the foregoing embodiment, the task management system is connected to a human-computer interaction interface through an interface, and a user can manage each task in the task management system based on the human-computer interaction interface, where the task management system includes task execution, task state query, task rerun, task stop, and the like. The unique identification refers to an identification which can be specifically corresponding to a unique specific task; the general identifier refers to an identifier common to a specific task, such as the execution time of a task executed within a certain time period, and may be used to query the states of multiple tasks executed within the time period; the generic identification can be used for fuzzy queries and precise queries.
Step S602, determining a task to be executed based on the task management operation, calling a task storage interface to store the task to be executed to a cloud end, and calling a task calling interface to submit the task to be executed to a cloud computing platform.
In the embodiment of the application, the task scheduling interface encapsulates the minimum general requirement for task scheduling execution and is used for scheduling the task to be executed, and the minimum general requirement for task scheduling execution is used for standardizing the functional requirements of each computing platform for the task management system. The task storage interface is packaged with a minimum general requirement for task storage and execution and used for storing the tasks to be executed, and the minimum general requirement for task storage and execution is used for standardizing the functional requirements of each storage space for accessing the task management system. After the task to be executed is determined, based on the task to be executed, a task storage interface is called to store the task degree to a cloud end, and a task scheduling interface is called to submit the task to be executed to a cloud computing platform.
Step S603, allocating a container space for the task to be executed, loading a task shell program to be executed, and parsing the workflow description language of the task to be executed to obtain task data of the task to be executed.
In the embodiment of the application, during task execution, an independent container space needs to be allocated for a task, a preset task shell program is loaded, a workflow description language of the task to be executed is analyzed, task data of the task is obtained, and an execution sequence and a dependency relationship of the task are determined.
Step S604, executing the task data, and uploading an execution result to a cloud for storage.
In the embodiment of the application, task data are executed in the independent container space, and the execution result is uploaded to the cloud storage, so that the task is executed.
According to the embodiment of the application, tasks are scheduled and stored through independent interfaces, various cloud computing platforms and cloud storage platforms can be connected in an optional mode, the same tasks are called and executed in a mixed environment, the universality and the expansibility of a task management system are enhanced, the granularity of task management is increased to a container level through the design of a shell program, all preparation/ending work which is different from the cloud environment is processed by the shell program, the difference of service tools of different cloud manufacturers in the mixed cloud environment is shielded, a containerization technology is adapted to the outer layer of a computing program, cloud computing resources can be effectively utilized, the computing cost is reduced, and the computing amount is reduced.
As a possible implementation manner of the present application, in the implementation manner, after uploading the execution result to the cloud storage, the task management method further includes:
detecting whether the task to be executed is executed completely, and when the task to be executed is not executed completely, continuing to execute the task to be executed; and after the task to be executed is executed, detecting whether a subsequent task exists after the task to be executed, if so, calling the subsequent task, and continuing to execute the subsequent task.
In the embodiment of the application, the provided shell program is wrapped before and after the algorithm script or the computing software submitted by the user is executed, and when the shell program is executed, the shell program can be further used for updating the task state of the task to be executed according to the execution result of the task to be executed. For convenience of description, for a specific example, the tasks to be executed include joba 1 and joba 2, and when the task data of joba 1 is calculated in the container, the task execution shell program is executed according to the task state of joba 1.
In the embodiment of the application, the task execution shell program can increase the granularity of task management to a container level, all preparation and ending work which are different from the cloud environment are processed by the shell program, the difference between the mixed cloud environment and a cloud manufacturer service tool is shielded, and a containerization technology is adapted to the outer layer of a computing program.
As a possible implementation manner of the present application, in this implementation manner, in the task management method, a task includes a primary task, a secondary task, and a tertiary task, where each of the primary tasks includes one or more of the secondary tasks, each of the secondary tasks includes one or more of the tertiary tasks, and the interface method layer includes:
the first-level task interface module is used for providing management query service of the first-level task;
the second-level task interface module is used for providing management query service of the second-level task;
and the third-level task interface module is used for providing management query services of the third-level tasks, wherein the management query services comprise task submission services, task state query services, task rerun services and task stop services.
In the task management method provided by the embodiment of the application, when the task is managed through the interface method layer, the interface method layer provides a third-level task interface module, wherein the third-level task interface module comprises a first-level task interface module used for providing a management query service of the first-level task; the second-level task interface module is used for providing management query service of the second-level task; and the third-level task interface module is used for providing management query services of the third-level tasks, wherein the management query services comprise task submission services, task state query services, task rerun services and task stop services.
In an embodiment of the present application, when determining a workflow description language of the task to be executed and analyzing the workflow description language to obtain an execution sequence of the task to be executed, the task management method includes:
for a primary task to be executed, analyzing a workflow description language of the primary task to be executed, and determining an execution sequence of each secondary task to be executed in the primary task to be executed and a dependency relationship of each secondary task to be executed; and/or
And for one to-be-executed secondary task, determining the execution sequence of each to-be-executed tertiary task in the to-be-executed secondary task and the dependency relationship among the to-be-executed tertiary tasks.
In an embodiment of the present application, when a task is managed by using an analysis layer provided in the foregoing embodiment, the analysis layer includes a workflow analyzer, configured to analyze a workflow description language of the task to be executed, and determine an order of the task to be executed, a dependency relationship between the tasks, a parameter serialization method, and a parameter deserialization method; the task scheduling interface is encapsulated with a minimum general requirement for task scheduling execution and is used for scheduling the task to be executed, and the minimum general requirement for task scheduling execution is used for standardizing the functional requirements of each computing platform for the task management system; the task storage interface is packaged with a minimum general requirement for task storage and execution and used for storing the tasks to be executed, and the minimum general requirement for task storage and execution is used for standardizing the functional requirements of each storage space for accessing the task management system.
In an embodiment of the present application, when executing the task execution shell program, the provided task management method includes:
updating the task state of the task to be executed based on the execution result of the task to be executed;
detecting whether the task to be executed is executed completely, and when the task to be executed is not executed completely, continuing to execute the task to be executed; and after the task to be executed is executed, detecting whether a subsequent task exists after the task to be executed, if so, calling the subsequent task, and continuing to execute the subsequent task.
In the embodiment of the application, the provided task management method can also be used for receiving a task state query request input by a user based on a human-computer interaction interface; and sending the state of the corresponding task to the human-computer interaction interface for displaying based on the task state query request.
The implementation principle of the task management method provided in the embodiment of the present application is the same as the implementation principle of the task management system provided in the foregoing embodiment, and details are not repeated herein. Corresponding to the embodiment of the application function implementation method, the application also provides a task management device, electronic equipment and a corresponding embodiment.
Fig. 7 is a schematic structural diagram of a task management device according to an embodiment of the present application.
Referring to fig. 7, the task management device 70 includes a human-computer interaction module 710, a task calling module 720, a parsing module 730, and a task execution module 740, wherein:
the human-computer interaction module 710 is used for receiving task management operation input by a user based on a human-computer interaction interface;
the task calling module 720 is used for determining a task to be executed based on the task management operation, calling a task storage interface to store the task to be executed to a cloud end, and calling a task calling interface to submit the task to be executed to a cloud computing platform;
the analysis module 730 is configured to allocate a container space for the task to be executed, load a task shell program to be executed, analyze a workflow description language of the task to be executed, and obtain task data of the task to be executed;
and the task execution module 740 is configured to execute the task data and upload an execution result to the cloud for storage.
As a possible implementation manner of the present application, in this implementation manner, after the execution result is uploaded to the cloud storage, the task execution module 740 may further be configured to:
detecting whether the task to be executed is executed completely, and when the task to be executed is not executed completely, continuing to execute the task to be executed; and after the task to be executed is executed, detecting whether a subsequent task exists after the task to be executed, if so, calling the subsequent task, and continuing to execute the subsequent task.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Fig. 8 is a schematic structural diagram of an electronic device shown in an embodiment of the present application.
Referring to fig. 8, the electronic device 1000 includes a memory 1010 and a processor 1020.
The Processor 1020 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 1010 may include various types of storage units, such as system memory, Read Only Memory (ROM), and permanent storage. Wherein the ROM may store static data or instructions that are needed by the processor 1020 or other modules of the computer. The persistent storage device may be a read-write storage device. The persistent storage may be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the persistent storage device employs a mass storage device (e.g., magnetic or optical disk, flash memory) as the persistent storage device. In other embodiments, the permanent storage may be a removable storage device (e.g., floppy disk, optical drive). The system memory may be a read-write memory device or a volatile read-write memory device, such as a dynamic random access memory. The system memory may store instructions and data that some or all of the processors require at runtime. Further, the memory 1010 may comprise any combination of storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic and/or optical disks, may also be employed. In some embodiments, memory 1010 may include a removable storage device that is readable and/or writable, such as a Compact Disc (CD), a digital versatile disc read only (e.g., DVD-ROM, dual layer DVD-ROM), a Blu-ray disc read only, an ultra-dense disc, a flash memory card (e.g., SD card, min SD card, Micro-SD card, etc.), a magnetic floppy disk, or the like. The storage medium does not contain carrier waves and momentary electrical signals transmitted by wireless or wired transmission.
The memory 1010 has stored thereon executable code that, when processed by the processor 1020, may cause the processor 1020 to perform some or all of the methods described above.
Furthermore, the method according to the present application may also be implemented as a computer program or computer program product comprising computer program code instructions for performing some or all of the steps of the above-described method of the present application.
Alternatively, the present application may also be embodied as a read storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) having stored thereon executable code (or a computer program or computer instruction code) which, when executed by a processor of an electronic device (or a server, etc.), causes the processor to perform part or all of the steps of the above-described methods according to the present application.
Having described embodiments of the present application, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (11)
1. A task management system, characterized in that the system comprises:
the interface method layer is used for providing an interface connected with a human-computer interaction interface, receiving task management operation input by a user through the human-computer interaction interface based on the interface and determining a task to be executed;
the analysis layer is used for determining a workflow description language of the task to be executed, analyzing the workflow description language to obtain an execution sequence of the task to be executed, providing a task scheduling interface for scheduling the task to be executed, and providing a task storage interface for storing the task to be executed;
and the execution layer is used for providing computing resources for executing the tasks to be executed and storage resources for storing the tasks to be executed, providing a task execution shell program for downloading and executing the data of the tasks to be executed, and uploading an execution result after the execution is finished.
2. The task management system of claim 1, wherein tasks include primary tasks, secondary tasks, and tertiary tasks, wherein each of the primary tasks includes one or more of the secondary tasks, wherein each of the secondary tasks includes one or more of the tertiary tasks, and wherein the interface method layer comprises:
the first-level task interface module is used for providing management query service of the first-level task;
the second-level task interface module is used for providing management query service of the second-level task;
and the third-level task interface module is used for providing management query services of the third-level tasks, wherein the management query services comprise task submission services, task state query services, task rerun services and task stop services.
3. The task management system according to claim 2, wherein the parsing layer, when determining a workflow description language of the task to be executed, parsing the workflow description language, and obtaining an execution sequence of the task to be executed, is configured to:
for a primary task to be executed, analyzing a workflow description language of the primary task to be executed, and determining an execution sequence of each secondary task to be executed in the primary task to be executed and a dependency relationship of each secondary task to be executed; and/or
And for one to-be-executed secondary task, determining the execution sequence of each to-be-executed tertiary task in the to-be-executed secondary task and the dependency relationship among the to-be-executed tertiary tasks.
4. The task management system of claim 1, wherein the resolution layer comprises:
the workflow parser is used for parsing the workflow description language of the tasks to be executed and determining the sequence of the tasks to be executed, the dependency relationship among the tasks, a parameter serialization method and a parameter deserialization method;
the task scheduling interface is encapsulated with a minimum general requirement for task scheduling execution and is used for scheduling the task to be executed, and the minimum general requirement for task scheduling execution is used for standardizing the functional requirements of each computing platform for the task management system;
the task storage interface is packaged with a minimum general requirement for task storage and execution and used for storing the tasks to be executed, and the minimum general requirement for task storage and execution is used for standardizing the functional requirements of each storage space for accessing the task management system.
5. The task management system of claim 1, wherein the execution layer, when executing the task execution shell, is further configured to:
updating the task state of the task to be executed based on the execution result of the task to be executed;
detecting whether the task to be executed is executed completely, and when the task to be executed is not executed completely, continuing to execute the task to be executed; and after the task to be executed is executed, detecting whether a subsequent task exists after the task to be executed, if so, calling the subsequent task, and continuing to execute the subsequent task.
6. The task management system of claim 1, wherein the interface method layer is further operable to:
receiving a task state query request input by a user based on a human-computer interaction interface;
and sending the state of the corresponding task to the human-computer interaction interface for displaying based on the task state query request.
7. A method for task management, the method comprising:
receiving task management operation input by a user based on a human-computer interaction interface;
determining a task to be executed based on the task management operation, calling a task storage interface to store the task to be executed to a cloud end, and calling a task calling interface to submit the task to be executed to a cloud computing platform;
allocating container space for the task to be executed, loading a task shell program to be executed, analyzing a workflow description language of the task to be executed, and acquiring task data of the task to be executed;
and executing the task data, and uploading an execution result to a cloud end for storage.
8. The task management method according to claim 7, wherein after uploading the execution result to a cloud storage, the method further comprises:
detecting whether the task to be executed is executed completely, and when the task to be executed is not executed completely, continuing to execute the task to be executed; and after the task to be executed is executed, detecting whether a subsequent task exists after the task to be executed, if so, calling the subsequent task, and continuing to execute the subsequent task.
9. A task management apparatus, characterized in that the task management apparatus comprises:
the human-computer interaction module is used for receiving task management operation input by a user based on a human-computer interaction interface;
the task calling module is used for determining a task to be executed based on the task management operation, calling a task storage interface to store the task to be executed to a cloud end, and calling the task calling interface to submit the task to be executed to a cloud computing platform;
the analysis module is used for allocating container space for the task to be executed, loading a task shell program to be executed, analyzing the workflow description language of the task to be executed and acquiring task data of the task to be executed;
and the task execution module is used for executing the task data and uploading an execution result to the cloud storage.
10. An electronic device, comprising:
a processor; and
memory having stored thereon executable code which, when executed by the processor, causes the processor to perform a method of task management according to any of claims 7-8.
11. A storage medium having stored thereon executable code which, when executed by a processor of an electronic device, causes the processor to perform a task management method according to any one of claims 7-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111506788.XA CN114185660A (en) | 2021-12-10 | 2021-12-10 | Task management system, method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111506788.XA CN114185660A (en) | 2021-12-10 | 2021-12-10 | Task management system, method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114185660A true CN114185660A (en) | 2022-03-15 |
Family
ID=80543074
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111506788.XA Pending CN114185660A (en) | 2021-12-10 | 2021-12-10 | Task management system, method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114185660A (en) |
-
2021
- 2021-12-10 CN CN202111506788.XA patent/CN114185660A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9015724B2 (en) | Job dispatching with scheduler record updates containing characteristics combinations of job characteristics | |
| JP4891225B2 (en) | Dependency graph parameter scoping | |
| CN101233489B (en) | Adaptive process dispatch in a computer system having a plurality of processors | |
| CN100559348C (en) | Be used to provide operating system service method and computing system based on strategy | |
| CN113243005A (en) | Performance-based hardware emulation in on-demand network code execution systems | |
| JP5147728B2 (en) | Qualitatively annotated code | |
| US9535754B1 (en) | Dynamic provisioning of computing resources | |
| US20080148300A1 (en) | Providing Policy-Based Operating System Services in a Hypervisor on a Computing System | |
| TW201409228A (en) | Profiling application code to identify code portions for FPGA implementation | |
| US20170048008A1 (en) | Method and apparatus for verification of network service in network function virtualization environment | |
| CA2637749A1 (en) | Method, system, and program product for deploying a platform dependent application in a grid environment | |
| CN111176818B (en) | Distributed prediction method, device, system, electronic equipment and storage medium | |
| CN112835676A (en) | Deployment method and device of containerized application, computer equipment and medium | |
| KR101751515B1 (en) | Apparatus, method, and computer program for testing | |
| US7721278B2 (en) | Modular server architecture for multi-environment HTTP request processing | |
| CN113010265A (en) | Pod scheduling method, scheduler, memory plug-in and system | |
| US20160179570A1 (en) | Parallel Computing Without Requiring Antecedent Code Deployment | |
| US7539992B2 (en) | Scheduling method, program product for use in such method, and task scheduling apparatus | |
| Spinner et al. | Online model learning for self-aware computing infrastructures | |
| EP3724776A1 (en) | Method, function manager and arrangement for handling function calls | |
| Harichane et al. | KubeSC‐RTP: Smart scheduler for Kubernetes platform on CPU‐GPU heterogeneous systems | |
| WO2022109932A1 (en) | Multi-task submission system based on slurm computing platform | |
| EP3855313B1 (en) | Os optimized workflow allocation | |
| CN112749062A (en) | Server program monitoring method and device, computer equipment and storage medium | |
| CN117111948A (en) | Distributed compiling method and system, node and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |