CN115269140A - Container-based cloud computing workflow scheduling method, system and equipment - Google Patents

Container-based cloud computing workflow scheduling method, system and equipment Download PDF

Info

Publication number
CN115269140A
CN115269140A CN202210782251.4A CN202210782251A CN115269140A CN 115269140 A CN115269140 A CN 115269140A CN 202210782251 A CN202210782251 A CN 202210782251A CN 115269140 A CN115269140 A CN 115269140A
Authority
CN
China
Prior art keywords
task
workflow
container
tasks
ready
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210782251.4A
Other languages
Chinese (zh)
Inventor
沈玉龙
师瑞谦
吕亚龙
郭泽伟
陶玉东
李云豆
李泽玉
严煜昆
刘磊
薛天琳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qingdao Institute Of Computing Technology Xi'an University Of Electronic Science And Technology
Original Assignee
Qingdao Institute Of Computing Technology Xi'an University Of Electronic Science And Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qingdao Institute Of Computing Technology Xi'an University Of Electronic Science And Technology filed Critical Qingdao Institute Of Computing Technology Xi'an University Of Electronic Science And Technology
Priority to CN202210782251.4A priority Critical patent/CN115269140A/en
Publication of CN115269140A publication Critical patent/CN115269140A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5077—Logical partitioning of resources; Management or configuration of virtualized resources
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45562—Creating, deleting, cloning virtual machine instances
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45595—Network integration; Enabling network access in virtual machine instances
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/50—Indexing scheme relating to G06F9/50
    • G06F2209/5021—Priority

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention belongs to the technical field of cloud computing data processing, and discloses a container-based cloud computing workflow scheduling method, a system and equipment, wherein the container-based cloud computing workflow scheduling method comprises the steps that firstly, a container configuration decision module obtains the optimal container configuration of micro-service examples corresponding to various tasks through analysis according to the historical records of resources required by various tasks during processing; secondly, the workflow initialization module determines the initial priority of the tasks in the reached workflow and calculates the sub-deadline of each task according to the priority and the deadline constraint of the workflow; scheduling the tasks of the workflow according to a scheduling method based on the emergency degree of the ready tasks to generate a scheduling scheme; and finally, the processing result feedback module arranges the subsequent waiting tasks according to the scheduling result. The invention can quickly process the request of the user and return the result, thereby reducing the cost and improving the resource utilization rate under the condition of ensuring the service quality of the user.

Description

Container-based cloud computing workflow scheduling method, system and equipment
Technical Field
The invention belongs to the technical field of cloud computing data processing, and particularly relates to a container-based cloud computing workflow scheduling method, system and device.
Background
At present, most of application programs in the cloud are in a traditional single-body architecture system mode, and the applied scheduling method is designed for a single-body architecture. Due to the problems of high coupling, poor expansibility, low development efficiency and the like in the traditional single application architecture, the micro-service architecture is provided by software developers as a new application program development mode. Under the micro-service architecture, the original single system is split and designed into a group of small services, and each small service runs in the own process and communicates by using a lightweight mechanism (HTTP request/resource API call). These services are built around the business and can be independently developed, deployed, run, and iteratively updated. Each microservice may be written in a different programming language and use different data storage technologies. By decoupling the services, different service units are decomposed into micro-services with single functions, and developers only need to pay attention to the business logic of the services. Development teams of all service modules can work in parallel, and development efficiency and robustness of the system are improved. With the development of micro-service technology, a micro-service architecture is widely applied to application programs in the cloud, but most of scheduling methods for the architecture are that a micro-service sub-module independently occupies one virtual machine to operate, so that resource waste is easily caused; if the micro-service sub-modules share the virtual machine, the micro-service execution environments are easy to interfere with each other.
Through the above analysis, the problems and defects of the prior art are as follows: in the scheduling method in the prior art, the micro-service sub-module independently occupies one virtual machine for operation, so that more idle time slots are formed, and resource waste is easily caused; if the micro-service sub-modules share the virtual machine, the micro-service execution environments are easy to interfere with each other.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a container-based cloud computing workflow scheduling method, system and equipment.
The invention is realized in such a way, a container-based cloud computing workflow scheduling method, which comprises the following steps:
firstly, a container configuration decision module obtains the optimal container configuration of the micro-service instances corresponding to various tasks through analysis according to the historical records of the resources required by various tasks during processing; secondly, determining the initial priority of the tasks in the arrival workflow by a workflow initialization module, and calculating the sub-deadline of each task according to the priority and the deadline constraint of the workflow; scheduling the tasks of the workflow according to a scheduling method based on the emergency degree of the ready tasks to generate a scheduling scheme so as to reduce the cost; and finally, the processing result feedback module arranges the subsequent waiting task according to the scheduling result so as to quickly perform subsequent processing and reduce the waiting time of the subsequent ready task.
Further, the container-based cloud computing workflow scheduling method specifically includes:
firstly, obtaining the optimal container configuration of the micro-service instances corresponding to various tasks through analysis according to the historical records of resources required by the calculation work of the various tasks; through the configuration, the most appropriate resources can be allocated when the container is initialized, and waste caused by excessive resource allocation is avoided;
secondly, converting the request of the user into a DAG graph of the workflow by the workflow initialization module, and calculating the priority of the task in the workflow and the sub-deadline of the task in the workflow; setting a precursor task set and a successor task set for each task, and putting the precursor task set and the successor task set into corresponding ready queues and task pools; by the method, the influence of the execution time fluctuation among the tasks can be reduced, the execution sequence among the tasks is coordinated, and the resource utilization rate is improved;
step three, the task scheduling module based on the urgency degree is triggered to distribute the ready tasks when a new workflow arrives or a new ready task is generated after an allocated task is completed, so that the execution sequence among the tasks can be adjusted in real time, the occupied time of the waiting task on container resources is avoided, and the waiting time slot and idle gaps are reduced;
step four, after a ready task is executed, uncertainty of execution time disappears, the micro-service instance of the executed task updates the ready time, the execution time of the task can be adjusted in real time, optimal resources are distributed in real time according to the ready time and the sub-deadline of the task to execute, and the success rate of executing the task is improved;
step five, creating a new container and a virtual machine in the step three, and performing two-layer expansion of the container and the virtual machine so as to utilize virtual machine resources to the maximum extent, reduce the service time of the virtual machine and reduce the cost of the virtual machine; the two layers of expansion are used to facilitate independent execution of each task, interference in task execution is avoided, resources can be divided in a fine-grained manner, and waste of large blocks of resources is avoided.
Further, the step one specific process is as follows:
when f-type micro-service examples are shared, obtaining a container configuration scheme CC = { CC = { (CC) }jJ =1,2, ·, f }, where ccjAnd indicating the configuration scheme of the container corresponding to the type-II micro-service type.
Further, the second step specifically comprises the following steps:
when a user request reaches a cloud center, a workflow initialization module converts the user request into a DAG graph of a workflow, calculates the priority of tasks in the workflow according to a key path of the DAG graph, and calculates the sub-deadline of the tasks in the workflow according to the deadline, the arrival time and the calculated priority; setting a precursor task set and a successor task set for each task, defining the task with the precursor task set as an empty task as a ready task, putting the ready task into a ready queue Q, and putting the rest tasks as waiting tasks into a task pool PT.
Further, the third specific process comprises:
the task scheduling module based on the urgency degree is triggered to distribute the ready tasks after a new workflow arrives or a new ready task is generated after an assigned task is completed; calculating urgency U for all ready tasksiAs the basis of the ready task scheduling, the problem of scheduling priority of a plurality of ready tasks is solved;
one ready task is a task which is completed by a precursor task, and an entry task is always a ready task; and after the calculation is finished, selecting the task with the minimum urgency degree for scheduling, wherein the task scheduling fully utilizes the existing resources and how to create a new container, and different scheduling schemes are required under different conditions.
Further, the different scheduling schemes required by the different situations specifically include:
when an instance capable of completing the task on time exists, selecting one running task with least cost increase; when none of the existing instances is satisfied, initializing a new container operation task with CC configuration; when the mirror image required by creating the instance exists on a certain virtual machine, the container is preferentially deployed on the virtual machine.
Further, the fourth specific process of the step is as follows:
after a ready task is executed, the uncertainty of the execution time disappears, and the micro-service instance of the execution task updates the ready time; when the instance has the waiting task, updating the relevant parameters of the waiting task, executing the waiting task after receiving the parameters of the preposed task, and updating the dependency information of the subsequent task of the task; and when all the prepositive tasks of the subsequent tasks are completed, changing the tasks into new ready tasks, putting the new ready tasks into the ready queue Q, and executing the task scheduling process again.
Further, in the fourth step, the specific process of performing two-layer expansion of the container and the virtual machine includes:
1) Deploying the newly added containers to the virtual machines through a best fit algorithm, and adding new virtual machines to deploy when the virtual machines cannot meet the requirements of the remaining containers;
2) Sorting the rest containers in descending order according to the size of the required resources and the size of each virtual machine type according to the owned resources, and setting the type with the most resources as the type1(ii) a Deploy all remaining vessels in type1Obtaining a feasible solution S on the type virtual machine1And calculating the cost;
3) Type of right removing1Every other type of virtual machine typeiLet i be 2 for the first time, find a feasible solution Si-1Set to vm in the last created virtual machinexMethod deployment to typeiIn the first time, i is 2, a feasible solution S is obtained2Calculating the cost of a feasible solution;
4) Repeating the process 3) to obtain a feasible solution S with the lowest cost, judging whether resources are left for each virtual machine in the solution, replacing the resources with a lower virtual machine configuration container to further reduce the cost, and finally obtaining the feasible solution S.
Another object of the present invention is to provide a container-based cloud computing workflow scheduling system implementing the container-based cloud computing workflow scheduling method, the container-based cloud computing workflow scheduling system including:
the user sends a request and receives a return result;
the workflow layer is characterized in that a workflow initialization module determines the initial priority of tasks in the workflow and calculates the sub-deadline of each task according to the priority and the deadline constraint of the workflow; scheduling the tasks of the workflow according to a scheduling method based on the urgency of the ready tasks to generate a scheduling scheme so as to reduce the cost;
in the micro-service embodiment layer, a container configuration decision module obtains the optimal container configuration of the micro-service examples corresponding to various tasks through analysis according to the historical records of the resources required by the various tasks during processing;
and in the virtual machine layer, the processing result feedback module arranges the subsequent waiting task according to the scheduling result so as to quickly perform subsequent processing and reduce the waiting time of the subsequent ready task.
It is a further object of the invention to provide a computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of:
firstly, according to the historical records of resources required by the calculation work of various tasks, obtaining the optimal container configuration of the micro-service instances corresponding to the various tasks through analysis;
secondly, converting the request of the user into a DAG graph of the workflow by the workflow initialization module, and calculating the priority of the task in the workflow and the sub-deadline of the task in the workflow; setting a precursor task set and a successor task set for each task, and putting the precursor task set and the successor task set into corresponding ready queues and task pools;
step three, the task scheduling module based on the urgency degree is triggered to distribute the ready tasks after a new workflow arrives or a new ready task is generated after an allocated task is completed;
step four, after a ready task is executed, the uncertainty of the execution time disappears, and the micro-service instance of the execution task updates the ready time;
and step five, creating a new container and a new virtual machine in the step three, and performing two-layer expansion of the container and the virtual machine so as to utilize the virtual machine resources to the maximum extent, reduce the service time of the virtual machine and reduce the cost of the virtual machine.
In combination with the above technical solutions and the technical problems to be solved, please analyze the advantages and positive effects of the technical solutions to be protected in the present invention from the following aspects:
first, aiming at the technical problems existing in the prior art and the difficulty in solving the problems, the technical problems to be solved by the technical scheme of the present invention are closely combined with results, data and the like in the research and development process, and some creative technical effects are brought after the problems are solved. The specific description is as follows:
in order to solve the problems of high coupling, poor expansibility, low development efficiency, difficult maintenance and the like in the traditional single application architecture, a micro-service architecture is proposed and adopted by many commercial companies. In the microservice architecture, an application is split into a series of servlets that can be independently developed, deployed, run, and iterated. Lightweight container virtualization technology is the technology of choice for deploying microservice instances due to its fast start-up and low overhead. There are few optimizations for microservice workflows in current approaches and there is no integration of microservices with existing container technology to maximize resource utilization in application scenarios. Most methods only consider the traditional workflow scheduling problem in grid computing and cloud computing, and schedule workflow tasks onto virtual machines in infrastructure, i.e., service, mode. The invention aims to realize the combination of workflow task scheduling and container expansion and contraction and ensure the service quality of users; and the resource utilization rate in the cloud is improved, and the execution cost is reduced. The invention adopts the container as a carrier for the operation of the micro-service, isolates the interference of the operation environment through the container, and greatly reduces the initialization time of the micro-service and improves the execution efficiency by utilizing the characteristics of quick deployment and operation of the container. In addition, the execution of a single workflow is assumed for most task scheduling algorithms at the present stage, and the characteristics of the workflow are known in advance; the uncertainty in the real-time and task execution times of the workflow is not considered in the multi-workflow-aware algorithm, resulting in decisions made by the scheduling algorithm violating the workflow's expiration date and increasing costs. Therefore, the invention focuses on the task scheduling problem of real-time multi-workflow, and aims to make the scaling decision made by the scheduling algorithm (such as the configuration and the quantity of new service instances) optimal as much as possible on the basis of ensuring the QoS of the user so as to reduce the resource renting cost.
Secondly, considering the technical scheme as a whole or from the perspective of products, the technical effect and advantages of the technical scheme to be protected by the invention are specifically described as follows:
the invention can quickly process the request of the user and return the result, thereby reducing the cost and improving the resource utilization rate under the condition of ensuring the service quality of the user; the method can provide a mode of charging according to requirements for the user, and allocate appropriate resources according to the type of the resources required by the user task, thereby further improving the execution efficiency.
Third, as inventive supplementary proof of the claims of the present invention, the following important aspects are also presented:
the technical scheme of the invention overcomes the technical prejudice that: in the traditional scheduling scheme, a virtual machine is mostly used for resource allocation, although the scheduling scheme is simple, the cost is too high, and huge performance waste exists, so that the situation can be avoided by using a double-layer structure of the virtual machine and a container, a faster execution environment is provided for the initialization of micro-services, the execution waiting time is shortened, and the task execution success rate is improved.
Drawings
Fig. 1 is a flowchart of a container-based cloud computing workflow scheduling method according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a container-based cloud computing workflow scheduling system according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a container cloud platform architecture provided in an embodiment of the present invention;
FIG. 4 is a schematic diagram of an application workflow provided by an embodiment of the invention;
in the figure: 1. a user; 2. a workflow layer; 3. a microservice embodiment layer; 4. and a virtual machine layer.
FIG. 5 is a partial project diagram of the operation provided by an embodiment of the present invention;
FIG. 6 is a partial component schematic of a platform provided by an embodiment of the present invention;
fig. 7 is a schematic diagram of resource usage of each node in a cluster according to an embodiment of the present invention;
fig. 8 is a schematic diagram of information of each cluster provided in the embodiment of the present invention;
FIG. 9 is a schematic diagram of an add cluster interface provided by an embodiment of the present invention;
FIG. 10 is a diagram illustrating the overall usage of a cluster resource provided by an embodiment of the present invention;
FIG. 11 is a schematic diagram of a create service interface provided by an embodiment of the invention;
fig. 12 is a schematic diagram illustrating all service specific information of cluster management provided by an embodiment of the present invention;
fig. 13 is a schematic diagram of container group information executed by the cluster in the platform according to the embodiment of the present invention;
fig. 14 is a schematic diagram illustrating node information in a cluster according to an embodiment of the present invention;
fig. 15 is a schematic diagram of information such as allocated resources and usage provided by an embodiment of the present invention;
FIG. 16 is a diagram illustrating a scheduling situation of a scheduler according to an embodiment of the present invention;
fig. 17 is a schematic diagram illustrating monitoring of user request information according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
1. Illustrative embodiments are explained. This section is an explanatory embodiment expanding on the claims so as to fully understand how the present invention is embodied by those skilled in the art.
As shown in fig. 1, a method for scheduling a cloud computing workflow based on a container according to an embodiment of the present invention includes:
s101: and obtaining the optimal container configuration of the micro-service instances corresponding to the various tasks through analysis according to the historical records of the resources required by the computing work of the various tasks.
S102: the workflow initialization module converts the request of the user into a DAG graph of the workflow, and calculates the priority of the task in the workflow and the sub-deadline of the task in the workflow; and simultaneously setting a precursor task set and a subsequent task set for each task, and putting the precursor task set and the subsequent task set into the corresponding ready queue and the corresponding task pool.
S103: the task scheduling module based on the urgency is triggered to allocate a ready task when a new workflow arrives or a new ready task is generated after an allocated task is completed.
S104: after a ready task is executed, the uncertainty of the execution time disappears, and the micro-service instance executing the task updates the ready time.
S105: and creating a new container and a new virtual machine in the third step, and performing two-layer expansion of the container and the virtual machine so as to utilize the virtual machine resources to the maximum extent, reduce the service time of the virtual machine and reduce the cost of the virtual machine.
The specific process of S101 provided by the embodiment of the present invention is:
assuming that f-type micro-service instances are shared, a container configuration scheme CC = { CC = { can be obtainedjJ =1,2, ·, f }, where ccjAnd indicating the configuration scheme of the container corresponding to the type-II micro-service type.
The specific process of S102 provided by the embodiment of the present invention is:
when a user request reaches the cloud center, the workflow initialization module converts the user request into a DAG graph of the workflow, calculates the priority of the task in the workflow according to the key path of the DAG graph, and calculates the sub-deadline of the task in the workflow according to the deadline, the arrival time and the calculated priority. And then setting a precursor task set and a successor task set for each task, defining the task with the precursor task set as an empty task as a ready task, putting the ready task into a ready queue Q, and putting the rest tasks as waiting tasks into a task pool PT.
The specific process of S103 provided by the embodiment of the present invention is:
the task scheduling module based on the urgency degree is triggered to distribute the ready tasks after a new workflow arrives or a new ready task is generated after an assigned task is completed. In order to solve the scheduling priority problem of a plurality of ready tasks, all the ready tasks are processedCalculating the urgency UiAs the basis for the ready task scheduling. A ready task refers to a task whose predecessor has completed, and an ingress task is always a ready task. And after the calculation is finished, selecting the task with the minimum urgency degree for scheduling, wherein the task scheduling needs to consider how to fully utilize the existing resources and how to create a new container.
Different scheduling schemes are required for different situations, and the schemes for the three known situations are as follows: 1) If there are instances where the task can be completed on time, selecting the one with the least cost increase to run the task; 2) If none of the existing instances are satisfied, initializing a new container with CC configuration to run the task; 3) If the mirror image required for creating the instance exists on a certain virtual machine, the container is preferentially deployed on the virtual machine.
The specific process of S104 provided by the embodiment of the present invention is:
after a ready task is executed, the uncertainty of its execution time will disappear, and the micro-service instance executing the task updates its ready time. When the waiting task exists on the instance, the relevant parameters of the waiting task are updated, the execution is carried out after the parameters of the pre-task are received, the dependency information of the subsequent task of the task is updated, and when all the pre-tasks of the subsequent task are completed, the subsequent task is changed into a new ready task and is put into a ready queue Q, and the task scheduling process is carried out again.
In S104 provided by the embodiment of the present invention, a specific process of performing two-layer extension of a container and a virtual machine is as follows:
1) And deploying the newly added container to the virtual machine through a best fit algorithm, and adding the new virtual machine to deploy when the virtual machine can not meet the rest containers.
2) Sorting the rest containers in descending order according to the size of the required resources and the size of each virtual machine type according to the owned resources, and setting the type with the most resources as the type1(ii) a Deploy all remaining containers at type1Obtaining a feasible solution S on the virtual machine with the highest configuration1And calculates its cost.
3) For type1Of every other typeVirtual machine typeiLet i be 2 for the first time, find a feasible solution Si-1Set to vm in the last created virtual machinex(this virtual machine may have too many remaining resources, and needs to be replaced with a smaller virtual machine to reduce costs), deploy the method in step 2) to typeiIn (i is 2 for the first time), a feasible solution S is obtained2The cost of this feasible solution is calculated.
4) And repeating the process 3) to obtain a feasible solution S with the lowest cost, judging whether resources are left for each virtual machine in the solution, replacing the virtual machines with lower virtual machines, and arranging the containers to further reduce the cost to finally obtain the feasible solution S.
As shown in fig. 2, the cloud computing workflow scheduling system based on a container provided in the embodiment of the present invention includes:
user 1, sends the request and receives the return result.
A workflow layer 2, wherein a workflow initialization module determines the initial priority of the tasks in the arriving workflow and calculates the sub-deadline of each task according to the priority and the deadline constraint of the workflow; and scheduling the tasks of the workflow according to a scheduling method based on the emergency degree of the ready tasks to generate a scheduling scheme so as to reduce the cost.
And in the micro-service embodiment layer 3, the container configuration decision module obtains the optimal container configuration of the micro-service examples corresponding to various tasks through analysis according to the history of the resources required during the processing of various tasks.
And in the virtual machine layer 4, the processing result feedback module arranges the subsequent waiting task according to the scheduling result so as to quickly perform subsequent processing and reduce the waiting time of the subsequent ready task.
2. Application examples. In order to prove the creativity and the technical value of the technical scheme of the invention, the part is the application example of the technical scheme of the claims on specific products or related technologies.
The scheme has a specific product, wherein part of running projects are shown in fig. 5, part of components of the platform are shown in fig. 6, and the resource utilization condition of each node in the cluster is shown in fig. 7, such as the indexes of CPU utilization, CPU average load, memory utilization, container group utilization and the like.
3. Evidence of the relevant effects of the examples. The embodiment of the invention achieves some positive effects in the process of research and development or use, and has great advantages compared with the prior art, and the following contents are described by combining data, diagrams and the like in the test process.
The computing cluster in the container service cloud platform can provide convenient and fast cluster computing tasks for users. The user only needs to upload the target data and the task file, the cluster calculation can be automatically carried out by the platform, and the calculation result is provided for the user. The user does not need to be concerned with cluster creation, environment configuration, etc. work that is not related to the computing task, and focuses on the computing itself. The test platform uses 32 virtual machines to build three clusters, the clusters are managed through a federal mechanism, one cluster is a Host cluster, the rest clusters are Member clusters, and information of each cluster, such as cluster names, the number of nodes in the clusters, kubernets version used by the clusters and the like, is shown in figure 8. When a new cluster needs to be managed, an addition to an existing cluster may be performed on an add cluster page to dynamically expand management of more clusters, and an add cluster interface is shown in fig. 9.
Clicking the cluster name can enter a corresponding cluster management interface, each cluster management comprises the functions of node management, service components, project management, application load, storage management, monitoring alarm and the like, convenient management operation is provided for service providers, application developers and software system administrators, and the overview interface displays the overall use condition of cluster resources in a centralized manner, such as the cluster CPU utilization rate, the memory utilization rate, the container group utilization rate and the local storage utilization rate. The specific information is shown in fig. 10.
In micro-service task scheduling and container scaling, in order to process a user's request, a service for executing a task needs to be issued to a platform first. The service is a micro-service mirror image developed by a group of application developers, the application developers can add own services on the platform, information such as service names, function introduction, belonging types, jar packages of the services and the like needs to be provided during adding, the platform can manage the services in all directions after the adding is completed, and an adding interface is shown in fig. 11. All service specific information of cluster management is shown in fig. 12.
In a Kubernets cluster, two components, namely a kube-scheduler and a kube-controller-manager, are used for managing functions such as task scheduling and elastic expansion of a pod, the components have self-defined interfaces, and a developer can modify rules of a scheduler and a controller according to project requirements to achieve customized management of a project. The proposed solution requires decisions to be made in case the state of existing containers and the real-time information of nodes are both clear, and therefore the container group and node operation needs to be monitored. The container group information for all runs in the platform is shown in fig. 13. In the micro-service task scheduling process, the scheduling algorithm can preferentially select the existing container service to execute the corresponding task, and when the container group does not meet the condition, the kube-scheduler can create a new container group and select the most appropriate node for deployment through the algorithm. During deployment, nodes are selected according to various information comprehensive evaluations of each node, the node information includes CPU utilization, memory utilization, and container group usage, the specific information is as shown in fig. 14, each node has a different container group to execute tasks, the load of the container group corresponds to the operating state of the node, and the information such as allocated resources and usage is as shown in fig. 15.
In the scheduling process, the platform also monitors the scheduler and the user request, checks the scheduling condition and the request pressure at different moments, and dynamically adjusts the scheduling strategy. Scheduler scheduling is shown in fig. 16, and user request information monitoring is shown in fig. 17.
It should be noted that the embodiments of the present invention can be realized by hardware, software, or a combination of software and hardware. The hardware portions may be implemented using dedicated logic; the software portions may be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or specially designed hardware. It will be appreciated by those skilled in the art that the apparatus and methods described above may be implemented using computer executable instructions and/or embodied in processor control code, for example such code provided on a carrier medium such as a diskette, CD-or DVD-ROM, a programmable memory such as read-only memory (firmware) or a data carrier such as an optical or electronic signal carrier. The apparatus and its modules of the present invention may be implemented by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of hardware circuits and software, e.g., firmware.
The above description is only for the purpose of illustrating the present invention and the appended claims are not to be construed as limiting the scope of the invention, which is intended to cover all modifications, equivalents and improvements that are within the spirit and scope of the invention as defined by the appended claims.

Claims (10)

1. A container-based cloud computing workflow scheduling method is characterized by comprising the following steps:
firstly, a container configuration decision module obtains the optimal container configuration of the micro-service instances corresponding to various tasks through analysis according to the historical records of the required resources during the processing of various tasks; secondly, determining the initial priority of the tasks in the arrival workflow by a workflow initialization module, and calculating the sub-deadline of each task according to the priority and the deadline constraint of the workflow; scheduling the tasks of the workflow according to a scheduling method based on the emergency degree of the ready tasks to generate a scheduling scheme so as to reduce the cost; and finally, the processing result feedback module arranges the subsequent waiting task according to the scheduling result so as to quickly perform subsequent processing and reduce the waiting time of the subsequent ready task.
2. The container-based cloud computing workflow scheduling method according to claim 1, wherein the container-based cloud computing workflow scheduling method specifically comprises:
firstly, according to the historical records of resources required by the calculation work of various tasks, obtaining the optimal container configuration of the micro-service instances corresponding to the various tasks through analysis;
step two, the workflow initialization module converts the request of the user into a DAG graph of the workflow, and calculates the priority of the task in the workflow and the sub-deadline of the task in the workflow; setting a precursor task set and a successor task set for each task, and putting the precursor task set and the successor task set into corresponding ready queues and task pools;
step three, the task scheduling module based on the urgency degree is triggered to distribute the ready tasks after a new workflow arrives or a new ready task is generated after an allocated task is completed;
step four, after a ready task is executed, the uncertainty of the execution time disappears, and the micro-service instance of the execution task updates the ready time;
and step five, creating a new container and a new virtual machine in the step three, and performing two-layer expansion of the container and the virtual machine so as to utilize the virtual machine resources to the maximum extent, reduce the service time of the virtual machine and reduce the cost of the virtual machine.
3. The container-based cloud computing workflow scheduling method according to claim 2, wherein the step one specific process is as follows:
when f-type micro-service examples are shared, obtaining a container configuration scheme CC = { CC = { (CC) }jJ =1,2, ·, f }, where ccjAnd indicating the configuration scheme of the container corresponding to the type-II micro-service type.
4. The cloud computing workflow scheduling method based on containers according to claim 2, wherein the specific process of the second step is as follows:
when a user request reaches a cloud center, a workflow initialization module converts the user request into a DAG graph of a workflow, calculates the priority of tasks in the workflow according to a key path of the DAG graph, and calculates the sub-deadline of the tasks in the workflow according to the deadline, the arrival time and the calculated priority; setting a precursor task set and a successor task set for each task, defining the task with the precursor task set as an empty task as a ready task, putting the ready task into a ready queue Q, and putting the rest tasks as waiting tasks into a task pool PT.
5. The container-based cloud computing workflow scheduling method according to claim 2, wherein the specific process of the third step is as follows:
the task scheduling module based on the urgency degree is triggered to allocate a ready task after a new workflow arrives or a new ready task is generated after an allocated task is completed; calculating urgency U for all ready tasksiThe scheduling priority problem of a plurality of ready tasks is solved as the basis of the scheduling of the ready tasks; one ready task is a task that the predecessor task has completed, and the entry task is always one ready task; and after the calculation is finished, selecting the task with the minimum urgency degree for scheduling, wherein the task scheduling fully utilizes the existing resources and how to create a new container, and different scheduling schemes are required under different conditions.
6. The container-based cloud computing workflow scheduling method of claim 5, wherein the different scheduling schemes required for the different situations specifically comprise:
when an instance capable of completing the task on time exists, selecting one running task with least cost increase; when none of the existing instances is satisfied, initializing a new container operation task with CC configuration; when the mirror image required by creating the instance exists on a certain virtual machine, the container is preferentially deployed on the virtual machine.
7. The container-based cloud computing workflow scheduling method according to claim 2, wherein the specific process of the fourth step is as follows:
after a ready task is executed, the uncertainty of the execution time disappears, and the micro-service instance of the execution task updates the ready time; when the instance has the waiting task, updating the relevant parameters of the waiting task, executing the parameters after receiving the parameters of the preposed task, and updating the dependency information of the successor task of the task; and when all the prepositive tasks of the subsequent tasks are completed, changing the tasks into new ready tasks, putting the new ready tasks into the ready queue Q, and executing the task scheduling process again.
8. The method for scheduling the cloud computing workflow based on the container according to claim 2, wherein in the fourth step, the specific process of performing two-layer expansion of the container and the virtual machine is as follows:
1) Deploying the newly added containers to the virtual machines through a best fit algorithm, and adding new virtual machines to deploy when the virtual machines cannot meet the requirements of the remaining containers;
2) Sorting the rest containers in descending order according to the size of the required resources and the size of each virtual machine type according to the owned resources, and setting the type with the most resources as the type1(ii) a Deploy all remaining vessels in type1Obtaining a feasible solution S on the type virtual machine1And calculating the cost;
3) For type1Every other type of virtual machine typeiIf i is 2 for the first time, a feasible solution S is foundi-1Set to vm for the last created virtual machinexMethod deployment to typeiIn the first time, i is 2, a feasible solution S is obtained2Calculating the cost of a feasible solution;
4) Repeating the process 3) to obtain a feasible solution S with the lowest cost, judging whether resources are left for each virtual machine in the solution, replacing the resources with a lower virtual machine configuration container to further reduce the cost, and finally obtaining the feasible solution S.
9. A container-based cloud computing workflow scheduling system implementing the container-based cloud computing workflow scheduling method according to any one of claims 1 to 8, wherein the container-based cloud computing workflow scheduling system comprises:
the user sends a request and receives a return result;
the workflow layer is characterized in that a workflow initialization module determines the initial priority of tasks in the workflow and calculates the sub-deadline of each task according to the priority and the deadline constraint of the workflow; scheduling the tasks of the workflow according to a scheduling method based on the emergency degree of the ready tasks to generate a scheduling scheme so as to reduce the cost;
in the micro-service embodiment layer, a container configuration decision module obtains the optimal container configuration of the micro-service examples corresponding to various tasks through analysis according to the historical records of the resources required by the various tasks during processing;
and in the virtual machine layer, the processing result feedback module arranges the subsequent waiting task according to the scheduling result so as to quickly perform subsequent processing and reduce the waiting time of the subsequent ready task.
10. A computer device, characterized in that the computer device comprises a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to carry out the steps of:
firstly, obtaining the optimal container configuration of the micro-service instances corresponding to various tasks through analysis according to the historical records of resources required by the calculation work of the various tasks;
step two, the workflow initialization module converts the request of the user into a DAG graph of the workflow, and calculates the priority of the task in the workflow and the sub-deadline of the task in the workflow; setting a precursor task set and a successor task set for each task, and putting the precursor task set and the successor task set into corresponding ready queues and task pools;
step three, the task scheduling module based on the urgency degree is triggered to distribute the ready tasks after a new workflow arrives or a new ready task is generated after an allocated task is completed;
step four, after a ready task is executed, the uncertainty of the execution time disappears, and the micro-service instance of the execution task updates the ready time;
and step five, creating a new container and a new virtual machine in the step three, and performing two-layer expansion of the container and the virtual machine so as to utilize the virtual machine resources to the maximum extent, reduce the service time of the virtual machine and reduce the cost of the virtual machine.
CN202210782251.4A 2022-07-05 2022-07-05 Container-based cloud computing workflow scheduling method, system and equipment Pending CN115269140A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210782251.4A CN115269140A (en) 2022-07-05 2022-07-05 Container-based cloud computing workflow scheduling method, system and equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210782251.4A CN115269140A (en) 2022-07-05 2022-07-05 Container-based cloud computing workflow scheduling method, system and equipment

Publications (1)

Publication Number Publication Date
CN115269140A true CN115269140A (en) 2022-11-01

Family

ID=83763350

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210782251.4A Pending CN115269140A (en) 2022-07-05 2022-07-05 Container-based cloud computing workflow scheduling method, system and equipment

Country Status (1)

Country Link
CN (1) CN115269140A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116149828A (en) * 2023-04-19 2023-05-23 南京信息工程大学 Cloud workflow scheduling method based on dynamically adjustable sub-expiration date

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116149828A (en) * 2023-04-19 2023-05-23 南京信息工程大学 Cloud workflow scheduling method based on dynamically adjustable sub-expiration date

Similar Documents

Publication Publication Date Title
US8656404B2 (en) Statistical packing of resource requirements in data centers
US8423646B2 (en) Network-aware virtual machine migration in datacenters
Hashem et al. MapReduce scheduling algorithms: a review
US20140156847A1 (en) Service Allocation in a Distributed Computing Platform
US11650654B2 (en) Managing power resources for pools of virtual machines
CN109992418B (en) SLA-aware resource priority scheduling method and system for multi-tenant big data platform
CN114741207A (en) GPU resource scheduling method and system based on multi-dimensional combination parallelism
CN113110914A (en) Internet of things platform construction method based on micro-service architecture
CN115934362B (en) Deep learning-oriented server non-perception computing cluster scheduling method and product
Luo et al. Erms: Efficient resource management for shared microservices with SLA guarantees
Baresi et al. PAPS: A serverless platform for edge computing infrastructures
CN116010064A (en) DAG job scheduling and cluster management method, system and device
Nastic et al. A serverless computing fabric for edge & cloud
Faragardi et al. A time-predictable fog-integrated cloud framework: One step forward in the deployment of a smart factory
CN115269140A (en) Container-based cloud computing workflow scheduling method, system and equipment
Chen et al. Task partitioning and offloading in IoT cloud-edge collaborative computing framework: a survey
Li et al. Cost-efficient fault-tolerant workflow scheduling for deadline-constrained microservice-based applications in clouds
Hung et al. Task scheduling for optimizing recovery time in cloud computing
US8813088B2 (en) Scheduling flows in a multi-platform cluster environment
CN113515356B (en) Lightweight distributed resource management and task scheduler and method
Kraemer et al. Reducing the number of response time service level objective violations by a cloud‐HPC convergence scheduler
Zhang et al. Cost-efficient and latency-aware workflow scheduling policy for container-based systems
Filippini et al. SPACE4AI-R: a runtime management tool for AI applications component placement and resource scaling in computing continua
CN113742646A (en) Compiling a single language compound function into a single entity
Stavrinides et al. Workload scheduling in fog and cloud environments: emerging concepts and research directions

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination