WO2024036940A1 - 一种容器管理方法及相关设备 - Google Patents

一种容器管理方法及相关设备 Download PDF

Info

Publication number
WO2024036940A1
WO2024036940A1 PCT/CN2023/081285 CN2023081285W WO2024036940A1 WO 2024036940 A1 WO2024036940 A1 WO 2024036940A1 CN 2023081285 W CN2023081285 W CN 2023081285W WO 2024036940 A1 WO2024036940 A1 WO 2024036940A1
Authority
WO
WIPO (PCT)
Prior art keywords
node
life cycle
container
management system
container set
Prior art date
Application number
PCT/CN2023/081285
Other languages
English (en)
French (fr)
Inventor
艾拓
王雷博
孙涛
黄毽
Original Assignee
华为云计算技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202211507530.6A external-priority patent/CN117632464A/zh
Application filed by 华为云计算技术有限公司 filed Critical 华为云计算技术有限公司
Publication of WO2024036940A1 publication Critical patent/WO2024036940A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]

Definitions

  • This application relates to the field of cloud computing technology, and in particular to a container management method, system, computing device cluster, computer-readable storage medium, and computer program product.
  • Containers are executable units of software that encapsulate application code and its libraries and dependencies in a common way, so the container can be run anywhere and at any time.
  • a container orchestration platform can be used to manage the entire life cycle of containers.
  • the container orchestration platform can perform image distribution, redundant deployment, health monitoring, resource allocation, elastic scaling, load balancing and scheduling of containers.
  • Container orchestration platforms can usually classify multiple containers into “container sets", recorded as pods, and then use pods as the smallest scheduling unit to run workloads and provide required networking and storage services for these pods. Considering that workloads can constantly change, the container orchestration platform can adjust the number of pods in real time so that the overall number of pods is sufficient to support business pressure. Furthermore, when the container orchestration platform adjusts the number of pods, it can also adjust the number of nodes used to deploy pods.
  • This application provides a container management method that schedules container sets at nodes based on the life cycle of the container set and the life cycle of the node, thereby realizing on-demand use of node resources, avoiding resource waste, and reducing business costs.
  • This application also provides corresponding container management systems, computing device clusters, computer-readable storage media, and computer program products.
  • this application provides a container management method.
  • This method is executed by the container management system.
  • the container management system may be a system used to manage container sets (pods) deployed to the business cluster or pods to be deployed to the business cluster.
  • the container management system can be a plug-in, component or module integrated into the container orchestration platform, or it can be independent software.
  • the software system can be deployed in a computing device cluster, and the computing device cluster executes the software.
  • the program code of the system is used to execute the container management method of this application.
  • the container management system is a hardware system, such as a computing device cluster with container management functions, the container management system can execute the container management method of the present application at runtime.
  • the container management system can obtain the life cycle of at least one node and the life cycle of at least one pod in the business cluster.
  • the node is used to deploy pods.
  • the node can be a virtual machine (VM) node, and then the container management The system determines the target node based on the life cycle of at least one node and the life cycle of at least one pod.
  • the target node is the node where the pod is to be deployed, or the node where the pod is to be deleted. Then the container management system scales the above pod on the target node. .
  • the container management system combines the life cycle of the nodes in the business cluster and the life cycle of the pod to be deployed or the deployed pod to scale the pod, avoiding the separation of the elastic scaling of the pod and the elastic scaling of the node, making the cluster
  • the elastic scaling controller cluster autoscaler, CA
  • CA can realize the on-demand use of node resources and reduce business costs.
  • At least one pod includes a pod to be deployed.
  • the container management system can determine the life cycle of the pod to be deployed and at least one node. The proximity of the life cycle, and then determine the target node from at least one node based on the proximity. Accordingly, the container management system can schedule the pod to be deployed to the target node.
  • This method schedules the pod to be deployed to a target node whose remaining life cycle is close to the pod's life cycle, so that when the pod is deleted (shrunk), other pods on the target node have also been deleted or are about to be deleted.
  • the target Nodes can be released as soon as possible, reducing resource waste and business costs.
  • the container management system provides multiple ways to determine the target node. Specifically, the container management system may sort the at least one node according to proximity, and determine the target node from the at least one node according to the sorting result. For example, the container management system can determine the closest node as the target node based on the sorting results. The container management system may also score the at least one node based on proximity, and determine the target node from the at least one node based on the score of the at least one node. For example, the container management system may determine the node with the highest score or a score greater than a preset score as the target node.
  • the method of determining the target node based on sorting is relatively simple, easy to implement, and requires lower computing power.
  • the method of determining the target node based on scoring is relatively more accurate and can determine a more reasonable target node and schedule the pod to the target node. It can minimize resource waste and reduce business costs.
  • At least one node includes the first node, and at least one pod includes the first pod.
  • the score of the first node is positively related to the first proximity.
  • the first proximity is determined based on the ratio of the life cycle of the first pod to the life cycle of the first node.
  • the score of the first node is positively related to the second proximity.
  • the second proximity is determined based on the ratio of the life cycle of the first node to the life cycle of the first pod. .
  • This method uses corresponding rules to determine the score of the node when the pod life cycle is less than the life cycle of the node and when the pod life cycle is not less than the life cycle of the node, which improves the accuracy of the node score and recommends appropriate target nodes. Lay the foundation.
  • the container management system may determine a candidate second node based on the life cycle of the at least one node and the life cycle of the container set on the at least one node, and then determine the candidate second node. At least one candidate deletion order of the container set on the candidate second node is used to predict the revenue of deleting the container set on the second node according to the candidate deletion order. Among them, the revenue can be determined based on the resource utilization on the cluster. Then, the container management system determines the target deletion order according to the profit, and determines the target node from the candidate second nodes. In this way, when the container management system scales the pod, it can adjust the target according to the target deletion order. The deletion order of the second container set on the node, and the second container set on the target node is deleted according to the adjusted deletion order.
  • the container management system intelligently analyzes the global shrinking order of pods and optimizes the shrinking order to fundamentally solve the problem of node resource fragmentation, improve resource utilization, and reduce business costs.
  • the life cycle of the candidate second node is greater than the first duration, and the life cycle of the container set on the candidate second node is greater than the second duration.
  • the candidate second node may be a long-cycle node
  • the pod on the candidate second node may be a long-cycle pod.
  • a candidate second node could be a long-lived pod remaining in the trough.
  • the life cycle of a long-life node is greater than the first duration
  • the life cycle of a long-life pod is greater than the second duration.
  • the first duration and the second duration can be set based on experience values. In some examples, the first duration and the second duration may be set to be equal, or set to be unequal.
  • long-term nodes and long-term pods are usually not deleted during the low period of business. In contrast, elastic nodes and elastic pods can be deleted during the low period of business.
  • This method can reduce the number of traversals during shrinkage sequence optimization and improve the efficiency of shrinkage optimization by identifying long-period nodes as candidate nodes.
  • the container management system supports periodic shrinkage optimization or real-time shrinkage optimization. Specifically, when adjusting the deletion order of the second pod on the target node, the container management system may periodically adjust the deletion order of the second pod during the low period of business. The container management system can also adjust the deletion order of the second pod according to the deletion order adjustment strategy analyzed in real time before the business reaches a low period.
  • the amount of calculation required for periodic shrinkage optimization is relatively small and can improve resource utilization at a small cost.
  • Real-time shrinkage optimization requires real-time calculation of the deletion sequence adjustment strategy to obtain better optimization results.
  • the container management system can obtain the survival time distribution of the replicas in the replica set corresponding to the at least one pod in the historical time period, and then based on the survival time distribution of the replicas in the replica set corresponding to the at least one pod in the historical time period.
  • the lifetime distribution of the pod is used to predict the life cycle of at least one pod through statistical strategies.
  • This method realizes the life cycle portrait of pod based on the survival time distribution of historical time periods, has high reliability, and provides a basis for life cycle-based scheduling.
  • the statistical strategy includes one or more of machine learning, quantile, mean, maximum, or probability distribution.
  • the container management system can select statistical strategies corresponding to the business based on business characteristics, thereby accurately predicting the life cycle of pods and providing a reference for life cycle-based scheduling strategies.
  • the container management system determines the life cycle of the at least one node based on the life cycle of the pod on the at least one node and the creation time of the pod on the at least one node.
  • life cycle portraits of nodes can be achieved with high reliability, providing a basis for life cycle-based scheduling.
  • the container management system is deployed in a scheduler.
  • the scheduler provides container management capabilities such as life cycle-based scheduling and shrinking sequence optimization, which can reduce the impact on other businesses and reduce intrusion.
  • the container management system can be deployed on different devices in a distributed manner, and different modules in the container management system interact through an application programming interface API server. In this way, risks can be dispersed and the reliability of the entire container management system can be improved.
  • the sequence optimization module in the container management system can be an independent plug-in, or can be obtained by modifying the kernel of the container orchestration platform.
  • independent plug-ins have good compatibility, can be applied to different platforms, and can meet the needs of users on different platforms. Modifying the kernel of the container orchestration platform to implement a sequence optimization module can simplify user operations and improve user experience.
  • this application provides a container management system.
  • the container management system is used to manage a container set deployed to a business cluster or a container set to be deployed to the business cluster.
  • the container set includes a group of containers.
  • the system includes:
  • a life cycle portrait module used to obtain the life cycle of at least one node in the business cluster and the life cycle of at least one container set, and the node is used to deploy the container set;
  • a life cycle scheduling module configured to determine a target node based on the life cycle of the at least one node and the life cycle of the at least one container set.
  • the target node is the node where the container set is to be deployed, or the node where the container set is to be deleted.
  • the life cycle scheduling module is also used to scale the container set on the target node.
  • the at least one container set includes the container set to be deployed, and the life cycle scheduling module is specifically used to:
  • the life cycle scheduling module is specifically used for:
  • the life cycle scheduling module is specifically used to:
  • the at least one node is scored according to the proximity, and a target node is determined from the at least one node according to the score of the at least one node.
  • the at least one node includes a first node, and the at least one container set includes a first container set;
  • the score of the first node is positively correlated with the first proximity, and the first proximity is based on the life cycle of the first container set.
  • the ratio of the period to the life cycle of the first node is determined;
  • the score of the first node is positively correlated with the second proximity, and the second proximity is based on the life cycle of the first node.
  • the ratio of the cycle to the life cycle of the first container set is determined.
  • the system also includes:
  • a sequence optimization module configured to determine a candidate second node based on the life cycle of the at least one node and the life cycle of the container set on the at least one node; determine at least one container set on the candidate second node Candidate deletion order, predict the revenue from deleting the container set on the second node according to the candidate deletion order, the revenue is determined based on the resource utilization on the cluster; based on the revenue, determine the target deletion order;
  • the life cycle scheduling module is specifically used for:
  • the life cycle scheduling module is specifically used for:
  • Adjust the deletion order of the second container set on the target node according to the target deletion order and delete the second container set on the target node according to the adjusted deletion order.
  • the life cycle scheduling module is specifically used to:
  • the deletion order of the second container set is adjusted according to the real-time analysis deletion order adjustment strategy.
  • the life cycle portrait module is specifically used to:
  • the life cycle of the at least one container set is predicted through a statistical strategy.
  • the life cycle portrait module is specifically used to:
  • the life cycle of the at least one node is determined according to the life cycle of the container set on the at least one node and the creation time of the container set on the at least one node.
  • the container management system is deployed in a scheduler.
  • the container management system is deployed on different devices in a distributed manner, and different modules in the container management system interact through an application programming interface API server.
  • the sequence optimization module in the container management system is an independent plug-in, or is obtained by modifying the kernel of the container orchestration platform.
  • this application provides a computing device cluster.
  • the cluster of computing devices includes at least one computing device including at least one processor and at least one memory.
  • the at least one processor and the at least one memory communicate with each other.
  • the at least one processor is configured to execute instructions stored in the at least one memory, so that the computing device or the computing device cluster executes the container management method as described in the first aspect or any implementation of the first aspect.
  • the present application provides a computer-readable storage medium in which instructions are stored, and the instructions instruct a computing device or a cluster of computing devices to execute the above-mentioned first aspect or any one of the first aspects. Implement the container management method described in the method.
  • the present application provides a computer program product containing instructions that, when run on a computing device or a cluster of computing devices, cause the computing device or a cluster of computing devices to execute the first aspect or any one of the first aspects. Implement the container management method described in the method.
  • Figure 1 is a schematic diagram of a pod horizontal elastic scaling controller controlling the number of pods provided by an embodiment of the present application
  • Figure 2 is a schematic diagram of pod elastic scaling and node elastic scaling provided by the embodiment of the present application.
  • Figure 3 is a schematic diagram of pod scheduling based on a boxing scheduling strategy provided by an embodiment of the present application
  • Figure 4 is a schematic diagram of scaling down based on a scaling down strategy provided by an embodiment of the present application
  • Figure 5 is a schematic diagram of pod scheduling based on life cycle provided by the embodiment of the present application.
  • Figure 6 is a schematic diagram of pod scheduling based on life cycle provided by the embodiment of the present application.
  • Figure 7 is a schematic diagram of life cycle-based shrinkage provided by an embodiment of the present application.
  • Figure 8 is a system architecture diagram of a container management system provided by an embodiment of the present application.
  • Figure 9 is a schematic structural diagram of a scheduler-based container management system provided by an embodiment of the present application.
  • Figure 10 is a schematic structural diagram of a plug-in-based container management system provided by an embodiment of the present application.
  • Figure 11 is a schematic structural diagram of a kernel-based container management system provided by an embodiment of the present application.
  • Figure 12 is a flow chart of a container management method provided by an embodiment of the present application.
  • Figure 13 is a schematic diagram of statistical analysis of pod survival time in a replica set provided by an embodiment of the present application.
  • Figure 14 is a schematic diagram of a pod copy using stack tracing provided by an embodiment of the present application.
  • Figure 15 is a schematic diagram of the life length distribution of a pod copy provided by an embodiment of the present application.
  • Figure 16 is a schematic diagram of a periodic optimization shrinking sequence provided by an embodiment of the present application.
  • Figure 17 is a schematic diagram of a real-time optimization shrinking sequence provided by an embodiment of the present application.
  • Figure 18 is a schematic structural diagram of a computing device provided by an embodiment of the present application.
  • Figure 19 is a schematic structural diagram of a computing device cluster provided by an embodiment of the present application.
  • Figure 20 is a schematic structural diagram of a computing device cluster provided by an embodiment of the present application.
  • Figure 21 is a schematic structural diagram of a computing device cluster provided by an embodiment of the present application.
  • first and second in the embodiments of this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, features defined as “first” and “second” may explicitly or implicitly include one or more of these features.
  • Node The smallest computing hardware unit.
  • a node can be a separate computer (also called a computing device).
  • the computer can be a physical host such as a server or terminal.
  • the server can be a cloud server, edge server or local server.
  • Cloud server refers to a server in a cloud environment, such as a central server in a central computing cluster.
  • Edge servers refer to servers in edge environments, such as edge servers in edge computing clusters.
  • a local server refers to a server in a local data center. Terminals include but are not limited to desktop computers, laptop computers, and smartphones.
  • the computer can also be a virtual host virtualized by a virtualization service on the physical host, also called a virtual machine (VM).
  • VM virtual machine
  • Cluster A collection of nodes. Nodes in a cluster usually work together, so the cluster can be considered a single system. Each node in the cluster can be set up to perform the same tasks and controlled and scheduled by software, thereby increasing availability and scalability. In this application, each node in the cluster can provide the same service, so the cluster can also be called a service cluster.
  • Container A collection of one or more processes (including all files needed to run) that is portable between computers.
  • a process refers to a program (computer program) that is executing in a computer.
  • Container set A collection of containers that share the same computing resources.
  • a group of containers sharing the same computing resources may include one or more containers, and the computing resources may include a processor, such as a central processing unit (CPU).
  • the computing resources of different container sets are pooled to form several business clusters. These business clusters can provide more powerful and intelligent distributed systems for executing corresponding applications.
  • Container orchestration refers to the automated deployment, management, scaling and networking of containers.
  • Container orchestration can usually be implemented by a container orchestration platform.
  • Container orchestration platforms also known as container orchestration tools, are used to manage a large number of containers during their life cycle, including image distribution, redundant deployment, health monitoring, resource allocation, elastic scaling, load balancing and scheduling.
  • Container orchestration platforms include but are not limited to Apache Mesos, Nomad, Docker Swarm or kubernetes (k8s for short). For ease of description, the following uses kubernetes examples.
  • Container orchestration platforms usually run workloads with pods as the smallest scheduling unit, and provide required networking and storage services for pods.
  • the load balancing function of the container orchestration platform can achieve load balancing among pods, and the elastic scaling function of the container orchestration platform can enable the number of pods to meet business needs.
  • the container orchestration platform can adjust the number of pods through the pod horizontal elastic scaling controller.
  • HPA can create a replica set (replica set) object or a deployment (deployment) object. Both the replica set object or the deployment object can be regarded as a replication controller (RC).
  • the replication controller is used to provide the specified Pod maintains a stable set of copies (instances). For example, set the number of pods in the deploy object to N, and N is a positive integer. If the number of pods is greater than N, you can instruct the deletion of redundant pods. If the number of pods is less than N, you can instruct the creation of new pods.
  • HPA can adjust the replica set object or deployment object to deploy more pods or reduce deployed pods to match observed indicators, such as average CPU utilization, average memory utilization, or other custom indicators.
  • the metric Take the metric as an example of average CPU utilization. In this example, if the current average CPU utilization is 20% and the expected average CPU utilization is 10%, then the expected number of copies will double the current number of copies. If the current average CPU utilization is 5%, then The expected number of replicas is halved from the current number of replicas.
  • HPA is mainly used to achieve elastic scaling at the pod level (instance level).
  • the container orchestration platform can also perform node-level elastic scaling based on the cluster autoscaler (CA).
  • elastic scaling includes elastic expansion and elastic reduction. Expansion refers to expanding pods or nodes, and scaling down refers to deleting (reducing) pods or deleting nodes.
  • CA cluster autoscaler
  • HPA observes the resource utilization of the replica set object or deployment object. When the resource utilization is too high, HPA creates a new pod to cope with the high load pressure. As pods increase and node resources are insufficient to schedule pods, CA triggers cluster expansion to expand nodes. On the contrary, when HPA observes that the resource utilization of the replica set object or deployment object is too low, HPA shrinks the pod to reduce resource consumption. As pods decrease, node resource utilization decreases accordingly. When the node resource utilization is lower than the shrinkage threshold, CA can trigger cluster shrinkage to delete nodes and save resources.
  • CA can adopt different reduction strategies for interruptible services and non-interruptible services.
  • interruptible services such as web services such as information query systems
  • CA can interrupt the remaining pods on the node, release the node, and reschedule the interrupted pods.
  • uninterruptible services such as transcoding services in live broadcast scenarios
  • CA can provide the following configuration parameters: container centralized disruption budget (pod disruption budget, PDB) to ensure that the working or active pods in the business cluster are not less than the PDB. If releasing a node causes the number of active pods in the business cluster to be smaller than the PDB, the node will not be released.
  • PDB container centralized disruption budget
  • CA The purpose of CA is to dynamically adjust the size of the business cluster following HPA as business peaks or troughs change, so as to achieve the effect of on-demand use of node resources.
  • HPA's elastic scaling of pods and CA's elastic scaling of nodes are usually separated.
  • HPA can calculate the expected number of copies of the pod and send corresponding scaling instructions to the corresponding execution units. Among them, if the expected number of copies of a pod is greater than the current number of copies, it indicates the need for expansion.
  • the execution unit corresponding to the expansion can be a scheduler, such as volcano scheduler.
  • the scheduler can respond to scaling instructions and select nodes for pod scheduling through a scheduling algorithm. If the expected number of copies of a pod is less than the current number of copies, it indicates the need for scaling down.
  • the execution unit corresponding to the scaling down can be a deployment controller, such as a k8s controller.
  • the replication controller can respond to scaling instructions and select pods for release through a scaling strategy.
  • CA elastically scales nodes based on node resource utilization.
  • the scheduler uses the boxing scheduling strategy to schedule pods.
  • the goal of the box-packing scheduling strategy is to use fewer nodes to accommodate more pods.
  • the box-packing scheduling strategy is usually an optimal solution for resources at a single time (scheduling time), but in the timing dimension, the box-packing scheduling strategy is not the global optimal solution. The reason is that the number of copies of a pod can change accordingly with business peaks and troughs, and each pod has a different life cycle. Since the scheduler usually packs boxes based on the two dimensions of CPU and memory, without considering the life cycle of the pod, pods with different life cycles will be evenly distributed among various nodes.
  • pods with shorter life spans are gradually released, and eventually during low business periods, long-lived pods will be scattered across various nodes.
  • node resource utilization is low, it is difficult to reduce the capacity.
  • the reduction process can cause a large number of pod interruptions.
  • the reduction strategy also does not consider the life cycle distribution of pods on the nodes.
  • the number of copies of HPA-controlled pods decreases.
  • the pods on each node will show a balanced decrease, and ultimately the same problem will occur, that is, for non-interruptible services, although the node resource utilization is low, it is difficult to shrink, and for interruptible services Business, the shrinking process can cause a large number of pod interruptions.
  • this application provides a container management method.
  • This method can be executed by the container management system.
  • the container management system can be a system integrated with the container orchestration platform.
  • the container management system is used to manage pods deployed to business clusters or pods to be deployed to business clusters.
  • the container management system may be a software system, and the computing device cluster executes the program code of the software system, thereby executing the container management method of the present application.
  • the container management system may also be a hardware system. When the hardware system is running, the container management method of the present application is executed.
  • the following uses a container management system as an example of a software system.
  • the container management system can obtain the life cycle of at least one node and the life cycle of at least one pod in the business cluster, where at least one pod can be a pod to be deployed, or a pod that has been deployed in at least one node. pod, and then the container management system can determine the target node based on the life cycle of at least one node and the life cycle of at least one pod.
  • the target node can be the node where the pod is to be deployed, or the node where the pod is to be deleted. Then the container management system can scale the above pod on the target node.
  • the container management system combines the life cycle of the nodes in the business cluster and the life cycle of the pod to be deployed or the deployed pod to scale the pod, avoiding the separation of the elastic scaling of the pod and the elastic scaling of the node, making CA It can realize the on-demand use of node resources and reduce business costs.
  • the container management system can add the time dimension for pod scheduling based on resource utilization such as CPU and memory. Specifically, the container management system can determine the target node for the pod to be deployed based on the life cycle of the node and the life cycle of the pod to be deployed, and then schedule the pod to be deployed to the target node.
  • this application refers to the above scheduling strategy as a life cycle-based scheduling strategy.
  • the life cycle-based scheduling strategy can schedule pods with close life cycles to the same node. When the low period of the business reaches, for example, after xx hours, the pods on the same node can be deleted first. All pods on the same node are deleted and the node can be released, thus reducing resource waste and business costs.
  • the container management system can obtain the pod to be deployed according to the scaling instructions, and then determine the target node based on the life cycle of the pod to be deployed and the life cycle of the node (such as the remaining life cycle of the VM) , and then schedule the pod to be deployed to the target node close to its life cycle.
  • the target node can be VM2
  • the container management system can schedule the pod to be deployed to VM2.
  • each pod determines its deletion order during scheduling, which is also called the shrinking order.
  • the default setting of this application is that the shrinking order is opposite to the expansion order. The later the pods are expanded, the more priority they will be released. In this way, during the scheduling period, the life cycle of each pod can be determined based on the portrait, specifically the life length of each pod.
  • this application In the initial stage of business deployment, when there is a lack of data and the portrait is not accurate enough, this application also designs a transitional strategy. Based on the default shrinking order of this application, it can be seen that the pods expanded in the early stage survive longer. When the number of pod copies increases, the closer the business peak period is, the shorter the pod survival time is. Based on this feature, the transitional nature of this application
  • the strategy can be: divide it into several stages sooner or later according to the order of pod expansion, and the pods in each stage are prioritized to be scheduled to the same node.
  • the container management system can also dynamically adjust the shrinking order and increase the priority of pods scheduled to inappropriate nodes so that the pods can be deleted or released first.
  • Figure 7 also provides an example for illustration.
  • the business cluster includes 4 VM nodes.
  • the horizontal axis is the timeline, and the horizontal length of the pod represents its survival time, that is, its life cycle.
  • the left picture in Figure 7 shows that a copy of the specified pod, pod1 (shown as a dark gray rectangular block), is incorrectly scheduled to VM4, resulting in low resource utilization of VM4 and VM1 and unable to be released.
  • the container management system can dynamically optimize the shrinking order. Specifically, before shrinking another copy of the specified pod, pod6 (shown as the dark gray rectangular block in VM1), the shrinking order of pod1 and pod is exchanged, and the shrinking order is prioritized.
  • VM4 in this way, after all the pods in VM4 are reduced, VM4 can be released by CA.
  • the right picture in Figure 7 shows the resource usage after the reduction sequence is exchanged. Compared with the left picture, a large amount of resources are saved and business costs are reduced.
  • the container management system 10 includes a life cycle portrait module 100 , a life cycle scheduling module 200 and a sequence optimization module 300 .
  • the life cycle scheduling module 200 and the sequence optimization module 300 are optional modules.
  • the container management system 10 may include a life cycle portrait module 100 and a life cycle scheduling module 200 .
  • the container management system 10 may include a life cycle profiling module 100 and a sequence optimization module 300 .
  • HPA can determine the expected number of replicas of a pod based on the resource utilization of the replica set object or deployment object. For example, in the example of Figure 8, the expected number of replicas of the pod corresponding to job1 can be 3, the expected number of replicas corresponding to job2 can be 2, the expected number of replicas corresponding to job3 can be 4, and the expected number of replicas corresponding to job4 can be 2. .
  • HPA can modify workload resources such as replica set objects or deployment objects, such as k8s resources, and the container management system 10 can respond to changes in workload resources and perform pod scaling based on the life cycle.
  • the life cycle portrait module 100 is used to profile the node and the pod to be deployed, obtain the life cycle of the node and the life cycle of the pod to be deployed, and then life cycle
  • the periodic scheduling module 200 is used to determine the target node according to the life cycle of the node and the life cycle of the pod to be deployed, for example, determine the target node from the cluster resource pool, and then schedule the pod to be deployed to the target node.
  • the cluster resource pool may include one or more of periodic node pools or pay-on-demand node pools.
  • the periodic node pool can be a long-term node pool with annual or monthly billing.
  • the life cycle portrait module 100 is used to profile the node and the deployed pod, obtain the life cycle of the node and the life cycle of the deployed pod, and then The sequence optimization module 300 can determine the target node based on the life cycle of the node and the life cycle of the deployed pods, and then adjust the shrinking order (that is, the deletion order) of the deployed pods in the target node, and follow the adjusted shrinking order. Delete the deployed pods on the target node sequentially. When all deployed pods on the target node are deleted, CA can also release the target node to control the number of nodes.
  • Modules in the container orchestration platform can interact through the interface server.
  • modules in kubernetes can interact through kube-apiserver.
  • the container orchestration platform can provide a variety of native pod orchestration and management methods such as deployment, replica set or statefulset.
  • the corresponding controller executes control logic by interacting with kube-apiserver.
  • the controller also provides an external interface, and the outside can control the shrinkage through kube-apiserver. order.
  • the scheduler senses pod changes in the business layer through kube-apiserver and binds the pod to the corresponding node.
  • the container orchestration platform can also provide customizable expansion capabilities (Custom Resource Define, CRD) for personalized orchestration needs.
  • CRD supports developers to customize resources to improve scalability.
  • the container management system 10 of the present application may include a variety of product forms.
  • the container management system 10 may be in the form of a scheduler-based product.
  • the container management system 10 may also be in the form of a product based on a plug-in of a container orchestration platform, such as a kubernetes plug-in.
  • the container management system 10 may be a product form based on kernel modification, specifically a product form based on kubernetes kernel modification.
  • the life cycle portrait module 100, life cycle scheduling module 200 and sequence optimization module 300 of the container management system 10 are all Implemented in the scheduler.
  • the life cycle profiling module 100 can perceive upper-layer business changes through the apiserver, thereby performing life cycle profiling.
  • the life cycle scheduling module 200 is executed by the scheduler, and the sequence optimization module 300 can Adjust the shrinking order of pods through apiserver.
  • the container management system 10 also supports management of pods in CRD resources developed by users (such as developers). For example, the container management system 10 can perform interface adaptation with CRD resources. The form of the interface can be consistent with the native interface, or a uniformly customized delivery interface can be used. In this way, the container management system 10 can uniformly manage pods in CRD resources and native resources such as pods in deployment resources, for example, uniformly schedule them according to the life cycle, or uniformly shrink them according to the shrinkage sequence adjusted based on the life cycle.
  • each module of the container management system 10 in Figure 10 is deployed independently and interacts through the apiserver.
  • the life cycle scheduling module 200 relies on the scheduler to execute, and the life cycle portrait module 100 and the sequence optimization module 300 are independent plug-ins.
  • the sequence optimization module 300 is implemented in the kubernetes kernel, such as the kubernetes apiserver. Based on this, whether it is a native resource such as a deployment resource or a customized CRD resource, the scaling instructions used for scaling in the scaling instructions can be intercepted. The sequence optimization module 300 optimizes the scaling sequence, and then issues the optimized reduction instructions so that the corresponding controller can execute the optimized reduction instructions.
  • the container management system 10 has been described in detail above. Next, the container management method of the embodiment of the present application will be described in detail from the perspective of the container management system 10 .
  • the method includes:
  • the container management system 10 obtains the life cycle of at least one node and the life cycle of at least one pod in the business cluster.
  • At least one pod can be a pod to be deployed or a deployed pod.
  • HPA instructs to expand a pod (that is, pod-level expansion)
  • at least one pod can be a pod to be deployed.
  • the pod to be deployed can be created based on the pod template defined in the deployment object or replica set object.
  • HPA instructs pod deletion (that is, pod-level scaling)
  • at least one pod can be a deployed pod.
  • the container management system 10 can obtain the life cycle of at least one node in the business cluster and the life cycle of at least one pod through the life cycle portrait.
  • the following describes the life cycle portrait of the pod and the life cycle portrait of the node respectively.
  • the container management system 10 can obtain the survival time distribution of the replicas in the replica set corresponding to at least one pod in the historical time period, as shown in the upper figure of Figure 13.
  • the horizontal axis of the duration distribution is time, and the vertical axis is the number of copies, indicating the pod copies that are working or active at the corresponding moment.
  • the container management system 10 can also perform conversion according to the above-mentioned survival time distribution, thereby determining the change of the life length with the expansion sequence.
  • the container management system 10 can predict the life cycle of at least one pod through statistical strategies. Among them, statistical strategies include but are not limited to one or more of machine learning, quantile, mean, maximum or probability distribution.
  • the container management system 10 can use stack tracking to track the life cycle of each pod copy.
  • stack tracking In order to facilitate understanding, the following is explained with a specific example.
  • the stack can record the life cycle changes of each pod copy such as each deployment pod.
  • the time is 8:00
  • the number of pods in the deployment is 1, and the time 8:00 can be pushed onto the stack.
  • the time is 12:00
  • the deployment adds a pod, and continues to add 12 pods. :00 is pushed onto the stack.
  • the time is 14:00
  • the deployment adds a pod and continues to push 14:00 onto the stack.
  • the time is 16:00
  • the workload decreases and the deployment decreases one pod. For example, 14:00 on the top of the stack is added. Get out of the stack.
  • the container management system 10 can draw inventory duration distribution based on the above records.
  • the container management system 10 can calculate the time difference between pushing and popping, and uses the time difference as the life length of the pod copy. As shown in Figure 15, for a pod copy with recorded stacking time and popping time, the container management system 10 can calculate the life length of the pod copy. Among them, the pod copy can be scheduled multiple times during the entire historical time period, and the container management system 10 can calculate multiple life lengths. For example, the life span of the third expanded pod replica (denoted as replicas 3) in the container management system 10 includes 20:00, 21:00 or 19:51.
  • pod copies that do not record the de-stack time in the stack can be regarded as pod copies with a longer life cycle.
  • the first pod copy (denoted as replicas 1)
  • the second pod copy (denoted as replicas 2) Can have a long life cycle.
  • the container management system 10 can predict based on the life length of each pod copy through statistical strategies, such as maximum value, minimum value, mean value, quantile (such as median, P99, etc.), mean value, probability distribution, or machine learning.
  • statistical strategies such as maximum value, minimum value, mean value, quantile (such as median, P99, etc.), mean value, probability distribution, or machine learning.
  • the life cycle of each pod copy For example, when the deployment expands a third pod replica, if you use the median prediction, you can predict that the pod will survive for 20 hours, and if you use the maximum value, you can predict that it will survive for 21 hours.
  • the container management system 10 may determine the life cycle of at least one node based on the life cycle of the pod on at least one node and the creation time of the pod on at least one node. For example, the container management system 10 can calculate the remaining survival time of each pod on at least one node, and determine the life cycle of the node based on the remaining survival time. Similarly, the container management system 10 can determine the life cycle of the node based on the remaining survival time and based on the statistical policy. Similarly, statistical strategies may include one or more of machine learning, quantile, mean, maximum, probability distribution. In some examples, the container management system 10 may determine the maximum value of the remaining survival time as the life cycle of the node.
  • the container management system 10 determines the target node based on the life cycle of the at least one node and the life cycle of the at least one pod.
  • At least one pod is a pod to be deployed, and the target node is the node where the pod is to be deployed.
  • the target node is the node where the pod is to be deleted.
  • the container management system 10 may determine the proximity of the life cycle of the pod to be deployed to the life cycle of at least one node, and then determine the target node from the at least one node based on the proximity.
  • the proximity between the life cycle of a pod and the life cycle of a node can be determined based on the difference in life cycles or the ratio of life cycles.
  • the proximity between the life cycle of a pod and the life cycle of a node can be the ratio of the life cycle of the pod to the life cycle of the node, or the reciprocal of the above ratio, that is, the ratio of the life cycle of the node to the life cycle of the pod.
  • the container management system 10 can sort at least one node according to the proximity of the pod's life cycle to at least one node, and then determine the target node from the at least one node according to the sorting result.
  • the container management system 10 can filter nodes whose proximity is lower than a preset value according to the sorting results, and determine the target node from the remaining nodes.
  • the resources of the target node can accommodate the pod to be deployed.
  • the container management system 10 may score at least one node based on proximity, and determine the target node from at least one node based on the score of the at least one node.
  • the first node among at least one node is used as an example for description below.
  • the pod to be deployed includes the first pod.
  • the score of the first node is positively correlated with the first proximity.
  • the first proximity is determined based on the ratio of the life cycle of the first pod to the life cycle of the first node.
  • the score of the first node is positively correlated with the second proximity.
  • the second proximity is determined based on the ratio of the life cycle of the first node and the life cycle of the first container set.
  • the score of the first node can be seen in the following formula:
  • a, b, c, and d are coefficients
  • score is the score
  • podlife is the life cycle of the pod
  • nodelife is the life cycle of the node.
  • the scoring strategy is not limited to the above methods. It only needs to ensure that: on the premise that the pod does not extend the life cycle of the node, the closer the life cycle is, the higher the score will be; on the premise that the pod extends the life cycle of the node, the more it is extended, the lower the score will be.
  • the container management system 10 may select the node with the highest score among the nodes whose resources can accommodate the first pod as the target node. In some embodiments, the container management system 10 may also select a node whose resources are capable of accommodating the first pod and whose score is greater than the set score as the target node.
  • the container management system 10 may determine an optimizeable fragmented node as a target node based on the life cycle of at least one node and the life cycle of the pod on the node. Specifically, the container management system 10 may first determine a candidate second node based on the life cycle of at least one node and the life cycle of the pod on at least one node.
  • the candidate second node may be a long-cycle node, and the pod on the candidate second node may be a long-cycle pod.
  • a candidate second node could be a long-lived pod remaining in the trough.
  • the life cycle of a long-life node is greater than the first duration
  • the life cycle of a long-life pod is greater than the second duration.
  • the first duration and the second duration can be set based on experience values. In some examples, the first duration and the second duration may be set to be equal, or set to be unequal.
  • long-term nodes and long-term pods are usually not deleted during the low period of business. In contrast, elastic nodes and elastic pods can be deleted during the low period of business.
  • the container management system 10 determines at least one candidate deletion order of the pod on the candidate second node, and predicts the revenue of deleting the container set on the second node according to the candidate deletion order.
  • the revenue is determined based on the resource utilization of the cluster.
  • the target deletion sequence can be determined based on the above benefits, and the target node can be determined from the candidate second nodes.
  • the container management system 10 may determine the candidate deletion order that maximizes revenue as the target deletion order, and determine the node that can be deleted among the candidate second nodes as the target node based on the target deletion order.
  • the container management system 10 may select some of the nodes that can be optimized as target nodes. Specifically, the container management system 10 can select the candidate second section according to the number of pods. Point sorting is, for example, sorting in ascending order of the number of pods. The container management system 10 sequentially determines whether the nodes can be optimized according to the sorting results, and determines the nodes that can be optimized as the target nodes.
  • the container management system 10 can predict, based on statistical analysis, the total number of long-term pod resources on the node after adjusting the deletion order of the second pod on the node according to the candidate deletion order, or the number of long-term pods in each deployment. If the number of long-term pods with deployment is greater than the number of elastic pods, the node will be skipped and the next node will be judged as to whether it can be optimized. Furthermore, when a node is optimized and the total accumulated long-term pod resources exceed the remaining space of the cluster, node screening can be terminated.
  • S1206 The container management system 10 scales the container set on the target node.
  • the container management system 10 may schedule the pod to be deployed (such as the first pod) to the target node.
  • the container management system 10 may delete the pod to be deleted (such as the second pod) from the target node. This allows the container management system 10 to scale the pod on the target node.
  • the container management system 10 can adjust the deletion order of the second pod on the target node according to the target deletion order, and then delete the second pod from the target node according to the adjusted deletion order. In this way, when all pods on the target node are deleted, CA can release the target node, thereby reducing resource waste and business costs.
  • the container management system 10 may periodically adjust the shrinking order of the second pod, or adjust the shrinking order of the second pod in real time.
  • periodic optimization refers to analyzing the life cycle distribution of pods in the cluster during the low period of business, determining the fragmented nodes that can be optimized as the target node, and adjusting the deletion order of the second pod on the target node (also called the shrinking order, Scale-down priority), when the business enters the next low period, the target node can be released.
  • the container management system 10 can analyze the life cycle distribution of deployed pods in the business cluster during the low period of the business, and select optimizable ones based on the life cycle distribution. Fragmented nodes.
  • new pods are added during the peak period of the business.
  • the pods scheduled to VM2 are The scaling priority is -3
  • the scaling priority of the pod scheduled to VM4 is -1
  • the scaling priority of the pod scheduled to VM5 is -1.
  • the container management system 10 prioritizes shrinking the pods on VM2 according to the shrinkage priority.
  • the container management system 10 determines that VM4 is an optimizeable fragmented node by analyzing the life cycle distribution of deployed pods in the business cluster, and can determine VM4 as the target node.
  • the container management system 10 adjusts the shrinking order of the pods on VM4. For example, during the next peak period, when a new pod is expanded on VM2, the shrinking order of the pods on VM4 is exchanged with the shrinking order of the new pods on VM2. . In this way, the container management system 10 can first shrink the pods on VM4 during the next low period of business. When all the pods on VM4 are deleted, VM4 can be released. Further, the container management system 10 can mark the shrinkage priority of the pod on VM5 as -3 during the next trough period, so as to shrink the pod on VM5 during the next trough period. When the pod on VM5 are deleted and VM5 can be released.
  • the container management system 10 can analyze the shrinking order of the pods in the business cluster in real time, and determine the shrinking order adjustment strategy through mathematical optimization methods. For example, during the peak period of business, the container management system 10 can determine the shrinking order adjustment strategy to exchange the shrinking order of the pods on VM2 with the shrinking order of the pods on VM4. Then, the container management system 10 can, before the business trough period arrives, according to Real-time analysis of the shrinking order adjustment strategy, adjusting the shrinking order of pods (i.e. shrinking priority). The container management system 10 can reduce the capacity according to the adjusted reduction sequence during the low period of the business, so that the nodes can be released smoothly. Compared with periodic optimization of the shrinking sequence, real-time optimization of the shrinking sequence takes effect faster and has better results.
  • this application also provides a container management system.
  • the container management system is used to manage the container set deployed to the business cluster or the container set to be deployed to the business cluster.
  • the container management system 10 As shown in Figure 8, the container management system 10:
  • the life cycle portrait module 100 is used to obtain the life cycle of at least one node in the business cluster and the life cycle of at least one container set, and the node is used to deploy the container set;
  • the life cycle scheduling module 200 is configured to determine a target node according to the life cycle of the at least one node and the life cycle of the at least one container set.
  • the target node is the node where the container set is to be deployed, or the node to be deleted.
  • the life cycle scheduling module 200 is also used to scale the container set on the target node.
  • life cycle portrait module 100 and life cycle scheduling module 200 can be implemented by hardware, or can be implemented by software.
  • life cycle scheduling module 200 is used as an example below.
  • the life cycle scheduling module 200 may be an application program running on a computing device, such as a computing engine.
  • the application can be provided to users as a virtualized service.
  • Virtualization services can include virtual machine (VM) services, bare metal server (BMS) services, and container (container) services.
  • the VM service can be a service that uses virtualization technology to virtualize a virtual machine (VM) resource pool on multiple physical hosts (such as computing devices) to provide users with VMs for use on demand.
  • the BMS service is a service that virtualizes BMS resource pools on multiple physical hosts to provide users with BMS on demand.
  • Container service is a service that virtualizes container resource pools on multiple physical hosts to provide users with containers on demand.
  • VM is a simulated virtual computer, that is, a logical computer.
  • BMS is an elastically scalable high-performance computing service. Its computing performance is the same as that of traditional physical machines, and it has the characteristics of safe physical isolation.
  • Containers are a kernel virtualization technology that can provide lightweight virtualization to isolate user space, processes and resources. It should be understood that the VM service, BMS service and container service in the above virtualization services are only specific examples. In actual applications, the virtualization service can also be other lightweight or heavyweight virtualization services, which are not discussed here. Specific limitations.
  • the life cycle scheduling module 200 may include at least one computing device, such as a server.
  • the life cycle scheduling module 200 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD).
  • ASIC application-specific integrated circuit
  • PLD programmable logic device
  • the above-mentioned PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (field-programmable gate array, FPGA), a general array logic (generic array logic, GAL), or any combination thereof.
  • CPLD complex programmable logical device
  • FPGA field-programmable gate array
  • GAL general array logic
  • the at least one container set includes the container set to be deployed, and the life cycle scheduling module 200 is specifically used to:
  • the life cycle scheduling module 200 is specifically used to:
  • life cycle scheduling module 200 is specifically used to:
  • the at least one node is scored according to the proximity, and a target node is determined from the at least one node according to the score of the at least one node.
  • the at least one node includes a first node, and the at least one container set includes a first container set;
  • the score of the first node is positively correlated with the first proximity, and the first proximity is based on the life cycle of the first container set.
  • the ratio of the period to the life cycle of the first node is determined;
  • the score of the first node is positively correlated with the second proximity, and the second proximity is based on the life cycle of the first node.
  • the ratio of the cycle to the life cycle of the first container set is determined.
  • system 10 further includes:
  • the sequence optimization module 300 is configured to determine a candidate second node according to the life cycle of the at least one node and the life cycle of the container set on the at least one node; determine at least one of the container sets on the candidate second node.
  • a candidate deletion order is used to predict the revenue from deleting the container set on the second node according to the candidate deletion order, and the revenue is determined based on the resource utilization on the cluster; based on the revenue, the target deletion order is determined;
  • the life cycle scheduling module 200 is specifically used to:
  • the life cycle scheduling module 200 is specifically used to:
  • Adjust the deletion order of the second container set on the target node according to the target deletion order and delete the second container set on the target node according to the adjusted deletion order.
  • the sequence optimization module 300 can be implemented by hardware or can be implemented by software.
  • the sequence optimization module 300 may be an application program running on a computing device, such as a computing engine or the like. The application can be provided to users as a virtualized service.
  • the sequence optimization module 300 may include at least one computing device, such as a server.
  • the sequence optimization module 300 may also be a device implemented using an application specific integrated circuit (ASIC) or a programmable logic device (PLD).
  • ASIC application specific integrated circuit
  • PLD programmable logic device
  • life cycle scheduling module 200 is specifically used to:
  • the deletion order of the second container set is adjusted according to the real-time analysis deletion order adjustment strategy.
  • the life cycle portrait module 100 is specifically used to:
  • the life cycle of the at least one container set is predicted through a statistical strategy.
  • the life cycle portrait module 100 is specifically used to:
  • the life cycle of the at least one node is determined according to the life cycle of the container set on the at least one node and the creation time of the container set on the at least one node.
  • the container management system 10 is deployed in a scheduler.
  • the container management system 10 is distributed and deployed on different devices, and different modules in the container management system 10 interact through an application programming interface API server.
  • the sequence optimization module in the container management system 10 is an independent plug-in, or is obtained by modifying the kernel of the container orchestration platform.
  • computing device 1800 includes: bus 1802, processor 1804, memory 1806, and communication interface 1808.
  • the processor 1804, the memory 1806 and the communication interface 1808 communicate through a bus 1802.
  • Computing device 1800 may be a server or terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1800.
  • the bus 1802 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc.
  • the bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one line is used in Figure 18, but it does not mean that there is only one bus or one type of bus.
  • Bus 1802 may include a path that carries information between various components of computing device 1800 (eg, memory 1806, processor 1804, communications interface 1808).
  • the processor 1804 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP). any one or more of them.
  • CPU central processing unit
  • GPU graphics processing unit
  • MP microprocessor
  • DSP digital signal processor
  • Memory 1806 may include volatile memory, such as random access memory (RAM).
  • the processor 1804 may also include non-volatile memory (non-volatile memory), such as read-only memory (ROM), flash memory, mechanical hard disk drive (hard disk drive, HDD) or solid state drive (solid state drive). drive, SSD).
  • ROM read-only memory
  • flash memory such as hard disk drive (hard disk drive, HDD) or solid state drive (solid state drive). drive, SSD).
  • the memory 1806 stores executable program code, and the processor 1804 executes the executable program code to implement the aforementioned container management method. Specifically, the memory 1806 stores instructions for the container management system 10 to execute the container management method.
  • the communication interface 1808 uses transceiver modules such as, but not limited to, network interface cards and transceivers to implement communication between the computing device 1800 and other devices or communication networks.
  • An embodiment of the present application also provides a computing device cluster.
  • the computing device cluster includes at least one computing device.
  • the computing device may be a server, such as a central server, an edge server, or a local server in a local data center.
  • the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
  • the computing device cluster includes at least one computing device 1800. Instructions for performing the container management method of the same container management system 10 may be stored in the memory 1806 of one or more computing devices 1800 in the computing device cluster.
  • one or more computing devices 1800 in the computing device cluster may also be used to The container management system 10 is used to execute part of the instructions of the container management method.
  • a combination of one or more computing devices 1800 may collectively execute instructions of the container management system 10 for performing the container management method.
  • the memory 1806 in different computing devices 1800 in the computing device cluster can store different instructions for executing some functions of the container management system 10 .
  • Figure 20 shows a possible implementation.
  • two computing devices 1800A and 1800B are connected through a communication interface 1808.
  • Stored on memory in computing device 1800A are instructions for performing the functions of life cycle profiling module 100 and .
  • Stored on memory in computing device 1800B are instructions for performing the functions of life cycle scheduling module 200 .
  • instructions for the functions of the sequence optimization module 300 are also stored on the memory in the computing device 1800B.
  • the memories 1806 of the computing devices 1800A and 1800B collectively store instructions for the container management system 10 to perform the container management method.
  • connection method between computing device clusters shown in Figure 20 can be based on the fact that the container management method provided by this application requires a large amount of computing power to determine the deletion sequence and then determine the target node. Therefore, it is considered that the functions implemented by the sequence optimization module 300 are also executed by the computing device 1800B.
  • computing device 1800A shown in FIG. 20 may also be performed by multiple computing devices 1800.
  • computing device 1800B may also be performed by multiple computing devices 1800.
  • one or more computing devices in a cluster of computing devices may be connected through a network.
  • the network may be a wide area network or a local area network, etc.
  • Figure 21 shows a possible implementation. As shown in Figure 21, two computing devices 1800C and 1800D are connected through a network. Specifically, the connection to the network is made through a communication interface in each computing device.
  • instructions for performing the functions of the life cycle profiling module 100 are stored in the memory 1806 of the computing device 1800C.
  • instructions for performing the functions of the life cycle scheduling module 200 are stored in the memory 1806 in the computing device 1800D.
  • instructions for the functions of the sequence optimization module 300 are also stored on the memory in the computing device 1800B.
  • connection method between computing device clusters shown in Figure 21 can be: Considering that the container management method provided by this application requires a large amount of computing power to determine the deletion sequence and then determine the target node, it is considered to combine the life cycle scheduling module 200 and the sequence optimization module
  • the functions implemented by 300 are executed by the computing device 1800D.
  • computing device 1800C shown in FIG. 21 may also be performed by multiple computing devices 1800.
  • the functions of computing device 1800D may also be performed by multiple computing devices 1800.
  • An embodiment of the present application also provides a computer-readable storage medium.
  • the computer-readable storage medium may be any available medium that a computing device can store or a data storage device such as a data center that contains one or more available media.
  • the usable media may be magnetic media (eg, floppy disk, hard disk, tape), optical media (eg, DVD), or semiconductor media (eg, solid state drive), etc.
  • the computer-readable storage medium includes instructions that instruct the computing device to execute the above-described application to the container management system 10 for performing the container management method.
  • An embodiment of the present application also provides a computer program product containing instructions.
  • the computer program product may be a software or program product containing instructions capable of running on a computing device or stored in any available medium.
  • the computer program product is run on at least one computing device, at least one computing device is caused to execute the above container management method.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本申请提供了一种容器管理方法，由容器管理系统执行，该系统用于对部署至业务集群的容器集pod或待部署至业务集群的pod进行管理，该方法包括：获取业务集群中至少一个节点(用于部署pod)的生命周期以及至少一个pod的生命周期，然后根据至少一个节点的生命周期以及至少一个pod的生命周期，确定目标节点，该目标节点为待部署pod的节点，或者是待删除pod的节点，接着容器管理系统在目标节点对pod进行伸缩。该方法结合业务集群中节点的生命周期以及待部署的pod或已部署的pod的生命周期，对pod进行伸缩，避免了pod的弹性伸缩与节点的弹性伸缩的割裂，能够实现节点资源按需使用，降低业务成本。

Description

一种容器管理方法及相关设备
本申请要求于2022年08月16日提交中国国家知识产权局、申请号为202210983171.5、发明名称为“一种容器管理方法及相关系统”的中国专利申请的优先权，以及要求于2022年11月29日提交中国国家知识产权局、申请号为202211507530.6、发明名称为“一种容器管理方法及相关设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及云计算技术领域，尤其涉及一种容器管理方法、系统、计算设备集群、计算机可读存储介质、计算机程序产品。
背景技术
随着云计算的不断发展，越来越多的开发者开始采用容器进行应用开发、部署。容器是软件的可执行单元，它采用通用方式封装了应用程序代码及其库和依赖项，因此可以随时随地运行容器。
考虑到一些应用可以包括数百甚至数千个容器，可以采用容器编排平台对容器进行全生命周期的管理。例如，容器编排平台可以对容器进行镜像分发、冗余部署、健康监测、资源分配、弹性伸缩、负载均衡和调度。
容器编排平台通常可以将多个容器分类组成"容器集"，记作pod，然后以pod为最小调度单元运行工作负载，并为这些pod提供所需的联网和存储等服务。考虑到工作负载可以是不断变化的，容器编排平台可以通过实时调整pod的数量，使得pod总体数量足够支撑业务压力。进一步地，容器编排平台在调整pod的数量时，还可以调整用于部署pod的节点的数量。
然而，容器编排平台在对pod的数量或节点的数量进行调整(也称作弹性伸缩)时，很难实现节点资源按需使用，进而导致业务成本居高不下。
发明内容
本申请提供了一种容器管理方法，该方法通过基于容器集的生命周期以及节点的生命周期在节点对容器集进行调度，从而实现节点资源按需使用，避免资源浪费，降低了业务成本。本申请还提供了对应的容器管理系统、计算设备集群、计算机可读存储介质以及计算机程序产品。
第一方面，本申请提供一种容器管理方法。该方法由容器管理系统执行。容器管理系统可以是用于对部署至业务集群的容器集(pod)或待部署至业务集群的pod进行管理的系统。当容器管理系统为软件系统时，容器管理系统可以是集成于容器编排平台的插件、组件或者模块，也可以是独立的软件，该软件系统可以部署在计算设备集群中，计算设备集群执行该软件系统的程序代码，从而执行本申请的容器管理方法。当容器管理系统为硬件系统时，例如为具有容器管理功能的计算设备集群时，该容器管理系统可以在运行时执行本申请的容器管理方法。
具体地，容器管理系统可以获取业务集群中至少一个节点的生命周期以及至少一个pod的生命周期，其中，节点用于部署pod，例如节点可以是虚拟机(virtual machine，VM)节点，然后容器管理系统根据至少一个节点的生命周期以及至少一个pod的生命周期，确定目标节点，该目标节点为待部署pod的节点，或者是待删除pod的节点，接着容器管理系统在目标节点对上述pod进行伸缩。
该方法中，容器管理系统结合业务集群中节点的生命周期以及待部署的pod或已部署的pod的生命周期，对pod进行伸缩，避免了pod的弹性伸缩与节点的弹性伸缩的割裂，使得集群弹性伸缩控制器(cluster autoscaler，CA)能够实现节点资源按需使用，降低业务成本。
在一些可能的实现方式中，至少一个pod包括待部署的pod，容器管理系统在根据节点的生命周期以及pod的生命周期，确定目标节点时，可以确定待部署的pod的生命周期与至少一个节点的生命周期的接近程度，然后根据接近程度，从至少一个节点中确定目标节点。相应地，容器管理系统可以将待部署的pod调度至目标节点。
该方法通过将待部署的pod调度至剩余生命周期与pod的生命周期接近的目标节点，使得该pod被删除(缩容)时，该目标节点上其他pod也已经被删除或即将被删除，目标节点可以尽快被释放，减少了资源浪费，降低了业务成本。
在一些可能的实现方式中，容器管理系统提供多种方式确定目标节点。具体地，容器管理系统可以根据接近程度对所述至少一个节点排序，根据排序结果从所述至少一个节点中确定目标节点。例如，容器管理系统可以根据排序结果，将最接近的节点确定为目标节点。容器管理系统也可以根据接近程度对所述至少一个节点进行评分，根据至少一个节点的评分从所述至少一个节点中确定目标节点。例如，容器管理系统可以将评分最高或者评分大于预设分值的节点确定为目标节点。
其中，基于排序确定目标节点的方式比较简单，易于实现，对算力要求较低，基于评分确定目标节点的方式相对更为精确，能够确定出更合理的目标节点，将pod调度至该目标节点可以较大程度地减少资源浪费，降低业务成本。
在一些可能的实现方式中，至少一个节点包括第一节点，至少一个pod包括第一pod。第一pod的生命周期小于第一节点的生命周期时，第一节点的评分与第一接近程度正相关，第一接近程度根据第一pod的生命周期和第一节点的生命周期的比值确定。第一pod的生命周期不小于第一节点的生命周期时，第一节点的评分与第二接近程度正相关，第二接近程度根据第一节点的生命周期和第一pod的生命周期的比值确定。
该方法针对pod生命周期小于节点的生命周期的情况、pod生命周期不小于节点的生命周期的情况，分别采用相应规则确定节点的评分，提高了节点评分的准确度，为推荐出合适的目标节点奠定基础。
在一些可能的实现方式中，容器管理系统在确定目标节点时，可以根据所述至少一个节点的生命周期以及所述至少一个节点上容器集的生命周期，确定候选的第二节点，然后确定所述候选的第二节点上容器集的至少一种候选删除顺序，预测按照所述候选删除顺序删除所述第二节点上容器集的收益。其中，收益可以根据集群上的资源利用率确定。接着容器管理系统根据所述收益，确定目标删除顺序，并从所述候选的第二节点中确定目标节点。如此，容器管理系统在对pod进行伸缩时，可以按照所述目标删除顺序调整所述目标 节点上第二容器集的删除顺序，并按照调整后的删除顺序对所述目标节点上所述第二容器集进行删除。
在该方法中，容器管理系统通过智能分析pod全局缩容顺序，并对缩容顺序进行优化，从根源上解决节点资源碎片化问题，提高资源利用率，降低业务成本。
在一些可能的实现方式中，候选的第二节点的生命周期大于第一时长，所述候选的第二节点上容器集的生命周期大于第二时长。
如此，候选的第二节点可以为长周期节点，候选的第二节点上的pod为长周期pod。例如，候选的第二节点可以在低谷器剩余长周期pod。长周期节点的生命周期大于第一时长，长周期pod的生命周期大于第二时长。该第一时长和第二时长可以根据经验值设置。在一些示例中，第一时长和第二时长可以设置为相等，或者设置为不等。其中，长周期节点、长周期pod通常不会在业务的低谷期被删除，与之相对的弹性节点、弹性pod可以在业务的低谷期被删除。
该方法通过确定长周期节点作为候选的节点，可以减少缩容顺序优化时的遍历次数，提升缩容优化效率。
在一些可能的实现方式中，所述容器管理系统支持周期性缩容优化，或者实时缩容优化。具体地，容器管理系统在调整所述目标节点上第二pod的删除顺序时，可以在业务的低谷期，周期性调整所述第二pod的删除顺序。容器管理系统也可以在业务的低谷期到达之前，根据实时分析的删除顺序调整策略，调整所述第二pod的删除顺序。
其中，周期性缩容优化所需的计算量相对较小，能够以较小的代价实现资源利用率的提升，实时性缩容优化需要实时计算删除顺序调整策略，从而获得较好的优化效果。
在一些可能的实现方式中，容器管理系统可以获取所述至少一个pod对应的副本集中的副本在历史时间段的存活时长分布，然后根据所述至少一个pod对应的副本集中的副本在历史时间段的存活时长分布，通过统计策略，预测所述至少一个pod的生命周期。
该方法基于历史时间段的存活时长分布实现pod的生命周期画像，具有较高可靠性，为基于生命周期的调度提供了依据。
在一些可能的实现方式中，统计策略包括机器学习、分位数、均值、最大值或概率分布中的一种或多种。具体实现时，容器管理系统可以根据业务特征选择业务对应的统计策略，从而精准预测pod的生命周期，为基于生命周期的调度策略提供参考。
在一些可能的实现方式中，所述容器管理系统根据所述至少一个节点上pod的生命周期以及所述至少一个节点上pod的创建时间，确定所述至少一个节点的生命周期。
如此，可以实现对节点进行生命周期画像，并且具有较高可靠性，为基于生命周期的调度提供了依据。
在一些可能的实现方式中，所述容器管理系统部署在调度器中。通过调度器提供基于生命周期的调度、缩容顺序优化等容器管理能力，可以降低对其他业务的影响，并且减少侵入性。
在一些可能的实现方式中，容器管理系统可以分布式部署在不同设备，并且容器管理系统中的不同模块通过应用程序编程接口API服务器交互。如此，可以分散风险，提升整个容器管理系统的可靠性。
在一些可能的实现方式中，所述容器管理系统中的顺序优化模块可以为独立插件，或者通过对容器编排平台的内核改造获得。其中，独立插件具有较好兼容性，可以适用于不同平台，能够满足不同平台的用户需求。对容器编排平台的内核进行改造实现顺序优化模块，则可以简化用户操作，提升用户体验。
第二方面，本申请提供一种容器管理系统。所述容器管理系统用于对部署至业务集群的容器集或待部署至所述业务集群的容器集进行管理，所述容器集包括一组容器，所述系统包括：
生命周期画像模块，用于获取所述业务集群中至少一个节点的生命周期以及至少一个容器集的生命周期，所述节点用于部署所述容器集；
生命周期调度模块，用于根据所述至少一个节点的生命周期以及所述至少一个容器集的生命周期，确定目标节点，所述目标节点为待部署所述容器集的节点，或者是待删除所述容器集的节点；
所述生命周期调度模块，还用于在所述目标节点对所述容器集进行伸缩。
在一些可能的实现方式中，所述至少一个容器集包括待部署的所述容器集，所述生命周期调度模块具体用于：
确定待部署的所述容器集的生命周期与所述至少一个节点的生命周期的接近程度；
根据所述接近程度，从所述至少一个节点中确定目标节点；
所述生命周期调度模块具体用于：
将待部署的所述容器集调度至所述目标节点。
在一些可能的实现方式中，所述生命周期调度模块具体用于：
根据所述接近程度对所述至少一个节点排序，根据排序结果从所述至少一个节点中确定目标节点；或者，
根据所述接近程度对所述至少一个节点进行评分，根据至少一个节点的评分从所述至少一个节点中确定目标节点。
在一些可能的实现方式中，所述至少一个节点包括第一节点，所述至少一个容器集包括第一容器集；
所述第一容器集的生命周期小于所述第一节点的生命周期时，所述第一节点的评分与第一接近程度正相关，所述第一接近程度根据所述第一容器集的生命周期和所述第一节点的生命周期的比值确定；
所述第一容器集的生命周期不小于所述第一节点的生命周期时，所述第一节点的评分与第二接近程度正相关，所述第二接近程度根据所述第一节点的生命周期和所述第一容器集的生命周期的比值确定。
在一些可能的实现方式中，所述系统还包括：
顺序优化模块，用于根据所述至少一个节点的生命周期以及所述至少一个节点上容器集的生命周期，确定候选的第二节点；确定所述候选的第二节点上容器集的至少一种候选删除顺序，预测按照所述候选删除顺序删除所述第二节点上容器集的收益，所述收益根据所述集群上的资源利用率确定；根据所述收益，确定目标删除顺序；
所述生命周期调度模块具体用于：
根据所述收益，从所述候选的第二节点中确定目标节点；
所述生命周期调度模块具体用于：
按照所述目标删除顺序调整所述目标节点上第二容器集的删除顺序，并按照调整后的删除顺序对所述目标节点上所述第二容器集进行删除。
在一些可能的实现方式中，所述生命周期调度模块具体用于：
在业务的低谷期，周期性调整所述第二容器集的删除顺序；或者，
在所述业务的低谷期到达之前，根据实时分析的删除顺序调整策略，调整所述第二容器集的删除顺序。
在一些可能的实现方式中，所述生命周期画像模块具体用于：
获取所述至少一个容器集对应的副本集中的副本在历史时间段的存活时长分布；
根据所述至少一个容器集对应的副本集中的副本在历史时间段的存活时长分布，通过统计策略，预测所述至少一个容器集的生命周期。
在一些可能的实现方式中，所述生命周期画像模块具体用于：
根据所述至少一个节点上容器集的生命周期以及所述至少一个节点上容器集的创建时间，确定所述至少一个节点的生命周期。
在一些可能的实现方式中，所述容器管理系统部署在调度器中。
在一些可能的实现方式中，所述容器管理系统分布式部署在不同设备，并且所述容器管理系统中的不同模块通过应用程序编程接口API服务器交互。
在一些可能的实现方式中，所述容器管理系统中的顺序优化模块为独立插件，或者通过对容器编排平台的内核改造获得。
第三方面，本申请提供一种计算设备集群。所述计算设备集群包括至少一台计算设备，所述至少一台计算设备包括至少一个处理器和至少一个存储器。所述至少一个处理器、所述至少一个存储器进行相互的通信。所述至少一个处理器用于执行所述至少一个存储器中存储的指令，以使得计算设备或计算设备集群执行如第一方面或第一方面的任一种实现方式所述的容器管理方法。
第四方面，本申请提供一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，所述指令指示计算设备或计算设备集群执行上述第一方面或第一方面的任一种实现方式所述的容器管理方法。
第五方面，本申请提供了一种包含指令的计算机程序产品，当其在计算设备或计算设备集群上运行时，使得计算设备或计算设备集群执行上述第一方面或第一方面的任一种实现方式所述的容器管理方法。
本申请在上述各方面提供的实现方式的基础上，还可以进行进一步组合以提供更多实现方式。
附图说明
为了更清楚地说明本申请实施例的技术方法，下面将对实施例中所需使用的附图作以简单地介绍。
图1为本申请实施例提供的一种pod水平弹性伸缩控制器控制pod数量的示意图；
图2为本申请实施例提供的一种pod弹性伸缩和节点弹性伸缩的示意图；
图3为本申请实施例提供的一种基于装箱调度策略进行pod调度的示意图；
图4为本申请实施例提供的一种基于缩容策略进行缩容的示意图；
图5为本申请实施例提供的一种基于生命周期进行pod调度的示意图；
图6为本申请实施例提供的一种基于生命周期进行pod调度的示意图；
图7为本申请实施例提供的一种基于生命周期进行缩容的示意图；
图8为本申请实施例提供的一种容器管理系统的系统架构图；
图9为本申请实施例提供的一种基于调度器的容器管理系统的结构示意图；
图10为本申请实施例提供的一种基于插件的容器管理系统的结构示意图；
图11为本申请实施例提供的一种基于内核的容器管理系统的结构示意图；
图12为本申请实施例提供的一种容器管理方法的流程图；
图13为本申请实施例提供的一种副本集中pod存活时长统计分析示意图；
图14为本申请实施例提供的一种采用栈跟踪pod副本的示意图；
图15为本申请实施例提供的一种pod副本的生命长度分布的示意图；
图16为本申请实施例提供的一种周期性优化缩容顺序的示意图；
图17为本申请实施例提供的一种实时优化缩容顺序的示意图；
图18为本申请实施例提供的一种计算设备的结构示意图；
图19为本申请实施例提供的一种计算设备集群的结构示意图；
图20为本申请实施例提供的一种计算设备集群的结构示意图；
图21为本申请实施例提供的一种计算设备集群的结构示意图。
具体实施方式
本申请实施例中的术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。
首先对本申请实施例中所涉及到的一些技术术语进行介绍。
节点(node)：最小的计算硬件单元。通常情况下，节点可以是一台单独的计算机(也称计算设备)。该计算机可以是物理主机，如服务器或终端。其中，服务器可以是云服务器、边缘服务器或本地服务器。云服务器是指云环境中的服务器，例如是中心计算集群中的中心服务器。边缘服务器是指边缘环境中的服务器，例如是边缘计算集群中的边缘服务器。本地服务器是指本地数据中心中的服务器。终端包括但不限于台式机、笔记本电脑、智能手机。进一步地，该计算机也可以是物理主机上通过虚拟化服务虚拟出的虚拟主机，也称作虚拟机(virtual machine，VM)。
集群(cluster)：节点的集合。集群中的节点通常是协同工作，因此集群可以视为单个系统。集群中的各节点可以被设置为执行相同的任务，并由软件控制和调度，由此提高可用性和可缩放性。在本申请中，集群中各节点可以提供相同的业务，因此，集群也可以称作业务集群。
容器(container)：一个或多个进程(process)的集合(包括运行所需的所有文件)，可在计算机之间进行移植。进程是指计算机中在执行的程式(计算机程序)。
容器集(pod)：一组共享相同计算资源的容器的集合。其中，一组共享相同计算资源的容器可以包括一个或多个容器，计算资源可以包括处理器，如中央处理器(central processing unit，CPU)。不同容器集的计算资源汇集，形成若干业务集群，这些业务集群可以提供功能更强大、智能化程度更高的分布式系统，用于执行相应的应用。
容器编排是指自动化容器的部署、管理、扩展和联网。容器编排通常可以由容器编排平台实现。容器编排平台也称作容器编排工具，用于在生命周期内管理大量的容器，包括镜像分发、冗余部署、健康监测、资源分配、弹性伸缩、负载均衡和调度。容器编排平台包括但不限于Apache Mesos、Nomad、Docker Swarm或kubernetes(简称为k8s)。为了便于描述，下文以kubernetes示例说明。
容器编排平台通常以pod为最小调度单元运行工作负载，并为pod提供所需的联网和存储等服务。容器编排平台的负载均衡功能可以使得pod之间达成负载均衡，容器编排平台的弹性伸缩功能可以使得pod数量能够满足业务需求。
具体地，容器编排平台可以通过pod水平弹性伸缩控制器调整pod数量。如图1所示，HPA可以创建副本集(replica set)对象或部署(deployment)对象，replica set对象或deployment对象均可以视为复制控制器(replication controller，RC)，复制控制器用于为指定的pod维护一个副本(实例)数量稳定的集合。例如，deploy对象中设置pod数量为N，N为正整数，则pod数量大于N时，可以指示删除多余的pod，pod数量小于N时，可以指示创建新的pod。
其中，HPA可以调整replica set对象或deployment对象，以部署更多的pod或者缩减已部署的pod，从而匹配观察到的指标，如平均CPU利用率、平均内存利用率或其他自定义指标。具体地，HPA可以根据当前指标和期望指标计算期望副本数，如下所示：
期望副本数＝当前副本数*(当前指标/期望指标)   (1)
以指标为平均CPU利用率示例说明。该示例中，若当前的平均CPU利用率为20％，期望的平均CPU利用率为10％，则期望副本数在当前副本数的基础上加倍，若当前的平均CPU利用率为5％，则期望副本数在当前副本数的基础上减半。
HPA主要用于实现pod级别(实例级别)的弹性伸缩。容器编排平台还可以基于集群弹性伸缩控制器(cluster autoscaler，CA)进行节点级别的弹性伸缩。其中，弹性伸缩包括弹性扩容和弹性缩容。扩容是指扩充pod或扩充节点，缩容是指删除(缩减)pod或删除节点。当业务集群的容量不足时，CA可以创建新的节点，当业务集群中节点在长时间(如10分钟)的资源利用率较低时，则可以删除该节点，以缩减成本。
目前，容器编排平台中的HPA和CA通常配合使用。HPA观察replica set对象或deployment对象的资源利用率，当资源利用率过高，HPA新建pod以应对高负载压力。随着pod增多，节点资源不够调度pod时，CA触发集群扩容，以扩充节点。反之，当HPA观察到replica set对象或deployment对象的资源利用率过低，HPA缩减pod减少资源消耗。随着pod减少，节点资源利用率相应降低。当节点资源利用率低于缩容阈值时，CA可以触发集群缩容，以删除节点进而节省资源。
进一步地，针对可中断的业务和不可中断的业务，CA可以采用不同的缩容策略。具体地，针对可中断业务，如信息查询系统等web业务，当节点资源利用率低于缩容阈值时，CA可以中断节点上剩余的pod，释放该节点，并重新调度被中断的pod。针对不可中断的业务，如直播场景中的转码业务，CA可以提供如下配置参数：容器集中断预算(pod disruption budget，PDB)，保证业务集群中工作或活动的pod不小于该PDB。如果释放一个节点导致业务集群中活动的pod小于PDB，则不释放该节点。
CA的目的是随着业务高峰或低谷的变化，跟随HPA相应动态调整业务集群的规模，以达到节点资源按需使用的效果。然而，HPA对pod的弹性伸缩与CA对节点的弹性伸缩通常是割裂的。
参见图2所示的pod弹性伸缩和节点弹性伸缩的示意图，HPA可以计算pod的期望副本数，并向相应的执行单元发送相应的伸缩指令。其中，pod的期望副本数大于当前副本数，则表征需要进行扩容，扩容对应的执行单元可以是调度器scheduler，如volcano scheduler，调度器可以响应于伸缩指令，通过调度算法选择节点进行pod调度。pod的期望副本数小于当前副本数，则表征需要进行缩容，缩容对应的执行单元可以是deployment controller，如k8s controller，复制控制器可以响应于伸缩指令，通过缩容策略选择pod释放。CA根据节点资源利用率，对节点进行弹性伸缩。
HPA对pod进行弹性伸缩时，并未考虑节点的弹性，使得CA难以实现节点资源按需使用。
如图3所示，HPA下发伸缩指令至调度器后，调度器采用装箱调度策略进行pod调度。装箱调度策略的目标是使用更少的节点容纳更多的pod。装箱调度策略通常是一种单时刻(调度时刻)的资源上的最优解，但是在时序的维度，装箱调度策略并不是全局最优。其原因在于，pod的副本数可以随着业务的高峰、低谷相应变化，每个pod具有不同的生命周期。由于调度器通常是基于CPU和内存两个维度装箱，未考虑pod的生命周期，生命周期长短不一的pod会均衡分散在各个节点。随着时间推移，生命长度较短的pod陆续释放，最终在业务低谷期，长期存活的pod会分散在各个节点。对于不可中断的业务，节点资源利用率虽然较低，但是难以缩容，对于可中断的业务，缩容过程可以造成大量pod中断。
如图4所示，缩容策略也同样未考虑节点上pod的生命周期分布。随着业务从高峰期至低谷期，HPA控制pod的副本数减少。在当前缩容策略下，每个节点上的pod则会表现出均衡减少，最终表现出同样的问题，即对于不可中断业务，节点资源利用率虽然较低，但是难以缩容，对于可中断的业务，缩容过程可以造成大量pod中断。
有鉴于此，本申请提供一种容器管理方法。该方法可以由容器管理系统执行。容器管理系统可以是集成于容器编排平台的系统。容器管理系统用于对部署至业务集群的pod或待部署至业务集群的pod进行管理。容器管理系统可以是软件系统，计算设备集群执行软件系统的程序代码，从而执行本申请的容器管理方法。容器管理系统也可以是硬件系统，该硬件系统运行时，执行本申请的容器管理方法。为了便于描述，下文以容器管理系统为软件系统示例说明。
具体地，容器管理系统可以获取业务集群中至少一个节点的生命周期以及至少一个pod的生命周期，其中，至少一个pod可以是待部署的pod，或者是至少一个节点中已部署的 pod，然后容器管理系统可以根据至少一个节点的生命周期以及至少一个pod的生命周期，确定目标节点。目标节点可以是待部署pod的节点，或者是待删除pod的节点。接着容器管理系统可以在目标节点对上述pod进行伸缩。
该方法中，容器管理系统结合业务集群中节点的生命周期以及待部署的pod或已部署的pod的生命周期，对pod进行伸缩，避免了pod的弹性伸缩与节点的弹性伸缩的割裂，使得CA能够实现节点资源按需使用，降低业务成本。
其中，节点的生命周期和pod的生命周期可以通过画像预测得到。如图5所示，容器管理系统可以在CPU、内存等资源利用率基础上，增加时间维度进行pod调度。具体地，容器管理系统可以结合节点的生命周期、待部署的pod的生命周期，为待部署的pod确定目标节点，然后将该待部署的pod调度至目标节点。为了便于描述，本申请将上述调度策略称作基于生命周期的调度策略。基于生命周期的调度策略可以实现将生命周期接近的pod调度至相同节点，当业务的低谷期到达时，例如是xx小时后，可以先删除同一节点上的pod。同一节点上的pod均被删除，该节点可以被释放，如此减少了资源浪费，降低了业务成本。
进一步地，当业务的高峰期到达，容器管理系统可以根据伸缩指令，获得待部署的pod，然后根据该待部署的pod的生命周期与节点的生命周期(如VM的剩余生命周期)确定目标节点，然后将待部署的pod调度至与其生命周期接近的目标节点。在图6的实例中，目标节点可以为VM2，容器管理系统可以将待部署的pod调度至VM2。
基于生命周期的调度策略实现的前提是每个pod在调度时就确定其删除顺序，也称作缩容顺序。本申请默认设置缩容顺序与扩容顺序相反，越晚扩容的pod越优先释放，如此，在调度期可以根据画像确定每个pod的生命周期，具体是每个pod的生命长度。
而在业务部署的初始阶段，缺乏数据，画像不够准确时，本申请还设计了一种过渡性策略。基于本申请的默认缩容顺序，可以表现出早期扩出的pod存活较长，当pod副本数越多，越接近业务高峰期，pod存活时间越短，基于这一特性，本申请的过渡性策略可以为：根据pod扩容顺序早晚划分为若干阶段，每个阶段的pod优先调度至相同节点。
当生命周期画像不准确，过渡策略不准确等因素导致pod被调度至不合适的节点(例如是长周期pod被调度至短周期节点)，而pod被调度至不合适的节点可以使得节点残留少许长周期pod，无法释放。因此，在缩容过程中，容器管理系统还可以动态调整缩容顺序，将被调度至不合适的节点的pod的优先级调高，使得该pod能够被优先删除或释放。
图7还提供了一个示例进行说明。在该示例中，业务集群包括4个VM节点，横向为时间轴，pod横向长度代表其存活时长，也即生命周期。图7中左图表示指定pod的一个副本pod1(如深灰色矩形块所示)被错误调度至VM4，导致VM4和VM1的资源利用率低，无法释放。容器管理系统可以动态优化缩容顺序，具体地，在指定pod的另一个副本pod6(如VM1中的深灰色矩形块所示)缩容之前，将pod1和pod缩容顺序互换，优先缩容pod1，如此，VM4中的pod均被缩容后，VM4可以被CA释放。图7中右图展示了缩容顺序互换后的资源使用情况，与左图相比，节省了大量资源，降低了业务成本。
为了使得本申请的技术方案更加清楚、易于理解，下面对本申请实施例的容器管理系 统的系统架构进行介绍。
参见图8所示的容器管理系统的系统架构图，容器管理系统10包括生命周期画像模块100、生命周期调度模块200和顺序优化模块300。其中，生命周期调度模块200、顺序优化模块300为可选模块。例如，容器管理系统10可以包括生命周期画像模块100和生命周期调度模块200。又例如，容器管理系统10可以包括生命周期画像模块100和顺序优化模块300。
具体地，HPA可以根据replica set对象或deployment对象的资源利用率，确定pod的期望副本数。例如，在图8的示例中，job1对应的pod的期望副本数可以为3，job2对应的期望副本数可以为2，job3对应的期望副本数可以为4，job4对应的期望副本数可以为2。相应地，HPA可以修改replica set对象或deployment对象等工作负载资源，例如是k8s resource，容器管理系统10可以响应于工作负载资源的变化，基于生命周期进行pod伸缩。
例如，期望副本数大于当前副本数时，则表示需要扩充pod，生命周期画像模块100用于对节点以及待部署的pod进行画像，获得节点的生命周期以及待部署的pod的生命周期，然后生命周期调度模块200用于根据节点的生命周期以及待部署的pod的生命周期确定目标节点，例如是从集群资源池中确定目标节点，然后将待部署的pod调度至目标节点。其中，集群资源池可以包括周期节点池或按需计费节点池中的一种或多种。周期节点池可以是包年或包月计费的长周期节点池。
又例如，期望副本数小于当前副本数时，则表示需要删除pod，生命周期画像模块100用于对节点以及已部署的pod进行画像，获得节点的生命周期以及已部署的pod的生命周期，然后顺序优化模块300可以根据节点的生命周期以及已部署的pod的生命周期，确定目标节点，然后调整目标节点中已部署的pod的缩容顺序(也即删除顺序)，并按照调整后的缩容顺序对目标节点上已部署的pod进行删除。当目标节点上已部署的pod被全部删除，CA还可以释放该目标节点，从而控制节点数量。
容器编排平台中的模块可以通过接口服务器交互，例如kubernetes中的模块可以通过kube-apiserver交互。容器编排平台可以提供deployment、replica set或者statefulset多种原生pod编排管理方法，其对应的控制器通过与kube-apiserver交互执行控制逻辑，controller还对外提供了接口，外部可以通过kube-apiserver控制缩容顺序。调度器通过kube-apiserver感知业务层的pod变化，并将pod绑定至相应节点。
需要说明的是，容器编排平台还可以针对个性化编排需求，提供可定制扩展能力(Custom Resource Define，CRD)。CRD支持开发者自定义资源，以提高可扩展性。
本申请的容器管理系统10可以包括多种产品形态。例如，该容器管理系统10可以是基于调度器的产品形态。又例如，该容器管理系统10也可以是基于容器编排平台的插件如kubernetes的插件的产品形态。还例如，该容器管理系统10可以是基于内核修改的产品形态，具体为基于kubernetes内核修改的产品形态。下面结合附图对上述产品形态进行介绍。
首先，参见图9所示的一种基于调度器的容器管理系统10的结构示意图，如图9所示，容器管理系统10的生命周期画像模块100、生命周期调度模块200和顺序优化模块300均在调度器中实现。其中，生命周期画像模块100可以通过apiserver感知上层的业务变化，从而进行生命周期画像。生命周期调度模块200通过调度器执行，顺序优化模块300可以 通过apiserver调整pod的缩容顺序。
在该架构下，容器管理系统10还支持对用户(如开发者)开发的CRD资源中的pod进行管理。例如，容器管理系统10可以CRD资源进行接口适配，该接口的形式可以与原生接口一致，也可以采用统一定制的交付接口。如此，容器管理系统10可以将CRD资源中的pod和原生资源如deployment资源中的pod进行统一管理，例如统一按照生命周期进行调度，或者统一按照基于生命周期调整后的缩容顺序进行缩容。
然后，参见图10所示的一种基于插件的容器管理系统10的结构示意图，区别于图9中的容器管理系统10，图10中的容器管理系统10的各个模块独立部署，通过apiserver交互。其中，生命周期调度模块200依托于调度器执行，生命周期画像模块100、顺序优化模块300为独立插件。
接着，参见图11所示的一种基于内核的容器管理系统10的结构示意图，区别于图10的容器管理系统10，顺序优化模块300实现在kubernetes内核中，例如是kubernetes apiserver中。基于此，无论是原生资源如deployment资源，还是自定义的CRD资源，伸缩指令中用于缩容的缩容指令都可以被拦截，顺序优化模块300进行缩容顺序优化化，再下发优化后的缩容指令，以便于相应的控制器可以执行优化后的缩容指令。
上文对容器管理系统10进行了详细说明，接下来，将从容器管理系统10的角度，对本申请实施例的容器管理方法进行详细说明。
参见图12所示的容器管理方法的流程图，该方法包括：
S1202：容器管理系统10获取业务集群中至少一个节点的生命周期以及至少一个pod的生命周期。
至少一个pod可以是待部署的pod或者是已部署的pod。例如，HPA指示扩充pod(即pod级别扩容)时，至少一个pod可以是待部署的pod，该待部署的pod可以根据deployment对象或者replica set对象中定义的pod模板创建得到。又例如，HPA指示删除pod(即pod级别缩容)时，至少一个pod可以是已部署的pod。
具体地，容器管理系统10可以通过生命周期画像，获取业务集群中至少一个节点的生命周期，以及至少一个pod的生命周期。下面对pod的生命周期画像和节点的生命周期画像分别进行说明。
参见图13所示的副本集中pod存活时长统计分析示意图，容器管理系统10可以获取至少一个pod对应的副本集中的副本在历史时间段的存活时长分布，如图13中的上图所示，存活时长分布的横轴为时间，纵轴为副本数量，表示在相应时刻工作或活动的pod副本。如图13中的下图所示，容器管理系统10还可以根据上述存活时长分布进行转换，从而确定生命长度随着扩容顺序的变化。基于图13中的下图可知，副本集中副本的扩容顺序越接近峰值，pod的生命长度越短，副本的扩容顺序越接近低谷，pod的生命长度越长。容器管理系统10可以通过统计策略，预测至少一个pod的生命周期。其中，统计策略包括但不限于机器学习、分位数、均值、最大值或概率分布中的一种或多种。
其中，容器管理系统10可以采用栈跟踪每个pod副本的生命周期。为了便于理解，下面结合一具体示例进行说明。
参见图14所示的采用栈跟踪pod副本的示意图，如图14所示，栈可以记录每个pod副本如每个deployment pod的生命周期变化规律。具体地，在图14的示例中，在时刻为8：00时，deployment的pod数为1，可以将8：00时间入栈，在时刻为12:00时，deployment增加一个pod，继续将12:00入栈，在时刻为14:00时，deployment增加一个pod，继续将14:00入栈，在时刻为16:00，工作负载减少，deployment减少一个pod，例如将栈顶的14:00出栈。容器管理系统10可以基于上述记录绘制存货时长分布。
进一步地，容器管理系统10可以计算入栈和出栈的时间差，将该时间差作为pod副本的生命长度。如图15所示，针对记录有入栈时间、出栈时间的pod副本，容器管理系统10可以计算出该pod副本的生命长度。其中，pod副本在整个历史时间段可以被调度多次，容器管理系统10可以计算出多个生命长度。例如，容器管理系统10第三个扩容的pod副本(记作replicas 3)的生命长度包括20：00、21:00或19:51。
需要说明的是，栈中未记录出栈时间的pod副本可以视为具有较长生命周期的pod副本，例如，第一个pod副本(记作replicas 1)、第二个pod副本(记作replicas 2)可以具有较长生命周期。
容器管理系统10可以根据各个pod副本的生命长度，通过统计策略，如最大值，最小值，均值，分位数(例如中位数，P99等)，均值，概率分布，或者是机器学习，预测各个pod副本的生命周期。例如，当该deployment扩充第三个pod副本，如果使用中位值预测，可以预知该pod将存活20小时，如果使用最大值则可以预测将存活21个小时。
在获得deployment下的pod的生命周期，容器管理系统10可以根据至少一个节点上pod的生命周期以及至少一个节点上pod的创建时间，确定至少一个节点的生命周期。例如，容器管理系统10可以计算至少一个节点上各个pod的剩余存活时间，基于该剩余存活时间确定节点的生命周期。类似地，容器管理系统10可以基于剩余存活时间，基于统计策略，确定节点的生命周期。类似地，统计策略可以包括机器学习，分位数，均值，最大值，概率分布中的一种或多种。在一些示例中，容器管理系统10可以确定剩余存活时间的最大值作为节点的生命周期。
S1204：容器管理系统10根据所述至少一个节点的生命周期以及所述至少一个pod的生命周期，确定目标节点。
在扩容阶段，至少一个pod为待部署的pod，目标节点为待部署pod的节点。在缩容阶段，至少一个pod为已部署的pod时，目标节点为待删除pod的节点。下面分别对不同阶段确定目标节点的具体实现方式进行示例说明。
在扩充pod的情况下，容器管理系统10可以确定待部署的pod的生命周期与至少一个节点的生命周期的接近程度，然后根据该接近程度，从至少一个节点中确定目标节点。
其中，pod的生命周期与节点的生命周期的接近程度可以根据生命周期的差值或生命周期的比值确定。例如，pod的生命周期与节点的生命周期的接近程度可以为pod的生命周期和节点的生命周期的比值，或者是上述比值的倒数，即节点的生命周期和pod的生命周期的比值。
在一些可能的实现方式中，容器管理系统10可以根据pod的生命周期与至少一个节点的接近程度对至少一个节点排序，然后根据排序结果从至少一个节点中确定目标节点。例 如，容器管理系统10可以根据排序结果将接近程度低于预设值的节点过滤，从剩余的节点中确定目标节点。该目标节点的资源能够容纳待部署的pod。
在另一些可能的实现方式中，容器管理系统10可以根据接近程度对至少一个节点进行评分，根据至少一个节点的评分从至少一个节点中确定目标节点。为了便于描述，下面以至少一个节点中的第一节点示例说明。
假设待部署的pod包括第一pod。第一pod的生命周期小于第一节点的生命周期时，第一节点的评分与第一接近程度正相关。其中，第一接近程度根据第一pod的生命周期和第一节点的生命周期的比值确定。第一pod的生命周期不小于第一节点的生命周期时，第一节点的评分与第二接近程度正相关。第二接近程度根据所述第一节点的生命周期和所述第一容器集的生命周期的比值确定。在一些示例中，第一节点的评分具体可以参见如下公式：
其中，a、b、c、d为系数，score为评分，podlife为pod的生命周期，nodelife为节点的生命周期。
评分策略不局限于以上方法，只需要保证：pod不延长node的生命周期前提下，生命周期越接近评分越高；pod在延长node的生命周期前提下，延长越多评分越低。
其中，容器管理系统10可以选择资源能够容纳第一pod的节点中评分最高的节点，作为目标节点。在一些实施例中，容器管理系统10也可以选择资源能够容纳第一pod的节点中评分大于设定评分的节点，作为目标节点。
在删除pod的情况下，容器管理系统10可以根据至少一个节点的生命周期以及节点上pod的生命周期，确定可优化的碎片化节点作为目标节点。具体地，容器管理系统10可以先根据至少一个节点的生命周期以及至少一个节点上pod的生命周期，确定候选的第二节点。
其中，候选的第二节点可以为长周期节点，候选的第二节点上的pod为长周期pod。例如，候选的第二节点可以在低谷器剩余长周期pod。长周期节点的生命周期大于第一时长，长周期pod的生命周期大于第二时长。该第一时长和第二时长可以根据经验值设置。在一些示例中，第一时长和第二时长可以设置为相等，或者设置为不等。其中，长周期节点、长周期pod通常不会在业务的低谷期被删除，与之相对的弹性节点、弹性pod可以在业务的低谷期被删除。
然后，容器管理系统10确定候选的第二节点上pod的至少一种候选删除顺序，预测按照候选删除顺序删除第二节点上容器集的收益，该收益根据集群的资源利用率确定，容器管理系统10可以根据上述收益确定目标删除顺序，并从候选的第二节点中确定目标节点。例如，容器管理系统10可以确定使得收益最大化的候选删除顺序为目标删除顺序，基于该目标删除顺序确定候选的第二节点中能够被删除的节点为目标节点。
考虑到候选的第二节点中可以存在不能被优化的节点，容器管理系统10可以选择部分可优化的节点作为目标节点。具体地，容器管理系统10可以按照pod数量对候选的第二节 点排序，例如是按照pod数量由小到大的顺序进行排序，容器管理系统10按照排序结果依次判断节点是否能够被优化，将能够被优化的节点确定为目标节点。
其中，容器管理系统10可以基于统计分析，预测按照候选删除顺序调整节点上第二pod删除顺序后，节点上长周期pod资源总和，或者各个deployment中长周期pod数量。如果存在deployment的长周期pod数量大于弹性pod数量，则跳过该节点，对下一个节点是否能够被优化进行判断。进一步地，当某个节点被优化后，导致累计的长周期pod资源总和超过集群的剩余空间，则可以终止节点筛选。
S1206：容器管理系统10在所述目标节点对所述容器集进行伸缩。
在扩容阶段，容器管理系统10可以将待部署的pod(如第一pod)调度至目标节点。在缩容阶段，容器管理系统10可以将待删除的pod(如第二pod)从目标节点删除。由此可以实现容器管理系统10在目标节点对pod进行伸缩。
需要说明的是，在缩容阶段，容器管理系统10可以按照目标删除顺序，调整目标节点上第二pod的删除顺序，然后按照调整后的删除顺序从目标节点删除第二pod。如此，当目标节点上的pod均被删除时，CA可以释放该目标节点，从而减少资源浪费，降低业务成本。
其中，容器管理系统10可以周期性地调整第二pod的缩容顺序，或者实时调整第二pod的缩容顺序。其中，周期性优化是指在业务的低谷期分析集群中pod的生命周期分布，确定可优化的碎片化节点作为目标节点，调整目标节点上第二pod的删除顺序(也称作缩容顺序、缩容优先级)，当进入业务的下一个低谷期，目标节点可以被释放。
下面结合附图分别对不同调整方式进行详细说明。
首先，参见图16所示的周期性优化缩容顺序的示意图，容器管理系统10可以在业务的低谷期，分析业务集群中已部署的pod的生命周期分布，根据该生命周期分布选择可优化的碎片化节点。
具体地，在该示例中，业务的高峰期扩充了新的pod，该业务的pod共包括3个，这3个pod分别被调度至VM2、VM4和VM5，其中，被调度至VM2的pod的缩容优先级为-3，被调度至VM4的pod的缩容优先级为-1，被调度至VM5的pod的缩容优先级为-1。在业务的低谷期，容器管理系统10按照该缩容优先级，优先对VM2上的pod进行缩容。容器管理系统10通过分析业务集群中已部署的pod的生命周期分布，确定VM4为可优化的碎片化节点，可以将VM4确定为目标节点。容器管理系统10调整VM4上pod的缩容顺序，例如是在下一个高峰期，在VM2上扩充新的pod时，将VM4上pod的缩容顺序与VM2上新的pod的缩容顺序进行互换。如此，容器管理系统10可以在业务的下一个低谷期先对VM4上的pod进行缩容，当VM4上的pod均被删除，VM4可以被释放。进一步地，容器管理系统10可以在上述下一个低谷期，将VM5上pod的缩容优先级标记为-3，以便于在下下个低谷期对VM5上的pod进行缩容，当VM5上的pod均被删除，VM5可以被释放。
其次，参见图17所示的实时优化缩容顺序的示意图，容器管理系统10可以实时分析业务集群中pod的缩容顺序，通过数学优化方法确定缩容顺序调整策略。例如，在业务的高峰期，容器管理系统10即可确定缩容顺序调整策略为将VM2上pod的缩容顺序与VM4上pod的缩容顺序进行互换。然后，容器管理系统10可以在业务的低谷期到达之前，根据 实时分析的缩容顺序调整策略，调整pod的缩容顺序(即缩容优先级)。容器管理系统10可以在业务的低谷期，按照调整后的缩容顺序进行缩容，使得节点可以被顺利释放。与周期性优化缩容顺序相比，实时优化缩容顺序的方式生效更快，具有较好的效果。
基于上述实施例的容器管理方法，本申请还提供一种容器管理系统。该容器管理系统用于对部署至业务集群的容器集或待部署至所述业务集群的容器集进行管理，如图8所示，该容器管理系统10：
生命周期画像模块100，用于获取所述业务集群中至少一个节点的生命周期以及至少一个容器集的生命周期，所述节点用于部署所述容器集；
生命周期调度模块200，用于根据所述至少一个节点的生命周期以及所述至少一个容器集的生命周期，确定目标节点，所述目标节点为待部署所述容器集的节点，或者是待删除所述容器集的节点；
所述生命周期调度模块200，还用于在所述目标节点对所述容器集进行伸缩。
示例性地，上述生命周期画像模块100、生命周期调度模块200可以通过硬件实现，或者可以通过软件实现。为了便于描述，下面以生命周期调度模块200示例说明。
当通过软件实现时，生命周期调度模块200可以是运行在计算设备上的应用程序，如计算引擎等。该应用程序可以以虚拟化服务的方式提供给用户使用。虚拟化服务可以包括虚拟机(virtual machine，VM)服务、裸金属服务器(bare metal server，BMS)服务以及容器(container)服务。其中，VM服务可以是通过虚拟化技术在多个物理主机(如计算设备)上虚拟出虚拟机(virtual machine，VM)资源池以为用户按需提供VM进行使用的服务。BMS服务是在多个物理主机上虚拟出BMS资源池以为用户按需提供BMS进行使用的服务。容器服务是在多个物理主机上虚拟出容器资源池以为用户按需提供容器进行使用的服务。VM是模拟出来的一台虚拟的计算机，也即逻辑上的一台计算机。BMS是一种可弹性伸缩的高性能计算服务，计算性能与传统物理机无差别，具有安全物理隔离的特点。容器是一种内核虚拟化技术，可以提供轻量级的虚拟化，以达到隔离用户空间、进程和资源的目的。应理解，上述虚拟化服务中的VM服务、BMS服务以及容器服务仅仅是作为具体的示例，在实际应用中，虚拟化服务还可以是其他轻量级或者重量级的虚拟化服务，此处不作具体限定。
当通过硬件实现时，生命周期调度模块200中可以包括至少一个计算设备，如服务器等。或者，生命周期调度模块200也可以是利用专用集成电路(application-specific integrated circuit，ASIC)实现、或可编程逻辑器件(programmable logic device，PLD)实现的设备等。其中，上述PLD可以是复杂程序逻辑器件(complex programmable logical device，CPLD)、现场可编程门阵列(field-programmable gate array，FPGA)、通用阵列逻辑(generic array logic，GAL)或其任意组合实现。
在一些可能的实现方式中，所述至少一个容器集包括待部署的所述容器集，所述生命周期调度模块200具体用于：
确定待部署的所述容器集的生命周期与所述至少一个节点的生命周期的接近程度；
根据所述接近程度，从所述至少一个节点中确定目标节点；
所述生命周期调度模块200具体用于：
将待部署的所述容器集调度至所述目标节点。
在一些可能的实现方式中，所述生命周期调度模块200具体用于：
根据所述接近程度对所述至少一个节点排序，根据排序结果从所述至少一个节点中确定目标节点；或者，
根据所述接近程度对所述至少一个节点进行评分，根据至少一个节点的评分从所述至少一个节点中确定目标节点。
在一些可能的实现方式中，所述至少一个节点包括第一节点，所述至少一个容器集包括第一容器集；
所述第一容器集的生命周期小于所述第一节点的生命周期时，所述第一节点的评分与第一接近程度正相关，所述第一接近程度根据所述第一容器集的生命周期和所述第一节点的生命周期的比值确定；
所述第一容器集的生命周期不小于所述第一节点的生命周期时，所述第一节点的评分与第二接近程度正相关，所述第二接近程度根据所述第一节点的生命周期和所述第一容器集的生命周期的比值确定。
在一些可能的实现方式中，所述系统10还包括：
顺序优化模块300，用于根据所述至少一个节点的生命周期以及所述至少一个节点上容器集的生命周期，确定候选的第二节点；确定所述候选的第二节点上容器集的至少一种候选删除顺序，预测按照所述候选删除顺序删除所述第二节点上容器集的收益，所述收益根据所述集群上的资源利用率确定；根据所述收益，确定目标删除顺序；
所述生命周期调度模块200具体用于：
根据所述收益，从所述候选的第二节点中确定目标节点；
所述生命周期调度模块200具体用于：
按照所述目标删除顺序调整所述目标节点上第二容器集的删除顺序，并按照调整后的删除顺序对所述目标节点上所述第二容器集进行删除。
与生命周期画像模块100、生命周期调度模块200类似，顺序优化模块300可以通过硬件实现，或者可以通过软件实现。当通过软件实现时，顺序优化模块300可以是运行在计算设备上的应用程序，如计算引擎等。该应用程序可以以虚拟化服务的方式提供给用户使用。当通过硬件实现时，顺序优化模块300中可以包括至少一个计算设备，如服务器等。或者，顺序优化模块300也可以是利用专用集成电路ASIC实现、或可编程逻辑器件PLD实现的设备等。
在一些可能的实现方式中，所述生命周期调度模块200具体用于：
在业务的低谷期，周期性调整所述第二容器集的删除顺序；或者，
在所述业务的低谷期到达之前，根据实时分析的删除顺序调整策略，调整所述第二容器集的删除顺序。
在一些可能的实现方式中，所述生命周期画像模块100具体用于：
获取所述至少一个容器集对应的副本集中的副本在历史时间段的存活时长分布；
根据所述至少一个容器集对应的副本集中的副本在历史时间段的存活时长分布，通过统计策略，预测所述至少一个容器集的生命周期。
在一些可能的实现方式中，所述生命周期画像模块100具体用于：
根据所述至少一个节点上容器集的生命周期以及所述至少一个节点上容器集的创建时间，确定所述至少一个节点的生命周期。
在一些可能的实现方式中，所述容器管理系统10部署在调度器中。
在一些可能的实现方式中，所述容器管理系统10分布式部署在不同设备，并且所述容器管理系统10中的不同模块通过应用程序编程接口API服务器交互。
在一些可能的实现方式中，所述容器管理系统10中的顺序优化模块为独立插件，或者通过对容器编排平台的内核改造获得。
本申请还提供一种计算设备1800。如图18所示，计算设备1800包括：总线1802、处理器1804、存储器1806和通信接口1808。处理器1804、存储器1806和通信接口1808之间通过总线1802通信。计算设备1800可以是服务器或终端设备。应理解，本申请不限定计算设备1800中的处理器、存储器的个数。
总线1802可以是外设部件互连标准(peripheral component interconnect，PCI)总线或扩展工业标准结构(extended industry standard architecture，EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图18中仅用一条线表示，但并不表示仅有一根总线或一种类型的总线。总线1802可包括在计算设备1800各个部件(例如，存储器1806、处理器1804、通信接口1808)之间传送信息的通路。
处理器1804可以包括中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、微处理器(micro processor，MP)或者数字信号处理器(digital signal processor，DSP)等处理器中的任意一种或多种。
存储器1806可以包括易失性存储器(volatile memory)，例如随机存取存储器(random access memory，RAM)。处理器1804还可以包括非易失性存储器(non-volatile memory)，例如只读存储器(read-only memory，ROM)，快闪存储器，机械硬盘(hard disk drive，HDD)或固态硬盘(solid state drive，SSD)。存储器1806中存储有可执行的程序代码，处理器1804执行该可执行的程序代码以实现前述容器管理方法。具体的，存储器1806上存有容器管理系统10用于执行容器管理方法的指令。
通信接口1808使用例如但不限于网络接口卡、收发器一类的收发模块，来实现计算设备1800与其他设备或通信网络之间的通信。
本申请实施例还提供了一种计算设备集群。该计算设备集群包括至少一台计算设备。该计算设备可以是服务器，例如是中心服务器、边缘服务器，或者是本地数据中心中的本地服务器。在一些实施例中，计算设备也可以是台式机、笔记本电脑或者智能手机等终端设备。
如图19所示，所述计算设备集群包括至少一个计算设备1800。计算设备集群中的一个或多个计算设备1800中的存储器1806中可以存有相同的容器管理系统10用于执行容器管理方法的指令。
在一些可能的实现方式中，该计算设备集群中的一个或多个计算设备1800也可以用于 执行容器管理系统10用于执行容器管理方法的部分指令。换言之，一个或多个计算设备1800的组合可以共同执行容器管理系统10用于执行容器管理方法的指令。
需要说明的是，计算设备集群中的不同的计算设备1800中的存储器1806可以存储不同的指令，用于执行容器管理系统10的部分功能。
图20示出了一种可能的实现方式。如图20所示，两个计算设备1800A和1800B通过通信接口1808实现连接。计算设备1800A中的存储器上存有用于执行生命周期画像模块100和的功能的指令。计算设备1800B中的存储器上存有用于执行生命周期调度模块200的功能的指令。进一步地，计算设备1800B中的存储器上还存有顺序优化模块300的功能的指令。换言之，计算设备1800A和1800B的存储器1806共同存储了容器管理系统10用于执行容器管理方法的指令。
图20所示的计算设备集群之间的连接方式可以是考虑到本申请提供的容器管理方法需要采用大量算力确定删除顺序，进而确定目标节点。因此，考虑将顺序优化模块300实现的功能也交由计算设备1800B执行。
应理解，图20中示出的计算设备1800A的功能也可以由多个计算设备1800完成。同样，计算设备1800B的功能也可以由多个计算设备1800完成。
在一些可能的实现方式中，计算设备集群中的一个或多个计算设备可以通过网络连接。其中，所述网络可以是广域网或局域网等等。图21示出了一种可能的实现方式。如图21所示，两个计算设备1800C和1800D之间通过网络进行连接。具体地，通过各个计算设备中的通信接口与所述网络进行连接。在这一类可能的实现方式中，计算设备1800C中的存储器1806中存有执行生命周期画像模块100的功能的指令。同时，计算设备1800D中的存储器1806中存有执行生命周期调度模块200的功能的指令。进一步地，计算设备1800B中的存储器上还存有顺序优化模块300的功能的指令。
图21所示的计算设备集群之间的连接方式可以是考虑到本申请提供的容器管理方法需要采用大量算力确定删除顺序，进而确定目标节点，因此考虑将生命周期调度模块200和顺序优化模块300实现的功能交由计算设备1800D执行。
应理解，图21中示出的计算设备1800C的功能也可以由多个计算设备1800完成。同样，计算设备1800D的功能也可以由多个计算设备1800完成。
本申请实施例还提供了一种计算机可读存储介质。所述计算机可读存储介质可以是计算设备能够存储的任何可用介质或者是包含一个或多个可用介质的数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘)等。该计算机可读存储介质包括指令，所述指令指示计算设备执行上述应用于容器管理系统10用于执行容器管理方法。
本申请实施例还提供了一种包含指令的计算机程序产品。所述计算机程序产品可以是包含指令的，能够运行在计算设备上或被储存在任何可用介质中的软件或程序产品。当所述计算机程序产品在至少一个计算设备上运行时，使得至少一个计算设备执行上述容器管理方法。
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以 对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的保护范围。

Claims (25)

  1. 一种容器管理方法，其特征在于，应用于容器管理系统，所述容器管理系统用于对部署至业务集群的容器集或待部署至所述业务集群的容器集进行管理，所述容器集包括一组容器，所述方法包括：
    所述容器管理系统获取所述业务集群中至少一个节点的生命周期以及至少一个容器集的生命周期，所述节点用于部署所述容器集；
    所述容器管理系统根据所述至少一个节点的生命周期以及所述至少一个容器集的生命周期，确定目标节点，所述目标节点为待部署所述容器集的节点，或者是待删除所述容器集的节点；
    所述容器管理系统在所述目标节点对所述容器集进行伸缩。
  2. 根据权利要求1所述的方法，其特征在于，所述至少一个容器集包括待部署的所述容器集，所述容器管理系统根据所述节点的生命周期以及所述容器集的生命周期，确定目标节点，包括：
    所述容器管理系统确定待部署的所述容器集的生命周期与所述至少一个节点的生命周期的接近程度；
    所述容器管理系统根据所述接近程度，从所述至少一个节点中确定目标节点；
    所述容器管理系统在所述目标节点对所述容器集进行伸缩，包括：
    所述容器管理系统将待部署的所述容器集调度至所述目标节点。
  3. 根据权利要求2所述的方法，其特征在于，所述容器管理系统根据所述接近程度，从所述至少一个节点中确定目标节点，包括：
    所述容器管理系统根据所述接近程度对所述至少一个节点排序，根据排序结果从所述至少一个节点中确定目标节点；或者，
    所述容器管理系统根据所述接近程度对所述至少一个节点进行评分，根据至少一个节点的评分从所述至少一个节点中确定目标节点。
  4. 根据权利要求3所述的方法，其特征在于，所述至少一个节点包括第一节点，所述至少一个容器集包括第一容器集；
    所述第一容器集的生命周期小于所述第一节点的生命周期时，所述第一节点的评分与第一接近程度正相关，所述第一接近程度根据所述第一容器集的生命周期和所述第一节点的生命周期的比值确定；
    所述第一容器集的生命周期不小于所述第一节点的生命周期时，所述第一节点的评分与第二接近程度正相关，所述第二接近程度根据所述第一节点的生命周期和所述第一容器集的生命周期的比值确定。
  5. 根据权利要求1所述的方法，其特征在于，所述容器管理系统根据所述至少一个节点的生命周期以及所述至少一个容器集的生命周期，确定目标节点，包括：
    所述容器管理系统根据所述至少一个节点的生命周期以及所述至少一个节点上容器集的生命周期，确定候选的第二节点；
    所述容器管理系统确定所述候选的第二节点上容器集的至少一种候选删除顺序，预测按照所述候选删除顺序删除所述第二节点上容器集的收益，所述收益根据所述集群上的资 源利用率确定；
    所述容器管理系统根据所述收益，确定目标删除顺序，并从所述候选的第二节点中确定目标节点；
    所述容器管理系统在所述目标节点对所述容器集进行伸缩，包括：
    所述容器管理系统按照所述目标删除顺序调整所述目标节点上第二容器集的删除顺序，并按照调整后的删除顺序对所述目标节点上所述第二容器集进行删除。
  6. 根据权利要求5所述的方法，其特征在于，所述容器管理系统调整所述目标节点上第二容器集的删除顺序，包括：
    在业务的低谷期，所述容器管理系统周期性调整所述第二容器集的删除顺序；或者，
    在所述业务的低谷期到达之前，所述容器管理系统根据实时分析的删除顺序调整策略，调整所述第二容器集的删除顺序。
  7. 根据权利要求1至6任一项所述的方法，其特征在于，所述容器管理系统获取至少一个容器集的生命周期，包括：
    所述容器管理系统获取所述至少一个容器集对应的副本集中的副本在历史时间段的存活时长分布；
    所述容器管理系统根据所述至少一个容器集对应的副本集中的副本在历史时间段的存活时长分布，通过统计策略，预测所述至少一个容器集的生命周期。
  8. 根据权利要求1至7任一项所述的方法，其特征在于，所述容器管理系统获取所述业务集群中至少一个节点的生命周期，包括：
    所述容器管理系统根据所述至少一个节点上容器集的生命周期以及所述至少一个节点上容器集的创建时间，确定所述至少一个节点的生命周期。
  9. 根据权利要求1至8任一项所述的方法，其特征在于，所述容器管理系统部署在调度器中。
  10. 根据权利要求1至8任一项所述的方法，其特征在于，所述容器管理系统分布式部署在不同设备，并且所述容器管理系统中的不同模块通过应用程序编程接口API服务器交互。
  11. 根据权利要求10所述的方法，其特征在于，所述容器管理系统中的顺序优化模块为独立插件，或者通过对容器编排平台的内核改造获得。
  12. 一种容器管理系统，其特征在于，所述容器管理系统用于对部署至业务集群的容器集或待部署至所述业务集群的容器集进行管理，所述容器集包括一组容器，所述系统包括：
    生命周期画像模块，用于获取所述业务集群中至少一个节点的生命周期以及至少一个容器集的生命周期，所述节点用于部署所述容器集；
    生命周期调度模块，用于根据所述至少一个节点的生命周期以及所述至少一个容器集的生命周期，确定目标节点，所述目标节点为待部署所述容器集的节点，或者是待删除所述容器集的节点；
    所述生命周期调度模块，还用于在所述目标节点对所述容器集进行伸缩。
  13. 根据权利要求12所述的系统，其特征在于，所述至少一个容器集包括待部署的所 述容器集，所述生命周期调度模块具体用于：
    确定待部署的所述容器集的生命周期与所述至少一个节点的生命周期的接近程度；
    根据所述接近程度，从所述至少一个节点中确定目标节点；
    所述生命周期调度模块具体用于：
    将待部署的所述容器集调度至所述目标节点。
  14. 根据权利要求13所述的系统，其特征在于，所述生命周期调度模块具体用于：
    根据所述接近程度对所述至少一个节点排序，根据排序结果从所述至少一个节点中确定目标节点；或者，
    根据所述接近程度对所述至少一个节点进行评分，根据至少一个节点的评分从所述至少一个节点中确定目标节点。
  15. 根据权利要求14所述的系统，其特征在于，所述至少一个节点包括第一节点，所述至少一个容器集包括第一容器集；
    所述第一容器集的生命周期小于所述第一节点的生命周期时，所述第一节点的评分与第一接近程度正相关，所述第一接近程度根据所述第一容器集的生命周期和所述第一节点的生命周期的比值确定；
    所述第一容器集的生命周期不小于所述第一节点的生命周期时，所述第一节点的评分与第二接近程度正相关，所述第二接近程度根据所述第一节点的生命周期和所述第一容器集的生命周期的比值确定。
  16. 根据权利要求12所述的系统，其特征在于，所述系统还包括：
    顺序优化模块，用于根据所述至少一个节点的生命周期以及所述至少一个节点上容器集的生命周期，确定候选的第二节点；确定所述候选的第二节点上容器集的至少一种候选删除顺序，预测按照所述候选删除顺序删除所述第二节点上容器集的收益，所述收益根据所述集群上的资源利用率确定；根据所述收益，确定目标删除顺序；
    所述生命周期调度模块具体用于：
    根据所述收益，从所述候选的第二节点中确定目标节点；
    所述生命周期调度模块具体用于：
    按照所述目标删除顺序调整所述目标节点上第二容器集的删除顺序，并按照调整后的删除顺序对所述目标节点上所述第二容器集进行删除。
  17. 根据权利要求16所述的系统，其特征在于，所述生命周期调度模块具体用于：
    在业务的低谷期，周期性调整所述第二容器集的删除顺序；或者，
    在所述业务的低谷期到达之前，根据实时分析的删除顺序调整策略，调整所述第二容器集的删除顺序。
  18. 根据权利要求12至17任一项所述的系统，其特征在于，所述生命周期画像模块具体用于：
    获取所述至少一个容器集对应的副本集中的副本在历史时间段的存活时长分布；
    根据所述至少一个容器集对应的副本集中的副本在历史时间段的存活时长分布，通过统计策略，预测所述至少一个容器集的生命周期。
  19. 根据权利要求12至18任一项所述的系统，其特征在于，所述生命周期画像模块 具体用于：
    根据所述至少一个节点上容器集的生命周期以及所述至少一个节点上容器集的创建时间，确定所述至少一个节点的生命周期。
  20. 根据权利要求12至18任一项所述的系统，其特征在于，所述容器管理系统部署在调度器中。
  21. 根据权利要求12至18任一项所述的系统，其特征在于，所述容器管理系统分布式部署在不同设备，并且所述容器管理系统中的不同模块通过应用程序编程接口API服务器交互。
  22. 根据权利要求21所述的系统，其特征在于，所述容器管理系统中的顺序优化模块为独立插件，或者通过对容器编排平台的内核改造获得。
  23. 一种计算设备集群，其特征在于，所述计算设备集群包括至少一台计算设备，所述至少一台计算设备包括至少一个处理器和至少一个存储器，所述至少一个存储器中存储有计算机可读指令；所述至少一个处理器执行所述计算机可读指令，以使得所述计算设备集群执行如权利要求1至11中任一项所述的方法。
  24. 一种计算机可读存储介质，其特征在于，包括计算机可读指令；所述计算机可读指令用于实现权利要求1至11任一项所述的方法。
  25. 一种计算机程序产品，其特征在于，包括计算机可读指令；所述计算机可读指令用于实现权利要求1至11任一项所述的方法。
PCT/CN2023/081285 2022-08-16 2023-03-14 一种容器管理方法及相关设备 WO2024036940A1 (zh)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202210983171 2022-08-16
CN202210983171.5 2022-08-16
CN202211507530.6A CN117632464A (zh) 2022-08-16 2022-11-29 一种容器管理方法及相关设备
CN202211507530.6 2022-11-29

Publications (1)

Publication Number Publication Date
WO2024036940A1 true WO2024036940A1 (zh) 2024-02-22

Family

ID=89940515

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/081285 WO2024036940A1 (zh) 2022-08-16 2023-03-14 一种容器管理方法及相关设备

Country Status (1)

Country Link
WO (1) WO2024036940A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170257432A1 (en) * 2011-02-09 2017-09-07 Cliqr Technologies Inc. Apparatus, systems and methods for container based service deployment
CN107688322A (zh) * 2017-08-31 2018-02-13 天津中新智冠信息技术有限公司 一种容器化管理系统
CN112269640A (zh) * 2020-11-02 2021-01-26 浪潮云信息技术股份公司 一种实现容器云组件的生命周期管理的方法
CN112463290A (zh) * 2020-11-10 2021-03-09 中国建设银行股份有限公司 动态调整计算容器的数量的方法、系统、装置和存储介质
CN112948050A (zh) * 2019-11-26 2021-06-11 西安华为技术有限公司 一种部署pod的方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170257432A1 (en) * 2011-02-09 2017-09-07 Cliqr Technologies Inc. Apparatus, systems and methods for container based service deployment
CN107688322A (zh) * 2017-08-31 2018-02-13 天津中新智冠信息技术有限公司 一种容器化管理系统
CN112948050A (zh) * 2019-11-26 2021-06-11 西安华为技术有限公司 一种部署pod的方法及装置
CN112269640A (zh) * 2020-11-02 2021-01-26 浪潮云信息技术股份公司 一种实现容器云组件的生命周期管理的方法
CN112463290A (zh) * 2020-11-10 2021-03-09 中国建设银行股份有限公司 动态调整计算容器的数量的方法、系统、装置和存储介质

Similar Documents

Publication Publication Date Title
JP7138126B2 (ja) リソース配置を最適化するための適時性リソース移行
Ghomi et al. Load-balancing algorithms in cloud computing: A survey
US8972983B2 (en) Efficient execution of jobs in a shared pool of resources
US8423646B2 (en) Network-aware virtual machine migration in datacenters
US9921866B2 (en) CPU overprovisioning and cloud compute workload scheduling mechanism
US9405572B2 (en) Optimized resource allocation and management in a virtualized computing environment
US11966792B2 (en) Resource processing method of cloud platform, related device, and storage medium
US20140282540A1 (en) Performant host selection for virtualization centers
CN111381928B (zh) 一种虚拟机迁移方法、云计算管理平台和存储介质
US20220100551A1 (en) Virtual disk management for efficient bin packing across nodes
US20200042398A1 (en) Systems and methods for predictive data protection
US11586471B2 (en) Computer system and control method for computer system
US20200174816A1 (en) Intelligent assignment of virtual machines to compute only or hyper converged nodes
WO2024036940A1 (zh) 一种容器管理方法及相关设备
US11336519B1 (en) Evaluating placement configurations for distributed resource placement
US11388050B2 (en) Accelerating machine learning and profiling over a network
KR20160043706A (ko) 가상 머신 스케일링 장치 및 그 방법
CN117632464A (zh) 一种容器管理方法及相关设备
Hanif et al. Jargon of Hadoop MapReduce scheduling techniques: a scientific categorization
WO2023066248A1 (zh) 数据处理方法、装置、设备和系统
WO2023246709A1 (zh) 数据处理方法、装置、设备和系统
US20240176677A1 (en) Energy efficient scaling of multi-zone container clusters
Patel et al. Existing and Relevant Methodologies for Energy Efficient Cloud Data centers
Lin et al. A novel virtual machine consolidation algorithm with server power mode management for energy-efficient cloud data centers
CN116069739A (zh) 数据处理方法、装置、设备和系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23853857

Country of ref document: EP

Kind code of ref document: A1