WO2024077885A1 - Management method, apparatus and device for container cluster, and non-volatile readable storage medium - Google Patents

Abstract

The present application relates to the technical field of cloud computing. Specifically disclosed are a management method, apparatus and device for a container cluster, and a non-volatile readable storage medium. In the management method, by means of creating a Kubernetes management cluster in advance, designing a computing resource management module and a partition module, preparing a plurality of versions of virtual machine mirror images, a container mirror image and a container cluster application deployment template, and quickly creating a Kubernetes load cluster in a cluster declaration mode and on the basis of a cluster application program interface, repetitive operations for creating different types of infrastructure resource pools and deploying different types of basic resources are simplified, and unified management of the different types of infrastructure resource pools and the different types of basic resources is realized, thereby effectively solving the problem in the related art of it not being possible to manage the infrastructure of a load cluster, and thus making it possible to support dedicated cloud delivery scenarios such as multi-CPU architecture infrastructure, multi-cluster unified management, rapid deployment and delivery, high availability, and disaster recovery and backup support for container clusters.

Description

Container cluster management method, device, equipment and non-volatile readable storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to a Chinese patent application filed with the Chinese Patent Office on October 10, 2022, with application number 202211231347.8 and application name “Management method, device, equipment and computer-readable storage medium for container clusters”, all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of cloud computing technology, and in particular to a management method, device, equipment and non-volatile readable storage medium for a container cluster.

Background technique

Kubernetes (K8s for short) is an open source containerized application for managing multiple hosts in a cloud platform. It can easily manage containerized applications across machines. As more and more containerized applications are involved in production, derivative projects and technical architecture solutions based on Kubernetes and various needs are emerging in an endless stream. The unified management capabilities of multiple clusters based on Kubernetes can achieve scenarios such as rapid deployment and delivery, high availability, and disaster recovery of container clusters. However, most of these implementations are based on Kubernetes cluster federation, without overall planning and design based on the cloud platform, and lack the unified management capabilities of container clusters across CPU (Central Processing Unit) architectures, making it difficult to adapt to users' diverse infrastructure environments.

The current solution for multi-cluster management is to simply register the user-created clusters in the cloud platform and use the cloud platform to monitor the resource status of the clusters, without involving the management of the user cluster's infrastructure. This not only makes the user cluster creation process more cumbersome, but also leads to poor availability of clusters and applications.

Providing a unified management solution for multi-architecture clusters based on a cloud platform with rapid deployment and delivery and high availability is a technical problem that needs to be solved by those skilled in the art.

Summary of the invention

The purpose of this application is to provide a container cluster management method, device, equipment and non-volatile readable storage medium, which are configured to achieve rapid deployment and delivery, and unified management of high-availability multi-architecture clusters based on a cloud platform.

To solve the technical problem, the present application provides a container cluster management method, including:

Pre-deploy the cluster application program interface in the kubernetes cluster based on the cloud platform to create a management cluster; configure the computing resource management module for unified management of infrastructure resource pools of different types of cloud platforms based on the cloud platform, and the partition management module set to maintain different versions of virtual machines; push multiple versions of virtual machine images to the image warehouse of each infrastructure resource pool; push the container images of the auxiliary components required by the kubernetes load cluster to the first container image warehouse of the cloud platform, and push the container cluster application deployment template to the chart warehouse of the cloud platform;

When receiving a kubernetes load cluster creation request sent by a client, identifying an infrastructure resource pool type of a first target load cluster to be created and an infrastructure type required by the first target load cluster;

Based on the cluster application program interface, a cluster declaration management method is adopted to call the computing resource management module and the partition management module to create a first target load cluster in the first infrastructure resource pool corresponding to the infrastructure resource pool type and complete the infrastructure deployment for the first target load cluster.

In some embodiments, multiple versions of virtual machine images are pushed to the image repository of each infrastructure resource pool, including:

In the process of making a virtual machine image, a binary resource configuration file related to Kubernetes of a specific version corresponding to the CPU architecture is placed in the image to a specified directory, a second container image repository is installed, and Kubernetes-related images corresponding to the CPU architecture and the corresponding version are pushed to the second container image repository;

Push the completed virtual machine image to the image repository of each infrastructure resource pool.

In some embodiments, the partitions of the partition management module correspond one-to-one to the virtual machine images;

Correspondingly, the partition information of the partition includes the CPU architecture, operating system type, operating system version, kubernetes version, container runtime type, container runtime version, and virtual machine image ID corresponding to the virtual machine in the first target load cluster.

In some embodiments, based on the cluster application program interface, a cluster declaration is adopted to call a computing resource management module and a partition management module to create a first target load cluster in a first infrastructure resource pool corresponding to the infrastructure resource pool type and complete infrastructure deployment for the first target load cluster, including:

After calling the computing resource management module to obtain the image service component interface of the first infrastructure resource pool, calling the partition management module to obtain the virtual machine image ID required by the first target load cluster in the image warehouse of the first infrastructure resource pool;

Generate and create an extended resource file based on the input parameters, and create an extended resource for the first target load cluster to control custom resources in the Kubernetes cluster;

When a custom resource is monitored, based on a cluster application program interface and a resource provisioning interface corresponding to the first infrastructure resource pool type, a kubernetes orchestration tool is called to create and deploy a kubernetes load cluster in the first infrastructure resource pool;

When it is monitored that the application interface services of all custom resources of the first target load cluster are deployed, the identifier of the first target load cluster is obtained and recorded;

Based on the client, the label setting role of the working node is modified, the container image required by the first target load cluster is called from the container image repository, the container cluster application deployment template required by the first target load cluster is obtained from the chart repository, and the installation and startup of the auxiliary components are completed in the first target load cluster.

In some embodiments, calling the partition management module to obtain the virtual machine image ID required by the first target load cluster from the image repository of the first infrastructure resource pool includes:

The partition management module is called to query the ID of the virtual machine image that meets the CPU architecture and kubernetes version conditions required by the kubernetes load cluster creation request as the virtual machine image ID.

In some embodiments, generating and creating an extended resource file according to input parameters, and creating an extended resource for managing and controlling custom resources in a kubernetes cluster for a first target load cluster includes:

Generate and create an extended resource file based on the input parameters and create extended resources. The extended resource is a CRD (CustomResourceDefinition) resource used to manage and control the kubernetes cluster.

In some embodiments, when a custom resource is monitored, based on a cluster application program interface and a resource provisioning interface corresponding to the first infrastructure resource pool type, a kubernetes orchestration tool is called to create and deploy a kubernetes load cluster in the first infrastructure resource pool, including:

When custom resources are monitored, the resource supply interface corresponding to the cloud platform and the first infrastructure resource pool type is used to call the interface of the infrastructure resource pool to create virtual machines, security groups and cloud hard disk resources, and the cloud platform initialization script is injected into the virtual machine, where the cloud platform initialization script is used to implement Kubernetes initialization, create and modify Kubernetes configuration parameters, and run the cloud platform initialization script to call the Kubernetes orchestration tool preset in the virtual machine image to deploy the first target load cluster.

In some embodiments, after monitoring that the application interface services of all custom resources of the first target load cluster are deployed, obtaining and recording the identifier of the first target load cluster includes:

Monitor the status of cluster.x-k8s resources, wait for the application interface service to be ready, then query the node list. After the application interface service is deployed, call the cluster application program interface to obtain the identifier of the first target load cluster, and record the identifier in the management record corresponding to the first target load cluster.

In some embodiments, completing the installation and activation of the auxiliary components in the first target load cluster includes:

Using a software package management tool in a container cluster application deployment template to install the attached components into the first target load cluster;

After the installation of the subsidiary components is completed, the subsidiary components are started, and the deployment completion status is recorded in the management record corresponding to the first target load cluster.

In some embodiments, it also includes:

When a deletion command for the created second target load cluster is received based on the client, the computing resource management module and the resource supply interface of the second infrastructure resource pool where the second target load cluster is located are called to delete the infrastructure of the second target load cluster in the second infrastructure resource pool.

In some embodiments, before calling the computing resource management module and the partition management module to create a first target load cluster in the first infrastructure resource pool corresponding to the infrastructure resource pool type and completing infrastructure deployment for the first target load cluster based on the cluster application program interface and managing the cluster declaration, the method further includes:

A one-to-one correspondence between the first target load cluster and the user is created, and a security authentication rule is configured for the first target load cluster.

In some embodiments, a one-to-one correspondence between the first target load cluster and the user is created, and a security authentication rule is configured for the first target load cluster, which may be:

The security authentication service interface of the first infrastructure resource pool is called to create a user, and corresponding security authentication rules are configured for the user.

In some embodiments, when the first infrastructure resource pool is an OpenStack infrastructure resource pool, the corresponding security authentication rules configured for the user may be:

Use nova, a computing component of the OpenStack infrastructure resource pool, to create a key pair.

In some embodiments, it also includes:

When receiving a deletion command for the created second target load cluster sent by the client, calling the computing resource management module and the resource supply interface of the second infrastructure resource pool where the second target load cluster is located, and deleting the infrastructure of the second target load cluster in the second infrastructure resource pool;

Call the computing resource management module and the resource supply interface of the second infrastructure resource pool, obtain the authentication information of the second target load cluster, delete the volume of the corresponding user under the second target load cluster, delete the corresponding security authentication rules, and delete the user information and the record of the second target load cluster in the cloud platform database.

In some embodiments, calling the computing resource management module and the resource supply interface of the second infrastructure resource pool where the second target load cluster is located, and deleting the infrastructure of the second target load cluster in the second infrastructure resource pool may include:

Invoke the computing resource management module to execute a deletion operation on the custom resources of the second target load cluster in the second infrastructure resource pool;

After the deletion operation is detected based on the cluster application program interface, the infrastructure of the second target load cluster is deleted in the second infrastructure resource pool through the resource supply interface of the second infrastructure resource pool.

In some embodiments, before calling the computing resource management module and the partition management module to create a first target load cluster in the first infrastructure resource pool corresponding to the infrastructure resource pool type and completing infrastructure deployment for the first target load cluster based on the cluster application program interface and managing the cluster declaration, the method further includes:

Perform permission verification and request parameter verification on kubernetes load cluster creation requests;

If the verification is passed, the process proceeds to the step of managing cluster declarations based on the cluster application program interface, calling the computing resource management module and the partition management module to complete the infrastructure resource pool deployment and infrastructure deployment steps for the first target load cluster.

In some embodiments, it also includes:

A corresponding management record is created for the first target load cluster in the database of the cloud platform, and the cluster information of the first target load cluster is synchronously updated to the management record.

To solve the technical problem, the present application also provides a container cluster management device, including:

The environment preparation unit is configured to pre-deploy a cluster application program interface in a kubernetes cluster based on a cloud platform to create a management cluster; configure a computing resource management module for unified management of infrastructure resource pools of different types of cloud platforms based on the cloud platform, and a partition management module for maintaining different versions of virtual machines; push multiple versions of virtual machine images to the image warehouse of each infrastructure resource pool; push the container image of the auxiliary components required by the kubernetes load cluster to the first container image warehouse of the cloud platform, and push the container cluster application deployment template to the chart warehouse of the cloud platform;

an identification unit, configured to, upon receiving a kubernetes load cluster creation request sent by a client, identify an infrastructure resource pool type of a first target load cluster to be created and an infrastructure type required by the first target load cluster;

The creation unit is configured to use a management cluster declaration method based on a cluster application program interface to call a computing resource management module and a partition management module to complete infrastructure resource pool deployment and infrastructure deployment for the first target load cluster.

To solve the technical problem, the present application also provides a container cluster management device, including:

a memory arranged to store a computer program;

The processor is configured to execute a computer program, and when the computer program is executed by the processor, the steps of any container cluster management method are implemented.

To solve the technical problem, the present application also provides a non-volatile readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned container cluster management methods are implemented.

The management method of the container cluster provided in the present application pre-deploys a cluster application program interface in a kubernetes cluster based on a cloud platform, creates a management cluster, configures a computing resource management module for unified management of infrastructure resource pools of different types of cloud platforms, and a partition management module for maintaining different versions of virtual machines, pushes different versions of virtual machine images for each infrastructure resource pool, pre-prepares container images of attached components, and a container cluster application deployment template, and upon receiving a kubernetes load cluster creation request sent by a client, identifies the infrastructure resource pool type and infrastructure type of the first target load cluster, and then uses a management cluster declaration method based on the cluster application program interface to call the computing resource management module and the partition management module to create the first target load cluster and complete the infrastructure deployment, thereby simplifying the repetitive operations of creating different types of infrastructure resource pools and deploying different types of basic resources, maintaining consistency and repeatability among infrastructures of different architectures, and realizing unified management of different types of infrastructure resource pools and different types of basic resources, effectively solving the problem that the infrastructure of the load cluster cannot be managed in the related technology, making it possible to support proprietary cloud delivery scenarios such as multi-CPU architecture infrastructure, unified management of multiple clusters, rapid deployment and delivery, high availability and disaster recovery and backup support for container clusters.

The present application also provides a management device, equipment and non-volatile readable storage medium for a container cluster, which have beneficial effects and are not described in detail here.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a flow chart of a method for managing a container cluster provided in an embodiment of the present application;

FIG. 2 is a flow chart of an optional implementation of S103 provided in an embodiment of the present application;

FIG3 is a schematic diagram of the structure of a container cluster management device provided in an embodiment of the present application;

FIG4 is a schematic diagram of the structure of a container cluster management device provided in an embodiment of the present application.

Detailed ways

The core of this application is to provide a container cluster management method, device, equipment and non-volatile readable storage medium for realizing unified management of multi-architecture clusters based on cloud platforms with rapid deployment and delivery and high availability.

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

FIG1 is a flowchart of a method for managing a container cluster provided in an embodiment of the present application.

As shown in FIG1 , the container cluster management method provided in the embodiment of the present application includes:

S101: Pre-deploy the cluster API in the Kubernetes cluster based on the cloud platform to create a management cluster; configure the computing resource management module based on the cloud platform to uniformly manage the infrastructure resource pools of different types of cloud platforms, and the computing resource management module for maintaining different versions. The partition management module of the virtual machine; pushes multiple versions of virtual machine images to the image repository of each infrastructure resource pool; pushes the container images of the auxiliary components required by the Kubernetes load cluster to the first container image repository of the cloud platform, and pushes the container cluster application deployment template to the chart repository of the cloud platform.

S102: When receiving a kubernetes load cluster creation request sent by a client, identify the infrastructure resource pool type of the first target load cluster to be created and the infrastructure type required by the first target load cluster.

S103: Based on the cluster application program interface, a cluster declaration method is adopted to call the computing resource management module and the partition management module to create a first target load cluster in the first infrastructure resource pool corresponding to the infrastructure resource pool type and complete infrastructure deployment for the first target load cluster.

The mainstream architecture design in the relevant technology is to manage the life cycle of servers and operating systems separately from that of kubernetes nodes: usually, a single-architecture machine is specified to pre-configure the operating system (OS), and then the kubernetes components are initialized into the running system (using kubeadm, kops or other kubernetes orchestration tools). Moreover, if these infrastructures and their operating systems are not part of kubernetes management, the availability of clusters and applications will be reduced. For example, when a user needs to create a kubernetes load cluster, although the load cluster can be created based on the existing infrastructure, operations such as virtual machine creation and operating system installation require the user to create the load cluster and then upload the installation package to the cloud platform for creation. This requires a large number of repetitive tasks such as uploading and installing. Different users may use different process operations. The creation processes of different load clusters are not repeatable, and consistency and repeatability cannot be maintained. Moreover, these infrastructures are not and cannot be included in the unified management of the cloud platform, and the consistency of multi-architecture CPUs cannot be guaranteed, which brings a burden to cluster lifecycle management.

In order to realize delivery scenarios such as multi-architecture infrastructure, rapid deployment and delivery, high availability, and container cluster disaster recovery and backup support, and quickly build and realize unified management of multi-architecture clusters based on cloud platforms, the container cluster management method provided in the embodiment of the present application is designed by designing computing resource management modules, partition management modules, and other modules of the cloud platform. Through a series of preliminary preparations, the user only needs to send a kubernetes load cluster creation request based on the client to provide the infrastructure resource pool type of the first target load cluster to be created and the infrastructure type required by the first target load cluster. Based on the cluster application interface, a multi-architecture kubernetes container cluster can be quickly created by using the management cluster declaration method, which simplifies the repetitive tasks of the kubernetes life cycle, and at the same time maintains consistency and repeatability in the infrastructure of multiple CPU architectures, reducing the actual burden brought by the cluster life cycle management work. It makes it possible to support proprietary cloud delivery scenarios such as multi-CPU architecture infrastructure, unified management of multiple clusters, rapid deployment and delivery, high availability, and container cluster disaster recovery and backup support.

In an optional implementation, for S101, firstly, a cluster application program interface is deployed on an existing kubernetes cluster based on a cloud platform, that is, a kubernetes management cluster (referred to as management cluster) is obtained. The cluster declared and created by the management cluster is called a kubernetes load cluster (referred to as load cluster), that is, a cluster that actually runs the load. The management cluster is usually only used for the management of the load cluster and does not run other applications.

The computing resource management module is configured based on the cloud platform. This module is a unified infrastructure management module. For different types of cloud platforms, the infrastructure may cover the business process management development platform AWS, the cloud platform-based operating system Azure, the application container engine Docker, the Google Cloud Platform GCP, the Apple Cloud Platform Metal3, VMware vSphere Integrated Container (VIC), the cloud computing management platform OpenStack and even bare metal. The infrastructure resource pools corresponding to different types of cloud platforms are pre-configured and all are managed uniformly using the computing resource management module, so that the cloud platform can deploy and manage kubernetes clusters across various infrastructures in a consistent manner. Taking OpenStack as an example, if the cloud platform configures OpenStack as an infrastructure resource pool, the required components are pre-configured for the OpenStack infrastructure resource pool, such as the computing service component nova, the network as a service component neutron, the security authentication service component keysnote (responsible for user authentication and service directory), the high-availability distributed object storage service component swift, the persistent block storage function component cinder, and the image service component glance. Among them, the image service component glance is used to pull the virtual machine image required by the first target load cluster from the storage of the OpenStack infrastructure resource pool (that is, the image warehouse stores virtual machine images required by various CPU architectures, operating system types and versions, kubernetes versions or container runtimes such as amd64-mirror-template-1, amd64-mirror-template-2, arm64-mirror-template-1, arm64-mirror-template-2, etc.) and create a virtual machine in the OpenStack infrastructure resource pool.

Furthermore, different versions of virtual machine images are made and pushed to the image warehouse of each infrastructure resource pool. In order to meet the needs of various CPU architectures, operating system types and versions, kubernetes versions or container runtimes, virtual machine images of different CPU architectures (x86, ARM, etc.), different operating system types and different operating system versions, different kubernetes versions or container runtimes (Docker, containerd, etc.) are made in advance and pushed to the image warehouse.

Multiple versions of virtual machine images are pushed to the image repositories of each infrastructure resource pool, which can include:

In the process of making a virtual machine image, place the binary resource configuration file related to Kubernetes of a specific version corresponding to the CPU architecture in the image to the specified directory, install the second container image repository (container image repository harbor, registry, etc.), and push the Kubernetes-related images corresponding to the CPU architecture and the corresponding version to the second container image repository;

Push the completed virtual machine image to the image repository of each infrastructure resource pool.

That is to say, the virtual machine image with pre-installed operating system and kubernetes orchestration tool is pushed to the image repository of each infrastructure resource pool, so that the infrastructure deployment can be completed at the same time when the first target load cluster is deployed, and the cloud platform can realize unified management of operating system, virtual machine, etc.

Push the container images related to the ancillary components required to create a kubernetes load cluster (such as Prometheus, Jenkins, Ingress-controller, localpath-provisioner, Istio, CNI plug-in, Cluster-Agent, etc.) to the first container image repository of the cloud platform, and push the container cluster application deployment (helm-chart) template from the helm client to the chart (chart) repository of the cloud platform.

A partition management module is set up based on the cloud platform to maintain virtual machines of different versions. The partitions of the partition management module can be set to correspond one to one with the virtual machine images.

Correspondingly, the partition information of the partition may include the CPU architecture, operating system type, operating system version, kubernetes version, container runtime type, container runtime version, and virtual machine image ID corresponding to the virtual machine in the first target load cluster.

That is, the partition management module is set to maintain the operating system type, operating system version, kubernetes/container runtime version, virtual machine image ID and other information of different CPU architectures, that is, each partition corresponds to a CPU structure, and each CPU architecture corresponds to a variety of different operating system types, operating system versions, kubernetes versions, container runtime types, container runtime versions, virtual machine image ID and other information. Set up multiple partitions, that is, maintain multiple partition information records, each partition corresponds to a virtual machine image with a specified CPU architecture, kubernetes/version, operating system type/version, container runtime type/version and other information.

Based on the cloud platform, the computing resource management module and the partition management module can be managed based on the pre-deployed cluster management module. In addition, the cluster management module is also configured to manage the custom resources (CDR resources, such as cluster.x-k8s) of the load cluster through the resource supply interface that interfaces with the infrastructure resource pool through the cluster application program interface.

After completing the above preparations, for S102, when receiving a kubernetes load cluster creation request sent by a user based on a client, identify the infrastructure resource pool type (such as using an OpenStack infrastructure resource pool) of the first target load cluster to be created carried by the kubernetes load cluster creation request and the infrastructure type (kubernetes version, CPU architecture, partition, etc.) required by the first target load cluster. To ensure the executability of the kubernetes load cluster creation request, in S103: before calling the computing resource management module and the partition management module to complete the infrastructure resource pool deployment and infrastructure deployment for the first target load cluster based on the cluster application program interface using the management cluster declaration method, the container cluster management method provided in the embodiment of the present application also includes:

Perform permission verification and request parameter verification on kubernetes load cluster creation requests;

If the verification is passed, the process proceeds to S103: based on the cluster application program interface, the computing resource management module and the partition management module are called to complete the infrastructure resource pool deployment and infrastructure deployment steps for the first target load cluster by managing the cluster declaration.

The cloud platform is set as a module for registering (registry) management of load clusters, which can provide application interface services (api server), storage services (etcd), proxy services (proxy), wireless communication management services (coredns), cluster record services (scheduler), etc. The client sends a request for creating a new load cluster to the cluster management module with the selected infrastructure resource pool type and infrastructure type. The cluster management module performs permission verification and request parameter verification. After the verification is completed, the cluster record service can create a corresponding management record for the first target load cluster in the database of the above-mentioned cloud platform, and synchronize the cluster information of the first target load cluster to the management record. In the process of life cycle management of the first target load cluster, the management record can be used to maintain the creation progress information of the first target load cluster, the maintenance change information in operation, and even the deletion information, so as to form maintenance management of the information of the multi-architecture cluster.

For S103, the cluster management module calls the computing resource management module and the partition management module respectively, finds the first infrastructure resource pool corresponding to the infrastructure resource pool type, pulls the virtual machine image that meets the infrastructure type requirements from the second container image repository of the first infrastructure resource pool, creates the first target load cluster, creates the virtual machine, and creates the corresponding operating system in the first infrastructure resource pool, and then Pull the container image of the auxiliary components that meet the infrastructure type requirements from the first container image repository of the cloud platform, pull the container cluster application deployment template that meets the infrastructure type requirements from the chart repository of the cloud platform, and complete the deployment of the first target load cluster in the first infrastructure resource pool.

After completing the above creation process of the first target load cluster, the deployment completion status of the first target load cluster is recorded in the management record corresponding to the first target load cluster in the database.

Since the infrastructure resource pool deployment and infrastructure deployment of the first target load cluster are both created by using the cluster application program interface in a manner of managing cluster declarations, the infrastructure of the kubernetes load cluster is managed.

FIG. 2 is a flow chart of an optional implementation of S103 provided in an embodiment of the present application.

Based on the above embodiments, the embodiments of the present application further describe in detail the steps of creating a Kubernetes load balancing cluster.

As shown in FIG. 2 , based on the above embodiment, in the container cluster management method provided in the embodiment of the present application, S103: based on the cluster application program interface, using the management cluster declaration method, calling the computing resource management module and the partition management module to create a first target load cluster in the first infrastructure resource pool corresponding to the infrastructure resource pool type and completing infrastructure deployment for the first target load cluster, may include:

S201: After calling the computing resource management module to obtain the image service component interface of the first infrastructure resource pool, calling the partition management module to obtain the virtual machine image ID required by the first target load cluster in the image repository of the first infrastructure resource pool.

In an optional implementation, after completing the verification of the kubernetes load cluster creation request, creating the creation progress of the first target load cluster and synchronously updating it to the database, the cluster management module is run to call the computing resource management module to obtain the authentication information of the first infrastructure resource pool, and call the image service component interface of the first infrastructure resource pool according to the request parameters of the kubernetes load cluster creation request, and call the partition management module to query the ID of the virtual machine image that meets the CPU architecture, kubernetes version and other conditions required by the kubernetes load cluster creation request. For example, when the infrastructure resource pool type is an OpenStack infrastructure resource pool, the cluster management module obtains the authentication information of the OpenStack infrastructure resource pool and calls the interface of the image service component glance of the OpenStack infrastructure resource pool according to the request parameters, so as to query the virtual machine image that meets the infrastructure type requirements through the interface and obtain the ID of the virtual machine image.

It should be noted that the lifecycle management of the container cluster includes not only cluster creation, but also cluster deletion. In order to facilitate the cleanup of cloud hard disk resources when deleting a load cluster, when performing S103: based on the cluster application program interface, the computing resource management module and the partition management module are called to create a first target load cluster in the first infrastructure resource pool corresponding to the infrastructure resource pool type and complete the infrastructure deployment for the first target load cluster, or before performing S202, the container cluster management method provided in the embodiment of the present application also includes: creating a one-to-one correspondence between the first target load cluster and the user, and configuring security authentication rules for the first target load cluster.

In some embodiments, the cluster management module can call the security authentication service interface of the first infrastructure resource pool to create a user and configure corresponding security authentication rules for the user. For example, when the infrastructure resource pool type is an OpenStack infrastructure resource pool, the cluster management module calls the interface of the security authentication service component keysnote of the OpenStack infrastructure resource pool to create a user.

Furthermore, the security authentication rule may be to use a key pair for authentication. When the first infrastructure resource pool is an OpenStack infrastructure resource pool, configuring a corresponding security authentication rule for the user may be to use nova, a computing service component of the OpenStack infrastructure resource pool, to create a key pair.

It is understandable that, according to actual needs, other types of security authentication rules may be configured for users of the first target load cluster to ensure the security of user information and data of the first target load cluster.

S202: Generate and create an extended resource file according to the input parameters, and create an extended resource for the first target load cluster for managing custom resources in the Kubernetes cluster.

In an optional implementation, the cluster management module is run to generate and create a cluster.x-k8s extended resource file in YAML format based on the input parameters and create an extended resource. The extended resource is a CRD resource used to manage and control the kubernetes cluster, that is, a custom resource used to define a load cluster.

S203: When a custom resource is monitored, based on the cluster application program interface and the resource supply interface corresponding to the first infrastructure resource pool type, the Kubernetes orchestration tool is called to create and deploy a Kubernetes load cluster in the first infrastructure resource pool.

In an optional implementation, when the cloud platform's cluster application program interface listens to new custom resources (i.e., cluster.x-k8s resources), it The platform uses the resource supply interface corresponding to the first infrastructure resource pool type (i.e., the supply interface used by the cluster application interface to connect to the infrastructure resource pool, which can be implemented by a set of functions to manage the infrastructure resource pool according to cluster parameters) to call the infrastructure resource pool interface to create virtual machines, security groups, and cloud hard disk resources, and inject the cloud platform initialization (cloud-init) script into the virtual machine. For example, when the first infrastructure resource pool is an OpenStack infrastructure resource pool, the cluster application interface calls the OpenStack interface through Provider-OpenStack (the Provider used by the cluster application interface to connect to the OpenStack infrastructure resource pool, which implements a set of functions to manage OpenStack infrastructure resources according to cluster parameters) to perform the operation of deploying infrastructure.

The cloud platform initialization script is used to implement functions such as Kubernetes initialization, creation, and modification of Kubernetes configuration parameters. The cloud platform initialization script is run to call the Kubernetes orchestration tool (such as kubeadm) preset in the virtual machine image to deploy the first target load cluster.

S204: After monitoring that the application interface services of all the custom resources of the first target load cluster are deployed, obtain and record the identifier of the first target load cluster.

In some embodiments, the cluster management module is run to monitor the status of cluster.x-k8s resources, and the node list is queried after the application interface service is ready (api server ready). After the application interface service (api server) is deployed, the cluster management module is run to call the cluster application program interface to obtain the identifier (kubeconfig) of the first target load cluster, and the identifier is recorded in the management record corresponding to the first target load cluster in the database.

S205: Modify the label setting role of the working node based on the client, call the container image required by the first target load cluster from the container image repository, obtain the container cluster application deployment template required by the first target load cluster from the chart repository, and complete the installation and startup of the auxiliary components in the first target load cluster.

In some embodiments, the cluster management module is run to modify the label (label) of the worker node through the client (k8s-client) to set the role (such as node-role.kubernetes.io/node), call the container image of the auxiliary component required by the first target load cluster from the first container image warehouse, obtain the container cluster application deployment template (helm-chart) required by the first target load cluster from the chart warehouse, and use the helm in the container cluster application deployment template to install auxiliary components (such as prometheus, Ingress-controller, localpath-provisioner, Istio, CNI plug-in, Cluster-Agent, etc.) to the first target load cluster. After the auxiliary components are installed, the deployment completion status is recorded in the management record corresponding to the first target load cluster in the database.

Since the lifecycle management task of a container cluster includes not only the creation of a load cluster but also the deletion of a load cluster, based on the above embodiment, the container cluster management method provided in the embodiment of the present application also includes:

When a deletion command for the created second target load cluster is received based on the client, the computing resource management module and the resource supply interface of the second infrastructure resource pool where the second target load cluster is located are called to delete the infrastructure of the second target load cluster in the second infrastructure resource pool.

In an optional implementation, the user sends a deletion command for the created second target load cluster to the cluster management module of the cloud platform based on the client, and the cluster management module updates the status of the second target load cluster to being deleted.

Run the cluster management module to call the computing resource management module, and delete the infrastructure of the second target load cluster (such as cluster.x-k8s CRD resources) in the second infrastructure resource pool from the resource supply interface of the second infrastructure resource pool where the second target load cluster is located, and then complete the deletion operation of the second target load cluster.

After completing the above-mentioned deletion operation on the second target load cluster, the management record corresponding to the second target load cluster is deleted from the cloud platform database.

In the above embodiment of the present application, it is mentioned that the creation process of the load cluster includes the occupation of cloud hard disk resources. In order to facilitate the cleanup of cloud hard disk resources when deleting the load cluster, the load cluster is created as a one-to-one correspondence with the user when the load cluster is created. On this basis, for the deletion process of the load cluster, on the basis of the above embodiment, the management method of the container cluster provided in the embodiment of the present application also includes:

When receiving a deletion command for the created second target load cluster sent by the client, calling the computing resource management module and the resource supply interface of the second infrastructure resource pool where the second target load cluster is located, and deleting the infrastructure of the second target load cluster in the second infrastructure resource pool;

Call the computing resource management module and the resource supply interface of the second infrastructure resource pool, obtain the authentication information of the second target load cluster, delete the volume of the corresponding user under the second target load cluster, delete the corresponding security authentication rules, and delete the user information and the record of the second target load cluster in the cloud platform database.

On the premise of setting a one-to-one correspondence between load clusters and users when creating a load cluster, after completing the deletion of the infrastructure of the second target load cluster, the user information corresponding to the second load cluster is further deleted, thereby completely clearing the resources occupied by the second target load cluster in the second infrastructure resource pool, and then deleting the status information and user information of the second target load cluster in the cloud platform database.

In some embodiments, after the infrastructure of the second target load cluster is deleted, the cluster management module is run to call the computing resource management module to obtain the authentication information of the second infrastructure resource pool, and then the component service interface of the second infrastructure resource pool is called to delete the volume under the user corresponding to the second target load cluster, and delete the security authentication rules corresponding to the user. For example, when the second infrastructure resource pool is an OpenStack infrastructure resource pool, the persistent block storage function component cinder is called to delete the volume under the user corresponding to the second target load cluster, and the computing service component nova is called to delete the key pair corresponding to the user, and finally the security authentication service component keysnote is called to delete the user information. After completing the above-mentioned deletion operation on the second target load cluster, the management record corresponding to the second target load cluster is deleted from the cloud platform database.

With respect to the two methods for deleting a load cluster provided in the above embodiments of the present application, calling the computing resource management module and the resource supply interface of the second infrastructure resource pool where the second target load cluster is located, and deleting the infrastructure of the second target load cluster in the second infrastructure resource pool may include:

Invoke the computing resource management module to execute a deletion operation on the custom resources of the second target load cluster in the second infrastructure resource pool;

After the deletion operation is detected based on the cluster application program interface, the infrastructure of the second target load cluster is deleted in the second infrastructure resource pool through the resource supply interface of the second infrastructure resource pool.

That is to say, when deleting the infrastructure of the second target load cluster, delete the custom resources (cluster.x-k8s CRD resources) first, and then delete the corresponding infrastructure.

For example, when the second infrastructure resource pool is an OpenStack infrastructure resource pool, the cluster management module is run to call the computing resource management module to execute the operation of deleting custom resources (cluster.x-k8s CRD resources).

After the cluster application program interface of the cloud platform monitors the above deletion operation, it calls OpenStack through Provider-OpenStack to delete the infrastructure of the second target load cluster and waits for the deletion to be completed.

The above describes in detail various embodiments corresponding to the container cluster management method. On this basis, the present application also discloses a container cluster management device, equipment and non-volatile readable storage medium corresponding to the above method.

FIG3 is a schematic diagram of the structure of a container cluster management device provided in an embodiment of the present application.

As shown in FIG3 , the container cluster management device provided in the embodiment of the present application includes:

The environment preparation unit 301 is configured to pre-deploy a cluster application program interface in a kubernetes cluster based on a cloud platform to create a management cluster; configure a computing resource management module for unified management of infrastructure resource pools of different types of cloud platforms based on the cloud platform, and a partition management module configured to maintain different versions of virtual machines; push multiple versions of virtual machine images to the image warehouse of each infrastructure resource pool; push the container image of the auxiliary components required by the kubernetes load cluster to the first container image warehouse of the cloud platform, and push the container cluster application deployment template to the chart warehouse of the cloud platform;

The identification unit 302 is configured to, upon receiving a kubernetes load cluster creation request sent by a client, identify an infrastructure resource pool type of a first target load cluster to be created and an infrastructure type required by the first target load cluster;

The creation unit 303 is configured to use a management cluster declaration method based on the cluster application program interface to call the computing resource management module and the partition management module to create a first target load cluster in the first infrastructure resource pool corresponding to the infrastructure resource pool type and complete infrastructure deployment for the first target load cluster.

Furthermore, the environment preparation unit 301 pushes multiple versions of virtual machine images to the image warehouse of each infrastructure resource pool, which may include:

When making a virtual machine image, place the binary resource configuration file related to the specific version of Kubernetes corresponding to the CPU architecture in the image to the specified directory, install the second container image repository, and push the corresponding CPU architecture and corresponding version of Kubernetes related images to the second Container image repository;

Push the completed virtual machine image to the image repository of each infrastructure resource pool.

Furthermore, the partitions of the partition management module correspond one-to-one with the virtual machine images;

Correspondingly, the partition information of the partition may include the CPU architecture, operating system type, operating system version, kubernetes version, container runtime type, container runtime version, and virtual machine image ID corresponding to the virtual machine in the first target load cluster.

Furthermore, the creation unit 303 may include:

The infrastructure pulling subunit is configured to call the computing resource management module to obtain the image service component interface of the first infrastructure resource pool, and then call the partition management module to obtain the virtual machine image ID required by the first target load cluster from the image repository of the first infrastructure resource pool;

The extended resource creation subunit is configured to generate and create an extended resource file according to input parameters, and to create an extended resource for the first target load cluster for managing custom resources in the kubernetes cluster;

The cluster orchestration subunit is configured to, when monitoring a custom resource, call a kubernetes orchestration tool to create and deploy a kubernetes load cluster in the first infrastructure resource pool based on a cluster application program interface and a resource provisioning interface corresponding to the first infrastructure resource pool type;

The cluster registration subunit is configured to obtain and record the identifier of the first target load cluster after monitoring that the application interface services of all custom resources of the first target load cluster are deployed;

The cluster deployment subunit is configured to modify the label setting role of the working node based on the client, call the container image required by the first target load cluster from the container image repository, obtain the container cluster application deployment template required by the first target load cluster from the chart repository, and complete the installation and startup of the auxiliary components in the first target load cluster.

Furthermore, the container cluster management device provided in the embodiment of the present application also includes:

The deletion unit is configured to call the computing resource management module and the resource supply interface of the second infrastructure resource pool where the second target load cluster is located when receiving a deletion command for the created second target load cluster sent by the client, and delete the infrastructure of the second target load cluster in the second infrastructure resource pool.

Furthermore, the container cluster management device provided in the embodiment of the present application also includes:

The account management unit is configured to create a one-to-one correspondence between the first target load cluster and the user, and configure security authentication rules for the first target load cluster before calling the computing resource management module and the partition management module to create a first target load cluster in the first infrastructure resource pool corresponding to the infrastructure resource pool type and completing infrastructure deployment for the first target load cluster in a management cluster declaration manner based on the cluster application program interface.

Furthermore, a one-to-one correspondence between the first target load cluster and the user is created, and a security authentication rule is configured for the first target load cluster, which may be:

The security authentication service interface of the first infrastructure resource pool is called to create a user, and corresponding security authentication rules are configured for the user.

Furthermore, when the first infrastructure resource pool is an OpenStack infrastructure resource pool, the corresponding security authentication rules are configured for the user, which may be:

Use nova, a computing component of the OpenStack infrastructure resource pool, to create a key pair.

Furthermore, the container cluster management device provided in the embodiment of the present application also includes:

The infrastructure deletion unit is configured to, when receiving a deletion command for the created second target load cluster sent based on the client, call the computing resource management module and the resource supply interface of the second infrastructure resource pool where the second target load cluster is located, and delete the infrastructure of the second target load cluster in the second infrastructure resource pool;

The account information deletion unit is configured to call the computing resource management module and the resource supply interface of the second infrastructure resource pool, obtain the authentication information of the second target load cluster, delete the volume of the corresponding user under the second target load cluster, delete the corresponding security authentication rules, and delete the user information and the record of the second target load cluster in the cloud platform database.

Further, the deletion unit or the infrastructure deletion unit calls the computing resource management module and the second infrastructure where the second target load cluster is located. The resource supply interface of the infrastructure resource pool is used to delete the infrastructure of the second target load cluster in the second infrastructure resource pool, including:

Invoke the computing resource management module to execute a deletion operation on the custom resources of the second target load cluster in the second infrastructure resource pool;

After the deletion operation is detected based on the cluster application program interface, the infrastructure of the second target load cluster is deleted in the second infrastructure resource pool through the resource supply interface of the second infrastructure resource pool.

Furthermore, the container cluster management device provided in the embodiment of the present application also includes:

The verification unit is configured to call the computing resource management module and the partition management module to create a first target load cluster in the first infrastructure resource pool corresponding to the infrastructure resource pool type and complete the infrastructure deployment for the first target load cluster in a management cluster declaration manner based on the cluster application program interface, and perform permission verification and request parameter verification on the kubernetes load cluster creation request; if the verification is passed, enter the creation unit 303.

Furthermore, the container cluster management device provided in the embodiment of the present application also includes:

The recording unit is configured to create a corresponding management record for the first target load cluster in the database of the cloud platform, and synchronously update the cluster information of the first target load cluster to the management record.

Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

FIG4 is a schematic diagram of the structure of a container cluster management device provided in an embodiment of the present application.

As shown in FIG4 , the management device of the container cluster provided in the embodiment of the present application includes:

A memory 410 configured to store a computer program 411;

The processor 420 is configured to execute a computer program 411. When the computer program 411 is executed by the processor 420, the steps of the container cluster management method according to any one of the above embodiments are implemented.

The processor 420 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 420 may be implemented in at least one of the following hardware forms: a digital signal processor DSP (Digital Signal Processing), a field-programmable gate array FPGA (Field-Programmable Gate Array), and a programmable logic array PLA (Programmable Logic Array). The processor 420 may also include a main processor and a coprocessor. The main processor is a processor configured to process data in an awake state, also known as a central processing unit CPU (Central Processing Unit); the coprocessor is a low-power processor configured to process data in a standby state. In some embodiments, the processor 420 may be integrated with a graphics processor GPU (Graphics Processing Unit), and the GPU is configured to be responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 420 may also include an artificial intelligence AI (Artificial Intelligence) processor, which is configured to process computing operations related to machine learning.

The memory 410 may include one or more non-volatile readable storage media, which may be non-transitory. The memory 410 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 410 is at least configured to store the following computer program 411, wherein the computer program 411, after being loaded and executed by the processor 420, can implement the relevant steps in the management method of the container cluster disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 410 may also include an operating system 412 and data 413, etc., and the storage method may be short-term storage or permanent storage. Among them, the operating system 412 may be Windows. Data 413 may include, but is not limited to, the data involved in the above method.

In some embodiments, the management device of the container cluster may further include a display screen 430 , a power supply 440 , a communication interface 450 , an input/output interface 460 , a sensor 470 , and a communication bus 480 .

Those skilled in the art will appreciate that the structure shown in FIG. 4 does not constitute a limitation on the management device of the container cluster, and may include more or fewer components than shown in the figure.

The container cluster management device provided in an embodiment of the present application includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the container cluster management method as described above, and the effect is the same as above.

It should be noted that the above-described device and equipment embodiments are merely illustrative. For example, the division of modules is merely a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or a Some features can be ignored or not performed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a non-volatile readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a non-volatile readable storage medium to execute all or part of the steps of the above methods of each embodiment of the present application.

To this end, an embodiment of the present application further provides a non-volatile readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the container cluster management method are implemented.

The non-volatile readable storage medium may include: a U disk, a mobile hard disk, a read-only memory ROM (Read-Only Memory), a random access memory RAM (Random Access Memory), a magnetic disk or an optical disk, and other media that can store program codes.

The computer program included in the non-volatile readable storage medium provided in this embodiment can implement the steps of the above-mentioned container cluster management method when executed by the processor, and the effect is the same as above.

The above is a detailed introduction to a container cluster management method, device, equipment and non-volatile readable storage medium provided by the present application. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the devices, equipment and non-volatile readable storage medium disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the above elements.

Claims (20)

1. A method for managing a container cluster, comprising:

   Pre-deploy a cluster application program interface in the kubernetes cluster based on the cloud platform to create a management cluster; configure a computing resource management module for unified management of different types of infrastructure resource pools of the cloud platform based on the cloud platform, and a partition management module for maintaining different versions of virtual machines; push multiple versions of virtual machine images to the image warehouse of each infrastructure resource pool; push the container image of the auxiliary components required by the kubernetes load cluster to the first container image warehouse of the cloud platform, and push the container cluster application deployment template to the chart warehouse of the cloud platform;

   When receiving a kubernetes load cluster creation request sent by a client, identifying an infrastructure resource pool type of a first target load cluster to be created and an infrastructure type required by the first target load cluster;

   Based on the cluster application program interface, the computing resource management module and the partition management module are called to create the first target load cluster in the first infrastructure resource pool corresponding to the infrastructure resource pool type and complete infrastructure deployment for the first target load cluster.
2. The method for managing a container cluster according to claim 1, wherein pushing multiple versions of virtual machine images to the image repository of each of the infrastructure resource pools specifically includes:

   In the process of making the virtual machine image, a binary resource configuration file related to Kubernetes of a specific version corresponding to the CPU architecture is placed in the image to a specified directory, a second container image repository is installed, and the Kubernetes-related image corresponding to the CPU architecture and the corresponding version is pushed to the second container image repository;

   The produced virtual machine image is pushed to the image warehouse of each infrastructure resource pool.
3. The container cluster management method according to claim 2, characterized in that the partitions of the partition management module correspond one-to-one to the virtual machine images;

   Correspondingly, the partition information of the partition specifically includes the CPU architecture, operating system type, operating system version, kubernetes version, container runtime type, container runtime version, and virtual machine image ID corresponding to the virtual machine in the first target load cluster.
4. The container cluster management method according to claim 1 is characterized in that the method of managing cluster declaration based on the cluster application program interface calls the computing resource management module and the partition management module to create the first target load cluster in the first infrastructure resource pool corresponding to the infrastructure resource pool type and completes infrastructure deployment for the first target load cluster, specifically comprising:

   After calling the computing resource management module to obtain the image service component interface of the first infrastructure resource pool, calling the partition management module to obtain the virtual machine image ID required by the first target load cluster from the image repository of the first infrastructure resource pool;

   Generate and create an extended resource file according to the input parameters, and create an extended resource for the first target load cluster for managing custom resources in the kubernetes cluster;

   When the custom resource is monitored, based on the cluster application program interface and the resource supply interface corresponding to the first infrastructure resource pool type, calling the kubernetes orchestration tool to create and deploy a kubernetes load cluster in the first infrastructure resource pool;

   When it is monitored that the application interface services of all the custom resources of the first target load cluster are deployed, obtaining and recording the identifier of the first target load cluster;

   Based on the client modifying the label setting role of the working node, the container image required by the first target load cluster is called from the container image repository, the container cluster application deployment template required by the first target load cluster is obtained from the chart repository, and the installation and startup of the auxiliary components are completed in the first target load cluster.
5. The container cluster management method according to claim 4, characterized in that the calling of the partition management module to obtain the virtual machine image ID required by the first target load cluster from the image repository of the first infrastructure resource pool includes:

   The partition management module is called to query the ID of the virtual machine image that meets the CPU architecture and kubernetes version conditions required by the kubernetes load cluster creation request as the virtual machine image ID.
6. The method for managing a container cluster according to claim 4, wherein the step of generating and creating an extended resource file according to the input parameters And create extended resources for the first target load cluster to manage custom resources in the Kubernetes cluster, including:

   Generate and create an extended resource file based on the input parameters and create an extended resource, where the extended resource is a CRD resource used to manage and control the kubernetes cluster.
7. The method for managing a container cluster according to claim 4, characterized in that when the custom resource is monitored, based on the cluster application program interface and the resource supply interface corresponding to the first infrastructure resource pool type, calling a kubernetes orchestration tool to create and deploy a kubernetes load cluster in the first infrastructure resource pool comprises:

   When the custom resource is monitored, the interface of the infrastructure resource pool is called through the resource supply interface corresponding to the cloud platform and the first infrastructure resource pool type to create virtual machine, security group and cloud hard disk resources, and the cloud platform initialization script is injected into the virtual machine, wherein the cloud platform initialization script is used to implement Kubernetes initialization, create and modify Kubernetes configuration parameters, and run the cloud platform initialization script to call the Kubernetes orchestration tool preset in the virtual machine image to deploy the first target load cluster.
8. The method for managing a container cluster according to claim 4, wherein after monitoring that the application interface services of all the custom resources of the first target load cluster are deployed, obtaining and recording an identifier of the first target load cluster comprises:

   Monitor the status of cluster.x-k8s resources, wait for the application interface service to be ready and then query the node list, after the application interface service is deployed, call the cluster application program interface to obtain the identifier of the first target load cluster, and record the identifier in the management record corresponding to the first target load cluster.
9. The method for managing a container cluster according to claim 4, wherein completing the installation and startup of the auxiliary component in the first target load cluster comprises:

   Using a software package management tool in the container cluster application deployment template to install ancillary components into the first target load cluster;

   After the installation of the subsidiary component is completed, the subsidiary component is started, and the deployment completion status is recorded in the management record corresponding to the first target load cluster.
10. The method for managing a container cluster according to claim 1, further comprising:

    When a deletion command for the created second target load cluster sent by the client is received, the computing resource management module and the resource supply interface of the second infrastructure resource pool where the second target load cluster is located are called to delete the infrastructure of the second target load cluster in the second infrastructure resource pool.
11. The method for managing a container cluster according to claim 1, characterized in that before the method of managing cluster declarations based on the cluster application program interface, calling the computing resource management module and the partition management module to create the first target load cluster for the first infrastructure resource pool corresponding to the infrastructure resource pool type and completing infrastructure deployment for the first target load cluster, further comprises:

    A one-to-one correspondence between the first target load cluster and the user is created, and a security authentication rule is configured for the first target load cluster.
12. The container cluster management method according to claim 11 is characterized in that the step of creating a one-to-one correspondence between the first target load cluster and the user and configuring a security authentication rule for the first target load cluster is as follows:

    Call the security authentication service interface of the first infrastructure resource pool to create a user, and configure corresponding security authentication rules for the user.
13. The method for managing a container cluster according to claim 12, wherein when the first infrastructure resource pool is an OpenStack infrastructure resource pool, the security authentication rules corresponding to the configuration for the user are specifically:

    A key pair is created using nova, a computing service component of the OpenStack infrastructure resource pool.
14. The method for managing a container cluster according to claim 11, further comprising:

    When receiving a deletion command for the created second target load cluster sent by the client, calling the computing resource management module and the resource provision interface of the second infrastructure resource pool where the second target load cluster is located, and deleting the infrastructure of the second target load cluster in the second infrastructure resource pool;

    Call the computing resource management module and the resource supply interface of the second infrastructure resource pool, obtain the authentication information of the second target load cluster, delete the volume of the corresponding user under the second target load cluster, delete the corresponding security authentication rules, and delete the user information And the record of the second target load cluster in the cloud platform database.
15. The container cluster management method according to claim 10 or 14, characterized in that the calling of the computing resource management module and the resource supply interface of the second infrastructure resource pool where the second target load cluster is located, and deleting the infrastructure of the second target load cluster in the second infrastructure resource pool, specifically includes:

    Calling the computing resource management module to execute a deletion operation on the custom resources of the second target load cluster in the second infrastructure resource pool;

    After the deletion operation is detected based on the cluster application program interface, the infrastructure of the second target load cluster is deleted from the second infrastructure resource pool through the resource supply interface of the second infrastructure resource pool.
16. The method for managing a container cluster according to claim 1, characterized in that before the method of managing cluster declarations based on the cluster application program interface, calling the computing resource management module and the partition management module to create the first target load cluster for the first infrastructure resource pool corresponding to the infrastructure resource pool type and completing infrastructure deployment for the first target load cluster, further comprises:

    Performing permission verification and request parameter verification on the kubernetes load cluster creation request;

    If the verification is passed, the process proceeds to the step of managing cluster declarations based on the cluster application program interface, calling the computing resource management module and the partition management module to complete infrastructure resource pool deployment and infrastructure deployment for the first target load cluster.
17. The method for managing a container cluster according to claim 1, further comprising:

    A corresponding management record is created for the first target load cluster in the database of the cloud platform, and cluster information of the first target load cluster is synchronously updated to the management record.
18. A container cluster management device, characterized by comprising:

    The environment preparation unit is configured to pre-deploy a cluster application program interface in a kubernetes cluster based on a cloud platform to create a management cluster; configure a computing resource management module for unified management of infrastructure resource pools of different types of the cloud platform based on the cloud platform, and a partition management module for maintaining virtual machines of different versions; push multiple versions of virtual machine images to the image warehouse of each infrastructure resource pool; push the container image of the auxiliary components required by the kubernetes load cluster to the first container image warehouse of the cloud platform, and push the container cluster application deployment template to the chart warehouse of the cloud platform;

    an identification unit, configured to, upon receiving a kubernetes load cluster creation request sent by a client, identify an infrastructure resource pool type of a first target load cluster to be created and an infrastructure type required by the first target load cluster;

    The creation unit is configured to call the computing resource management module and the partition management module to create the first target load cluster in the first infrastructure resource pool corresponding to the infrastructure resource pool type and complete infrastructure deployment for the first target load cluster in a management cluster declaration manner based on the cluster application program interface.
19. A management device for a container cluster, comprising:

    a memory arranged to store a computer program;

    A processor is configured to execute the computer program, and when the computer program is executed by the processor, the steps of the method for managing a container cluster as claimed in any one of claims 1 to 17 are implemented.
20. A non-volatile readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the container cluster management method as described in any one of claims 1 to 17 are implemented.