CN113918270A

CN113918270A - Cloud resource scheduling method and system based on Kubernetes

Info

Publication number: CN113918270A
Application number: CN202010653672.8A
Authority: CN
Inventors: 李嘉敏; 王鹏达; 陈乐翀; 刘照德; 郑叶
Original assignee: Diankeyun Beijing Technology Co ltd
Current assignee: Diankeyun Beijing Technology Co ltd
Priority date: 2020-07-08
Filing date: 2020-07-08
Publication date: 2022-01-11

Abstract

The invention provides a cloud resource scheduling method and system based on Kubernetes, wherein the method comprises the following steps: creating a Pod according to the resource object creation request so as to record the type of the required scheduler in the resource object creation request into a configuration file of the created Pod; after the created Pod is monitored, a scheduler is selected from a scheduler group of a required scheduler type which is deployed in advance, and the identifier of the selected scheduler is updated to a configuration file of the Pod; the scheduler group comprises a plurality of schedulers of the same type; after monitoring the identifier of the selected scheduler, determining cluster nodes distributed to the Pod and sending a binding request; after the admission controller receives the binding request, the admission controller verifies that the resource can still meet the requirement of the information of the required resource, and records the identification of the cluster node to Pod; and after the corresponding cluster node monitors that the identification of the cluster node is recorded by the Pod, starting the Pod and creating a corresponding container. By the scheme, more schedulers can schedule cloud resources in parallel, and resource scheduling efficiency is improved.

Description

Cloud resource scheduling method and system based on Kubernetes

Technical Field

The invention relates to the technical field of cloud resource scheduling, in particular to a Kubernetes-based cloud resource scheduling method and system.

Background

The container cloud platform is a cloud resource management platform implemented based on a container technology, and many tasks do not need to be communicated or synchronized in resource scheduling tasks needing to be executed, for example, tasks based on a MapReduce framework in a big data scene. If only a single scheduler is provided, the resource scheduling efficiency is low when the scale of the cloud resource cluster is large.

Native kubernets default to a single scheduler to handle the user's tasks, and although the added multi-scheduler function can make multiple schedulers work simultaneously, the number and types of the schedulers are fixed, and there is only one scheduler in each type.

Disclosure of Invention

In view of this, the invention provides a cloud resource scheduling method and system based on Kubernetes, so as to achieve parallel scheduling of cloud resources by more schedulers and further improve resource scheduling efficiency.

In order to achieve the purpose, the invention is realized by adopting the following scheme:

according to an aspect of the embodiments of the present invention, a cloud resource scheduling method based on Kubernetes is provided, including:

acquiring a resource object creation request, creating a Pod according to the resource object creation request, and recording a required scheduler type in the resource object creation request into a configuration file of the created Pod;

after the created Pod is monitored, a scheduler is selected from a scheduler group which is deployed in advance and belongs to the type of the required scheduler, and the identifier of the selected scheduler is updated to a configuration file of the created Pod; the scheduler group comprises a plurality of schedulers of the same type;

after monitoring that the self identification is recorded in the configuration file of the created Pod, the selected dispatcher determines the cluster node allocated to the created Pod according to the information of the required resource in the configuration file of the created Pod and sends a corresponding request for binding the Pod and the cluster node;

under the condition that the admission controller verifies that the determined resources of the cluster nodes distributed to the created Pod can still meet the requirement of the information of the required resources after receiving the request of the bound Pod and the cluster nodes, recording the determined identifications of the cluster nodes distributed to the created Pod;

and after the determined cluster node allocated to the created Pod monitors that the created Pod records the identifier of the determined cluster node allocated to the created Pod, starting the created Pod to create a corresponding container.

In some embodiments, acquiring a resource object creation request, and creating a Pod according to the resource object creation request, so as to record a required scheduler type in the resource object creation request into a configuration file of the created Pod, includes: acquiring a resource object creation request, creating a Pod according to the resource object creation request, and recording a required scheduler type and a required scheduling policy identifier in the resource object creation request into a configuration file of the created Pod. The method further comprises the following steps: a step of selecting a scheduler from a scheduler group which is deployed in advance and belongs to the type of the required scheduler after monitoring the created Pod and updating the identifier of the selected scheduler into the configuration file of the created Pod under the condition that the required scheduling policy identifier existing in the configuration file of the created Pod is determined as a set scheduling policy identifier; or, when determining that the required scheduling policy identifier in the created Pod configuration file is the original scheduling policy identifier or the corresponding field is empty, executing a default scheduling flow in kubernets.

In some embodiments, while creating a Pod according to the resource object creation request to record a required scheduler type in the resource object creation request into a configuration file of the created Pod, the method further includes: and storing the relevant information of the created Pod to the ETCD. The method further comprises the following steps: and judging whether the required scheduling policy identifier in each piece of relevant information of the created Pod stored in the ETCD is a set scheduling policy identifier or not through the API Server, so as to judge whether the required scheduling policy identifier in the configuration file of the created Pod is the set scheduling policy identifier or not. After the created Pod is monitored, a scheduler is selected from a scheduler group which is deployed in advance and belongs to the type of the required scheduler, and the identification of the selected scheduler is updated to a configuration file of the created Pod, wherein the method comprises the following steps: and monitoring each piece of relevant information of the created Pod stored in the ETCD through the API Server, confirming to monitor the created Pod, triggering to select a scheduler from a scheduler group which is deployed in advance and belongs to the type of the required scheduler, and recording the identifier of the selected scheduler into a configuration file of the created Pod. While updating the identity of the selected scheduler into the profile of the created Pod, the method further comprises: and storing the identification update of the selected scheduler into a corresponding field in each piece of relevant information of the Pod created in the ETCD. After monitoring that the self identification is recorded in the configuration file of the created Pod, the selected scheduler determines the cluster node allocated to the created Pod according to the information of the required resource in the configuration file of the created Pod, including: and monitoring the ETCD by the API Server, updating the identifier of the selected scheduler, and confirming that the identifier of the selected scheduler is recorded in the configuration file of the created Pod, and determining the cluster node allocated to the created Pod by the selected scheduler according to the information of the required resource in the configuration file of the created Pod. Recording the determined identification of the cluster node allocated to the created Pod, and simultaneously, the method further comprises: and recording the determined identification of the cluster node allocated to the created Pod into a corresponding field in each piece of relevant information of the Pod created in the ETCD. After the determined cluster node allocated to the created Pod listens that the created Pod records the identifier of the determined cluster node allocated to the created Pod, starting the created Pod to create a corresponding container, including: and monitoring the identification of the cluster node stored in the ETCD and allocated to the created Pod through the API Server, confirming that the created Pod records the identification of the cluster node allocated to the created Pod, and starting the created Pod by the determined cluster node allocated to the created Pod to create a corresponding container.

In some embodiments, the method further comprises: the admission controller verifies whether the determined resources allocated to the cluster nodes of the created Pod can still meet the requirement of the information of the required resources according to the determined relevant information allocated to the cluster nodes of the created Pod and stored in the ETCD; and/or, the method further comprises: and under the condition that the admission controller receives the request of the bound Pod and the cluster node and verifies that the determined resources distributed to the cluster node of the created Pod cannot meet the requirement of the information of the required resources, deleting the identifier of the selected scheduler in the relevant information of the Pod created in the ETCD, reselecting the scheduler from a scheduler group which is deployed in advance and belongs to the type of the required scheduler, and updating the identifier of the reselected scheduler into a corresponding field in the relevant information of the Pod created in the ETCD.

In some embodiments, selecting a scheduler from a group of schedulers that are pre-deployed and belong to the desired scheduler type comprises: selecting a scheduler from a group of schedulers that are pre-deployed and belong to the desired scheduler type by a pre-deployed scheduling controller according to a round-robin scheduling algorithm; wherein, the pre-deployed scheduling controller comprises the set scheduling policy identifier; and/or the admission controller verifies whether the determined resources allocated to the cluster nodes of the created Pod still can meet the requirement of the information of the required resources according to the determined related information allocated to the cluster nodes of the created Pod, which is stored in the ETCD, and the method comprises the following steps: reading all information stored in the ETCD to a cache, and locking the resources of the cluster nodes distributed by each scheduler in the cache; and the admission controller judges whether the determined cluster node allocated to the created Pod can still meet the requirement of the information of the required resource according to the determined resource locking information in the cache of the cluster node allocated to the created Pod.

In some embodiments, the method further comprises: acquiring kubel data of each cluster node, cluster node monitoring data and metrics data of each scheduler in a scheduler group, acquiring the number and load conditions of the cluster nodes according to the kubel data of each cluster node and the cluster node monitoring data, and acquiring the throughput condition of each scheduler in the scheduler group according to the metrics data of each scheduler in each scheduler group; and automatically adjusting the number of the schedulers in the corresponding scheduler group according to the number and the throughput condition of the cluster nodes and the throughput condition of each scheduler in the scheduler group.

In some embodiments, obtaining kubelet data of each cluster node, cluster node monitoring data, and metrics data of each scheduler in the scheduler group, obtaining the number of cluster nodes and the load condition according to the kubelet data of each cluster node and the cluster node monitoring data, and obtaining the throughput condition of each scheduler in the scheduler group according to the metrics data of each scheduler in each scheduler group, includes: acquiring the metrics data of corresponding schedulers by using services which are deployed between a PromatheusOperator and each scheduler in each scheduler group in advance through Promatheus to obtain the metrics data of each scheduler in each scheduler group; acquiring data of kubel from each cluster node through Prometous and acquiring cluster node monitoring data; and obtaining the number and the load condition of the cluster nodes through Prometous according to the kubel data and the cluster node monitoring data of each cluster node, and obtaining the throughput condition of each scheduler in each scheduler group according to the metrics data of each scheduler in each scheduler group. Automatically adjusting the number of schedulers in a corresponding scheduler group according to the number and throughput condition of cluster nodes and the throughput condition of each scheduler in the scheduler group, comprising: obtaining the number and throughput condition of cluster nodes and the throughput condition of each scheduler in a scheduler group through an API Server aggregation layer and providing the cluster nodes and the throughput condition of each scheduler in the scheduler group for HPA access; and under the condition that the total throughput of all schedulers in the scheduler group is judged to be lower than a first expected condition and the number of cluster nodes and the load condition are judged to be lower than a second expected condition through the HPA, the number of the schedulers in the corresponding scheduler group is increased, and under the condition that the total throughput of all the schedulers in the scheduler group is judged to be higher than the first expected condition and the number of the cluster nodes and the load condition are higher than the second expected condition, the number of the schedulers in the corresponding scheduler group is reduced.

According to another aspect of the embodiments of the present invention, there is provided a cloud resource scheduling system based on Kubernetes, including:

the resource object creating module is used for acquiring a resource object creating request and creating a Pod according to the resource object creating request so as to record the type of the required scheduler in the resource object creating request into a configuration file of the created Pod;

the scheduling controller module is used for selecting a scheduler from a scheduler group which is deployed in advance and belongs to the type of the required scheduler after monitoring the created Pod, and updating the identifier of the selected scheduler into a configuration file of the created Pod;

each scheduler group comprises a plurality of schedulers of the same type and is used for providing a selected scheduler, monitoring that the self identification is recorded in the configuration file of the created Pod, determining the cluster node allocated to the created Pod according to the information of the required resource in the configuration file of the created Pod, and sending a corresponding request for binding the Pod and the cluster node;

the API Server comprises an admission controller module, and is used for recording the determined identification of the cluster node allocated to the created Pod under the condition that the determined resource allocated to the cluster node of the created Pod can still meet the requirement of the information of the required resource after receiving the request of the bound Pod and the cluster node;

the cluster node module is used for providing the determined cluster nodes allocated to the created Pod, and starting the created Pod to create a corresponding container after monitoring that the created Pod records the determined identifiers of the cluster nodes allocated to the created Pod;

and the monitoring interface module is used for monitoring whether the created Pod exists, recording the identifier of the selected scheduler in a configuration file of the created Pod, and recording the determined identifier of the cluster node allocated to the created Pod.

In some embodiments, the system further comprises:

the core monitoring module is used for acquiring kubel data of each cluster node, cluster node monitoring data and metrics data of each scheduler in the scheduler group, acquiring the number and load condition of the cluster nodes according to the kubel data of each cluster node and the cluster node monitoring data, and acquiring the throughput condition of each scheduler in the scheduler group according to the metrics data of each scheduler in each scheduler group;

the API Server also comprises an aggregation layer module which is used for providing the number and the throughput condition of the cluster nodes and the throughput condition of each scheduler in the scheduler group to the HPA management module of the controller instance;

and the HPA management module of the controller instance is used for automatically adjusting the number of the schedulers in the corresponding scheduler group according to the number and the throughput condition of the cluster nodes and the throughput condition of each scheduler in the scheduler group.

According to a further aspect of embodiments of the present invention, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method according to any one of the embodiments described above.

According to the cloud resource scheduling method based on Kubernetes, the cloud resource scheduling system based on Kubernetes and the computer readable storage medium, different scheduler groups are set aiming at different scheduler types, one scheduler group can have a plurality of schedulers, and schedulers of the same type can be used for scheduling in parallel. Furthermore, the system can automatically scale according to the condition of the cloud resource cluster, so that the requirement of improving the throughput of the scheduler under an emergency condition can be met.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:

fig. 1 is a schematic flowchart of a cloud resource scheduling method based on Kubernetes according to an embodiment of the present invention;

fig. 2 is a schematic flow chart of a cloud resource scheduling method based on Kubernetes according to another embodiment of the present invention;

fig. 3 is a schematic structural diagram of a cloud resource scheduling system based on Kubernetes according to an embodiment of the present invention;

fig. 4 is a schematic structural diagram of a cloud resource scheduling system based on Kubernetes according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.

The native Kubernetes' multi-scheduler function, while allowing multiple schedulers to work simultaneously, is fixed in number and type and has only one scheduler of each type. In order to further improve the resource scheduling efficiency, the invention provides a cloud resource scheduling method based on Kubernetes, which can perform parallel resource scheduling based on a plurality of schedulers in a multi-scheduler group, and can solve the problem of scheduling resource conflict possibly existing among the schedulers, thereby further improving the resource scheduling efficiency.

Fig. 1 is a schematic flow chart of a cloud resource scheduling method based on Kubernetes according to an embodiment of the present invention. As shown in fig. 1, the cloud resource scheduling method based on kubernets according to the embodiments may include the following steps S110 to S150.

Specific embodiments of steps S110 to S150 will be described in detail below.

Step S110: acquiring a resource object creation request, and creating a Pod according to the resource object creation request so as to record the type of the required scheduler in the resource object creation request into a configuration file of the created Pod.

In the above step S110, creating Pod (or may be written as Pod) may be implemented by a Client (Client) of the kubernets cluster. The resource object creation request may contain various information required to create the resource object, such as the required scheduler type, and may also include, for example, information on the required resources, a required scheduling policy identification, and the like. Wherein the required scheduler type may be, for example, scheduled cluster node association, scheduled cluster node disassociation, and the like. The information of the required resource may include specification, type, etc. of the required resource; of course, the user may not specify the information of the required resource in the resource object creation request, in which case the resource may be created according to default settings in or set by kubernets. In addition, a scheduling policy may be marked with a policy identification; the user can select the required scheduling policy by giving the required scheduling policy identifier in the resource object creation request, for example, the scheduling policy realized based on the following steps S120 to S150 in the present invention can be marked by the set scheduling policy identifier, and the original scheduling process in kubernets can be marked by the default scheduling policy identifier or null; of course, if there is only one scheduling policy, the user may adopt the scheduling policy without specifying the given required scheduling policy identifier.

The created Pod (which may be referred to as a container set) may be used to contain one or more containers, and the number of pods, the number of containers in each Pod, may be selected by the user as desired. The configuration file of the created Pod may contain various fields, for example, a required scheduler type field, and may further include an information field of a required resource, a required scheduling policy identification field, and the like, and information in the resource object creation request may be transmitted as a field value to a corresponding field in the configuration file. If some fields in the configuration file of the created Pod do not have corresponding field values transmitted, the field values of the fields may be null.

Step S120: after the created Pod is monitored, a scheduler is selected from a scheduler group which is deployed in advance and belongs to the type of the required scheduler, and the identifier of the selected scheduler is updated to a configuration file of the created Pod; the scheduler group includes a plurality of schedulers of the same type.

In the step S120, the scheduling controller deployed in the present invention monitors whether a new Pod is created through the API server in kubernets (the Pod created here refers to the step created in the step S110, and the scheduler is not determined yet, and the cluster node is not bound yet). For example, after a new Pod is created, Pod-related information may be fixed in an ETCD (writable ETCD or ETCD) database (also called an ETCD component) in kubernets, and the new Pod may be determined to be created according to a data change condition of the ETCD database.

A plurality of scheduler groups, each of which may correspond to a scheduler type and may include a plurality of schedulers, may be deployed by a scheduling controller (which may be implemented based on an original, similar, or modified scheduling controller) deployed in kubernets according to the present invention. The number of schedulers included in different scheduler groups may be the same or different, without excluding that there is only one scheduler in some of the scheduler groups. In the case that the scheduling process according to steps S120 to S150 of the present invention needs to be started according to the set scheduling policy identifier, the scheduling controller also includes the set scheduling policy identifier, so as to manage the set of schedulers (e.g. selection of a scheduler) according to the present invention by using the scheduling controller. Compared with the original Kubernetes multi-scheduler, the multi-scheduler system has the advantages that each type of scheduler can be provided with a plurality of schedulers, and therefore the parallel scheduling requirements of the same type can be better met.

When a new Pod is found to be created, the identity of the appropriate selected scheduler may be transferred by the scheduling controller to the Pod's profile via an API Server (Server of the application program interface in Kubernetes) and entered into the scheduler-related field. In addition, the ETCD database can be used as a Kubernets master database and can store various information through the API Server, so that the selected identifier of the scheduler can be stored in the corresponding field of the relevant information of the corresponding Pod in the ETCD database.

In some embodiments, in step S120, selecting a scheduler from a scheduler group that is pre-deployed and belongs to the required scheduler type may specifically include: selecting a scheduler from a group of schedulers that are pre-deployed and belong to the desired scheduler type by a pre-deployed scheduling controller according to a round-robin scheduling algorithm; wherein the pre-deployed scheduling controller includes the set scheduling policy identifier. The schedulers are selected through a Round-robin strategy, so that the number of scheduling tasks of each scheduler can be ensured to be balanced as much as possible, and the scheduling efficiency is improved.

Step S130: after monitoring that the self identification is recorded in the configuration file of the created Pod, the selected scheduler determines the cluster node allocated to the created Pod according to the information of the required resource in the configuration file of the created Pod, and sends a corresponding request for binding the Pod and the cluster node.

In the above step S130, the selected scheduler may listen, through API Server in kubernets, whether the identity of the selected scheduler is recorded in the configuration file of the created Pod (i.e. the required scheduler is selected (corresponding field value is recorded in the field in the configuration file)). For example, after the scheduler is determined in the created Pod, the information of the selected scheduler may be fixed to the ETCD database in Kubernetes, and it may be determined that the created Pod has determined the scheduler and has not bound to the cluster node according to the data change condition of the ETCD database.

After finding that the created Pod is selected by the scheduler, the scheduler may select a suitable cluster node (node) for the Pod according to information (which may include a resource specification, a type, and the like) of a required resource, and after the selection is completed, may initiate a binding request to the API Server to verify whether a resource of the cluster node determined by the selected scheduler conflicts with a resource of the cluster node allocated by the parallel scheduler (i.e., whether the determined resource of the cluster node is already allocated by another scheduler at this time, because the other schedulers may be scheduling the resource in parallel with the selected scheduler, and if the processing scheduling speed of the other schedulers is fast, the other schedulers may have allocated the cluster node resource in advance).

Step S140: and under the condition that the admission controller verifies that the determined resources of the cluster nodes distributed to the created Pod can still meet the requirement of the information of the required resources after receiving the request of the bound Pod and the cluster nodes, recording the determined identifications of the cluster nodes distributed to the created Pod.

In step S140, the Admission Controller may be an original Admission Controller (Admission Controller) in the API Server in the kubernets cluster, and the code is written based on the Admission Controller to implement the verification logic. Since the API Server can easily take various required data (e.g. the API Server can read the required data from the etc data interfaced with the API Server), the validation of resource conflicts by the admission controller can reduce the workload of improving kubernets to implement the present invention.

After the scheduler prepares to bind the selected cluster node and the created Pod, the scheduler does not directly bind but sends a binding request to the admission controller for resource conflict verification, so that the problem that the scheduler prepares to allocate the resource of the cluster node of the created Pod, and the resource is not enough when the resource is allocated by other schedulers of the parallel scheduling resource in advance (for example, the resource is allocated to other pods) to start the resource can be avoided.

In addition, whether enough resources are left can be judged by comparing the predetermined cluster nodes corresponding to the Pod with the current resources left by the cluster nodes; or alternatively. Whether enough resources are left can be judged by judging whether the resources of the cluster nodes change, if so, the determined resources of the cluster nodes are considered to be insufficient, and if not, the determined resources of the cluster nodes are considered to be enough.

Step S150: and after the determined cluster node allocated to the created Pod monitors that the created Pod records the identifier of the determined cluster node allocated to the created Pod, starting the created Pod to create a corresponding container.

In the above step S150, the cluster node may implement, through an API Server in kubernets, to listen whether the created Pod (the Pod in the above step S140, the required scheduler and the cluster node are selected (the field in the configuration file has recorded the corresponding field value)) records the determined identifier of the cluster node allocated to the created Pod. If the selected cluster node finds that the selected cluster node is allocated to the Pod, the Pod can be started, so that the Pod is finally put into the cluster node, and the corresponding resource is started.

In the above steps S120 to S150, different scheduler groups are set for different scheduler types, and one scheduler group may have a plurality of schedulers, and schedulers of the same type may perform scheduling in parallel, so that more schedulers may be allowed to schedule resources in parallel. Moreover, the condition of sufficient resources is verified before the Pod and the cluster node are bound, so that the problem of scheduling failure caused by the fact that the scheduler schedules resources in a collision mode can be avoided, and the efficiency of scheduling resources in parallel by more schedulers can be higher.

In some embodiments, in order to allow the scheduling process to be implemented based on the above steps S120 to S150, other scheduling processes, such as a default scheduling process of the kubernets cluster, may be reserved, and an identifier may be set for different scheduling processes for a user to select.

In this case, when a user requests to create a resource object, the user may provide a required scheduling policy identifier to select a scheduling process to be used by the user. For example, the step S110, that is, acquiring a resource object creation request, and creating a Pod according to the resource object creation request, so as to update a required scheduler type in the resource object creation request into a configuration file of the created Pod, may specifically include: acquiring a resource object creation request, creating a Pod according to the resource object creation request, and recording a required scheduler type and a required scheduling policy identifier in the resource object creation request into a configuration file of the created Pod. In this embodiment, the user may select the desired scheduling policy by inputting the desired scheduling policy identifier as a value of the corresponding field.

Further, the method shown in fig. 1 may further include the steps of: s161, when determining that the required scheduling policy identifier existing in the configuration file of the created Pod is the set scheduling policy identifier, executing step S120, that is, after monitoring the created Pod, selecting a scheduler from the scheduler group which is deployed in advance and belongs to the type of the required scheduler, and updating the identifier of the selected scheduler into the configuration file of the created Pod. The set scheduling policy identifier may be set as needed, as long as the scheduling process of the present invention can be identified, for example, the set scheduling policy identifier may be an identifier of a scheduling controller deployed in the present invention for implementing the scheduling process of the present invention.

Alternatively, the method shown in fig. 1 may further include the steps of: and S162, executing a default scheduling flow in Kubernetes under the condition that the required scheduling policy identifier in the created configuration file of the Pod is determined to be the original scheduling policy identifier or the corresponding field is empty. The default scheduling flow in kubernets may be implemented by referring to the existing kubernets cluster. The original scheduling policy identifier may be set as needed, as long as it can identify the default scheduling flow in Kubernetes. In this embodiment, the identification of the required scheduling policy may allow the scheduling policy of the present invention to be used while retaining the default scheduling flow in kubernets. Therefore, the scheduling process of the invention can be obtained by less improvement work on the original Kubernets, and various requirements of users can be met.

In some embodiments, under the kubernets framework, the etc. database in the kubernets may be used to store Pod-related creation, modification and other information, so that the API Server may conveniently retrieve Pod-related information from the database, thereby performing required operations. The interaction with the ETCD database can be realized through an API Server, and the ETCD can realize the operations of data increase, deletion, modification, check and the like based on the original main database of Kubernetes. The change of the specified field data can be monitored by monitoring the change of the ETCD database.

For example, in the step S110, a Pod is created according to the resource object creation request, so as to record the required scheduler type in the resource object creation request into the configuration file of the created Pod, and the method shown in fig. 1 may further include the steps of: and storing the relevant information of the created Pod to the ETCD. Meanwhile, the synchronization does not necessarily mean the same time, and may mean the synchronization of the execution timings, and specifically, when the required scheduler type in the resource object creation request is recorded in the profile of the created Pod, the storage of each piece of information on the created Pod into the etc may be performed.

Further, in some embodiments, if a plurality of different scheduling flows are included, in order to select a desired scheduling flow, the method shown in fig. 1 may further include the steps of: and judging whether the required scheduling policy identifier in each piece of relevant information of the created Pod stored in the ETCD is a set scheduling policy identifier or not through the API Server, so as to judge whether the required scheduling policy identifier in the configuration file of the created Pod is the set scheduling policy identifier or not.

Further, in other embodiments, it may be determined that a new Pod is created by using information related to the determination that the ETCD stores the new Pod, so that the scheduler may be selected.

For example, in the step S120, after the created Pod is monitored, selecting a scheduler from a scheduler group that is deployed in advance and belongs to the required scheduler type, and updating an identifier of the selected scheduler into a configuration file of the created Pod, specifically, the step S may include: and monitoring each piece of relevant information of the created Pod stored in the ETCD through the API Server, confirming to monitor the created Pod, triggering to select a scheduler from a scheduler group which is deployed in advance and belongs to the type of the required scheduler, and recording the identifier of the selected scheduler into a configuration file of the created Pod.

Further, in still other embodiments, the Pod-related information stored in the ETCD may be modified or updated.

For example, in step S120, while updating the identifier of the scheduler in the configuration file, that is, while updating the identifier of the selected scheduler into the configuration file of the created Pod, the method shown in fig. 1 may specifically include: and storing the identification update of the selected scheduler into a corresponding field in each piece of relevant information of the Pod created in the ETCD.

In this case, the API Server may listen to a change in data in the ETCD to obtain a desired listening result. For example, in step S130, after monitoring that the identifier of the scheduler is recorded in the configuration file of the created Pod, the selected scheduler determines the cluster node allocated to the created Pod according to the information of the required resource in the configuration file of the created Pod, which may specifically include: and monitoring the ETCD by the API Server, updating the identifier of the selected scheduler, and confirming that the identifier of the selected scheduler is recorded in the configuration file of the created Pod, and determining the cluster node allocated to the created Pod by the selected scheduler according to the information of the required resource in the configuration file of the created Pod.

For another example, in step S140, while updating the cluster node in the configuration file, that is, while recording the determined identifier of the cluster node allocated to the created Pod, the method shown in fig. 1 may include: and recording the determined identification of the cluster node allocated to the created Pod into a corresponding field in each piece of relevant information of the Pod created in the ETCD. For example, the etc may store fields of various Pod-related information, the fields regarding the cluster nodes may default to null initially, and thereafter, an identification (e.g., name) of the selected cluster node may be recorded as a value of the corresponding field in the etc.

In this case, it can be known by the cluster node that the Pod is bound to the cluster node by the API Server listening for changes in data in the ETCD. For example, in the step S150, after the determined cluster node allocated to the created Pod monitors that the created Pod records the identifier of the determined cluster node allocated to the created Pod, the created Pod is started to create the corresponding container, which may specifically include: and monitoring the identification of the cluster node stored in the ETCD and allocated to the created Pod through the API Server, confirming that the created Pod records the identification of the cluster node allocated to the created Pod, and starting the created Pod by the determined cluster node allocated to the created Pod to create a corresponding container. The creation of the resource object may thus be accomplished.

Further, in still other embodiments, it may be verified whether the resources to be allocated are sufficient by using the cluster node-related information stored in the etc.

Illustratively, the method shown in FIG. 1 may include the steps of: and S160, the admission controller verifies whether the determined resources allocated to the cluster nodes of the created Pod can still meet the requirement of the information of the required resources according to the determined relevant information allocated to the cluster nodes of the created Pod, which is stored in the ETCD.

More specifically, the step S160, that is, the admission controller verifies whether the determined resource allocated to the cluster node of the created Pod still satisfies the requirement of the information of the required resource according to the determined information related to the cluster node allocated to the created Pod stored in the etc d, may include the steps of: s161, reading all the information stored in the ETCD to a cache, and locking the resources of the cluster nodes distributed by each scheduler in the cache; and S162, the admission controller judges whether the determined cluster node allocated to the created Pod can still meet the requirement of the information of the required resource according to the determined resource locking information in the cache of the cluster node allocated to the created Pod. Therefore, the conflict of the parallel scheduling resources of a plurality of schedulers can be better avoided.

In the above step S161 and step S162, all the information in the ETCD may be read into a cache (cache) of the API Server memory. The resource locking can be realized by adding a memory lock. After each scheduler allocates the cluster node resource to the Pod (for example, it can be determined that the cluster node resource has been allocated through successful binding between the Pod and the cluster node), the allocated resource can be locked, and the cluster node resource cannot be allocated again after being locked, so that the problem of failure caused by once allocation of the same cluster node resource can be avoided. For example, after a cluster node binds a Pod (e.g., the identifier of the cluster node is written into the configuration file of the Pod), if the Pod has not been started, the cluster node is allocated to another Pod and the start of the other Pod is completed, in this case, when the previous Pod starts the Pod, the resource is insufficient, and this situation can be avoided as much as possible by locking the resource of the cluster node in the cache after the cluster node is allocated (e.g., the cluster node and the cluster node are bound). If the locking is successful, it is generally guaranteed that no resource conflicts will occur. Whether the resources are sufficient can be judged by whether the corresponding resources are changed. And under the condition of locking the resources, judging whether the resources conflict or not, and avoiding the resource conflict caused by the failure of locking, thereby better avoiding the resource conflict.

After the step S160 is executed, if the verification is passed, the step S140 may be executed; if the verification fails, the record of the identity of the chosen scheduler may be deleted.

Illustratively, the method shown in FIG. 1 may include the steps of: and under the condition that the admission controller receives the request of binding the Pod and the cluster node and verifies that the determined resources of the cluster node distributed to the created Pod cannot meet the requirement of the information of the required resources, deleting the identifier of the selected scheduler in various related information of the Pod created in the ETCD, reselecting the scheduler from a scheduler group which is deployed in advance and belongs to the type of the required scheduler, updating the identifier of the reselected scheduler to the configuration file of the created Pod, and storing the configuration file into corresponding fields in various related information of the Pod created in the ETCD. In this embodiment, since the user mainly cares about the resource given to the user and does not care which scheduler is selected, even if the scheduler identifier that is updated at the beginning is recorded in the profile of the Pod created, when the scheduler needs to be reselected, the identifier of the scheduler does not need to be updated. Of course, in the case where there is an identifier of the scheduler in the Pod (its profile) created, the identifier of the scheduler in the corresponding field in both the ETCD and the Pod may be deleted at the same time. In other embodiments, in particular, if the identifier of the selected scheduler is recorded only to the created Pod, and the identifier of the scheduler is not recorded in the ETCD, only the identifier of the scheduler in the corresponding field in the created Pod (its configuration file) may be deleted when the resources are insufficient.

In the native Kubernetes multi-scheduler function, the number of schedulers needs to be determined in advance according to the existing experience, and the schedulers cannot be automatically expanded or contracted according to the cloud resource cluster condition, so that the requirement of improving the throughput of the schedulers under the emergency condition is difficult to meet. In this regard, in some embodiments, the problem may be solved by adjusting the number of schedulers by taking both the data of the number of cluster nodes and the load condition and the throughput condition of the schedulers, and combining these two data.

Fig. 2 is a schematic flowchart of a cloud resource scheduling method based on kubernets according to another embodiment of the present invention, and referring to fig. 2, the method shown in fig. 1 may further include:

step S170: acquiring kubel data of each cluster node, cluster node monitoring data and metrics data of each scheduler in a scheduler group, acquiring the number and load conditions of the cluster nodes according to the kubel data of each cluster node and the cluster node monitoring data, and acquiring the throughput condition of each scheduler in the scheduler group according to the metrics data of each scheduler in each scheduler group;

step S180: and automatically adjusting the number of the schedulers in the corresponding scheduler group according to the number and the throughput condition of the cluster nodes and the throughput condition of each scheduler in the scheduler group.

In step S170, Prometheus in the kubernets cluster may be used to obtain various required data. Prometheus is suitable for acquiring or monitoring other data in original Kubernets, and in the invention, the Prometheus is improved based on acquiring cluster related data and scheduler related data. Therefore, the data acquisition can be realized by reserving and fully utilizing the original Kubernetes architecture as much as possible. In addition, a setting rule (which can be set according to experience) can be added to Prometheus to obtain the number of cluster nodes and the throughput condition of each scheduler in the scheduler group according to the acquired data. The number of cluster nodes and the throughput of each scheduler in the scheduler group may be exposed through the aggregation layer of the API Server via k8s-promethus-adapter for further analysis by HPA acquisition.

Exemplarily, the step S170 may specifically include: acquiring the metrics data of corresponding schedulers by using services which are deployed between a PromatheusOperator and each scheduler in each scheduler group in advance through Promatheus to obtain the metrics data of each scheduler in each scheduler group; acquiring data of kubel from each cluster node through Prometous and acquiring cluster node monitoring data; and obtaining the number and the load condition of the cluster nodes through Prometous according to the kubel data and the cluster node monitoring data of each cluster node, and obtaining the throughput condition of each scheduler in each scheduler group according to the metrics data of each scheduler in each scheduler group. Further, the number and load condition of cluster nodes and the throughput condition of each scheduler in the scheduler group can be provided for external service access through an API Server aggregation layer.

In step S180, the number of schedulers may be adjusted by using the HPA module of the controller component in the kubernets cluster. The HPA is a function realized in the controller manager, the original Kubernets are used as a scheduler example for managing users outside, and the invention utilizes the Kubernets to manage the internal scheduler of the cluster, thereby reserving and fully utilizing the original architecture of the Kubernets and reducing the workload of improving the cluster.

For example, the step S180 of automatically adjusting the number of schedulers in the corresponding scheduler group according to the number of cluster nodes and the throughput condition of each scheduler in the scheduler group may specifically include: obtaining the number and throughput conditions of cluster nodes and the throughput conditions of all schedulers in a scheduler group through an API Server aggregation layer and providing the cluster nodes and the throughput conditions for HPA access; and under the condition that the total throughput of all schedulers in the scheduler group is judged to be lower than a first expected condition and the number of cluster nodes and the load condition are judged to be lower than a second expected condition through the HPA, the number of the schedulers in the corresponding scheduler group is increased, and under the condition that the total throughput of all the schedulers in the scheduler group is judged to be higher than the first expected condition and the number of the cluster nodes and the load condition are higher than the second expected condition, the number of the schedulers in the corresponding scheduler group is reduced.

Based on the same inventive concept as the cloud resource scheduling method based on Kubernetes shown in fig. 1, the embodiment of the present application further provides a cloud resource scheduling system based on Kubernetes, as described in the following embodiments. Because the principle of solving the problems of the cloud resource scheduling system based on the Kubernetes is similar to that of the cloud resource scheduling method based on the Kubernetes, the implementation of the cloud resource scheduling system based on the Kubernetes can refer to the implementation of the cloud resource scheduling method based on the Kubernetes, and repeated parts are not described again.

Fig. 3 is a schematic structural diagram of a cloud resource scheduling system based on Kubernetes according to an embodiment of the present invention. As shown in fig. 3, the kubernets-based cloud resource scheduling system of the embodiments may include: a resource object creation module 210, a schedule controller module 220, a plurality of scheduler groups 230, an API Server 240, a cluster node module 250, a listening interface module 260, and the like. Each scheduler group contains a plurality of schedulers 231 of the same type. The API Server 240 includes an admission controller module 241.

The resource object creating module 210 is configured to obtain a resource object creating request, create a Pod according to the resource object creating request, and record a required scheduler type in the resource object creating request into a configuration file of the created Pod.

The scheduling controller module 220 is configured to monitor the created Pod, select a scheduler from a scheduler group that is pre-deployed and belongs to the required scheduler type, and update an identifier of the selected scheduler to a configuration file of the created Pod.

And a plurality of scheduler groups 230, each of which includes a plurality of schedulers 231 of the same type, for providing a selected scheduler, and after monitoring that the identifier of the scheduler (the selected scheduler) is recorded in the configuration file of the created Pod, determining the cluster node allocated to the created Pod according to the information of the required resource in the configuration file of the created Pod, and sending a request for binding the Pod and the cluster node.

The admission controller module 241 is configured to, after receiving the request for binding the Pod and the cluster node, verify that the determined resource allocated to the created Pod still meets the requirement of the information of the required resource, and record the determined identifier allocated to the cluster node of the created Pod to the created Pod.

The cluster node module 250 is configured to provide the determined cluster node allocated to the created Pod, and start the created Pod to create a corresponding container after monitoring that the created Pod records the identifier of the determined cluster node allocated to the created Pod.

The monitoring interface module 260 is configured to implement whether monitoring has a created Pod, implement whether monitoring records an identifier of a selected scheduler in a configuration file of the created Pod, and implement whether monitoring records an identifier of a determined cluster node allocated to the created Pod.

Further, the system shown in fig. 3 may further include: core monitor module 270, controller instance HPA management module 280, and so on. The API Server 240 may also include an aggregation layer module 242.

The core monitoring module 270 is configured to obtain kubel data of each cluster node, cluster node monitoring data, and metrics data of each scheduler in the scheduler group, obtain the number of cluster nodes and load conditions according to the kubel data of each cluster node and the cluster node monitoring data, and obtain throughput conditions of each scheduler in the scheduler group according to the metrics data of each scheduler in each scheduler group.

Aggregation layer module 242 is used to provide the number of cluster nodes and throughput conditions and the throughput conditions of each scheduler in the scheduler group to controller instance HPA management module 280.

The HPA management module 280 of the controller instance is configured to automatically adjust the number of schedulers in the corresponding scheduler group according to the number of cluster nodes and the throughput condition of each scheduler in the scheduler group.

In some embodiments, the resource object creating module 210 may be specifically configured to obtain a resource object creating request, and create a Pod according to the resource object creating request, so as to record a required scheduler type and a required scheduling policy identifier in the resource object creating request into a configuration file of the created Pod.

The system shown in fig. 3, may further include: a scheduling policy selection module to: and under the condition that the needed scheduling policy identifier existing in the configuration file of the created Pod is determined to be the set scheduling policy identifier, after the created Pod is monitored, a scheduler is selected from a scheduler group which is deployed in advance and belongs to the type of the needed scheduler, and the identifier of the selected scheduler is updated into the configuration file of the created Pod. Or, the scheduling policy selection module may be configured to execute a default scheduling procedure in kubernets when it is determined that a required scheduling policy identifier in the profile of the created Pod is an original scheduling policy identifier or a corresponding field is empty.

In some embodiments, the API Server 240 may further be configured to store the created Pod information to the ETCD.

In some embodiments, the listening interface module 260 may be further configured to determine, through the API Server, whether a required scheduling policy identifier existing in each piece of information related to the created Pod stored in the ETCD is a set scheduling policy identifier, to determine whether a required scheduling policy identifier existing in a configuration file of the created Pod is a set scheduling policy identifier.

In some embodiments, the listening interface module 260 may be further configured to monitor, through the API Server, that the ETCD stores the relevant information of the created Pod, confirm to monitor the created Pod, trigger selection of a scheduler from a scheduler group that is pre-deployed and belongs to the type of the required scheduler, and record an identifier of the selected scheduler in a configuration file of the created Pod.

In some embodiments, the API Server 240 may be further configured to store the identifier update of the selected scheduler in a corresponding field in each piece of relevant information of the Pod created in the ETCD.

In some embodiments, the monitoring interface module 260 may be further configured to monitor, through the API Server, that the ETCD updates the identifier of the selected scheduler, and confirm that the identifier of the selected scheduler is recorded in the configuration file of the created Pod, and then the selected scheduler determines the cluster node allocated to the created Pod according to the information of the required resource in the configuration file of the created Pod.

In some embodiments, the API Server 240 may further be configured to record the determined identifier of the cluster node assigned to the created Pod into a corresponding field in each piece of related information of the Pod created in the etc.

In some embodiments, the listening interface module 260 may be further configured to listen, through the API Server, that the ETCD stores the identifier of the determined cluster node allocated to the created Pod, and confirm that the created Pod records the identifier of the determined cluster node allocated to the created Pod, and then the determined cluster node allocated to the created Pod starts the created Pod to create the corresponding container.

In some embodiments, the admission controller module 241 may be further configured to verify whether the determined resource allocated to the cluster node of the created Pod still satisfies the requirement of the information of the required resource according to the determined information related to the cluster node allocated to the created Pod stored in the etc d.

In some embodiments, the admission controller module 241 is further configured to, in a case where it is verified that the determined resource allocated to the cluster node of the created Pod cannot meet the requirement of the information of the required resource after receiving the request of the bound Pod and the cluster node, delete the identifier of the selected scheduler in the relevant information of the Pod created in the etc d, and reselect the scheduler from the group of schedulers that are deployed in advance and belong to the type of the required scheduler, and update the identifier of the reselected scheduler into a corresponding field in the relevant information of the Pod created in the etc d.

In some embodiments, the scheduling controller module is specifically operable to select, by the pre-deployed scheduling controller, a scheduler from a group of schedulers pre-deployed and belonging to the desired scheduler type in a round robin scheduling algorithm. Wherein the pre-deployed scheduling controller may include the set scheduling policy identifier.

In some embodiments, the admission controller module may be further configured to read all information stored in the ETCD to a cache (specifically, may locally cache all resource object information stored in the ETCD through the API Server), and lock, in the cache, a resource of the cluster node allocated by each scheduler; and the admission controller judges whether the determined cluster node allocated to the created Pod can still meet the requirement of the information of the required resource according to the determined resource locking information in the cache of the cluster node allocated to the created Pod.

In some embodiments, the core monitoring module may be specifically configured to obtain metrics data of each scheduler in the scheduler group by using a service, which is previously deployed between the promemeus Operator and each scheduler in each scheduler group, by the promemeus; acquiring data of kubel from each cluster node through Prometous and acquiring cluster node monitoring data; and obtaining the number and the load condition of the cluster nodes through Prometous according to the kubel data and the cluster node monitoring data of each cluster node, and obtaining the throughput condition of each scheduler in each scheduler group according to the metrics data of each scheduler in each scheduler group. The number and the load condition of the cluster nodes and the throughput condition of each scheduler in the scheduler group can be provided for external service access through an API Server aggregation layer.

In some embodiments, the aggregation layer module may be specifically configured to provide the number of cluster nodes and throughput conditions and the throughput conditions of each scheduler in the scheduler group to the HPA access. And when the HPA judges that the total throughput condition of all schedulers in the scheduler group is lower than a first expected condition and the number of cluster nodes and the load condition are lower than a second expected condition, the number of the schedulers in the corresponding scheduler group is increased, and when the total throughput condition of all the schedulers in the scheduler group is judged to be higher than the first expected condition and the number of the cluster nodes and the load condition are higher than the second expected condition, the number of the schedulers in the corresponding scheduler group is reduced.

In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the method according to any of the above embodiments.

In order that those skilled in the art will better understand the present invention, embodiments of the present invention will be described below with reference to specific examples.

Fig. 4 is a schematic structural diagram of a cloud resource scheduling system based on Kubernetes according to an embodiment of the present invention. Referring to fig. 4, the kubernets-based cloud resource scheduling system of this embodiment may include: an Admission Controller and a kube-aggregation layer set in the API Server, a plurality of Scheduler groups, a scheduling Controller, cluster nodes (nodes 1-Node n), a promemeus (core monitoring system), a k8s-promethus-adapter, a Controller manager, an etc., and a Default Scheduler. And the method can also comprise monitoring an open interface of the software package, creating a client of the Pod and the like.

The method comprises the steps of writing codes based on an Admission controller in Kubernets so as to verify resource scheduling conflicts. The scheduler group is different from the original Kubernetes multi-scheduler, not only each scheduler group corresponds to one type, and n scheduler groups correspond to n types (type 1-type) one by one, but also each scheduler group can be provided with a plurality of schedulers, and furthermore, the number of schedulers in each scheduler group is scalable. The scheduling controller functions similarly to the kubernets default scheduler, but can be modified to adjust the schedulers in the scheduler group of the present invention. The cluster node can be realized by using the original cluster node in Kubernetes. The ETCD database can be realized by utilizing the original ETCD database in Kubernetes, is used for storing various kinds of relevant information of Pod, and can learn the progress change of the scheduling process by monitoring the change of the database data. The improvement is carried out based on Prometheus in Kubernetes, so that the Prometheus can acquire cluster scale and node monitoring data from cluster nodes and acquire metrics data from each scheduler, and the data can be converted into data capable of measuring the conditions of the schedulers and indexes capable of reflecting the conditions of the cluster nodes according to the acquired data according to set rules. The indexes can be transmitted to an aggregation layer exposed to an API Server through k8s-promethus-adapter in Kubernets, so that the application of the aggregation layer is improved, and a controller manager can take the index data from the aggregation layer to serve as a basis for adjusting the number of schedulers in a scheduler group. The controller manager may have HPA (scheduler instance management) function, where the original HPA is only used to adjust the scheduler instance according to the user's requirement, and here, the HPA is improved to adjust the number of schedulers (i.e. scheduler instances) in the scheduler group of the present invention, thereby realizing scalability of the scheduler. In addition, the module creating the Pod can have the scheduler identifier of the invention transmitted in the field (scheduler name) of the scheduler, and can update the scheduler identifier after the scheduler is selected.

Referring to fig. 4, the scheduling method based on the system may include the following processes:

1. the scheduler of the specified type and its HPA object are created by the deployment controller of Kubernetes, and then HPA will decide whether to adjust the number of schedulers of the corresponding type according to the scheduler instance situation, cluster node number situation and load situation taken by Prometheus. The number of schedulers is expanded when the existing schedulers cannot meet the user's desired throughput and is reduced if the throughput is higher than the user's desired throughput for a long time. In addition, when the cluster is small or the utilization rate of all nodes of the large-scale cluster is in a high level, the number of schedulers can be reduced, because the throughput cannot be improved by a plurality of schedulers under the condition, the throughput is even lower than that of a small number of schedulers, and the probability of resource competition caused by multiple schedulers is obviously increased.

2. And creating a scheduling controller (scheduler-controller) which has the function of selecting the schedulers from the scheduler group of the specified type according to the stored.

3. When a user creates a resource object (Pod), if a scheduling controller is employed, the schedulerName and store. If the kubernets default scheduler is used, no settings need to be made.

4. When the API Server (application program interface Server) solidifies the Pod to the ETCD.

5. The scheduling controller notices (monitors) that the schedulerName is created for the Pod of the scheduler-controller, selects a proper scheduler in the corresponding type of scheduler group according to the value of the store.

6. And then the scheduler identification schedulerName of the Pod is saved in the ETCD through the API Server.

7. And monitoring (Watch), after the dispatcher of the schedulers name finds the Pod allocated to the dispatcher, if the dispatcher is a dispatcher defined by a user, selecting a proper cluster node according to a dispatching strategy defined by the user.

8. And (3) sending a binding request (binding Pod and node) to the API Server, and if the API Server is a Kubernet native scheduler, selecting a proper node according to the flow of the native scheduler and then sending the binding request.

The API Server, upon receiving the binding request, will validate the plug-in that called this scheme when passing through the admission controller (admission controller). The plug-in is used for preventing resource competition under the condition of multiple schedulers, because different schedulers do not know the requirements of the opposite side when selecting nodes at the same time, the decision that the nodes all meet the resource requirements can be made, but the nodes are found to be insufficient in resources after respective Pod falls into the nodes, although one layer of verification is carried out after the Pod falls into the nodes, the verification in advance is more favorable for avoiding the overhead brought by resource conflict.

10. After the API Server binds a node to a Pod, the kubel of the corresponding node finds the Pod and verifies the Pod with a predicate argument and the like.

11. If so, Pod will be launched by the container runtime.

Referring back to fig. 4, more specifically, the scheduling method based on the system may include the following processes:

first, the components or configurations involved in the present invention need to be deployed on a kubernets cluster (if the kubernets cluster is not created, the kubernets cluster needs to be established first, and then the scheme is deployed), and then a new scheduling scheme can be used when a user creates a resource object. The scheme is deployed without affecting the existing applications on the existing cluster, for example, the applications are temporarily unavailable, and the specific deployment steps are as follows:

1. a docker image of the scheduler is prepared and a scheduler instance is created by the deployment controller.

The created scheduler instance only listens for the creation of pod with its own name (through pod object spec. schedulername), and here and after steps are described by taking the example of creating a type1, name my-schedulers, single copy of the resource object of the deployment resource. In addition, a ServiceAccount, clusterrollingwhich gives the scheduler instance permission to access cluster resources is created for it.

2. Creating HPAs enables flexible scaling of the number of scheduler instances.

First, in order to collect the related metrics data of each scheduler instance through Prometheus, a service needs to be deployed through Prometheus Operator to communicate with my-schedulers. Then Prometheus will gather the following data exposed by the scheduler through this service:

schedule _ e2e _ scheduling _ latency _ microspheres (scheduling delay);

schedule _ binding _ latency _ microspheres (allocation/binding elapsed time);

scheduler _ scheduling _ algorithm _ latency _ policies (scheduling algorithm);

in addition to the data for the scheduler instance, Prometheus also requires data to be taken from the various nodes of the cluster to kubel and monitoring data for the nodes, both of which are typically configured by default.

1)kubelet

kubelet _ running _ container _ count (number of containers)

kubel _ running _ Pod _ count (Pod number)

kubel _ running _ operations _ latency _ semiconductors (container running speed)

2) Node monitoring data

node _ load1 (average load of node in the past one minute)

node _ load5 (average load of nodes in the past five minutes)

node _ load15 (node average load in the past fifteen minutes)

node _ cpu _ seconds _ total (cpu working time)

node _ memory _ MemAvailable _ bytes (available memory)

node _ memory _ bytes (total memory)

node _ memory _ Buffers _ bytes (cache)

node _ memory _ SwapCached _ bytes (exchange area)

node _ memory _ Cached _ bytes (cache)

node _ memory _ MemFree _ bytes (free memory)

node _ memory _ SwapFree _ bytes (available switching zone)

Secondly, adding a rule to Prometeus to generate a new index, wherein the rule calculates the collected index information according to a certain strategy and stores the result into a new index Pod _ requests _ per _ second and a current _ cluster _ load, and then the k8s-promethus-adapter exposes the new index to an aggregation layer (kube-aggregator) of the APIserver through an API (application programming interface).

Finally, the HPA controller is set to query the pop _ requests _ per _ second and current _ cluster _ load every two seconds (the kube-controller-manager service initiation parameter horizontal-Pod-autoscaler-sync-period is set to 2 seconds, default 15 seconds) through the custom. If the HPA controller finds from these two data that the total throughput of all scheduler instances of a given type does not reach the desired value within 1 second, the number of scheduler instances of that type will be increased, and will not change if equal to or above the desired value. If the Pod _ requests _ per _ second is below the set value or the current _ cluster _ load is above the desired threshold, the number of scheduler instances will be reduced.

3. A scheduling controller is created.

After compiling the source code of the scheduling controller into a binary file and packaging into a docker image (assuming the name of scheduler-controller), a Pod resource object is started, and its yacml file is shown below.

4. And deploying an admission controller plug-in of the APIServer.

In order to avoid influencing the existing Kubernetes cluster, the plug-in solves the problem of resource competition of a multi-scheduler through the ValidatingAdmissionWebhook of the API Server. Because the validatingaddissessionwebhook can be used to call a remote plug-in registered in the system, some user-defined admission control operations are performed by the remote plug-in, which avoids the following problems:

the starting parameter of the API Server needs to be specified;

compiling a new plug-in into a binary file of the API Server;

dynamic loading of plug-ins during runtime is also not possible.

After the scheme is deployed to the kubernets cluster, when a user creates a resource object, the specific workflow of the scheme is as follows:

1. a deployment resource object using a scheduler-controller for scheduling is created, the YAML file of which is shown below, and then sent to the API Server via kubecect.

2, the API server receives the request of the client and stores the resource content in the database etcd;

the Deployment controller monitors and reacts to the resource change, checks the database change and prepares to create a desired number of Pod instances;

a scheduler-controller checks the database change, finds a Pod which is not allocated to a specific node, and updates the name (such as my-scheduler-01) of a scheduler group (my-schedulers) corresponding to a type1 (an identification example of the scheduler type) into spec.schedulername of a nginx-default resource object according to a scheduler which is suitable for selecting a Round-robin policy in accordance with the "store.cetcloud.cn/type: type 1";

monitoring a scheduler my-scheduler-01 (identifier/name example of the scheduler) in my-schedulers to find that the database is changed and the scheduler name is a Pod of the my-scheduler-01 which is not allocated to a specific node, allocating the Pod to a node which can run the node according to a set of relevant rules, updating a database record through a Bind interface, and recording the Pod allocation condition;

and 6, the API Server receives the Bind operation, calls a remote plug-in to verify whether the resource conflict occurs on the distributed nodes or not through the validatingAdmissionWebhook, returns a failure if the resource conflict is found, and otherwise, the Bind operation is successful.

Kubelet monitors database changes, manages subsequent Pod lifecycles, and discovers the Pod assigned to the node where it is located. Before the Pod is started, a second confirmation operation (submit) is performed, and various pre-selection (preset) admission checks are performed on the Pods, including checking whether the node has enough resources such as CPUs, memories and the like, so that the resource competition condition is further prevented from occurring.

In this embodiment, on the basis of the kubernets framework, a scheme that a plurality of schedulers can schedule in parallel is implemented without modifying component source codes thereof, and the number of actual schedulers is scaled according to the current state of the existing schedulers, the current number of nodes of a cluster and the load condition, and a user can select different types of schedulers to process different types of tasks. The method is further improved on the basis of a Kubernetes multi-scheduler, and adopts a Horizon PodAutoschaler (HPA) to manage the scheduler examples, so that the number of the scheduler examples can be rapidly adjusted according to the cluster condition, and elastic scaling is realized. On the other hand, the scheme is based on a Kubernetes native framework, is realized by utilizing the existing mechanism of Kubernetes, and has no any invasive change to the code of the Kubernetes.

In summary, in the cloud resource scheduling method based on Kubernetes, the cloud resource scheduling system based on Kubernetes, and the computer-readable storage medium according to the embodiments of the present invention, different scheduler groups are set for different scheduler types, and one scheduler group may have multiple schedulers, and schedulers of the same type may perform scheduling in parallel, so that more schedulers may be allowed to schedule resources in parallel. Moreover, the condition of sufficient resources is verified before the Pod and the cluster node are bound, so that the problem of scheduling failure caused by the fact that the scheduler schedules resources in a collision mode can be avoided, and the efficiency of scheduling resources in parallel by more schedulers can be higher. Furthermore, the number of the schedulers in the corresponding scheduler group is adjusted according to the number and the throughput condition of the cluster nodes and the throughput condition of each scheduler in the scheduler group, so that the scheduling can be automatically expanded according to the condition of the cloud resource cluster, and the requirement of improving the throughput of the schedulers in an emergency can be met.

In the description herein, reference to the description of the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," "an example," "a particular example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. The sequence of steps involved in the various embodiments is provided to schematically illustrate the practice of the invention, and the sequence of steps is not limited and can be suitably adjusted as desired.

The present invention can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A cloud resource scheduling method based on Kubernetes is characterized by comprising the following steps:

2. The Kubernetes-based cloud resource scheduling method of claim 1,

acquiring a resource object creation request, creating a Pod according to the resource object creation request, and recording a required scheduler type in the resource object creation request into a configuration file of the created Pod, wherein the method comprises the following steps:

acquiring a resource object creation request, creating a Pod according to the resource object creation request, and recording a required scheduler type and a required scheduling policy identifier in the resource object creation request into a configuration file of the created Pod;

the method further comprises the following steps:

a step of selecting a scheduler from a scheduler group which is deployed in advance and belongs to the type of the required scheduler after monitoring the created Pod and updating the identifier of the selected scheduler into the configuration file of the created Pod under the condition that the required scheduling policy identifier existing in the configuration file of the created Pod is determined as a set scheduling policy identifier; or, when determining that the required scheduling policy identifier in the created Pod configuration file is the original scheduling policy identifier or the corresponding field is empty, executing a default scheduling flow in kubernets.

3. The Kubernetes-based cloud resource scheduling method of claim 2,

creating a Pod according to the resource object creation request, so as to record the required scheduler type in the resource object creation request into a configuration file of the created Pod, and simultaneously, the method further comprises the following steps:

storing each piece of relevant information of the created Pod to the ETCD;

the method further comprises the following steps:

judging whether a required scheduling policy identifier existing in each piece of relevant information of the created Pod stored in the ETCD is a set scheduling policy identifier or not through the API Server, and judging whether the required scheduling policy identifier existing in a configuration file of the created Pod is the set scheduling policy identifier or not;

after the created Pod is monitored, a scheduler is selected from a scheduler group which is deployed in advance and belongs to the type of the required scheduler, and the identification of the selected scheduler is updated to a configuration file of the created Pod, wherein the method comprises the following steps:

monitoring each piece of relevant information of the created Pod stored in the ETCD through the API Server, confirming to monitor the created Pod, triggering to select a scheduler from a scheduler group which is deployed in advance and belongs to the type of the required scheduler, and recording the identifier of the selected scheduler into a configuration file of the created Pod;

while updating the identity of the selected scheduler into the profile of the created Pod, the method further comprises:

updating and storing the identifier of the selected scheduler into corresponding fields in each piece of relevant information of the Pod established in the ETCD;

after monitoring that the self identification is recorded in the configuration file of the created Pod, the selected scheduler determines the cluster node allocated to the created Pod according to the information of the required resource in the configuration file of the created Pod, including:

monitoring that the ETCD updates and stores the identifier of the selected scheduler through the API Server, and confirming that the identifier of the selected scheduler is recorded in the configuration file of the created Pod, and determining the cluster node allocated to the created Pod by the selected scheduler according to the information of the required resource in the configuration file of the created Pod;

recording the determined identification of the cluster node allocated to the created Pod, and simultaneously, the method further comprises:

recording the determined identification of the cluster node allocated to the created Pod into corresponding fields in each piece of relevant information of the Pod created in the ETCD;

after the determined cluster node allocated to the created Pod listens that the created Pod records the identifier of the determined cluster node allocated to the created Pod, starting the created Pod to create a corresponding container, including:

and monitoring the identification of the cluster node stored in the ETCD and allocated to the created Pod through the API Server, confirming that the created Pod records the identification of the cluster node allocated to the created Pod, and starting the created Pod by the determined cluster node allocated to the created Pod to create a corresponding container.

4. The Kubernetes-based cloud resource scheduling method of claim 3,

the method further comprises the following steps:

the admission controller verifies whether the determined resources allocated to the cluster nodes of the created Pod can still meet the requirement of the information of the required resources according to the determined relevant information allocated to the cluster nodes of the created Pod and stored in the ETCD;

and/or the presence of a gas in the gas,

the method further comprises the following steps:

and under the condition that the admission controller receives the request of the bound Pod and the cluster node and verifies that the determined resources distributed to the cluster node of the created Pod cannot meet the requirement of the information of the required resources, deleting the identifier of the selected scheduler in the relevant information of the Pod created in the ETCD, reselecting the scheduler from a scheduler group which is deployed in advance and belongs to the type of the required scheduler, and updating the identifier of the reselected scheduler into a corresponding field in the relevant information of the Pod created in the ETCD.

5. The Kubernetes-based cloud resource scheduling method of claim 2,

selecting a scheduler from a group of schedulers pre-deployed and belonging to the desired scheduler type, comprising:

selecting a scheduler from a group of schedulers that are pre-deployed and belong to the desired scheduler type by a pre-deployed scheduling controller according to a round-robin scheduling algorithm; wherein, the pre-deployed scheduling controller comprises the set scheduling policy identifier;

and/or the presence of a gas in the gas,

the admission controller verifies whether the determined resources allocated to the cluster nodes of the created Pod can still meet the requirement of the information of the required resources according to the determined relevant information allocated to the cluster nodes of the created Pod, which is stored in the ETCD, and comprises the following steps:

reading all information stored in the ETCD to a cache, and locking the resources of the cluster nodes distributed by each scheduler in the cache;

and the admission controller judges whether the determined cluster node allocated to the created Pod can still meet the requirement of the information of the required resource according to the determined resource locking information in the cache of the cluster node allocated to the created Pod.

6. The Kubernetes-based cloud resource scheduling method according to any one of claims 1 to 5, further comprising:

acquiring kubel data of each cluster node, cluster node monitoring data and metrics data of each scheduler in a scheduler group, acquiring the number and load conditions of the cluster nodes according to the kubel data of each cluster node and the cluster node monitoring data, and acquiring the throughput condition of each scheduler in the scheduler group according to the metrics data of each scheduler in each scheduler group;

and automatically adjusting the number of the schedulers in the corresponding scheduler group according to the number and the throughput condition of the cluster nodes and the throughput condition of each scheduler in the scheduler group.

7. The Kubernetes-based cloud resource scheduling method of claim 6,

acquiring kubel data, cluster node monitoring data and metrics data of all schedulers in a scheduler group of each cluster node, acquiring the number and load conditions of the cluster nodes according to the kubel data and the cluster node monitoring data of all cluster nodes, and acquiring the throughput conditions of all schedulers in the scheduler group according to the metrics data of all schedulers in each scheduler group, wherein the method comprises the following steps:

acquiring metrics data of a corresponding scheduler by using a service which is deployed between a Prometheus Operator and each scheduler in each scheduler group in advance through Prometheus to obtain the metrics data of each scheduler in each scheduler group; acquiring data of kubel from each cluster node through Prometous and acquiring cluster node monitoring data; obtaining the number and the load condition of cluster nodes through Prometous according to kubel data and cluster node monitoring data of each cluster node, and obtaining the throughput condition of each scheduler in each scheduler group according to metrics data of each scheduler in each scheduler group;

automatically adjusting the number of schedulers in a corresponding scheduler group according to the number and throughput condition of cluster nodes and the throughput condition of each scheduler in the scheduler group, comprising:

obtaining the number and throughput conditions of cluster nodes and the throughput conditions of all schedulers in a scheduler group through an API Server aggregation layer and providing the cluster nodes and the throughput conditions for HPA access; and under the condition that the total throughput of all schedulers in the scheduler group is judged to be lower than a first expected condition and the number of cluster nodes and the load condition are judged to be lower than a second expected condition through the HPA, the number of the schedulers in the corresponding scheduler group is increased, and under the condition that the total throughput of all the schedulers in the scheduler group is judged to be higher than the first expected condition and the number of the cluster nodes and the load condition are higher than the second expected condition, the number of the schedulers in the corresponding scheduler group is reduced.

8. A cloud resource scheduling system based on Kubernetes is characterized by comprising:

9. The Kubernetes-based cloud resource scheduling system of claim 8, further comprising:

10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.