CN117785382A

CN117785382A - Resource monitoring and dynamic scheduling method, equipment and medium based on k8s cluster

Info

Publication number: CN117785382A
Application number: CN202311825699.0A
Authority: CN
Inventors: 华芮; 汪中原; 苏洪明
Original assignee: Nanchang Keneng Urban Rail Technology Co ltd; Hefei Technological University Intelligent Robot Technology Co ltd; CSG Smart Science and Technology Co Ltd
Current assignee: Nanchang Keneng Urban Rail Technology Co ltd; Hefei Technological University Intelligent Robot Technology Co ltd; CSG Smart Science and Technology Co Ltd
Priority date: 2023-12-26
Filing date: 2023-12-26
Publication date: 2024-03-29

Abstract

The invention relates to a resource monitoring and dynamic scheduling method, equipment and medium based on k8s clusters, which comprises the steps of constructing a set of Kubernetes which are hereinafter referred to as k8s on a plurality of linux servers, and at least comprising a cluster environment of a main node and a plurality of computing nodes; installing a precursor-Operator service component, a node-exporters acquisition component, an alert manager alarm component, a pushgateway push component and a grafana visualization tool component in a k8s cluster environment; cleaning various indexes collected by prometaus, and screening monitoring index data required by business; supporting to set platform resource scheduling rules to reschedule cluster services; supporting setting an alarm strategy to alarm index data monitored by a server; the system is displayed to clients in a graph mode through statistical analysis of various indexes of the cluster server. The invention realizes the unified management of k8s cluster resource monitoring and scheduling rules, realizes the automatic resource scheduling and management of the platform, reduces the artificial operation and maintenance cost and the error rate, improves the operation stability of cluster service, and greatly improves the cluster resource utilization rate.

Description

Resource monitoring and dynamic scheduling method, equipment and medium based on k8s cluster

Technical Field

The invention relates to the technical field of server virtualization containers, in particular to a method, equipment and a storage medium for resource monitoring and dynamic scheduling based on a k8s cluster.

Background

With development of cloud computing and containerization technologies, more and more enterprises choose to use docker and k8s to realize efficient application programming deployment and management, and simultaneously promethaus is used as a preferred k8s cluster monitoring scheme, however, in actual enterprise service deployment, for example, a metro operation company power supply intelligent operation and maintenance system, the power supply management requirement is strict, the power supply environment is complex, equipment systems are numerous, each system is required to provide efficient and stable service for 7 x 24 hours, the operation and maintenance cost of the power supply system is high, the overall planning capability is weak, the problems of insufficient resources and balanced load of a cluster server cannot be found in time, and the resource monitoring and scheduling rule of a k8s cluster is a key problem.

In order to provide a flexible and effective method for managing and optimizing the scheduling rules and resource monitoring of k8s cluster resources, in the prior art, k8s cluster container services are independently deployed on physical machines with different production environments, each pod container is mutually communicated by adopting HTTP and TCP communication mechanisms and is a stateless service, how to monitor the use conditions of various hardware resources of the physical machine in real time, and how to deploy the service states of each container on the physical machine and the resource utilization conditions, under the architecture of a plurality of physical machines and a large number of micro-services, the scheduling rules of k8s cluster container scheduling are dynamically and effectively adjusted according to indexes such as k8s cluster server resource monitoring, virtual container instance monitoring and the like, and the k8s cluster container scheduling rules are not fast and effectively improved.

Disclosure of Invention

The method, the device and the storage medium for resource monitoring and dynamic scheduling based on the k8s cluster can at least solve one of the technical problems in the background technology.

In order to achieve the above purpose, the present invention adopts the following technical scheme:

a resource monitoring and dynamic scheduling method based on k8s cluster comprises the following steps,

and (3) building a set of k8s and dock virtualization container environments on a plurality of linux physical machines to be monitored, wherein the k8s large version can be 1.23, the dock version can be 20.10, and after the installation is finished, verifying whether the cluster is available.

And deploying a promethaus-Operator controller component for k8s cluster resource monitoring, and monitoring the variation of the customized resources related to the promethaus, and automatically executing corresponding operations according to the variation.

The deployment node-exporters component is used for collecting resource data of server cpu occupancy rate, memory use, disk use and virtualized containers, the official network provides a plurality of exporter collectors for different middleware, the exporter collectors can be downloaded and decompressed according to project requirements to operate during use, and a user can also use Clinet Libraries of prometaus to customize exporter collection indexes.

The alert manager component is deployed for monitoring the alert function and supporting custom rules for alert.

And deploying a pushgateway component, and acquiring monitoring index data by adopting passive pushing, so as to ensure that precursor can collect target indexes and ensure the integrity of resource data.

The grafana visual display component is deployed, so that the resource data analysis, the data statistics, the display and the report can be intuitively displayed to the resource utilization condition of the client.

How to acquire various resource index data exposed by prometaus, the platform can write an acquisition interface by using an http protocol, and the real-time acquisition server monitors index display.

For the cleaning and persistence of the prometaus acquisition index, the platform can acquire the meta interface data at regular time intervals in a manner of a regular task, edit code cleaning logic, store the cleaned data into a distributed file storage mongdb database, and provide data source support for customizing historical resource analysis requirements for clients.

In yet another aspect, the invention also discloses a computer readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of the method as described above.

In yet another aspect, the invention also discloses a computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of the method as above.

According to the technical scheme, based on the operation steps, a user can clearly and intuitively see the resource use condition of each server in the k8s cluster, and the problem of unbalanced idle load of certain nodes due to high use rate of certain node servers in the k8s cluster is solved; aiming at the problem that the resource utilization rate of CPU, memory and the like allocated by some pod container services is low, a user can set different resource scheduling strategies on a platform, so that the overall service quality of a k8s cluster is improved, and the utilization rate of cluster server resources is improved;

a user can set a reasonable scheduling rule strategy according to cluster node resource conditions, so as to avoid resource bottleneck problems, such as k8s node load imbalance, the user can set various resource use thresholds of a cluster node server, a platform can judge that the selected resource is relatively sufficient, splice a container service. Yaml, and call an Api interface of k8s to pull up the container;

for pod container service of cluster nodes, the utilization condition of the container for distributing resources such as cpu, memory and the like can be monitored, a pod container resource utilization rate threshold value is set, and for the problem of low container resource utilization rate, a platform can prompt threshold value alarm information to remind operation and maintenance personnel, and can splice service. Yaml again to set a reasonable resource value and pull up the container service again;

for the node server alarm data pushed by the alarm strategy set by the platform, the node scheduling strategy can be set as well, if the calling response time of a certain service api interface is too long, it is inferred that the container is caused by cpu core number and insufficient memory resources, the resources of the allocation container can be increased, and the container is pulled up again.

The invention provides a resource monitoring and dynamic scheduling method based on k8s clusters, which supports real-time monitoring and historical data analysis, realizes unified management of k8s cluster resource monitoring and scheduling rules, is related to each other, influences a platform to realize automatic resource scheduling and management, reduces the labor operation cost and error rate, improves the running stability of cluster service, and greatly improves the cluster resource utilization rate.

Drawings

FIG. 1 is a flow chart of monitoring index data according to the present invention;

FIG. 2 is a flow chart of the resource scheduling rules of the present invention;

fig. 3 is a system frame diagram of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention.

As shown in fig. 1, fig. 2 and fig. 3, in the resource monitoring and dynamic scheduling method based on the k8s cluster, a set of k8s and dock virtualized container environments are built on a plurality of linux physical machines to be monitored, the large k8s version can be 1.23, the dock version can be 20.10, and after the installation is completed, whether the cluster is available is verified.

As shown in fig. 1, based on the above operation steps, a user can clearly and intuitively see the resource use condition of each server in the k8s cluster, and aims at the problems that the use rate of certain node servers in the k8s cluster is very high and the idle load of certain nodes is unbalanced; aiming at the problem that the resource utilization rate of CPU, memory and the like allocated by some pod container services is low, a user can set different resource scheduling strategies on a platform, so that the overall service quality of a k8s cluster is improved, and the utilization rate of cluster server resources is improved;

as shown in fig. 2, a user can set a reasonable scheduling rule policy according to the cluster node resource condition, so as to avoid resource bottleneck problems, such as k8s node load imbalance, and can set various resource use thresholds of a cluster node server, a platform can judge that the selected resource is relatively sufficient, splice container service. Yaml, and call an Api interface of k8s to pull up a container;

Based on the above, the following examples,

the method for resource monitoring and dynamic scheduling based on k8s cluster in the embodiment of the invention comprises the following steps:

building a set of Linux serversKubernetes(hereinafter referred to as k8 s) a cluster environment comprising at least one master node and a plurality of computing nodes;

installing a precursor-Operator service component, a node-exporters acquisition component, an alert manager alarm component, a pushgateway push component and a grafana visualization tool component in a k8s cluster environment;

cleaning various indexes acquired by prometaus, screening monitoring index data required by a service, performing persistence operation on the cleaned index data, and storing the data in a mongdb database;

supporting to set platform resource scheduling rules to reschedule cluster services;

supporting setting an alarm strategy to alarm index data monitored by a server;

through the statistical analysis of various indexes of the cluster server, the index is displayed to the client in a chart mode, and the most visual cluster server resource use condition is provided for the client.

Specifically, a set of K8s cluster environment is firstly built on a plurality of linux servers, a large version of dock can be selected to be 20.10 versions, and a large version of K8s can be selected to be 1.23, and the specific operation steps are as follows:

configuration of an alicloud dock source:

Wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo-O/etc/yum.repos.d/docker-ce.repo，

installing docker version:

yuminstall-y docker-ce-20.10.21docker-ce-cli-20.10.21containerd.io

installing kubuead related components:

yuminstall-y kubelet-1.23.0kubeadm-1.23.0kubectl-1.23.0

after the command execution is completed, verifying whether the k8s cluster is successfully deployed.

The k8s cluster is used for deploying a promethaus-Operator component and is used for deploying and managing the cluster promethaus, and a promethaus instance is automatically set and managed through the deployed promethaus-Operator component;

the node-exporters acquisition component is mainly used for monitoring indexes of a Linux system, such as the number of processes of a current server, CPU consumption, memory, disk space, http, tcp connection number and other resource information, exporters are the sum of Prometaus data acquisition components, and the official network provides a large number of exporter collectors, and can be operated by downloading and decompressing according to the requirement;

the alert manager component mainly realizes the function of monitoring the alarm, and the alert manager mainly receives the alarm information sent by the promethaus, supports the custom rule of the alarm, judges whether the index exceeds a threshold according to the rule of the rule, and sends the alarm to the alert manager if the index exceeds the threshold and is not recovered for a period of time;

the pushgateway is another data acquisition mode, and a prometaus plug-in for acquiring monitoring data by adopting passive pushing can customize a monitoring index script to send to the pushgateway, and the pushgateway pushes the data to prometaus service;

grafana is an open source data visualization tool developed by the Go language, which can perform visual display on data collected by prometaus, support various chart types and visual options, including line drawings, bar charts, instrument panels, maps and the like, and can select the most suitable mode for displaying the data according to the needs;

and cleaning various indexes acquired by the prometaus, writing an acquisition interface by using an http protocol, starting a timing task, calling a metrics interface exposed by the prometaus every other time period, acquiring server monitoring index data, storing the required index data into a mongdb database, customizing the requirements for development, and providing data source support.

The method supports the setting of platform resource scheduling rules to reschedule cluster services, and comprises the following specific operation steps: based on k8s cluster virtualization deployment, resources of scheduling service, such as cpu number, gpu core number, memory size and the like, can be dynamically set according to a monitoring resource index analysis result of a promethaus server, parameters are spliced to generate a service. Yaml file, and an Api interface provided by k8s is called to pull up a service container.

The method supports setting an alarm strategy to alarm index data monitored by a server, and comprises the following specific operation steps: threshold warning can be set for each item of server index data collected by prometaus, and when the index data is pushed by the timing http collection interface, logic judges whether the index data is warned or not.

Through index analysis of the k8s cluster server, the method presents visual resource utilization conditions to clients in a graph mode, and is characterized in that: by setting up a real-time resource data monitoring and analyzing platform, a grafana presentation page of a resource data analysis, data statistics, presentation and report can be integrated into a web project of the user, the grafana is set to allow an iframe page to access, the back-end service of the java platform starts auth.proxy, servlet proxy is carried out on the grafana service, and the front-end vue page iframe accesses a grafana panel which is processed by the back-end proxy.

In summary, as shown in fig. 3, the embodiment of the invention provides a resource monitoring and dynamic scheduling method based on k8s clusters, which supports real-time monitoring and historical data analysis, realizes unified management of k8s cluster resource monitoring and scheduling rules, and interrelates influence, realizes automatic resource scheduling and management by a platform, reduces the labor operation cost and error rate, improves the operation stability of cluster service, and greatly improves the cluster resource utilization rate.

In yet another embodiment provided herein, there is also provided a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the k8s cluster-based resource monitoring and dynamic scheduling methods of the above embodiments.

It may be understood that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and explanation, examples and beneficial effects of the related content may refer to corresponding parts in the above method.

The embodiment of the application also provides an electronic device, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus,

a memory for storing a computer program;

and the processor is used for realizing the resource monitoring and dynamic scheduling method based on the k8s cluster when executing the program stored in the memory.

The communication bus mentioned by the above electronic device may be a peripheral component interconnect standard (english: peripheral Component Interconnect, abbreviated: PCI) bus or an extended industry standard architecture (english: extended Industry Standard Architecture, abbreviated: EISA) bus, or the like. The communication bus may be classified as an address bus, a data bus, a control bus, or the like.

The communication interface is used for communication between the electronic device and other devices.

The Memory may include random access Memory (Random Access Memory, abbreviated as RAM) or nonvolatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.

The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; it may also be a digital signal processor (English: digital Signal Processing; DSP; for short), an application specific integrated circuit (English: application Specific Integrated Circuit; ASIC; for short), a Field programmable gate array (English: field-Programmable Gate Array; FPGA; for short), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A resource monitoring and dynamic scheduling method based on k8s clusters is characterized by comprising the following steps of constructing a set of Kubernetes below k8s on a plurality of linux servers, wherein the cluster environment at least comprises a main node and a plurality of computing nodes;

supporting setting an alarm strategy to alarm index data monitored by a server;

2. The k8s cluster-based resource monitoring and dynamic scheduling method according to claim 1, wherein: firstly, a set of K8s cluster environment is built on a plurality of linux servers, 20.10 versions are selected from a large version of a dock, 1.23 versions of the K8s are selected, and the specific operation steps are as follows:

configuration of an alicloud dock source:

wgethttps://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo-O/etc/yum.repos.d/docker-ce.repo，

installing docker version:

yuminstall-y docker-ce-20.10.21docker-ce-cli-20.10.21containerd.io

installing kubuead related components:

yuminstall-y kubelet-1.23.0kubeadm-1.23.0kubectl-1.23.0

3. The k8s cluster-based resource monitoring and dynamic scheduling method according to claim 1, wherein:

the k8s cluster deploys a promethaus-Operator component for deployment and management of the cluster, and a promethaus instance is automatically set and managed through the deployed promethaus-Operator component;

the node-exporters acquisition component is used for monitoring indexes of the Linux system, and comprises resource information such as the number of processes, CPU consumption, memory, disk space, http and tcp connection number of a current server, wherein exporter is the sum of Prometaus data acquisition components, and the official network provides a large number of exporter collectors, and can be operated by downloading and decompressing according to the requirement;

the alert manager component is used for realizing a monitoring alarm function, the alert manager is used for receiving alarm information sent by promethaus, supporting a custom rule of rule alarm, judging whether an index exceeds a threshold value according to the rule of rule, and sending an alarm to the alert manager if the index exceeds the threshold value and is not recovered for a period of time;

grafana is an open source data visualization tool developed by the Go language, and can visually display data acquired by prometaus, support various chart types and visualization options, including a line graph, a bar chart, a dashboard and a map, and select the most suitable mode for displaying the data according to the needs.

4. The k8s cluster-based resource monitoring and dynamic scheduling method according to claim 1, wherein: and cleaning various indexes acquired by the prometaus, writing an acquisition interface by using an http protocol, starting a timing task, calling a metrics interface exposed by the prometaus every other time period, acquiring server monitoring index data, storing the required index data into a mongdb database, customizing the requirements for development, and providing data source support.

5. The k8s cluster-based resource monitoring and dynamic scheduling method according to claim 1, wherein: the method also comprises the following specific operation steps of: based on k8s cluster virtualization deployment, resources of scheduling service, including cpu number, gpu core number, memory size and parameter splicing, can be dynamically set according to the analysis result of monitoring resource indexes by a promethaus server, a service. Yaml file is generated by parameter splicing, and an Api interface provided by k8s is called to pull up a service container.

6. The k8s cluster-based resource monitoring and dynamic scheduling method according to claim 1, wherein: the method also comprises the following specific operation steps of: threshold warning can be set for each item of server index data collected by prometaus, and when the index data is pushed by the timing http collection interface, logic judges whether the index data is warned or not.

7. The k8s cluster-based resource monitoring and dynamic scheduling method according to claim 1, wherein: the index analysis of the k8s cluster server is carried out, the visual resource utilization condition is shown to a client in a graph mode, and the method specifically comprises the steps of integrating a grafana display page of resource data analysis and data statistics, display and report forms into a web project by setting up a real-time resource data monitoring and analysis platform, setting up grafana to allow iframe page access, starting an auth.proxy for java platform back-end service, carrying out servlet proxy on the grafana service, and accessing a grafana panel which is proxied by the back-end through the front-end vue page iframe.

8. A computer readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of the method of any one of claims 1 to 6.

9. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method of any of claims 1 to 6.