CN115297112A - Dynamic resource quota and scheduling component based on Kubernetes - Google Patents
Dynamic resource quota and scheduling component based on Kubernetes Download PDFInfo
- Publication number
- CN115297112A CN115297112A CN202210912913.5A CN202210912913A CN115297112A CN 115297112 A CN115297112 A CN 115297112A CN 202210912913 A CN202210912913 A CN 202210912913A CN 115297112 A CN115297112 A CN 115297112A
- Authority
- CN
- China
- Prior art keywords
- resource
- node
- pod
- quota
- scheduling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- YHXISWVBGDMDLQ-UHFFFAOYSA-N moclobemide Chemical compound C1=CC(Cl)=CC=C1C(=O)NCCN1CCOCC1 YHXISWVBGDMDLQ-UHFFFAOYSA-N 0.000 claims abstract description 11
- 230000006978 adaptation Effects 0.000 claims abstract description 5
- 238000012544 monitoring process Methods 0.000 claims description 45
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 238000007726 management method Methods 0.000 claims description 5
- 238000005311 autocorrelation function Methods 0.000 claims description 4
- 238000013523 data management Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000003745 diagnosis Methods 0.000 claims description 2
- 210000002569 neuron Anatomy 0.000 claims description 2
- 238000000034 method Methods 0.000 abstract description 3
- 230000001131 transforming effect Effects 0.000 abstract 2
- 230000006870 function Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011002 quantification Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/50—Testing arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a dynamic resource quota and scheduling component based on Kubernets, which comprises: acquiring historical resource usage amount of application service in a Kubernetes cluster; analyzing the usage amount of each application resource by adopting an ARIMA and LSTM prediction model and dynamically adjusting quota; acquiring the actual usage amount of each resource of each Node; evaluating the adaptation degree of the Node and the Pod to be scheduled from multiple dimensions; and scheduling the Pod to be scheduled to the target Node. The assembly provided by the invention is used for transforming the traditional Kubernetes resource quota method, predicting the use amount of Pod resources by using a time sequence prediction model and dynamically adjusting quota; and transforming a Kubernetnes scheduling module, and estimating and scheduling the Pod to the target Node from dimensions such as a CPU, a memory, a network bandwidth, a disk IO, a priority and the like by utilizing the actual resource usage amount of the Node. The difficulty of determining the resource usage amount when a Kubernets user deploys the application is reduced, and the problem of unbalanced scheduling of data intensive or I/O intensive applications is solved.
Description
Technical Field
The invention belongs to the technical field of cloud computing, and particularly relates to a dynamic resource quota and scheduling component based on Kubernetes.
Background
Kubernetes is an open-source container cluster management system, provides functions of arranging, automatically deploying, discovering services and scheduling resources for large-scale container groups, and provides a whole set of container application solution for users. The application scenes are very wide, and the method becomes an industry standard. However, kubernetes currently has the following disadvantages in resource allocation and scheduling:
1) Kubernets give the right to allocate resources to Pod to users, and the users can request the resources according to the requirements. In a practical scenario, it is statistical that in about 70% of cases, the resources requested by the user exceed the required resources, and the application resource over-allocation results in a decrease in throughput and cluster resource utilization, while under-allocation, although more services can be deployed in each node, will cause the application to compete for resources when the services are busy, resulting in an increase in task latency.
2) When measuring the node resource capacity, the Kubernets default scheduler applies for the sum of resources through applications already deployed on the nodes, belongs to a static value, cannot accurately represent the actual load of the nodes, and only pays attention to a CPU and a memory during scheduling, so that when file storage, a mirror image center and some data intensive applications are scheduled, a plurality of applications of the type can be scheduled to a few repeated nodes, and the node resource bottleneck is caused.
Disclosure of Invention
The invention aims to provide a dynamic resource quota and scheduling component based on Kubernetes, so as to solve the problems in the background technology.
In order to solve the technical problems, the invention provides the following technical scheme: a dynamic resource quota and scheduling component based on Kubernetes comprises the steps of predicting the resource usage amount of a Pod in the future for a period of time through a time sequence model and dynamically adjusting, selecting the most appropriate Node for the Pod to deploy from five aspects of CPU, memory, network bandwidth, disk I/O and Node high-priority application deployment conditions by utilizing the actual load information of the Node, and further comprising the following steps:
the resource monitoring module is used for monitoring, alarming and collecting data, particularly monitoring the actual use condition of various resources of each node in the Kubernetes cluster, sending alarm information to a manager when the load of the node is overhigh, and providing a resource load inquiring function for a user;
the dynamic resource quota module is used for predicting the use amount of the Pod resources, comparing the predicted value with the current Pod resource quota value, and if the predicted value exceeds or is lower than a threshold interval, adjusting the resource amount applied by the Pod, specifically providing an application service automatic quota adjusting function for a user;
and the dynamic scheduling module is used for scheduling the Pod of the unbound nodes, measuring the adaptation degree of the Pod and each Node from multiple dimensions through the Node information acquired by the monitoring module, and finally selecting the most suitable Node to deploy the Pod on the Node.
Preferably, the resource monitoring module includes:
the index monitoring unit is used for monitoring the CPU, the memory, the network bandwidth and the real-time use condition of the disk I/O of each node in the Kubernetes cluster, and monitoring the actual use condition of resources of each Pod on the node and the working state of the Pod;
the alarm management unit is used for alarming the Node and the Pod with abnormal monitoring data, and the alarm rules established by the user comprise whether the resource usage of the Node and the Pod exceeds a threshold value and whether the working state of the Pod interface can normally provide service to the outside;
and the data management unit is used for acquiring and monitoring the acquired historical data of each resource, processing abnormal data, packaging the data and inquiring an interface for a user.
Preferably, the dynamic resource quota module includes:
the resource prediction unit is used for predicting the resource usage of the Pod in the next period of time by using a time series prediction model according to the historical resource usage of the Pod obtained by the monitoring module;
the dynamic adjustment unit is used for comparing the Pod resource usage amount obtained by the resource prediction unit with the current Pod resource usage amount and realizing whether to carry out dynamic quota on the Pod or not according to the error between the Pod resource usage amount and the current Pod resource usage amount;
preferably, the resource prediction unit further includes:
acquiring resource usage including CPU, memory, network bandwidth and disk I/O;
linear prediction, namely analyzing the obtained use amount of each resource of Pod by adopting an ARIMA prediction model;
performing nonlinear prediction, namely predicting residual errors by adopting an LSTM (least squares metric) model according to the difference between a true value and a linear predicted value;
model checking, namely checking the prediction model through a root mean square error, an average absolute error and an average absolute percentage error;
and predicting the result, and obtaining a final predicted value through the sum of linearity and nonlinearity.
Preferably, the dynamic scheduling module includes:
the pre-selection scheduling unit is used for continuously pre-checking all nodes, including the surplus of network bandwidth, the occupation of ports and the sufficiency of disk space, and eliminating the nodes which do not meet the requirements;
the optimal scheduling unit is used for selecting the node most suitable for Pod in a quantification way for the nodes meeting the conditions, and describing the load balancing degree of each resource by adopting each resource variance;
and the binding unit is used for deploying the Pod to the Node with the highest score.
Compared with the prior art, the invention has the following beneficial effects: aiming at the situation that the quota value cannot be determined before the Pod is deployed, the resource usage amount of the Pod in a future period of time is predicted by constructing a model, and dynamic adjustment is carried out, so that the service quality is ensured, and the user experience is improved; when disk I/O and data intensive applications are faced, a plurality of applications of the type are dispatched to the same node, the real-time load of the node is obtained through monitoring, and the node is evaluated from more dimensions, so that the resource load of each node is more balanced, and the resource utilization rate and the load balancing degree of an enterprise cluster are improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate preferred embodiments of the invention and together with the description serve to explain the principles of the invention in which:
FIG. 1 is a general architecture diagram of a dynamic resource quota and scheduling component based on Kubernetes in accordance with the present invention;
FIG. 2 is a flow chart of a dynamic resource quota and scheduling component based on Kubernetes in accordance with the present invention;
FIG. 3 is a flow diagram of a resource monitoring module according to an embodiment of the present invention;
FIG. 4 is a flow diagram of a dynamic resource quota module in accordance with an embodiment of the present invention;
FIG. 5 is a flow chart of a dynamic scheduling module according to an embodiment of the present invention;
FIG. 6 is a flow chart of the ARIMA and LSTM combined prediction model.
Detailed Description
The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the description in the present specification. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The term Kubernets is used herein, is an open-source cross-platform container cluster management system, and can be used for automatically deploying, expanding and managing containerized applications, and in architectural design, kubernets defines a series of expansion points for design structures, so that users can expand according to their own needs, and thus, the Kubernets can meet various different workloads;
the term "Prometheus" is used herein, which is a set of open source system monitoring alarm framework that can monitor resources in kubernets cluster and display data in a time-series database.
The term "Pod" is used herein, a Pod being the smallest deployable computing unit that can be created and managed in kubernets, which is a combination of one or more containers.
The term "Node" is used herein, a Node being a working Node of a kubernets cluster, which may be a physical machine or a virtual machine.
The term "ARIMA" is used herein, and ARIMA is a time series modeling method for performing optimization expansion on an autoregressive moving average model, because the autoregressive moving average model requires that a time series meets stationarity, but most data in real life does not have stationarity, so that the autoregressive moving average model cannot be directly used, but an original time series can be differentiated, and if the differentiated sequence can pass stability check and become stable, the autoregressive moving average model can be continuously used.
The term "LSTM" is used herein, and long-short term memory neural networks (LSTM) are an improved model proposed in 1997 to address the problems of gradient disappearance and gradient explosion that occur with the cyclic neural network model.
Example 1: a dynamic resource quota and scheduling assembly based on Kubernetes, the assembly overall architecture diagram is as shown in figure 1, the assembly generally comprises 4 layers of infrastructure layer, data storage layer, service logic layer and interaction layer, the main part of the invention is located in the service logic layer, the function comprises predicting the resource usage amount of Pod in a period of time and dynamically adjusting through a time sequence model, and the most appropriate Node is selected for Pod to be deployed from five aspects of CPU, memory, network bandwidth, disk I/O and Node high-priority application deployment condition by utilizing the actual load information of the Node, further comprising:
the resource monitoring module is used for monitoring, alarming and collecting data, particularly monitoring the actual use condition of various resources of each node in the Kubernetes cluster, sending alarm information to a manager when the load of the node is overhigh, and providing a resource load inquiring function for a user;
the dynamic resource quota module is used for predicting the use amount of the Pod resources, comparing the predicted value with the current Pod resource quota value, and if the predicted value exceeds or is lower than a threshold interval, adjusting the resource amount applied by the Pod, specifically providing an automatic quota adjustment function of the application service for the user;
and the dynamic scheduling module is used for scheduling the Pod of the Node which is not bound yet, measuring the adaptation degree of the Pod and each Node from multiple dimensions through the Node information acquired by the monitoring module, and finally selecting the most suitable Node to deploy the Pod on the Node.
The resource monitoring module includes:
the index monitoring unit is used for monitoring the CPU, the memory, the network bandwidth and the real-time use condition of the disk I/O of each node in the Kubernets cluster, and monitoring the actual use condition of resources of each Pod on the node and the working state of the Pod;
the alarm management unit is used for alarming the Node nodes and the Pod with abnormal monitoring data, and the alarm rules established by the user comprise whether the resource usage amount of the Node nodes and the Pod exceeds a threshold value and whether the working state of the Pod interface can normally provide services to the outside;
and the data management unit is used for collecting and monitoring the acquired historical data of each resource, processing abnormal data, packaging the data and inquiring an interface for a user.
The dynamic resource quota module includes:
the resource prediction unit is used for predicting the resource usage of the Pod in the next period of time by using a time sequence prediction model according to the historical resource usage of the Pod obtained by the monitoring module;
the dynamic adjustment unit is used for comparing the Pod resource usage amount obtained by the resource prediction unit with the current Pod resource usage amount and realizing whether to carry out dynamic quota on the Pod or not according to the error between the Pod resource usage amount and the current Pod resource usage amount;
the resource prediction unit further includes:
acquiring resource usage including CPU, memory, network bandwidth and disk I/O;
linear prediction, namely analyzing the usage amount of each acquired Pod resource by adopting an ARIMA prediction model;
performing nonlinear prediction, namely predicting residual errors by adopting an LSTM (least squares metric) model according to the difference between a true value and a linear predicted value;
model checking, namely checking the prediction model through a root mean square error, an average absolute error and an average absolute percentage error;
and predicting the result, and obtaining a final predicted value through the sum of linearity and nonlinearity.
Preferably, the dynamic scheduling module includes:
the pre-selection scheduling unit is used for continuously pre-checking all nodes, including the surplus of network bandwidth, the occupation of ports and the sufficiency of disk space, and eliminating the nodes which do not meet the requirements;
the optimal scheduling unit is used for selecting the node most suitable for the Pod in a quantification manner for the nodes meeting the conditions, and describing the load balancing degree of each resource by adopting each resource variance;
and the binding unit is used for deploying the Pod to the Node with the highest score.
Example 2: fig. 1 is a general architecture diagram of a dynamic resource quota and scheduling component based on kubernets according to the present invention, and fig. 2 is a flowchart of a dynamic resource quota and scheduling component based on kubernets according to the present invention, where the component uses a user as a main body, and performs a dynamic quota on a specific Pod according to a user requirement, so as to provide a disk I/O and data intensive scheduler for the user, and includes the following steps:
and S100, monitoring the resource usage amount of each node and Pod on the node in the cluster, giving an alarm according to a predefined rule, and collecting data required to be used.
As shown in fig. 3, step S100 specifically includes the steps of:
step S101, a monitoring module is set up, and nodes and application indexes needing to be monitored, such as a CPU, a memory, a network bandwidth, a disk I/O and the like, are configured;
step S102, configuring alarm names, alarm rules and alarm sending ways, such as high load of resources like a CPU, a memory and the like;
step S103, operating an alarm rule according to the monitoring data, and sending alarm information;
step S104, collecting monitoring data and providing an inquiry interface for the outside;
step S200, inquiring the Pod needing dynamic quota in the cluster, obtaining the historical resource usage amount of the service from the monitoring module, predicting and dynamically adjusting through a prediction model, and specifically comprising the following steps:
as shown in fig. 4, step S200 specifically includes the steps of:
step S201, obtaining CPU, memory, network bandwidth and historical usage amount of disk I/O of Pod in the cluster;
step S210, predicting a time sequence formed by the use amount of the Pod historical resources by a combined prediction model;
as shown in fig. 6, step S210 further specifically includes the steps of:
step S211, firstly, carrying out stationarity check on the time sequence of resource usage, if not, carrying out difference operation until the stationarity check is passed, recording the difference times as d, determining P and q of ARIMA (p, d, q) types through an autocorrelation function and a partial autocorrelation function, carrying out parameter estimation and model diagnosis on the model to obtain a final model, finally, predicting historical resource usage by using a prediction model to obtain a predicted value, and recording the predicted value as a predicted value
Step S212, calculating the real value y and the predicted value of the previous step
Residual error between, noted as e t ；
Step S213, establishing LSTM model pair residual e t Predicting, wherein the model is divided into four layers, namely an input layer, an LSTM layer 1, an LSTM layer 2 and an output layer, the neuron of the output layer is 1, the input layer is selected to be more than 5, and the number of the intermediate layers is determined according to an empirical formula
To select, obtain the final model to predict the residual error, and record as
Step S214, the ARIMA model prediction value is obtained
And LSTM prediction value
Adding the obtained data to obtain a final predicted value
Step S220, comparing the predicted results
And the current resource application amount of Pod, setting 0.2 as a threshold value, and if the application value is greater than the predicted value
Multiply by 1.2 or less than predicted
Multiplying by 0.8, then adjusting; if the quota is not required to be adjusted in the interval;
step S230, updating the quota value of each resource for Pod, and waiting for entering the next dynamic quota cycle;
step S300, selecting an optimal node for deployment through a pre-selection strategy and an optimal strategy according to the resource amount applied by the Pod and the actual load of each node;
as shown in fig. 5, step S300 specifically includes the steps of:
step S301, acquiring actual usage amounts of a CPU, a memory, a network bandwidth, a disk I/O and the like of each node, and acquiring a Pod resource application amount to be scheduled;
step S302, in a preselection stage, nodes with insufficient resources or conflicts, such as insufficient disk space, port conflicts, insufficient network bandwidth and the like, are filtered, so that the nodes do not participate in the next stage, and the calculated amount is reduced;
step S303, in the optimization stage, after the Pod to be scheduled is scheduled to each node, the node resource load balance degree is calculated, and the calculation formulas of the balance degree and the score are respectively as follows:
and step S304, selecting the node with the highest score, deploying the Pod to the node, and updating the node information.
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that various changes, modifications, substitutions and equivalents can be made without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. A dynamic resource quota and scheduling component based on Kubernetes comprises the steps of dynamically adjusting application resource application amount by utilizing monitoring data of all nodes of a cluster and a time sequence prediction model, measuring node and application adaptation degree from multiple dimensions by utilizing actual resource load of nodes, and scheduling, and is characterized by comprising the following steps:
the resource monitoring module is used for monitoring the use condition of each resource of each node in the cluster, monitoring the use condition and the working state of the resource applied on the node, giving an alarm according to load and providing data for other modules;
the dynamic resource quota module is used for dynamically adjusting various resource application amounts of application services on the nodes, acquiring the actual use condition of each resource of the application, predicting by using a combined prediction model and determining whether to adjust;
and the dynamic scheduling module is used for scheduling the application service of the unbound nodes to the node which is most suitable for the application service, and balancing the resource use of each node by measuring the adaptation degree from a plurality of dimensions.
2. The dynamic resource quota and scheduling assembly based on Kubernetes of claim 1, wherein the resource monitoring module comprises:
the index monitoring unit is used for monitoring the real-time use conditions of the CPU, the memory, the network bandwidth and the disk I/O of each node and application in the cluster;
the alarm management unit is used for alarming Node nodes and Pod with abnormal monitoring data;
and the data management unit is used for acquiring and monitoring the acquired resource use data, processing abnormal data, packaging the data and inquiring an interface for a user.
3. The dynamic resource quota and schedule component based on Kubernetes of claim 1, wherein the dynamic resource quota module comprises:
the resource prediction unit is used for predicting the resource usage of the Pod in the next period of time by using a time series prediction model according to the application historical resource usage obtained by the monitoring module;
and the dynamic adjusting unit is used for comparing and determining whether to adjust according to the predicted value and the current value.
4. The Kubernets-based dynamic resource quota and scheduling component of claim 3, wherein the resource prediction unit comprises:
acquiring resource usage including CPU, memory, network bandwidth and disk I/O;
linear prediction, i.e. prediction of linear part of future resource usage by using ARIMA model
Non-linear prediction, namely predicting residual errors by adopting an LSTM model according to the difference between a true value and a linear predicted value;
model checking for checking model accuracy;
and predicting the result, and obtaining a final predicted value through the sum of linearity and nonlinearity.
5. The dynamic resource quota and schedule component based on Kubernetes of claim 1, wherein the dynamic scheduling module comprises:
the pre-selection scheduling unit is used for continuously pre-checking all nodes, including the surplus of network bandwidth, occupied ports and sufficient disk space, and eliminating the nodes which do not meet the requirements;
the optimal scheduling unit is used for selecting the optimal node from the nodes meeting the conditions for deployment;
and the binding unit is used for binding the Pod with the optimal node and updating the binding information.
6. The dynamic resource quota and scheduling component based on kubernets of claim 1, wherein the component is based on a user, performs dynamic quota on a specific Pod according to user demand, and provides a disk I/O and data intensive scheduler for the user, comprising the steps of:
step S100, monitoring the resource usage amount of each node and Pod on the node in the cluster, giving an alarm according to a predefined rule, collecting data required to be used,
step S200, inquiring the Pod needing dynamic quota in the cluster, obtaining the historical resource usage amount of the service from the monitoring module, predicting and dynamically adjusting through a prediction model,
and step S300, selecting the optimal node for deployment through a pre-selection strategy and an optimal strategy according to the resource amount applied by the Pod and the actual load of each node.
7. The dynamic resource quota and scheduling assembly based on Kubernetes as claimed in claim 1, wherein step S100 specifically comprises the steps of:
step S101, a monitoring module is set up, and nodes and application indexes needing to be monitored, such as a CPU, a memory, a network bandwidth, a disk I/O and the like, are configured;
step S102, configuring alarm names, alarm rules and alarm sending ways, such as high load of resources like a CPU, a memory and the like;
step S103, operating an alarm rule according to the monitoring data, and sending alarm information;
and step S104, collecting monitoring data and providing an inquiry interface for the outside.
8. The dynamic resource quota and scheduling assembly based on Kubernetes according to claim 1, wherein step S200 specifically comprises the steps of:
step S201, obtaining CPU, memory, network bandwidth and historical usage amount of disk I/O of Pod in the cluster;
step S210, predicting a time sequence formed by the use amount of the Pod historical resources by a combined prediction model;
step S220, comparing the predicted results
And the current resource application amount of Pod, setting 0.2 as a threshold value, and if the application value is greater than the predicted value
Multiply by 1.2 or less than predicted
Multiplying by 0.8, then adjusting; if the quota is not adjusted in the interval, the quota is not required to be adjusted;
step S230, updating the quota value of each resource for Pod, and waiting for entering a next dynamic quota cycle;
step S210 further specifically includes the steps of:
step S211, firstly, the stationarity check is carried out on the time sequence of resource usage, if the time sequence of resource usage is not stationary, the difference operation is needed until the stationarity check is passed, the difference times are marked as d, p and q of ARIMA (p, d, q) types are determined through the autocorrelation function and the partial autocorrelation function, and the parameter estimation and the model are carried out on the modelType diagnosis, obtaining a final model, finally using the prediction model to predict the usage amount of the historical resources to obtain a predicted value, and recording the predicted value as
Step S212, calculating the real value y and the predicted value of the previous step
Residual error between them, denoted as e t ；
Step S213, establishing LSTM model pair residual e t Predicting, wherein the model is divided into four layers, namely an input layer, an LSTM layer 1, an LSTM layer 2 and an output layer, the neuron of the output layer is 1, the input layer is selected to be more than 5, and the number of the intermediate layers is determined according to an empirical formula
To select, obtain the final model to predict the residual error, and record as
Step S214, the ARIMA model prediction value is calculated
And LSTM prediction value
Adding to obtain the final predicted value
9. The dynamic resource quota and scheduling assembly based on Kubernetes as claimed in claim 1, wherein step S300 specifically includes the steps of:
step S301, acquiring actual usage amounts of a CPU, a memory, a network bandwidth, a disk I/O and the like of each node, and acquiring a Pod resource application amount to be scheduled;
step S302, in a preselection stage, nodes with insufficient resources or conflicts, such as insufficient disk space, port conflicts, insufficient network bandwidth and the like, are filtered, so that the nodes do not participate in the next stage, and the calculated amount is reduced;
step S303, in the optimization stage, after the Pod to be scheduled is scheduled to each node, the node resource load balance degree is calculated, and the calculation formulas of the balance degree and the score are respectively as follows:
and step S304, selecting the node with the highest score, deploying the Pod to the node, and updating the node information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210912913.5A CN115297112A (en) | 2022-07-31 | 2022-07-31 | Dynamic resource quota and scheduling component based on Kubernetes |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210912913.5A CN115297112A (en) | 2022-07-31 | 2022-07-31 | Dynamic resource quota and scheduling component based on Kubernetes |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115297112A true CN115297112A (en) | 2022-11-04 |
Family
ID=83825824
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210912913.5A Pending CN115297112A (en) | 2022-07-31 | 2022-07-31 | Dynamic resource quota and scheduling component based on Kubernetes |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115297112A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115550371A (en) * | 2022-12-05 | 2022-12-30 | 安超云软件有限公司 | Pod scheduling method and system based on Kubernetes and cloud platform |
| CN115665157A (en) * | 2022-11-14 | 2023-01-31 | 杭州谐云科技有限公司 | Balanced scheduling method and system based on application resource types |
| CN116112495A (en) * | 2023-04-12 | 2023-05-12 | 中国人民解放军国防科技大学 | Data-driven decentralized autonomous infrastructure service platform |
| CN116244085A (en) * | 2023-05-05 | 2023-06-09 | 江苏博云科技股份有限公司 | Kubernetes cluster container group scheduling method, device and medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109960585A (en) * | 2019-02-02 | 2019-07-02 | 浙江工业大学 | A kind of resource regulating method based on kubernetes |
| CN110990159A (en) * | 2019-12-25 | 2020-04-10 | 浙江大学 | Historical data analysis-based container cloud platform resource quota prediction method |
| CN111984381A (en) * | 2020-07-10 | 2020-11-24 | 西安理工大学 | Kubernetes resource scheduling optimization method based on historical data prediction |
| CN113010270A (en) * | 2021-04-08 | 2021-06-22 | 桂林电子科技大学 | Kubernetes platform-based dynamic resource load balancing scheduling method and system |
| CN113391913A (en) * | 2021-07-12 | 2021-09-14 | 中国科学技术大学 | Distributed scheduling method and device based on prediction |
| CN113986479A (en) * | 2021-09-29 | 2022-01-28 | 武汉纺织大学 | Resource scheduling strategy optimization method based on Kubernetes cluster |
-
2022
- 2022-07-31 CN CN202210912913.5A patent/CN115297112A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109960585A (en) * | 2019-02-02 | 2019-07-02 | 浙江工业大学 | A kind of resource regulating method based on kubernetes |
| CN110990159A (en) * | 2019-12-25 | 2020-04-10 | 浙江大学 | Historical data analysis-based container cloud platform resource quota prediction method |
| CN111984381A (en) * | 2020-07-10 | 2020-11-24 | 西安理工大学 | Kubernetes resource scheduling optimization method based on historical data prediction |
| CN113010270A (en) * | 2021-04-08 | 2021-06-22 | 桂林电子科技大学 | Kubernetes platform-based dynamic resource load balancing scheduling method and system |
| CN113391913A (en) * | 2021-07-12 | 2021-09-14 | 中国科学技术大学 | Distributed scheduling method and device based on prediction |
| CN113986479A (en) * | 2021-09-29 | 2022-01-28 | 武汉纺织大学 | Resource scheduling strategy optimization method based on Kubernetes cluster |
Non-Patent Citations (5)
| Title |
|---|
| 杨鹏飞: "《基于Kubernetes的资源动态调度的研究与实现》", 中国优秀硕士学位论文全文库 信息科技辑, no. 12, 15 December 2017 (2017-12-15), pages 19 - 30 * |
| 杨鹏飞: "《基于Kubernetes的资源动态调度的研究与实现》", 中国优秀硕士学位论文全文库 信息科技辑, no. 12, 5 December 2012 (2012-12-05), pages 19 - 30 * |
| 杨鹏飞: "基于Kubernetes的资源动态调度的研究与实现_杨鹏飞", 中国优秀硕士学位论文全文库 信息科技辑, no. 12, 15 December 2017 (2017-12-15), pages 19 - 30 * |
| 马航: "《基于 Kubernetes 的容器云平台资源动态调度的研究与实现》", 中国优秀硕士学位论文全文库 信息科技辑, no. 03, 15 March 2020 (2020-03-15), pages 28 - 34 * |
| 马航: "基于Kubernetes的...台资源动态调度的研究与实现", 中国优秀硕士学位论文全文库 信息科技辑, no. 03, 13 March 2020 (2020-03-13), pages 28 - 34 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115665157A (en) * | 2022-11-14 | 2023-01-31 | 杭州谐云科技有限公司 | Balanced scheduling method and system based on application resource types |
| CN115550371A (en) * | 2022-12-05 | 2022-12-30 | 安超云软件有限公司 | Pod scheduling method and system based on Kubernetes and cloud platform |
| CN116112495A (en) * | 2023-04-12 | 2023-05-12 | 中国人民解放军国防科技大学 | Data-driven decentralized autonomous infrastructure service platform |
| CN116112495B (en) * | 2023-04-12 | 2023-06-23 | 中国人民解放军国防科技大学 | Data-driven decentralized autonomous infrastructure service platform |
| CN116244085A (en) * | 2023-05-05 | 2023-06-09 | 江苏博云科技股份有限公司 | Kubernetes cluster container group scheduling method, device and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115297112A (en) | Dynamic resource quota and scheduling component based on Kubernetes | |
| Hulshof et al. | Tactical resource allocation and elective patient admission planning in care processes | |
| US7054934B2 (en) | Tailorable optimization using model descriptions of services and servers in a computing environment | |
| US7072960B2 (en) | Generating automated mappings of service demands to server capacities in a distributed computer system | |
| US11579933B2 (en) | Method for establishing system resource prediction and resource management model through multi-layer correlations | |
| CN111045820B (en) | Container scheduling method based on time sequence prediction | |
| CN109753356A (en) | A kind of container resource regulating method, device and computer readable storage medium | |
| CN113110914A (en) | Internet of things platform construction method based on micro-service architecture | |
| CN108366082A (en) | Expansion method and flash chamber | |
| CN113806018A (en) | Kubernetes cluster resource hybrid scheduling method based on neural network and distributed cache | |
| CN114356587B (en) | Calculation power task cross-region scheduling method, system and equipment | |
| CN113037877A (en) | Optimization method for time-space data and resource scheduling under cloud edge architecture | |
| CN116467082A (en) | Big data-based resource allocation method and system | |
| CN114205317B (en) | SDN and NFV-based service function chain SFC resource allocation method and electronic equipment | |
| Xiao et al. | Dscaler: A horizontal autoscaler of microservice based on deep reinforcement learning | |
| CN117056018A (en) | Resource scheduling method, apparatus, device, program product and storage medium | |
| CN114978913B (en) | Cross-domain deployment method and system for service function chains based on cut chains | |
| CN112328395A (en) | Cloud resource capacity planning method and system | |
| CN115562841B (en) | Cloud video service self-adaptive resource scheduling system and method | |
| CN111796933A (en) | Resource scheduling method, device, storage medium and electronic equipment | |
| CN115525394A (en) | Method and device for adjusting number of containers | |
| CN115658319A (en) | Resource scheduling method, system, device and storage medium | |
| CN115658287A (en) | Method, apparatus, medium, and program product for scheduling execution units | |
| Toka et al. | Resource provisioning for highly reliable and ultra-responsive edge applications | |
| Li et al. | An adaptive read/write optimized algorithm for Ceph heterogeneous systems via performance prediction and multi-attribute decision making |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |