CN112527448A - Openstack-based dynamic load adjustment method and system - Google Patents
Openstack-based dynamic load adjustment method and system Download PDFInfo
- Publication number
- CN112527448A CN112527448A CN202010896885.3A CN202010896885A CN112527448A CN 112527448 A CN112527448 A CN 112527448A CN 202010896885 A CN202010896885 A CN 202010896885A CN 112527448 A CN112527448 A CN 112527448A
- Authority
- CN
- China
- Prior art keywords
- load
- openstack
- data
- nodes
- dynamic load
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
Abstract
The invention relates to a dynamic load adjusting method and system based on openstack. The method comprises the following steps: acquiring characteristic data of a physical machine and a virtual machine serving as a monitored object in real time based on a zabbix monitoring node; a prediction step of predicting an abnormal load of the feature data based on a predetermined prediction model; and a scheduling step of performing load balancing processing when the abnormal load is predicted in the predicting step. According to the invention, the problem of insufficient Openstack dynamic resource scheduling in the prior art can be solved, the resources are balanced in load by constructing the load regulator, and the dynamic migration scheduling can be carried out on the virtual machine according to the load condition of the actual running node.
Description
Technical Field
The invention relates to a cloud technology, in particular to a dynamic load adjusting method based on openstack and a dynamic load adjusting system based on openstack.
Background
OpenStack (cloud computing management platform) is an open-source cloud computing management platform project, provides scalable and flexible cloud computing services for private clouds and public clouds, and aims to provide a cloud computing management platform which is simple to implement, can be expanded in a large scale, is rich and has a unified standard.
Openstack comprises two main modules, Nova and Swift. The former is a virtual server deployment and service computation module, and the latter is a distributed cloud storage module.
In the prior art, a virtual machine scheduling policy used in openstack is a static scheduling policy, and resource scheduling is realized by performing three steps of host filtering, weight selection and host selection through a nova-scheduler, so that the problem of how to distribute a newly-built virtual machine to a physical server is solved.
However, the static scheduling policy used by Openstack does not perform automatic migration after the virtual machine is initialized and scheduled, which will cause that Openstack cannot perform dynamic migration and scheduling on the virtual machine according to the load condition of the actually running node, although the service requirement of the user can be ensured, the actual resource utilization condition is ignored, and the resource utilization rate is low.
Because the migration of the virtual machine is manually completed due to the static resource scheduling strategy in the Opnestack, the dynamic migration according to the change of the node load condition is not realized; in the actual operation of the Openstack, the demands of the virtual machines running on the Openstack on physical resources are dynamically changed, a situation of local aggregation is easily caused in continuous operation, some nodes may have too high load, and some nodes are idle, so that a situation of load imbalance occurs in a system, and resource waste is serious.
Disclosure of Invention
In view of the foregoing problems, the present invention is directed to a dynamic load adjustment method based on openstack and a dynamic load adjustment system based on openstack, which are capable of achieving load dynamic balance.
The openstack-based dynamic load adjustment method is characterized by comprising the following steps of:
acquiring characteristic data of a physical machine and a virtual machine serving as a monitored object in real time based on a zabbix monitoring node;
a prediction step of predicting an abnormal load of the feature data based on a predetermined prediction model; and
and a scheduling step of performing load balancing processing when the abnormal load is predicted in the predicting step.
Optionally, the acquiring step comprises:
building a zabbix monitoring node;
accessing a monitored object to the zabbix monitoring node; and
and establishing a CUM node to summarize the characteristic data obtained by all zabbix monitoring nodes.
Optionally, in the acquiring step, the zabbix monitoring nodes are set by using distributed nodes.
Optionally, in the acquiring step, different zabbix monitoring nodes are set for different servers, and different CUM nodes are set for different zabbix monitoring node clusters.
Optionally, in the acquiring step, one or more of CPU occupancy, memory occupancy, disk occupancy, and network traffic of the physical machine and the virtual machine are acquired in real time as the feature data of the monitored object.
Optionally, the acquiring step further comprises: all the obtained feature data are normalized so as to be mapped to an order of magnitude to eliminate singular samples.
Optionally, the characteristic data is normalized using the following formula,
wherein the normalization process is on yjJ e (1, 2, 3.. 4) to generate a new sequence xjAnd j belongs to (1, 2, 3.. 4), the variance is 1, the mean value is 0, and the value of the characteristic data is mapped to [0,1]On the interval of the time interval, the time interval is,
wherein j represents the characteristic data of the monitored object, yiFor the actual value of the feature data corresponding to j,
xito correspond to y after standardizationiAnd (4) obtaining the scaled value.
Optionally, between the acquiring step and the training step, further comprising:
and a weft descending step, namely performing data weft descending on the characteristic data by adopting a specified algorithm.
Optionally, in the predicting step, the abnormal load prediction is performed on the feature data subjected to data reduction based on a composite prediction model.
Optionally, the composite predictive model comprises N predictive models,
the predicting step includes:
respectively carrying out model training on the N prediction models by adopting data after data weft reduction;
determining a weight for each prediction model in the composite prediction model against prescribed features of the N prediction models;
building a composite prediction model based on the weights; and
performing abnormal load prediction on the characteristic data based on the constructed composite prediction model,
wherein N is a natural number of 2 or more.
Optionally, when N is equal to 2, the 2 prediction models are respectively an SVM model and a random forest model.
Optionally, when N is equal to 2, the 2 prediction models are a k-means model and a decision tree model, respectively.
Optionally, the scheduling step includes:
under the condition that the abnormal load is predicted in the predicting step, acquiring a virtual machine load sequence with the predicted abnormal load by using a bubble array method;
calculating comprehensive load coefficients of physical nodes with no predicted abnormality and sequencing the physical nodes by using a bubble arrangement method;
selecting physical nodes with the highest matching degree from the physical nodes with the unpredicted abnormality and calculating a load variation coefficient; and
and judging whether to execute virtual machine migration or not based on the calculated value of the load variation coefficient.
Optionally, the comprehensive load factor is calculated based on the resource usage proportion of the physical node and the weight possessed by the different characteristic data.
The openstack-based dynamic load adjustment system in one aspect of the present invention is characterized by including:
the data acquisition module is used for acquiring characteristic data of the load serving as the monitored object in real time; and
and the load adjusting module is used for performing abnormal load prediction on the characteristic data based on a specified prediction model and performing load balancing processing when abnormal load is predicted.
Optionally, the data acquisition module comprises: a central collector, a subdata collector and a cluster,
the plurality of clusters collect characteristic data of the physical machine and the virtual machine in real time and gather the characteristic data to the sub-data collector, and the sub-data collector further gathers the characteristic data to the central collector.
Optionally, each of the plurality of clusters comprises a plurality of zabbix monitor nodes, each of the plurality of sub data collectors comprises a plurality of CUM nodes,
the method comprises the steps of building zabbix monitoring nodes, accessing monitored objects to the zabbix monitoring nodes, building CUM nodes to collect characteristic data obtained by all the zabbix monitoring nodes, and collecting the characteristic data collected by the CUM nodes to the central collector.
Optionally, the plurality of zabbix monitoring nodes are set by using distributed nodes.
Optionally, different zabbix monitoring nodes are set for different servers, and different CUM nodes are set for different zabbix monitoring node clusters.
Optionally, the zabbix monitoring node acquires one or more of CPU occupancy, memory occupancy, disk occupancy, and network traffic of the physical machine and the virtual machine in real time as feature data of the monitored object.
Optionally, the central collector normalizes all the obtained feature data to map to an order of magnitude and eliminate singular samples.
Optionally, further comprising:
and the data weft descending module is used for carrying out data weft descending on the characteristic data by adopting a PCA algorithm.
Optionally, in the load adjustment module, the abnormal load prediction is performed on the feature data after the data weft reduction is performed based on a composite prediction model.
Optionally, the composite prediction model is an SVM model and a random forest model, respectively.
Optionally, the composite prediction model is a k-means model and a decision tree model, respectively.
Optionally, in the load adjustment module, when an abnormal load is predicted, a virtual machine load sequence with a predicted load abnormality is obtained by using a bubble permutation method, a comprehensive load coefficient is calculated for physical nodes without a predicted abnormality and is ranked by using the bubble permutation method, a physical node with a highest matching degree is selected from the physical nodes without a predicted abnormality and is calculated as a load variation coefficient, and whether to execute virtual machine live migration is determined based on a value of the calculated load variation coefficient.
The computer-readable medium of the present invention has stored thereon a computer program which, when executed by a processor, implements the openstack-based dynamic load adjustment method as set forth in the claims.
The computer device of the present invention includes a storage module, a processor, and a computer program stored on the storage module and executable on the processor, and is characterized in that the processor implements the openstack-based dynamic load adjustment method when executing the computer program.
As described above, according to the Openstack-based dynamic load adjustment method and the Openstack-based dynamic load adjustment system of the present invention, the problem of insufficient Openstack dynamic resource scheduling in the prior art can be solved, resources are load-balanced by constructing a load regulator, dynamic migration scheduling can be performed on a virtual machine according to the load condition of an actually operating node, and the situation that some nodes may be overloaded during operation and some nodes are idle with unbalanced load can be prevented, so that resource utilization is improved.
Drawings
Fig. 1 is a schematic flowchart illustrating a method for adjusting a dynamic load based on openstack according to the present invention.
Fig. 2 is a block diagram showing a configuration of an openstack-based dynamic load adjustment system according to the present invention.
Fig. 3 is a flowchart illustrating a method for adjusting dynamic load based on openstack according to an embodiment of the present invention.
FIG. 4 is a schematic diagram illustrating one example of implementing feature data acquisition.
Fig. 5 shows the specific steps of resource scheduling.
FIG. 6 is a diagram illustrating a virtual machine load sequence.
Fig. 7 is a diagram showing a load sequence of a physical node.
Fig. 8 is a block diagram showing a configuration of an openstack-based dynamic load adjustment system according to an embodiment of the present invention.
Detailed Description
The following description is of some of the several embodiments of the invention and is intended to provide a basic understanding of the invention. It is not intended to identify key or critical elements of the invention or to delineate the scope of the invention.
For the purposes of brevity and explanation, the principles of the present invention are described herein with reference primarily to exemplary embodiments thereof. However, those skilled in the art will readily recognize that the same principles are equally applicable to all types of openstack-based dynamic load adjustment methods and openstack-based dynamic load adjustment systems, and that these same principles, as well as any such variations, may be implemented therein without departing from the true spirit and scope of the present patent application.
Moreover, in the following description, reference is made to the accompanying drawings that illustrate certain exemplary embodiments. Electrical, mechanical, logical, and structural changes may be made to these embodiments without departing from the spirit and scope of the invention. In addition, while a feature of the invention may have been disclosed with respect to only one of several implementations/embodiments, such feature may be combined with one or more other features of the other implementations/embodiments as may be desired and/or advantageous for any given or identified function. The following description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims and their equivalents.
Terms such as "comprising" and "comprises" mean that, in addition to having elements (modules) and modules directly and explicitly stated in the description and claims, the solution of the invention does not exclude the presence of other elements (modules) and modules not directly or explicitly stated.
Fig. 1 is a schematic flowchart illustrating a method for adjusting a dynamic load based on openstack according to the present invention.
As shown in fig. 1, the openstack-based dynamic load adjustment method of the present invention includes:
a collection step S100: acquiring characteristic data of a load serving as a monitoring object in real time;
weft lowering step S200: performing data weft reduction on the characteristic data by adopting a PCA algorithm;
prediction step S300: performing abnormal load prediction on the characteristic data based on a specified prediction model;
scheduling step S400: when the abnormal load is predicted in the prediction step S300, load balancing processing is performed.
Wherein the step S200 of lowering the weft is an optional step.
In the acquisition step S100, characteristic data of the physical machine and the virtual machine are acquired in real time based on the zabbix monitoring node. Specifically, the acquisition step S200 includes:
building a zabbix monitoring node;
accessing a monitored object to the zabbix monitoring node; and
and establishing a CUM node to summarize the characteristic data obtained by all zabbix monitoring nodes.
Here, the zabbix monitoring nodes are set up using distributed nodes. Moreover, it is preferable that different zabbix monitoring nodes are set for different servers, and different CUM nodes are set for different zabbix monitoring node clusters.
As the acquired feature data, there are, for example, in general: one or more of CPU occupancy rates, memory occupancy rates, disk occupancy rates, and network traffic of the physical machine and the virtual machine.
The acquisition step S100 further includes: all the obtained feature data are normalized so as to map to an order of magnitude and eliminate singular samples.
The prediction step S300 includes:
respectively carrying out model training on the first prediction model and the second prediction model by adopting data after data weft reduction;
determining the weight of two prediction models in the composite prediction model by comparing specified characteristics of the two prediction models;
building a composite prediction model based on the weights; and
and performing abnormal load prediction on the characteristic data based on the constructed composite prediction model.
As an example, the first predictive model and the second predictive model are SVM and random forest, respectively.
The scheduling step S400 includes:
acquiring a virtual machine load sequence with a predicted load abnormality by using a bubble array method when an abnormal load is predicted in the predicting step S300;
calculating comprehensive load coefficients of physical nodes with no predicted abnormality and sequencing the physical nodes by using a bubble arrangement method;
selecting physical nodes with the highest matching degree from the physical nodes with the unpredicted abnormality and calculating a load variation coefficient; and
and judging whether to execute virtual machine migration or not based on the calculated value of the load variation coefficient.
Fig. 2 is a block diagram showing a configuration of an openstack-based dynamic load adjustment system according to the present invention.
As shown in fig. 2, the openstack-based dynamic load adjustment system of the present invention includes:
a data acquisition module 100 which acquires characteristic data of a load as a monitoring object in real time;
the data weft descending module 200 is used for performing data weft descending on the characteristic data by adopting a PCA algorithm; and
and the load adjusting module 300 is used for performing abnormal load prediction on the characteristic data based on a specified prediction model and performing load balancing processing when abnormal load is predicted.
The data weft descending module 200 is an optional module.
In the load adjusting module 300, the data after data weft reduction is adopted to respectively perform model training on a first prediction model and a second prediction model, the specified characteristics of the two prediction models are compared to determine the weights of the two prediction models in the composite prediction model, the composite prediction model is constructed based on the weights, and the abnormal load prediction is performed on the characteristic data based on the constructed composite prediction model.
As an example, the first and second predictive models are SVM models and random forest films, respectively.
In the load adjusting module 300, when an abnormal load is predicted, a virtual machine load sequence with a predicted load abnormality is obtained by using a bubble permutation method, comprehensive load coefficients are calculated for physical nodes without a predicted abnormality and are ranked by using the bubble permutation method, a physical node with the highest matching degree is selected from the physical nodes without a predicted abnormality and is calculated with a load variation coefficient, and whether virtual machine live migration is executed or not is judged based on the calculated value of the load variation coefficient.
Next, a dynamic load adjustment method by an openstack and a dynamic load adjustment system by an openstack according to an embodiment of the present invention will be described.
Fig. 3 is a flowchart illustrating a method for adjusting dynamic load based on openstack according to an embodiment of the present invention.
As shown in fig. 3, at step S1, feature data of the load is extracted (i.e., collected);
in step S2, data weft reduction is performed on the characteristic data of the load;
in step S3, training a composite detection model;
in step S4, data acquisition is performed in real time;
in step S5, performing abnormal load prediction on the collected feature data by using the trained composite prediction model, determining whether the load state is abnormal, and if so, proceeding to step S6;
in step S6, an alarm is issued;
in step S7, resource scheduling is performed after the alarm is received;
in step S8, it is determined whether or not it is necessary to reallocate resources, and if yes, the resources are reallocated and the flow returns to step S5, and if no, the flow ends.
Next, the details of each step will be described.
First, the extraction of the feature data of the load in step S1 will be described.
As an example in the present application, the information mainly collected is state information of CPUs, memories, disks, and networks of the physical machines and the virtual machines. Monitoring a server and a virtual machine by building a zabbix monitoring node; after the monitored object is accessed to the zabbix monitoring node, the zabbix monitoring node can collect real-time load data of the object; and then establishing a CUM node and summarizing monitoring data of all zabbix nodes by using interfaces of the zabbix monitoring nodes. For example, FIG. 4 shows a schematic diagram that is one example of implementing feature data acquisition.
As shown in fig. 4, a plurality of zabbix monitor nodes form a plurality of clusters, which are shown as cluster 1 to cluster n in fig. 4, a plurality of zabbix monitor nodes in cluster 1 are gathered to CUM node (1) of the sub-data collector, a plurality of zabbix monitor nodes in cluster 2 are gathered to CUM node (2) of the sub-data collector, a plurality of zabbix monitor nodes in cluster n … … are gathered to CUM node (n) of the sub-data collector, and CUM node (1) of the sub-data collector to CUM node (n) of the sub-data collector are gathered to the central collector, thereby forming the data collector to be mentioned later.
Here, zabbix itself does not occupy too many resources of the system, and basically does not affect the load condition of the platform. In addition, the problem that the pressure of the central node is too high due to too many monitoring objects in the data acquisition and summarization process of the central collector is not avoided, a distributed node monitoring mode is adopted in the invention, monitoring nodes zabbix are arranged aiming at different servers, and CUM nodes are arranged for different clusters, so that the service pressure of the central collector is dispersed, the network bandwidth from the central acquisition server to each area is solved, and although the defect is that the data is slightly delayed and increased, the increase of the data delay is worthy of being compared with the increase of the whole benefit.
By using the data acquisition mode shown in fig. 4, the historical operating data of the physical machine and the virtual machine is selected, and eight load characteristics extracted from the historical operating data are as follows: { { physical machine: CPU occupancy, memory occupancy, disk occupancy, network traffic }, { virtual machine: CPU occupancy rate, memory occupancy rate, disk occupancy rate and network flow rate } }.
Because the different representative meanings and the differences of the values of different characteristic data are different, in order to eliminate the influence of the values of different characteristics on the prediction, the invention can also additionally arrange a data standardization process to map all the data to an order of magnitude and eliminate the singularity sample, and the adopted standardization method comprises the following steps:
as an example, the characteristic data is normalized using the following formula.
Wherein the normalization process is on yjJ e (1, 2, 3.. n) is transformed to generate a new sequence xjAnd j belongs to (1, 2, 3.. n) with the variance of 1 and the mean value of 0, so that the value of the characteristic data is mapped to [0,1]On the interval, where j represents the characteristic data of the monitored object, yiCorresponding to the actual value of the characteristic data represented by j, for example, j can represent characteristic CPU occupancy rate, memory occupancy rate, disk occupancy rate, network traffic and the like, and xjTo correspond to y after standardizationiAnd (4) obtaining the scaled value.
In step S2, data reduction is performed on the feature data of the load, and as an example, data dimension reduction is performed on the constructed feature data through a PCA algorithm. This reduces the time cost for machine learning, improves the computation efficiency, and reduces the loss rate of feature information.
Next, training of the composite prediction model is performed.
In the invention, a multi-algorithm composite prediction model under different weights is set, and the model is updated by periodically using new data, so that the adaptability of the model is improved, and the problem that the traditional single algorithm model is easily influenced by the specificity of the data and the characteristics of the algorithm can be avoided. Here, N prediction models may be used as the composite prediction model, where N is a natural number of 2 or more.
As an example, when N is 2, an SVM model and a random forest model are selected as the underlying algorithm. SVM (support Vector machines) refers to a support Vector machine, one of the most robust and accurate methods in all known data mining algorithms, belongs to a binary classification algorithm, and can support linear and nonlinear classification. The SVM model is a generalized linear classifier for binary classification of data in a supervised learning mode, and a decision boundary of the SVM model is a maximum margin hyperplane for solving learning samples. The random forest model is a classifier which trains and predicts a sample by using a plurality of trees. In machine learning, a random forest is a classifier that contains multiple decision trees, and the class of its output is determined by the mode of the class output by the individual trees.
The SVM model has strong generalization capability, is not easy to generate overfitting condition, can process noise data, has low speed on large-scale data and is sensitive to missing data and kernel functions; the random forest model can process large-scale data at a high training speed, can process missing data with high data adaptability, and is easy to generate overfitting condition when processing noisy data. The two algorithm models are combined to complement a single algorithm, so that the accuracy of the model for data prediction is improved.
The information entropy calculation method used by the random forest model is as follows:
the information gain calculation method is as follows:
the SVM model adopts a linear kernel function formula as follows:
f(i，j)＝iTj+c
wherein A represents the object of information entropy to be calculated, which refers to the sample set, Ent (A) represents the information entropy of A,
k represents characteristic data (namely CPU occupancy rate, memory occupancy rate, disk occupancy rate, network traffic and the like),
pkrepresenting the probability of the characteristic k being at the value of Morse, or at a certain segment,
d represents a set of samples and a set of samples,
a is the basis of dividing the sample set, the sample division is carried out according to the set a,
ent (A) the entropy of the information before partitioning,
i is a classification vector, namely the value of the characteristic data in the training set,
j is a classification mark and is a mark of the classification,
c is a constant.
Then, a composite prediction model result is constructed based on the weights according to the following formula:
result＝Weightsa*model(a)+(1-Weightsa)model(b)
wherein, model(a)Is the probability that the random forest model predicts as an abnormal state, model(b)Is the probability that the SVM model predicts as an anomaly, WeightsaPerforming model evaluation on the trained random forest model and SVM model through a test data set to obtain specified characteristics of the random forest model and SVM model, and comparing the specified characteristics of the two models to determine Weights of the composite prediction modela。
Also, as another example, when N ═ 2, a k-means model and a decision tree model may be employed. The K-means model is a K-means clustering algorithm (K-means clustering algorithm), which is an iterative solution clustering analysis algorithm and comprises the steps of dividing data into K groups in advance, randomly selecting K objects as initial clustering centers, calculating the distance between each object and each seed clustering center, allocating each object to the nearest clustering center, wherein the clustering centers and the objects allocated to the clustering centers represent a cluster, allocating a sample per cluster, and recalculating the clustering centers of the clusters according to the existing objects in the clusters. The decision tree model is a simple and easy-to-use nonparametric classifier, is a tree diagram formed by decision points, strategy points (event points) and results, is generally applied to sequence decision, usually takes maximum expected value of profit or minimum expected cost as a decision criterion, solves benefit values of various schemes under different conditions in a graphical mode, and then makes a decision through comparison.
As described above, in the openstack-based dynamic load adjustment method according to the present embodiment, the characteristic data such as CPU occupancy, memory occupancy, disk occupancy, network traffic, and the like of the physical machine and the virtual machine is extracted from the historical load characteristic data, and the positive sample and the white sample are marked, respectively, to divide the training set and the test set.
In order to avoid resource consumption caused by excessive data volume, data reduction needs to be performed on original feature data, for example, a PCA algorithm is selected to perform data dimension reduction on constructed feature data, so that repeated features are eliminated and feature characteristics are maximally retained. And respectively carrying out model training on the random forest model and the SVM model by using the training data set after dimensionality reduction or adopting other models (such as a k-means model, a decision tree model and the like), obtaining the accuracy, the precision, the recall rate, the false alarm rate and an ROC curve of the obtained model by testing the data set, and comparing the five characteristics of the two prediction models to determine the prediction weight occupied by the two prediction models in the composite prediction model. And setting a time T value, acquiring data of the cloud platform by the data acquisition unit according to the T value, performing load prediction by using a composite prediction model after dimension reduction by the same PCA, outputting the marked abnormal virtual machine abnormally by the system after predicting the abnormality, and performing load balancing treatment, namely resource scheduling.
Next, a specific case of resource scheduling will be described.
And performing abnormal load prediction on the acquired characteristic data by using the composite prediction model obtained by training, and sending an alarm and performing resource scheduling under the condition that the abnormal load is predicted. Fig. 5 shows the specific steps of resource scheduling. The specific steps of resource scheduling are shown in fig. 5.
As shown in fig. 5, in step a1, a virtual machine load sequence with a predicted load abnormality is obtained;
in step a2, calculating and sequencing comprehensive load coefficients of the physical nodes with no predicted abnormality;
in step a3, selecting the physical node with the highest matching degree from the physical nodes with no predicted abnormality and calculating the load variation coefficient;
at step a4, verifying feasibility;
in step a5, it is determined whether or not the value of the load variation coefficient is negative, and if the determination result is yes, it is determined whether or not virtual machine migration is to be executed;
in step a6, it is determined whether all the abnormal virtual machines complete the processing, if all the abnormal virtual machines complete the processing, the process is ended, otherwise, the process returns to step a 1.
The steps of fig. 5 will be specifically described next.
In the invention, after abnormal load prediction is carried out on load characteristics by adopting a composite prediction model, corresponding load abnormality needs to be solved, a resource scheduling scheme is designed according to the core idea of load balancing, the load of each node finally reaches a dynamic balance value, and the following load weighing method is designed according to the load characteristics, wherein the comprehensive load coefficient CL of the comprehensive load coefficient is calculated based on the resource utilization proportion of the physical nodes and the weights owned by different characteristic datanodeThe method comprises the following steps:
first, compute physical node R (x)nodeThe resource usage of (2) is as follows:
R(x)node＝∑use(x)m。
secondly, calculating the total resource amount T (x) of the physical nodenodeThe following formula:
T(x)node＝∑total(x)m。
thus, the resource utilization ratio f (m) of the physical nodenodeThe following formula:
physical node comprehensive load coefficient CLnodeThe calculation method of (2) is as follows:
CLnode＝∑Weightsm×f(m)
＝Weightscpu×f(cpu)+Weightsram×f(ram)+Weightsdisk×f(disk)+WeightsNetwork×f(Network)
the load unbalance coefficient NBCL is represented by the following equation:
the load variation coefficient Changes is as follows:
wherein, T (X) a is the total resource amount of the physical node a, the node a may contain a plurality of physical machine resources, the total resource amount of a needs to be accumulated to contain each physical machine resource, Use represents the resource consumed by one of the physical machines, T (X) b is the total resource amount of the physical node b, Use (x)mRepresents the consumption of x servers on node on m resources, total (x)mThe total amount of m resources owned by a server on a node is represented, m represents load characteristics, and CPU, ram, disk and Network respectively represent the CPU occupancy rate, the memory occupancy rate, the disk occupancy rate and the Network traffic of a physical node.
CLnThe comprehensive load coefficient of all the nodes n is expressed, and the calculation mode is specifically shown in the CLnodeIs calculated by the formula CLnodeIs the comprehensive load factor of the node. f (m) is the consumption proportion of m resources at a certain node; weightsmRepresenting the weight coefficients possessed by different m load characteristics, different load weight coefficients representing eachThe magnitude of the impact of the load information on the overall load, WeightsmThe larger the value is, the larger the influence of the m resource on the load is; finally determining the comprehensive load coefficient CL of the physical nodenode(ii) a NBCL represents the load condition of the entire system; changes are used to measure the load change after thermal migration, when Changes are negative, the load after migration is reduced, when Changes are 0, the load after migration is unchanged, and when Changes are positive, the load after migration is increased.
After receiving the load abnormality alarm of the composite prediction model, collecting the virtual machines with the predicted abnormality, performing load comparison on the collected virtual machines, and finding the virtual machine with a severe load through a virtual machine load sequence constructed by a bubble sorting method, as shown in fig. 6. FIG. 6 is a diagram illustrating a virtual machine load sequence. In fig. 6, the virtual machines 1, 2, 3, and … … are arranged in the load reduction direction, and the most heavily loaded virtual machine can be found most quickly by the pointer Max, and the load can be reduced to the lowest quickly by the Max iteration each time.
On the other hand, normal physical nodes are screened according to the abnormal alarm information, and sorting is performed by using a bubble sorting algorithm based on the value of the comprehensive load coefficient, so that a load sequence of the physical nodes from small to large according to the load is obtained, as shown in fig. 7. Fig. 7 is a diagram showing a load sequence of a physical node. In fig. 7, nodes 1, 2, 3, and … … are arranged in the direction of increasing load. The pointer await points to the node that is best suited for migration to.
In step a5, before performing the virtual machine live migration operation, the load variation coefficients change are calculated by using the above formula, and if the change coefficient is negative, it indicates that the virtual machine migration is feasible, otherwise, the await pointer points to the next physical node for determination.
After the virtual machine is subjected to hot migration once, the Max pointer can find the next virtual machine with the most serious load through a bubbling algorithm, and meanwhile, the load sequence of the physical nodes can be regenerated due to the fact that the loads of different physical nodes are changed.
As described above, in the case where the load abnormality is detectedThen, the abnormal virtual machine is subjected to thermal migration to reduce the load of the corresponding physical node, according to CLvmThe calculation result uses a bubble algorithm (CL)vmThe combined load factor representing vm virtual machine, its calculation and CLnodeSimilarly, refer to CLnodeThe calculation formula) of the virtual machine, constructing a virtual machine load sequence, and preferentially arranging the virtual machine on the physical node with higher load pressure to perform thermal migration; according to CLnodeConstructing a physical machine node load sequence by using bubble sorting according to a calculation result, finding a physical node marked pre-migration node which has no abnormal load and has the lowest load, calculating load variation coefficients Changes by using a formula, and if the calculation result is a negative value, indicating that the load balance can be realized by the migration; otherwise, canceling the mark, searching the next selected physical node again until the live migration is completed, and after the first live migration is completed, repeating the above operations to continue the live migration until all the abnormal virtual machines are processed completely if the abnormal virtual machine list is not empty.
Fig. 8 is a block diagram showing a configuration of an openstack-based dynamic load adjustment system according to an embodiment of the present invention.
As shown in fig. 8, the openstack-based dynamic load adjustment system according to an embodiment of the present invention includes:
a data collector 10 for collecting characteristic data of a load as a monitoring object in real time; and
the load regulator 20 performs abnormal load prediction on the feature data based on a predetermined prediction model, and performs load balancing processing when an abnormal load is predicted.
The structure of data collector 10 can be referred to the structure shown in fig. 4. The data acquisition device 10 adopts a multi-layer data acquisition structure to acquire the load condition of the operation of the bearing node in real time, compared with a single data acquisition point in the prior art, the stability and the pressure resistance are stronger, and moreover, the consumption of resources by the used zabbix server is relatively lower, and the load of a platform can hardly be increased.
The load regulator 20 is a regulator for implementing automated load balancing in response to the problem of insufficient Openstack dynamic resource scheduling.
The load regulator 20 includes:
the prediction submodule 21 is used for monitoring the cloud platform operation data by constructing a composite prediction model through analysis on historical data, and finding out or predicting load abnormal points in advance; and
and the load regulation submodule 22 is used for constructing a load measurement formula according to the structural characteristics of the node resources, calculating a load variation coefficient and a load imbalance coefficient, finding an optimal resource scheduling scheme and processing the load abnormal node.
In this way, dynamic balancing of resources is ultimately achieved with the load regulator 20 to improve resource utilization.
In the prediction sub-module 21, the first prediction model and the second prediction model are respectively subjected to model training by using feature data, the weights of the two prediction models in the composite prediction model are determined by comparing the specified features of the two prediction models, the composite prediction model is constructed based on the weights, and the abnormal load prediction is performed on the feature data based on the constructed composite prediction model. The process of constructing the composite prediction module can refer to the above description.
In the load adjustment submodule 22, when an abnormal load is predicted, a virtual machine load sequence with a predicted load abnormality is obtained by using a bubble permutation method, comprehensive load coefficients are calculated for physical nodes without a predicted abnormality and are sorted by using the bubble permutation method, a physical node with the highest matching degree is selected from the physical nodes without a predicted abnormality and is calculated with a load variation coefficient, and whether virtual machine live migration is executed or not is judged based on the calculated value of the load variation coefficient. The specific load adjustment process can refer to the above description.
As described above, in the prediction sub-module 21, for example, the random forest model and the SVM model are used as the bottom-layer algorithm to train and construct the composite prediction model, compared with the conventional single algorithm multi-algorithm, the defect of the algorithm itself can be made up, data with stronger adaptability can be predicted accurately, load state monitoring and prediction alarm can be performed on the cloud bearing node in real time, and the passive condition that the conventional load abnormality occurs first and then is solved is changed.When the load of the predicted node is abnormal, the load adjustment submodule 22 will further analyze the data collector 10 and the data processed by the composite prediction model, and use CLnodeAnd analyzing the load condition of each node by parameters such as Changes, constructing a scheduling sequence, finding out a corresponding service transfer node through a normal load sequence and an abnormal load sequence, verifying the feasibility before transferring, ensuring the implementation stability, and sequencing the sequences again after transferring until all the nodes reach load balance.
The present invention provides a computer readable medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the openstack-based dynamic load adjustment method described above.
The invention further provides a computer device, which comprises a storage module, a processor and a computer program stored on the storage module and capable of running on the processor, wherein the processor implements the openstack-based dynamic load adjustment method when executing the computer program.
The above examples mainly illustrate the openstack-based dynamic load adjustment method and the openstack-based dynamic load adjustment system according to the present invention. Although only a few embodiments of the present invention have been described in detail, those skilled in the art will appreciate that the present invention may be embodied in many other forms without departing from the spirit or scope thereof. Accordingly, the present examples and embodiments are to be considered as illustrative and not restrictive, and various modifications and substitutions may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims.
Claims (28)
1. An openstack-based dynamic load adjustment method is characterized by comprising the following steps:
acquiring characteristic data of a physical machine and a virtual machine serving as a monitored object in real time based on a zabbix monitoring node;
a prediction step of predicting an abnormal load of the feature data based on a predetermined prediction model; and
and a scheduling step of performing load balancing processing when the abnormal load is predicted in the predicting step.
2. The openstack-based dynamic load adjustment method of claim 1,
the step of collecting comprises:
building a zabbix monitoring node;
accessing a monitored object to the zabbix monitoring node; and
and establishing a CUM node to summarize the characteristic data obtained by all zabbix monitoring nodes.
3. The openstack-based dynamic load adjustment method of claim 2,
in the acquisition step, distributed nodes are adopted to set the zabbix monitoring nodes.
4. The openstack-based dynamic load adjustment method of claim 2,
in the acquisition step, different zabbix monitoring nodes are set for different servers, and different CUM nodes are set for different zabbix monitoring node clusters.
5. The openstack-based dynamic load adjustment method of claim 1,
in the acquisition step, one or more items of CPU occupancy rate, memory occupancy rate, disk occupancy rate and network flow of the physical machine and the virtual machine are acquired in real time to serve as characteristic data of the monitored object.
6. The openstack-based dynamic load adjustment method of claim 1,
further comprising in the step of acquiring: all the obtained feature data are normalized so as to be mapped to an order of magnitude to eliminate singular samples.
7. The openstack-based dynamic load adjustment method of claim 6,
the feature data is normalized using the following formula,
wherein the normalization process is on yjJ e (1, 2, 3.. 4) to generate a new sequence xjAnd j belongs to (1, 2, 3.. 4), the variance is 1, the mean value is 0, and the value of the characteristic data is mapped to [0,1]On the interval of the time interval, the time interval is,
wherein j represents the characteristic data of the monitored object, yiFor the actual value of the feature data corresponding to j,
xito correspond to y after standardizationjAnd (4) obtaining the scaled value.
8. The openstack-based dynamic load adjustment method according to any one of claims 1 to 7, further comprising, between the acquiring step and the training step:
and a weft descending step, namely performing data weft descending on the characteristic data by adopting a specified algorithm.
9. The openstack-based dynamic load adjustment method of claim 8,
in the predicting step, abnormal load prediction is performed on the feature data subjected to data weft reduction based on a composite prediction model.
10. The openstack-based dynamic load adjustment method of claim 9,
the composite predictive model includes N predictive models,
the predicting step includes:
respectively carrying out model training on the N prediction models by adopting data after data weft reduction;
determining a weight for each prediction model in the composite prediction model against prescribed features of the N prediction models;
building a composite prediction model based on the weights; and
performing abnormal load prediction on the characteristic data based on the constructed composite prediction model,
wherein N is a natural number of 2 or more.
11. The openstack-based dynamic load adjustment method of claim 10,
and when the N is equal to 2, the 2 prediction models are respectively an SVM model and a random forest model.
12. The openstack-based dynamic load adjustment method of claim 10,
and when the N is equal to 2, wherein the 2 prediction models are a k-means model and a decision tree model respectively.
13. The openstack-based dynamic load adjustment method of claim 9, wherein the scheduling step comprises:
under the condition that the abnormal load is predicted in the predicting step, acquiring a virtual machine load sequence with the predicted abnormal load by using a bubble array method;
calculating comprehensive load coefficients of physical nodes with no predicted abnormality and sequencing the physical nodes by using a bubble arrangement method;
selecting physical nodes with the highest matching degree from the physical nodes with the unpredicted abnormality and calculating a load variation coefficient; and
and judging whether to execute virtual machine migration or not based on the calculated value of the load variation coefficient.
14. The openstack-based dynamic load adjustment method of claim 13,
and calculating the comprehensive load coefficient based on the resource utilization ratio of the physical node and the weight possessed by different characteristic data.
15. An openstack-based dynamic load adjustment system, comprising:
the data acquisition module is used for acquiring characteristic data of the load serving as the monitored object in real time; and
and the load adjusting module is used for performing abnormal load prediction on the characteristic data based on a specified prediction model and performing load balancing processing when abnormal load is predicted.
16. The openstack-based dynamic load adjustment system of claim 15,
the data acquisition module comprises: a central collector, a subdata collector and a cluster,
the plurality of clusters collect characteristic data of the physical machine and the virtual machine in real time and gather the characteristic data to the sub-data collector, and the sub-data collector further gathers the characteristic data to the central collector.
17. The openstack-based dynamic load adjustment system of claim 16,
each of the plurality of clusters includes a plurality of zabbix monitoring nodes,
each of the plurality of sub data collectors includes a plurality of CUM nodes,
the method comprises the steps of building zabbix monitoring nodes, accessing monitored objects to the zabbix monitoring nodes, building CUM nodes to collect characteristic data obtained by all the zabbix monitoring nodes, and collecting the characteristic data collected by the CUM nodes to the central collector.
18. The openstack-based dynamic load adjustment system of claim 17,
and setting the plurality of zabbix monitoring nodes by adopting distributed nodes.
19. The openstack-based dynamic load adjustment system of claim 18,
different zabbix monitoring nodes are set for different servers, and different CUM nodes are set for different zabbix monitoring node clusters.
20. The openstack-based dynamic load adjustment system of claim 19,
the zabbix monitoring node acquires one or more of CPU occupancy rate, memory occupancy rate, disk occupancy rate and network flow of the physical machine and the virtual machine in real time as characteristic data of the monitored object.
21. The openstack-based dynamic load adjustment system of claim 20,
the central collector normalizes all the acquired feature data so as to map to an order of magnitude and eliminate singular samples.
22. The openstack-based dynamic load adjustment system of any one of claims 15-21, further comprising:
and the data weft descending module is used for carrying out data weft descending on the characteristic data by adopting a PCA algorithm.
23. The openstack-based dynamic load adjustment system of claim 22,
and in the load adjusting module, performing abnormal load prediction on the characteristic data subjected to data weft reduction based on a composite prediction model.
24. The openstack-based dynamic load adjustment system of claim 23,
the composite prediction model is respectively an SVM model and a random forest model.
25. The openstack-based dynamic load adjustment system of claim 23,
the composite prediction models are respectively a k-means model and a decision tree model.
26. The openstack-based dynamic load adjustment system of claim 23,
in the load adjusting module, under the condition that abnormal loads are predicted, a virtual machine load sequence with the predicted load abnormality is obtained by using a bubble permutation method, comprehensive load coefficients are calculated for physical nodes without the predicted abnormality and are ranked by using the bubble permutation method, the physical node with the highest matching degree is selected from the physical nodes without the predicted abnormality and is used for calculating a load variation coefficient, and whether the virtual machine thermal migration is executed or not is judged based on the value of the calculated load variation coefficient.
27. A computer-readable medium, having stored thereon a computer program,
the computer program is executed by a processor to realize the openstack-based dynamic load adjustment method according to any one of claims 1 to 14.
28. A computer device comprising a storage module, a processor and a computer program stored on the storage module and executable on the processor, wherein the processor executes the computer program to implement the openstack-based dynamic load adjustment method according to any one of claims 1 to 14.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010896885.3A CN112527448A (en) | 2020-08-31 | 2020-08-31 | Openstack-based dynamic load adjustment method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010896885.3A CN112527448A (en) | 2020-08-31 | 2020-08-31 | Openstack-based dynamic load adjustment method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112527448A true CN112527448A (en) | 2021-03-19 |
Family
ID=74978813
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010896885.3A Pending CN112527448A (en) | 2020-08-31 | 2020-08-31 | Openstack-based dynamic load adjustment method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112527448A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113051131A (en) * | 2021-03-23 | 2021-06-29 | 北京沃东天骏信息技术有限公司 | Acquisition terminal, management control platform, and Prometheus service adjusting method and system |
| CN113157440A (en) * | 2021-03-23 | 2021-07-23 | 北京云上曲率科技有限公司 | Self-adaptive load balancing and high availability guaranteeing method applied to mobile terminal |
| CN114265671A (en) * | 2022-03-03 | 2022-04-01 | 浙江省邮电工程建设有限公司 | Hybrid expansion method of virtual machine room |
| CN117421131A (en) * | 2023-12-18 | 2024-01-19 | 武汉泽塔云科技股份有限公司 | Intelligent scheduling method and system for monitoring power consumption load of server |
-
2020
- 2020-08-31 CN CN202010896885.3A patent/CN112527448A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113051131A (en) * | 2021-03-23 | 2021-06-29 | 北京沃东天骏信息技术有限公司 | Acquisition terminal, management control platform, and Prometheus service adjusting method and system |
| CN113157440A (en) * | 2021-03-23 | 2021-07-23 | 北京云上曲率科技有限公司 | Self-adaptive load balancing and high availability guaranteeing method applied to mobile terminal |
| CN113157440B (en) * | 2021-03-23 | 2023-06-27 | 北京云上曲率科技有限公司 | Self-adaptive load balancing and high availability ensuring method applied to mobile terminal |
| CN114265671A (en) * | 2022-03-03 | 2022-04-01 | 浙江省邮电工程建设有限公司 | Hybrid expansion method of virtual machine room |
| CN117421131A (en) * | 2023-12-18 | 2024-01-19 | 武汉泽塔云科技股份有限公司 | Intelligent scheduling method and system for monitoring power consumption load of server |
| CN117421131B (en) * | 2023-12-18 | 2024-03-26 | 武汉泽塔云科技股份有限公司 | Intelligent scheduling method and system for monitoring power consumption load of server |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112527448A (en) | Openstack-based dynamic load adjustment method and system | |
| CN106933650B (en) | Load management method and system of cloud application system | |
| CN104978236B (en) | HDFS load source destination node choosing methods based on more measurement indexs | |
| CN111459617B (en) | Automatic allocation optimization system and method for containerized application based on cloud platform | |
| Jia et al. | A grid and density-based clustering algorithm for processing data stream | |
| CN110505114B (en) | Method for judging node abnormity in cloud computing environment | |
| Canali et al. | Improving scalability of cloud monitoring through PCA-based clustering of virtual machines | |
| CN107220108B (en) | Method and system for realizing load balance of cloud data center | |
| Shekhawat et al. | Datacenter workload classification and characterization: An empirical approach | |
| CN113177857B (en) | Resource elastic allocation method based on typhoon disaster estimation | |
| CN112835698A (en) | Heterogeneous cluster-based dynamic load balancing method for request classification processing | |
| CN109460301B (en) | Method and system for configuring elastic resources of streaming data load | |
| WO2004053726A2 (en) | Apparatus and methods for classification of web sites | |
| US7461037B2 (en) | Clustering technique for cyclic phenomena | |
| CN116010228B (en) | Time estimation method and device for network security scanning | |
| CN111327480B (en) | Method for monitoring multiple QoS of Web service under mobile edge environment | |
| CN112882805A (en) | Profit optimization scheduling method based on task resource constraint | |
| CN116431346A (en) | Compensation method for main memory capacity of electronic equipment | |
| CN113723514B (en) | Mixed sampling-based secure access log data balance processing method | |
| CN115292303A (en) | Data processing method and device | |
| JP2023101234A (en) | Cloud application deployment apparatus and cloud application deployment method | |
| CN110134575B (en) | Method and device for calculating service capacity of server cluster | |
| CN115242797A (en) | Client load balancing method and system for micro-service architecture | |
| CN112148483A (en) | Container migration method and related device | |
| CN111598390A (en) | Server high availability evaluation method, device, equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |