CN109787855A - Server Load Prediction method and system based on Markov chain and time series models - Google Patents
Server Load Prediction method and system based on Markov chain and time series models Download PDFInfo
- Publication number
- CN109787855A CN109787855A CN201811544236.6A CN201811544236A CN109787855A CN 109787855 A CN109787855 A CN 109787855A CN 201811544236 A CN201811544236 A CN 201811544236A CN 109787855 A CN109787855 A CN 109787855A
- Authority
- CN
- China
- Prior art keywords
- load
- value
- sequence
- time series
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The Server Load Prediction method and system based on Markov chain and time series models that the invention discloses a kind of.This method comprises the following steps: step 1 carries out periodic sampling update using load information of the slip window sampling to server, and generates load value time series；Step 2 establishes ARIMA model according to load value time series, is modified using Markov chain to ARIMA model prediction load error, obtains revised load estimation value；Step 3, each host report revised load estimation value to the scheduler of place server cluster, carry out task distribution for scheduler and provide decision-making foundation.The present invention is that one kind can provide the Server Load Prediction method based on Markov chain and time series models of decision-making foundation for the subsequent rational management for realizing computing resource.
Description
Technical field
The present invention relates to server technology fields, in particular to a kind of can be the subsequent rational management for realizing computing resource
The Server Load Prediction method and system based on Markov chain and time series models of decision-making foundation are provided.
Background technique
With the universal of cloud computing technology and development, more and more mechanisms and enterprise using trustship on internet long-range clothes
Business device network builds cloud platform, in order to store, manage and processing business data.In cloud platform operational process, server
The utilization rate moment of the resources such as CPU and memory is in upper and lower fluctuation status.If server maintains always in high load condition,
The response speed of access request will be impacted；, whereas if server maintains always in low load condition, Jiu Huilang
Take computing resource.In view of the above-mentioned problems, the present invention proposes that a kind of server based on Markov chain and time series models is negative
Prediction technique is carried, provides decision-making foundation for the subsequent rational management for realizing computing resource.
Cloud computing is a kind of provided by way of servicing in terms of the dynamically resource of telescopic virtualization Internet
Calculation mode.Cloud computing service provider integrates a large amount of computing resource and uses for multi-user, and user can be during peak traffic
More resources are easily requested, just can be discharged extra resource again later to peak period.Therefore user does not need because short
The temporary peak traffic phase just buys a large amount of resource, is in the spare time so as to avoid resource in the long-time in addition to the peak traffic phase
State is set, is improved resource utilization, economic cost has been saved.Realize the critical issue of above-mentioned on-demand dynamic dispatching computing resource
Be: scheduling of resource needs the regular hour, if demand sharp increase, during this period of time resource allocation is insufficient；Conversely, if
Demand declines to a great extent, then during this period of time resource is idle.In order to solve this problem, it needs to use and needs a kind of server negative
Prediction technique is carried, by obtaining the predicted value of subsequent time server load to historic load progress modeling analysis, according to
Predicted value deploys computing resource ahead of time, timely responds to resource use demand or recycles idle allocated resource.
Markov chain is in state space by the random process of the conversion from a state to another state.It should
Cross range request and have Markov property: the probability distribution of NextState can only determine by current state, in time series it
The event of front has no truck with.In each step of Markov chain, system can change according to probability distribution from a state
To another state, current state can also be kept.The change of state is called transfer, due to state transfer be it is random,
A possibility that state shifts size must be described with transition probability.If system has n kind state, each state correspondence just has
To the transition probability of n kind state, the transition probability of n kind state in system is once arranged, the transfer of an available n rank is general
Rate matrix.By the original state and transition probability matrix of certain object, can predict that the object is transferred to NextState can
It can property size.
Difference ARMA model (ARIMA (p, d, q)) model, one of time series forecasting analysis method.It will
The data sequence that prediction object is formed over time is considered as a random sequence, carries out d rank if it is non-stationary series
Difference tranquilization, according to recognition rule, with certain mathematical model come this sequence of approximate description.The model mainly includes following
3 kinds: autoregression model AR (p), wherein p is autoregression item；Moving average model(MA model) MA (q), wherein q is rolling average item number；From
It returns moving average model(MA model) ARMA (p, q).This model can from the past value of time series and now once be determined
Value predicts future value.
There are two main classes for the prior art.One kind is single traditional prediction method, such as the method for moving average, exponential smoothing
With gray model etc..Since the load on host computers variation in cloud platform has the characteristics that non-linear, non-stationary and dynamic random, so
Conventional method is not high to the fitting degree of loading trends, and precision of prediction is lower.In addition one kind is based on the pre- of neural network model
Survey method, such method improves precision of prediction compared to traditional prediction method, but neural network model is by sample complexity
It is affected, and its training needs to spend more time, that there are convergence rates is slow, parameter selection is sensitive and is easily trapped into part
The deficiencies of optimal solution.
Summary of the invention
The present invention is directed to overcome the deficiencies of existing technologies, providing a kind of to be the subsequent rational management for realizing computing resource
The Server Load Prediction method based on Markov chain and time series models of decision-making foundation is provided.
To achieve the above object, the invention adopts the following technical scheme: providing a kind of based on Markov chain and time sequence
The Server Load Prediction method of column model, which comprises the steps of:
S1 carries out periodic sampling update using load information of the slip window sampling to server, and generates the load value time
Sequence；
S2 establishes ARIMA model according to load value time series, by ARIMA model computational load predicted value, utilizes horse
Markov's chain is modified ARIMA model prediction load error, obtains revised load estimation value；
S3, each host report revised load estimation value to the scheduler of place server cluster, carry out for scheduler
Task distribution provides decision-making foundation.
Step S1 includes:
S11 acquires a machine every one section of preset sampling period when certain server starting in server cluster
The load information including CPU usage, memory usage, generate length be n load value initial time sequence；
S12 is updated load value time series by slip window sampling, when reaching the sampling period by load value
Earliest load information replaces with freshly harvested load information in time series, and forming new length is n load value time series.
Step S2 includes:
S21, i data determine whether stationary sequence before intercepting load value time series, if non-stationary series, carry out flat
Steadyization pretreatment；
S22 calculates the load estimation at remaining moment as a result, obtaining historical forecast error sequence using ARIMA model；
S23 estimates following prediction error using Markov chain from historical forecast error information, and using this estimation
Error is modified original predictive value, obtains revised load estimation value.
Step S21 includes:
The preceding i data of load value time series are intercepted as inputting, ADF unit root test is used to determine it whether to be flat
Steady sequence is allowed to become stationary sequence by difference if non-stationary series, and the number of difference is ARIMA (p, q, d) at this time
Parameter d, d in model are so that input data sequence is become the difference number of stationary sequence in the step, and p is autoregression item, and q is
Rolling average item number；
Step S22 includes:
ARIMA (p, q, d) model after determining difference number d, can be reduced to ARMA (p, q) model by S221, be model
Identification is prepared, and in order to carry out model identification, is judged using auto-correlation function ACF and partial autocorrelation function PACF value；
S222, ACF the and PACF value obtained by previous step carry out model identification；
S223, using ACF and PACF value tentatively judge parameter p and q on the basis of, by minimum information criterion AIC with
Schwartz criterion SC carries out determining rank, is opposite optimal models when AIC and SC functional value reaches the smallest model；
S224 carries out parameter Estimation using least square method；
S225 carries out hypothesis testing, carries out evaluation fitting, match value and reality using the ARIMA model for having determined that parameter
The sequence that the difference of value is formed forms residual sequence, if probability P > significance value together in the LBQ statistic of residual sequence
0.05, then think that model is acceptable；
S226, using passed through examine model carry out forecast analysis, obtain the load estimation at rear n-i moment as a result,
The load estimation error at n-i moment is calculated, historical forecast error sequence is formed.
Step S23 includes:
S231 handles the wide discrete method of historical forecast error sequence, the codomain of sequence is converted into from-∞ to+∞
The m section with same widths, and carry out state demarcation；
S232: according to state demarcation as a result, calculating and generating state transition probability matrix P；
S233 generates forecast set according to state transition probability matrix P, determines the prediction error initial state distribution of forecast set,
Assuming that n-th of moment prediction error is in state Si, then the distribution of its state can use the row vector e of 1 × m0It indicates, e0=[0 ...
1 ... 0], it is 1 at the i of position, is 0 at remaining；
S234: the then prediction error state distribution at (n+1)th moment are as follows:
e1=e0P=[p1…pi…pm], max (pi) corresponding SiThe shape that as (n+1)th moment is most possibly transferred to
State section, the prediction error at (n+1)th moment for taking the median of the state interval to estimate as Markov chain, e0It is n-th
The prediction error state distribution vector at a moment, e1For the prediction error state distribution vector at (n+1)th moment, p1When being current
It is transferred to state interval S quarter1Probability, piState interval S is transferred to for current timeiProbability, pmIt is transferred to for current time
State interval SmProbability；
S235: load estimation value is corrected according to the estimated value of prediction error, correction formula is
Wherein V (n+1) is the load estimation value at (n+1)th moment, and v is the prediction error that Markov chain estimates (n+1)th moment.
The Server Load Prediction system based on Markov chain and time series models that the present invention also provides a kind of, packet
It includes:
Load value time series generation unit, for periodically being adopted using load information of the slip window sampling to server
Sample updates, and generates load value time series；
Load estimation value amending unit passes through ARIMA model for establishing ARIMA model according to load value time series
Computational load predicted value is modified ARIMA model prediction load error using Markov chain, obtains revised load
Predicted value；
Task distributes decision-making foundation unit, revised negative to the report of the scheduler of place server cluster for each host
Predicted value is carried, task distribution is carried out for scheduler and decision-making foundation is provided.
Load value time series generation unit, comprising:
Load value time series generates subelement, is used for when certain server starting in server cluster, every one section
The preset sampling period acquires the load information including CPU usage, memory usage an of the machine, generates length and is
The load value initial time sequence of n；
Load value time series updating unit is used for when reaching the sampling period, when by slip window sampling to load value
Between sequence be updated, load information earliest in load value time series is replaced with into freshly harvested load information, composition is new
Length be n load value time series；
The load estimation value amending unit, specifically includes:
I data determine whether stationary sequence before smoothing preprocessing unit interception load value time series, if non-flat
Steady sequence carries out smoothing preprocessing historical forecast error sequence generation unit, calculates the negative of remaining moment using ARIMA model
Prediction result is carried, obtains historical forecast error sequence；
Load estimation value revise subelemen, for estimating future from historical forecast error information using Markov chain
It predicts error, and original predictive value is modified using this evaluated error, obtain revised load estimation value.
The load estimation value amending unit, further includes:
The smoothing preprocessing unit, is specifically used for:
The preceding i data of load value time series are intercepted as inputting, ADF unit root test is used to determine it whether to be flat
Steady sequence is allowed to become stationary sequence by difference if non-stationary series, and the number of difference is ARIMA (p, q, d) at this time
Parameter d, d in model are so that input data sequence is become the difference number of stationary sequence in the step, and p is autoregression item, and q is
Rolling average item number；
The historical forecast error sequence generation unit, is specifically specifically used for:
After determining difference number d, ARIMA (p, q, d) model can be reduced to ARMA (p, q) model, identified for model
It prepares, in order to carry out model identification, is judged using auto-correlation function ACF and partial autocorrelation function PACF value；
ACF the and PACF value obtained by previous step carries out model identification；
Using ACF and PACF value on the basis of tentatively judging parameter p and q, pass through minimum information criterion AIC and Schwartz
Criterion SC carries out determining rank, is opposite optimal models when AIC and SC functional value reaches the smallest model；
Parameter Estimation is carried out using least square method；
Hypothesis testing is carried out, evaluation fitting, the difference of match value and actual value are carried out using the ARIMA model for having determined that parameter
The sequence of formation forms residual sequence, if probability P > significance value 0.05 together in the LBQ statistic of residual sequence,
Think that model is acceptable；
Forecast analysis is carried out using the model examined has been passed through, obtains the load estimation at rear n-i moment as a result, calculating n-
The load estimation error at i moment forms historical forecast error sequence.
The load value time series updating unit, is specifically used for:
The wide discrete method of historical forecast error sequence is handled, the codomain of sequence is converted into from-∞ to+∞ with phase
With m section of width, and carry out state demarcation；
According to state demarcation as a result, calculating and generating state transition probability matrix P；
Forecast set is generated according to state transition probability matrix P, determines the prediction error initial state distribution of forecast set, it is assumed that
N-th of moment prediction error is in state Si, then the distribution of its state can use the row vector e of 1 × m0It indicates, e0=[0 ... 1 ...
0], it is 1 at the i of position, is 0 at remaining；
The prediction error state distribution at (n+1)th moment are as follows: e1=e0P=[p1…pi…pm], max (pi) corresponding SiI.e.
For the state interval that (n+1)th moment is most possibly transferred to, the median of the state interval is taken to estimate as Markov chain
(n+1)th moment prediction error, e0For the prediction error state distribution vector at n-th of moment, e1For (n+1)th moment
Predict error state distribution vector, p1State interval S is transferred to for current time1Probability, piState is transferred to for current time
Section SiProbability, pmState interval S is transferred to for current timemProbability；
Load estimation value is corrected according to the estimated value of prediction error, correction formula isWherein V
It (n+1) is the load estimation value at (n+1)th moment, v is the prediction error that Markov chain estimates (n+1)th moment.
The beneficial effects of the present invention are: it is combined using ARIMA model with Markov chain the invention proposes a kind of
Server Load Prediction method, using the prediction result of the predicted error amendment ARIMA model of Markov chain, compared to single
The fitting degree to loading trends is improved for conventional method, improves the precision of load estimation；Compared to neural network model
For, this model modeling information needed is few, and operation is convenient, and prediction the time it takes cost is lower.
Detailed description of the invention
Fig. 1 show the structural block diagram of the Server Load Prediction method based on Markov chain and time series models.
Fig. 2 show the overall process schematic diagram of server load rolling forecast.
Fig. 3 show the server tentative prediction process schematic that load prediction module is based on ARIMA (p, q, d) model.
Fig. 4 show the process schematic that the load estimation value at (n+1)th moment is obtained using Markov chain.
Fig. 5 show the structure chart of the Server Load Prediction system based on Markov chain and time series models.
Fig. 6 show the structure chart of load value time series generation unit.
Fig. 7 show the structure chart of load estimation value amending unit.
Fig. 8 show the structure chart of load estimation value generation unit.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, below in conjunction with attached drawing and specific implementation
Example, the present invention will be described in further detail.It should be appreciated that specific embodiment described herein is only to explain this hair
It is bright, but not to limit the present invention.
There are two main classes for the prior art.One kind is single traditional prediction method, such as the method for moving average, exponential smoothing
With gray model etc..Since the load on host computers variation in cloud platform has the characteristics that non-linear, non-stationary and dynamic random, so
Conventional method is not high to the fitting degree of loading trends, and precision of prediction is lower.In addition one kind is based on the pre- of neural network model
Survey method, such method improves precision of prediction compared to traditional prediction method, but neural network model is by sample complexity
It is affected, and its training needs to spend more time, that there are convergence rates is slow, parameter selection is sensitive and is easily trapped into part
The deficiencies of optimal solution.
The invention proposes a kind of Server Load Prediction method combined using ARIMA model with Markov chain,
Using the prediction result of the predicted error amendment ARIMA model of Markov chain, compared to being improved for single conventional method pair
The fitting degree of loading trends improves the precision of load estimation；For neural network model, letter needed for this model modeling
Breath is few, and operation is convenient, and prediction the time it takes cost is lower.
With reference to Fig. 1, it is negative that the embodiment of the invention discloses a kind of servers based on Markov chain and time series models
Carry prediction technique characterized by comprising
S1 carries out periodic sampling update using load information of the slip window sampling to server, and generates load value time sequence
Column；
S2 establishes ARIMA model according to load value time series, by ARIMA model computational load predicted value, utilizes horse
Markov's chain is modified ARIMA model prediction load error, obtains revised load estimation value；
Each host of S3 reports revised load estimation value to the scheduler of place server cluster, is appointed for scheduler
Business distribution provides decision-making foundation.
Fig. 2 describes the overall process of server load rolling forecast:
Step S1 is specifically included:
When certain server starting in server cluster, load detecting module starts, and the module is preset every one section
Sampling period just acquires the load informations such as the CPU usage an of the machine, memory usage, at the beginning of generating the load value that length is n
Beginning time series；
When reaching the sampling period, it is updated, will be loaded by load value time series of the slip window sampling to server
Earliest load information replaces with freshly harvested load information in value time series, forms the load value time that new length is n
Sequence.
Whenever generating new load value time series, which submits to load prediction module as input data and carries out
New load estimation value, i.e. step S2 are obtained after processing.
Step S2 includes:
S21, i data determine whether stationary sequence before intercepting load value time series, if non-stationary series, carry out flat
Steadyization pretreatment；
S22 calculates the load estimation at remaining moment as a result, obtaining historical forecast error information using ARIMA model；
S23 estimates following prediction error using Markov chain from historical forecast error information, and using this estimation
Error is modified original predictive value, obtains revised load estimation value.
Step S2 is further detailed below in conjunction with Fig. 3 and Fig. 4.
Further, step S3 includes:
The scheduler of cluster where new load estimation value will be sent to the server；
The scheduler of place cluster safeguards that a server node loaded list, scheduler receive the new of server report
When load estimation value, write the values into loaded list；
After scheduler receives the request of user service, selection load is optimal in existing server node loaded list
Server node, and forward requests to the node.
Fig. 3 describes the server tentative prediction process that load prediction module is based on ARIMA (p, q, d) model.
Step S21 includes:
Whether i data determine it using ADF unit root test as input before intercepting the interception of load value time series
Be allowed to become stationary sequence by difference if non-stationary series for stationary sequence, at this time the number of difference be ARIMA (p,
Q, d) parameter d, d in model be so that input data sequence is become the difference number of stationary sequence in the step, p is autoregression
, q is rolling average item number.
Step S22 includes:
After determining difference number d, ARIMA (p, q, d) model can be reduced to ARMA (p, q) model, in order to carry out model
Identification will use auto-correlation function ACF and partial autocorrelation function PACF to be judged:
Calculate YtWith Yt-kACF:Calculate Yt..., Yt-kPACF:YtIndicate the load value of t moment, Yt-kIndicate the load value at a distance of k time interval with t moment,
ρkIndicate YtWith Yt-kACF coefficient, E (x) indicate x desired value,Indicate YtSquare value and Yt-kThe product of square value,Indicate Yt..., Yt-kPACF coefficient.
ACF the and PACF value obtained by previous step carries out model identification, and recognition rule is as shown in the table:
| AR(p) | MA(q) | ARMA(p,q) | |
| ACF | Hangover/concussion | Q walks truncation | Hangover/concussion |
| PACF | P walks truncation | Hangover/concussion | Hangover/concussion |
Wherein ARMA (p, q) model expression are as follows:
As q=0, which becomes AR (p) model:
The formula indicates the load value of current t moment by p history value Y of pastt-1、Yt-2、…、Yt-pWeighted sum composition,Indicate the corresponding weight of p history value of past of current t moment, εtIndicate current t moment
Load value error term；As p=0, which becomes MA (q) model: Yt=εt+θ1εt-1+θ2εt-2+…+θqεt-q
The formula indicates the load value of current t moment by the weighted average of several white noises and forms, wherein εt、
εt-1、…、εt-qIndicate white Gaussian noise, θ1、θ2、…、θqIndicate the weight of corresponding white Gaussian noise.
Using ACF and PACF on the basis of tentatively judging parameter p and q, then pass through minimum information criterion AIC and Shi Wa
Hereby criterion SC carries out determining rank, and it is opposite optimal models that AIC and SC functional value, which reaches the smallest model,.
Parameter Estimation is carried out using least square method.
Hypothesis testing is carried out, evaluation fitting, the difference of match value and actual value are carried out using the ARIMA model for having determined that parameter
The sequence of formation forms residual sequence, and whether diagnosis residual sequence is white noise, if if the LBQ of the residual sequence of residual sequence unites
Accompany probability P > significance value 0.05 in metering, then thinks that model is acceptable.
The load estimation at remaining moment is calculated as a result, obtaining historical forecast error information using ARIMA model；I.e. using
Forecast analysis is carried out by the model of inspection, obtains the load estimation result at rear n-i moment.
Fig. 4 describes the process that the load estimation value at (n+1)th moment is obtained using Markov chain.
Step S23 includes:
Historical forecast error sequence is inputted, the wide discrete method of historical forecast error sequence is handled, by the codomain of sequence
The m section with same widths is converted into from-∞ to+∞, as state demarcation { Si, i=1,2 ..., m }.
According to state demarcation as a result, calculating and generating state transition probability matrix P, state siTo state sjTransition probabilityWherein nijIndicate SiOne step is transferred to SjNumber, thus generate state transition probability matrix:
Forecast set is generated according to state transition probability matrix P, determines the prediction error initial state distribution of forecast set, it is assumed that
N-th of moment prediction error is in state Si, then the distribution of its state can use the row vector e of 1 × m0It indicates, e0=[0 ... 1 ...
0], it is 1 at the i of position, is 0 at remaining.
Then the prediction error state at (n+1)th moment is distributed as e1=e0P=[p1…pi…pm], max (pi) corresponding Si
The state interval that as (n+1)th moment is most possibly transferred to.If max (pi) have multiple values identical or very close, it adopts
Go bail for and keep the maximum prediction error state of policy selection, avoid server because predicted value by estimation less than normal wrong distribution task
Amount, the prediction error at (n+1)th moment for taking the median of the state interval to estimate as Markov chain, e0When being n-th
The prediction error state distribution vector at quarter, e1For the prediction error state distribution vector at (n+1)th moment, p1Turn for current time
Move on to state interval S1Probability, piState interval S is transferred to for current timeiProbability, pmState is transferred to for current time
Section SmProbability.
Load estimation value is corrected according to the estimated value of prediction error, correction formula is
Wherein V (n+1) is the load estimation value at (n+1)th moment, and v is that Markov chain estimates the pre- of (n+1)th moment
Survey error.
It should be noted that the load estimation model of combination ARIMA model and Markov chain, utilizes Markov chain pair
ARIMA model prediction load error is modified, and improves precision of prediction.
With reference to Fig. 5, the embodiment of the invention also discloses a kind of server based on Markov chain and time series models
Load estimation system 100, comprising:
Load value time series generation unit 1, it is regular for being carried out using load information of the slip window sampling to server
Sampling updates, and generates load value time series；
Load estimation value amending unit 2 passes through ARIMA model for establishing ARIMA model according to load value time series
Computational load predicted value is modified ARIMA model prediction load error using Markov chain, obtains revised load
Predicted value；
Task distributes decision-making foundation unit 3, revised to the report of the scheduler of place server cluster for each host
Load estimation value carries out task distribution for scheduler and provides decision-making foundation.
With reference to Fig. 6, load value time series generation unit 1, comprising:
Load value time series generates subelement 10, is used for when certain server starting in server cluster, Mei Geyi
The section preset sampling period acquires the load information including CPU usage, memory usage an of the machine, generates new
Length is the load value initial time sequence of n；
Load value time series updating unit 11 is used for when reaching the sampling period, by slip window sampling to load value
Time series is updated, and load information earliest in load value time series is replaced with freshly harvested load information, composition
New length is n load value time series；
Whenever generating new load value time series, which submits to load prediction module as input data and carries out
New load estimation value is obtained after processing, the scheduler of cluster where new predicted value will be sent to the server；
Scheduler safeguards that a server node loaded list, scheduler receive the new load estimation value of server report
When, it writes the values into loaded list；
After scheduler receives the request of user service, selection load is optimal in existing server node loaded list
Server node, and forward requests to the node.
With reference to Fig. 7, the load estimation value amending unit 2 is specifically included:
I data determine whether stationary sequence before smoothing preprocessing unit 21 intercepts load value time series, if not
Stationary sequence carries out smoothing preprocessing；
Historical forecast error sequence generation unit 22 calculates the load estimation at remaining moment using ARIMA model as a result, obtaining
Historical forecast error sequence out；
Load estimation value revise subelemen 23, for estimating future from historical forecast error information using Markov chain
Prediction error, and original predictive value is modified using this evaluated error, obtains revised load estimation value.
Smoothing preprocessing unit 21, specifically includes:
Preceding i data for the load value time series that intercepted length is n are as input, using ADF unit root test
Determine whether it is stationary sequence, if non-stationary series, is allowed to become stationary sequence by difference, the number of difference is at this time
It is the difference number for making input data sequence become stationary sequence in the step, p for the parameter d, d in ARIMA (p, q, d) model
It is autoregression item, q is rolling average item number；
Historical forecast error sequence generation unit 22, specifically includes:
It is ARMA (p, q) model by ARIMA (p, q, d) model simplification after determining difference number d；
ACF and PACF value for being obtained by previous step carries out model identification, and recognition rule is as shown in the table:
| AR(p) | MA(q) | ARMA(p,q) | |
| ACF | Hangover/concussion | Q walks truncation | Hangover/concussion |
| PACF | P walks truncation | Hangover/concussion | Hangover/concussion |
Wherein ARMA (p, q) model expression are as follows:
As q=0, which becomes AR (p) model:The formula indicates that the load value of current t moment was gone through by past p
History value Yt-1、Yt-2、…、Yt-pWeighted sum composition,Indicate p history value difference of past of current t moment
Corresponding weight, εtIndicate the error term of the load value of current t moment；As p=0, which becomes MA (q) model:
Yt=εt+θ1εt-1+θ2εt-2+…+θqεt-q,
The formula indicates the load value of current t moment by the weighted average of several white noises and forms, wherein εt、
εt-1、…、εt-qIndicate white Gaussian noise, θ1、θ2、…、θqIndicate the weight of corresponding white Gaussian noise；
For application ACF and PACF tentatively judge parameter p and q on the basis of, then by minimum information criterion AIC with
Schwartz criterion SC carries out determining rank, AIC and SC functional value is selected to reach the smallest model for opposite optimal models；
For carrying out parameter Estimation using least square method；
For carrying out hypothesis testing, evaluation fitting, match value and actual value are carried out using the ARIMA model for having determined that parameter
Difference formed sequence form residual sequence, if in the LBQ statistic of residual sequence together probability P > significance value
0.05, then it is assumed that model is acceptable.
Forecast analysis is carried out using the model examined has been passed through, obtains the load estimation at rear n-i moment as a result, calculating n-
The load estimation error at i moment forms historical forecast error sequence.
Load estimation value revise subelemen 23, is specifically used for:
The load estimation at remaining moment is calculated as a result, obtaining historical forecast error information using ARIMA model；I.e. for benefit
With carrying out forecast analysis by the model examined, the load estimation at rear n-i moment is obtained as a result, n-i moment of calculating
Load estimation error is formed historical forecast error sequence, is modified using Markov chain.
With reference to Fig. 8, the load estimation value generation unit 3 is specifically included:
Wide discrete method processing unit 31, for handling the wide discrete method of historical forecast error sequence, by sequence
Codomain is converted into the m section with same widths from-∞ to+∞, carries out state demarcation { Si, i=1,2 ..., m }；
State transition probability matrix generation unit 32: according to state demarcation as a result, calculating and generating state transition probability square
Battle array P, state siTo state sjTransition probabilityWherein nijIndicate SiOne step is transferred to SjNumber, thus give birth to
At state transition probability matrix:
It predicts that error state is distributed generation unit 33, forecast set is generated according to state transition probability matrix P, it is pre- for determining
Survey the prediction error initial state distribution of collection, it is assumed that n-th of moment prediction error is in state Si, then the distribution of its state is available
The row vector e of 1 × m0It indicates, e0=[0 ... 1 ... 0], is 1 at the i of position, is 0 at remaining；
Obtain the prediction error state distribution at (n+1)th moment are as follows:
e1=e0P=[p1…pi…pm], max (pi) corresponding SiThe shape that as (n+1)th moment is most possibly transferred to
State section, if max (pi) have multiple values identical or very close, take conservative strategy to select maximum prediction error state,
Server is avoided because predicted value is wrong by estimation less than normal to distribute task amount, takes the median of the state interval can as Ma Er
The prediction error at (n+1)th moment of husband's chain estimation, e0For the prediction error state distribution vector at n-th of moment, e1It is (n+1)th
The prediction error state distribution vector at a moment, p1State interval S is transferred to for current time1Probability, piTurn for current time
Move on to state interval SiProbability, pmState interval S is transferred to for current timemProbability；
Load estimation value amending unit 34, for correcting load estimation value, correction formula according to the estimated value of prediction error
ForWherein V (n+1) is the load estimation value at (n+1)th moment, and v is Markov chain estimation n-th
The prediction error at+1 moment.
The above described specific embodiments of the present invention are not intended to limit the scope of the present invention..Any basis
Any other various changes and modifications made by technical concept of the invention should be included in the guarantor of the claims in the present invention
It protects in range.
Claims (8)
1. a kind of Server Load Prediction method based on Markov chain and time series models, which is characterized in that including such as
Lower step:
S1 carries out periodic sampling update using load information of the slip window sampling to server, and generates load value time series；
S2 establishes ARIMA model according to load value time series, can using Ma Er by ARIMA model computational load predicted value
Husband's chain is modified ARIMA model prediction load error, obtains revised load estimation value；
S3, each host report revised load estimation value to the scheduler of place server cluster, carry out task for scheduler
Distribution provides decision-making foundation.
2. the Server Load Prediction method based on Markov chain and time series models as described in claim 1, special
Sign is that step S1 includes:
S11 acquires the packet an of the machine every one section of preset sampling period when certain server starting in server cluster
The load information including CPU usage, memory usage is included, the load value initial time sequence that length is n is generated；
S12 is updated load value time series by slip window sampling, when reaching the sampling period by the load value time
Earliest load information replaces with freshly harvested load information in sequence, and forming new length is n load value time series.
3. the Server Load Prediction method based on Markov chain and time series models as claimed in claim 2, special
Sign is that step S2 includes:
S21, i data determine whether stationary sequence before intercepting load value time series, if non-stationary series, carry out tranquilization
Pretreatment；
S22 calculates the load estimation at remaining moment as a result, obtaining historical forecast error sequence using ARIMA model；
S23 is estimated following prediction error from historical forecast error information using Markov chain, and uses this evaluated error
Original predictive value is modified, revised load estimation value is obtained.
4. the Server Load Prediction method based on Markov chain and time series models as claimed in claim 3, special
Sign is,
Step S21 includes:
The preceding i data of load value time series are intercepted as input, ADF unit root test is used to determine it whether for steady sequence
Column, if non-stationary series, are allowed to become stationary sequence by difference, the number of difference is ARIMA (p, q, d) model at this time
In parameter d, d is so that input data sequence is become the difference number of stationary sequence in the step, and p is autoregression item, and q is mobile
Average item number；
Step S22 includes:
ARIMA (p, q, d) model after determining difference number d, can be reduced to ARMA (p, q) model by S221, be identified for model
It prepares, in order to carry out model identification, is judged using auto-correlation function ACF and partial autocorrelation function PACF and calculate ACF
Value and PACF value；
S222, ACF the and PACF value obtained by previous step carry out model identification；
S223 passes through minimum information criterion AIC and Shi Wa using ACF and PACF value on the basis of tentatively judging parameter p and q
Hereby criterion SC carries out determining rank, is opposite optimal models when AIC and SC functional value reaches the smallest model；
S224 carries out parameter Estimation using least square method；
S225 carries out hypothesis testing, using having determined that the ARIMA model of parameter carries out evaluation fitting, match value and actual value it
The sequence that difference is formed forms residual sequence, if the probability P > significance value 0.05 that accompanies in the LBQ statistic of residual sequence
Think that model is acceptable；
S226 carries out forecast analysis using the model examined has been passed through, obtains the load estimation at rear n-i moment as a result, calculating
The load estimation error at n-i moment afterwards forms historical forecast error sequence；
Step S23 includes:
S231 handles the wide discrete method of historical forecast error sequence, the codomain of sequence is converted into having from-∞ to+∞
M section of same widths, and carry out state demarcation；
S232: according to state demarcation as a result, calculating and generating state transition probability matrix P；
S233 generates forecast set according to state transition probability matrix P, determines the prediction error initial state distribution of forecast set, it is assumed that
N-th of moment prediction error is in state Si, then the distribution of its state can use the row vector e of 1 × m0It indicates, e0=
[0...1...0] is 1 at the i of position, is 0 at remaining；S234: the then prediction error state distribution at (n+1)th moment are as follows:
e1=e0P=[p1…pi…pm], max (pi) corresponding SiThe state area that as (n+1)th moment is most possibly transferred to
Between, the prediction error at (n+1)th moment for taking the median of the state interval to estimate as Markov chain, e0When being n-th
The prediction error state distribution vector at quarter, e1For the prediction error state distribution vector at (n+1)th moment, p1Turn for current time
Move on to state interval S1Probability, piState interval S is transferred to for current timeiProbability, pmState is transferred to for current time
Section SmProbability；
S235: load estimation value is corrected according to the estimated value of prediction error, correction formula is Wherein V
It (n+1) is the load estimation value at (n+1)th moment, v is the prediction error that Markov chain estimates (n+1)th moment.
5. a kind of Server Load Prediction system based on Markov chain and time series models characterized by comprising
Load value time series generation unit, for carrying out periodic sampling more using load information of the slip window sampling to server
Newly, and load value time series is generated；
Load estimation value amending unit is calculated for establishing ARIMA model according to load value time series by ARIMA model
Load estimation value is modified ARIMA model prediction load error using Markov chain, obtains revised load estimation
Value；
Task distributes decision-making foundation unit, pre- to the revised load of the scheduler of place server cluster report for each host
Measured value carries out task distribution for scheduler and provides decision-making foundation.
6. system as claimed in claim 5, which is characterized in that load value time series generation unit, comprising:
Load value time series generates subelement, for being preset every one section when certain server starting in server cluster
Sampling period acquire the load information including CPU usage, memory usage including of the machine, generating length is n
Load value initial time sequence；
Load value time series updating unit is used for when reaching the sampling period, by slip window sampling to load value time sequence
Column are updated, and load information earliest in load value time series is replaced with freshly harvested load information, forms new length
Degree is n load value time series.
7. system as claimed in claim 5, which is characterized in that the load estimation value amending unit specifically includes:
I data determine whether stationary sequence before smoothing preprocessing unit interception load value time series, if non-stationary sequence
Column carry out smoothing preprocessing historical forecast error sequence generation unit, and the load for calculating the remaining moment using ARIMA model is pre-
It surveys as a result, obtaining historical forecast error sequence；
Load estimation value revise subelemen, for estimating following prediction from historical forecast error information using Markov chain
Error, and original predictive value is modified using this evaluated error, obtain revised load estimation value.
8. system as claimed in claim 7, which is characterized in that the load estimation value amending unit, further includes:
The smoothing preprocessing unit, is specifically used for:
The preceding i data of load value time series are intercepted as input, ADF unit root test is used to determine it whether for steady sequence
Column, if non-stationary series, are allowed to become stationary sequence by difference, the number of difference is ARIMA (p, q, d) model at this time
In parameter d, d is so that input data sequence is become the difference number of stationary sequence in the step, and p is autoregression item, and q is mobile
Average item number；
The historical forecast error sequence generation unit, is specifically used for:
After determining difference number d, ARIMA (p, q, d) model can be reduced to ARMA (p, q) model, do standard for model identification
It is standby, in order to carry out model identification, judged using auto-correlation function ACF and partial autocorrelation function PACF and calculate ACF value and
PACF value；
ACF the and PACF value obtained by previous step carries out model identification；
Using ACF and PACF value on the basis of tentatively judging parameter p and q, pass through minimum information criterion AIC and Schwartz criterion
SC carries out determining rank, is opposite optimal models when AIC and SC functional value reaches the smallest model；
Parameter Estimation is carried out using least square method；
Hypothesis testing is carried out, using the ARIMA model progress evaluation fitting for having determined that parameter, the difference of match value and actual value is formed
Sequence form residual sequence, if thinking mould if probability P > significance value 0.05 together in the LBQ statistic of residual sequence
Type is acceptable；
Using passed through examine model carry out forecast analysis, obtain the load estimation at rear n-i moment as a result, calculating after n-i
The load estimation error at a moment forms historical forecast error sequence；
The load value time series updating unit, is specifically used for:
The wide discrete method of historical forecast error sequence is handled, the codomain of sequence is converted into have identical width from-∞ to+∞
M section of degree, and carry out state demarcation；
According to state demarcation as a result, calculating and generating state transition probability matrix P；
Forecast set is generated according to state transition probability matrix P, determines the prediction error initial state distribution of forecast set, it is assumed that n-th
A moment prediction error is in state Si, then the distribution of its state can use the row vector e of 1 × m0It indicates, e0=[0...1...0],
It is 1 at the i of position, is 0 at remaining；
The prediction error state distribution at (n+1)th moment are as follows: e1=e0P=[p1…pi…Pm], max (pi) corresponding SiAs n-th
The state interval that+1 moment is most possibly transferred to, take the median of the state interval as Markov chain estimate n-th+
The prediction error at 1 moment, e0For the prediction error state distribution vector at n-th of moment, e1It is missed for the prediction at (n+1)th moment
Poor state distribution vector, p1State interval S is transferred to for current time1Probability, piState interval S is transferred to for current timei
Probability, PmState interval S is transferred to for current timemProbability；
Load estimation value is corrected according to the estimated value of prediction error, correction formula isWherein V (n+1)
For the load estimation value at (n+1)th moment, v is the prediction error that Markov chain estimates (n+1)th moment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811544236.6A CN109787855A (en) | 2018-12-17 | 2018-12-17 | Server Load Prediction method and system based on Markov chain and time series models |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811544236.6A CN109787855A (en) | 2018-12-17 | 2018-12-17 | Server Load Prediction method and system based on Markov chain and time series models |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109787855A true CN109787855A (en) | 2019-05-21 |
Family
ID=66498038
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811544236.6A Pending CN109787855A (en) | 2018-12-17 | 2018-12-17 | Server Load Prediction method and system based on Markov chain and time series models |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109787855A (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110149237A (en) * | 2019-06-13 | 2019-08-20 | 东北大学 | A kind of Hadoop platform calculate node load predicting method |
| CN110389820A (en) * | 2019-06-28 | 2019-10-29 | 浙江大学 | A kind of private clound method for scheduling task carrying out resources based on v-TGRU model |
| CN110471768A (en) * | 2019-08-13 | 2019-11-19 | 北京计算机技术及应用研究所 | A kind of load predicting method based on fastPCA-ARIMA |
| CN110851333A (en) * | 2019-11-14 | 2020-02-28 | 北京金山云网络技术有限公司 | Monitoring method and device of root partition and monitoring server |
| CN111242348A (en) * | 2019-12-30 | 2020-06-05 | 安徽先兆科技有限公司 | Electrical safety monitoring method and system based on time sequence |
| CN111561930A (en) * | 2020-04-28 | 2020-08-21 | 南京工业大学 | Method for restraining random drift error of vehicle-mounted MEMS gyroscope |
| CN111913803A (en) * | 2020-07-21 | 2020-11-10 | 哈尔滨工程大学 | Service load fine granularity prediction method based on AKX hybrid model |
| CN112000459A (en) * | 2020-03-31 | 2020-11-27 | 华为技术有限公司 | Method for expanding and contracting service and related equipment |
| CN113190429A (en) * | 2021-06-03 | 2021-07-30 | 河北师范大学 | Server performance prediction method and device and terminal equipment |
| CN113347014A (en) * | 2020-03-02 | 2021-09-03 | 中国科学院沈阳自动化研究所 | Industrial control system situation combined prediction method based on time sequence |
| CN113641960A (en) * | 2021-08-30 | 2021-11-12 | 北京航空航天大学 | Component data time sequence prediction method and system based on Givens transformation |
| CN114489944A (en) * | 2022-01-24 | 2022-05-13 | 合肥工业大学 | Kubernetes-based prediction type elastic expansion method |
| CN114499934A (en) * | 2021-12-16 | 2022-05-13 | 西安交通大学 | Intrusion detection method and system based on fusion learning in industrial Internet of things |
| CN115242797A (en) * | 2022-06-17 | 2022-10-25 | 西北大学 | Client load balancing method and system for micro-service architecture |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040249776A1 (en) * | 2001-06-28 | 2004-12-09 | Microsoft Corporation | Composable presence and availability services |
| CN103545846A (en) * | 2013-11-11 | 2014-01-29 | 湖南大学 | Microgrid economic operation method based on generalized load prediction |
| CN106933650A (en) * | 2017-03-03 | 2017-07-07 | 北方工业大学 | load management method and system of cloud application system |
| CN107273262A (en) * | 2017-05-23 | 2017-10-20 | 深圳先进技术研究院 | The Forecasting Methodology and system of a kind of hardware event |
-
2018
- 2018-12-17 CN CN201811544236.6A patent/CN109787855A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040249776A1 (en) * | 2001-06-28 | 2004-12-09 | Microsoft Corporation | Composable presence and availability services |
| CN103545846A (en) * | 2013-11-11 | 2014-01-29 | 湖南大学 | Microgrid economic operation method based on generalized load prediction |
| CN106933650A (en) * | 2017-03-03 | 2017-07-07 | 北方工业大学 | load management method and system of cloud application system |
| CN107273262A (en) * | 2017-05-23 | 2017-10-20 | 深圳先进技术研究院 | The Forecasting Methodology and system of a kind of hardware event |
Non-Patent Citations (4)
| Title |
|---|
| JOHANN LEITHON，ET AL.,: ""Renewable Energy Management in Networks:An Online Strategy based on ARIMA Forecasting and a Markov Chain Cellular Model"", 《IEEE WIRELESS CONFERENCE AND NETWORKING CONFERENCE (WCNC 2016) TRACK 1:PHY AND FUNDAMENTALS》 * |
| 卢飞强等，: ""基于时间序列分析法的软土路基沉降预测研究"", 《路基工程》 * |
| 焦娇等，: ""基于ARIMA和马尔可夫链的风速中期预测模型"", 《电测与仪表》 * |
| 蒋堃等，: ""监控与预测的云资源优化配置"", 《华侨大学学报(自然科学版)》 * |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020248228A1 (en) * | 2019-06-13 | 2020-12-17 | 东北大学 | Computing node load prediction method in a hadoop platform |
| CN110149237A (en) * | 2019-06-13 | 2019-08-20 | 东北大学 | A kind of Hadoop platform calculate node load predicting method |
| CN110149237B (en) * | 2019-06-13 | 2021-06-22 | 东北大学 | Hadoop platform computing node load prediction method |
| CN110389820A (en) * | 2019-06-28 | 2019-10-29 | 浙江大学 | A kind of private clound method for scheduling task carrying out resources based on v-TGRU model |
| CN110471768A (en) * | 2019-08-13 | 2019-11-19 | 北京计算机技术及应用研究所 | A kind of load predicting method based on fastPCA-ARIMA |
| CN110471768B (en) * | 2019-08-13 | 2023-02-03 | 北京计算机技术及应用研究所 | FastPCA-ARIMA-based load prediction method |
| CN110851333A (en) * | 2019-11-14 | 2020-02-28 | 北京金山云网络技术有限公司 | Monitoring method and device of root partition and monitoring server |
| CN110851333B (en) * | 2019-11-14 | 2023-09-01 | 北京金山云网络技术有限公司 | Root partition monitoring method and device and monitoring server |
| CN111242348A (en) * | 2019-12-30 | 2020-06-05 | 安徽先兆科技有限公司 | Electrical safety monitoring method and system based on time sequence |
| CN113347014A (en) * | 2020-03-02 | 2021-09-03 | 中国科学院沈阳自动化研究所 | Industrial control system situation combined prediction method based on time sequence |
| CN113347014B (en) * | 2020-03-02 | 2023-06-20 | 中国科学院沈阳自动化研究所 | Industrial control system situation combination prediction method based on time sequence |
| CN112000459A (en) * | 2020-03-31 | 2020-11-27 | 华为技术有限公司 | Method for expanding and contracting service and related equipment |
| CN111561930A (en) * | 2020-04-28 | 2020-08-21 | 南京工业大学 | Method for restraining random drift error of vehicle-mounted MEMS gyroscope |
| CN111913803A (en) * | 2020-07-21 | 2020-11-10 | 哈尔滨工程大学 | Service load fine granularity prediction method based on AKX hybrid model |
| CN111913803B (en) * | 2020-07-21 | 2023-12-29 | 哈尔滨工程大学 | Service load fine granularity prediction method based on AKX hybrid model |
| CN113190429A (en) * | 2021-06-03 | 2021-07-30 | 河北师范大学 | Server performance prediction method and device and terminal equipment |
| CN113641960B (en) * | 2021-08-30 | 2023-09-22 | 北京航空航天大学 | Givens transformation-based component data time sequence prediction method and system |
| CN113641960A (en) * | 2021-08-30 | 2021-11-12 | 北京航空航天大学 | Component data time sequence prediction method and system based on Givens transformation |
| CN114499934B (en) * | 2021-12-16 | 2022-12-09 | 西安交通大学 | Intrusion detection method and system based on fusion learning in industrial Internet of things |
| CN114499934A (en) * | 2021-12-16 | 2022-05-13 | 西安交通大学 | Intrusion detection method and system based on fusion learning in industrial Internet of things |
| CN114489944A (en) * | 2022-01-24 | 2022-05-13 | 合肥工业大学 | Kubernetes-based prediction type elastic expansion method |
| CN115242797A (en) * | 2022-06-17 | 2022-10-25 | 西北大学 | Client load balancing method and system for micro-service architecture |
| CN115242797B (en) * | 2022-06-17 | 2023-10-27 | 西北大学 | Micro-service architecture-oriented client load balancing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109787855A (en) | Server Load Prediction method and system based on Markov chain and time series models | |
| Liu et al. | A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning | |
| Bhattacharjee et al. | Barista: Efficient and scalable serverless serving system for deep learning prediction services | |
| US9002774B2 (en) | Systems and methods for generating a forecasting model and forecasting future values | |
| CN108664378A (en) | A kind of most short optimization method for executing the time of micro services | |
| CN106020933B (en) | Cloud computing dynamic resource scheduling system and method based on ultralight amount virtual machine | |
| CN113037877B (en) | Optimization method for time-space data and resource scheduling under cloud edge architecture | |
| CN106776005A (en) | A kind of resource management system and method towards containerization application | |
| Tiemessen et al. | Dynamic demand fulfillment in spare parts networks with multiple customer classes | |
| CN106528280A (en) | Task allocation method and system | |
| CN109960573B (en) | Cross-domain computing task scheduling method and system based on intelligent perception | |
| Liu et al. | Quantitative workload analysis and prediction using Google cluster traces | |
| US20120221373A1 (en) | Estimating Business Service Responsiveness | |
| CN107404409A (en) | Towards the container cloud elastic supply number of containers Forecasting Methodology and system of mutation load | |
| CN109377291A (en) | Task price expectation method, apparatus, electronic equipment and computer storage medium | |
| CN113672846A (en) | Network appointment scheduling method and device, electronic equipment and storage medium | |
| CN109889391A (en) | A kind of network short term traffic forecasting method based on built-up pattern | |
| CN114895773A (en) | Energy consumption optimization method, system and device of heterogeneous multi-core processor and storage medium | |
| US20170139754A1 (en) | A mechanism for controled server overallocation in a datacenter | |
| Devi et al. | Time series-based workload prediction using the statistical hybrid model for the cloud environment | |
| Zhang et al. | A dynamic resource overbooking mechanism in fog computing | |
| CN107608781B (en) | Load prediction method, device and network element | |
| Rossi et al. | Bayesian uncertainty modelling for cloud workload prediction | |
| CN112819215B (en) | Recommendation strategy training method and device, electronic equipment and readable storage medium | |
| CN105491079A (en) | Method and device for adjusting resources needed by application in cloud computing environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190521 |