CN116702044A - Capacity prediction method, system, storage medium and processor - Google Patents
Capacity prediction method, system, storage medium and processor Download PDFInfo
- Publication number
- CN116702044A CN116702044A CN202310769144.2A CN202310769144A CN116702044A CN 116702044 A CN116702044 A CN 116702044A CN 202310769144 A CN202310769144 A CN 202310769144A CN 116702044 A CN116702044 A CN 116702044A
- Authority
- CN
- China
- Prior art keywords
- data
- capacity
- index
- prediction
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 70
- 238000003860 storage Methods 0.000 title claims abstract description 19
- 238000007781 pre-processing Methods 0.000 claims abstract description 12
- 238000012549 training Methods 0.000 claims description 56
- 238000012545 processing Methods 0.000 claims description 27
- 238000012937 correction Methods 0.000 claims description 21
- 239000013598 vector Substances 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 18
- 238000012795 verification Methods 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 14
- 230000003068 static effect Effects 0.000 claims description 12
- 238000013528 artificial neural network Methods 0.000 claims description 8
- 230000004913 activation Effects 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 5
- 230000002779 inactivation Effects 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 3
- 230000008439 repair process Effects 0.000 claims description 3
- 230000001960 triggered effect Effects 0.000 claims description 3
- 239000010410 layer Substances 0.000 description 45
- 238000010586 diagram Methods 0.000 description 12
- 230000008859 change Effects 0.000 description 11
- 230000008569 process Effects 0.000 description 10
- 238000012423 maintenance Methods 0.000 description 8
- 238000011144 upstream manufacturing Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 210000002569 neuron Anatomy 0.000 description 3
- 230000002829 reductive effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012731 temporal analysis Methods 0.000 description 2
- 238000000700 time series analysis Methods 0.000 description 2
- 230000003213 activating effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000415 inactivating effect Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 238000012886 linear function Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 239000002356 single layer Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The embodiment of the application provides a capacity prediction method, a capacity prediction system, a capacity prediction processor and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring index data of capacity to be predicted, and preprocessing the index data to obtain preprocessed index data; wherein the index data includes: index history data, index real-time data, business data, associated resource data and application performance data; classifying basic data based on the preprocessed index data; based on the classified basic data, carrying out capacity prediction of a future preset time section in a capacity prediction model to obtain a prediction result; and determining and pushing the predicted value of each capacity index based on the predicted result. The method solves the problem that the existing capacity prediction scheme is low in prediction precision.
Description
Technical Field
The present application relates to the field of computer technology, and in particular, to a capacity prediction method, a capacity prediction system, a storage medium, and a processor
Background
In order to ensure stable operation of the system, it is necessary to ensure that the capacity of the system meets the requirements, and along with the diversified development of service types, the nonlinear characteristics of capacity prediction are more and more obvious, if the accuracy of capacity prediction can be ensured, early warning, capacity cleaning and extension can be performed based on the capacity prediction result, and the stable operation performance of the system can be greatly ensured. Aiming at capacity early warning, the existing operation and maintenance team generally relies on monitoring tools such as zabbix, configures user attention index monitoring items and alarm triggers, monitors the use condition of relevant capacity in real time, and passively expands the capacity according to self operation and maintenance experience. And secondly, judging the capacity expansion trend by the supervisor according to the history operation and maintenance experience of operation and maintenance personnel, and carrying out capacity management. In addition, a capacity for a future period of time is predicted by using a machine learning method, and capacity management is performed according to a future capacity trend.
In the current capacity prediction method, regression prediction is carried out by using capacity indexes to be predicted, a regression model is generally a linear model, highly complex data are difficult to express, and the influence of abnormal points is easy to cause. In the current capacity prediction method, in the modeling process, index historical data of the capacity to be predicted is generally used as input to predict the future capacity. The capacity prediction method does not consider the related parties such as business factors, upstream and downstream data, network topology related configuration data and the like, and the selected data is single, so that the capacity prediction method of the traditional method cannot effectively predict the capacity change caused by business change, related resource change and upstream and downstream business change. The time characteristics are not added in the modeling process of the regression prediction method. Capacity prediction is a typical application of time series analysis prediction, and the time characteristic is a key factor affecting the capacity trend. Aiming at the problem of insufficient prediction precision of the existing capacity prediction scheme, a new capacity prediction method needs to be created.
Disclosure of Invention
The embodiment of the application aims to provide a capacity prediction method, a system, a storage medium and a processor, so as to solve the problem of low capacity prediction precision in the existing scheme.
In order to achieve the above object, a first aspect of the present application provides a capacity prediction method, including: acquiring index data of capacity to be predicted, and preprocessing the index data to obtain preprocessed index data; wherein the index data includes: index history data, index real-time data, business data, associated resource data and application performance data; classifying basic data based on the preprocessed index data; based on the classified basic data, carrying out capacity prediction of a future preset time section in a capacity prediction model to obtain a prediction result; and determining and pushing the predicted value of each capacity index based on the predicted result.
In an embodiment of the present application, the associated resource data includes: one or more of the total number of cores of the CPU, the allocated amount of the CPU, the remaining allocated amount of the CPU, the total amount of the memory, the allocated amount of the memory, and the remaining usable amount of the memory.
In an embodiment of the present application, the preprocessing the index data includes: performing one or more of outlier rejection processing, deduplication processing, index missing segment repair processing and data rejection processing on the index data; and identifying the time sequence data, and carrying out future covariate extraction processing on the identified time sequence data.
In an embodiment of the present application, the method further includes: performing capacity prediction model training, comprising: splitting the index history data into training data and verification data, the training data being chronologically earlier than the verification data; based on the training data, reading auxiliary prediction time sequences, known future covariates and static covariates corresponding to the same time period, and forming a training set together with the training data; in a pre-constructed neural network, taking the data of the same period of verification data as a training target, and carrying out model training based on the training set to obtain an initial model; comparing the training result of the initial model with the verification data, and carrying out model correction based on deviation, wherein each time model correction is completed, comparing the training result of the correction model with the verification data again until the difference value of the training result of the correction model and the verification data meets the error minimization condition, stopping model correction, and taking the correction model obtained by the latest correction as a capacity prediction model.
In an embodiment of the present application, the pre-constructed neural network includes: input module, encoder, decoder and output module.
In an embodiment of the present application, the input data of the input module includes: a first portion of data comprising a capacity history sequence to be predicted and auxiliary prediction sequence data; the second part of data is static covariate data after pretreatment; and the third part of data is data obtained by passing the multi-layer perceptron residual error module through the future known covariates.
In an embodiment of the present application, the encoder is formed by stacking a plurality of multi-layer perceptron residual modules.
In the embodiment of the application, the multi-layer perceptron residual error module comprises a short connection branch and a trunk branch; the input data of the input module passes through a first full connection layer on the short connection branch to obtain a first intermediate result vector; the input data of the input module passes through a second full-connection layer, a leakage type rectification linear unit activation function module, a third full-connection layer and a random inactivation layer one by one on the main branch to obtain a second intermediate result vector; and the output vector of the input module is obtained by layer normalization based on the first intermediate result vector and the second intermediate result vector.
In an embodiment of the present application, the performing capacity prediction of a future preset time zone in a capacity prediction model based on the classified basic data to obtain a prediction result includes: and taking the index real-time data, the service data, the associated resource data and the application performance data as input parameters, and training based on the capacity prediction model to obtain a training result.
In the embodiment of the present application, the determining and pushing the predicted value of each capacity index based on the predicted result includes: identifying predicted values of each capacity index based on the training results; respectively comparing the predicted value of each capacity index with the associated resource data under the corresponding capacity index, and judging whether an overrun capacity index exists or not; when the overrun capacity index exists, information pushing is performed in a differentiated mode aiming at the overrun capacity index, and alarm information is triggered.
A second aspect of the present application provides a capacity prediction system comprising: the acquisition unit is used for acquiring index data of the capacity to be predicted, and preprocessing the index data to obtain preprocessed index data; wherein the index data includes: index history data, index real-time data, business data, associated resource data and application performance data; the processing unit is used for classifying basic data based on the preprocessed index data; the prediction unit is used for performing capacity prediction for a preset time section in a capacity prediction model based on the basic data to obtain a prediction result; and the pushing unit is used for determining and pushing the predicted value of each capacity index based on the predicted result.
A third aspect of the present application provides a processor configured to perform the capacity prediction method described above.
A fourth aspect of the application provides a machine-readable storage medium having stored thereon instructions which, when executed by a processor, cause the processor to be configured to perform the capacity prediction method described above.
A fifth aspect of the application provides a computer program product comprising a computer program which, when executed by a processor, implements the capacity prediction method described above.
According to the technical scheme, when the model is built, historical data is taken as input, associated service data, upstream and downstream data and network topology associated configuration data are taken as auxiliary prediction data to be input into the model, a data distribution relation is fitted through a deep learning model, and accurate prediction is carried out on capacity future trend. Through comprehensive collection and prediction of related data, the problem that the training accuracy is low due to the fact that an existing scheme only depends on single historical data for training is avoided.
Additional features and advantages of embodiments of the application will be set forth in the detailed description which follows.
Drawings
The accompanying drawings are included to provide a further understanding of embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain, without limitation, the embodiments of the application. In the drawings:
FIG. 1 schematically shows a flow diagram of a capacity prediction method according to an embodiment of the application;
FIG. 2 schematically illustrates a block diagram of an input module according to an embodiment of the application;
FIG. 3 schematically illustrates a block diagram of a multi-layer perceptron residual module, in accordance with an embodiment of the present application;
FIG. 4 schematically illustrates a block diagram of a capacity prediction system according to an embodiment of the present application;
fig. 5 schematically shows an internal structural view of a computer device according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it should be understood that the detailed description described herein is merely for illustrating and explaining the embodiments of the present application, and is not intended to limit the embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
It should be noted that, if directional indications (such as up, down, left, right, front, and rear) are involved in the embodiment of the present application, the directional indications are merely used to explain the relative positional relationship, movement conditions, etc. between the components in a specific posture (as shown in the drawings), and if the specific posture is changed, the directional indications are correspondingly changed.
In addition, if there is a description of "first", "second", etc. in the embodiments of the present application, the description of "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present application.
In order to ensure stable operation of the system, it is necessary to ensure that the capacity of the system meets the requirements, and along with the diversified development of service types, the nonlinear characteristics of capacity prediction are more and more obvious, if the accuracy of capacity prediction can be ensured, early warning, capacity cleaning and extension can be performed based on the capacity prediction result, and the stable operation performance of the system can be greatly ensured.
Aiming at capacity early warning, the existing operation and maintenance team generally relies on monitoring tools such as zabbix, configures user attention index monitoring items and alarm triggers, monitors the use condition of relevant capacity in real time, and passively expands the capacity according to self operation and maintenance experience. And secondly, judging the capacity expansion trend by the supervisor according to the history operation and maintenance experience of operation and maintenance personnel, and carrying out capacity management. In addition, a machine learning method is used for predicting capacity for a period of time in the future, capacity management is carried out according to the trend of the capacity in the future, and the specific flow of capacity prediction is as follows:
1) Input data: acquiring single capacity data of a capacity index to be predicted in a preset time period before the current moment;
2) Data preprocessing: preprocessing time sequence data within a preset time period of the capacity index to be predicted, including abnormal point elimination, judging whether the data length meets the model training requirement or not, repairing index missing values and the like;
3) Characteristic engineering: the method comprises the steps of performing data feature mining on indexes to be predicted, including sequence periodicity, outlier periodicity, data capacity change types, upper and lower bounds of data change, holiday effect and the like, and processing data according to time sequence data features;
4) Model training: the processed capacity index data to be predicted is used as training data, a regression prediction model is used for training, the regression prediction model is optimized by utilizing the deviation between the model prediction result and the real data, and an optimal model is reserved;
5) Model prediction: and inputting time sequence data of the capacity index to be predicted within a preset time period to the regression prediction model for prediction, and obtaining a prediction result.
In the current capacity prediction method, regression prediction is carried out by using capacity indexes to be predicted, a regression model is generally a linear model, highly complex data are difficult to express, and the influence of abnormal points is easy to cause. In the current capacity prediction method, in the modeling process, index historical data of the capacity to be predicted is generally used as input to predict the future capacity. The capacity prediction method does not consider the related parties such as business factors, upstream and downstream data, network topology related configuration data and the like, and the selected data is single, so that the capacity prediction method of the traditional method cannot effectively predict the capacity change caused by business change, related resource change and upstream and downstream business change. The time characteristics are not added in the modeling process of the regression prediction method. Capacity prediction is a typical application of time series analysis prediction, and the time characteristic is a key factor affecting the capacity trend.
Aiming at the problem of low capacity prediction precision in the existing scheme, the scheme of the application provides a new capacity prediction method, when the scheme of the application is used for training model construction, historical data is taken as input, associated service data, upstream and downstream data and network topology associated configuration data are taken as auxiliary prediction data input models, and data distribution relations are fitted through a deep learning model to accurately predict future trend of the capacity. Through comprehensive collection and prediction of related data, the problem that the training accuracy is low due to the fact that an existing scheme only depends on single historical data for training is avoided.
Fig. 1 schematically shows a flow diagram of a capacity prediction method according to an embodiment of the application. As shown in fig. 1, in an embodiment of the present application, there is provided a capacity prediction method, including the steps of:
step S10: and acquiring index data of the capacity to be predicted, and preprocessing the index data to obtain preprocessed index data.
Specifically, the index data comprises index history data, index real-time data, service data, associated resource data and application performance data; the associated resource data includes: one or more of the total number of cores of the CPU, the allocated amount of the CPU, the remaining allocated amount of the CPU, the total amount of the memory, the allocated amount of the memory, and the remaining usable amount of the memory.
In the embodiment of the application, capacity prediction is carried out by using the multi-variable time sequence, not only the historical time sequence of the capacity to be predicted is used, but also static covariates, associated service data, associated resource data, application performance data and the like are introduced, so that a help model can learn more information for capacity prediction, and the deviation between the capacity prediction and a real result is reduced.
Specifically, the preprocessing the index data includes: performing one or more of outlier rejection processing, deduplication processing, index missing segment repair processing and data rejection processing on the index data; and identifying the time sequence data, and carrying out future covariate extraction processing on the identified time sequence data.
Step S20: and classifying basic data based on the preprocessed index data.
Specifically, the scheme of the application not only needs to carry out model training based on the collected index data, but also needs to carry out subsequent capacity prediction based on the real-time data. Therefore, index data differentiation is needed, the effectiveness of subsequent data processing is ensured, the output processing time is reduced, and the data processing efficiency is improved. Based on this, the present application distinguishes based on the data specified by the index data, and uses each data set distinguished as the base data.
Step S30: and carrying out capacity prediction of a future preset time section in a capacity prediction model based on the classified basic data to obtain a prediction result.
Specifically, the method further comprises the following steps: performing capacity prediction model training, comprising: splitting the index history data into training data and verification data, the training data being chronologically earlier than the verification data; based on the training data, reading auxiliary prediction time sequences, known future covariates and static covariates corresponding to the same time period, and forming a training set together with the training data; in a pre-constructed neural network, taking the verification data and the data of the same period as training targets, and carrying out model training based on the training set to obtain an initial model; comparing the training result of the initial model with the verification data, carrying out model correction based on deviation, comparing the training result of the correction model with the verification data every time the model correction is completed until the difference value of the training result of the correction model and the verification data meets the error minimization condition, stopping model correction, and taking the correction model obtained by the latest correction as a capacity prediction model.
Among these, future covariates are known, for example, as covariates, variables related to the date that each point in time belongs to the day of the week, belongs to which month, and even whether it is a legal holiday. These covariates vary over time, but are known in the future to be predicted, that is, such covariates are known virtually without prediction. The static covariates are covariates that are invariant with respect to capacity, such as the class to which the predicted capacity belongs (traffic, iaas layer resources, paas layer resources, etc.).
In the scheme of the application, time characteristics are introduced into known future covariates, and variables related to the date, such as hours, days, months and even legal holidays, of each time point are taken as covariate input models, so that effective characteristics are provided for modeling of capacity prediction, and the accuracy of capacity prediction is improved. According to the scheme, capacity prediction is carried out by using the multi-variable time sequence, not only is the historical time sequence of the capacity to be predicted used, but also static covariates, associated service data, associated resource data, application performance data and the like are introduced, so that a help model can learn more information for capacity prediction, and the deviation between the capacity prediction and a real result is reduced.
Preferably, the pre-constructed neural network includes: input module, encoder, decoder and output module.
Preferably, the input data of the input module includes: a first portion of data comprising a capacity history sequence to be predicted and auxiliary prediction sequence data; the second part of data is static covariate data after pretreatment; and the third part of data is data obtained by passing the multi-layer perceptron residual error module through the future known covariates.
In one possible implementation, as in fig. 2, the input data can be divided into three major parts: 1. the method comprises the steps that a capacity history sequence to be predicted and an auxiliary prediction sequence with the length of L respectively pass through a full connection layer, the results of the capacity history sequence to be predicted and the auxiliary prediction sequence are stacked and pass through the full connection layer, then the results are sequentially sent into convolution layers with the step length of C (C is a super parameter), and the output of each convolution layer is spliced and sent into the full connection layer to obtain the output of a first part; 2. the preprocessed static covariates are used as second partial data; 3. the future known covariates with length L+H (L is the length of the historical data and H is the length of the predicted data) are processed by the multi-layer perceptron residual error module to obtain the third part of data. And splicing the three parts of data to obtain the final input.
Preferably, the encoder is constructed of a stack of multiple multi-layer perceptron residual modules.
Preferably, as shown in fig. 3, the multi-layer perceptron residual module includes two branches: short connecting branches and trunk branches; the input data passes through a first full connection layer on a short connection branch to obtain a first intermediate result vector; the input data sequentially passes through a second full-connection layer, a leakage type rectification linear unit activation function module, a third full-connection layer and a random inactivation layer on the trunk branch to obtain a second intermediate result vector; and the output vector of the input module is obtained by layer normalization based on the first intermediate result vector and the second intermediate result vector.
In one possible implementation, in the multi-layer perceptron residual module, the data is split into two paths after input: the method comprises the steps of activating a function by using a leakage-type rectifying linear unit (leakage-ReLU) after passing through a full-connection layer, and then inactivating immediately after passing through the full-connection layer to obtain a result vector; and the second is to obtain an intermediate result vector through the full connection layer. And adding the result vectors of the two paths, and carrying out layer normalization to obtain an output vector.
Wherein, the full connection layer is that each node is connected with all nodes of the upper layer and is used for integrating the features extracted from the front edge. The parameters of the fully connected layer are also generally the most due to their fully connected nature. The leakage rectifying linear unit (leakage ReLU) function is a specialization of the ReLU function, whose function value is no longer equal to 0 when x <0, but has a small slope that decreases slowly. The activation function selects an alpha value; common values are between 0.01 and 0.1. Random inactivation is the random omission or masking of some neurons in a proportion (the proportion parameter can be set) during training. These neurons are randomly "discarded", that is, they temporarily disappear during the forward propagation of their contribution to the downstream neurons, nor will they have any weight updates during the backward propagation. So, by the propagation process dropout will produce the same effect of shrink weights as the L2 norm. Layer normalization is to normalize all features of each sample. The convolution layers, namely each layer of convolution layer in the convolution neural network, are composed of a plurality of convolution units, and the parameters of each convolution unit are obtained through optimization of a back propagation algorithm. The purpose of convolution operations is to extract different features of the input, and the first layer of convolution may only extract some low-level features such as edges, lines, and corners, and more layers of the network may iteratively extract more complex features from the low-level features.
In the embodiment of the application, in the construction process of the capacity prediction model, the nonlinear activation function is added after the full connection layer, and a nonlinear activation function is superimposed after the linear transformation, so that the multilayer network is prevented from being equivalent to a single-layer linear function, the identification capacity of the model is enhanced, the problem that the linear model cannot solve is solved, and therefore, larger learning and fitting capacity is obtained, and the model prediction precision is improved. According to the scheme, a deep learning technology is introduced, a capacity prediction neural network is constructed by using a full-connection layer and a convolution layer as bases, and a non-linear factor is added by introducing a Leaky-ReLU as a model to increase the non-linear fitting capacity. The model provided by the application has higher efficiency than a cyclic neural network model and a transducer model, and the model prediction accuracy is equivalent or better, so that the model is more suitable for being applied to production.
Further, in the decoder, the encoder output enters a plurality of stacked multi-layer perceptron residual modules to obtain an intermediate result I, and the H part after the result of the multi-layer perceptron residual modules of the future known covariates is intercepted is taken as an intermediate result II. And stacking the two intermediate results and sending the two intermediate results to a multi-layer perceptron residual error module.
Specifically, the real-time index data, the business data, the associated resource data and the application performance data are used as input parameters, and training is performed based on the capacity prediction model to obtain a training result. The capacity prediction is performed through the neural network model according to the capacity historical data to be predicted, auxiliary prediction time sequences (including business data, associated resource data, application performance data and the like), known future covariates and static covariates.
Step S40: and determining and pushing the predicted value of each capacity index based on the predicted result.
Specifically, identifying predicted values of each capacity index based on the training results; respectively comparing the predicted value of each capacity index with the associated resource data under the corresponding capacity index, and judging whether an overrun capacity index exists or not; when the overrun capacity index exists, information pushing is performed in a differentiated mode aiming at the overrun capacity index, and alarm information is triggered.
In the embodiment of the application, for example, if the predicted residual CPU demand is greater than the residual CPU distributable quantity, early warning is needed so that a user can clean the CPU conveniently to release more residual CPU distributable quantity, and the system is prevented from being out of limit and down or affecting the normal execution of user services.
In one embodiment, as shown in FIG. 4, a capacity prediction system is provided, wherein the system comprises: the acquisition unit is used for acquiring index data of the capacity to be predicted, and preprocessing the index data to obtain preprocessed index data; the processing unit is used for classifying basic data based on the preprocessed index data; the prediction unit is used for carrying out capacity prediction of a future preset time section in a capacity prediction model based on the basic data to obtain a prediction result; and the pushing unit is used for determining and pushing the predicted value of each capacity index based on the predicted result.
The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel can be provided with one or more than one, and the capacity prediction method is realized by adjusting kernel parameters.
The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.
An embodiment of the present application provides a storage medium having stored thereon a program which, when executed by a processor, implements the capacity prediction method described above.
The embodiment of the application provides a processor for running a program, wherein the capacity prediction method is executed when the program runs.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor a01, a network interface a02, a memory (not shown) and a database (not shown) connected by a system bus. Wherein the processor a01 of the computer device is adapted to provide computing and control capabilities. The memory of the computer device includes internal memory a03 and nonvolatile storage medium a04. The nonvolatile storage medium a04 stores an operating system B01, a computer program B02, and a database (not shown in the figure). The internal memory a03 provides an environment for the operation of the operating system B01 and the computer program B02 in the nonvolatile storage medium a04. The network interface a02 of the computer device is used for communication with an external terminal through a network connection. The computer program B02, when executed by the processor a01, implements a capacity prediction method.
It will be appreciated by those skilled in the art that the structure shown in FIG. 5 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, the capacity prediction system provided by the present application may be implemented in the form of a computer program that is executable on a computer device as shown in fig. 5. The memory of the computer device may store therein respective program modules constituting the capacity prediction system, and the computer program constituted by the respective program modules causes the processor to execute the steps in the capacity prediction method of the respective embodiments of the present application described in the present specification.
The present application also provides a computer program product adapted to perform the above-mentioned capacity prediction method when executed on a data processing device.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
Computer-readable media include both permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.
Claims (14)
1. A method of capacity prediction, the method comprising:
acquiring index data of capacity to be predicted, and preprocessing the index data to obtain preprocessed index data; wherein the index data includes: index history data, index real-time data, business data, associated resource data and application performance data;
classifying basic data based on the preprocessed index data;
based on the classified basic data, carrying out capacity prediction of a future preset time section in a capacity prediction model to obtain a prediction result;
and determining and pushing the predicted value of each capacity index based on the predicted result.
2. The method of claim 1, wherein the associated resource data comprises:
one or more of the total number of cores of the CPU, the allocated amount of the CPU, the remaining allocated amount of the CPU, the total amount of the memory, the allocated amount of the memory, and the remaining usable amount of the memory.
3. The method of claim 1, wherein the preprocessing the index data comprises:
performing one or more of outlier rejection processing, deduplication processing, index missing segment repair processing and data rejection processing on the index data;
and identifying the time sequence data, and carrying out future covariate extraction processing on the identified time sequence data.
4. The method according to claim 2, wherein the method further comprises:
performing capacity prediction model training, comprising:
splitting the index history data into training data and verification data, the training data being chronologically earlier than the verification data;
based on the training data, reading auxiliary prediction time sequences, known future covariates and static covariates corresponding to the same time period, and forming a training set together with the training data;
in a pre-constructed neural network, taking the data of the same period of verification data as a training target, and carrying out model training based on the training set to obtain an initial model;
comparing the training result of the initial model with the verification data, and carrying out model correction based on deviation, wherein each time model correction is completed, comparing the training result of the correction model with the verification data again until the difference value of the training result of the correction model and the verification data meets the error minimization condition, stopping model correction, and taking the correction model obtained by the latest correction as a capacity prediction model.
5. The method of claim 4, wherein the pre-constructed neural network comprises: input module, encoder, decoder and output module.
6. The method of claim 5, wherein the input data of the input module comprises:
a first portion of data comprising a capacity history sequence to be predicted and auxiliary prediction sequence data;
the second part of data is static covariate data after pretreatment;
and the third part of data is data obtained by passing the multi-layer perceptron residual error module through the future known covariates.
7. The method of claim 6, wherein the encoder is comprised of a stack of multiple multi-layer perceptron residual modules.
8. The method of claim 7, wherein the multi-layer perceptron residual module includes a short connection branch and a backbone branch;
the input data of the input module passes through a first full connection layer on the short connection branch to obtain a first intermediate result vector;
the input data of the input module passes through a second full-connection layer, a leakage type rectification linear unit activation function module, a third full-connection layer and a random inactivation layer one by one on the main branch to obtain a second intermediate result vector;
and the output vector of the input module is obtained by layer normalization based on the first intermediate result vector and the second intermediate result vector.
9. The method according to claim 2, wherein the performing capacity prediction of the future preset time zone in the capacity prediction model based on the classified basis data to obtain a prediction result comprises:
and taking the index real-time data, the service data, the associated resource data and the application performance data as input parameters, and training based on the capacity prediction model to obtain a training result.
10. The method according to claim 2, wherein determining and pushing the predicted value of each capacity index based on the predicted result comprises:
identifying predicted values of each capacity index based on the training results;
respectively comparing the predicted value of each capacity index with the associated resource data under the corresponding capacity index, and judging whether an overrun capacity index exists or not;
when the overrun capacity index exists, information pushing is performed in a differentiated mode aiming at the overrun capacity index, and alarm information is triggered.
11. A capacity prediction system, the system comprising:
the acquisition unit is used for acquiring index data of the capacity to be predicted, and preprocessing the index data to obtain preprocessed index data; wherein the index data includes: index history data, index real-time data, business data, associated resource data and application performance data;
the processing unit is used for classifying basic data based on the preprocessed index data;
the prediction unit is used for performing capacity prediction for a preset time section in a capacity prediction model based on the basic data to obtain a prediction result;
and the pushing unit is used for determining and pushing the predicted value of each capacity index based on the predicted result.
12. A processor configured to perform the capacity prediction method according to any one of claims 1 to 10.
13. A machine-readable storage medium having instructions stored thereon, which when executed by a processor cause the processor to be configured to perform the capacity prediction method according to any one of claims 1 to 10.
14. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the capacity prediction method according to any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310769144.2A CN116702044A (en) | 2023-06-27 | 2023-06-27 | Capacity prediction method, system, storage medium and processor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310769144.2A CN116702044A (en) | 2023-06-27 | 2023-06-27 | Capacity prediction method, system, storage medium and processor |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116702044A true CN116702044A (en) | 2023-09-05 |
Family
ID=87835632
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310769144.2A Pending CN116702044A (en) | 2023-06-27 | 2023-06-27 | Capacity prediction method, system, storage medium and processor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116702044A (en) |
-
2023
- 2023-06-27 CN CN202310769144.2A patent/CN116702044A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20190268283A1 (en) | Resource Demand Prediction for Distributed Service Network | |
| Polato et al. | Data-aware remaining time prediction of business process instances | |
| Shcherbakov et al. | Proactive and predictive maintenance of cyber-physical systems | |
| CN104423968B (en) | It designs the method for service logic, execute its server and storage medium | |
| CN111177485A (en) | Method, device and medium for predicting device failure based on parameter rule matching | |
| CN116467674B (en) | Intelligent fault processing fusion updating system and method for power distribution network | |
| CN112508695A (en) | Financial risk prediction method and device based on market risk and electronic equipment | |
| WO2018182442A1 (en) | Machine learning system and method for generating a decision stream and automonously operating device using the decision stream | |
| CN111177876B (en) | Community discovery method and device and electronic equipment | |
| CN114185761B (en) | Log acquisition method, device and equipment | |
| Overbeck et al. | Development and analysis of digital twins of production systems | |
| CN108711074A (en) | Business sorting technique, device, server and readable storage medium storing program for executing | |
| CN107203545B (en) | Data processing method and device | |
| Kildeeva et al. | Models and methods of forecasting and tasks distribution by performers in electronic document management systems | |
| CN116702044A (en) | Capacity prediction method, system, storage medium and processor | |
| KR20140146437A (en) | Apparatus and method for forecasting business performance based on patent information | |
| Pourbafrani et al. | Data-Driven Simulation In Process Mining: Introducing A Reference Model. | |
| CN114385705A (en) | Data importance identification method, device, equipment and medium | |
| US20230376796A1 (en) | Method and system for knowledge-based process support | |
| CN110210518B (en) | Method and device for extracting dimension reduction features | |
| CN114817209A (en) | Monitoring rule processing method and device, processor and electronic equipment | |
| CN116933451A (en) | Multi-dimensional architecture mapping relation model evaluation method, device and equipment based on Petri network | |
| CN116011698B (en) | Method, device, computer equipment and storage medium for determining unit combination | |
| KR102719797B1 (en) | Method and system for company future forecast | |
| CN111062816B (en) | Account asset supervision method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |