CN117390465A

CN117390465A - Workload prediction method

Info

Publication number: CN117390465A
Application number: CN202311686615.XA
Authority: CN
Inventors: 路亚彬; 任景彪; 李晨光; 赵伟
Original assignee: Tianjin Nankai University General Data Technologies Co ltd
Current assignee: Tianjin Nankai University General Data Technologies Co ltd
Priority date: 2023-12-11
Filing date: 2023-12-11
Publication date: 2024-01-12
Anticipated expiration: 2043-12-11
Also published as: CN117390465B

Abstract

The invention provides a workload prediction method, which comprises the following steps: acquiring current load information; inputting current load information into a pre-trained prediction model; the prediction model comprises a linear model and a nonlinear model; calculating the ratio of the nonlinear model to the prediction result of the linear model, and if the ratio is greater than a specified ratio threshold, outputting the prediction result of the nonlinear model; otherwise, outputting the prediction result of the linear model. In the invention, the prediction result with higher accuracy can be obtained in various workload predictions such as periodic, peak-type, gradual change and the like, and the prediction method does not adopt an extra neural network architecture, so that the training time is shorter and the training cost is lower.

Description

Workload prediction method

Technical Field

The invention belongs to the technical field of computers, and particularly relates to a workload prediction method.

Background

The goal of workload prediction is to enable the system to predict what the workload of an application will look like in the future over a period of time in the future. Because the workload of the actual application is never static. The system may then select an optimization to prepare based on this predicted workload, such as by indexing in advance.

It should be noted that unsupervised workload prediction is challenging. The first point is that queries of an application may have distinct arrival rates. Thus, an effective predictive model must be able to identify and describe each of these arrival rate patterns. The second point is that the composition and number of queries in the application workload may change over time. The ability of a DBMS to implement workload prediction depends largely on its knowledge of queries and patterns in the application workload. If the workload deviates too much from the past, the predictive model becomes inaccurate and must be retrained.

In practice, the modes commonly used for the workload include the following ones, which are periodic, peak-type, gradual, and the like. The periodic embodiment is in many interactive programs, such as bus taking APP, which acquire data by interacting with humans, and thus the workload has a fixed law, and more queries occur in the morning and afternoon during peak hours. Such cycles have a significant period, such as 24 hours. Another common workload pattern is that the amount of queries increases over time, but the amount of queries increases crazy over the last few days, which is typical in applications with a specific expiration date. Yet another mode is that the workload evolves gradually over time. Generally this is the result of a change in user and a change in query arrival rate pattern, such as the addition of new functionality. It is difficult to accurately predict the load of the above modes at the same time, and most of the work is concentrated on one to two kinds.

Previous work has rarely modeled the workload, and more has focused on modeling the resource requirements of the system than directly representing the workload itself. They are more concerned with the scheduling of resources such as CPUs than with the actual query patterns within the database. There are some approaches to simulate the performance of a DBMS by answering hypothetical questions about changes in the OLTP workload and thereby modeling the workload as a mixture of different types of transactions with a fixed ratio. Still other efforts may use hidden Markov models or regression models to predict how the workload will change over time. The method combines the advantages of software and hardware, aggregates the query into a fixed template, aggregates the templates into clusters, and predicts the query effect by using the clusters.

Early work also modeled database workloads using more formal methods with predefined transaction types and arrival rates. All of these methods have the disadvantage that some only use lossy compression schemes that only maintain high-level statistics, such as average query delay and resource utilization. Other work assumes that the tools provide a static workload, or they only generate new models when the workload changes, so that the amount of queries and the workload cannot be captured. The method for clustering according to the template has the defects of high data requirement (clustering), difficult training (load change needs to be retrained) and the like.

Disclosure of Invention

Therefore, the invention aims to provide a workload prediction method to solve the problems of low prediction precision and high training cost of the existing workload prediction algorithm.

In order to achieve the above purpose, the technical scheme of the invention is realized as follows:

a method of workload prediction, comprising the steps of:

acquiring current load information;

inputting current load information into a pre-trained prediction model; the prediction model comprises a linear model and a nonlinear model;

calculating the ratio of the nonlinear model to the prediction result of the linear model, and if the ratio is greater than a specified ratio threshold, outputting the prediction result of the nonlinear model; otherwise, outputting the prediction result of the linear model.

Further, the training method of the prediction model comprises the following steps:

preprocessing the existing workload, and abstracting the query operation with a certain format into a query template;

inputting the condition columns of a plurality of query templates into a frequent item set mining algorithm, and screening out frequent item query templates with occurrence frequencies exceeding a given threshold value by the frequent item set mining algorithm;

and constructing a new cluster by target query operation represented by the frequent item query template, and inputting the new cluster into the prediction model as a training sample.

Further, the method for constructing a new cluster by the target query operation represented by the frequent item query template comprises the following steps:

pruning the existing workload, and removing query operations except the target query operation;

classifying the target query operation according to the frequent item query templates;

and splicing the classified target query operations according to the sequence of the frequent item query templates to form the new cluster.

Further, the method for constructing the prediction model comprises the following steps:

and shearing the new cluster according to the sequence of the frequent item query templates, and simultaneously predicting the arrival rate of a plurality of the frequent item query templates.

Further, the linear model is an XGboost model; the nonlinear model is a KR-kernel model.

An electronic device comprising a processor and a memory communicatively coupled to the processor for storing instructions executable by the processor for performing the workload prediction method described above.

A server comprising at least one processor, and a memory communicatively coupled to the processor, the memory storing instructions executable by the at least one processor, the instructions being executable by the processor to cause the at least one processor to perform the workload prediction method described above.

A computer readable storage medium storing a computer program which when executed by a processor implements the workload prediction method described above.

Compared with the prior art, the workload prediction method provided by the invention has the following advantages:

the workload prediction method provided by the invention predicts the workload by using the linear model and the nonlinear model, and determines the final output result by the ratio of the two prediction results, and the prediction results with higher accuracy can be obtained in various workload predictions such as periodic type, peak type, gradual change type and the like.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute an undue limitation on the invention. In the drawings:

FIG. 1 illustrates a flow chart of a workload prediction method according to an inventive embodiment of the present invention;

FIG. 2 is a schematic diagram of a prediction process of a prediction model in an inventive embodiment of the present invention;

fig. 3 is a schematic diagram of a new cluster construction process in an embodiment of the invention.

Detailed Description

It should be noted that, without conflict, the embodiments of the present invention and features of the embodiments may be combined with each other.

The invention will be described in detail below with reference to the drawings in connection with embodiments.

FIG. 1 illustrates a flow chart of a workload prediction method of an embodiment of the present disclosure. As shown in fig. 1, the workload prediction method S100 includes the steps of:

step S110, current load information is acquired.

The load information is a batch of inquiry operations with time labels and inquiry volume, the time labels record the time period when the inquiry operations are executed, and the inquiry volume records the inquiry operation times in the time period. The time stamp and the query volume combine to form an arrival rate, where the arrival rate is specifically the arrival of a certain type of query operation over a period of time in the future.

In a real-world scenario, there may be tens of millions of query operations per second, and it is not practical to record the arrival rates of these query operations and predict them separately, because the resources are too huge, and where a significant portion of the query operations occur only a few times, there is no benefit in optimizing too many of these small queries for the overall load. Therefore, it is a primary task to select representative, e.g., more frequently occurring, query operations for workload prediction.

Step S120, inputting current load information into a pre-trained prediction model; the predictive models include linear models and nonlinear models.

In an embodiment of the present application, the prediction model is trained by the following method: 1) Preprocessing the existing workload, and abstracting the query with a certain format into a query template;

2) Extracting condition columns appearing in the query templates, and dynamically screening frequent item query templates by adopting an FP-Growth algorithm;

3) Pruning and reconstructing the existing workload to construct a new cluster;

4) And inputting the new cluster as a training sample into a prediction model to predict the arrival rate of the existing workload.

The following describes three steps one by way of example:

1) For table a in the database, for a query operation shaped as select name from A where sex =rule and age=18, the actual data appearing in the query operation, such as rule, 18, is replaced with a x to characterize a query template. The reason for this is: workload prediction is commonly used in the field of database optimization-databases make their own optimizations based on upcoming queries to better adapt to the queries, thereby speeding up the processing efficiency of the queries, which in many scenarios are made without the need for actual data. In addition, for a column such as age that has a large number of discrete values, all queries on the column may involve tens of values, and there may be hundreds or even more on a larger column, tracking and predicting tracks for these different values is costly and less profitable, so it is more appropriate to abstract the query to a query template, such as select name from A where sex = and age=.

2) And screening the extracted query templates, wherein the frequent item query templates are screened, and determining the query operation represented by the frequent item query templates as a new cluster. The column where the query condition is located in the query operation is defined as a condition column, and for example, the conditions in select name from A where sex =malee and age=18 are set and age. Extracting a condition column from the query template extracted in the step 1), and inputting the condition column into an FP-Growth algorithm. The FP-Growth algorithm is a method for quickly filtering frequent items, and given a threshold k, the FP-Growth algorithm can find a query template with a number of occurrences greater than k in a short time. The query templates are the query templates with the largest occurrence times in the whole workload, and the arrival rate rules of the query templates influence the arrival rate rules of the whole workload to a great extent, so that query operations represented by the query templates can be used as representative new clusters, and the arrival rate results of the whole workload can be represented by predicting the arrival rate of the new clusters.

3) And reconstructing and pruning the existing workload, removing query operations with low occurrence frequency, classifying the rest query operations according to the query templates, and splicing the classified query operations according to the sequence of the query templates to form a new cluster. The classification and splicing aims at redistributing the query operations in the new cluster according to all the query templates so as to avoid mutual interference among the query templates. Because different query templates have different arrival rates, predicting the arrival rate of a new cluster is essentially predicting the arrival rates of all query templates separately.

4) The new clusters are used as training samples of the prediction model, the prediction model cuts the input new clusters according to the sequence of the frequent item query templates, reversely restores the query operation represented by each frequent item query template, and predicts the arrival rate of each frequent item query template respectively.

In this embodiment, the prediction model includes a linear model and a nonlinear model.

A good predictive model is one that is capable of adequate applicability and accuracy, both of which are indispensable. In a real-world scenario, different query operations may have different arrival rules. For example, for a bus scene, when a bus occupant gets on, a bus company needs to check the balance of the bus card of the bus occupant and deduct fees, so that a query cluster for the bus card may be periodic with days, i.e. people go to work and get off work, and the process is repeated. In this scenario, the predictive model needs to record data of several cycles in the past according to the past law to predict the current possible query operation. Still other scenarios, such as the annual growth of the seller, although data from the past several months continue to be low, may be due to the sudden increase in sales over several days of the event, which in turn requires a predictive model with extremely high responsiveness, requiring attention to the data in the short term, and reacting to immediate changes in the data.

However, a single model cannot meet all of these requirements. A common model cannot handle both linear relationships between inputs and outputs, such as periodic rules reached by a query operation in a bus scenario, and nonlinear relationships between inputs and outputs, such as peak-type rules of a growth in the end of year. In contrast, different models are suited for different scenarios. When the input and output of the model have linear relation, the traditional linear model fitting has high accuracy, short training time and convenient fitting, but the actual data sometimes show nonlinear relation, and although some nonlinear models are good at learning more complex nonlinear relation, the nonlinear models consume long time, require more training data and cannot be interfered by the linear data.

An ideal situation should be to use the optimal method correspondingly in different time ranges according to different forms of the workload, so the prediction model provided by the invention is actually an integrated model. The integrated model comprises a linear model and a nonlinear model, and the arrival rate of the clusters under all conditions cannot be predicted by the linear model and the nonlinear model, so that the method selects to simultaneously incorporate the linear model and the nonlinear model into the same integrated model for comprehensive prediction, and the integrated model is compared to obtain an optimal result, which is helpful for improving the applicability of the method under more scenes and improving the prediction accuracy under corresponding scenes.

In this embodiment, the main body of the linear model is an XGboost model, also called a linear autoregressive model, which is often used in the statistics and time series prediction literature, because they have a closed form solution, i.e. they do not require an extra optimization step to find a globally optimal solution, which has the disadvantage that the above mentioned peak situation cannot be handled, and they cannot be simulated without having seen peak data. For example, if the sales of a store per month is to be predicted for 1-11 months, the sales increase per month due to good store operation, then the XGboost model can predict sales for several months in the future well after knowing sales for the previous months, but if the sales increase is caused by the activity of "double 11" in 11 months, a high peak value occurs for several months, the accuracy of the XGboost model drops because it cannot predict such peak value which is not seen, and its prediction result is more conservative. Since peak form is a very common working model in real life, the invention proposes to build an integrated prediction model, i.e. the prediction model not only comprises a linear model, but also needs to comprise a nonlinear model, and a scene without periodicity, such as a peak scene, is processed so that the prediction model has enough multi-scene applicability.

In this embodiment, the nonlinear model is a KR-kernel model, also called a kernel regression model, which is necessary for the present invention to better predict the query peak condition in advance. The kernel regression model is a nonlinear variant of the linear regression model that uses a Nadaraya-Watson estimator to achieve its nonlinearity without iterative training. The prediction of a given input is a weighted average of the training outputs, where the weights decrease as the distance between the given input and the corresponding training input decreases. Kernel regression is a non-parametric method, meaning that it does not assume a specific functional form, but rather that it only assumes that the function is smooth, which provides it with the necessary flexibility to model different non-linear functions between input and output. Thus, even if the peak occurs only a few times or even does not occur in the past, it can predict when the peak will occur repeatedly in the future. However, the kernel regression model performs worse than the linear model in terms of average prediction accuracy. This is why the present invention is to make predictions in combination with both models at the same time.

Step S130, calculating the ratio of the nonlinear model to the prediction result of the linear model, and if the ratio is greater than a specified threshold, outputting the prediction result of the nonlinear model; otherwise, outputting the prediction result of the linear model.

In this embodiment, the input of the prediction model is cluster load information of the query operation to be predicted, that is, current load information and the arrival rate of the new cluster, and the output is the arrival rate of each frequent query template and the query operation cluster represented by the same in a future period of time.

When data is input, the prediction model trains a linear model and a nonlinear model internally at the same time, and gives prediction results of the two models respectively. The predictive model internally compares the predicted results of the linear model and the nonlinear model. The nonlinear model is suitable for nonlinear data prediction scenes, but when it is applied to linear scenes, the prediction results are unstable and the average accuracy is low. For this feature we give a ratio threshold P and calculate the ratio of the nonlinear model to the predicted result of the linear model. When the ratio is greater than P, the load generates peak input, so that the nonlinear model prediction result is increased rapidly along with the input, and the linear model prediction result is more conservative, so that the ratio of the two is greater than the threshold value P, and the prediction model selects the prediction result of the nonlinear model as output. Otherwise, if the ratio is smaller than P, the nonlinear model prediction result is in the fluctuation range, and peak input does not occur, at the moment, the linear model average precision is higher, and the prediction model selects the prediction result of the linear model as output.

In general, the prediction model judges whether the current scene is a linear scene or a nonlinear scene according to the proportion of the prediction results, thereby adopting the prediction effect of the corresponding model. The prediction model supports internal dual-model parallel training, increases the application range of the model under the condition of reducing training time as much as possible, and enhances the prediction capability under various scenes.

The prediction model only adopts an XGboost model and a kernel regression model, does not introduce excessive other models, and does not adopt an additional neural network architecture. Even the integrated model, compared with other deep learning methods, the method has less expenditure and lower training difficulty, and can support large-scale business.

Furthermore, the method supports the actual verification within the database. The method selects AutoAdmin algorithm of Microsoft to carry out index establishment simulation, and confirms reliability of query prediction. According to the method, based on the traditional index recommendation method, the time prediction range horizons is set, and the improvement brought by dynamic index establishment is determined to be greater than that brought by the index establishment in advance by comparing performance advantages and disadvantages of the conditions of not establishing the index, establishing the index in advance and establishing the index dynamically in a period of time in the future, so that the method is effective and reliable.

Those of ordinary skill in the art will appreciate that the elements and method steps of each example described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the elements and steps of each example have been described generally in terms of functionality in the foregoing description to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

In the several embodiments provided in this application, it should be understood that the disclosed methods and systems may be implemented in other ways. For example, the above-described division of units is merely a logical function division, and there may be another division manner when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted or not performed. The units may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present invention.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention, and are intended to be included within the scope of the appended claims and description.

The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.

Claims

1. A method of workload prediction, characterized by: the method comprises the following steps:

acquiring current load information;

2. The workload prediction method according to claim 1, wherein: the training method of the prediction model comprises the following steps:

3. The workload prediction method according to claim 2, wherein: the method for constructing a new cluster by the target query operation represented by the frequent item query template comprises the following steps:

4. A workload prediction method according to claim 3, characterized in that: the training method of the prediction model comprises the following steps:

5. The workload prediction method according to claim 1, wherein: the linear model is an XGboost model; the nonlinear model is a KR-kernel model.

6. An electronic device comprising a processor and a memory communicatively coupled to the processor for storing processor-executable instructions, characterized in that: the processor is configured to perform the workload prediction method according to any one of the preceding claims 1-5.

7. A server, characterized by: comprising at least one processor, and a memory communicatively coupled to the processor, the memory storing instructions executable by the at least one processor to cause the at least one processor to perform the workload prediction method of any of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that: the computer program, when executed by a processor, implements the workload prediction method of any of claims 1-5.