CN111459761A - Redis configuration method, device, storage medium and equipment - Google Patents
Redis configuration method, device, storage medium and equipment Download PDFInfo
- Publication number
- CN111459761A CN111459761A CN202010251295.5A CN202010251295A CN111459761A CN 111459761 A CN111459761 A CN 111459761A CN 202010251295 A CN202010251295 A CN 202010251295A CN 111459761 A CN111459761 A CN 111459761A
- Authority
- CN
- China
- Prior art keywords
- monitoring index
- redis
- linear
- model
- memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000012544 monitoring process Methods 0.000 claims abstract description 137
- 230000008859 change Effects 0.000 claims abstract description 41
- 238000012417 linear regression Methods 0.000 claims abstract description 31
- 238000004458 analytical method Methods 0.000 claims abstract description 28
- 230000015654 memory Effects 0.000 claims description 58
- 238000012549 training Methods 0.000 claims description 19
- 230000008569 process Effects 0.000 claims description 13
- 230000006870 function Effects 0.000 claims description 12
- 230000004044 response Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 6
- 230000008030 elimination Effects 0.000 claims description 5
- 238000003379 elimination reaction Methods 0.000 claims description 5
- 230000001419 dependent effect Effects 0.000 claims description 4
- 230000000737 periodic effect Effects 0.000 claims description 4
- 238000011156 evaluation Methods 0.000 claims description 3
- 238000012886 linear function Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 7
- 238000012545 processing Methods 0.000 abstract description 7
- 230000002411 adverse Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 12
- 238000012423 maintenance Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000003933 environmental pollution control Methods 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012731 temporal analysis Methods 0.000 description 1
- 238000000700 time series analysis Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present specification provides a method, apparatus, storage medium, and device for Redis configuration. According to the method, historical monitoring indexes in the Redis cluster are obtained, the historical monitoring indexes are linear or nonlinear according to the change trend of the historical monitoring indexes, a linear regression model or a time series model is correspondingly adopted to predict the predicted values of the monitoring indexes in a preset time period, whether the Redis cluster is expanded or not is analyzed according to the predicted values, accordingly, trend analysis of the Redis is achieved, adjustment is timely made according to the predicted trend, reasonable configuration is obtained, and adverse effects of business caused by lagging alarm processing are solved.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method, an apparatus, a storage medium, and a device for Redis configuration.
Background
Redis (Remote Dictionary service) is a key-value storage system, and is often used as a cache. At present, in the using process of Redis, the problems that the Redis is unavailable due to insufficient memory capacity or the Redis instance is insufficient in performance, so that the response time is too long or even overtime is caused, and the service use is influenced are often encountered. In the related art, a common solution is to set an alarm threshold, trigger an alarm when the memory or performance reaches the alarm threshold, and then manually intervene to expand the memory. This approach often results in processing that is significantly behind the on-line demand.
Disclosure of Invention
To overcome the problems in the related art, the present specification provides a method, apparatus, storage medium, and device for Redis configuration.
According to a first aspect of embodiments herein, there is provided a method of Redis configuration, the method comprising:
acquiring a historical monitoring index in a Redis cluster, and judging whether the change trend of the historical monitoring index is linear;
when the change trend of the historical monitoring index is linear, predicting the predicted value of the monitoring index in a preset time period through a linear regression model, wherein the linear regression model is obtained by training sample data of the monitoring index based on a linear regression algorithm; when the change trend of the historical monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a time series model, wherein the time series model is obtained by training sample data of the monitoring index based on a time series method;
and analyzing whether the Redis cluster is expanded or not according to the predicted value.
In some examples, the monitoring indicator includes at least one of: memory usage capacity of the machine, throughput of the machine, CPU usage of the machine, user response time.
In some examples, the determining whether the change trend of the historical monitoring index is linear includes:
fitting the change trend of the historical monitoring indexes based on a simple linear regression algorithm, and calculating the goodness of fit;
when the goodness of fit exceeds a first preset value, judging that the change trend of the historical monitoring index is linear;
and when the goodness of fit does not exceed a first preset value, judging that the change trend of the historical monitoring index is nonlinear.
In some examples, the monitoring indicator includes a memory usage capacity of the machine; when it is determined to expand the Redis cluster according to the predictive value analysis, the method further comprises:
increase usable memory, or use memory elimination strategies.
In some examples, the monitoring indicator includes at least one of: throughput of the machine, CPU usage of the machine, and user response time; when it is determined to expand the Redis cluster according to the predictive value analysis, the method further comprises:
and building a new Redis slave library and updating the configuration of the distribution flow.
In some examples, the linear regression model includes a ridge regression model, and the training process of the ridge regression model includes:
taking the monitoring index as a dependent variable, taking the corresponding time when the monitoring index is collected as an independent variable, and obtaining a ridge regression model by using a ridge regression algorithm;
determining a mean square error corresponding to the ridge regression model by using an evaluation function;
and when the value of the mean square error is smaller than a second preset value, determining that the ridge regression model is qualified.
In some examples, the time series model includes a Prophet model; the training process of the Prophet model comprises the following steps:
and taking the monitoring index and the time corresponding to the acquired monitoring index as sample data of the Prophet model, and respectively determining a breakpoint, a piecewise linear function, a periodic function and a holiday function corresponding to the breakpoint to obtain the trained Prophet model.
According to a second aspect of embodiments herein, there is provided a Redis-configured apparatus, the apparatus comprising:
the acquisition module is used for acquiring historical monitoring indexes in the Redis cluster and judging whether the change trend of the historical monitoring indexes is linear or not;
the prediction module is used for predicting the predicted value of the monitoring index in a preset time period through a linear regression model when the change trend of the historical monitoring index is linear, and the linear regression model is obtained by training sample data of the monitoring index based on a linear regression algorithm; when the change trend of the historical monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a time series model, wherein the time series model is obtained by training sample data of the monitoring index based on a time series method;
and the analysis module is used for analyzing whether the Redis cluster is expanded or not according to the predicted value.
According to a third aspect of embodiments of the present specification, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs any one of the methods of the embodiments of the specification.
According to a fourth aspect of embodiments herein, there is provided a computer apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements any of the methods in the embodiments herein when executing the program.
The technical scheme provided by the embodiment of the specification can have the following beneficial effects:
in the embodiment of the specification, a method, a device, a storage medium and equipment for Redis configuration are disclosed. According to the method, historical monitoring indexes in the Redis cluster are obtained, the historical monitoring indexes are linear or nonlinear according to the change trend of the historical monitoring indexes, a linear regression model or a time series model is correspondingly adopted to predict the predicted values of the monitoring indexes in a preset time period, whether the Redis cluster is expanded or not is analyzed according to the predicted values, accordingly, trend analysis of the Redis is achieved, adjustment is timely made according to the predicted trend, reasonable configuration is obtained, and adverse effects of business caused by lagging alarm processing are solved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the specification.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present specification and together with the description, serve to explain the principles of the specification.
FIG. 1 is a flow chart illustrating a method of Redis configuration according to an exemplary embodiment of the present description;
FIG. 2 is a hardware configuration diagram of a computer device in which a Redis configured apparatus according to an embodiment of the present disclosure is located;
FIG. 3 is a block diagram of an apparatus of a Redis configuration shown in the present specification according to an exemplary embodiment;
FIG. 4 is a schematic diagram of the architecture of a Redis configuration system shown in the present specification in accordance with an exemplary embodiment;
FIG. 5A is a schematic diagram illustrating an architecture of a first subsystem in a Redis configuration system in accordance with an exemplary embodiment of the present description;
FIG. 5B is a schematic diagram of the architecture of a second subsystem in a Redis configuration system shown in accordance with an exemplary embodiment;
fig. 5C is a schematic diagram illustrating an effect of obtaining a predicted value of a monitoring index according to an exemplary embodiment.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the specification, as detailed in the appended claims.
The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the description. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of the present specification. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
Redis (Remote Dictionary Server) is a key-value storage system, can support read/write operations tens of thousands of times per second, has performance far exceeding that of a database, and also supports configurations such as cluster, distributed, master-slave synchronization and the like, and is often used as a cache. At present, in the using process of Redis, the problems that the Redis is unavailable due to insufficient memory capacity or the Redis instance is insufficient in performance, so that the response time is too long or even overtime is caused, and the service use is influenced are often encountered. In the related art, a common solution is to set an alarm threshold, trigger an alarm when the memory or performance reaches the alarm threshold, and then manually intervene to expand the memory. As can be appreciated, this tends to cause the process to fall significantly behind the on-line demand.
The following provides a detailed description of examples of the present specification.
As shown in fig. 1, fig. 1 is a flow chart illustrating a method of Redis configuration according to an exemplary embodiment of the present description, the method comprising:
in step 101, obtaining a historical monitoring index in a Redis cluster, and judging whether the change trend of the historical monitoring index is linear;
it is generally recognized that a Redis Cluster has three modes, a master-slave mode, a Sentinel mode, and a Cluster mode. A Redis cluster may be composed of a plurality of Redis instances, and in the master-slave mode, the instances are divided into two types, i.e., a master database and a slave database, or simply referred to as a master database and a slave database. The master library can provide read-write service, and when data change is caused by read-write operation, the data can be automatically synchronized to the slave library; the slave library typically provides read-only services and receives data synchronized by the master library. The Sentinel mode is established on the basis of a master-slave mode, a Sentinel process is provided for monitoring the working state of a master library in the Redis cluster, and when the master library fails, the master library and a slave library can be switched, so that the high availability of a system is ensured. The Cluster mode can be regarded as a combination of a master-slave mode and a Sentinel mode, wherein a plurality of Redis instances are interconnected in a network and share data, a client can be connected with any one master library for reading and writing, and the slave library does not provide services and only serves as a standby. In some examples, the Redis cluster in this step may refer to a Redis cluster in a master-slave mode, which is not limited in this specification.
In some examples, the historical monitoring index in this step refers to a historical monitoring index of each instance in the Redis cluster, where the monitoring index may include at least one of the following: memory usage capacity of the machine, throughput of the machine, CPU usage of the machine, user response time. The memory usage capacity of the machine may reflect the memory state of the Redis instance, and the throughput of the machine, the CPU usage of the machine, or the user response time may reflect the performance state of the Redis instance. It is understood that the memory of the Redis instance is generally configured, and the performance of the Redis instance is limited, so that when the memory state or the performance state of a certain Redis instance continuously approaches the upper limit in a period of time, it indicates that the processing pressure of the Redis instance is large, and capacity needs to be expanded.
In some examples, the determining whether the variation trend of the historical monitoring index is linear in this step may include: fitting the change trend of the historical monitoring indexes based on a simple linear regression algorithm, and calculating the goodness of fit; when the goodness of fit exceeds a first preset value, judging that the change trend of the historical monitoring index is linear; and when the goodness of fit does not exceed a first preset value, judging that the change trend of the historical monitoring index is nonlinear. The historical monitoring index in this step may be collected according to a certain period, and the period may be set according to the needs of a specific scene, and may be one minute, two minutes, one hour, one day, and so on. The concept of goodness of fit is the degree of fit of the regression line to the predicted value, typically by the coefficient R2As a statistic of the metric, the closer the value of the coefficient to 1, the better the fitting degree of the regression line to the predicted value. Historical monitoring of acquisition, assuming a one hour periodThe indexes are memory use capacities of machines corresponding to four points to eight points, which are respectively 4.2G, 4.8G, 5.2G, 5.5G and 5.7G, for example, sample data can be recorded as (4, 4.2), (5, 4.8), (6, 5.2), (7, 5.5) and (8, 5.7), the sample data are substituted for fitting based on a simple linear regression algorithm, the simple linear regression algorithm can include a unary linear regression and a least square method, a coefficient of decision is calculated after fitting to be 0.96, when a first preset value is 0.97, the goodness of fit 0.96 does not exceed the first preset value 0.97, and the change trend of the historical monitoring index is judged to be nonlinear.
In step 102, when the change trend of the historical monitoring index is linear, predicting a predicted value of the monitoring index in a preset time period through a linear regression model, wherein the linear regression model is obtained by training sample data of the monitoring index based on a linear regression algorithm; when the change trend of the historical monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a time series model, wherein the time series model is obtained by training sample data of the monitoring index based on a time series method;
after the change trend of the historical monitoring index is judged to be linear or nonlinear, the predicted value of the monitoring index in a preset time period is predicted through the corresponding model. In some examples, the linear regression model in this step may refer to: ridge regression model. The ridge regression model can be implemented based on R language, and also can be implemented based on SPSS (Statistical Product and service Solutions). The training process of the ridge regression model comprises the following steps: taking the monitoring index as a dependent variable, taking the corresponding time when the monitoring index is collected as an independent variable, and obtaining a ridge regression model by using a ridge regression algorithm; determining a mean square error corresponding to the ridge regression model by using an evaluation function; and when the value of the mean square error is smaller than a second preset value, determining that the ridge regression model is qualified. In general, after sample data corresponding to independent variables and dependent variables are input, ridge parameter K values are selected so that coefficient R can be determined2The value of (a) is as large as possible and is as stable as possible, and a function-form ridge regression model can be obtained after the value is determined; the greenThe mean square error MSE of the regression model may be determined by the following equation:
wherein, observediPredicted as sample dataiSubstituting the independent variables into the ridge regression model to obtain data, wherein n is the number of samples;
the Mean Square Error (MSE) evaluates the change degree of data, and the smaller the value of the MSE is, the better the accuracy of the model in describing sample data is shown, and the model also has better effect in predicting the future trend of the sample data. It can be understood that the second preset value can be set according to the type of the monitoring index, the required accuracy of the model, and other requirements of specific scenarios.
The embodiment of the specification aims at the nonlinear monitoring index and is based on a time series method for modeling and predicting. The time series method, also called time series analysis, is a theory and method for establishing a mathematical model through curve fitting and parameter estimation according to time series data obtained by system observation, and is commonly used in aspects of national economy macro control, enterprise operation management, weather forecast, environmental pollution control and the like. In some examples, the time series model indicated in the embodiments of the present specification includes a Prophet model, which is an open-source time series prediction tool and can be applied to build the Prophet model. In some examples, the training process of the Prophet model includes: and taking the monitoring index and the time corresponding to the acquired monitoring index as sample data of the Prophet model, and respectively determining a breakpoint, a piecewise linear function, a periodic function and a holiday function corresponding to the breakpoint to obtain the trained Prophet model. In general, the principle of Prophet is to analyze various time series characteristics: periodicity, trending, holiday effects, and partial outliers. In the aspect of trend, the method supports the addition of change points and realizes piecewise linear fitting; in terms of period, it uses Fourier series to establish periodic function, in terms of holidays and emergencies, holidays can be specified, and several days before and after holidays can be specified. Therefore, the established Prophet model has better prediction accuracy when used for predicting the future trend of the monitoring index.
When the predicted value of the monitoring index in the preset time period is predicted, the corresponding model is input only by taking the preset time period as the input quantity, and the output quantity obtained later is the required predicted value. The preset time period here may be set according to the predicted demand.
Of course, in some examples, the linear regression model mentioned in this step may also be an L asso regression model, and the time series model may be a L STM (L ong Short Term Memory) model, which may be selected according to the selected monitoring index, and this specification does not limit this.
And step 103, analyzing whether the Redis cluster is expanded or not according to the predicted value.
After the predicted value of the monitoring index in the preset time period is obtained, how to configure Redis can be analyzed according to the predicted value. Generally, different thresholds may be set for different monitoring indexes, where the threshold is determined based on a percentage of an upper limit of the monitoring index, and when a predicted value exceeds the threshold, the analysis determines to expand the Redis. For example, the monitoring index includes a memory usage capacity of the machine, and the threshold of the monitoring index may be set to eighty percent of the upper limit, if the maximum memory usage capacity of the allocated machine is 10G and the threshold thereof is 8G, when the predicted value of the monitoring index in the preset time period exceeds 8G, the analysis determines to expand the Redis.
The embodiment of the present specification provides the following capacity expansion schemes for two corresponding monitoring indexes reflecting the Redis memory state and the performance state:
aiming at the memory state, the monitoring index comprises the memory use capacity of the machine; when the capacity of the Redis cluster is determined to be expanded according to the predictive value analysis, firstly trying to increase the available memory, and when the available memory is zero, using a memory elimination strategy to eliminate the key value pair with the minimum access frequency, thereby expanding the Redis memory;
for the performance state, the monitoring indicator includes at least one of: throughput of the machine, CPU usage of the machine, and user response time; and when the Redis cluster is determined to be expanded according to the predicted value analysis, a new slave library is built, the configuration is updated after the building is completed, the flow is distributed in a balanced manner, namely, the service flow is divided into a part to the new slave library, and the purpose of expanding the Redis performance is achieved.
After capacity expansion, the state records before and after capacity expansion can be recorded and notified to related personnel.
In addition, when the predicted value of the monitoring index is obtained, whether Redis capacity expansion is needed or not is analyzed, and whether Redis capacity expansion can be conducted or not is also analyzed. For example, the monitoring index includes the CPU utilization of the machine, and when the predicted value shows that the CPU utilization of the machine is continuously higher than 95%, which is far higher than the set threshold value of 60%, an alarm may be triggered to notify the relevant personnel, and the relevant personnel intervene to troubleshoot the problem.
In the embodiment of the description, the historical monitoring index in the Redis cluster is acquired, the historical monitoring index is linear or nonlinear according to the change trend of the historical monitoring index, the predicted value of the monitoring index in the preset time period is predicted by correspondingly adopting a ridge regression model or a time sequence model, and whether the Redis cluster is subjected to capacity expansion is analyzed according to the predicted value, so that the trend analysis of the Redis is realized, the adjustment is timely made according to the predicted trend, the elastic capacity expansion is realized, the reasonable configuration is obtained, and the adverse effect of service caused by the backward alarm processing is solved.
Those skilled in the art will appreciate that the methods of the embodiments of the present description may also be applied to achieve elastic scalping of Redis.
Corresponding to the embodiments of the foregoing method, the present specification also provides embodiments of a Redis configured apparatus and a terminal applied thereto.
The embodiments of the apparatus configured by Redis in this specification can be applied to a computer device, such as a server or a terminal device. The device embodiments may be implemented by software, or by hardware, or by a combination of hardware and software. The software implementation is taken as an example, and as a logical device, the device is formed by reading corresponding computer program instructions in the nonvolatile memory into the memory for operation through the processor in which the file processing is located. From a hardware aspect, as shown in fig. 2, a hardware structure diagram of a computer device in which a device configured in Redis according to this description embodiment is located is shown in fig. 2, except for the processor 510, the memory 530, the network interface 520, and the nonvolatile memory 540 shown in fig. 2, a server or an electronic device in which the device 531 is located in this embodiment may also include other hardware according to an actual function of the computer device, which is not described again.
Accordingly, the embodiments of the present specification also provide a computer storage medium, in which a program is stored, and the program, when executed by a processor, implements the method in any of the above embodiments.
Embodiments of the present description may take the form of a computer program product embodied on one or more storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having program code embodied therein. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of the storage medium of the computer include, but are not limited to: phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technologies, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic tape storage or other magnetic storage devices, or any other non-transmission medium, may be used to store information that may be accessed by a computing device.
As shown in fig. 3, fig. 3 is a block diagram of an apparatus of a Redis configuration shown in the present specification according to an exemplary embodiment, the apparatus comprising:
the obtaining module 31 is configured to obtain a historical monitoring index in a Redis cluster, and determine whether a change trend of the historical monitoring index is linear;
the prediction module 32 is configured to predict a predicted value of the monitoring index in a preset time period through a linear regression model when the change trend of the historical monitoring index is linear, where the linear regression model is obtained by training sample data of the monitoring index based on a linear regression algorithm; when the change trend of the historical monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a time series model, wherein the time series model is obtained by training sample data of the monitoring index based on a time series method;
and the analysis module 33 is configured to analyze whether to expand the Redis cluster according to the predicted value.
The implementation process of the functions and actions of each module in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, wherein the modules described as separate parts may or may not be physically separate, and the parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution in the specification. One of ordinary skill in the art can understand and implement it without inventive effort.
In order to provide a further understanding of the method of the embodiments of the present disclosure, a further description of an application example is provided below:
as shown in fig. 4, fig. 4 is a schematic diagram of an architecture of a Redis configuration system shown in this specification according to an exemplary embodiment, and the Redis configuration system applies the method of the embodiment of the specification to configure the Redis. The architecture of the Redis configuration system mainly comprises the following parts: the system comprises a monitoring index acquisition unit, a trend analysis unit, an analysis decision unit, a configuration unit, a Redis agent unit, a Redis cluster and an operation and maintenance platform.
The monitoring index acquisition unit is configured to acquire a historical monitoring index in a Redis cluster, and specifically, may acquire each monitoring index of each instance in the Redis cluster, where the monitoring index acquisition unit includes at least one of the following: the method comprises the steps of reporting collected data to a Kafka cluster by using the memory usage capacity of a machine, the throughput of the machine, the CPU usage rate of the machine and the user response time, and storing the collected data into a ClickHouse database by the Kafka cluster through a consumption queue mode.
The trend analysis unit is used for analyzing the service conditions of the memory and the performance through an algorithm and giving a trend numerical value of a period of time in the future, specifically, the trend analysis unit can acquire a historical monitoring index from a ClickHouse database and judge whether the change trend of the historical monitoring index is linear, and if the change trend is linear, the predicted value of the monitoring index in a preset period of time is predicted through a ridge regression model; and if the non-linearity exists, predicting the predicted value of the monitoring index in a preset time period through a Prophet model. Similarly, sample data of the ridge regression model and the Prophet model during training is obtained from the ClickHouse database.
The analysis decision unit is used for analyzing whether the Redis needs to be expanded according to the predicted value obtained by the trend analysis unit, if so, informing the configuration unit to perform corresponding expansion operation, and simultaneously informing the operation and maintenance platform of the expansion state; and whether the Redis can be expanded or not is analyzed, and if the Redis cannot be expanded, an alarm is triggered to notify the operation and maintenance platform.
The configuration unit is used for executing capacity expansion operation on the Redis cluster, specifically, the capacity expansion operation is divided into two types, namely memory capacity expansion operation and performance capacity expansion operation, the memory capacity expansion is to try to increase the available memory, and when the available memory is zero, a memory elimination strategy is used to eliminate the key value pair with the least access frequency; and performance capacity expansion is to build a new slave library, inform a Redis agent unit after the building is completed, update Redis configuration and balance distribution flow.
The Redis agent layer is used for load balancing and read-write separation of the Redis.
Specifically, the Redis configuration system may be considered to include a first subsystem and a second subsystem, where the first subsystem is for a memory direction, and the second subsystem is for a performance direction, as shown in fig. 5A and 5B, fig. 5A is a schematic diagram of an architecture of the first subsystem shown in this embodiment; fig. 5B is a schematic diagram of an architecture of a second subsystem shown in an embodiment of the present specification. Wherein:
in a first subsystem, a monitoring index acquisition unit monitors and acquires historical monitoring indexes of a Redis cluster and reports the historical monitoring indexes to a Kalfa cluster, wherein the monitoring indexes comprise the memory use capacity of a machine; the Kalfa cluster stores the reported data, and stores the result to a ClickHouse database through a consumption queue mode; the ClickHouse database provides data for the trend analysis unit; the trend analysis unit analyzes the use condition of the memory through an algorithm and provides trend data of a period of time in the future; the analysis decision unit analyzes whether the Redis needs to be expanded according to the trend data, if yes, the analysis decision unit informs the configuration unit of carrying out corresponding expansion operation, and simultaneously informs the operation and maintenance platform of the expansion state, analyzes whether the Redis can be expanded, and if the Redis cannot be expanded, triggers an alarm to inform the operation and maintenance platform; the configuration unit executes memory capacity expansion operation, tries to increase the available memory first, and uses a memory elimination strategy to eliminate the key value pair with the least access frequency when the available memory is zero.
In the second subsystem, the logical relations of the monitoring index acquisition unit, the Kalfa cluster, the ClickHouse, the trend analysis unit and the analysis decision unit are the same as those of the first subsystem, except that the monitoring index comprises at least one of the following items: throughput of the machine, CPU usage of the machine, user response time; the configuration unit executes performance capacity expansion operation, builds a new slave library, informs the Redis agent unit after the building is completed, updates Redis configuration and balances and distributes flow; and the Redis agent layer performs load balancing and read-write separation on the Redis.
The trend analysis unit of the first subsystem analyzes the use condition of the memory through an algorithm, and the trend analysis unit of the second subsystem analyzes the use condition of the performance through the algorithm. The algorithm may be a linear regression algorithm or a time series algorithm, and specifically, the algorithm may be a method of judging whether a change trend of a historical monitoring index is linear, and if the change trend is linear, predicting a predicted value of the monitoring index in a preset time period through a ridge regression model; and if the non-linearity exists, predicting the predicted value of the monitoring index in a preset time period through a Prophet model. As shown in fig. 5C, fig. 5C is a schematic diagram illustrating an effect of obtaining a predicted value of a monitoring index according to an exemplary embodiment, where the monitoring index is a QPS (query per second) of a machine, belongs to an index reflecting performance, and is one of important parameters of throughput of the machine; in the graph, a curve 1 is a curve for monitoring collected sample data, a point P represents a current time point, a curve 2 is a curve obtained by fitting the sample data before the current time point according to an algorithm, a point Q is a predicted value of a QPS of a machine, which is predicted by the algorithm, corresponding to a time point 5:30, and the predicted value is 101.8K, while an actual value of the QPS is 68.6K at the time point 5:30 and an actual value of the QPS is 103.4K at the time point 5:40 according to data collected later, so that the system has high feasibility and accuracy in decision making according to the predicted value.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Other embodiments of the present description will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This specification is intended to cover any variations, uses, or adaptations of the specification following, in general, the principles of the specification and including such departures from the present disclosure as come within known or customary practice within the art to which the specification pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the specification being indicated by the following claims.
It will be understood that the present description is not limited to the precise arrangements described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.
The above description is only a preferred embodiment of the present disclosure, and should not be taken as limiting the present disclosure, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (10)
1. A method of Redis configuration, the method comprising:
acquiring a historical monitoring index in a Redis cluster, and judging whether the change trend of the historical monitoring index is linear;
when the change trend of the historical monitoring index is linear, predicting the predicted value of the monitoring index in a preset time period through a linear regression model, wherein the linear regression model is obtained by training sample data of the monitoring index based on a linear regression algorithm; when the change trend of the historical monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a time series model, wherein the time series model is obtained by training sample data of the monitoring index based on a time series method;
and analyzing whether the Redis cluster is expanded or not according to the predicted value.
2. The method of claim 1, wherein the monitoring metrics comprise at least one of: memory usage capacity of the machine, throughput of the machine, CPU usage of the machine, user response time.
3. The method of claim 1, wherein the determining whether the trend of the historical monitoring indicator is linear comprises:
fitting the change trend of the historical monitoring indexes based on a simple linear regression algorithm, and calculating the goodness of fit;
when the goodness of fit exceeds a first preset value, judging that the change trend of the historical monitoring index is linear;
and when the goodness of fit does not exceed a first preset value, judging that the change trend of the historical monitoring index is nonlinear.
4. The method of claim 1, wherein the monitoring metrics include a memory usage capacity of the machine; when it is determined to expand the Redis cluster according to the predictive value analysis, the method further comprises:
increase usable memory, or use memory elimination strategies.
5. The method of claim 1, wherein the monitoring metrics comprise at least one of: throughput of the machine, CPU usage of the machine, and user response time; when it is determined to expand the Redis cluster according to the predictive value analysis, the method further comprises:
and building a new Redis slave library and updating the configuration of the distribution flow.
6. The method of claim 1, wherein the linear regression model comprises a ridge regression model, and wherein the training process of the ridge regression model comprises:
taking the monitoring index as a dependent variable, taking the corresponding time when the monitoring index is collected as an independent variable, and obtaining a ridge regression model by using a ridge regression algorithm;
determining a mean square error corresponding to the ridge regression model by using an evaluation function;
and when the value of the mean square error is smaller than a second preset value, determining that the ridge regression model is qualified.
7. The method of claim 1, wherein the time series model comprises a Prophet model; the training process of the Prophet model comprises the following steps:
and taking the monitoring index and the time corresponding to the acquired monitoring index as sample data of the Prophet model, and respectively determining a breakpoint, a piecewise linear function, a periodic function and a holiday function corresponding to the breakpoint to obtain the trained Prophet model.
8. An apparatus of Redis configuration, the apparatus comprising:
the acquisition module is used for acquiring historical monitoring indexes in the Redis cluster and judging whether the change trend of the historical monitoring indexes is linear or not;
the prediction module is used for predicting the predicted value of the monitoring index in a preset time period through a linear regression model when the change trend of the historical monitoring index is linear, and the linear regression model is obtained by training sample data of the monitoring index based on a linear regression algorithm; when the change trend of the historical monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a time series model, wherein the time series model is obtained by training sample data of the monitoring index based on a time series method;
and the analysis module is used for analyzing whether the Redis cluster is expanded or not according to the predicted value.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the method of any one of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010251295.5A CN111459761B (en) | 2020-04-01 | 2020-04-01 | Redis configuration method, device, storage medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010251295.5A CN111459761B (en) | 2020-04-01 | 2020-04-01 | Redis configuration method, device, storage medium and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111459761A true CN111459761A (en) | 2020-07-28 |
| CN111459761B CN111459761B (en) | 2024-03-01 |
Family
ID=71685775
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010251295.5A Active CN111459761B (en) | 2020-04-01 | 2020-04-01 | Redis configuration method, device, storage medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111459761B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112598291A (en) * | 2020-12-25 | 2021-04-02 | 中国农业银行股份有限公司 | Prophet-based operation and maintenance intelligent scheduling method and device |
| CN113342561A (en) * | 2021-06-18 | 2021-09-03 | 上海哔哩哔哩科技有限公司 | Task diagnosis method and system |
| CN114051000A (en) * | 2021-11-17 | 2022-02-15 | 中国工商银行股份有限公司 | Service flow switching method and device based on time series model |
| CN114637656A (en) * | 2022-05-13 | 2022-06-17 | 飞狐信息技术（天津）有限公司 | Redis-based monitoring method, device, storage medium and equipment |
| CN115080356A (en) * | 2022-07-21 | 2022-09-20 | 支付宝(杭州)信息技术有限公司 | Abnormity warning method and device |
| CN115474165A (en) * | 2021-05-24 | 2022-12-13 | 中移动金融科技有限公司 | Self-monitoring dynamic adjustment method and system for short message service |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150332311A1 (en) * | 2014-05-15 | 2015-11-19 | Coupons.Com Incorporated | Optimized placement of digital offers |
| CN107800551A (en) * | 2016-08-31 | 2018-03-13 | 北京优朋普乐科技有限公司 | Redis group systems and its method, the client for improving reliability |
| CN108920324A (en) * | 2018-06-08 | 2018-11-30 | 广东轩辕网络科技股份有限公司 | The method of the trend analysis of information technoloy equipment memory capacity and early warning, system and electronic device |
| CN109842563A (en) * | 2017-11-24 | 2019-06-04 | 中国电信股份有限公司 | Content delivery network flow dispatching method, device and computer readable storage medium |
| CN110086649A (en) * | 2019-03-19 | 2019-08-02 | 深圳壹账通智能科技有限公司 | Detection method, device, computer equipment and the storage medium of abnormal flow |
| CN110912737A (en) * | 2019-11-14 | 2020-03-24 | 国网浙江省电力有限公司信息通信分公司 | Dynamic perception performance early warning method based on hybrid model |
-
2020
- 2020-04-01 CN CN202010251295.5A patent/CN111459761B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150332311A1 (en) * | 2014-05-15 | 2015-11-19 | Coupons.Com Incorporated | Optimized placement of digital offers |
| CN107800551A (en) * | 2016-08-31 | 2018-03-13 | 北京优朋普乐科技有限公司 | Redis group systems and its method, the client for improving reliability |
| CN109842563A (en) * | 2017-11-24 | 2019-06-04 | 中国电信股份有限公司 | Content delivery network flow dispatching method, device and computer readable storage medium |
| CN108920324A (en) * | 2018-06-08 | 2018-11-30 | 广东轩辕网络科技股份有限公司 | The method of the trend analysis of information technoloy equipment memory capacity and early warning, system and electronic device |
| CN110086649A (en) * | 2019-03-19 | 2019-08-02 | 深圳壹账通智能科技有限公司 | Detection method, device, computer equipment and the storage medium of abnormal flow |
| CN110912737A (en) * | 2019-11-14 | 2020-03-24 | 国网浙江省电力有限公司信息通信分公司 | Dynamic perception performance early warning method based on hybrid model |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112598291A (en) * | 2020-12-25 | 2021-04-02 | 中国农业银行股份有限公司 | Prophet-based operation and maintenance intelligent scheduling method and device |
| CN112598291B (en) * | 2020-12-25 | 2023-10-13 | 中国农业银行股份有限公司 | Prophet-based operation and maintenance intelligent scheduling method and device |
| CN115474165A (en) * | 2021-05-24 | 2022-12-13 | 中移动金融科技有限公司 | Self-monitoring dynamic adjustment method and system for short message service |
| CN115474165B (en) * | 2021-05-24 | 2024-05-07 | 中移动金融科技有限公司 | Self-monitoring dynamic adjustment method and system for short message service |
| CN113342561A (en) * | 2021-06-18 | 2021-09-03 | 上海哔哩哔哩科技有限公司 | Task diagnosis method and system |
| CN114051000A (en) * | 2021-11-17 | 2022-02-15 | 中国工商银行股份有限公司 | Service flow switching method and device based on time series model |
| CN114637656A (en) * | 2022-05-13 | 2022-06-17 | 飞狐信息技术（天津）有限公司 | Redis-based monitoring method, device, storage medium and equipment |
| CN114637656B (en) * | 2022-05-13 | 2022-09-20 | 飞狐信息技术（天津）有限公司 | Redis-based monitoring method and device, storage medium and equipment |
| CN115080356A (en) * | 2022-07-21 | 2022-09-20 | 支付宝(杭州)信息技术有限公司 | Abnormity warning method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111459761B (en) | 2024-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111459761B (en) | Redis configuration method, device, storage medium and equipment | |
| US6311175B1 (en) | System and method for generating performance models of complex information technology systems | |
| CN110413227B (en) | Method and system for predicting remaining service life of hard disk device on line | |
| US20210019211A1 (en) | Method and device for determining a performance indicator value for predicting anomalies in a computing infrastructure from values of performance indicators | |
| US7107187B1 (en) | Method for modeling system performance | |
| US20130318022A1 (en) | Predictive Analytics for Information Technology Systems | |
| CN105323111A (en) | Operation and maintenance automation system and method | |
| CN111045894B (en) | Database abnormality detection method, database abnormality detection device, computer device and storage medium | |
| CN106776288B (en) | A kind of health metric method of the distributed system based on Hadoop | |
| CN110289994B (en) | Cluster capacity adjusting method and device | |
| Luo et al. | Intelligent Virtual Machine Provisioning in Cloud Computing. | |
| Duan et al. | Multi-phase sequential preventive maintenance scheduling for deteriorating repairable systems | |
| CN111563022B (en) | Centralized memory monitoring method and device | |
| US20220245010A1 (en) | Time-series anomaly detection using an inverted index | |
| US20240028830A1 (en) | Machine learning -based selection of metrics for anomaly detection | |
| CN114896121A (en) | Monitoring method and device of distributed processing system | |
| CN108021484A (en) | The extension method and its system of disk life expectancy value in cloud service system | |
| CN111849544A (en) | Hydrocracking product quality automatic control method, device and storage | |
| CN114398235A (en) | Memory recovery trend early warning device and method based on fusion learning and hypothesis testing | |
| US11556451B2 (en) | Method for analyzing the resource consumption of a computing infrastructure, alert and sizing | |
| CN117971488A (en) | Storage management method and related device for distributed database cluster | |
| CN116701491A (en) | Method and device for determining static threshold of operation index, storage medium and processor | |
| CN116780755A (en) | Cloud resource scheduling operation system | |
| CN117971384A (en) | Automatic operation and maintenance method based on container and big data | |
| CN112600705A (en) | Method for automatic operation and maintenance of network equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |