CN111459761B - Redis configuration method, device, storage medium and equipment - Google Patents
Redis configuration method, device, storage medium and equipment Download PDFInfo
- Publication number
- CN111459761B CN111459761B CN202010251295.5A CN202010251295A CN111459761B CN 111459761 B CN111459761 B CN 111459761B CN 202010251295 A CN202010251295 A CN 202010251295A CN 111459761 B CN111459761 B CN 111459761B
- Authority
- CN
- China
- Prior art keywords
- monitoring index
- redis
- model
- predicted value
- change trend
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000012544 monitoring process Methods 0.000 claims abstract description 133
- 230000008859 change Effects 0.000 claims abstract description 39
- 238000012417 linear regression Methods 0.000 claims abstract description 32
- 238000004458 analytical method Methods 0.000 claims abstract description 28
- 230000015654 memory Effects 0.000 claims description 58
- 238000012549 training Methods 0.000 claims description 19
- 230000006870 function Effects 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 12
- 230000004044 response Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 6
- 230000001419 dependent effect Effects 0.000 claims description 4
- 230000000737 periodic effect Effects 0.000 claims description 4
- 238000011156 evaluation Methods 0.000 claims description 3
- 238000012886 linear function Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 abstract description 8
- 230000000694 effects Effects 0.000 abstract description 7
- 230000002411 adverse Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 12
- 238000012423 maintenance Methods 0.000 description 5
- 238000013459 approach Methods 0.000 description 3
- 230000008030 elimination Effects 0.000 description 3
- 238000003379 elimination reaction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 238000012300 Sequence Analysis Methods 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 238000003933 environmental pollution control Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The specification provides a method, a device, a storage medium and equipment for Redis configuration. In the method, a historical monitoring index in the Redis cluster is obtained, linearity or nonlinearity is presented according to the change trend, a linear regression model or a time sequence model is correspondingly adopted to predict a predicted value of the monitoring index in a preset time period, so that whether the Redis cluster is expanded or not is analyzed according to the predicted value, the trend analysis of the Redis is realized, adjustment is timely made according to the predicted trend, reasonable configuration is obtained, and adverse effects of business due to alarm processing lag are further solved.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method, an apparatus, a storage medium, and a device for Redis configuration.
Background
Redis (Remote Dictionary Server, remote dictionary service) is a key-value storage system, often used as a cache. At present, in the use process of Redis, the problems of unavailability caused by insufficient capacity of the Redis memory or overlong response time even overtime caused by insufficient performance of a Redis instance, thereby influencing service use, are often encountered. In the related art, a common solution is to set an alarm threshold, trigger an alarm when the memory or performance reaches the alarm threshold, and then manually intervene to expand the memory. This approach often results in processing that is severely behind the on-line requirements.
Disclosure of Invention
To overcome the problems in the related art, the present specification provides a method, apparatus, storage medium, and device for Redis configuration.
According to a first aspect of embodiments of the present specification, there is provided a method of dis configuration, the method comprising:
acquiring a history monitoring index in a Redis cluster, and judging whether the change trend of the history monitoring index is linear or not;
when the change trend of the historical monitoring index is linear, predicting a predicted value of the monitoring index in a preset time period through a linear regression model, wherein the linear regression model is obtained after training sample data of the monitoring index based on a linear regression algorithm; when the change trend of the historical monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a time sequence model, wherein the time sequence model is obtained after training sample data of the monitoring index based on a time sequence method;
and analyzing whether the Redis cluster is subjected to capacity expansion or not according to the predicted value.
In some examples, the monitoring indicator includes at least one of: memory usage capacity of the machine, throughput of the machine, CPU usage of the machine, user response time.
In some examples, determining whether the trend of the historical monitoring index is linear includes:
fitting the change trend of the history monitoring index based on a simple linear regression algorithm, and calculating the fitting goodness;
when the fitting goodness exceeds a first preset value, judging that the change trend of the history monitoring index is linear;
and when the goodness of fit does not exceed a first preset value, judging that the change trend of the history monitoring index is nonlinear.
In some examples, the monitoring metrics include memory usage capacity of the machine; when the capacity expansion of the Redis cluster is determined according to the predicted value analysis, the method further comprises:
increasing the available memory, or using a memory culling strategy.
In some examples, the monitoring indicator includes at least one of: throughput of the machine, CPU usage of the machine, and user response time; when the capacity expansion of the Redis cluster is determined according to the predicted value analysis, the method further comprises:
a new Redis slave library is built and the configuration of the allocated traffic is updated.
In some examples, the linear regression model includes a ridge regression model, and the training process of the ridge regression model includes:
taking the monitoring index as a dependent variable, taking the corresponding time when the monitoring index is acquired as an independent variable, and obtaining a ridge regression model by using a ridge regression algorithm;
determining a mean square error corresponding to the ridge regression model by using an evaluation function;
and when the value of the mean square error is smaller than a second preset value, determining that the ridge regression model is qualified.
In some examples, the time series model includes a propset model; the training process of the propset model comprises the following steps:
and respectively determining a breakpoint, a piecewise linear function, a periodic function and a holiday function corresponding to the breakpoint by taking the monitoring index and the time corresponding to the collected monitoring index as sample data of the Prophet model, so as to obtain the trained Prophet model.
According to a second aspect of embodiments of the present specification, there is provided an apparatus of a Redis configuration, the apparatus comprising:
the acquisition module is used for acquiring the history monitoring index in the Redis cluster and judging whether the change trend of the history monitoring index is linear or not;
the prediction module is used for predicting the predicted value of the monitoring index in a preset time period through a linear regression model when the change trend of the historical monitoring index is linear, and the linear regression model is obtained after training the sample data of the monitoring index based on a linear regression algorithm; when the change trend of the historical monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a time sequence model, wherein the time sequence model is obtained after training sample data of the monitoring index based on a time sequence method;
and the analysis module is used for analyzing whether the Redis cluster is subjected to capacity expansion according to the predicted value.
According to a third aspect of the embodiments of the present description, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements any of the methods of the embodiments of the present description.
According to a fourth aspect of embodiments of the present specification, there is provided a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements any of the methods of the embodiments of the present specification when the program is executed.
The technical scheme provided by the embodiment of the specification can comprise the following beneficial effects:
in an embodiment of the specification, a method, a device, a storage medium and equipment for Redis configuration are disclosed. In the method, a historical monitoring index in the Redis cluster is obtained, linearity or nonlinearity is presented according to the change trend, a linear regression model or a time sequence model is correspondingly adopted to predict a predicted value of the monitoring index in a preset time period, so that whether the Redis cluster is expanded or not is analyzed according to the predicted value, the trend analysis of the Redis is realized, adjustment is timely made according to the predicted trend, reasonable configuration is obtained, and adverse effects of business due to alarm processing lag are further solved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and together with the description, serve to explain the principles of the specification.
FIG. 1 is a flow chart illustrating a method of Redis configuration according to an exemplary embodiment of the present description;
FIG. 2 is a hardware configuration diagram of a computer device in which the Redis configuration apparatus according to the embodiment of the present disclosure is located;
FIG. 3 is a block diagram of an apparatus of one Redis configuration shown in accordance with an exemplary embodiment of the present description;
FIG. 4 is a schematic diagram of the architecture of a Redis configuration system as illustrated in this specification according to an example embodiment;
FIG. 5A is a schematic diagram of the architecture of a first subsystem in a Redis configuration system, as illustrated in the present specification, according to an example embodiment;
FIG. 5B is a schematic diagram of the architecture of a second subsystem in a Redis configuration system, as illustrated in the present specification, according to an example embodiment;
fig. 5C is a schematic diagram showing the effect of obtaining the predicted value of the monitor index according to an exemplary embodiment of the present specification.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present specification. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present description as detailed in the accompanying claims.
The terminology used in the description presented herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the description. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.
Redis (Remote Dictionary Server, remote dictionary service) is a key-value storage system, can support read/write operations of hundreds of thousands of times per second, has performance far exceeding that of a database, and also supports configuration such as cluster, distributed, master-slave synchronization and the like, and is often used as a cache. At present, in the use process of Redis, the problems of unavailability caused by insufficient capacity of the Redis memory or overlong response time even overtime caused by insufficient performance of a Redis instance, thereby influencing service use, are often encountered. In the related art, a common solution is to set an alarm threshold, trigger an alarm when the memory or performance reaches the alarm threshold, and then manually intervene to expand the memory. As can be appreciated, this tends to lead to processing that is severely behind the on-line requirements.
Next, embodiments of the present specification will be described in detail.
As shown in fig. 1, fig. 1 is a flowchart illustrating a method of dis configuration according to an exemplary embodiment of the present specification, the method including:
101, acquiring a history monitoring index in a Redis cluster, and judging whether the change trend of the history monitoring index is linear or not;
redis clusters are generally considered to have three modes, master-slave, sentinel and Cluster. A dis cluster may be composed of multiple dis instances, which are classified into two types, a master database and a slave database, also referred to simply as a master library and a slave library, in a master-slave mode. The master library can provide read-write service, and when the read-write operation causes data change, the data can be automatically synchronized to the slave library; the slave library typically provides only read services and receives data synchronized by the master library. The Sentinel (Sentinel) mode is based on a master-slave mode, and a Sentinel process is provided for monitoring the working state of a master library in a Redis cluster, so that when the master library fails, the switching between the master library and a slave library can be realized, and the high availability of the system is ensured. The Cluster mode can be regarded as a combination of a master-slave mode and a Sentinel mode, wherein a plurality of Redis instance networks are connected, data are shared, a client can be connected with any one of the master libraries for reading and writing, and the slave libraries do not provide service and only serve as standby. In some examples, the Redis cluster in this step may refer to a Redis cluster in a master-slave mode, which is, of course, not limited in this specification.
In some examples, the historical monitoring index in this step refers to the historical monitoring index of each instance in the dis cluster, where the monitoring index may include at least one of the following: memory usage capacity of the machine, throughput of the machine, CPU usage of the machine, user response time. The memory usage capacity of the machine may reflect the memory state of the Redis instance, and the throughput of the machine, CPU usage of the machine, or user response time may reflect the performance state of the Redis instance. It will be appreciated that the memory of a Redis instance is generally configured and its performance is limited, so when the memory state or performance state of a certain Redis instance continues to approach an upper limit for a period of time, it indicates that the processing pressure of the Redis instance is high, and expansion is required.
In some examples, determining whether the trend of the change in the history monitor indicator is linear in this step may include: fitting the change trend of the history monitoring index based on a simple linear regression algorithm, and calculating the fitting goodness; when the fitting goodness exceeds a first preset value, judging that the change trend of the history monitoring index is linear; and when the goodness of fit does not exceed a first preset value, judging that the change trend of the history monitoring index is nonlinear. The history monitoring index in this step may be collected according to a certain period, and the period may be set according to the needs of a specific scene, and may be one minute, two minutes, one hour, one day, or the like. The goodness of fit concept is the degree of fit of the regression line to the predicted value, typically by a determinable coefficient R 2 As a statistic of the measure, the closer the value of the determinable coefficient is to 1, the better the fitting degree of the regression line to the predicted value is explained. Assuming that the period is one hour, the obtained historical monitoring index is the memory usage capacity of the machine corresponding to four to eight points, which is respectively 4.2G, 4.8G, 5.2G, 5.5G and 5.7G, for example, the sample data can be recorded as (4,4.2), (5,4.8), (6,5.2), (7,5.5) and (8,5.7), the sample data are substituted for the simple linear regression algorithm to perform fitting, the simple linear regression algorithm can comprise a unitary linear regression and a least square method, the calculated determinable coefficient after fitting is 0.96, when the first preset value is 0.97, the fitting goodness of fit is 0.96 and does not exceed the first preset value of 0.97, and the change trend of the historical monitoring index is judged to be nonlinear.
102, when the change trend of the historical monitoring index is linear, predicting a predicted value of the monitoring index in a preset time period through a linear regression model, wherein the linear regression model is obtained after training sample data of the monitoring index based on a linear regression algorithm; when the change trend of the historical monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a time sequence model, wherein the time sequence model is obtained after training sample data of the monitoring index based on a time sequence method;
and after judging whether the change trend of the historical monitoring index is linear or nonlinear, predicting the predicted value of the monitoring index in a preset time period through a corresponding model. In some examples, the linear regression model in this step may refer to: a ridge regression model. The ridge regression model here can be implemented based on the R language, or based on SPSS (Statistical Product and Service Solutions ). The training process of the ridge regression model comprises the following steps: taking the monitoring index as a dependent variable, taking the corresponding time when the monitoring index is acquired as an independent variable, and obtaining a ridge regression model by using a ridge regression algorithm; determining a mean square error corresponding to the ridge regression model by using an evaluation function; and when the value of the mean square error is smaller than a second preset value, determining that the ridge regression model is qualified. In colloquial terms, after inputting the independent variable and the sample data corresponding to the dependent variable, the value of the ridge parameter K is selected to make the coefficient R determinable 2 The values of the (2) are as large as possible, and are as stable as possible, and the ridge regression model in the function form can be obtained after the determination; the mean square error MSE of the ridge regression model can be determined by the following equation:
wherein, the observed i Is sample data, predicted i Substituting the independent variable into the data obtained after the ridge regression model, wherein n is the number of samples;
the degree of change of the data is evaluated by the MSE, and the smaller the value of the MSE is, which shows that the model has better accuracy in describing the sample data, and has better effect in predicting the future trend of the sample data. It will be appreciated that the second preset value may be set according to the type of monitoring indicator, the accuracy required for the model, and other specific scenario requirements.
The embodiments of the present specification are directed to monitoring indicators that are nonlinear, modeled and predicted based on a time series approach. The time sequence method, also called time sequence analysis, is a theory and method for establishing a mathematical model through curve fitting and parameter estimation according to time sequence data obtained by system observation, and is commonly used in the aspects of national economy macro control, enterprise management, weather forecast, environmental pollution control and the like. In some examples, the time series model indicated in the embodiments of the present specification includes a propset model, which is an open source time series prediction tool that can be used to build the propset model. In some examples, the training process of the propset model includes: and respectively determining a breakpoint, a piecewise linear function, a periodic function and a holiday function corresponding to the breakpoint by taking the monitoring index and the time corresponding to the collected monitoring index as sample data of the Prophet model, so as to obtain the trained Prophet model. Colloquially, the principle of Prophet is to analyze various time series characteristics: periodicity, trending, holiday effects, and partial outliers. In the aspect of trend, the method supports the addition of change points to realize piecewise linear fitting; in terms of periodicity, it uses a fourier series to build a periodic function, and in terms of holidays and emergencies, holidays can be specified, and several days in tandem. Thus, the established propset model has better prediction precision when being used for predicting the future trend of the monitoring index.
When the predicted value of the monitoring index in the preset time period is predicted, the preset time period is only required to be used as input quantity, the corresponding model is input, and the obtained output quantity is the required predicted value. The preset time period here may be set according to the predicted demand.
Of course, in some examples, the linear regression model mentioned in this step may also be a Lasso regression model, and the time series model may be an LSTM (Long Short Term Memory, long-short-term memory network) model, which may be selected according to the monitoring index selected, which is not limited in this specification.
And in step 103, analyzing whether the Redis cluster is subjected to capacity expansion according to the predicted value.
After obtaining the predicted value of the monitoring index in the preset time period, how to configure the Redis according to the predicted value analysis. In general, different thresholds may be set for different monitoring indicators, where the thresholds are determined based on a percentage of the upper limit of the monitoring indicator, and when the predicted value exceeds the threshold, the analysis determines to expand the Redis. For example, the monitoring index includes a memory usage capacity of the machine, and the threshold of the monitoring index may be set to eighty percent of the upper limit, if the maximum memory usage capacity of the machine is allocated to be 10G, the threshold is 8G, and when the predicted value of the monitoring index in the preset time period exceeds 8G, the analysis determines that the Redis is expanded.
The embodiment of the specification provides the following expansion scheme aiming at the two monitoring indexes reflecting the Redis memory state and the performance state:
for the memory state, the monitoring index comprises the memory use capacity of the machine; when the capacity expansion of the Redis cluster is determined according to the analysis of the predicted value, firstly attempting to increase the usable memory, and when the usable memory is zero, using a memory elimination strategy to eliminate the key value pair with the minimum access frequency, thereby expanding the Redis memory;
for the performance state, the monitoring index includes at least one of: throughput of the machine, CPU usage of the machine, and user response time; when the capacity expansion of the Redis cluster is determined according to the predicted value analysis, a new slave library is built, the configuration is updated after the building is completed, the traffic is distributed in an equalizing mode, namely, the traffic is separated into a part of the new slave library, and the purpose of expanding the Redis performance is achieved.
After capacity expansion, the state before and after capacity expansion can be recorded and notified to related personnel.
In addition, when the predicted value of the monitoring index is obtained, it is not only analyzed whether the capacity of the Redis is required to be expanded, but also whether the capacity of the Redis can be expanded. For example, the monitoring index comprises the CPU utilization rate of the machine, when the predicted value shows that the CPU utilization rate of the machine is continuously higher than 95%, the predicted value far exceeds the set threshold value of 60%, and an alarm can be triggered to inform related personnel, so that the related personnel intervene in troubleshooting the problem.
According to the embodiment of the specification, the historical monitoring index in the Redis cluster is obtained, linearity or nonlinearity is presented according to the change trend of the historical monitoring index, the predicted value of the monitoring index in the preset time period is correspondingly predicted by adopting a ridge regression model or a time sequence model, so that whether the Redis cluster is expanded or not is analyzed according to the predicted value, the trend analysis of the Redis is realized, the adjustment is timely made according to the predicted trend, the elastic expansion is realized, the reasonable configuration is obtained, and the adverse effect of the service caused by the lag of alarm processing is further solved.
Those skilled in the art will appreciate that the methods of the embodiments of the present description may also be applied to achieve elastic contraction of Redis.
Corresponding to the embodiments of the foregoing method, the present specification also provides embodiments of the apparatus for dis configuration and the terminal to which the apparatus is applied.
The embodiment of the Redis configuration apparatus of the present specification may be applied to a computer device, such as a server or a terminal device. The apparatus embodiments may be implemented by software, or may be implemented by hardware or a combination of hardware and software. Taking software implementation as an example, the device in a logic sense is formed by reading corresponding computer program instructions in the nonvolatile memory into the memory through a processor of the file processing where the device is located. In terms of hardware, as shown in fig. 2, a hardware structure diagram of a computer device where a device configured by Redis in this embodiment of the present disclosure is located is shown in fig. 2, and in addition to the processor 510, the memory 530, the network interface 520, and the nonvolatile memory 540 shown in fig. 2, a server or an electronic device where the device 531 is located in the embodiment may generally include other hardware according to the actual function of the computer device, which is not described herein again.
Accordingly, the present specification embodiment also provides a computer storage medium having a program stored therein, which when executed by a processor, implements the method in any of the above embodiments.
Embodiments of the present description may take the form of a computer program product embodied on one or more storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having program code embodied therein. Computer-usable storage media include both permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to: phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, may be used to store information that may be accessed by the computing device.
As shown in fig. 3, fig. 3 is a block diagram of an apparatus of a dis configuration, which is illustrated in the present specification according to an exemplary embodiment, the apparatus includes:
the obtaining module 31 is configured to obtain a history monitoring indicator in the Redis cluster, and determine whether a variation trend of the history monitoring indicator is linear;
the prediction module 32 is configured to predict a predicted value of the monitoring index in a preset time period through a linear regression model when the change trend of the historical monitoring index is linear, where the linear regression model is obtained after training sample data of the monitoring index based on a linear regression algorithm; when the change trend of the historical monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a time sequence model, wherein the time sequence model is obtained after training sample data of the monitoring index based on a time sequence method;
and an analysis module 33, configured to analyze whether to perform capacity expansion on the dis cluster according to the predicted value.
The implementation process of the functions and roles of each module in the above device is specifically shown in the implementation process of the corresponding steps in the above method, and will not be described herein again.
For the device embodiments, reference is made to the description of the method embodiments for the relevant points, since they essentially correspond to the method embodiments. The apparatus embodiments described above are merely illustrative, wherein the modules illustrated as separate components may or may not be physically separate, and the components shown as modules may or may not be physical, i.e., may be located in one place, or may be distributed over a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purposes of the present description. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
For a more complete understanding of the method of the embodiments of the present disclosure, a further description of an example application will follow:
as shown in fig. 4, fig. 4 is a schematic diagram of an architecture of a dis configuration system according to an exemplary embodiment of the present disclosure, where the dis configuration system configures a dis by applying the method of the embodiment of the present disclosure. The architecture of the Redis configuration system mainly comprises the following parts: the system comprises a monitoring index acquisition unit, a trend analysis unit, an analysis decision unit, a configuration unit, a Redis agent unit, a Redis cluster and an operation and maintenance platform.
The monitoring index collection unit is configured to obtain a history monitoring index in the Redis cluster, and specifically may collect each monitoring index of each instance in the Redis cluster, where the monitoring index includes at least one of the following: the method comprises the steps of reporting collected data to a Kafka cluster, and storing the collected data into a ClickHouse database through a mode of a consumption queue by the Kafka cluster.
The trend analysis unit is used for analyzing the use condition of the memory and the performance through an algorithm to give trend values of a future period, specifically, the trend analysis unit can be used for acquiring a history monitoring index from a ClickHouse database, judging whether the change trend of the history monitoring index is linear or not, and if the change trend of the history monitoring index is linear, predicting a predicted value of the history monitoring index in a preset period through a ridge regression model; and if the monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a Prophet model. Likewise, sample data for the ridge regression model and propset model at the time of training is obtained from the clickHouse database.
The analysis decision unit is used for analyzing whether the Redis is required to be expanded according to the predicted value obtained by the trend analysis unit, if so, the configuration unit is informed to perform corresponding capacity expansion operation, and meanwhile, the capacity expansion state is informed to the operation and maintenance platform; and analyzing whether the Redis can be expanded, and if the Redis can not be expanded, triggering an alarm to inform the operation and maintenance platform.
The configuration unit is used for executing the capacity expansion operation on the Redis cluster, and particularly, the capacity expansion operation is divided into two types of memory capacity expansion operation and performance capacity expansion operation, wherein the memory capacity expansion is to try to increase the usable memory firstly, and when the usable memory is zero, a memory elimination strategy is used to eliminate the key value pair with the minimum access frequency; and the performance capacity expansion is to build a new slave library, notify the Redis proxy unit after the building is completed, update the Redis configuration and balance the distribution flow.
The Redis proxy layer is used for carrying out load balancing and read-write separation on the Redis.
Specifically, the above-mentioned Redis configuration system may be considered to include a first subsystem and a second subsystem, where the first subsystem is directed to a memory direction, and the second subsystem is directed to a performance direction, as shown in fig. 5A and 5B, and fig. 5A is a schematic diagram of an architecture of the first subsystem shown in the embodiment of the present disclosure; fig. 5B is a schematic diagram of an architecture of a second subsystem shown in an embodiment of the present disclosure. Wherein:
in the first subsystem, a monitoring index acquisition unit monitors and acquires historical monitoring indexes of a Redis cluster and reports the historical monitoring indexes to a Kalfa cluster, wherein the monitoring indexes comprise the memory use capacity of a machine; the Kalfa cluster stores the reported data, and stores the result to the ClickHouse database through a consumption queue mode; the ClickHouse database provides data for the trend analysis unit; the trend analysis unit analyzes the use condition of the memory through an algorithm and gives trend data of a period of time in the future; the analysis decision unit analyzes whether the capacity expansion of the Redis is required according to the trend data, if so, the configuration unit is notified to perform corresponding capacity expansion operation, meanwhile, the capacity expansion state is notified to the operation and maintenance platform, whether the capacity expansion of the Redis can be performed is analyzed, and if the capacity expansion can not be performed, an alarm is triggered to notify the operation and maintenance platform; the configuration unit executes memory expansion operation, firstly tries to increase available memory, when the available memory is zero, a memory elimination strategy is used, and key value pairs with the minimum access frequency are eliminated.
In the second subsystem, the logic relationship of the monitoring index acquisition unit, the Kalfa cluster, the ClickHouse, the trend analysis unit and the analysis decision unit is the same as that of the first subsystem, except that the monitoring index comprises at least one of the following: throughput of the machine, CPU utilization of the machine, user response time; the configuration unit executes performance capacity expansion operation, builds a new slave library, notifies the Redis proxy unit after building, updates the Redis configuration, and distributes flow evenly; the Redis agent layer performs load balancing and read-write separation on the Redis.
The trend analysis unit of the first subsystem analyzes the use condition of the memory through an algorithm, and the trend analysis unit of the second subsystem analyzes the use condition of the performance through the algorithm. The algorithm may be a linear regression algorithm or a time series algorithm, specifically, may be to judge whether the variation trend of the historical monitoring index is linear, and if so, predict the predicted value of the monitoring index in the preset time period through a ridge regression model; and if the monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a Prophet model. As shown in fig. 5C, fig. 5C is a schematic diagram illustrating an effect of obtaining a predicted value of a monitoring index according to an exemplary embodiment, where the monitoring index is QPS (query-per-second) of a machine, belongs to an index reflecting performance, and is one of important parameters of throughput of the machine; in the figure, a curve 1 is a curve for monitoring collected sample data, a point P represents a current time point, a curve 2 is a curve obtained by fitting sample data before the current time point according to an algorithm, a point Q is a predicted value of a machine which is predicted by the algorithm and corresponds to the QPS at a time point of 5:30, the predicted value is 101.8K, an actual value of the machine at the time point of 5:30 is 68.6K according to data collected later, and an actual value of the machine at the time point of 5:40 is 103.4K, so that the system has higher feasibility and accuracy for making decisions according to the predicted value.
The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
Other embodiments of the present description will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This specification is intended to cover any variations, uses, or adaptations of the specification following, in general, the principles of the specification and including such departures from the present disclosure as come within known or customary practice within the art to which the specification pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the specification being indicated by the following claims.
It is to be understood that the present description is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.
The foregoing description of the preferred embodiments is provided for the purpose of illustration only, and is not intended to limit the scope of the disclosure, since any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the disclosure are intended to be included within the scope of the disclosure.
Claims (9)
1. A method of dis configuration, the method comprising:
acquiring a history monitoring index in a Redis cluster;
fitting the change trend of the history monitoring index based on a simple linear regression algorithm, and calculating the fitting goodness; when the fitting goodness exceeds a first preset value, judging that the change trend of the history monitoring index is linear; when the goodness of fit does not exceed a first preset value, judging that the change trend of the history monitoring index is nonlinear;
when the change trend of the historical monitoring index is linear, predicting a predicted value of the monitoring index in a preset time period through a linear regression model, wherein the linear regression model is obtained after training sample data of the monitoring index based on a linear regression algorithm; when the change trend of the historical monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a time sequence model, wherein the time sequence model is obtained after training sample data of the monitoring index based on a time sequence method;
and analyzing whether the Redis cluster is subjected to capacity expansion or not according to the predicted value.
2. The method of claim 1, wherein the monitoring indicator comprises at least one of: memory usage capacity of the machine, throughput of the machine, CPU usage of the machine, user response time.
3. The method of claim 1, wherein the monitoring indicator comprises a memory usage capacity of the machine; when the capacity expansion of the Redis cluster is determined according to the predicted value analysis, the method further comprises:
increasing the available memory, or using a memory culling strategy.
4. The method of claim 1, wherein the monitoring indicator comprises at least one of: throughput of the machine, CPU usage of the machine, and user response time; when the capacity expansion of the Redis cluster is determined according to the predicted value analysis, the method further comprises:
a new Redis slave library is built and the configuration of the allocated traffic is updated.
5. The method of claim 1, wherein the linear regression model comprises a ridge regression model, and wherein the training process of the ridge regression model comprises:
taking the monitoring index as a dependent variable, taking the corresponding time when the monitoring index is acquired as an independent variable, and obtaining a ridge regression model by using a ridge regression algorithm;
determining a mean square error corresponding to the ridge regression model by using an evaluation function;
and when the value of the mean square error is smaller than a second preset value, determining that the ridge regression model is qualified.
6. The method of claim 1, wherein the time series model comprises a propset model; the training process of the propset model comprises the following steps:
and respectively determining a breakpoint, a piecewise linear function, a periodic function and a holiday function corresponding to the breakpoint by taking the monitoring index and the time corresponding to the collected monitoring index as sample data of the Prophet model, so as to obtain the trained Prophet model.
7. An apparatus of a Redis configuration, the apparatus comprising:
the acquisition module is used for acquiring the history monitoring index in the Redis cluster; fitting the change trend of the history monitoring index based on a simple linear regression algorithm, and calculating the fitting goodness; when the fitting goodness exceeds a first preset value, judging that the change trend of the history monitoring index is linear; when the goodness of fit does not exceed a first preset value, judging that the change trend of the history monitoring index is nonlinear;
the prediction module is used for predicting the predicted value of the monitoring index in a preset time period through a linear regression model when the change trend of the historical monitoring index is linear, and the linear regression model is obtained after training the sample data of the monitoring index based on a linear regression algorithm; when the change trend of the historical monitoring index is nonlinear, predicting a predicted value of the monitoring index in a preset time period through a time sequence model, wherein the time sequence model is obtained after training sample data of the monitoring index based on a time sequence method;
and the analysis module is used for analyzing whether the Redis cluster is subjected to capacity expansion according to the predicted value.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 6 when the program is executed by the processor.
9. A computer readable storage medium, characterized in that a computer program is stored thereon, which program, when being executed by a processor, implements the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010251295.5A CN111459761B (en) | 2020-04-01 | 2020-04-01 | Redis configuration method, device, storage medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010251295.5A CN111459761B (en) | 2020-04-01 | 2020-04-01 | Redis configuration method, device, storage medium and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111459761A CN111459761A (en) | 2020-07-28 |
| CN111459761B true CN111459761B (en) | 2024-03-01 |
Family
ID=71685775
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010251295.5A Active CN111459761B (en) | 2020-04-01 | 2020-04-01 | Redis configuration method, device, storage medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111459761B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112598291B (en) * | 2020-12-25 | 2023-10-13 | 中国农业银行股份有限公司 | Prophet-based operation and maintenance intelligent scheduling method and device |
| CN115474165B (en) * | 2021-05-24 | 2024-05-07 | 中移动金融科技有限公司 | Self-monitoring dynamic adjustment method and system for short message service |
| CN113342561A (en) * | 2021-06-18 | 2021-09-03 | 上海哔哩哔哩科技有限公司 | Task diagnosis method and system |
| CN114051000A (en) * | 2021-11-17 | 2022-02-15 | 中国工商银行股份有限公司 | Service flow switching method and device based on time series model |
| CN114637656B (en) * | 2022-05-13 | 2022-09-20 | 飞狐信息技术（天津）有限公司 | Redis-based monitoring method and device, storage medium and equipment |
| CN115080356B (en) * | 2022-07-21 | 2022-12-13 | 支付宝(杭州)信息技术有限公司 | Abnormity warning method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107800551A (en) * | 2016-08-31 | 2018-03-13 | 北京优朋普乐科技有限公司 | Redis group systems and its method, the client for improving reliability |
| CN108920324A (en) * | 2018-06-08 | 2018-11-30 | 广东轩辕网络科技股份有限公司 | The method of the trend analysis of information technoloy equipment memory capacity and early warning, system and electronic device |
| CN109842563A (en) * | 2017-11-24 | 2019-06-04 | 中国电信股份有限公司 | Content delivery network flow dispatching method, device and computer readable storage medium |
| CN110086649A (en) * | 2019-03-19 | 2019-08-02 | 深圳壹账通智能科技有限公司 | Detection method, device, computer equipment and the storage medium of abnormal flow |
| CN110912737A (en) * | 2019-11-14 | 2020-03-24 | 国网浙江省电力有限公司信息通信分公司 | Dynamic perception performance early warning method based on hybrid model |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150332311A1 (en) * | 2014-05-15 | 2015-11-19 | Coupons.Com Incorporated | Optimized placement of digital offers |
-
2020
- 2020-04-01 CN CN202010251295.5A patent/CN111459761B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107800551A (en) * | 2016-08-31 | 2018-03-13 | 北京优朋普乐科技有限公司 | Redis group systems and its method, the client for improving reliability |
| CN109842563A (en) * | 2017-11-24 | 2019-06-04 | 中国电信股份有限公司 | Content delivery network flow dispatching method, device and computer readable storage medium |
| CN108920324A (en) * | 2018-06-08 | 2018-11-30 | 广东轩辕网络科技股份有限公司 | The method of the trend analysis of information technoloy equipment memory capacity and early warning, system and electronic device |
| CN110086649A (en) * | 2019-03-19 | 2019-08-02 | 深圳壹账通智能科技有限公司 | Detection method, device, computer equipment and the storage medium of abnormal flow |
| CN110912737A (en) * | 2019-11-14 | 2020-03-24 | 国网浙江省电力有限公司信息通信分公司 | Dynamic perception performance early warning method based on hybrid model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111459761A (en) | 2020-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111459761B (en) | Redis configuration method, device, storage medium and equipment | |
| US11403164B2 (en) | Method and device for determining a performance indicator value for predicting anomalies in a computing infrastructure from values of performance indicators | |
| US10467533B2 (en) | System and method for predicting response time of an enterprise system | |
| US7107187B1 (en) | Method for modeling system performance | |
| US8000932B2 (en) | System and method for statistical performance monitoring | |
| CN109120463B (en) | Flow prediction method and device | |
| US20190057307A1 (en) | Deep long short term memory network for estimation of remaining useful life of the components | |
| CN110413227A (en) | A kind of remaining life on-line prediction method and system of hard disc apparatus | |
| US11307916B2 (en) | Method and device for determining an estimated time before a technical incident in a computing infrastructure from values of performance indicators | |
| US20070030853A1 (en) | Sampling techniques | |
| Luo et al. | Intelligent Virtual Machine Provisioning in Cloud Computing. | |
| US11675643B2 (en) | Method and device for determining a technical incident risk value in a computing infrastructure from performance indicator values | |
| CN112148561B (en) | Method and device for predicting running state of business system and server | |
| JP2002268922A (en) | Performance monitoring device of www site | |
| US20220245010A1 (en) | Time-series anomaly detection using an inverted index | |
| US20160094392A1 (en) | Evaluating Configuration Changes Based on Aggregate Activity Level | |
| CN111563022A (en) | Centralized storage monitoring method and device | |
| CN114896121A (en) | Monitoring method and device of distributed processing system | |
| CN118153919B (en) | Equipment maintenance arrangement method and related equipment | |
| CN117971488A (en) | Storage management method and related device for distributed database cluster | |
| CN108021484A (en) | The extension method and its system of disk life expectancy value in cloud service system | |
| CN117971384A (en) | Automatic operation and maintenance method based on container and big data | |
| CN111614504A (en) | Power grid regulation and control data center service characteristic fault positioning method and system based on time sequence and fault tree analysis | |
| CN113849333B (en) | WN-Spline baseline domain algorithm-based data center self-driving obstacle removing method and system | |
| Jehangiri et al. | Distributed predictive performance anomaly detection for virtualised platforms |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |