CN114461948A - Web cache setting optimization method and electronic equipment - Google Patents
Web cache setting optimization method and electronic equipment Download PDFInfo
- Publication number
- CN114461948A CN114461948A CN202111598316.1A CN202111598316A CN114461948A CN 114461948 A CN114461948 A CN 114461948A CN 202111598316 A CN202111598316 A CN 202111598316A CN 114461948 A CN114461948 A CN 114461948A
- Authority
- CN
- China
- Prior art keywords
- trend
- value
- api
- array
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/957—Browsing optimisation, e.g. caching or content distillation
- G06F16/9574—Browsing optimisation, e.g. caching or content distillation of access to content, e.g. by caching
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of Web services, in particular to a Web cache setting optimization method and electronic equipment, wherein the method comprises the steps of obtaining access time intervals of all apis, and writing the access time intervals into a first queue; respectively calculating a long-term trend value, a short-term trend value and an average value by using the access time interval in the first queue to determine a long-term trend array and a short-term trend array; performing variance and calculation based on the long-term trend array, the short-term trend array and the average value to determine a trend difference array; calculating to obtain a current trend value based on the last value and the last trend value in the trend difference array; acquiring current performance data of a target server to determine a hot spot threshold; and determining a target api according to the magnitude relation between the current trend value corresponding to each api and the hot spot threshold, determining the target api as hot spot data, and caching the target api. Whether the visiting api belongs to the hotspot data or not is judged through the self-adaptive decision, so that the hotspot data hit rate is improved.
Description
Technical Field
The invention relates to the technical field of Web services, in particular to a Web cache setting optimization method and electronic equipment.
Background
In order to support massive large-scale http access, a plurality of Web service optimization technologies can be used, wherein a Web cache technology is one of the most important basic technologies. The excellent Web cache system can greatly reduce the load pressure of the back-end database, and the response speed of reading data from the cache system can be greatly accelerated while the load of the database is reduced without directly accessing the back-end database by caching hotspot data, so that the network response time of accessing Web services by users is shortened, and the customer experience is greatly improved.
Currently, a commonly-used web caching technology includes dynamic data caching, and in order to solve the problem of insufficient database performance in a high-concurrency and large-data scene, a Nosql technology is generally used for realizing a caching function. The Nosql database is a database storing data in a memory, is weaker than the traditional database in the aspect of data persistence function, is not suitable for persistence, but has extremely high access speed, so the Nosql database is naturally suitable for being used as a cache system.
The existing Web cache technical scheme is as follows: the url path set requiring the caching function is configured in the code, depending on the experience of the technician. When the http access arrives at the system: according to the api configuration query, if the api is judged to be the api without the cache function, directly accessing the database and returning data; if the api is configured with the cache function, accessing a corresponding cache key in the redis system and returning cache data; if the key does not exist, the database is accessed and the data is stored in the redis.
However, this approach may lead to inaccurate hot spot caching decisions: firstly, for deciding which apis need to use the caching function, the prior art usually depends on manual experience to prejudge hot spot caching. In an actual scenario, http traffic often changes frequently along with a time dimension, a url may be a hot spot api frequently accessed in some time periods of a day, but not a hot spot in other time periods, and thus, a real-time decision for whether a certain url starts a cache function also needs to be changed.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method for optimizing a Web cache setting and an electronic device, so as to solve the problem that hot-point data in an existing Web cache system is not accurately determined.
According to a first aspect, an embodiment of the present invention provides a method for optimizing a Web cache setting, including:
obtaining an access time interval of each api, and writing the access time interval into a first queue;
respectively calculating a long-term trend value, a short-term trend value and an average value by using the access time interval in the first queue to determine a long-term trend array and a short-term trend array;
performing variance and calculation based on the long-term trend array, the short-term trend array and the average value to determine a trend difference array;
calculating to obtain a current trend numerical value based on the last numerical value and the last trend numerical value in the trend difference array;
acquiring current performance data of a target server to determine a hot spot threshold;
and determining a target api according to the magnitude relation between the current trend value corresponding to each api and the hot spot threshold, determining the target api as hot spot data, and caching the target api.
According to the Web cache setting optimization method provided by the embodiment of the invention, whether the visiting api belongs to the hot data or not is judged through self-adaptive decision, so that the hit rate of the hot data is improved. In the prior art, only a hot spot url can be manually selected, but the invention can decide whether the current api should enter the hot spot range according to real-time access data. The function can greatly optimize the problem that a Nosql system occupies a large amount of memory; the traditional data uses a magnetic disk to store data, and cannot copy all hot data into a memory due to the limitation of the memory capacity Nosql.
Optionally, the current trend value is calculated using the following formula:
RLTN-1＝((M-1)*RLTN-2+2*DIFFN-1)÷(M+1)
wherein, RLTN-1For the current trend value, RLTN-2DIFF being said last trend valueN-1And M is a periodic constant which is the last numerical value in the trend difference array.
Optionally, the obtaining current performance data of the target server to determine the hotspot threshold includes:
acquiring a CPU (Central processing Unit) use ratio, a memory use ratio and a disk use ratio of the target server;
and performing weighted calculation based on the CPU usage ratio, the memory usage ratio and the disk usage ratio to determine the hot spot threshold.
The method for optimizing the Web cache setting provided by the embodiment of the invention fully considers the influence of the real-time performance data of the target server on the cache service, and the hot point threshold value can be dynamically changed under different server load conditions, so that the cache server is ensured to run with the highest performance as possible, and the system safety is also ensured.
Optionally, the determining a target api according to a magnitude relationship between the current trend value corresponding to each api and the hotspot threshold, determining the target api as hotspot data, and caching the target api, includes:
corresponding to each api, respectively judging whether the current trend value corresponding to the api is smaller than the hot spot threshold value;
and when the current trend value corresponding to the api is smaller than the hot spot threshold value, determining the api as a target api and the hot spot data and caching the target api.
Optionally, the determining a target api according to a magnitude relationship between the current trend value corresponding to each api and the hotspot threshold, determining the target api as hotspot data, and caching the target api, further includes:
and when the current trend numerical value corresponding to the api is greater than or equal to the hot spot threshold value, accessing the database corresponding to the api.
Optionally, before the step of obtaining the access time interval of each api, the method further includes:
acquiring an access request;
judging whether a corresponding interface keyword exists in the cache or not based on the access request;
and when the interface keyword does not exist, executing the step of acquiring the access time interval of each api.
Optionally, the method further comprises:
acquiring the updating time interval of each database, and writing the updating time interval into a second queue;
respectively calculating an updated long-term trend value, an updated short-term trend value and an updated average value by using the updating time interval in the second queue so as to determine an updated long-term trend array and an updated short-term trend array;
performing variance and calculation based on the updated long-term trend array, the updated short-term trend array and the updated average value, and determining an updated trend difference array;
calculating to obtain a current updating trend value based on the last value in the updating trend difference array and the last updating trend value;
and determining the cache expiration time of the database according to the current updating trend value and the updating average value.
According to the Web cache setting optimization method provided by the embodiment of the invention, scientific and reasonable cache expiration time is calculated by using a self-adaptive algorithm according to the database updating interval data. When the updating trend value of the database is continuously increased, the data change of the database is slow, and the self-adaptive algorithm can increase the cache expiration time. Otherwise, the cache expiration time is reduced when the database change is accelerated. Therefore, the performance of the cache system is optimized, and the cache efficacy is reasonably and maximally exerted.
Optionally, the cache expiration time is calculated by using the following formula:
wherein, RLT'n-1For the current update trend value, RATE is the update weight, n is the number of update time intervals in the second queue, dbiFor the ith update interval in the second queue, RANDOM is a RANDOM value.
According to the Web cache setting optimization method provided by the embodiment of the invention, the problems of avoiding cache breakdown and cache avalanche are considered, a random value is introduced in the calculation process, the system safety is improved, and the method is more suitable for being used as a scheme of a cache system in a high-concurrency scene.
According to a second aspect, an embodiment of the present invention provides an electronic device, including: the Web cache setting optimization method includes a memory and a processor, where the memory and the processor are communicatively connected to each other, and the memory stores computer instructions, and the processor executes the computer instructions to execute the Web cache setting optimization method described in the first aspect or any one of the embodiments of the first aspect.
According to a third aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions for causing a computer to execute the method for optimizing a Web cache setting described in the first aspect or any one of the implementation manners of the first aspect.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 shows a system architecture diagram of a caching system in the present invention;
FIG. 2 shows a functional block diagram of the inside of a monitoring system used in the present invention;
FIG. 3 shows a block diagram of the internal functions of the logging system used in the present invention;
FIG. 4 is a flow diagram of a method for Web cache setting optimization according to an embodiment of the present invention;
FIG. 5 is a flow diagram of a method for Web cache setting optimization according to an embodiment of the present invention;
FIG. 6 is a flow diagram of a method for Web cache setting optimization according to an embodiment of the present invention;
FIG. 7 is a flow diagram of a method for Web cache setting optimization according to an embodiment of the present invention;
fig. 8 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a system architecture embodiment of a cache system set forth in the present invention, and the whole system framework includes a web backend service, a cache system, a database, a monitoring system, and a log service system.
The cache system uses redis as an embodiment in the invention, and the redis data uses the memory to store data, so that the read-write speed is extremely high. Database mysql is used as an example in the present invention, which is one of the most popular mainstream databases with data persisted in local disk.
The invention uses Zabbix as the embodiment of the monitoring system, the monitoring function module diagram is shown in figure 2, and the Zabbix is divided into Zabbix server service and Zabbix agent service. The Zabbix agent can collect corresponding data at the server side according to the collection items configured by the Zabbix server, and can also support the script developed by the developer if the native Zabbix does not support the required collection indexes. After data are collected by the Zabbix agent, the Zabbix agent uploads the data to the Zabbix server; zabbix server may provide api access to provide collected data queries to the outside. The monitoring system also needs a background service, periodically accesses the Zabbix server api to acquire the acquired data, and then writes the acquired data into the redis.
The invention uses a log service system ELK as an embodiment, a log service function module diagram is shown in figure 3, and the ELK is a combination of three subsystems, namely elastic search, LogStash and Kibana. The LogStash is responsible for collecting and filtering the log, and finally storing the log for use; the ElasticSearch provides a full-text search engine with distributed multi-user capability and provides a Restful web interface; kibana can display an analysis-friendly Web interface for LogStash and the log provided by LogStash. And after the log system collects the database log file, storing the database update information matched with the rule. The log system needs a background service, and periodically accesses the elastic search api to write the needed data into the redis.
The Web backend services commonly include spring boot services, php, django of python, flash and other Web technical frameworks, and the flash is used as an embodiment in the invention.
In accordance with an embodiment of the present invention, there is provided a Web cache setting optimization method embodiment, it is noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
In this embodiment, a method for optimizing a Web cache setting is provided, which may be used in electronic devices, such as a terminal and a server, and fig. 4 is a flowchart of the method for optimizing a Web cache setting according to an embodiment of the present invention, and as shown in fig. 4, the flowchart includes the following steps:
s11, obtaining the access time interval of each api, and writing the access time interval into the first queue.
When network flow accesses and enters a Web back-end system, a program records a time stamp of url access, converts the time stamp data into time interval data according to the time stamp data, and continuously writes the time interval data of each visiting ApiKey into a first queue, namely a redis queue, corresponding to the ApiKey. The redis queue sets a fixed length N according to the actual service condition, the data mark variable in the queue is array data, the subscript is 0 to N-1, N-1 is the latest data, and the array data can be used as an input quantity parameter for determining the current trend value by a subsequent adaptive algorithm.
And S12, respectively calculating a long-term trend value, a short-term trend value and an average value by using the access time interval in the first queue to determine a long-term trend array and a short-term trend array.
In the present embodiment, the core idea of the adaptive algorithm is to consider the influence of the time dimension and the historical data on the current data. Based on the data trend, two trend data lines are designed, and one trend data line uses a long-period constant L to calculate data trend change; the other uses the short-period constant S to calculate the data trend change; the difference between the short-term trend data and the long-term trend data can reflect the whole trend change more accurately. Meanwhile, the sum of the variances of the whole data is considered, so that the consistency of data fluctuation is considered. In order to calculate the influence of the time dimension on the trend, the current numerical values of all key steps are calculated based on the data operation of the last time unit, so that the final trend data result of the time dimension can be achieved.
The electronic equipment calculates to obtain a current long-term trend value by using the access time interval, the long-period constant L and the last long-term trend value in the first queue; calculating to obtain a current short-term trend value by using the access time interval, the short-period constant S and the last short-term trend value in the first queue; the access time intervals in the first queue are then used to calculate an average.
And S13, performing variance and calculation based on the long-term trend array, the short-term trend array and the average value, and determining a trend difference array.
And the electronic equipment respectively calculates the variance sum by utilizing the long-term trend array, the short-term trend array and the average value to obtain a trend difference array. Each element in the trend difference array is obtained by calculating the corresponding element in the long-term trend array, the corresponding element in the short-term trend array and the average value.
And S14, calculating the current trend value based on the last value in the trend difference array and the last trend value.
The current trend value may also be considered to be the most recent trend value calculated using the last value in the trend difference array and the last trend value. Calculating the final value RLT of the trend dataN-1Representing the current trend value. If RLTN-1Greater than 0, indicating that the current api access interval value is in a prolonged trend, RLTN-1The larger the prediction, the longer the future interval, the less frequent the access; if RLTN-1Less than 0, indicating that the api access interval value is on a decreasing trend, RLTN-1Smaller means that the future interval is predicted to be more terminal, and more frequent access.
And S15, acquiring the current performance data of the target server to determine the hot spot threshold.
Reading performance index data of a back-end server written by a monitoring system from a redis, wherein more hot data enter a cache system when the server is idle, and a hot threshold value is increased; when the server load is higher, less hot data will enter the cache system, and the hot threshold will be adjusted to be smaller. Therefore, the electronic device determines the current hot spot threshold value by acquiring the current performance data of the target server in real time.
S16, determining the target api according to the size relationship between the current trend value corresponding to each api and the hot spot threshold, determining the target api as hot spot data, and caching the target api.
The electronic equipment compares the current trend value with the calculated hot spot threshold value, and if the current trend value is smaller than the calculated hot spot threshold value, the api is a target api and is hot spot data; otherwise, the data is not the hot data, and the data can be accessed normally.
According to the Web cache setting optimization method provided by the embodiment, whether the visiting api belongs to the hotspot data or not is judged through self-adaptive decision, so that the hotspot data hit rate is improved. In the prior art, only a hot spot url can be manually selected, but the invention can decide whether the current api should enter the hot spot range according to real-time access data. The function can greatly optimize the problem that a Nosql system occupies a large amount of memory; the traditional data uses a magnetic disk to store data, and cannot copy all hot data into a memory due to the limitation of the memory capacity Nosql.
In this embodiment, a method for optimizing a Web cache setting is provided, which may be used in electronic devices, such as a terminal and a server, and fig. 5 is a flowchart of the method for optimizing a Web cache setting according to an embodiment of the present invention, and as shown in fig. 5, the flowchart includes the following steps:
s21, obtaining the access time interval of each api, and writing the access time interval into the first queue.
Please refer to S11 in fig. 4 for details, which are not described herein.
And S22, respectively calculating a long-term trend value, a short-term trend value and an average value by using the access time interval in the first queue to determine a long-term trend array and a short-term trend array.
Specifically, the method comprises the following steps:
(1) and setting the long-term trend data period constant as L, the short-term trend data period constant as S and the period constant as M. Different L, S and M values are set according to actual use requirements, for example, if high sensitivity is pursued, the parameters S is 11, L is 27, and M is 9; if the parameter S is 131, L is 287 and M is 110, which can be selected for high accuracy.
(2) All the calculation parameters needed to initialize the adaptive algorithm:
SEMA0＝SEMA1＝0
LEMA0＝LEMA1＝0
LEMA0＝LEMA1＝0
RLT0＝0
(3) calculating a short term trend array SEMA, the period constant using S
(4) Calculating a long-term trend array LEMA, wherein the period constant uses L
(5) Calculating the mean value
And S23, performing variance and calculation based on the long-term trend array, the short-term trend array and the average value, and determining a trend difference array.
Specifically, the trend difference number group DIFF is calculated using the following formula:
and S24, calculating the current trend value based on the last value in the trend difference array and the last trend value.
The trend value array RLT is calculated by adopting the following formula:
RLTi＝((M-1)*RLTi-1+2*DIFFi)÷(M+1)
accordingly, the current trend value is calculated using the following formula:
RLTN-1＝((M-1)*RLTN-2+2*DIFFN-1)÷(M+1)
wherein, RLTN-1For the current trend value, RLTN-2DIFF being said last trend valueN-1And M is a periodic constant which is the last numerical value in the trend difference array.
S25, obtaining the current performance data of the target server to determine the hot spot threshold.
Specifically, S25 includes:
and S251, acquiring the CPU use ratio, the memory use ratio and the disk use ratio of the target server.
The monitoring system can sample key performance parameters of the target server in real time, such as a cpu, a memory, a disk and the like, and periodically write performance duty into redis.
And S252, performing weighted calculation based on the CPU usage ratio, the memory usage ratio and the disk usage ratio to determine a hot spot threshold.
And reading the performance index data of the back-end server written by the monitoring system from the redis, and calculating the influence of the current server performance on the cache strategy according to the following formula. The calculation formula of the hot spot threshold is as follows: threshold Max ((1-CPU)used)*Ratecpu+(1-MEMused)*Ratemem+(1-DISKused)*Ratedisk)
Max: threshold maximum value of [ -10, 0], adjusted according to service usage scenario
CPUused：[0，1.0]Ratio of cpu usage
Ratecpu：[0，1.0]The cpu ratio weight is adjusted according to the service usage scenario
MEMused：[0，1.0]Memory usage fraction
Ratemem：[0，1.0]Memory proportion weight is adjusted according to service use scene
DISKused：[0，1.0]disk usage fraction
Ratedisk：[0，1.0]disk proportion weight, according to service use scene adjustment
And adjusting the Rate according to the service use scene.
S26, determining the target api according to the size relationship between the current trend value corresponding to each api and the hot spot threshold, determining the target api as hot spot data, and caching the target api.
Specifically, S26 includes:
s261, corresponding to each api, respectively judging whether the current trend value corresponding to the api is smaller than the hot spot threshold value;
when the current trend value corresponding to the api is smaller than the hot spot threshold value, executing S262; otherwise, S263 is performed.
S262, determining the api as the target api and the hotspot data and caching the target api.
And S263, accessing the database corresponding to the api.
If RLTN-1If the value is smaller than the set hotspot threshold, the data can be listed as hotspot data and written into a redis cache, otherwise, the database is accessed to obtain the data.
The method for optimizing the Web cache setting provided by the embodiment fully considers the influence of the real-time performance data of the target server on the cache service, and the hot spot threshold value can be dynamically changed under the condition of different server loads, so that the cache server is ensured to operate as high as possible, and the system safety is also ensured.
In some optional implementations of this embodiment, the step S21 may further include:
(1) an access request is obtained.
(2) And judging whether the corresponding interface key words exist in the cache or not based on the access request.
(3) And when the interface key word does not exist, executing the step of acquiring the access time interval of each api.
The electronic equipment obtains a cache interface keyword (ApiKey) according to a character string which is obtained by adding a plurality of access parameters to an access request url and then converting the ordered access parameters into the character string; if the key has a direct read cache data return, if the ApiKey does not exist, the step S21 is entered.
When the flash service receives api access, each visiting api can acquire a url path and an access parameter list, and a character string converted after sorting parameters on url splicing is considered, wherein the fact that different url parameters cause different database retrieval data is considered as the api parameters; so that a unique cache key for the visiting api is available. And taking the cache key to perform query in redis, directly reading the cache content to return if the cache key of the visiting api exists, and starting the calculation of the self-adaptive algorithm to start judgment if the cache key of the visiting api does not exist.
In this embodiment, a method for optimizing a Web cache setting is provided, which may be used in electronic devices, such as a terminal and a server, and fig. 6 is a flowchart of the method for optimizing a Web cache setting according to an embodiment of the present invention, and as shown in fig. 6, the flowchart includes the following steps:
s301, obtaining the access time interval of each api, and writing the access time interval into the first queue.
Please refer to S21 in fig. 5, which is not repeated herein.
S302, the long-term trend value, the short-term trend value and the average value are respectively calculated by utilizing the access time interval in the first queue, so that a long-term trend array and a short-term trend array are determined.
Please refer to S22 in fig. 5, which is not repeated herein.
And S303, performing variance and calculation based on the long-term trend array, the short-term trend array and the average value, and determining a trend difference array.
Please refer to S23 in fig. 5, which is not repeated herein.
And S304, calculating to obtain a current trend value based on the last value and the last trend value in the trend difference array.
Please refer to S24 in fig. 5, which is not repeated herein.
S305, obtaining the current performance data of the target server to determine the hot spot threshold.
Please refer to S25 in fig. 5, which is not repeated herein.
S306, determining a target api according to the magnitude relation between the current trend value corresponding to each api and the hot spot threshold, determining the target api as hot spot data, and caching the target api.
Please refer to S26 in fig. 5, which is not repeated herein.
And S307, acquiring the updating time interval of each database, and writing the updating time interval into a second queue.
And for the acquisition of each database updating time interval, a log acquisition system is used, and a log acquisition agent program acquires database table updating timestamp data by analyzing the database log file in real time and reports the database table updating timestamp data to a log server side. And the log server program converts the time stamp data into updating time interval data. By utilizing the implicit relation between the url and the database table, a database update interval data sampling array corresponding to the url can be obtained and recorded as a variable db, subscripts are 0 to n-1, and n-1 is latest data, and the update time interval can be written into a second queue, namely a redis queue, by the log system in real time for calling during calculation of the adaptive algorithm.
And S308, respectively calculating an updated long-term trend value, an updated short-term trend value and an updated average value by using the updating time interval in the second queue so as to determine an updated long-term trend array and an updated short-term trend array.
S309, performing variance and calculation based on the updated long-term trend array, the updated short-term trend array and the updated average value, and determining an updated trend difference array.
And S310, calculating to obtain a current updating trend value based on the last value in the updating trend difference array and the last updating trend value.
The specific manner of calculating the current update trend values is similar to that of calculating the current trend values in S22-S24, except that the calculation data of the current update trend values is the database update time interval, and the calculation data of the current trend values in S22-S2 is the visit time interval.
S311, determining the cache expiration time of the database according to the current updating trend value and the updating average value.
Specifically, the cache expiration time is calculated by the following formula:
wherein, RLTn′-1For the current update trend value, RATE is the update weight, n is the number of update time intervals in the second queue, dbiFor the ith update interval in the second queue, RANDOM is a RANDOM value.
After calculating the cache expiration time, the electronic device may empty the cache based on the cache expiration time.
According to the Web cache setting optimization method provided by the embodiment, scientific and reasonable cache expiration time is calculated by using a self-adaptive algorithm according to database updating interval data. When the updating trend value of the database is continuously increased, the data change of the database is slow, and the self-adaptive algorithm can increase the cache expiration time. Otherwise, the cache expiration time is reduced when the database is changed more quickly. Therefore, the performance of the cache system is optimized, and the cache efficacy is reasonably and maximally exerted. Because the problems of avoiding cache breakdown and cache avalanche are considered, a random value is introduced in the calculation process, the system safety is improved, and the method is more suitable for being used as a scheme of a cache system in a high-concurrency scene.
In a specific embodiment of this embodiment, as shown in fig. 7, taking Python language as an example for explanation, the method includes:
step 1: and (3) realizing an interceptor function by using a decorator app.before _ request in a flash frame, acquiring an endpoint attribute, an uri attribute and a parameter object from the request object when a visitor api enters, and splicing the converted character strings after the uri and the parameter queue are sequenced to be used as a redis key which is recorded as an ApiKey. The redis database is accessed using a Python redis library. If the corresponding redis key has the cache data acquisition return; if not, go to step 2.
Step 2: and splicing the keyword Interval according to the ApiKey to be used as a redis key, recording the keyword Interval as ApiInterval Key to access a redis acquisition Interval data queue, and recording the keyword Interval as data.
And step 3: trend data was initially calculated according to the adaptive algorithm and the results were recorded as RLT.
Step 3.1: setting variable initialization values
SEMA0＝SEMA1＝0
LEMA0＝LEMA1＝0
DIFF0＝0
RLT0＝0
Step 3.2: calculating short term trend data using Python
Step 3.3: calculating Long term trend data using Python
Step 3.4: calculation of mean value Using Python
Step 3.5: calculation of trend differences and sum of variances using Python
Step 3.5: calculating trend values using Python
RLTi＝((M-1)*RLTi-1+2*DIFFN-1)÷(M+1)
And 4, step 4: trend data was initially calculated according to the adaptive algorithm and the results were recorded as RLT.
And 5: calculating and setting a buffer according to Threshold, specifically:
Ratecpu＝Ratemem＝Ratedisk＝1/3
CPUused，MEMused，DISKusedpython redis access monitoring system collects performance data Threshold Max
*((1-CPUused)*Ratecpu+(1--MEMused)*Ratemem+(1--DISKused)*Ratedisk)
if RLTi＜THRESHOLD：
Accessing mysql database using Python sql library
Writing the data into a redis cache according to the return value of the database by utilizing a Python redis library
else
Utilizing Python sqlalchemy to access mysql database and directly returning data
Step 6: obtaining the sampling data of the updating interval of the database by utilizing a Python redis library and recording the sampling data as DB
And 7: calculating to obtain new RLT 'by taking DB as an input data array according to the function in the step 3'
And 8: calculating the cache expiration time according to the following formula, and writing the cache expiration time into a redis cache system by using a Python redis library:
the Web cache setting optimization method is suitable for cache system optimization in a large-scale Internet http access scene, can greatly improve the hit rate of the cache system and the reasonability of cache expiration time setting, can ensure the healthy operation of the cache system, and can exert the maximum performance.
In this embodiment, a Web cache setting optimization apparatus is further provided, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, and details are not described again after the description. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
The present embodiment provides a Web cache setting optimization apparatus, including:
the first acquisition module is used for acquiring the access time interval of each api and writing the access time interval into a first queue;
the first calculation module is used for calculating a long-term trend value, a short-term trend value and an average value respectively by using the access time interval in the first queue so as to determine a long-term trend array and a short-term trend array;
the first determining module is used for performing variance and calculation on the basis of the long-term trend array, the short-term trend array and the average value to determine a trend difference array;
the second calculation module is used for calculating to obtain a current trend numerical value based on the last numerical value and the last trend numerical value in the trend difference array;
the second acquisition module is used for acquiring the current performance data of the target server to determine a hot spot threshold;
and the second determining module is used for determining a target api according to the magnitude relation between the current trend numerical value corresponding to each api and the hot spot threshold, determining the target api as hot spot data and caching the target api.
The Web cache setting optimization apparatus in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and a memory executing one or more software or fixed programs, and/or other devices that can provide the above-described functions.
Further functional descriptions of the modules are the same as those of the corresponding embodiments, and are not repeated herein.
An embodiment of the present invention further provides an electronic device, please refer to fig. 8, fig. 8 is a schematic structural diagram of an electronic device according to an alternative embodiment of the present invention, and as shown in fig. 8, the electronic device may include: at least one processor 41, such as a CPU (Central Processing Unit), at least one communication interface 43, memory 44, and at least one communication bus 42. Wherein a communication bus 42 is used to enable the connection communication between these components. The communication interface 43 may include a Display (Display) and a Keyboard (Keyboard), and the optional communication interface 43 may also include a standard wired interface and a standard wireless interface. The Memory 44 may be a high-speed RAM Memory (volatile Random Access Memory) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The memory 44 may alternatively be at least one memory device located remotely from the aforementioned processor 41. Wherein the processor 41 calls program code stored in the memory 44 for performing any of the above-mentioned method steps.
The communication bus 42 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 42 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 8, but this is not intended to represent only one bus or type of bus.
The memory 44 may include a volatile memory (RAM), such as a random-access memory (RAM); the memory may also include a non-volatile memory (english: non-volatile memory), such as a flash memory (english: flash memory), a hard disk (english: hard disk drive, abbreviated: HDD) or a solid-state drive (english: SSD); the memory 44 may also comprise a combination of the above-mentioned kinds of memories.
The processor 41 may be a Central Processing Unit (CPU), a Network Processor (NP), or a combination of CPU and NP.
The processor 41 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof.
Optionally, the memory 44 is also used to store program instructions. Processor 41 may invoke program instructions to implement a Web cache setting optimization method as shown in any of the embodiments of the present application.
The embodiment of the invention also provides a non-transitory computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the Web cache setting optimization method in any method embodiment. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (10)
1. A Web cache setting optimization method is characterized by comprising the following steps:
obtaining an access time interval of each api, and writing the access time interval into a first queue;
respectively calculating a long-term trend value, a short-term trend value and an average value by using the access time interval in the first queue to determine a long-term trend array and a short-term trend array;
performing variance and calculation based on the long-term trend array, the short-term trend array and the average value to determine a trend difference array;
calculating to obtain a current trend numerical value based on the last numerical value and the last trend numerical value in the trend difference array;
acquiring current performance data of a target server to determine a hot spot threshold;
and determining a target api according to the magnitude relation between the current trend value corresponding to each api and the hot spot threshold, determining the target api as hot spot data, and caching the target api.
2. The method of claim 1, wherein the current trend value is calculated using the formula:
RLTN-1＝((M-1)*RLTN-2+2*DIFFN-1)÷(M+1)
wherein, RLTN-1For the current trend value, RLTN-2DIFF being the last trend valueN-1And M is a periodic constant which is the last numerical value in the trend difference array.
3. The method of claim 1, wherein obtaining current performance data of a target server to determine a hotspot threshold comprises:
acquiring a CPU (Central processing Unit) use ratio, a memory use ratio and a disk use ratio of the target server;
and performing weighted calculation based on the CPU usage ratio, the memory usage ratio and the disk usage ratio to determine the hot spot threshold.
4. The method of claim 1, wherein the determining a target api according to a magnitude relationship between the current trend value corresponding to each api and the hotspot threshold, and determining the target api as hotspot data and caching the target api, comprises:
corresponding to each api, respectively judging whether the current trend value corresponding to the api is smaller than the hot spot threshold value;
and when the current trend value corresponding to the api is smaller than the hot spot threshold value, determining the api as a target api and the hot spot data and caching the target api.
5. The method of claim 4, wherein the determining a target api according to the magnitude relationship between the current trend value corresponding to each api and the hotspot threshold, and determining the target api as hotspot data and caching the target api, further comprises:
and when the current trend numerical value corresponding to the api is greater than or equal to the hot spot threshold value, accessing the database corresponding to the api.
6. The method of claim 1, wherein the step of obtaining the access time interval of each api is preceded by the step of:
acquiring an access request;
judging whether a corresponding interface keyword exists in the cache or not based on the access request;
and when the interface keyword does not exist, executing the step of acquiring the access time interval of each api.
7. The method according to any one of claims 1-6, further comprising:
acquiring the updating time interval of each database, and writing the updating time interval into a second queue;
respectively calculating an updated long-term trend value, an updated short-term trend value and an updated average value by using the updating time interval in the second queue so as to determine an updated long-term trend array and an updated short-term trend array;
performing variance and calculation based on the updated long-term trend array, the updated short-term trend array and the updated average value, and determining an updated trend difference array;
calculating to obtain a current updating trend value based on the last value in the updating trend difference array and the last updating trend value;
and determining the cache expiration time of the database according to the current updating trend value and the updating average value.
8. The method of claim 7, wherein the cache expiration time is calculated using the following equation:
wherein, RLT'n-1For the current update trend value, RATE is the update weight, n is the number of update time intervals in the second queue, dbiFor the ith update interval in the second queue, RANDOM is a RANDOM value.
9. An electronic device, comprising:
a memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the Web cache setting optimization method according to any one of claims 1 to 8.
10. A computer-readable storage medium storing computer instructions for causing a computer to perform the method for Web cache setting optimization according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111598316.1A CN114461948A (en) | 2021-12-24 | 2021-12-24 | Web cache setting optimization method and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111598316.1A CN114461948A (en) | 2021-12-24 | 2021-12-24 | Web cache setting optimization method and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114461948A true CN114461948A (en) | 2022-05-10 |
Family
ID=81407380
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111598316.1A Pending CN114461948A (en) | 2021-12-24 | 2021-12-24 | Web cache setting optimization method and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114461948A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113380008A (en) * | 2021-05-12 | 2021-09-10 | 四川新网银行股份有限公司 | Dynamic threshold value adjusting method based on number of hits and hit rate |
-
2021
- 2021-12-24 CN CN202111598316.1A patent/CN114461948A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113380008A (en) * | 2021-05-12 | 2021-09-10 | 四川新网银行股份有限公司 | Dynamic threshold value adjusting method based on number of hits and hit rate |
| CN113380008B (en) * | 2021-05-12 | 2022-07-08 | 四川新网银行股份有限公司 | Dynamic threshold value adjusting method based on number of hits and hit rate |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10503710B2 (en) | Webpage pre-reading method, apparatus and smart terminal | |
| CN109756230B (en) | Data compression storage method, data compression method, device, equipment and medium | |
| US8307164B2 (en) | Automatic determination of read-ahead amount | |
| EP2815335A1 (en) | Method of machine learning classes of search queries | |
| US10673722B2 (en) | Search result suggestions based on dynamic network latency classification | |
| CN105550338A (en) | HTML5 application cache based mobile Web cache optimization method | |
| WO2016115957A1 (en) | Method and device for accelerating computers and intelligent devices for users and applications | |
| CN111901377B (en) | AI training platform-based file transmission method, device, equipment and medium | |
| CN109413694B (en) | Small cell caching method and device based on content popularity prediction | |
| CN113705981A (en) | Big data based anomaly monitoring method and device | |
| CN111708720A (en) | Data caching method, device, equipment and medium | |
| CN114461948A (en) | Web cache setting optimization method and electronic equipment | |
| CN114356893A (en) | Metadata tuning method, device, equipment and storage medium based on machine learning | |
| CN113726341A (en) | Data processing method and device, electronic equipment and storage medium | |
| US11194504B2 (en) | Information processing device and data management method of information processing device | |
| CN110347546B (en) | Dynamic adjustment method, device, medium and electronic equipment for monitoring task | |
| US9736671B2 (en) | Data usage recommendation generator | |
| CN112650940A (en) | Recommendation method and device of application program, computer equipment and storage medium | |
| CN114153760B (en) | Method, system and storage medium for eliminating healthy value storage cache based on weight | |
| WO2022179294A1 (en) | Data pre-fetching method, device and system | |
| CN112182454A (en) | Webpage optimization method and device, storage medium and electronic equipment | |
| CN114637809A (en) | Method, device, electronic equipment and medium for dynamic configuration of synchronous delay time | |
| CN110674839B (en) | Abnormal user identification method and device, storage medium and electronic equipment | |
| CN114218134A (en) | Method and device for caching users | |
| CN112446501A (en) | Method, device and system for obtaining cache allocation model in real network environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |