CN116521688B

CN116521688B - Key prefix operation KEY value method based on Redis cluster

Info

Publication number: CN116521688B
Application number: CN202310810910.5A
Authority: CN
Inventors: 罗剑锋; 于光杰; 刘启铨; 曾力耕
Original assignee: Whale Cloud Technology Co Ltd
Current assignee: Whale Cloud Technology Co Ltd
Priority date: 2023-07-04
Filing date: 2023-07-04
Publication date: 2023-09-26
Anticipated expiration: 2043-07-04
Also published as: CN116521688A

Abstract

The invention provides a KEY prefix operation KEY value method based on a Redis cluster, which comprises the following steps: the user end initiates an operation request to the Redis through the access command; performing parameter validity verification after Redis receives the command request; redis searches the KEY slot index according to the designated slot and the KEY prefix; redis processes the searched KEY according to the operation request; redis continues the processing of the next slot according to the processing strategy; the Redis returns the command request processing result to the user terminal; the user end performs iterative operation KEY until all the processing is completed. The invention avoids inquiring and comparing the full quantity KEY; the request processing can be directly completed according to prefix statistics and deletion operation, so that a large amount of data interaction with a user is avoided; according to the slot iteration command mode, blockage of Redis service can be avoided, and service stability is guaranteed.

Description

Key prefix operation KEY value method based on Redis cluster

Technical Field

The invention relates to the field of distributed caching, in particular to a KEY prefix operation KEY value method based on a Redis cluster.

Background

With the development of big data and cloud computing, distributed systems are becoming more and more popular. Redis has been widely used in distributed cache clusters as a high performance Key-Value (Key-Value) storage system due to its rich data types and efficient accessibility by Key names.

In Redis, a Hash data structure is used to construct a KEY index, and the VALUE data can be quickly retrieved by exact matching of the KEY name, but such an unordered discrete data structure is not suitable for performing a range lookup per KEY prefix. Thus, redis is not good at for statistical analysis-like operations that need to be performed with the KEY prefix.

For example, if data deletion is required according to KEY prefix, the current method mainly performs full data scanning matching through KEY command or SCAN command to obtain an accurate KEY list, and then performs deletion through DEL or UNLINK command. However, the time consumption of the process is greatly influenced by the data quantity stored in the Redis, and the service quality of the Redis is influenced, so that the occupation of resources such as a CPU (Central processing Unit), a network and the like of the system is increased.

Therefore, how to efficiently perform operations such as statistics, inquiry, deletion and the like according to KEY prefixes on the premise of not affecting the Redis service quality becomes a problem to be solved urgently.

Disclosure of Invention

In order to overcome the defects of the prior art, the invention provides a KEY prefix operation KEY value method based on Redis clusters, which can quickly position the relevant KEYs according to KEY prefixes through the KEY slot indexes of Reids clusters, so as to avoid inquiring and comparing the total KEYs; the request processing can be directly completed according to prefix statistics and deletion operation, so that a large amount of data interaction with a user is avoided; according to the slot iteration command mode, blockage of Redis service can be avoided, and service stability is guaranteed.

In order to achieve the above object, the present invention provides a KEY prefix operation KEY value method based on a Redis cluster, which includes the following steps:

step S1: the user end initiates an operation request to the Redis through the KEY prefix access command;

step S2: performing validity verification of parameters after Redis receives a command request;

step S3: redis searches the KEY slot index according to the designated slot and the KEY prefix;

step S4: redis processes the searched KEY according to the operation request;

step S5: redis continues the processing of the next slot according to the processing strategy;

step S6: the Redis returns the command request processing result to the user terminal;

step S7: the user end performs iterative operation KEY until all the processing is completed.

Further, the step S2 specifically includes:

step S21: checking whether the inputted slot parameter value is a valid slot value or not;

step S22: checking whether the parameters are valid according to the input KEY prefix;

step S23: checking whether the current Redis instance is in cluster mode;

step S24: when the request command is to delete data by prefix, it is checked whether the current Redis instance is the master node.

Further, the step S3 specifically includes:

step S31: correcting the incoming slot position value according to the slot position owned by the current Redis;

step S32: and searching the KEY slot index based on the slots and the KEY prefix to obtain the matching KEY.

Further, step S4 specifically includes:

step S41: performing KEY quantity accumulation processing on the request counted according to the prefix;

step S42: storing the queried KEY into a return list according to the prefix query request;

step S43: and deleting the queried KEY according to the prefix deletion request, and accumulating the deletion quantity.

Further, step S5 specifically includes:

step S51: adding 1 to the current processing slot position, and preparing to continue iterative processing;

step S52: checking whether the continuous iteration processing condition is met according to the processing strategy;

step S521: if the condition is met, continuing to process in the step S3;

step S522: if not, stopping the command iteration process.

Further, the command request processing result in step S6 includes a slot value of the next iteration continuing request and a prefix operation result, and the specific steps are as follows:

step S61: correcting the slot position value to be 0 when the current processing slot position exceeds the maximum value;

step S62: returning the current processing slot position value and the matched KEY number according to the prefix statistic request;

step S63: returning the current processing slot position value and the matched KEY list to the query request according to the prefix;

step S64: and returning the current processing slot position value and the deleted KEY number according to the prefix deleting request.

Further, the method also comprises an automatic threshold adjustment mechanism, and comprises the following steps:

step S101: initializing a threshold;

step S201: starting a monitoring task;

step S301: acquiring the resource utilization rate;

step S401: calculating a new threshold value;

step S501: applying a new threshold;

step S601: and (5) threshold adjustment feedback.

Further, the method also comprises a log mechanism, and comprises the following steps:

step S102: initializing a log system;

step S202: defining a log format;

step S302: recording an operation start log;

step S402: recording an operation ending log;

step S502: asynchronously writing a log;

step S602: and (5) log maintenance.

Further, the operation request in step S4 supports three command operations of statistics by prefix, querying by prefix, and deleting by prefix.

Further, in step S7, the request may be initiated again by performing the iterative operation on other slots by executing the returned slot value last time, and if the returned slot value is 0, the specific steps are as follows:

step S71: the index is queried using the slot and KEY prefix combination condition,

step S72: for the number of KEYs that contain matches in the result returned by the prefix statistics request,

step S73: for KEY lists that contain matches in the results returned by the prefix query request,

step S74: for the number of KEYs to be deleted contained in the result returned by the prefix deletion request,

step S75: returning a slot value of 0 in the result to indicate that the slot traversal is completed; otherwise, request processing for the next slot may continue.

Compared with the prior art, the invention has the beneficial effects that:

1. the invention provides a KEY prefix operation KEY value method based on a Redis cluster, which can quickly position a relevant KEY according to the KEY prefix through the KEY slot index of the Reids cluster, so as to avoid inquiring and comparing the total KEYs;

2. the invention provides a KEY prefix operation KEY value method based on a Redis cluster, which can directly complete request processing according to prefix statistics and deletion operation and avoid a large amount of data interaction with a user;

3. the invention provides a KEY prefix operation KEY value method based on a Redis cluster, which can avoid blocking of Redis service and ensure service stability according to a slot iteration command mode;

4. the invention provides a KEY prefix operation KEY value method based on a Redis cluster, which can carry out detailed record and tracking on the execution condition of an operation request through a log mechanism, thereby facilitating subsequent problem investigation and performance optimization;

5. the invention provides a KEY prefix operation KEY value method based on a Redis cluster, which can dynamically adjust the execution time of an operation request and the number of processed KEY values according to the actual resource use condition of a Redis instance by the automatic threshold adjustment mechanism, thereby ensuring the stability of Redis service and improving the processing performance;

6. the invention provides a KEY prefix operation KEY value method based on a Redis cluster, which can avoid blocking the Redis service for a long time through a processing strategy, ensure the stability of the service, and simultaneously consider the processing performance and reduce the interaction times between a user and the Redis.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description will briefly explain the drawings needed in the embodiments or the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a method for implementing a KEY prefix-based Redis cluster KEY slot index technique in accordance with an embodiment of the present invention;

FIG. 2 is a timing diagram of the operation of an application to delete a 3-node Redis cluster by KEY prefix, according to an embodiment of the invention.

Detailed Description

The technical solution of the present invention will be more clearly and completely explained by the description of the preferred embodiments of the present invention with reference to the accompanying drawings.

As shown in fig. 1, the present invention includes the steps of:

step S1: the user end initiates an operation request to the Redis through the KEY prefix access command; the operation request comprises three command requests of statistics by prefix, inquiry by prefix and deletion by prefix

step S23: checking whether the current Redis instance is in cluster mode;

The parameters include slots and KEY prefix names.

The KEY slot index is an existing data structure of the Redis cluster, additional construction and maintenance are not needed, and memory occupation overhead is not increased newly;

step S4: redis processes the searched KEY according to the operation request;

step S41: for KEY number accumulation processing per prefix statistics request,

The operation request supports three command operations of statistics by prefix, query by prefix and deletion by prefix

Step S5: redis continues the processing of the next slot according to the processing strategy; step S51: adding 1 to the current processing slot position, and preparing to continue iterative processing;

step S521: if the condition is met, continuing to process in the step S3;

step S522: if not, stopping the command iteration process.

The processing strategy comprises operation duration and whether the number of the operated key values reaches a set threshold value, wherein the threshold value of the operation duration is 500 microseconds by default, the threshold value of the number of the operated key values is 1000 by default, and the operation is finished when the threshold value is reached.

The command request processing result comprises a slot value of the next iteration continuing request and a prefix operation result.

Step S7: the user end performs iterative operation KEY until all the processing is completed. And the other slots in the iterative operation can initiate a request again through the last time of executing the returned slot value, and if the returned slot value is 0, the completion of all iterations is indicated, and the specific steps are as follows:

step S75: returning a slot value of 0 in the result to indicate that the slot traversal is completed; otherwise, the request processing of the next slot position can be continued

The parameters in step S2 include slots and KEY prefix names.

The operation request in step S4 supports three command operations of statistics by prefix, inquiry by prefix, and deletion by prefix. The cluster KEY slot index technology is realized by adopting a Rax radix tree in Redis 3.2-6.2 version, is used for carrying out KEY KEY value inquiry owned by slots when cluster slots are migrated, and can initiate a request again by continuing to iterate other slots through the slot value returned by the last execution, and if the returned slot value is 0, the completion of all iterations is indicated

Also included is an automatic threshold adjustment mechanism, which, as a specific embodiment, may be performed as follows:

step S101: initializing a threshold value: initial values of an operation duration threshold and a key value number threshold are set. These two thresholds are used to control the execution time of the operation request and the number of key values processed.

Step S201: starting a monitoring task: a timing task is set, which is executed once per minute, for checking the CPU utilization and memory utilization of the Redis instance.

Step S301: acquiring the resource utilization rate: in the timing task, the CPU utilization rate and the memory utilization rate of the Redis instance are obtained. This may be accomplished by calling an API of the operating system or using a related library.

Step S401: calculating a new threshold: and calculating a new operation duration threshold value and a key value quantity threshold value according to the acquired CPU utilization rate and memory utilization rate. For example, if the CPU utilization is below 30%, the threshold is raised, and if the CPU utilization exceeds 70%, the threshold is lowered. The adjustment amplitude of the threshold value may be set according to the actual situation.

Step S501: applying a new threshold: the calculated new threshold is applied to subsequent operation requests. When processing an operation request, it is necessary to check whether the execution time of the operation and the number of processed key values exceeds a current threshold value, and if so, the operation is ended.

Step S501: threshold adjustment feedback: the condition of each threshold adjustment is recorded, including the threshold values before and after adjustment, the resource utilization rate and the reason for adjustment. This information can be used for subsequent performance optimization and problem investigation.

By the automatic threshold adjustment mechanism, the execution time of the operation request and the number of processed key values can be dynamically adjusted according to the actual resource use condition of the Redis instance, so that the stability of the Redis service can be ensured, and the processing performance can be improved.

Also included is a logging mechanism, as a specific embodiment, which may be performed as follows:

step S102: initializing a log system: the log function using Redis is selected.

Step S202: defining a log format: the log information format recorded at the beginning and end of the operation request is defined. The log information may include the type of operation request (per-prefix statistics, per-prefix queries, per-prefix deletions, etc.), the slot and KEY prefixes of the operation, the start and end times of the operation, the results of the operation, etc.

Step S302: recording an operation start log: when the opened log record level reaches the requirement, the relevant information is recorded to the log system according to the defined log format when each operation request starts.

Step S402: recording an operation end log: when the opened log record level reaches the requirement, at the end of each operation request, relevant information is recorded to the log system according to the defined log format. If the operation request is wrong, the error information needs to be recorded.

Step S502: asynchronous write log: in order to avoid that the log operation affects the performance of the Redis, log information can be written into the memory first, and then the log information in the memory is written into the disk when appropriate. This may be accomplished by using an asynchronous log function, or by implementing a log buffer itself.

Step S602: journal maintenance: and (3) checking and cleaning the log files regularly, so that the log files are prevented from occupying excessive disk space. At the same time, the log files can be backed up and archived for subsequent analysis and auditing.

Through the log mechanism, the execution condition of the operation request can be recorded and tracked in detail, and subsequent problem investigation and performance optimization are facilitated.

According to the embodiment of the invention, for a cluster with a total KEY value data volume of 2000 ten thousand, when a certain type of service data deleted according to a KEY prefix is only tens of pieces, the time is only tens of milliseconds, and 1 minute is needed in a conventional SCAN+DEL mode; when some service data deleted according to the KEY prefix has 1000 ten thousands of service data, the time is only tens of seconds, and the conventional SCAN+DEL mode needs tens of minutes. The performance improvement of the present invention is more pronounced when the cluster data is billions or even billions in size, as the performance of the present invention is primarily related to the number of KEY prefix KEYs that match, whereas conventional approaches are related to the total number of cluster KEY values.

The above detailed description is merely illustrative of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Various modifications, substitutions and improvements of the technical scheme of the present invention will be apparent to those skilled in the art from the description and drawings provided herein without departing from the spirit and scope of the invention. The scope of the invention is defined by the claims.

Claims

1. The KEY prefix operation KEY value method based on the Redis cluster is characterized by comprising the following steps:

the step S2 specifically comprises the following steps:

step S23: checking whether the current Redis instance is in cluster mode;

step S24: when the request command is to delete data by prefix, checking whether the current Redis instance is a master node;

the step S3 specifically comprises the following steps:

step S32: searching a KEY slot index based on the slot and the KEY prefix to obtain a matching KEY;

step S4: redis processes the searched KEY according to the operation request;

2. The KEY prefix operation KEY value method based on the Redis cluster as claimed in claim 1, wherein the step S4 is specifically:

3. The KEY prefix operation KEY value method based on the Redis cluster as claimed in claim 1, wherein step S5 specifically comprises:

step S521: if the condition is met, continuing to process in the step S3;

step S522: if not, stopping the command iteration process.

4. The KEY prefix operation KEY value method based on the Redis cluster as claimed in claim 1, wherein the command request processing result in step S6 includes a slot value of the next iteration continuation request and a prefix operation result, and the specific steps are as follows:

5. The KEY prefix operation KEY value method based on Redis cluster as recited in claim 1, further comprising an automatic threshold adjustment mechanism, comprising the steps of:

step S101: initializing a threshold;

step S201: starting a monitoring task;

step S301: acquiring the resource utilization rate;

step S401: calculating a new threshold value;

step S501: applying a new threshold;

step S601: and (5) threshold adjustment feedback.

6. The KEY prefix operation KEY value method based on Redis cluster as recited in claim 1, further comprising a log mechanism, comprising the steps of:

step S102: initializing a log system;

step S202: defining a log format;

step S302: recording an operation start log;

step S402: recording an operation ending log;

step S502: asynchronously writing a log;

step S602: and (5) log maintenance.

7. The KEY prefix operation KEY value method based on the Redis cluster according to claim 1, wherein the operation request in the step S4 supports three command operations of statistics by prefix, inquiry by prefix, and deletion by prefix.

8. The KEY prefix operation KEY value method based on the Redis cluster according to claim 1, wherein in the step S7, the request is initiated again by continuing the iterative operation of other slots by executing the returned slot value last time, if the returned slot value is 0, the specific steps are: