CN117118990A

CN117118990A - Redis cluster architecture system and data processing method

Info

Publication number: CN117118990A
Application number: CN202311025125.5A
Authority: CN
Inventors: 张银波; 陈良
Original assignee: Sichuan Hongmei Intelligent Technology Co Ltd
Current assignee: Sichuan Hongmei Intelligent Technology Co Ltd
Priority date: 2023-08-15
Filing date: 2023-08-15
Publication date: 2023-11-24

Abstract

The embodiment of the invention relates to a Redis cluster architecture system and a data processing method, wherein the system comprises a plurality of Redis cache node groups; different Redis cache node groups correspond to different hash slots, and each node in the same Redis cache node group corresponds to the same hash slot, so that different data are mapped into the corresponding Redis cache node group according to the corresponding hash slot. The embodiment of the invention realizes the conversion from the single-machine Redis service to the plurality of cluster master-slave Redis services, and the use of the cluster master-slave architecture can improve the Redis read-write performance, the storage capacity and the fault tolerance, thereby improving the performance and the availability of the system.

Description

Redis cluster architecture system and data processing method

Technical Field

The invention relates to the technical field of cache architecture, in particular to a Redis cluster architecture system and a data processing method.

Background

With the development and innovation of network technology, the growth of high concurrent read-write requests and data storage of application software has become a common challenge for most enterprises. Redis is favored by more and more businesses as a high-performance NoSQL database that can quickly access data and support complex command operations.

However, when the data storage amount reaches a certain scale, the single-machine Redis cannot meet the performance requirement, and meanwhile, the single-machine Redis also has a single-point fault problem. Referring to FIG. 1, multiple application systems access a Redis cache service that is deployed on one machine. When the number of users reaches a certain scale, more and more users frequently access an application system, and the application system continuously requests the Redis service to read and store data. When the concurrent requests are too many and exceed the Redis service processing capacity, the Redis service response delay and even timeout are caused, and the performance of the application system is reduced. Meanwhile, in the existing scheme, when the stored data exceeds the upper limit of the Redis storage capacity, the Redis service cannot write data continuously, so that system abnormality is caused.

Disclosure of Invention

In order to solve the above technical problems or at least partially solve the above technical problems, the present invention provides a Redis cluster architecture system and a data processing method.

According to a first aspect, a Redis cluster architecture system provided by an embodiment of the present invention includes a plurality of Redis cache node groups; different Redis cache node groups correspond to different hash slots, and each node in the same Redis cache node group corresponds to the same hash slot, so that different data are mapped into the corresponding Redis cache node group according to the corresponding hash slot.

According to a second aspect, an embodiment of the present invention provides a data processing method, which is implemented based on the system of the first aspect, where the method includes:

when a data processing request sent by an application system is received, determining a hash slot corresponding to the data;

determining a corresponding Redis cache node group according to the Ha Xicao;

and performing data processing operation based on the determined main node in the Redis cache node group.

According to the Redis cluster architecture system and the data processing method provided by the embodiment of the invention, a cluster is formed by a plurality of Redis cache node groups, different Redis cache node groups in the cluster correspond to different hash slots, corresponding hash slots are calculated for one piece of data to be processed, and then according to the calculated hash slots, which of the Redis cache node groups corresponds to the data to be processed is known, and corresponding processing is carried out through the Redis cache node groups, namely, different data are mapped into the corresponding Redis cache node groups according to the corresponding hash slots. Therefore, the embodiment of the invention creates the Redis cluster, realizes the conversion from the single-machine Redis service to the cluster master-slave multiple Redis services, and can improve the Redis read-write performance, the storage capacity and the fault tolerance by using the cluster master-slave architecture, thereby improving the performance and the availability of the system.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.

In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, and it will be obvious to a person skilled in the art that other drawings can be obtained from these drawings without inventive effort.

FIG. 1 is a schematic diagram of a prior art architecture of a stand-alone Redis;

FIG. 2 is a schematic diagram of a Redis cluster architecture system according to an embodiment of the invention;

FIG. 3 is a schematic diagram illustrating role transitions in a Redis cache node group according to one embodiment of the present invention;

FIG. 4 is a schematic diagram of the deployment of Redis services on each node in one embodiment of the invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In a first aspect, an embodiment of the present invention provides a Redis cluster architecture system, where the system includes a plurality of Redis cache node groups; different Redis cache node groups correspond to different hash slots, and each node in the same Redis cache node group corresponds to the same hash slot, so that different data are mapped into the corresponding Redis cache node group according to the corresponding hash slot.

Redis is a high-performance NoSQL database, and is favored by more and more enterprises because of the ability to quickly access data and support complex command operations.

Wherein the hash slot is a tuple of spaces. To solve the problem of data distribution, a layer is added between the data and the node, which is called a hash slot, and is used for managing the relationship between the data and the node, and the slot is equivalent to the node, and the data is placed in the slot. One cluster can only have 16384 slots, numbered 0-16383. The slots are allocated to the Redis cache node groups in the cluster, which numbered slots are allocated to which Redis cache node groups can be specified, and the cluster records the corresponding relationship between the nodes and the slots. For example, for a certain data, a hash value is obtained, then 16384 is obtained, and the remainder is the hash slot corresponding to the data, so that the data is allocated to the Redis cache node group corresponding to the hash slot.

Wherein, different Redis cache node groups correspond to different hash slots, so that one data can be ensured to be distributed into one Redis cache node group. Each node in the same dis cache node group corresponds to the same hash slot, for example, the hash slots corresponding to all nodes in a certain dis cache node group are 0-163, and since the master node interacts with the outside, the hash slots of the master node can be considered as 0-163.

It can be seen that a plurality of Redis cache node groups form a cluster, different Redis cache node groups in the cluster correspond to different hash slots, corresponding hash slots are calculated for one piece of data to be processed, and then according to the calculated hash slots, which of the Redis cache node groups corresponds to the data to be processed is known, and further corresponding processing is performed through the Redis cache node groups, namely, different data are mapped into the corresponding Redis cache node groups according to the corresponding hash slots.

In one embodiment, each Redis cache node group comprises a master node and at least one slave node, wherein the master node is used for carrying out data interaction with an application system; the at least one slave node is communicatively coupled to the master node, each slave node capable of automatically synchronizing data in the master node to maintain data synchronization between the slave node and the master node.

For example, a Redis cache node group includes a master node and a slave node, where the master node interacts with the application system, and the slave node connects with the master node, and the slave node automatically synchronizes data on the master node, so that the slave node and the master node maintain a data synchronization state.

In one embodiment, when a master node in a Redis cache node group fails, a slave node in the Redis cache node group plays the role of the master node, and the role of the failed original master node is switched to be a slave node needing repair.

That is, if a master node in one Redis cache node group fails and cannot perform normal operation, one slave node is selected from the slave nodes in the Redis cache node group to take the role of the master node, and the role of the original master node becomes the slave node, that is, at this time, a role transition is performed, so that one slave node which can perform normal operation becomes the master node.

Further, each slave node in the Redis cache node group acts as a master node by electing one slave node through an election algorithm.

That is, one slave node is selected from among the slave nodes as a master node by an election algorithm. Of course, if there is only one slave node, then no election algorithm need be employed.

In one embodiment, the data in the current master node is synchronized after the slave nodes needing to be repaired in the Redis cache node group are repaired.

That is, after the original master node becomes the slave node, the slave node needs to be repaired, so that the slave node can work normally, and after being repaired, the slave node still serves as the slave node to synchronize the data in the current master node, and the master-slave synchronization is maintained.

In one embodiment, communication links are provided between master nodes in each Redis cache node group.

The master nodes in each Redis cache node group are in communication connection, so that information interaction between the master nodes, such as interaction of respective hash slots, can be performed. After a certain data is allocated to an incorrect set of Redis cache nodes, it is known which set of Redis cache nodes should be actually allocated to through interaction between the master nodes, and the data is transferred to the correct set of Redis cache nodes.

In one embodiment, the Redis cluster architecture system maintains metadata through gossip protocol, the metadata including at least one of node basic information, role transformation information of master and slave nodes, node increase and decrease information, and node data change information.

Wherein basic information of the node, such as the identity, name, role, etc. of the node.

Wherein the role transformation information of the master and slave nodes, for example, whether the nodes have undergone the role transformation.

The node increment and decrement information, for example, which node is newly added or deleted in each Redis cache node group, how many nodes are, and the like.

The node data change information includes, for example, whether the data in the node changes, whether the data amount changes, and the like.

It can be appreciated that in the prior art, when the data storage amount reaches a certain scale, a single-machine Redis can have performance bottlenecks and single-point failure problems. Aiming at the two problems, the embodiment of the invention designs a high-performance and high-reliability Redis cluster architecture system, improves the read-write capability and the processing efficiency of the Redis, improves the service performance, can realize the fault transfer when one main node is down, ensures that the Redis service can continue to operate, and improves the usability of the system.

Referring to fig. 2, a set of Redis cache node groups A, B, C is provided in the system, each Redis cache node group includes a master node and a slave node, the data key1 corresponds to the Redis cache node group a, the data key2 corresponds to the Redis cache node group B, and the data key3 corresponds to the Redis cache node group C. master is the master node and slave is the slave node.

Because the Redis cluster architecture system is a plurality of master nodes which simultaneously provide data inquiry and reading and writing for the application system, the Redis service performance is improved in a laterally expanding mode, and in FIG. 2, the performance is improved by three times from a single machine to three machines. Meanwhile, the data storage capacity is improved by 3 times, so that mass data storage and high concurrency read-write of an upper application system are supported.

Referring to fig. 3, if a master node in the Redis cache node group a fails or is down, the network is interrupted, a slave node in the Redis cache node group a is switched to the master node to continue to provide services to the application system. And after the original master node is restored, the data of the new master node is synchronized as the slave node, and the consistency of the master and slave data is maintained. When the slave node fails or the machine is down and the network is interrupted, the data of the master node is not influenced by the application system, and after the slave node is recovered, the data of the master node is continuously synchronized, so that the consistency of the master data and the slave data is maintained.

Referring to fig. 4, for the Redis service on each node, its deployment process roughly includes:

1. downloading a Redis installation package in a Redis official network;

2. on each machine, after decompressing the installation package, entering a Redis directory, compiling and installing the Redis.

3. The configuration file, such as a port number, for each machine is modified and then the redis service is started.

4. A cluster is created using the command-/dis-trie. Rb create-redundant 1ip: port.

5. After the cluster is established, any service of redis is connected, data is read and written, and testing is carried out.

It can be understood that the embodiment of the invention creates the Redis cluster, realizes the conversion from the single-machine Redis service to the cluster master-slave multiple Redis services, and can improve the Redis read-write performance, the storage capacity and the fault tolerance by using the cluster master-slave architecture, thereby improving the performance and the availability of the system.

In a second aspect, an embodiment of the present invention provides a data processing method, where the method is implemented based on the system provided in the first aspect, and the method includes:

determining a corresponding Redis cache node group according to the Ha Xicao;

Where the data is processed in requests, e.g., read requests, write requests, etc.

For example, a data needs to be written, a hash slot of the data is calculated, and the calculated hash slot corresponds to the first Redis cache node group, so that the data is written into the master node in the first Redis cache node group, and the writing operation is completed. Of course, each slave node in the first Redis cache node group also writes the data, and keeps the data synchronous.

It may be appreciated that the method provided by the embodiment of the present invention may be performed by a management server in a cluster.

In one embodiment, further comprising:

receiving main node fault reporting information in any Redis cache node group;

selecting a slave node from all slave nodes in the Redis cache node group to take the role of a master node through an election algorithm;

and switching the role of the original master node with the fault into a slave node needing to be repaired.

It can be understood that when a master node in a Redis cache node group fails, reporting is performed, and when the master node in a Redis cache node group is known to fail to work normally, a slave node is selected to play the role of the master node through an election algorithm, and the original master node becomes a slave node, so that role conversion is realized, and normal work is ensured.

In one embodiment, the method provided by the embodiment of the invention can further include:

metadata is maintained through a gossip protocol, and the metadata comprises at least one of node basic information, role transformation information of master and slave nodes, node increase and decrease information and node data change information.

The Gossip protocol is also called Epidemic Protocol, epidemic protocol, and is mainly used for message transmission, and is a consistency algorithm. Protocols are also well understood, just like the names of protocols, by continuously spreading as epidemic by "infecting" nodes. The Gossip protocol was used: redis Cluster, consul, apache Cassandra, etc.

It may be appreciated that, in the method provided by the embodiment of the present invention, explanation, specific implementation, beneficial effects, examples and other content related to the content may refer to corresponding parts in the system provided by the first aspect, which are not described herein again.

In a third aspect, embodiments of the present invention provide a computer readable medium having stored thereon computer instructions which, when executed by a processor, cause the processor to perform the method provided in the second aspect.

Specifically, a system or apparatus provided with a storage medium on which a software program code realizing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or apparatus may be caused to read out and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium may realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code form part of the present invention.

Examples of the storage medium for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code may be downloaded from a server computer by a communication network.

Further, it should be apparent that the functions of any of the above-described embodiments may be implemented not only by executing the program code read out by the computer, but also by causing an operating system or the like operating on the computer to perform part or all of the actual operations based on the instructions of the program code.

Further, it is understood that the program code read out by the storage medium is written into a memory provided in an expansion board inserted into a computer or into a memory provided in an expansion module connected to the computer, and then a CPU or the like mounted on the expansion board or the expansion module is caused to perform part and all of actual operations based on instructions of the program code, thereby realizing the functions of any of the above embodiments.

It may be appreciated that, for explanation, specific implementation, beneficial effects, examples, etc. of the content in the computer readable medium provided by the embodiment of the present invention, reference may be made to corresponding parts in the method provided in the second aspect, and details are not repeated herein.

In a fourth aspect, one embodiment of the present specification provides a computing device comprising a memory having executable code stored therein and a processor that, when executing the executable code, performs the method of any one of the embodiments of the specification.

It may be appreciated that, for explanation, specific implementation, beneficial effects, examples, etc. of the content in the computing device provided by the embodiment of the present invention, reference may be made to corresponding parts in the method provided in the second aspect, which are not repeated herein.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments in part.

Those skilled in the art will appreciate that in one or more of the examples described above, the functions described in the present invention may be implemented in hardware, software, a pendant, or any combination thereof. When implemented in software, these functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium.

The foregoing embodiments have been provided for the purpose of illustrating the general principles of the present invention in further detail, and are not to be construed as limiting the scope of the invention, but are merely intended to cover any modifications, equivalents, improvements, etc. based on the teachings of the invention.

Claims

1. A dis cluster architecture system, comprising a plurality of dis cache node groups; different Redis cache node groups correspond to different hash slots, and each node in the same Redis cache node group corresponds to the same hash slot, so that different data are mapped into the corresponding Redis cache node group according to the corresponding hash slot.

2. The system of claim 1, wherein each set of Redis cache nodes includes a master node and at least one slave node, the master node being configured to interact with data of the application system; the at least one slave node is communicatively coupled to the master node, each slave node capable of automatically synchronizing data in the master node to maintain data synchronization between the slave node and the master node.

3. The system of claim 2, wherein when a master node in a set of Redis cache nodes fails, a slave node in the set of Redis cache nodes takes on the role of master node, and the role of the failed original master node switches to a slave node that needs repair.

4. A system according to claim 3, wherein each slave node in the set of dis cache nodes acts as a master node by electing one slave node by an election algorithm.

5. The system of claim 3, wherein the slave nodes in the set of Redis cache nodes that require repair are repaired to synchronize data in the current master node.

6. The system of claim 1, wherein the master nodes in each set of Redis cache nodes are communicatively coupled.

7. The system of claim 1, wherein the dis cluster architecture system maintains metadata via gossip protocol, the metadata including at least one of node basic information, role transformation information of master-slave nodes, node increase and decrease information, and node data change information.

8. A data processing method, characterized in that the method is implemented on the basis of the system according to any one of claims 1-7, the method comprising:

determining a corresponding Redis cache node group according to the Ha Xicao;

9. The method as recited in claim 8, further comprising:

receiving main node fault reporting information in any Redis cache node group;

10. The method as recited in claim 8, further comprising: