CN113794757A - Method and system for remote synchronization of redis cluster based on rabbitmq - Google Patents
Method and system for remote synchronization of redis cluster based on rabbitmq Download PDFInfo
- Publication number
- CN113794757A CN113794757A CN202111012818.1A CN202111012818A CN113794757A CN 113794757 A CN113794757 A CN 113794757A CN 202111012818 A CN202111012818 A CN 202111012818A CN 113794757 A CN113794757 A CN 113794757A
- Authority
- CN
- China
- Prior art keywords
- synchronous
- client
- information
- event
- rabbitmq
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 230000001360 synchronised effect Effects 0.000 claims abstract description 168
- 238000004806 packaging method and process Methods 0.000 claims abstract description 26
- 238000004458 analytical method Methods 0.000 claims abstract description 22
- 238000004519 manufacturing process Methods 0.000 claims description 53
- 238000012545 processing Methods 0.000 claims description 10
- 238000013507 mapping Methods 0.000 claims description 9
- 238000005516 engineering process Methods 0.000 description 15
- 230000000694 effects Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 7
- 238000005538 encapsulation Methods 0.000 description 6
- 238000013461 design Methods 0.000 description 5
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000004590 computer program Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 229910002056 binary alloy Inorganic materials 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Hardware Redundancy (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a method and a system for remote synchronization of redis cluster based on rabbitmq, wherein the method comprises the following steps: packaging the client redis component according to the first packaging instruction to obtain a first user-defined client; acquiring request information of a first client according to the first user-defined client; analyzing the request information of the first client to obtain a first event queue; constructing a first synchronous rabbitmq component; writing the first event queue to obtain first write-in information; analyzing the first written information to obtain a first analysis result; and according to the first event information, obtaining first client request information, requesting a first remote redis cluster, and placing a first local redis command into the first remote redis cluster. The method solves the technical problems that excessive redundant data can be synchronized and data needs to be rewritten or data is lost when a network fault occurs in the prior art.
Description
Technical Field
The invention relates to the technical field related to data processing, in particular to a method and a system for remote synchronization of redis cluster based on rabbitmq.
Background
An Internet Data Center of an Internet Data Center is abbreviated as IDC and is a place for a telecommunication department to provide trusteeship for enterprises, merchants or website servers. With the development of internet technology, especially the deep application of financial technology, more and more financial technology companies, especially internet financial technology companies, have made more stringent requirements on data security and service stability, and deploying application or synchronizing data across IDCs has become a technical problem that internet software or application must solve.
In order to improve the application experience of enterprises, the cache technology represented by rediscluster is applied and practiced in large scale in each large software architecture, and the cross-IDC allopatric synchronization technology of rediscluster is also valued by more and more enterprises along with the requirement of cross-IDC deployment. Some internet enterprises realize the customization of the redis-based log replication technology, most of the schemes are very complex to realize, and the requirements on the version of the redis high.
However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the inventors of the present application find that the above-mentioned technology has at least the following technical problems:
the technical problems that redundant data can be synchronously excessive and data needs to be rewritten or data is missing when a network fault occurs exist in the prior art.
Disclosure of Invention
The embodiment of the application provides a method and a system for remote synchronization of a redis cluster based on a rabbitmq, and solves the technical problems that redundant data can be synchronized excessively and data needs to be rewritten or data is missing when a network fault occurs in the prior art. Most versions can be compatible by packaging the custom client, synchronous data is determined according to the request of the client, and data redundancy is avoided; the request is packaged, the event queue is built, the event queue is written by the first synchronous production end, the phenomenon that local redis clusters lack data due to network faults can be avoided, the events are analyzed by the first synchronous consumption end, and the technical effect that data can be selected and stably synchronized among multiple internet data centers is achieved.
In view of the above problems, the embodiments of the present application provide a method and a system for remote synchronization of redis cluster based on rabbitmq.
In a first aspect, an embodiment of the present application provides a redis cluster allopatric synchronization method based on a rabbitmq, where the method is applied to a user-defined information allopatric synchronization system, and the first user-defined allopatric synchronization system includes a client redis component and a first synchronization rabbitmq component, where the method includes: packaging the client redis component according to a first packaging instruction to obtain a first user-defined client; obtaining first client request information according to the first user-defined client, wherein the first client request information is user-defined by a first service party; analyzing the first client request information through the first user-defined client to obtain a first event queue; constructing the first synchronous rabbitmq component according to a first local redis cluster, wherein the first synchronous rabbitmq component comprises a first synchronous production end and a first synchronous consumption end; writing the first event queue in the first synchronous production end to obtain first write-in information; analyzing the first written information through the first synchronous consuming terminal to obtain a first analysis result, wherein the first analysis result comprises the first event information, and the first event information corresponds to the first client request information; acquiring the first client request information according to the first event information; and requesting a first remote redis cluster according to the first client request information, and putting a first local redis command into the first remote redis cluster.
On the other hand, the embodiment of the application provides a redis cluster allopatric synchronization system based on rabbitmq, wherein the system comprises: the first obtaining unit is used for packaging the client redis component according to a first packaging instruction to obtain a first user-defined client; the first request unit is used for acquiring first client request information according to the first user-defined client, and the first client request information is defined by a first service party; a second obtaining unit, configured to analyze the first client request information through the first custom client to obtain a first event queue; the first building unit is used for building a first synchronous rabbitmq component according to a first local redis cluster, and the first synchronous rabbitmq component comprises a first synchronous production end and a first synchronous consumption end; a third obtaining unit, configured to write the first event queue in the first synchronous production end, and obtain first write information; a fourth obtaining unit, configured to analyze the first written information through the first synchronous consuming terminal to obtain a first analysis result, where the first analysis result includes the first event information, and the first event information corresponds to the first client request information; a fifth obtaining unit, configured to obtain the first client request information according to the first event information; a first execution unit, configured to request a first remote redis cluster according to the first client request information, and place a first local redis command in the first remote redis cluster.
In a third aspect, an embodiment of the present application provides a rabbitmq-based redis cluster allopatric synchronization system, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the method according to any one of the first aspect when executing the program.
One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:
packaging the client redis component according to a first packaging instruction to obtain a first user-defined client; obtaining first client request information according to the first user-defined client, wherein the first client request information is user-defined by a first service party; analyzing the first client request information through the first user-defined client to obtain a first event queue; constructing the first synchronous rabbitmq component according to a first local redis cluster, wherein the first synchronous rabbitmq component comprises a first synchronous production end and a first synchronous consumption end; writing the first event queue in the first synchronous production end to obtain first write-in information; analyzing the first written information through the first synchronous consuming terminal to obtain a first analysis result, wherein the first analysis result comprises the first event information, and the first event information corresponds to the first client request information; acquiring the first client request information according to the first event information; the technical scheme includes that a first remote redis cluster is requested according to first client request information, a first local redis command is placed in the first remote redis cluster, most versions can be compatible through packaging a user-defined client, synchronous data are determined according to client requests, and data redundancy is avoided; the request is packaged, the event queue is built, the event queue is written by the first synchronous production end, the situation that local redis clusters lack data due to network faults can be avoided, the events are analyzed by the first synchronous consumption end, and the technical effect that data can be selected and stably synchronized among multiple internet data centers is achieved.
The foregoing description is only an overview of the technical solutions of the present application, and the present application can be implemented according to the content of the description in order to make the technical means of the present application more clearly understood, and the following detailed description of the present application is given in order to make the above and other objects, features, and advantages of the present application more clearly understandable.
Drawings
FIG. 1 is a schematic flow chart of a redis cluster remote synchronization method based on rabbitmq in an embodiment of the present application;
fig. 2 is a schematic flowchart of a method for constructing the first synchronous rabbitmq component according to the first local redis cluster in the embodiment of the present application;
FIG. 3 is a schematic flow chart of another redis cluster remote synchronization method based on rabbitmq according to an embodiment of the present application;
FIG. 4 is a schematic structural diagram of a redis cluster remote synchronization system based on rabbitmq according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an exemplary electronic device according to an embodiment of the present application.
Description of reference numerals: the system comprises a first obtaining unit 11, a first requesting unit 12, a second obtaining unit 13, a first constructing unit 14, a third obtaining unit 15, a fourth obtaining unit 16, a fifth obtaining unit 17, a first executing unit 18, an electronic device 300, a memory 301, a processor 302, a communication interface 303 and a bus architecture 304.
Detailed Description
The embodiment of the application provides a method and a system for remote synchronization of a redis cluster based on a rabbitmq, and solves the technical problems that redundant data can be synchronized excessively and data needs to be rewritten or data is missing when a network fault occurs in the prior art. Most versions can be compatible by packaging the custom client, synchronous data is determined according to the request of the client, and data redundancy is avoided; the request is packaged, the event queue is built, the event queue is written by the first synchronous production end, data loss of a local redis cluster due to network faults can be avoided, the event is analyzed by the first synchronous consumption end, and the technical effect that data can be selected and stably synchronized among multiple internet data centers is achieved.
Summary of the application
An Internet Data Center of an Internet Data Center is abbreviated as IDC and is a place for a telecommunication department to provide trusteeship for enterprises, merchants or website servers. With the development of internet technology, especially the deep application of financial technology, more and more financial technology companies, especially internet financial technology companies, have made more stringent requirements on data security and service stability, and deploying application or synchronizing data across IDCs has become a technical problem that internet software or application must solve. In order to improve the application experience of enterprises, the cache technology represented by rediscluster is applied and practiced in large scale in each large software architecture, and the cross-IDC allopatric synchronization technology of rediscluster is also valued by more and more enterprises along with the requirement of cross-IDC deployment. Some internet enterprises realize the customization of the redis-based log replication technology, most of the schemes are very complex to realize, and the requirements on the version of the redis high. However, the prior art has the technical problems that redundant data can be synchronized, and data needs to be rewritten or data is missing when a network failure occurs.
In view of the above technical problems, the technical solution provided by the present application has the following general idea:
the embodiment of the application provides a redis cluster allopatric synchronization method based on rabbitmq, which is applied to a user-defined information allopatric synchronization system, wherein the first user-defined allopatric synchronization system comprises a client redis component and a first synchronization rabbitmq component, and the method comprises the following steps: packaging the client redis component according to a first packaging instruction to obtain a first user-defined client; obtaining first client request information according to the first user-defined client, wherein the first client request information is user-defined by a first service party; analyzing the first client request information through the first user-defined client to obtain a first event queue; constructing the first synchronous rabbitmq component according to a first local redis cluster, wherein the first synchronous rabbitmq component comprises a first synchronous production end and a first synchronous consumption end; writing the first event queue in the first synchronous production end to obtain first write-in information; analyzing the first written information through the first synchronous consuming terminal to obtain a first analysis result, wherein the first analysis result comprises the first event information, and the first event information corresponds to the first client request information; acquiring the first client request information according to the first event information; and requesting a first remote redis cluster according to the first client request information, and putting a first local redis command into the first remote redis cluster.
Having thus described the general principles of the present application, various non-limiting embodiments thereof will now be described in detail with reference to the accompanying drawings.
Example one
As shown in fig. 1, an embodiment of the present application provides a redis cluster allopatric synchronization method based on rabbitmq, where the method is applied to a user-defined information allopatric synchronization system, and the first user-defined allopatric synchronization system includes a client redis component and a first synchronization rabbitmq component, where the method includes:
s100: packaging the client redis component according to a first packaging instruction to obtain a first user-defined client;
specifically, the first encapsulating instruction refers to an instruction signal for encapsulating the client redis component, which is issued by the first customized remote synchronization system; the first user-defined client refers to a client which can be used for a service party to freely select a remote synchronous interface after the client redis component is encapsulated according to the first encapsulation instruction, and the encapsulation method is an example without limitation: because the jedis client is a universal client of the redis java language, the optimal api encapsulation client redis component of the simulated jedis client is used for obtaining the first customized client, and because the api encapsulation client is a client obtained by encapsulation of the simulated jedis client, the method also has universality, is suitable for most redis versions, and avoids the situation that the versions are incompatible.
S200: obtaining first client request information according to the first user-defined client, wherein the first client request information is user-defined by a first service party;
specifically, the first service party refers to an internet data center applied by the first customized remote synchronization system; the first client request information refers to that the first service party selects data needing different-place synchronization by the first user-defined client according to requirements, and can autonomously select a different-place synchronization interface. The request information of the first client selectively calls a different-place synchronous interface, so that the internet data center synchronization of some unreasonable synchronous data (such as data with extremely short TTL and data with extremely large byte number) is avoided, and the data synchronization efficiency is improved.
S300: analyzing the first client request information through the first user-defined client to obtain a first event queue;
further, based on the analyzing of the first client request information by the first custom client, a first event queue is obtained, and step S300 includes:
s310: analyzing the first client request information through the first custom client, and packaging the first client request information based on a protobuf protocol to obtain first event information;
s320: and calling the first client to request information to request the first local redis cluster, and inserting the first event information into the first event queue.
Specifically, the first event information refers to analyzing the first client request information through the redis component in the first custom client, an analyzing process is mainly data that can be identified and processed by the first custom client, and the analyzed first client request information is preferably encapsulated into a binary event by using the protobuf protocol; the protobuf protocol refers to a serialized framework of an open source, like XML and json, and is characterized in that the template is based on binary, is much shorter than the same segment of content in the traditional XML representation, and can define some optional fields for communication between a server and a client and data communication between distributed applications or data exchange in heterogeneous environments. As a binary data transmission format excellent in efficiency and compatibility, it can be used in various fields such as network transmission, profile, data storage, and the like.
Further, the first local redis cluster refers to a queue cluster used by the first customized remote synchronization system to store data to be synchronized in the internet data center; the first event queue refers to a queue cluster, and the insertion implementation mode is as follows: calling the first client request information to request the first local redis cluster through the redis component in the first custom client, and storing the packaged first event information into the first event queue. The first client request information is called and the first event information is inserted into the first event queue to form an inseparable transaction, if the local transaction fails, the first event information cannot be stored, the problem of partial failure cannot occur, the integrity of the transaction is guaranteed, and the data consistency of the redis cluster synchronized across the internet data center is also guaranteed.
S400: constructing the first synchronous rabbitmq component according to a first local redis cluster, wherein the first synchronous rabbitmq component comprises a first synchronous production end and a first synchronous consumption end;
in particular, the first synchronization rabbitmq component refers to a component that assists the first local redis cluster in synchronizing cached data; the first synchronous production end refers to a port for synchronizing data in the first local redis cluster, and the port is binary event information; the first synchronous consuming terminal calls written synchronous cache event information from the first synchronous producing terminal, analyzes the event information, obtains corresponding first client request information and synchronizes to a port in a remote redis cluster. The first synchronous production end and the first synchronous consumption end jointly form the first synchronous rabbitmq component of the first local redis cluster, and data is synchronously cached through the first synchronous rabbitmq component, so that even if a network fails, the data of the first local redis cluster cannot be lost or rewritten because the data is cached in the first synchronous rabbitmq component; because the first event information of the first local redis cluster is stored in the memory, data may be lost or rewritten when the amount of data is large or the network is interrupted, and the first synchronization rabbitmq component may store the first event information on a storage medium such as a disk, and the storage information is not affected by power failure and is relatively more stable.
S500: writing the first event queue in the first synchronous production end to obtain first write-in information;
further, based on the writing of the first event queue in the first synchronous production end, obtaining first write information, step S500 includes:
s510: writing the first event information into the first synchronous production end, and judging whether the writing of the first event information is finished;
s520: if the writing of the first event information is finished, popping up the first event queue;
s530: and taking the written information in the first synchronous production terminal as the first written information.
Specifically, in order to timely consume the message in the first event queue of the first local redis cluster to the first consuming side, the first synchronous producing side needs to distributively consume the plurality of pieces of first event information in the first event queue in the synchronization process, and the first event information stored in the first event queue is directly written into the first synchronous producing side without parsing, and is stopped when it is determined that the first event information corresponding to the first client request information is completely written, and the first event queue whose information is in contact with the cache data pressure is popped up; the first write-in information refers to the first event information which is directly written into the first synchronous production end in batches in the first event queue, and is stored in a plurality of queues in the first synchronous production end after the first event queue information is popped up. And a plurality of queues are arranged in the first synchronous production end to buffer messages, so that the technical effect of improving the buffer efficiency is achieved.
S600: analyzing the first written information through the first synchronous consuming terminal to obtain a first analysis result, wherein the first analysis result comprises the first event information, and the first event information corresponds to the first client request information;
s700: acquiring the first client request information according to the first event information;
s800: and requesting a first remote redis cluster according to the first client request information, and putting a first local redis command into the first remote redis cluster.
Specifically, the first analysis result refers to that the first synchronous consuming terminal is used for calling the first write-in information, and the first event information is analyzed according to a protobuf protocol to obtain the first client request information encapsulated in a binary event; the first local redis command refers to information representing data needing to be synchronized, and corresponds to the first request information; and sending the request information of the first client to the first remote redis cluster, and calling the first request information by a remote user-defined client to obtain the first local redis command information to perform data synchronization, thereby achieving the technical effect of data synchronization of the cross-internet data center.
Further, constructing the first synchronous rabbitmq component based on the first local redis cluster, as shown in fig. 2, step S400 includes:
s410: obtaining the number of first event queues according to the first local redis cluster;
s420: according to the number of the first event queues, matching the number of the first synchronous rabbitmq writing component queues with the number of the first synchronous rabbitmq consuming component queues;
s430: writing all the first synchronous rabbitmq into an assembly queue as the first synchronous production end;
s440: taking all the first synchronization rabbitmq consumption component queues as the first synchronization consumption ends;
s450: and the first synchronous rabbitmq component is formed by the first synchronous production end and the first synchronous consumption end.
Specifically, the first event queue number refers to the number of queues in the first local redis cluster called according to the first client request information, and is an example without limitation: if the distributor is selected to design the first local redis cluster, the number of slot memory slots designed by the distributor is 16384 when the first local redis cluster is designed, and the distributor refers to a controller for dividing the memory slots into the first local redis cluster; the slot memory slot refers to a storage location where the first event information can be stored after being divided, so that generally, at most, there are 0-16383 queues in the first local redis cluster, and 16384 queues in total, and the first event queues of the first local redis cluster with the same number are called according to the first event information.
Further, after the number of the first event queues is determined, the number of the first synchronous rabbitmq write component queues capable of being written into all the first event queues in parallel and the number of the first synchronous rabbitmq consumption component queues capable of analyzing all the first write information in parallel are matched, as an example without limitation: designing each queue storage amount of the first synchronous rabbitmq writing component and the first synchronous rabbitmq consuming component to be as large as the first event queue, matching twice the number of the first synchronous rabbitmq writing component queues and twice the number of the first synchronous rabbitmq consuming component queues, setting two queues for each group of writing or analyzing, one for work and one for standby, and when one of the queues fails, the other one replaces the work at any time.
Furthermore, all the first synchronous rabbitmq writing component queues are used as the first synchronous production ends, all the first synchronous rabbitmq consuming component queues are used as the first synchronous consuming ends, and then the first synchronous rabbitmq components of the first local redis cluster are constructed. Because the first synchronous rabbitmq component can be stored on a disk medium, but the storage is generally slow, a parallel queue parallel process is adopted here, and the design of a standby queue ensures the flow operation of writing or analyzing data, thereby achieving the technical effect of stable and synchronous data transmission.
Further, obtaining the first client request information based on the first synchronous consuming side according to the first event information, as shown in fig. 3, further includes step S900:
s910: acquiring a first preset processing instruction, processing the first event information and acquiring second event information;
s920: correlating the second event information and the first client request information.
Specifically, the second event refers to that when the first synchronization consuming side analyzes that the first event information is a special event, the specificity of the first event can be set by the remote synchronization system based on the customized information; processing the special first event information according to the first preset processing instruction, which includes, by way of example and not limitation: if the expiration time is to convert relative time to absolute time, the idempotent command is executed. And after the second event information is obtained, correlating the second event information with the first client request information, and executing corresponding first local redis command information. The accuracy of data synchronization is ensured by processing special events.
Further, the method further includes step S1000:
s1010: calling a first distributor through the first user-defined client to generate a first slotCache plug-in, wherein the first slotCache plug-in corresponds to the first event information one by one;
s1020: and maintaining a first mapping relation between the first event information and the first event queue through the first slotCache plug-in.
In particular, the first dispenser refers to the conventional dispenser described above; the first slotCache plug-in means a cache memory for storing the first event information in the slot memory slot. After the binary system first event is packaged, the first dispatcher is called by the first user-defined client to design the first slot cache plug-in, the first event information is stored in a slot memory slot, and the first event queue is formed by the inserted first event information. Further, the first slotCache plug-in may also store a mapping relationship between the first event and the first event queue, that is, the first functional relationship, preferably a hash function, to determine the functional relationship. Through the first functional relationship, the first event information can be quickly found in the first event queue.
Further, the method further includes step S1100:
s1110: the first synchronous production end obtains a slot queue of a first local redis cluster through the first event queue;
s1120: and maintaining a second mapping relation between the first slot queue and the first synchronous rabbitmq writing component queue through the first synchronous production end.
Specifically, the slot queue of the first local redis cluster refers to queue information corresponding to the first event queue one to one; after the first synchronous production end writes the first event queue information, the second mapping relationship refers to a mapping relationship between the first slot queue and the first synchronous rabbitmq writing component queue maintained by the first synchronous production end, that is, a queue corresponding relationship between the first event queue and the first synchronous rabbitmq writing component queue is represented, and preferably, a functional relationship is determined through a hash function. Through the second functional relationship, the stability of data can be protected, and whether the first event information is completely written can be judged.
To sum up, the redis cluster remote synchronization method and system based on rabbitmq provided by the embodiment of the application have the following technical effects:
1. packaging the client redis component according to a first packaging instruction to obtain a first user-defined client; obtaining first client request information according to the first user-defined client, wherein the first client request information is user-defined by a first service party; analyzing the first client request information through the first user-defined client to obtain a first event queue; constructing the first synchronous rabbitmq component according to a first local redis cluster, wherein the first synchronous rabbitmq component comprises a first synchronous production end and a first synchronous consumption end; writing the first event queue in the first synchronous production end to obtain first write-in information; analyzing the first written information through the first synchronous consuming terminal to obtain a first analysis result, wherein the first analysis result comprises the first event information, and the first event information corresponds to the first client request information; acquiring the first client request information according to the first event information; the technical scheme includes that a first remote redis cluster is requested according to first client request information, a first local redis command is placed in the first remote redis cluster, most versions can be compatible through packaging a user-defined client, synchronous data are determined according to client requests, and data redundancy is avoided; the request is packaged, the event queue is built, the event queue is written by the first synchronous production end, the phenomenon that local redis clusters lack data due to network faults can be avoided, the events are analyzed by the first synchronous consumption end, and the technical effect that data can be selected and stably synchronized among multiple internet data centers is achieved.
2. Because the first synchronous rabbitmq component can be stored on a disk medium, but the storage is generally slow, a parallel queue parallel process is adopted here, and the design of a standby queue ensures the flow operation of writing or analyzing data, thereby achieving the technical effect of stable and synchronous data transmission.
Example two
Based on the same inventive concept as the redis cluster allopatric synchronization method based on rabbitmq in the foregoing embodiment, as shown in fig. 4, the embodiment of the present application provides a redis cluster allopatric synchronization system based on rabbitmq, wherein the system includes:
a first obtaining unit 11, where the first obtaining unit 11 is configured to package a client redis component according to a first package instruction, and obtain a first custom client;
a first request unit 12, where the first request unit 12 is configured to obtain first client request information according to the first customized client, where the first client request information is customized by a first service party;
a second obtaining unit 13, where the second obtaining unit 13 is configured to analyze the first client request information through the first custom client, and obtain a first event queue;
a first building unit 14, the first building unit 14 being configured to build a first synchronous rabbitmq component according to a first local redis cluster, the first synchronous rabbitmq component comprising a first synchronous production end and a first synchronous consumption end;
a third obtaining unit 15, where the third obtaining unit 15 is configured to write the first event queue in the first synchronous production end, and obtain first write information;
a fourth obtaining unit 16, where the fourth obtaining unit 16 is configured to analyze the first written information through the first synchronous consuming end to obtain a first analysis result, where the first analysis result includes the first event information, and the first event information corresponds to the first client request information;
a fifth obtaining unit 17, where the fifth obtaining unit 17 is configured to obtain the first client request information according to the first event information;
a first executing unit 18, where the first executing unit 18 is configured to request a first remote redis cluster according to the first client request information, and place a first local redis command in the first remote redis cluster.
Further, the system further comprises:
a sixth obtaining unit, configured to analyze the first client request information through the first custom client, and encapsulate the first client request information based on a protobuf protocol to obtain first event information;
a second execution unit, configured to invoke the first client to request the information to request the first local redis cluster, and insert the first event information into the first event queue;
further, the system further comprises:
a seventh obtaining unit, configured to obtain, according to the first local redis cluster, a first event queue number;
the first matching unit is used for matching the number of the first synchronous rabbitmq writing component queues and the number of the first synchronous rabbitmq consuming component queues according to the number of the first event queues;
a first setting unit, configured to write all the first synchronous rabbitmq into an assembly queue as the first synchronous production end;
a second setting unit, configured to use all the first synchronization rabbitmq consumption component queues as the first synchronization consumption end;
a second building unit to construct the first synchronous rabbitmq assembly from the first synchronous production end and the first synchronous consumption end.
Further, the system further comprises:
a first judging unit, configured to write the first event information into the first synchronous production end, and judge whether the writing of the first event information is completed;
a first ejecting unit, configured to eject the first event queue if the first event information is written completely;
and the third setting unit is used for taking the written information in the first synchronous production terminal as the first written information.
Further, the system further comprises:
an eighth obtaining unit, configured to obtain a first preset processing instruction, process the first event information, and obtain second event information;
a first associating unit configured to associate the second event information and the first client request information with each other.
Further, the system further comprises:
the first generating unit is used for calling a first distributor through the first user-defined client to generate a first slotCache plug-in, and the first slotCache plug-in corresponds to the first event information one by one;
and the first maintenance unit is used for maintaining the first event information and the first mapping relation of the first event queue through the first slotCache plug-in.
Further, the system further comprises:
a ninth obtaining unit, configured to obtain, by the first synchronous production end, a slot queue of a first local redis cluster through the first event queue;
and the second maintenance unit is used for maintaining a second mapping relation between the first slot queue and the first synchronous rabbitmq write-in component queue through the first synchronous production end.
Exemplary electronic device
The electronic device of the embodiment of the present application is described below with reference to figure 5,
based on the same inventive concept as the redis cluster allopatric synchronization method based on rabbitmq in the previous embodiment, the embodiment of the present application further provides a redis cluster allopatric synchronization system based on rabbitmq, which includes: a processor coupled to a memory, the memory for storing a program that, when executed by the processor, causes a system to perform the method of any of the first aspects.
The electronic device 300 includes: processor 302, communication interface 303, memory 301. Optionally, the electronic device 300 may also include a bus architecture 304. Wherein, the communication interface 303, the processor 302 and the memory 301 may be connected to each other through a bus architecture 304; the bus architecture 304 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus architecture 304 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 5, but this is not intended to represent only one bus or type of bus.
The communication interface 303 may be any device, such as a transceiver, for communicating with other devices or communication networks, such as an ethernet, a Radio Access Network (RAN), a Wireless Local Area Network (WLAN), a wired access network, and the like.
The memory 301 may be, but is not limited to, a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, an electrically erasable Programmable read-only memory (EEPROM), a compact disk read-only memory (CD-ROM) or other optical disk storage, optical disk storage (including compact disk, laser disk, optical disk, digital versatile disk, blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory may be self-contained and coupled to the processor through a bus architecture 304. The memory may also be integral to the processor.
The memory 301 is used for storing computer-executable instructions for executing the present application, and is controlled by the processor 302 to execute. The processor 302 is configured to execute the computer-executable instructions stored in the memory 301, so as to implement a rabbitmq-based redis cluster remote synchronization method provided by the above-mentioned embodiment of the present application.
Optionally, the computer-executable instructions in the embodiments of the present application may also be referred to as application program codes, which are not specifically limited in the embodiments of the present application.
The embodiment of the application provides a redis cluster allopatric synchronization method based on rabbitmq, which is applied to a user-defined information allopatric synchronization system, wherein the first user-defined allopatric synchronization system comprises a client redis component and a first synchronization rabbitmq component, and the method comprises the following steps: packaging the client redis component according to a first packaging instruction to obtain a first user-defined client; obtaining first client request information according to the first user-defined client, wherein the first client request information is user-defined by a first service party; analyzing the first client request information through the first user-defined client to obtain a first event queue; constructing the first synchronous rabbitmq component according to a first local redis cluster, wherein the first synchronous rabbitmq component comprises a first synchronous production end and a first synchronous consumption end; writing the first event queue in the first synchronous production end to obtain first write-in information; analyzing the first written information through the first synchronous consuming terminal to obtain a first analysis result, wherein the first analysis result comprises the first event information, and the first event information corresponds to the first client request information; acquiring the first client request information according to the first event information; requesting a first remote redis cluster according to the first client request information, putting a first local redis command into the first remote redis cluster, enabling a user-defined client to be compatible with most versions through encapsulation, determining synchronous data according to the client request, and avoiding data redundancy; the request is packaged, the event queue is built, the event queue is written by the first synchronous production end, the situation that local redis clusters lack data due to network faults can be avoided, the events are analyzed by the first synchronous consumption end, and the technical effect that data can be selected and stably synchronized among multiple internet data centers is achieved.
Those of ordinary skill in the art will understand that: the various numbers of the first, second, etc. mentioned in this application are only used for the convenience of description and are not used to limit the scope of the embodiments of this application, nor to indicate the order of precedence. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one" means one or more. At least two means two or more. "at least one," "any," or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one (one ) of a, b, or c, may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or multiple.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device including one or more available media integrated servers, data centers, and the like. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The various illustrative logical units and circuits described in this application may be implemented or operated upon by design of a general purpose processor, a digital signal processor, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration.
The steps of a method or algorithm described in the embodiments herein may be embodied directly in hardware, in a software element executed by a processor, or in a combination of the two. The software cells may be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. For example, a storage medium may be coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC, which may be disposed in a terminal. In the alternative, the processor and the storage medium may reside in different components within the terminal. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Although the present application has been described in conjunction with specific features and embodiments thereof, it will be evident that various modifications and combinations can be made thereto without departing from the spirit and scope of the application. Accordingly, the specification and figures are merely exemplary of the present application as defined in the appended claims and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of the present application. It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations.
Claims (9)
1. A redis cluster allopatric synchronization method based on rabbitmq is applied to a user-defined information allopatric synchronization system, wherein the first user-defined allopatric synchronization system comprises a client redis component and a first synchronization rabbitmq component, and the method comprises the following steps:
packaging the client redis component according to a first packaging instruction to obtain a first user-defined client;
obtaining first client request information according to the first user-defined client, wherein the first client request information is user-defined by a first service party;
analyzing the first client request information through the first user-defined client to obtain a first event queue;
constructing the first synchronous rabbitmq component according to a first local redis cluster, wherein the first synchronous rabbitmq component comprises a first synchronous production end and a first synchronous consumption end;
writing the first event queue in the first synchronous production end to obtain first write-in information;
analyzing the first written information through the first synchronous consuming terminal to obtain a first analysis result, wherein the first analysis result comprises the first event information, and the first event information corresponds to the first client request information;
acquiring the first client request information according to the first event information;
and requesting a first remote redis cluster according to the first client request information, and putting a first local redis command into the first remote redis cluster.
2. The method of claim 1, wherein said parsing the first client request information by the first custom client to obtain a first event queue comprises:
analyzing the first client request information through the first custom client, and packaging the first client request information based on a protobuf protocol to obtain first event information;
and calling the first client to request information to request the first local redis cluster, and inserting the first event information into the first event queue.
3. The method of claim 1, wherein said building the first synchronous rabbitmq component from the first local redis cluster comprises:
obtaining the number of first event queues according to the first local redis cluster;
according to the number of the first event queues, matching the number of the first synchronous rabbitmq writing component queues with the number of the first synchronous rabbitmq consuming component queues;
writing all the first synchronous rabbitmq into an assembly queue as the first synchronous production end;
taking all the first synchronization rabbitmq consumption component queues as the first synchronization consumption ends;
and the first synchronous rabbitmq component is formed by the first synchronous production end and the first synchronous consumption end.
4. The method of claim 1, wherein said writing the first event queue in the first synchronous production peer, obtaining first write information, comprises:
writing the first event information into the first synchronous production end, and judging whether the writing of the first event information is finished;
if the writing of the first event information is finished, popping up the first event queue;
and taking the written information in the first synchronous production terminal as the first written information.
5. The method of claim 1, wherein the first synchronous consumer obtains the first client request information according to the first event information, further comprising:
acquiring a first preset processing instruction, processing the first event information and acquiring second event information;
correlating the second event information and the first client request information.
6. The method of claim 2, wherein the method further comprises:
calling a first distributor through the first user-defined client to generate a first slotCache plug-in, wherein the first slotCache plug-in corresponds to the first event information one by one;
and maintaining a first mapping relation between the first event information and the first event queue through the first slotCache plug-in.
7. The method of claim 6, wherein the method further comprises:
the first synchronous production end obtains a slot queue of a first local redis cluster through the first event queue;
and maintaining a second mapping relation between the first slot queue and the first synchronous rabbitmq writing component queue through the first synchronous production end.
8. A rabbitmq-based rediscluster allopatric synchronization system, wherein the system comprises:
the first obtaining unit is used for packaging the client redis component according to a first packaging instruction to obtain a first user-defined client;
the first request unit is used for acquiring first client request information according to the first user-defined client, and the first client request information is defined by a first service party;
a second obtaining unit, configured to analyze the first client request information through the first custom client to obtain a first event queue;
the first building unit is used for building a first synchronous rabbitmq component according to a first local redis cluster, and the first synchronous rabbitmq component comprises a first synchronous production end and a first synchronous consumption end;
a third obtaining unit, configured to write the first event queue in the first synchronous production end, and obtain first write information;
a fourth obtaining unit, configured to analyze the first written information through the first synchronous consuming terminal to obtain a first analysis result, where the first analysis result includes the first event information, and the first event information corresponds to the first client request information;
a fifth obtaining unit, configured to obtain the first client request information according to the first event information;
a first execution unit, configured to request a first remote redis cluster according to the first client request information, and place a first local redis command in the first remote redis cluster.
9. A redis cluster remote synchronization system based on rabbitmq comprises: a processor coupled with a memory for storing a program that, when executed by the processor, causes a system to perform the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111012818.1A CN113794757B (en) | 2021-08-31 | 2021-08-31 | Redis cluster off-site synchronization method and system based on rabitmq |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111012818.1A CN113794757B (en) | 2021-08-31 | 2021-08-31 | Redis cluster off-site synchronization method and system based on rabitmq |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113794757A true CN113794757A (en) | 2021-12-14 |
| CN113794757B CN113794757B (en) | 2024-03-19 |
Family
ID=78876750
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111012818.1A Active CN113794757B (en) | 2021-08-31 | 2021-08-31 | Redis cluster off-site synchronization method and system based on rabitmq |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113794757B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115514746A (en) * | 2022-09-21 | 2022-12-23 | 平安银行股份有限公司 | Instant messaging method, device, system, equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120278473A1 (en) * | 2011-04-27 | 2012-11-01 | Rackspace Us, Inc. | Event Queuing and Distribution System |
| US20140019626A1 (en) * | 2012-07-16 | 2014-01-16 | eZuce, Inc. | Providing unified communications services |
| US20190312778A1 (en) * | 2018-04-05 | 2019-10-10 | Aeris Communications, Inc. | GLOBAL DEVICE MANAGEMENT ARCHITECTURE FOR IoT DEVICES WITH REGIONAL AUTONOMY |
| CN110933145A (en) * | 2019-11-14 | 2020-03-27 | 光通天下网络科技股份有限公司 | Remote scheduling method, device, equipment and medium |
| US20200145344A1 (en) * | 2018-11-01 | 2020-05-07 | International Business Machines Corporation | Operating a message queue cluster having multiple nodes |
| CN111756811A (en) * | 2020-05-29 | 2020-10-09 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for actively pushing distributed system |
| CN112052104A (en) * | 2020-09-01 | 2020-12-08 | 掌阅科技股份有限公司 | Message queue management method based on multi-computer-room realization and electronic equipment |
| CN112615666A (en) * | 2020-12-19 | 2021-04-06 | 河南方达空间信息技术有限公司 | Micro-service high-availability deployment method based on RabbitMQ and HAproxy |
-
2021
- 2021-08-31 CN CN202111012818.1A patent/CN113794757B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120278473A1 (en) * | 2011-04-27 | 2012-11-01 | Rackspace Us, Inc. | Event Queuing and Distribution System |
| US20140019626A1 (en) * | 2012-07-16 | 2014-01-16 | eZuce, Inc. | Providing unified communications services |
| US20190312778A1 (en) * | 2018-04-05 | 2019-10-10 | Aeris Communications, Inc. | GLOBAL DEVICE MANAGEMENT ARCHITECTURE FOR IoT DEVICES WITH REGIONAL AUTONOMY |
| US20200145344A1 (en) * | 2018-11-01 | 2020-05-07 | International Business Machines Corporation | Operating a message queue cluster having multiple nodes |
| CN110933145A (en) * | 2019-11-14 | 2020-03-27 | 光通天下网络科技股份有限公司 | Remote scheduling method, device, equipment and medium |
| CN111756811A (en) * | 2020-05-29 | 2020-10-09 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for actively pushing distributed system |
| CN112052104A (en) * | 2020-09-01 | 2020-12-08 | 掌阅科技股份有限公司 | Message queue management method based on multi-computer-room realization and electronic equipment |
| CN112615666A (en) * | 2020-12-19 | 2021-04-06 | 河南方达空间信息技术有限公司 | Micro-service high-availability deployment method based on RabbitMQ and HAproxy |
Non-Patent Citations (1)
| Title |
|---|
| 袁佳;郭燕慧;: "基于rabbitmq的海量日志的分布式处理", 软件, no. 07 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115514746A (en) * | 2022-09-21 | 2022-12-23 | 平安银行股份有限公司 | Instant messaging method, device, system, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113794757B (en) | 2024-03-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021203968A1 (en) | Method for unified management of micro-services of multiple registration centers, and apparatus, device and medium | |
| EP3404542A1 (en) | Data pipeline architecture for analytics processing stack | |
| CN107463418B (en) | Configuration file generation method and device for server middleware | |
| US7506334B2 (en) | Common, multi-version interface that allows a distributed sybsystem to communicate with multiple versions of the distributed subsystem | |
| CA2747783C (en) | Extensibility for web based diagram visualization | |
| CN109254854A (en) | Asynchronous invoking method, computer installation and storage medium | |
| WO2022247201A1 (en) | Data query method and apparatus | |
| CN113193947B (en) | Method, apparatus, medium, and program product for implementing distributed global ordering | |
| CN112039701A (en) | Interface call monitoring method, device, equipment and storage medium | |
| CN113794757A (en) | Method and system for remote synchronization of redis cluster based on rabbitmq | |
| CN113010332A (en) | Remote service calling method, device, equipment and storage medium | |
| WO2021211717A1 (en) | System and method for reactive messaging client for use with a microservices environment | |
| CN108111630B (en) | Zookeeper cluster system and connection method and system thereof | |
| CN113132371A (en) | Message conversion method, device, computer readable storage medium and processor | |
| CN109062870A (en) | The method and storage medium that server, business datum verify | |
| CN116800616A (en) | Management method and related device of virtualized network equipment | |
| WO2013059887A1 (en) | Data interchange system | |
| WO2017084515A1 (en) | Method and device for transmitting data code stream | |
| CN115328457A (en) | Method and device for realizing form page based on parameter configuration | |
| CN112052104A (en) | Message queue management method based on multi-computer-room realization and electronic equipment | |
| CN112187916A (en) | Cross-system data synchronization method and device | |
| CN111679899A (en) | Task scheduling method, device, platform equipment and storage medium | |
| CN113778764B (en) | Hbase data dual-activity system and method | |
| WO2022022446A1 (en) | Log printing method, device and system | |
| CN114238479A (en) | Data import method, device, equipment and medium based on yellowbreak database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100000 floors 1-3, block a, global creative Plaza, No. 10, Furong street, Chaoyang District, Beijing Applicant after: Bairong Zhixin (Beijing) Technology Co.,Ltd. Address before: 100000 floors 1-3, block a, global creative Plaza, No. 10, Furong street, Chaoyang District, Beijing Applicant before: Bairong Zhixin (Beijing) credit investigation Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |