CN114745438A - Cache data processing method, device and system for multiple data centers and electronic equipment - Google Patents
Cache data processing method, device and system for multiple data centers and electronic equipment Download PDFInfo
- Publication number
- CN114745438A CN114745438A CN202210288959.4A CN202210288959A CN114745438A CN 114745438 A CN114745438 A CN 114745438A CN 202210288959 A CN202210288959 A CN 202210288959A CN 114745438 A CN114745438 A CN 114745438A
- Authority
- CN
- China
- Prior art keywords
- cache
- data
- written
- cluster
- cache data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims description 18
- 238000000034 method Methods 0.000 claims abstract description 41
- 238000012545 processing Methods 0.000 claims abstract description 37
- 230000010076 replication Effects 0.000 claims description 36
- 238000004590 computer program Methods 0.000 claims description 10
- 230000004044 response Effects 0.000 claims description 9
- 239000000126 substance Substances 0.000 claims 1
- 238000004891 communication Methods 0.000 description 8
- 230000000694 effects Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 239000003795 chemical substances by application Substances 0.000 description 6
- 230000001360 synchronised effect Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000005192 partition Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000003362 replicative effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method, a device and a system for processing cache data of multiple data centers and electronic equipment. Wherein, the method comprises the following steps: responding to the write-in operation of the business system to the cache data in the cache cluster, determining the cache data to be written in, and writing the cache data to be written in the cache cluster; and sending the cache data to be written to the first proxy node in other cache clusters through the MQ cluster, and enabling the first proxy node to replay the cache data to be written to other cache clusters. Therefore, cache data among data centers at a plurality of geographical positions far away from each other can be copied efficiently and accurately in near real time.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for processing cache data in multiple data centers, a cache data processing system, an electronic device, and a computer-readable storage medium.
Background
In order to better provide data services required by businesses to users, enterprises generally need to provide business support through a plurality of data centers located at different geographic locations and perform disaster recovery backup. In order to ensure the consistency of data stored in a plurality of data centers, efficient and accurate near real-time replication of cache data among the plurality of data centers is required.
Disclosure of Invention
The embodiment of the application provides a cache data processing method and device for a multi-data center, a cache data processing system, electronic equipment and a storage medium. The efficient and accurate near real-time copying of the cache data can be realized among data centers which are far away from each other in a plurality of geographic positions.
In a first aspect, an embodiment of the present application provides a method for processing cache data in multiple data centers, where each data center includes cache clusters, a replication relationship is established between each cache cluster through a message queue MQ cluster, each cache cluster includes a proxy node, and the method is applied to the proxy node, and the method includes: responding to the writing operation of a service system to the cache data in the cache cluster, determining cache data to be written, and writing the cache data to be written into the cache cluster; and sending the cache data to be written to a first proxy node in other cache clusters through the MQ cluster, and enabling the first proxy node to replay the cache data to be written to the other cache clusters.
By the technical scheme, the high-efficiency and accurate quasi-real-time copying of the cache data of the multiple data centers can be realized, so that low-delay local cache data access is provided for service subsystems in different data centers, and the method can be used for realizing remote multi-activity of the cache data.
In one implementation, each cache cluster further includes a data node, and the data node logically divides the whole data set according to a Key of the cache data and is responsible for different data subsets; the writing the cache data to be written into the cache cluster includes: and sending the cache data to be written to a first data node in the cache cluster, and enabling the first data node to store the cache data to be written which is in charge of the first data node according to a Key in the cache data to be written.
In one implementation, the sending, by the MQ cluster, the to-be-written cache data to the proxy nodes in other cache clusters includes: and selecting a copy synchronization path according to the cache data to be written, and sending the cache data to be written to proxy nodes in other cache clusters through the MQ cluster.
In an optional implementation manner, the selecting a copy synchronization path according to the to-be-written cache data, and sending the to-be-written cache data to a first proxy node in another cache cluster through the MQ cluster includes: and responding to that the Key fields in the cache data to be written are the same, and sending the cache data to be written with the same Key fields to the first proxy nodes in other cache clusters through the same replication synchronization path through the MQ cluster.
By the technical scheme, the orderliness of copying the multiple pieces of cache data to be written with the same Key fields among multiple data centers can be ensured, so that the cache data of the multiple data centers can be accurately copied.
In an optional implementation manner, the selecting a copy synchronization path according to the to-be-written cache data, and sending the to-be-written cache data to a first proxy node in another cache cluster through the MQ cluster includes: setting the same hash tag for the cache data to be written in response to the cache data to be written containing a plurality of different Key fields; and sending the cache data to be written with the same hash tag to the first proxy nodes in other cache clusters through the same replication synchronization path through the MQ cluster.
By the technical scheme, the orderliness of copying the different cache data to be written in among the data centers can be ensured, so that the accurate copying of the cache data of the data centers is realized.
In a second aspect, the present application provides a device for processing cache data in multiple data centers, where each data center includes cache clusters, a replication relationship is established between each cache cluster through a message queue MQ cluster, each cache cluster includes a proxy node, and the device is applied to the proxy node, and the device includes: the write-in module is used for responding to the write-in operation of the business system on the cache data in the cache cluster, determining the cache data to be written in and writing the cache data to be written in the cache cluster; and the sending module is used for sending the cache data to be written to the first proxy node in other cache clusters through the MQ cluster and enabling the first proxy node to replay the cache data to be written to the other cache clusters.
In one implementation, each cache cluster further includes a data node, and the data node logically divides the whole data set according to a Key of the cache data and is responsible for different data subsets; the write module is specifically configured to: and sending the cache data to be written to a first data node in the cache cluster, and enabling the first data node to store the cache data to be written which is in charge of the first data node according to a Key in the cache data to be written.
In one implementation, the sending module is specifically configured to: and selecting a copy synchronization path according to the cache data to be written, and sending the cache data to be written to the first proxy node in other cache clusters through the MQ cluster.
In an optional implementation manner, the sending module is specifically configured to: and responding to that the Key fields in the cache data to be written are the same, and sending the cache data to be written with the same Key fields to the first proxy nodes in other cache clusters through the same replication synchronization path through the MQ cluster.
In an optional implementation manner, the sending module is specifically configured to: and responding to that the Key fields in the cache data to be written are the same, and sending the cache data to be written with the same Key fields to the first proxy nodes in other cache clusters through the same replication synchronization path through the MQ cluster.
In an optional implementation manner, the sending module is specifically configured to: setting the same hash tag for the cache data to be written in response to the cache data to be written containing a plurality of different Key fields; and sending the cache data to be written with the same hash label to the first proxy nodes in other cache clusters through the same replication synchronization path through the MQ cluster.
In a third aspect, an embodiment of the present application provides a cache data processing system, including a plurality of data centers, where each data center includes a cache cluster, a replication relationship is established between each cache cluster through a message queue MQ cluster, and each cache cluster includes a proxy node; the proxy node is used for responding to the write-in operation of the business system to the cache data in the cache cluster and determining the cache data to be written; the proxy node is further configured to send the cache data to be written to a first proxy node in another cache cluster through the MQ cluster, and enable the first proxy node to replay the cache data to be written to the other cache cluster.
In one implementation, each cache cluster further includes a data node, and the data node logically divides the whole data set according to a Key of the cache data and is responsible for different data subsets; the proxy node is further configured to send the cache data to be written to the data node; and the data node is used for storing the cache data to be written which is in charge of the data node according to the Key in the cache data to be written.
In one implementation, the proxy node is specifically configured to: and selecting a copy synchronization path according to the cache data to be written, and sending the cache data to be written to the first proxy node in other cache clusters through the MQ cluster.
In an optional implementation manner, the proxy node is specifically configured to: and responding to that the Key fields in the cache data to be written are the same, and sending the cache data to be written with the same Key fields to the first proxy nodes in other cache clusters through the same replication synchronization path through the MQ cluster.
In an optional implementation manner, the proxy node is specifically configured to: responding to the cache data to be written containing a plurality of different Key fields, and setting the same hash tag for the cache data to be written; and sending the cache data to be written with the same hash tag to the first proxy nodes in other cache clusters through the same replication synchronization path through the MQ cluster.
In a fourth aspect, an embodiment of the present application provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of cache data processing for multiple data centers of the first aspect.
In a fifth aspect, the present application provides a storage medium storing instructions that, when executed, cause the method according to the first aspect to be implemented.
In a sixth aspect, an embodiment of the present application provides a computer program product including a computer program, where the computer program, when executed by a processor, implements the method for processing cache data in multiple data centers according to the first aspect.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present application, nor do they limit the scope of the present application. Other features of the present application will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
fig. 1 is a flowchart of a cache data processing method for multiple data centers according to an embodiment of the present disclosure;
fig. 2 is a flowchart of another method for processing cache data in multiple data centers according to an embodiment of the present application;
fig. 3 is a flowchart of another method for processing cache data in multiple data centers according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a cache data processing apparatus for multiple data centers according to an embodiment of the present application;
FIG. 5 is a diagram illustrating an exemplary cache data processing system according to an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of an electronic device for implementing a cache data processing method for multiple data centers according to an embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Where in the description of the present application, "/" indicates an OR meaning, for example, A/B may indicate A or B; in the present application, "and/or" is only an association relationship describing an associated object, and means that there may be three relationships, for example, a and/or B, and may mean: a exists alone, A and B exist simultaneously, and B exists alone. The various numbers of the first, second, etc. mentioned in this application are only used for the convenience of description and are not used to limit the scope of the embodiments of this application, nor to indicate the order of precedence.
First, there is a need for convention on the terms referred to in this application.
The term "Replication" in embodiments of the present application is a technique of replicating a set of data from one data source to one or more data sources, and is divided into synchronous Replication and asynchronous Replication.
Wherein, the term "synchronous replication" means: each write operation is required to be completed at both the source and target ends before the next operation process is performed. The characteristic is that the data loss is less.
The term "asynchronous replication" refers to: the data is not waited to be copied to the target system before processing the next operation. The characteristic is that the copied data has time difference with the source data, but the copying has less influence on the performance of the production system.
The term "cache data": mainly refers to data stored in a K (Key) V (Value) system represented by Redis (generally referred to as a data structure server), and is characterized in that a specific Key is specified when reading and writing.
The term "MQ (Message Queue) cluster," also known as an MQ component, encompasses multiple MQ nodes, an MQ component being a distributed Message Queue software that can provide Message storage and delivery functions.
The term "Proxy agent node": in order to realize the horizontal expansion capability of the cache system, cache data fragments need to be stored on different server (service) nodes, Proxy maintains the fragment mapping relationship of the data, and services access the cache data through the Proxy nodes.
The term "hashtag": in order to realize that data of a plurality of keys are mapped to the same node through hash calculation, the same tag (label) is added in different keys to guarantee.
Referring to fig. 1, fig. 1 is a flowchart of a cache data processing method for multiple data centers according to an embodiment of the present disclosure. In the embodiment of the present application, each data center may include a cache cluster, a replication relationship is established between each cache cluster through an MQ cluster, and each cache cluster includes a proxy node. For example, a separate cache cluster may be built in a plurality of data centers that need to be replicated and synchronized, and the cache cluster may employ a Proxy Based architecture (such as data node storage + Proxy node forwarding), wherein the Proxy node of the cache cluster is modified according to the present application, so that a replication relationship may be established between the cache clusters through the MQ cluster, wherein the MQ cluster supports a partition mechanism, messages may be transmitted in parallel between a plurality of partitions, and different partitions may be bound to different nodes in the MQ cluster, so that the MQ cluster may also linearly improve the concurrent processing capability of the system by increasing the number of nodes.
It should be noted that the cache data processing method for multiple data centers according to the embodiment of the present application may be applied to a proxy node. The following describes an implementation manner of the cache data processing method for multiple data centers according to the embodiment of the present application from a proxy node side on a cache cluster in a certain data center. As shown in fig. 1, the method for processing cache data in multiple data centers may include, but is not limited to, the following steps.
Step S101, responding to the writing operation of the business system to the cache data in the cache cluster, determining the cache data to be written, and writing the cache data to be written into the cache cluster.
In this embodiment, the service system may be a service system corresponding to a data center to which an execution main body "proxy node" of the cache data processing method described in this embodiment belongs.
In one implementation mode, the service system reads and writes local cache data through an agent node of the local cache system, when the data is read and written, the agent node judges whether the data is written, and if the data is written, the agent node writes cache data to be written into a cache cluster.
For example, when the service system needs to write the cache data of the data center, the service system may issue the data that needs to be written to the proxy node of any data center, and the proxy node writes the received data as the cache data to be written to the corresponding cache data of the data center.
In one implementation mode, each cache cluster further comprises data nodes, and the data nodes logically divide the whole data set according to a Key of cache data and are responsible for different data subsets; the above implementation manner of writing the cache data to be written into the cache cluster may be as follows: and sending the cache data to be written to a first data node in the cache cluster, and enabling the first data node to store the cache data to be written which is in charge of the first data node according to a Key in the cache data to be written.
For example, the cache cluster of each data center may include a plurality of data nodes, the plurality of data nodes may perform logical segmentation on the data set of the entire data center according to the difference of the Key fields of the cache data to obtain a plurality of data subsets, and each data node is responsible for one corresponding data subset, so that the entire cache cluster may improve the concurrent processing capability by increasing the number of nodes linearly. After the data node receives the cache data to be written, which is forwarded by the proxy node, the data node determines a data subset where the cache data corresponding to the cache data to be written is located according to the Key field of the cache data to be written, and the data node corresponding to the data subset writes the cache data to be written. Therefore, in the technical scheme of the application, the data node logically divides the whole data set according to the key of the cache data and is responsible for different data subsets, so that the whole cache cluster can linearly improve the concurrent processing capacity by increasing the number of the nodes.
Step S102, sending the cache data to be written to the first proxy node in other cache clusters through the MQ cluster, and enabling the first proxy node to replay the cache data to be written to other cache clusters.
The first proxy node refers to a proxy node of any one of the other data centers except for a proxy node which directly receives and executes write-in operation of the business system to the cache data in the cache cluster, among the proxy nodes corresponding to the plurality of data centers.
For example, the proxy node that receives and executes the write operation of the service system on the cache data in the cache cluster may send the cache data to be written to the first proxy node through the MQ cluster, and the first proxy node performs data playback on the cache data to be written after receiving the cache data to be written sent by the MQ cluster, so as to write the cache data to be written into the cache data of the data center.
In one implementation, the replication synchronization path may be selected according to the cache data to be written, and the cache data to be written is sent to the proxy nodes in other cache clusters through the MQ cluster.
For example, after the MQ cluster receives the cache data to be written, which is forwarded by the proxy node, the copy synchronization path of the cache data to be written during transmission in the MQ cluster may be selected according to the Key of the cache data to be written, so as to send the cache data to be written to the first proxy node.
By implementing the embodiment of the application, the high-efficiency and accurate quasi-real-time copying of the cache data of a plurality of data centers can be realized, so that low-delay local cache data access is provided for service subsystems in different data centers, and the method can also be used for realizing remote multi-activity of the cache data.
Referring to fig. 2, fig. 2 is a flowchart of another method for processing cached data in multiple data centers according to an embodiment of the present application. According to the method, the copy synchronization path of the cache data to be written can be selected according to the Key field of the cache data to be written, so that the cache data to be written can be sent to other data center agent nodes. As shown in fig. 2, the method for processing cache data in multiple data centers may include, but is not limited to, the following steps.
Step S201, in response to the writing operation of the service system on the cache data in the cache cluster, determining the cache data to be written, and writing the cache data to be written into the cache cluster.
In the embodiment of the present application, step S201 may be implemented by any one of the embodiments of the present application, which is not limited in this embodiment and is not described again.
Step S202, responding to that the Key fields in the cache data to be written are all the same, sending the cache data to be written with the same Key fields to the first proxy nodes in other cache clusters through the MQ cluster through the same replication synchronous path, and enabling the first proxy nodes to replay the cache data to be written to other cache clusters.
For example, when the proxy node receiving the write operation of the service system on the cache data in the cache cluster communicates with the MQ cluster, the MQ cluster may send, to the first proxy node, multiple pieces of to-be-written cache data corresponding to the write operation of the multiple service systems on the cache data in the cache cluster within the same session time. When the Key fields of the cache data to be written are the same, the copy synchronization paths for transmitting the cache data to be written with the same Key fields in the MQ cluster are also the same, so that the first proxy node replays the cache data to be written with the same Key fields according to the write operation sequence of the cache data by the service system and writes the cache data to be written into the corresponding cache cluster. And ensuring multiple write operations on the cached data of the same Key field, and enabling the cached cluster of the data center corresponding to the first proxy node to take effect according to the write operation sequence of the service system, so that the write operation order in the same session can be ensured in the copying process.
By implementing the embodiment of the application, the copy synchronization path of the cache data to be written, which is forwarded by the MQ cluster and has the same Key field, can be ensured to have uniqueness, so that all write operations on the cache data of the same Key field are ensured to take effect in other data center cache clusters strictly according to the write sequence of the business system, and the accurate copy of the cache data of a plurality of data centers is realized.
Referring to fig. 3, fig. 3 is a flowchart of another cache data processing method for multiple data centers according to an embodiment of the present application. According to the method, the copy synchronization path of the cache data to be written can be selected according to the Key field of the cache data to be written, so that the cache data to be written can be sent to other data center agent nodes. As shown in fig. 3, the method for processing cache data in multiple data centers may include, but is not limited to, the following steps.
Step S301, responding to the writing operation of the business system to the cache data in the cache cluster, determining the cache data to be written, and writing the cache data to be written into the cache cluster.
In the embodiment of the present application, step S301 may be implemented by any one of the embodiments of the present application, which is not limited in this embodiment and is not described again.
Step S302, in response to that the cache data to be written contains a plurality of different Key fields, setting the same hash tag for the cache data to be written.
For example, when Key fields of multiple pieces of cache data to be written, which are forwarded by the proxy node through the MQ cluster within the same session time, are different, the proxy node may add the same hash tag to the multiple pieces of cache data to be written.
As an example, a Key to be written into the cache data may be named in a agreed format, and a specific part in a Key field of the cache data to be written is designated as a hash tag.
Step S303, sending the to-be-written cache data with the same hash tag to the first proxy node in other cache clusters through the same replication synchronization path through the MQ cluster, and enabling the first proxy node to replay the to-be-written cache data to other cache clusters.
For example, when the proxy node receiving the write operation of the service system on the cache data in the cache cluster communicates with the MQ cluster, the proxy node may send, to the first proxy node, the plurality of pieces of cache data to be written corresponding to the write operation of the plurality of service systems on the cache data in the cache cluster within the same session time. In the transmission process, the MQ cluster transmits a plurality of pieces of cache data to be written, which contain the same Hash tag in the Key field, and the copy synchronization paths for transmission are also the same, so that the first proxy node replays the cache data to be written according to the write operation sequence of the business system to the cache data and writes the cache data to the corresponding cache cluster. And ensuring all write operations of a plurality of pieces of cache data to be written, and enabling the cache clusters to take effect in the data center corresponding to the first proxy node according to the sequence of the write operations of the service system, so that the ordering of the write operations in the same session can be ensured in the copying process.
By implementing the embodiment of the application, the uniqueness of the copy synchronization path of the cache data to be written, which is forwarded by the MQ cluster and has different Key fields, can be ensured, so that all write operations on the cache data of different Key fields are ensured, the cache data take effect in other data center cache clusters strictly according to the write sequence of the business system, and the accurate copy of the cache data of a plurality of data centers is realized.
Referring to fig. 4, as shown in fig. 4, a schematic structural diagram of a cache data processing apparatus for multiple data centers according to an embodiment of the present invention is shown. In the embodiment of the application, each data center comprises a cache cluster, a replication relationship is established among the cache clusters through a message queue MQ cluster, each cache cluster comprises a proxy node, and the device is applied to the proxy node. The cache data processing device of the multi-data center comprises a writing module 401 and a sending module 402.
The writing module 401 is configured to determine, in response to a write operation of the service system on the cache data in the cache cluster, to-be-written cache data, and write the to-be-written cache data into the cache cluster; the sending module 402 is configured to send the cache data to be written to the first proxy node in another cache cluster through the MQ cluster, and enable the first proxy node to replay the cache data to be written to another cache cluster.
In one implementation mode, each cache cluster further comprises data nodes, and the data nodes logically divide the whole data set according to keys of cache data and are responsible for different data subsets; the write module 401 is specifically configured to: and sending the cache data to be written to a first data node in the cache cluster, and enabling the first data node to store the cache data to be written which is in charge of the first data node according to a Key in the cache data to be written.
In one implementation, the sending module 402 is specifically configured to: and selecting a copy synchronization path according to the cache data to be written, and sending the cache data to be written to the first proxy nodes in other cache clusters through the MQ cluster.
In an optional implementation manner, the sending module 402 is specifically configured to: and responding to the fact that the Key fields in the cache data to be written are the same, and sending the cache data to be written with the same Key fields to the first proxy nodes in other cache clusters through the MQ cluster through the same replication synchronous path.
In an optional implementation manner, the sending module 402 is specifically configured to: setting the same hash tag for the cache data to be written in response to the fact that the cache data to be written contains a plurality of different Key fields; and sending the cache data to be written with the same hash label to the first proxy nodes in other cache clusters through the same replication synchronization path by the MQ cluster.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
According to the cache data processing device of the multiple data centers, the cache data of the multiple data centers can be efficiently and accurately copied in a quasi-real-time manner, so that low-delay local cache data access is provided for service subsystems in different data centers, and the cache data processing device can be used for realizing remote multi-activity of the cache data.
Referring to fig. 5, fig. 5 is a schematic diagram of a cache data processing system according to an embodiment of the present disclosure. As shown in fig. 5, the cache data processing system includes a plurality of data centers 501, where each data center includes a cache cluster, a replication relationship is established between each cache cluster through a message queue MQ cluster, and each cache cluster includes a proxy node 502; the proxy node 502 is configured to determine, in response to a write operation of the service system on cache data in the cache cluster, to-be-written cache data; the proxy node 502 is further configured to send the cache data to be written to a first proxy node in another cache cluster through the MQ cluster, so that the first proxy node replays the cache data to be written to the other cache cluster.
It should be noted that, in order to ensure the high performance requirement of the cache cluster, an asynchronous replication method may be adopted, that is, before the proxy node processes the next operation, it does not wait for the write-in cache data replication to complete the write-in other data centers.
In one implementation, each cache cluster further includes a data node 503, and the data node 503 logically divides the entire data set according to a Key of the cache data and is responsible for different data subsets; the proxy node 502 is further configured to send the cache data to be written to the data node; the data node 503 is configured to store the cache data to be written, which is in charge of the data node, according to the Key in the cache data to be written.
In one implementation, the proxy node 502 is specifically configured to: and selecting a copy synchronization path according to the cache data to be written, and sending the cache data to be written to the first proxy nodes in other cache clusters through the MQ cluster.
In an optional implementation, the proxy node 502 is specifically configured to: and responding to the fact that the Key fields in the cache data to be written are the same, and sending the cache data to be written with the same Key fields to the first proxy nodes in other cache clusters through the MQ cluster through the same replication synchronous path.
In an optional implementation, the proxy node 502 is specifically configured to: setting the same hash tag for the cache data to be written in response to the fact that the cache data to be written contains a plurality of different Key fields; and sending the cache data to be written with the same hash tag to the first proxy nodes in other cache clusters through the same replication synchronization path through the MQ cluster.
With regard to the systems in the above-described embodiments, the specific manner in which the components of the respective systems perform operations has been described in detail in relation to the embodiments of the method and will not be elaborated upon here.
According to the cache data processing system provided by the embodiment of the application, the cache data of a plurality of data centers can be efficiently and accurately copied in near real time, so that low-delay local cache data access is provided for service subsystems in different data centers, and the cache data processing system can be used for realizing remote multi-activity of the cache data.
Based on the embodiment of the present application, the present application further provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the cache data processing method of the multiple data centers according to any one of the embodiments.
Based on the embodiments of the present application, the present application also provides a computer readable storage medium storing computer instructions. The computer instructions are used for enabling a computer to execute the cache data processing method of the multi-data center according to any one of the previous embodiments provided by the embodiment of the application.
Referring to FIG. 6, shown in FIG. 6 is a schematic block diagram of an example electronic device that may be used to implement embodiments of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown in this application, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the applications described and/or claimed in this application.
As shown in fig. 6, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes in accordance with a computer program stored in a Read-Only Memory (ROM) 602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The calculation unit 601, the ROM602, and the RAM 603 are connected to each other via a bus 604. An Input/Output (I/O) interface 605 is also connected to bus 604.
A number of components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, a mouse, or the like; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing Unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and the like. The computing unit 601 performs the various methods and processes described above, such as a cache data processing method for multiple data centers. For example, in some embodiments, the multi-datacenter cache data processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM602 and/or the communication unit 609. When loaded into RAM 603 and executed by computing unit 601, may perform one or more of the steps of the multi-datacenter cache data processing method described above. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the cache data processing method of multiple data centers by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described above in this Application may be realized in digital electronic circuitry, Integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Parts (ASSPs), System On Chip (SOC), load Programmable Logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present application may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, causes the functions/acts specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM) or flash Memory), an optical fiber, a Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a Display device (e.g., a Cathode Ray Tube (CRT) or LCD (Liquid Crystal Display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), the internet, and blockchain networks.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The Server may be a cloud Server, which is also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS (Virtual Private Server) service. The server may also be a server of a distributed system, or a server incorporating a blockchain.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, and the present disclosure is not limited thereto as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A cache data processing method for multiple data centers is characterized in that each data center comprises cache clusters, a replication relationship is established among the cache clusters through message queue MQ clusters, each cache cluster comprises a proxy node, and the method is applied to the proxy nodes and comprises the following steps:
responding to the writing operation of a service system to the cache data in the cache cluster, determining cache data to be written, and writing the cache data to be written into the cache cluster;
and sending the cache data to be written to a first proxy node in other cache clusters through the MQ cluster, and enabling the first proxy node to replay the cache data to be written to the other cache clusters.
2. The method according to claim 1, wherein each of the cache clusters further comprises a data node, and the data node logically divides the whole data set according to a Key of the cache data and is responsible for different data subsets; the writing the cache data to be written into the cache cluster includes:
and sending the cache data to be written to a first data node in the cache cluster, and enabling the first data node to store the cache data to be written which is in charge of the first data node according to a Key in the cache data to be written.
3. The method as claimed in claim 1, wherein the sending the cached data to be written to the proxy node in the other cache cluster through the MQ cluster comprises:
and selecting a copy synchronization path according to the cache data to be written, and sending the cache data to be written to proxy nodes in other cache clusters through the MQ cluster.
4. The method as claimed in claim 3, wherein the selecting a replication synchronization path according to the to-be-written cache data, and sending the to-be-written cache data to the first proxy node in the other cache cluster through the MQ cluster, comprises:
and responding to that the Key fields in the cache data to be written are the same, and sending the cache data to be written with the same Key fields to the first proxy nodes in other cache clusters through the same replication synchronization path through the MQ cluster.
5. The method as claimed in claim 3, wherein the selecting a replication synchronization path according to the to-be-written cache data, and sending the to-be-written cache data to the first proxy node in the other cache cluster through the MQ cluster, comprises:
setting the same hash tag for the cache data to be written in response to the cache data to be written containing a plurality of different Key fields;
and sending the cache data to be written with the same hash tag to the first proxy nodes in other cache clusters through the same replication synchronization path through the MQ cluster.
6. A device for processing cache data of multiple data centers, wherein each data center includes cache clusters, a replication relationship is established between each cache cluster through a message queue MQ cluster, each cache cluster includes a proxy node, and the device is applied to the proxy node, and the device includes:
the write-in module is used for responding to the write-in operation of the business system on the cache data in the cache cluster, determining the cache data to be written in and writing the cache data to be written in the cache cluster;
and the sending module is used for sending the cache data to be written to the first proxy node in other cache clusters through the MQ cluster and enabling the first proxy node to replay the cache data to be written to the other cache clusters.
7. A cache data processing system is characterized by comprising a plurality of data centers, wherein each data center comprises cache clusters, replication relation is established among the cache clusters through message queue MQ clusters, and each cache cluster comprises a proxy node; wherein the content of the first and second substances,
the proxy node is used for responding to the write-in operation of the business system to the cache data in the cache cluster and determining the cache data to be written;
the proxy node is further configured to send the cache data to be written to a first proxy node in another cache cluster through the MQ cluster, and enable the first proxy node to replay the cache data to be written to the another cache cluster.
8. An electronic device, comprising:
at least one processor;
and a memory communicatively coupled to the at least one processor;
wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of cache data processing for multiple data centers of any of claims 1-5.
9. A computer-readable storage medium storing instructions that, when executed, cause the method of any of claims 1-5 to be implemented.
10. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the multi-data center cached data processing method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210288959.4A CN114745438B (en) | 2022-03-22 | 2022-03-22 | Cache data processing method, device and system of multi-data center and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210288959.4A CN114745438B (en) | 2022-03-22 | 2022-03-22 | Cache data processing method, device and system of multi-data center and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114745438A true CN114745438A (en) | 2022-07-12 |
| CN114745438B CN114745438B (en) | 2024-02-09 |
Family
ID=82276631
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210288959.4A Active CN114745438B (en) | 2022-03-22 | 2022-03-22 | Cache data processing method, device and system of multi-data center and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114745438B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110320403A1 (en) * | 2010-06-28 | 2011-12-29 | O'krafka Brian W | Approaches for the replication of write sets |
| US20120303737A1 (en) * | 2008-09-18 | 2012-11-29 | Avere Systems, Inc. | System and Method for Storing Data in Clusters Located Remotely From Each Other |
| CN103078927A (en) * | 2012-12-28 | 2013-05-01 | 合一网络技术(北京)有限公司 | Key-value data distributed caching system and method thereof |
| CN106357452A (en) * | 2016-09-29 | 2017-01-25 | 上海和付信息技术有限公司 | High-availability framework system with single-point heterogeneous data storage function and realizing method thereof |
| CN108183961A (en) * | 2018-01-04 | 2018-06-19 | 中电福富信息科技有限公司 | A kind of distributed caching method based on Redis |
| CN109492040A (en) * | 2018-11-06 | 2019-03-19 | 深圳航天智慧城市系统技术研究院有限公司 | A kind of system suitable for data center's magnanimity short message data processing |
| CN111475428A (en) * | 2020-04-03 | 2020-07-31 | 苏州浪潮智能科技有限公司 | Data reading and writing method, system, device and storage medium |
| CN111858190A (en) * | 2020-06-15 | 2020-10-30 | 福建天泉教育科技有限公司 | Method and system for improving cluster availability |
| CN112076464A (en) * | 2020-09-04 | 2020-12-15 | 腾讯科技（深圳）有限公司 | Data request processing method and device, computer equipment and storage medium |
| CN113127443A (en) * | 2020-01-14 | 2021-07-16 | 北京京东振世信息技术有限公司 | Method and device for updating cache data |
-
2022
- 2022-03-22 CN CN202210288959.4A patent/CN114745438B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120303737A1 (en) * | 2008-09-18 | 2012-11-29 | Avere Systems, Inc. | System and Method for Storing Data in Clusters Located Remotely From Each Other |
| US20110320403A1 (en) * | 2010-06-28 | 2011-12-29 | O'krafka Brian W | Approaches for the replication of write sets |
| CN103078927A (en) * | 2012-12-28 | 2013-05-01 | 合一网络技术(北京)有限公司 | Key-value data distributed caching system and method thereof |
| CN106357452A (en) * | 2016-09-29 | 2017-01-25 | 上海和付信息技术有限公司 | High-availability framework system with single-point heterogeneous data storage function and realizing method thereof |
| CN108183961A (en) * | 2018-01-04 | 2018-06-19 | 中电福富信息科技有限公司 | A kind of distributed caching method based on Redis |
| CN109492040A (en) * | 2018-11-06 | 2019-03-19 | 深圳航天智慧城市系统技术研究院有限公司 | A kind of system suitable for data center's magnanimity short message data processing |
| CN113127443A (en) * | 2020-01-14 | 2021-07-16 | 北京京东振世信息技术有限公司 | Method and device for updating cache data |
| CN111475428A (en) * | 2020-04-03 | 2020-07-31 | 苏州浪潮智能科技有限公司 | Data reading and writing method, system, device and storage medium |
| CN111858190A (en) * | 2020-06-15 | 2020-10-30 | 福建天泉教育科技有限公司 | Method and system for improving cluster availability |
| CN112076464A (en) * | 2020-09-04 | 2020-12-15 | 腾讯科技（深圳）有限公司 | Data request processing method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114745438B (en) | 2024-02-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10528527B2 (en) | File management in thin provisioning storage environments | |
| US20170339094A1 (en) | Message delivery in messaging networks | |
| US10129357B2 (en) | Managing data storage in distributed virtual environment | |
| WO2012081099A1 (en) | Data transfer program, computer, and data transfer method | |
| CN106855834B (en) | Data backup method, device and system | |
| CN109254873B (en) | Data backup method, related device and system | |
| US11079968B1 (en) | Queue management in multi-site storage systems | |
| US20150378854A1 (en) | Maintaining inactive copy relationships for secondary storages of active copy relationships having a common primary storage for use in case of a failure of the common primary storage | |
| CN110633046A (en) | Storage method and device of distributed system, storage equipment and storage medium | |
| CN109347899B (en) | Method for writing log data in distributed storage system | |
| US20200264796A1 (en) | Data copy amount reduction in data replication | |
| US9715477B2 (en) | Shared-bandwidth multiple target remote copy | |
| CN112052104A (en) | Message queue management method based on multi-computer-room realization and electronic equipment | |
| CN114745438B (en) | Cache data processing method, device and system of multi-data center and electronic equipment | |
| CN114490540B (en) | Data storage method, medium, device and computing equipment | |
| CN113326038B (en) | Method, apparatus, device, storage medium and program product for providing service | |
| US20200042631A1 (en) | Bandwidth management in a data storage system | |
| US11500701B1 (en) | Providing a global queue through replication | |
| US10979239B2 (en) | Updating data objects on a system | |
| US9223513B2 (en) | Accessing data in a dual volume data storage system using virtual identifiers | |
| CN112860796A (en) | Method, apparatus, device and storage medium for synchronizing data | |
| CN114442962B (en) | Data reading method, device, equipment and storage medium | |
| US10171388B2 (en) | Message retrieval in a distributed computing network | |
| US20160357479A1 (en) | Storage control apparatus | |
| US11080113B1 (en) | Fifo queue replication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |