CN111324664A - Method for realizing multi-computer-room cache synchronization - Google Patents
Method for realizing multi-computer-room cache synchronization Download PDFInfo
- Publication number
- CN111324664A CN111324664A CN202010075711.0A CN202010075711A CN111324664A CN 111324664 A CN111324664 A CN 111324664A CN 202010075711 A CN202010075711 A CN 202010075711A CN 111324664 A CN111324664 A CN 111324664A
- Authority
- CN
- China
- Prior art keywords
- cache
- key
- reading
- responsible
- machine room
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 23
- 230000009471 action Effects 0.000 claims abstract description 16
- 230000001360 synchronised effect Effects 0.000 description 4
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000011664 signaling Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for realizing multi-computer-room cache synchronization, which comprises the following steps: s1, deploying interceptors and message middleware in the machine room responsible for reading and writing, and deploying listeners in other machine rooms responsible for reading only, wherein the listeners subscribe instructions sent by the message middleware; the machine room responsible for reading and writing and the machine room responsible for reading only are positioned in different geographical areas and are connected through a network special line with limited bandwidth; s2, the interceptor intercepts the cache key and cache action of the machine room in charge of reading and writing, and sends corresponding instructions to the message middleware for broadcasting, and the other monitors of the machine room in charge of read-only service receive the instructions of the message middleware; and executing the instruction received by the listener, and deleting or updating the key in the corresponding cache. The invention can ensure that the cache data of multiple machine rooms can be quickly and consistently realized in real time under the condition of occupying extremely small bandwidth.
Description
Technical Field
The invention relates to a method for realizing multi-computer-room cache synchronization, belonging to the technical field of cache synchronization.
Background
With the continuous expansion of services, many companies deploy applications in a plurality of overseas data centers, wherein a domestic data center serves as a main data center to provide read-write services, other overseas centers only provide read-only services, and when the overseas data center performs read services, if corresponding data does not exist in a cache, the data read from a local database is stored in the cache. However, when the master data center updates data, how to ensure that other data centers read the latest data immediately is a challenge of data consistency of multiple data centers. Especially, under the condition that the bandwidth of the private line of the overseas data center and the domestic data center is limited, how to ensure the realization of the data consistency of multiple machine rooms under the condition of little occupied bandwidth becomes more complicated.
At present, the deployment mode of multiple data centers in the industry is generally a write-once read-many mode, that is, a main center provides read-write service, other data centers provide read-only service, and all writes are returned to the main data center. There are many technical schemes in the industry to ensure the final consistency of the cache data of multiple computer rooms, and the method mainly includes the following modes:
1) the method is characterized in that a secondary cache server is arranged on a cache server of a main data center, a plurality of cache servers of a read-only data center are secondary servers of the cache server of the main data center, and when data of the cache server of the main data center changes, the data are synchronized to cache servers of different data centers in near real time to achieve data consistency, however, the scheme has the defect that bandwidth is greatly occupied, for example: when the main center's cache data is in peak hours, there may be hundreds of megabytes of data change, and hundreds of megabytes of data change may result in the bandwidth occupation of many overseas data centers being very large.
2) When the write cache service occurs in the main data center, the cache is asynchronously and synchronously written into the cache servers of other data centers. However, the disadvantages of this solution are: the value content corresponding to the cache key (all synchronized to the caches of other data centers) occupies a very large bandwidth, and if the cached value is large and occupies a large bandwidth, the bandwidth is delayed when the peak period is full, and the data is updated in a non-real time manner; in addition, modification of the code is costly.
Disclosure of Invention
The invention aims to provide a method for realizing multi-computer-room cache synchronization, which can effectively solve the problems in the prior art, particularly the problems of large bandwidth occupation and non-real-time data updating during cache synchronization.
In order to solve the technical problems, the invention adopts the following technical scheme: a method for realizing multi-computer-room cache synchronization comprises the following steps:
s1, deploying interceptors and message middleware in the machine room responsible for reading and writing, and deploying listeners in other machine rooms responsible for reading only, wherein the listeners subscribe instructions sent by the message middleware; the machine room responsible for reading and writing and the machine room responsible for reading only are positioned in different geographical areas and are connected through a network special line with limited bandwidth;
s2, the interceptor intercepts the API method name and parameter name (the method name contains the name of the key) which are responsible for reading and writing the cache operation of the machine room, analyzes which action is performed on which key, then sends a corresponding instruction to the message middleware for broadcasting, and other listeners of the machine room which are responsible for the read-only service receive the instruction of the message middleware; executing the instruction received by the monitor, and deleting or updating the key in the corresponding cache; the data in the cache is stored in the memory in the form of Key and Value.
Preferably, step S1 further includes: performing instruction definition on the cache key and the cache action at the bottom layer of the interceptor to generate a configuration table; in step S2, after the interceptor intercepts the cache key and the cache action responsible for reading and writing the machine room, it queries the configuration table and then sends an instruction to the message middleware for broadcasting.
Preferably, the strategy of inquiring the type of key is fuzzified in the configuration table according to the information of the intercepted key, so that the matching efficiency can be improved. .
Preferably, in step S1, the local database of the machine room configuration responsible for reading and writing and the local databases of other machine room configurations responsible for reading and writing form a master-slave database; when data in the cache of the machine room in charge of reading and writing are updated or deleted, the data in the corresponding main database are correspondingly updated, and then other slave databases in charge of read-only machine room configuration are also updated in real time through a network private line.
Preferably, in step S2, other listeners in the machine room responsible for the read-only service receive the instruction of the message middleware, and if the instruction is a delete cache key, the corresponding cache key is directly deleted; and if the instruction is to update the cache key, reading corresponding data from the corresponding local database, and updating the key in the cache.
Compared with the prior art, the method has the advantages that the interceptor is used for intercepting the cache key and the cache action in charge of reading and writing the machine room and sending the corresponding instruction to the message middleware for broadcasting, and the other monitors in the machine room in charge of read-only service receive the instruction of the message middleware; executing the instruction received by the monitor, deleting or updating or resetting the Key in the corresponding cache, compared with the prior art (adopting the copy technology of the cache middleware to copy and transmit data of a multi-computer room (when the cache of the main computer room is updated, synchronously copying all cache data into the caches of other computer rooms), if the data volume is large, the occupation of the bandwidth is very large, the cache data can be synchronously jammed in the peak period of the traffic flow, and the timeliness of the cache is reduced), the invention only broadcasts the cache Key, and compared with the transmission of the complete data content, the data volume can be ignored in occupation of the bandwidth, and after broadcasting to each computer room in a signaling mode, the data can be independently pulled or the cache of the monitor can be set or deleted, so that the quick consistency of the cache data of the multiple computer rooms can be realized under the occupation of the extremely small bandwidth, the design based on the instructions (the instruction definition is carried out on the cache key and the cache action at the bottom layer of the interceptor) can conveniently realize the quasi-real-time transmission of the signaling of a plurality of rooms. In addition, the invention intercepts the cached key and broadcasts the key, after the listener of other machine rooms receives the instruction, the listener executes the instruction received by the listener, and performs corresponding operation on the local cache, the mode has better expansibility, and the code modification cost of the invention is lower (compared with the traditional modification mode, namely, the synchronous operation of the cache is directly written in the service code), only the definition of the instruction is needed to be performed at the bottom layer of the interceptor, the interceptor intercepts the API name and the parameter name (which can be the setting of deleting, updating or re-caching) which are responsible for reading and writing the cache operation of the machine rooms, and then sends the intercepted information to the message system for broadcasting (namely, the invention only performs unified processing at the interceptor layer).
The inventor applies the technical scheme of the invention to company business, and since the company business is distributed in a plurality of overseas countries, and the external service is deployed in a plurality of overseas machine rooms, the requirement on the real-time performance of data is higher, the special line from the overseas machine room to the domestic network is unstable, the bandwidth of the special line from the overseas machine room to the domestic machine room is smaller, and if the cache content is synchronized in real time, the bandwidth is brought with overlarge pressure, so after the overseas multi-machine room is deployed, the cache synchronization effect is very obvious, and the real-time performance is very good.
Drawings
FIG. 1 is a schematic flow chart of the method of the experimental example of the present invention.
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.
Detailed Description
The embodiment of the invention comprises the following steps: a method for realizing multi-computer-room cache synchronization comprises the following steps:
s1, deploying interceptors and message middleware in the machine room responsible for reading and writing, and deploying listeners in other machine rooms responsible for reading only, wherein the listeners subscribe instructions sent by the message middleware; the machine room responsible for reading and writing and the machine room responsible for reading only are positioned in different geographical areas and are connected through a network special line with limited bandwidth;
s2, the interceptor intercepts the API method name and parameter name (the method name contains the name of the key) which are responsible for reading and writing the cache operation of the machine room, analyzes which action is performed on which key, then sends a corresponding instruction to the message middleware for broadcasting, and other listeners of the machine room which are responsible for the read-only service receive the instruction of the message middleware; executing the instruction received by the monitor, and deleting or updating the key in the corresponding cache; the data in the cache is stored in the memory in the form of Key and Value.
Optionally, step S1 further includes: performing instruction definition on the cache key and the cache action at the bottom layer of the interceptor to generate a configuration table; in step S2, after the interceptor intercepts the cache key and the cache action responsible for reading and writing the machine room, it queries the configuration table and then sends an instruction to the message middleware for broadcasting.
Optionally, a policy of the type of key is fuzzily queried in the configuration table according to the intercepted key information, for example, the username001, and the username002 only needs to configure the rule corresponding to the username in the configuration table, that is, the username executes an update or delete operation in another machine room.
Optionally, in step S1, the local database of the machine room configuration responsible for reading and writing and the local databases of other machine room configurations responsible for reading and writing form a master-slave database; when data in the cache of the machine room in charge of reading and writing are updated or deleted, the data in the corresponding main database are correspondingly updated, and then other slave databases in charge of read-only machine room configuration are also updated in real time through a network private line.
Optionally, in step S2, other listeners in the machine room responsible for the read-only service receive the instruction of the message middleware, and if the instruction is a delete cache key, the corresponding cache key is directly deleted; and if the instruction is to update the cache key, reading corresponding data from the corresponding local database, and updating the key in the cache.
Experimental example: the technical scheme of the invention is suitable for a data center with one master and multiple slaves, wherein only the master data center is responsible for reading and writing, and other data centers are responsible for reading only, and for simplicity of expression, two machine rooms (one master and one slave) are exemplified, as shown in fig. 1.
In the upper diagram, the IDCA and the IDCB refer to two IDC machine rooms located in A, B two different geographical areas, the two machine rooms are connected by a network special line with limited bandwidth, and the phenomenon of packet loss occurs in a peak period; the IDCA, the application A and the IDCB, the application A are application software respectively deployed in machine rooms of two IDCs, wherein the IDCA, the application A is responsible for reading and writing operations, the IDCB, the application A is responsible for reading operations, services between the IDCA and the application A cannot be mutually called, and both the IDCA and the application A provide reading services; message represents message middleware of a geographical area, to which other IDCs can be connected through a listener; the database master library in the IDCA is used for reading operation and writing operation; b, the database of the geographic region is selected from the database and is only responsible for read-only operation; the idca, cachea and the idcb, cacheb belong to cache machines of A, B two IDC rooms respectively, and cannot access each other, and the idca, cachea is responsible for storing data in the application a in the geographic area of the application a (the data in the cache is stored in the memory in the form of Key and Value); because the applications of the two IDCs can both provide services to the outside and the caches of the two machine rooms are completely isolated, in order to ensure that the data read by the two machine rooms are consistent data, the cache data of the two machine rooms need to be processed. The detailed process flow is as follows:
(1) if the user A performs the updating operation of the data R1 on the application A in the IDCA;
(2) idca. application a first deletes (while updating the corresponding data in a machine master database) the r1.c1(r1.c1 is the data of the previous version of r1.c 2) cache data;
(3) the interceptor intercepts the cache operation in the IDCA application A;
(4) inquiring the cache operation intercepted by the configuration table, inquiring the instruction to be executed and sending the instruction to the message middleware;
(5) the message middleware executes the instructions queried from the configuration table: cmd ═ delete; key ═ r1.c 1; from ═ IDCA; action ═ publish, the instruction meaning is: deleting the cache with the key name of 'R1. C1' in the machine room of the IDCA, requesting other machine rooms to delete the data of the corresponding key, and locally updating the cache; this action performs a broadcast operation;
(6) listener subscribed to instructions sent from message middleware:
cmd＝delete；key＝R1.C1；from＝IDCA；action＝publish；
(7) analyzing the instruction: cmd ═ delete; key ═ r1.c 1; from ═ IDCA; action ═ publication, i.e.: the IDCB monitors the cache with the local room cache Key of R1.C1 to delete;
(8) applying A to read the data of R1, if the data does not exist in the cache, reading the data from the database, and updating the latest data in the database to the cache IDCA.R1. C2;
(9) the application A reads the data of the R1, if the data does not exist in the cache, the latest data in the database is updated to the cache IDCB.R1.C2 (wherein, the synchronization of the database data between the IDCA.MasterA.DB and the IDCB.Slave.DB is carried out based on a network private line);
(10) the cache data of the two computer rooms of IDCA and IDCB realize final consistency.
Claims (5)
1. A method for realizing multi-computer-room cache synchronization is characterized by comprising the following steps:
s1, deploying interceptors and message middleware in the machine room responsible for reading and writing, and deploying listeners in other machine rooms responsible for reading only, wherein the listeners subscribe instructions sent by the message middleware; the machine room responsible for reading and writing and the machine room responsible for reading only are positioned in different geographical areas and are connected through a network special line with limited bandwidth;
s2, the interceptor intercepts the API method name and the parameter name of the cache operation of the machine room, analyzes which key has performed what action, then sends corresponding instructions to the message middleware for broadcasting, and the other monitors of the machine room in charge of the read-only service receive the instructions of the message middleware; executing the instruction received by the monitor, and deleting or updating the key in the corresponding cache; the data in the cache is stored in the memory in the form of Key and Value, and the name of the method contains the name of the Key.
2. The method for implementing multi-computer room cache synchronization as claimed in claim 1, wherein the step S1 further comprises: performing instruction definition on the cache key and the cache action at the bottom layer of the interceptor to generate a configuration table; in step S2, after the interceptor intercepts the cache key and the cache action responsible for reading and writing the machine room, it queries the configuration table and then sends an instruction to the message middleware for broadcasting.
3. The method for implementing multi-computer room cache synchronization as claimed in claim 2, wherein the policy of inquiring the type of key is fuzzified in the configuration table according to the information of the intercepted key.
4. The method for implementing multi-computer room cache synchronization as claimed in claim 1, wherein in step S1, the local database of the computer room configuration responsible for reading and writing and the local databases of other computer rooms responsible for read-only form a master-slave database; when data in the cache of the machine room in charge of reading and writing are updated or deleted, the data in the corresponding main database are correspondingly updated, and then other slave databases in charge of read-only machine room configuration are also updated in real time through a network private line.
5. The method according to claim 4, wherein in step S2, the listeners in the computer room responsible for the read-only service receive the instruction from the message middleware, and if the instruction is a delete cache key, the corresponding cache key is directly deleted; and if the instruction is to update the cache key, reading corresponding data from the corresponding local database, and updating the key in the cache.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010075711.0A CN111324664A (en) | 2020-01-22 | 2020-01-22 | Method for realizing multi-computer-room cache synchronization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010075711.0A CN111324664A (en) | 2020-01-22 | 2020-01-22 | Method for realizing multi-computer-room cache synchronization |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111324664A true CN111324664A (en) | 2020-06-23 |
Family
ID=71165210
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010075711.0A Pending CN111324664A (en) | 2020-01-22 | 2020-01-22 | Method for realizing multi-computer-room cache synchronization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111324664A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106294033A (en) * | 2015-05-13 | 2017-01-04 | 阿里巴巴集团控股有限公司 | The method of testing of a kind of many machine rooms cache synchronization function and device |
| CN110309231A (en) * | 2019-07-12 | 2019-10-08 | 焦点科技股份有限公司 | A kind of method of data synchronization and system across computer room |
| CN110597910A (en) * | 2019-09-12 | 2019-12-20 | 聚好看科技股份有限公司 | Remote data synchronization method, device and system |
-
2020
- 2020-01-22 CN CN202010075711.0A patent/CN111324664A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106294033A (en) * | 2015-05-13 | 2017-01-04 | 阿里巴巴集团控股有限公司 | The method of testing of a kind of many machine rooms cache synchronization function and device |
| CN110309231A (en) * | 2019-07-12 | 2019-10-08 | 焦点科技股份有限公司 | A kind of method of data synchronization and system across computer room |
| CN110597910A (en) * | 2019-09-12 | 2019-12-20 | 聚好看科技股份有限公司 | Remote data synchronization method, device and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7499905B2 (en) | Method and apparatus for accessing information based on distributed file system (DFS) paths | |
| CN108833503B (en) | ZooKeeper-based Redis clustering method | |
| US10831612B2 (en) | Primary node-standby node data transmission method, control node, and database system | |
| CN111432025B (en) | Cloud edge cooperation-oriented distributed service directory management method and system | |
| CN111078426A (en) | High concurrency solution under back-end micro-service architecture | |
| CN102609479B (en) | A kind of memory database node clone method | |
| WO2018040168A1 (en) | Distributed-cache synchronization method, apparatus, and system | |
| CN101706781A (en) | Method and system for centralized management of database caches | |
| CN110727738B (en) | Global routing system based on data fragmentation, electronic equipment and storage medium | |
| CN112866119B (en) | Virtual extensible local area network communication method and device, electronic equipment and storage medium | |
| CN106874465A (en) | Method for efficiently managing cache based on data version | |
| CN115086330A (en) | Cross-cluster load balancing system | |
| CN113766027A (en) | Method and equipment for forwarding data by flow replication cluster node | |
| CN113259476B (en) | Message pushing method and system | |
| CN112261090B (en) | Web data processing method and device, computer equipment and readable storage medium | |
| CN110460482B (en) | Traffic acquisition method, device, server and medium | |
| CN111324664A (en) | Method for realizing multi-computer-room cache synchronization | |
| CN111931105A (en) | Kafka consumption appointed push time data processing method | |
| CN112749172A (en) | Data synchronization method and system between cache and database | |
| CN111600958B (en) | Service discovery system, service data management method, server, and storage medium | |
| CN114610509A (en) | Calling parameter processing method, system, device, storage medium and product | |
| CN111083182B (en) | Distributed Internet of things equipment management method and device | |
| CN113051091A (en) | Process-level cache data synchronization method and device | |
| CN114900449A (en) | Resource information management method, system and device | |
| CN113630317A (en) | Data transmission method and device, nonvolatile storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200623 |