CN114328618B - Cache data synchronization method, device, equipment and storage medium - Google Patents
Cache data synchronization method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114328618B CN114328618B CN202111445506.XA CN202111445506A CN114328618B CN 114328618 B CN114328618 B CN 114328618B CN 202111445506 A CN202111445506 A CN 202111445506A CN 114328618 B CN114328618 B CN 114328618B
- Authority
- CN
- China
- Prior art keywords
- data
- cache data
- server
- file
- target cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000004590 computer program Methods 0.000 claims description 15
- 238000006243 chemical reaction Methods 0.000 claims description 10
- 238000012545 processing Methods 0.000 claims description 10
- 238000010586 diagram Methods 0.000 description 10
- 230000007246 mechanism Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000002688 persistence Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Abstract
The application discloses a cache data synchronization method, a device, equipment and a storage medium, which are applied to a distributed message middleware system, wherein the method comprises the following steps: acquiring target cache data in a main server of a remote dictionary service cluster; the target cache data are cache data obtained by storing corresponding changed service data to the main server after the service data are changed by the application server; broadcasting target cache data to all clients of the distributed message middleware system through a server of the distributed message middleware system; and each client of the distributed message middleware system is controlled to respectively transmit the acquired target cache data to a corresponding slave server, so that each slave server can store the acquired target cache data to a local cache space. By the technical scheme, the pressure of the application server and the main server can be reduced, the real-time synchronization of the cache data is realized, and the consistency of the data is ensured.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a medium for synchronizing cache data.
Background
In order to increase the access speed of the system in the application system, a caching mechanism is often added at the application level, that is, the queried data is put in the memory, and Redis (Remote Dictionary Server, remote dictionary service) is an important memory database for managing the cached data. The management of data caching in the multi-service cluster mode ensures both the consistency of the data and the real-time performance of the data, and is a basic requirement for cache synchronization in time. The existing synchronization method of the cache data in the remote dictionary service can be divided into the following modes:
(1) As shown in fig. 1, an application server is connected to a plurality of dis servers, and the plurality of dis servers respectively correspond to a plurality of clients, so as to realize synchronization of cached data, which has the following disadvantages: the application server needs to connect multiple Redis services simultaneously, i.e. the pressure of the application server increases.
(2) As shown in fig. 2, a dis backup mechanism is adopted to realize one-to-many data synchronization, and for one master dis server, other slave dis servers realize data synchronization by synchronizing the master server cache data, which has the following disadvantages: the master server also needs to process the request of data synchronization of the slave Redis server when updating the cache, i.e. the pressure of the master Redis server increases.
(3) As shown in fig. 3, a plurality of master dis servers are added, and the pressure of synchronizing to the slave dis servers is reduced at this time, but the consistency of the cache data of each master dis server cannot be ensured.
Therefore, when the cache data is synchronized, the pressure of the application server and the main server is reduced, and the cache data consistency is realized
Disclosure of Invention
In view of the above, the present invention aims to provide a method, an apparatus, a device, and a medium for synchronizing cache data, which can solve the problem that when one master server connects a plurality of slave servers, the master server has high pressure, and ensure the consistency of the cache data, and the specific scheme is as follows:
in a first aspect, the present application discloses a method for synchronizing cached data, which is applied to a distributed message middleware system, and includes:
acquiring target cache data in a main server of a remote dictionary service cluster; the target cache data are cache data obtained by storing corresponding changed service data to the main server after the service data are changed by the application server;
broadcasting the target cache data to all clients of the distributed message middleware system through a server of the distributed message middleware system; different clients respectively correspond to different slave servers in the remote dictionary service cluster;
and controlling each client of the distributed message middleware system to respectively transmit the respectively acquired target cache data to the corresponding slave server so that each slave server can store the respectively acquired target cache data into the respective local cache space.
Optionally, the obtaining the target cache data in the main server of the remote dictionary service cluster includes:
and reading the cache data file in the main server of the remote dictionary service cluster, and performing file conversion processing on the read cache data file through a data file converter to obtain the target cache data.
Optionally, the reading the cached data file in the main server of the remote dictionary service cluster includes:
and reading the RDB file in the main server of the remote dictionary service cluster, and reading the AOF file in the main server after reading the RDB file.
Optionally, the reading the RDB file in the main server of the remote dictionary service cluster, and reading the AOF file in the main server after reading the RDB file, includes:
and based on a preset time period, periodically reading the RDB file in the main server of the remote dictionary service cluster, and reading the AOF file in the main server after reading the RDB file.
Optionally, the performing, by the data file converter, file conversion processing on the read cache data file to obtain the target cache data includes:
and respectively analyzing the read RDB file and the read AOF file through the data file converter to obtain corresponding first analyzed data and second analyzed data, and repackaging the first analyzed data and the second analyzed data in a key value pair as a data recording mode to obtain the target cache data.
Optionally, the cache data synchronization method further includes:
after the AOF file in the main server is read, the reading position of the corresponding AOF file is recorded.
Optionally, before broadcasting the target cache data to each client of the distributed message middleware system through the server of the distributed message middleware system, the method further includes:
setting a message theme for the target cache data through a server side of the distributed message middleware system, configuring a message queue for the message theme, and storing the target cache data in the message queue;
correspondingly, the broadcasting, by the server side of the distributed message middleware system, the target cache data to each client side of the distributed message middleware system includes:
and acquiring a subscription request of each client side for the message theme through the server side of the distributed message middleware system, and broadcasting the target cache data stored in the message queue to each client side of the distributed message middleware system.
In a second aspect, the present application discloses a cache data synchronization device, applied to a distributed message middleware system, including:
the data acquisition module is used for acquiring target cache data in a main server of the remote dictionary service cluster; the target cache data are cache data obtained by storing corresponding changed service data to the main server after the service data are changed by the application server;
the data broadcasting module is used for broadcasting the target cache data to all clients of the distributed message middleware system through the server side of the distributed message middleware system; different clients respectively correspond to different slave servers in the remote dictionary service cluster;
and the data sending module is used for controlling each client of the distributed message middleware system to respectively transmit the acquired target cache data to the corresponding slave server so that each slave server can store the acquired target cache data to the local cache space.
In a third aspect, the present application discloses an electronic device comprising:
a memory for storing a computer program;
and the processor is used for executing the computer program to realize the cache data synchronization method disclosed by the prior art.
In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the previously disclosed cache data synchronization method.
As can be seen, the present application proposes a method for synchronizing cache data, including: acquiring target cache data in a main server of a remote dictionary service cluster; the target cache data are cache data obtained by storing corresponding changed service data to the main server after the service data are changed by the application server; broadcasting the target cache data to all clients of the distributed message middleware system through a server of the distributed message middleware system; different clients respectively correspond to different slave servers in the remote dictionary service cluster; and controlling each client of the distributed message middleware system to respectively transmit the acquired target cache data to the corresponding slave server, so that each slave server stores the acquired target cache data to the local cache space, and therefore, the application server is only connected with one master server, and the pressure of the application server is relieved. In addition, the distributed message middleware system is added between the master server and the slave server, and the data synchronization request of the slave server is processed by the distributed message middleware system, so that the pressure of the master server is relieved. Finally, the system has a broadcast sending mechanism, and only one master server realizes the real-time synchronization of the cache data, thereby ensuring the consistency of the data on the slave servers.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a diagram illustrating a first conventional cache data synchronization method;
FIG. 2 is a schematic diagram of a second prior art method for synchronizing buffered data;
FIG. 3 is a schematic diagram of a third conventional cache data synchronization method;
FIG. 4 is a flowchart of a method for synchronizing cache data disclosed in the present application;
FIG. 5 is a flowchart of a method for synchronizing buffered data according to the present disclosure;
FIG. 6 is a schematic structural diagram of a specific method for synchronizing cache data disclosed in the present application;
fig. 7 is a schematic structural diagram of a buffer data synchronization device disclosed in the present application;
fig. 8 is a block diagram of an electronic device disclosed in the present application.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
When the cache data is synchronized, the prior art faces the problems that the pressure of an application server and a main server is high and the consistency of the cache data cannot be guaranteed. Therefore, the embodiment of the application provides a cache data synchronization scheme, which can relieve the pressure of an application server and a main server and realize cache data consistency.
The embodiment of the application discloses a cache data synchronization method, which is shown in fig. 4 and comprises the following steps:
step S11: acquiring target cache data in a main server of a remote dictionary service cluster; the target cache data are cache data obtained by storing corresponding changed service data to the main server after the service data are changed by the application server.
The remote dictionary service (Redis) in this implementation is a fully open-source, BSD (Berkeley Software Distribution, berkeley software release) compliant and high-performance Key-Value pair (Key-Value) database for storing cached data. It should be noted that, the application layer updates data to obtain cached data, and transmits the cached data to the main server of the remote dictionary service cluster, where the remote dictionary server can persist the cache in the memory to the disk for persistence. When the distributed message middleware system synchronizes the cache data, the target cache data in the main server of the remote dictionary service cluster needs to be acquired first, and it can be understood that the acquired target cache data is changed according to the actual scene, so that the target cache data is the cache data obtained by storing the corresponding changed service data of the application server to the main server after the service data is changed.
Specifically, in the process of performing persistence preservation on the remote dictionary server, a configuration file is first required to be configured in a main server of the remote dictionary service cluster, and a persistence function of the remote dictionary service is started by utilizing redis.
RDB persistence configuration:
save 900 1
save 300 10
save 60 10000；
AOF persistence configuration:
appendfsync always # is written to the AOF file each time a data modification occurs.
It should be noted that the remote dictionary service data converter is incorporated in the distributed message middleware system because there are two ways to store the data files in the remote dictionary server: one is a RDB (Redis Database) file, which exports cached data to the RDB file at regular time; the other is AOF (Append Only File) file, which is written with the operation log of the remote dictionary service in an additional mode; in the process of acquiring target cache data in a main server of a remote dictionary service cluster, a remote dictionary service data converter in a distributed message middleware system is utilized to perform file conversion processing on a cache data file in the main server of the read remote dictionary service cluster, the data converter performs data analysis on an RDB file or an AOF file to obtain corresponding first analyzed data and second analyzed data, and then a key-value pair (key-value) is used as a data record mode to repackage the first analyzed data and the second analyzed data so as to obtain the target cache data.
In the embodiment of the application, the remote dictionary service data converter in the distributed message middleware system is configured regularly, so that the data converter can periodically read the latest RDB file in the main server of the remote dictionary service cluster based on a preset time period, and read the AOF file in the main server after reading the RDB file. It can be understood that when the AOF file persistence configuration is performed, once the content of the buffer area changes, the buffer area is written into the AOF file, when the RDB file is read, full synchronization of the remote dictionary service is realized, when the application level buffer is updated, the AOF file in the main server is read again, incremental synchronization of the remote dictionary service is realized, and the speed of transmitting data and the efficiency of data synchronization can be further improved. It should be noted that after reading the AOF file in the main server, the reading position of the corresponding AOF file is recorded at any time.
Step S12: broadcasting the target cache data to all clients of the distributed message middleware system through a server of the distributed message middleware system; different clients respectively correspond to different slave servers in the remote dictionary service cluster.
In this embodiment, distributed message middleware is added between the Redis master server and the slave servers for data broadcasting. Specifically, after the target cache data is obtained, the target cache data is sent to a server side of a distributed message middleware system, then the server side of the distributed message middleware system broadcasts the target cache data to all clients of the distributed message middleware system, and only all clients of the distributed message middleware system registered in the server side of the distributed message middleware system can receive the target cache data. In the application, the consistency of receiving data from the server is realized by a broadcast transmission mode. It should be noted that in order to send the target cache data to the server of the distributed message middleware system, a client is deployed in the data converter.
In this embodiment, a plurality of slave server nodes are established for the master server, and different clients are set to respectively correspond to different slave servers in the remote dictionary service cluster, that is, each slave server node corresponds to a client of the distributed message middleware system.
Step S13: and controlling each client of the distributed message middleware system to respectively transmit the respectively acquired target cache data to the corresponding slave server so that each slave server can store the respectively acquired target cache data into the respective local cache space.
In this embodiment, since different clients respectively correspond to different slave servers in the remote dictionary service cluster, the distributed message middleware system controls each client of the distributed message middleware system to respectively transmit the target cache data acquired respectively to the corresponding slave servers, so that each slave server stores the target cache data acquired respectively to the local cache space.
As can be seen, the present application proposes a method for synchronizing cache data, including: acquiring target cache data in a main server of a remote dictionary service cluster; the target cache data are cache data obtained by storing corresponding changed service data to the main server after the service data are changed by the application server; broadcasting the target cache data to all clients of the distributed message middleware system through a server of the distributed message middleware system; different clients respectively correspond to different slave servers in the remote dictionary service cluster; and controlling each client of the distributed message middleware system to respectively transmit the acquired target cache data to the corresponding slave server, so that each slave server stores the acquired target cache data to the local cache space, and therefore, the application server is only connected with one master server, and the pressure of the application server is relieved. In addition, the distributed message middleware system is added between the master server and the slave server, and the data synchronization request of the slave server is processed by the distributed message middleware system, so that the pressure of the master server is relieved. Finally, the system has a broadcast sending mechanism, and only one master server realizes the real-time synchronization of the cache data, thereby ensuring the consistency of the data on the slave servers.
The embodiment of the application discloses a specific cache data synchronization method, which is shown in fig. 5, and includes:
step S21: acquiring target cache data in a main server of a remote dictionary service cluster; the target cache data are cache data obtained by storing corresponding changed service data to the main server after the service data are changed by the application server.
For more specific processing in step S21, reference may be made to the corresponding content disclosed in the foregoing embodiment, and no further description is given here.
Step S22: setting a message theme for the target cache data through the server side of the distributed message middleware system, configuring a message queue for the message theme, and storing the target cache data in the message queue.
In the embodiment of the present application, after target cache data in a main server of a remote dictionary service cluster is obtained, a server side of a distributed message middleware system needs to be deployed in a cluster, and then data is broadcasted. Specifically, a message theme is set for target cache data at a server side of the distributed message middleware system, a message queue is configured for the message theme, then the target cache data is stored in the message queue in a message form, so that each corresponding client side in the distributed message middleware cluster can acquire the target cache data, and the target cache data is broadcast to a slave server of a remote dictionary service cluster registered in the corresponding distributed message middleware client side.
It can be understood that, because the message sequence in each queue is certain, the sequence of sending data is sequential, a message theme is set, and the theme is set to only one queue, so that the sequence consistency of the backup data can be ensured based on the sequence of sending data during the restoration.
Step S23: and acquiring a subscription request of each client side for the message theme through the server side of the distributed message middleware system, and broadcasting the target cache data stored in the message queue to each client side of the distributed message middleware system.
In this embodiment, when each client of the distributed message middleware needs the target cache data, each client of the distributed message middleware sends a subscription request of the message theme corresponding to the target cache data to a server of the distributed message middleware system, and after the server of the distributed message middleware system obtains the subscription request of each client for the message theme, the server broadcasts the target cache data stored in the message queue to each client of the distributed message middleware system.
Step S24: and controlling each client of the distributed message middleware system to respectively transmit the respectively acquired target cache data to the corresponding slave server so that each slave server can store the respectively acquired target cache data into the respective local cache space.
In this embodiment, each client of the distributed message middleware system transmits the target cache data obtained by each client to the corresponding slave server, and it should be noted that the target cache data is stored in a message queue in a message form, so that the target cache data obtained by the slave server is still in a message form, and further, the target cache data in a message form needs to be parsed, so that each slave server stores the target cache data obtained by each slave server in a local cache space, thereby implementing synchronization of the cache data.
It can be seen that the embodiment of the present application provides a method for synchronizing cached data, firstly, obtaining target cached data in a main server of a remote dictionary service cluster; the target cache data are cache data obtained by storing corresponding changed service data to the main server after the service data are changed by the application server; setting a message theme for the target cache data through a server side of the distributed message middleware system, configuring a message queue for the message theme, and storing the target cache data in the message queue; acquiring a subscription request of each client side for the message theme through a server side of the distributed message middleware system, and broadcasting the target cache data stored in the message queue to each client side of the distributed message middleware system; and finally, controlling each client of the distributed message middleware system to respectively transmit the acquired target cache data to the corresponding slave server so that each slave server can store the acquired target cache data to the local cache space. Therefore, the target cache data is broadcasted to each client of the distributed message middleware system through a broadcast notification mechanism of the server of the distributed message middleware system, a message theme is set for the server, and the theme is set to be only one queue, so that the sequence consistency of the backup data is ensured. In addition, the application server in the application is only connected with one main server, so that the pressure of the application server is relieved. In addition, the distributed message middleware system is added between the master server and the slave server, and the data synchronization request of the slave server is processed by the distributed message middleware system, so that the pressure of the master server is relieved. Finally, the system has a broadcast sending mechanism, and only one master server realizes the real-time synchronization of the cache data, thereby ensuring the consistency of the data on the slave servers.
Fig. 6 shows a schematic structural diagram of a specific procedure of the cache data synchronization method.
It should be noted that, in this embodiment, a dockmq client is disposed in the data file converter, so that the target cache data obtained after conversion by the data file converter is sent to the dockmq server of the distributed message middleware system. Referring to fig. 6, in this embodiment, after service data is changed, an application server stores corresponding changed service data to the main Redis server, in the main Redis server, cache data is configured to obtain an RDB file and an AOF file, then a data file converter performs file conversion processing on the read RDB file and AOF file to obtain target cache data, and a preconfigured RocketMQ client in the data file converter sends the target cache data to a RocketMQ server of the distributed message middleware system. It should be noted that the target cache data is stored in the form of a message in a message queue in a dockmq server of the distributed message middleware system, and the target cache data corresponds to a message topic, so that based on the message topic subscribed by each dockmq client, the dockmq server of the distributed message middleware system broadcasts the target cache data stored in the message queue to each dockmq client of the distributed message middleware system, and finally the distributed message middleware system controls each dockmq client of the distributed message middleware system to transmit the respectively acquired target cache data to the corresponding slave Redis server so that each slave Redis server stores the respectively acquired target cache data to the respective local cache space.
Correspondingly, the embodiment of the application also discloses a buffer data synchronization device, as shown in fig. 7, which includes:
the data acquisition module 11: acquiring target cache data in a main server of a remote dictionary service cluster; the target cache data are cache data obtained by storing corresponding changed service data to the main server after the service data are changed by the application server;
the data broadcasting module 12: broadcasting the target cache data to all clients of the distributed message middleware system through a server of the distributed message middleware system; different clients respectively correspond to different slave servers in the remote dictionary service cluster;
the data transmission module 13: and controlling each client of the distributed message middleware system to respectively transmit the respectively acquired target cache data to the corresponding slave server so that each slave server can store the respectively acquired target cache data into the respective local cache space.
The more specific working process of each module may refer to the corresponding content disclosed in the foregoing embodiment, and will not be described herein.
As can be seen, the present application proposes a method for synchronizing cache data, including: acquiring target cache data in a main server of a remote dictionary service cluster; the target cache data are cache data obtained by storing corresponding changed service data to the main server after the service data are changed by the application server; broadcasting the target cache data to all clients of the distributed message middleware system through a server of the distributed message middleware system; different clients respectively correspond to different slave servers in the remote dictionary service cluster; and controlling each client of the distributed message middleware system to respectively transmit the acquired target cache data to the corresponding slave server, so that each slave server stores the acquired target cache data to the local cache space, and therefore, the application server is only connected with one master server, and the pressure of the application server is relieved. In addition, the distributed message middleware system is added between the master server and the slave server, and the data synchronization request of the slave server is processed by the distributed message middleware system, so that the pressure of the master server is relieved. Finally, the system has a broadcast sending mechanism, and only one master server realizes the real-time synchronization of the cache data, thereby ensuring the consistency of the data on the slave servers.
Further, the embodiment of the application also provides electronic equipment. Fig. 8 is a block diagram of an electronic device 20, according to an exemplary embodiment, and the contents of the diagram should not be construed as limiting the scope of use of the present application in any way.
Fig. 8 is a schematic structural diagram of an electronic device 20 according to an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a display screen 23, an input output interface 24, a communication interface 25, a power supply 26, and a communication bus 27. The memory 22 is configured to store a computer program, which is loaded and executed by the processor 21 to implement relevant steps in the cache data synchronization method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the present embodiment may be specifically an electronic computer.
In this embodiment, the power supply 26 is used to provide an operating voltage for each hardware device on the electronic device 20; the communication interface 25 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein; the input/output interface 24 is used for obtaining external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application needs, which is not limited herein.
The memory 22 may be a read-only memory, a random access memory, a magnetic disk, an optical disk, or the like, and the resources stored thereon may include the computer program 221, which may be stored in a temporary or permanent manner. The computer program 221 may further include a computer program for performing other specific tasks in addition to the computer program for performing the cache data synchronization method performed by the electronic device 20 as disclosed in any of the foregoing embodiments.
Further, the embodiment of the application also discloses a computer readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the previously disclosed cache data synchronization method.
For specific steps of the method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and no further description is given here.
In this application, various embodiments are described in a progressive manner, and each embodiment is mainly described in terms of differences from other embodiments, where the same or similar parts of the various embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing has described in detail a method, apparatus, device, and storage medium for synchronizing cache data, to which specific examples are applied to illustrate the principles and embodiments of the present application, where the foregoing examples are provided to assist in understanding the method and core ideas of the present application; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (7)
1. A method for synchronizing cached data, applied to a distributed message middleware system, comprising:
acquiring target cache data in a main server of a remote dictionary service cluster; the target cache data are cache data obtained by storing corresponding changed service data to the main server after the service data are changed by the application server;
broadcasting the target cache data to all clients of the distributed message middleware system through a server of the distributed message middleware system; different clients respectively correspond to different slave servers in the remote dictionary service cluster;
controlling each client of the distributed message middleware system to respectively transmit the respectively acquired target cache data to the corresponding slave server so that each slave server can store the respectively acquired target cache data into the respective local cache space;
the obtaining the target cache data in the main server of the remote dictionary service cluster comprises the following steps:
reading a cache data file in a main server of a remote dictionary service cluster, and performing file conversion processing on the read cache data file through a data file converter to obtain target cache data;
the reading the cached data file in the main server of the remote dictionary service cluster comprises the following steps:
reading an RDB file in a main server of a remote dictionary service cluster, and reading an AOF file in the main server after reading the RDB file;
the file conversion processing is performed on the read cache data file by a data file converter to obtain the target cache data, including:
and respectively analyzing the read RDB file and the read AOF file through the data file converter to obtain corresponding first analyzed data and second analyzed data, and repackaging the first analyzed data and the second analyzed data in a key value pair as a data recording mode to obtain the target cache data.
2. The method for synchronizing cached data according to claim 1, wherein the reading the RDB file in the primary server of the remote dictionary service cluster and reading the AOF file in the primary server after reading the RDB file comprises:
and based on a preset time period, periodically reading the RDB file in the main server of the remote dictionary service cluster, and reading the AOF file in the main server after reading the RDB file.
3. The method for synchronizing buffered data according to claim 2, further comprising:
after the AOF file in the main server is read, the reading position of the corresponding AOF file is recorded.
4. A method of synchronizing cached data according to any one of claims 1 to 3, wherein before broadcasting the target cached data to the clients of the distributed message middleware system by the server of the distributed message middleware system, the method further comprises:
setting a message theme for the target cache data through a server side of the distributed message middleware system, configuring a message queue for the message theme, and storing the target cache data in the message queue;
correspondingly, the broadcasting, by the server side of the distributed message middleware system, the target cache data to each client side of the distributed message middleware system includes:
and acquiring a subscription request of each client side for the message theme through the server side of the distributed message middleware system, and broadcasting the target cache data stored in the message queue to each client side of the distributed message middleware system.
5. A cache data synchronization apparatus for use in a distributed message middleware system, comprising:
the data acquisition module is used for acquiring target cache data in a main server of the remote dictionary service cluster; the target cache data are cache data obtained by storing corresponding changed service data to the main server after the service data are changed by the application server;
the data broadcasting module is used for broadcasting the target cache data to all clients of the distributed message middleware system through the server side of the distributed message middleware system; different clients respectively correspond to different slave servers in the remote dictionary service cluster;
the data sending module is used for controlling each client of the distributed message middleware system to respectively transmit the respectively acquired target cache data to the corresponding slave server so that each slave server can store the respectively acquired target cache data into the respective local cache space;
wherein, the data acquisition module includes:
the cache file reading unit is used for reading cache data files in a main server of the remote dictionary service cluster;
the file conversion processing unit is used for carrying out file conversion processing on the read cache data file through the data file converter so as to obtain the target cache data;
the cache file reading unit is specifically configured to read an RDB file in a main server of a remote dictionary service cluster, and read an AOF file in the main server after the RDB file is read;
the file conversion processing unit is specifically configured to parse the read RDB file and the read AOF file respectively through the data file converter to obtain corresponding first parsed data and second parsed data, and repackage the first parsed data and the second parsed data in a key value pair as a data recording manner, so as to obtain the target cache data.
6. An electronic device, comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the cache data synchronization method as claimed in any one of claims 1 to 4.
7. A computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements a cache data synchronization method as claimed in any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111445506.XA CN114328618B (en) | 2021-11-30 | 2021-11-30 | Cache data synchronization method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111445506.XA CN114328618B (en) | 2021-11-30 | 2021-11-30 | Cache data synchronization method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114328618A CN114328618A (en) | 2022-04-12 |
| CN114328618B true CN114328618B (en) | 2024-02-09 |
Family
ID=81049265
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111445506.XA Active CN114328618B (en) | 2021-11-30 | 2021-11-30 | Cache data synchronization method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114328618B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114547511B (en) * | 2022-04-27 | 2022-07-19 | 成都瑞华康源科技有限公司 | System data dictionary using method and system based on Internet of things MQTT protocol |
| CN114780643B (en) * | 2022-06-16 | 2022-10-21 | 恒生电子股份有限公司 | Data synchronization processing method and device, electronic equipment and readable storage medium |
| CN115118590A (en) * | 2022-06-22 | 2022-09-27 | 平安科技（深圳）有限公司 | Method, device, system, equipment and storage medium for managing configuration data |
| CN115412603B (en) * | 2022-11-02 | 2022-12-27 | 中国电子科技集团公司第十五研究所 | High-availability method and device for message client module of message middleware |
| CN116938961B (en) * | 2023-09-15 | 2024-03-01 | 北京趋动智能科技有限公司 | System and method for data synchronization and data reading method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105893543A (en) * | 2016-03-31 | 2016-08-24 | 微梦创科网络科技（中国）有限公司 | Data buffer service method and system |
| CN110633320A (en) * | 2018-05-30 | 2019-12-31 | 北京京东尚科信息技术有限公司 | Processing method, system, equipment and storage medium of distributed data service |
| CN113010549A (en) * | 2021-01-29 | 2021-06-22 | 腾讯科技（深圳）有限公司 | Data processing method based on remote multi-active system, related equipment and storage medium |
| CN113157466A (en) * | 2021-04-27 | 2021-07-23 | 上海销氪信息科技有限公司 | Message pushing method, device, system, electronic equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8495250B2 (en) * | 2009-12-16 | 2013-07-23 | International Business Machines Corporation | Asynchronous file operations in a scalable multi-node file system cache for a remote cluster file system |
-
2021
- 2021-11-30 CN CN202111445506.XA patent/CN114328618B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105893543A (en) * | 2016-03-31 | 2016-08-24 | 微梦创科网络科技（中国）有限公司 | Data buffer service method and system |
| CN110633320A (en) * | 2018-05-30 | 2019-12-31 | 北京京东尚科信息技术有限公司 | Processing method, system, equipment and storage medium of distributed data service |
| CN113010549A (en) * | 2021-01-29 | 2021-06-22 | 腾讯科技（深圳）有限公司 | Data processing method based on remote multi-active system, related equipment and storage medium |
| CN113157466A (en) * | 2021-04-27 | 2021-07-23 | 上海销氪信息科技有限公司 | Message pushing method, device, system, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114328618A (en) | 2022-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114328618B (en) | Cache data synchronization method, device, equipment and storage medium | |
| US10901812B2 (en) | Managing communication between cloud and heterogeneous devices across networks | |
| CN109542865B (en) | Method, device, system and medium for synchronizing configuration files of distributed cluster system | |
| EP3490224B1 (en) | Data synchronization method and system | |
| CN106453288B (en) | A kind of distributed micro services frame system that supporting asynchronous mode and its implementation | |
| CN106874424B (en) | A kind of collecting webpage data processing method and system based on MongoDB and Redis | |
| US20200336769A1 (en) | Video Live Broadcast Method and Apparatus | |
| EP3063925B1 (en) | Synchronizing event history for multiple clients | |
| US20150237113A1 (en) | Method and system for file transmission | |
| US20130339448A1 (en) | Subscriber device and subscription management that supports real-time communication | |
| US20120323990A1 (en) | Efficient state reconciliation | |
| EP2574004B1 (en) | Method, apparatus and system for improving synchronization efficiency of really simple syndication service | |
| CN108200219B (en) | Data synchronization method, device, server and storage medium | |
| US10069942B2 (en) | Method and apparatus for changing configurations | |
| CN107040563B (en) | Asynchronous service processing method and server | |
| CN111355986B (en) | Message processing method and device in live broadcast room and storage medium | |
| CN109547524B (en) | User behavior storage method, device, equipment and storage medium based on Internet of things | |
| US10972296B2 (en) | Messaging to enforce operation serialization for consistency of a distributed data structure | |
| CN110995566A (en) | Message data pushing method, system and device | |
| CN111212135A (en) | Message subscription method, device, system, electronic equipment and storage medium | |
| CN114448686B (en) | Cross-network communication device and method based on micro-service | |
| US20240015135A1 (en) | Domain management and synchronization system | |
| CN109542981B (en) | Data synchronization system and method, electronic device and storage medium | |
| CN109257448B (en) | Session information synchronization method and device, electronic equipment and storage medium | |
| CN115189931A (en) | Distributed key management method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |