CN112182095A - Method, device, equipment and medium for synchronizing service data among multiple machine rooms - Google Patents
Method, device, equipment and medium for synchronizing service data among multiple machine rooms Download PDFInfo
- Publication number
- CN112182095A CN112182095A CN202010896795.4A CN202010896795A CN112182095A CN 112182095 A CN112182095 A CN 112182095A CN 202010896795 A CN202010896795 A CN 202010896795A CN 112182095 A CN112182095 A CN 112182095A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- synchronization
- redis
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The invention provides a method, a device, equipment and a medium for synchronizing service data among multiple machine rooms, wherein the method comprises the following steps: business services and synchronization services; setting a data synchronization frame, and uniformly taking over a synchronization mode by code intrusion into each server end; if the data is modified, intercepting the data into a queue of the business service, and serializing the data; the service sends the serialized data to the synchronization service; the synchronization service carries out data deserialization operation on the received serialized data, writes the deserialized data into a remote server and supports multidirectional synchronization of data among different computer rooms.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a medium for service data synchronization between multiple computer rooms.
Background
When data among a plurality of computer rooms needs to be synchronized, most of the existing products on the market are completely synchronized aiming at a certain library/table, the fine granularity is coarse, and specific data cannot be synchronized according to business and requirements. And an overall solution for multi-directional data synchronization of data such as redis, es and the like is lacking.
Taking redis as an example, synchronization is mainly realized by performing master-slave configuration between nodes at present, one writing node and a plurality of slave nodes are allowed to exist, but if the nodes cross a computer room, multi-writing synchronization of multi-region data and master-slave configuration of redis can not be realized, data replication based on master-slave protocols is basically adopted at present, and transmission of the nodes cross the computer room is realized by adopting middleware such as MQ and the like, taking a synchronization tool XPipe of redis as an example, the process is as follows:
as shown in fig. 1, the Slave keeper (pretended to be a redis Slave) collects the redis logs, and then delivers the logs to proxy for compression and synchronization. Meanwhile, proxy is also responsible for receiving data from the other side and writing back to Masterkeeper (pretended to be redis master).
XPipe can't support two writing under the multizone (surpassing 2 computer rooms), has adopted the log under master slave agreement to go to update simultaneously, so also can't follow the business and control synchronous granularity, must whole redis synchronous together.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a method, an apparatus, a device and a medium for service data synchronization between multiple machine rooms, which support multidirectional synchronization of data between different machine rooms.
In a first aspect, the present invention provides a method for service data synchronization between multiple machine rooms, including: business services and synchronization services;
step 1, setting a data synchronization frame, and uniformly taking over a synchronization mode by code intrusion into each server;
step 2, if the data is modified, intercepting the data into a queue of the business service, and serializing the data;
step 3, the service sends the serialized data to the synchronization service;
and 4, the synchronization service performs data deserialization operation on the received serialized data and writes the data into a remote server.
Furthermore, an Elasticsearch component, a Rabbitmq component, a Redis component and a Mysql component are encapsulated in the data synchronization framework, and are used for providing Elasticsearch service, Rabbitmq service, Redis service and Mysql service, and providing an interface consistent with the original sdk.
Further, the step 4 is further specifically: the synchronization service carries out data deserialization on the received serialized data to obtain Mysql client, elastic search client, Redis client or mq client, and writes the data into corresponding elastic search service, Rabbitmq service, Redis service and Mysql service in a remote server.
In a second aspect, the present invention provides an apparatus for synchronizing service data between multiple machine rooms, including: business services and synchronization services;
the intrusion module is provided with a data synchronization frame, and uniformly takes over a synchronization mode through a code intrusion server side;
the serialization module is used for intercepting the data into a queue of the business service and serializing the data if the data is modified;
the sending module is used for sending the serialized data to the synchronous service by the business service;
and the synchronization module is used for performing data deserialization operation on the received serialized data by the synchronization service and writing the data deserialized data into a remote server.
Furthermore, an Elasticsearch component, a Rabbitmq component, a Redis component and a Mysql component are encapsulated in the data synchronization framework, and are used for providing Elasticsearch service, Rabbitmq service, Redis service and Mysql service, and providing an interface consistent with the original sdk.
Further, the synchronization module is further specifically: the synchronization service carries out data deserialization on the received serialized data to obtain Mysql client, elastic search client, Redis client or mq client, and writes the data into corresponding elastic search service, Rabbitmq service, Redis service and Mysql service in a remote server.
In a third aspect, the present invention provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of the first aspect when executing the program.
In a fourth aspect, the invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the method of the first aspect.
One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:
the method, the device, the equipment and the medium for service data synchronization among multiple machine rooms, provided by the embodiment of the application, realize fine synchronization control of data and simultaneously support multidirectional synchronization of data among different machine rooms.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
The invention will be further described with reference to the following examples with reference to the accompanying drawings.
FIG. 1 is a diagram illustrating a conventional data synchronization;
FIG. 2 is a schematic diagram of data synchronization according to the present invention;
FIG. 3 is a flow chart of a method according to one embodiment of the present invention;
fig. 4 is a schematic structural diagram of a device according to a second embodiment of the present invention.
Detailed Description
The embodiment of the application provides a method, a device, equipment and a medium for service data synchronization among multiple machine rooms, so that the problem that the synchronization granularity cannot be controlled from the service and the whole redis must be synchronized together is solved; the aim of supporting the multidirectional synchronization of data among different machine rooms is achieved.
The technical scheme in the embodiment of the application has the following general idea:
setting a data synchronization frame, and providing the following characteristics for data synchronization through a mode that a code invades a server end to uniformly take over synchronization:
supporting data synchronization and multi-write modes of an elastic search, Rabbitmq, Redis and Mysql; the formats of the data are different, but the data are serialized into json character strings, and the data can be deserialized back after being received by the Server end;
the business service controls the granularity of synchronization through code intrusion:
client.insert (data 1); // data1 will not be synchronized
client. createsnc (). insert (data 2); // data2 are synchronized
As shown in fig. 2, when the createSync generates modification, the modification is intercepted into a queue of mysql changes/es changes/redis changes/mq changes (a position for controlling synchronization is configured by a cloud), serialized by a synchronization processor, and sent to a synchronization service through a data transmitter, thereby completing a synchronization process of a client. Where the data sender is generated by a service monitor that determines how the data is sent to the synchronization service, e.g. in the form of a message queue.
The data synchronization service receives the information in the queue, deserializes the data into mysql client/es client/redis client/mq client operation according to the corresponding target in the synchronization processor, writes the operation into a remote server, and completes the cross-region data writing, namely completes the data synchronization; taking a redis client as an example, a redis client instance created in the synchronization service is shown, and the redis client instance is connected with a remote redis server (namely, a redis service). For example: json data, the operation of deserializing json into a client is equivalent to calling a method of the client, such as a set method of a redis client. The method writes the data into the remote redis server, and completes the synchronization of the data.
Similarly, by deploying the corresponding synchronization service in each area, multi-area data multi-writing can be realized.
Example one
As shown in fig. 3, this embodiment provides a method for service data synchronization between multiple machine rooms, including: business services and synchronization services;
step 1, setting a data synchronization frame, and taking over a synchronization mode uniformly by code intrusion into each server side, wherein an elastic search component, a Rabbitmq component, a Redis component and a Mysql component are encapsulated in the data synchronization frame, are used for providing an elastic search service, a Rabbitmq service, a Redis service and a Mysql service, and provide an interface consistent with original sdk;
step 2, if the data is modified, intercepting the data into a queue of the business service, and serializing the data;
step 3, the service sends the serialized data to the synchronization service;
and 4, the synchronization service carries out data deserialization on the received serialized data to obtain Mysql client, elastic search client, Redis client or mq client, and writes the Mysql client, the elastic search client, the Redis client or the mq client into a remote server corresponding to the elastic search service, the Rabbitmq service, the Redis service and the Mysql service.
Based on the same inventive concept, the application also provides a device corresponding to the method in the first embodiment, which is detailed in the second embodiment.
Example two
As shown in fig. 4, in this embodiment, an apparatus for performing service data synchronization between multiple machine rooms is provided, including: business services and synchronization services;
the system comprises an intrusion module, a data synchronization module and a code intrusion server side, wherein the data synchronization module is provided with a data synchronization frame, uniformly takes over a synchronization mode through the code intrusion server side, and is used for packaging an Elasticissearch component, a Rabbitmq component, a Redis component and a Mysql component, providing Elasticissearch service, Rabbitmq service, Redis service and Mysql service and providing an interface consistent with original sdk;
the serialization module is used for intercepting the data into a queue of the business service and serializing the data if the data is modified;
the sending module is used for sending the serialized data to the synchronous service by the business service;
and the synchronization module is used for performing data deserialization on the received serialized data by the synchronization service to obtain Mysql client, elastic search client, Redis client or mq client operation, and writing the operation into corresponding elastic search service, Rabbitmq service, Redis service and Mysql service in a remote server.
Since the apparatus described in the second embodiment of the present invention is an apparatus used for implementing the method of the first embodiment of the present invention, based on the method described in the first embodiment of the present invention, a person skilled in the art can understand the specific structure and the deformation of the apparatus, and thus the details are not described herein. All the devices adopted in the method of the first embodiment of the present invention belong to the protection scope of the present invention.
Based on the same inventive concept, the application provides an electronic device embodiment corresponding to the first embodiment, which is detailed in the third embodiment.
EXAMPLE III
The embodiment provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, any one of the embodiments may be implemented.
Since the electronic device described in this embodiment is a device used for implementing the method in the first embodiment of the present application, based on the method described in the first embodiment of the present application, a specific implementation of the electronic device in this embodiment and various variations thereof can be understood by those skilled in the art, and therefore, how to implement the method in the first embodiment of the present application by the electronic device is not described in detail herein. The equipment used by those skilled in the art to implement the methods in the embodiments of the present application is within the scope of the present application.
Based on the same inventive concept, the application provides a storage medium corresponding to the fourth embodiment, which is described in detail in the fourth embodiment.
Example four
The present embodiment provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any one of the first embodiment can be implemented.
The technical scheme provided in the embodiment of the application at least has the following technical effects or advantages: the methods, devices, systems, apparatuses, and media provided by embodiments of the present application,
as will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Although specific embodiments of the invention have been described above, it will be understood by those skilled in the art that the specific embodiments described are illustrative only and are not limiting upon the scope of the invention, and that equivalent modifications and variations can be made by those skilled in the art without departing from the spirit of the invention, which is to be limited only by the appended claims.
Claims (8)
1. A method for synchronizing service data among multiple machine rooms is characterized in that: the method comprises the following steps: business services and synchronization services;
step 1, setting a data synchronization frame, and uniformly taking over a synchronization mode by code intrusion into each server;
step 2, if the data is modified, intercepting the data into a queue of the business service, and serializing the data;
step 3, the service sends the serialized data to the synchronization service;
and 4, the synchronization service performs data deserialization operation on the received serialized data and writes the data into a remote server.
2. The method for synchronizing service data among multiple rooms as claimed in claim 1, wherein: the data synchronization framework is packaged with an Elasticissearch component, a Rabbitmq component, a Redis component and a Mysql component, and is used for providing Elasticissearch service, Rabbitmq service, Redis service and Mysql service and providing an interface consistent with original sdk.
3. The method for service data synchronization among multiple rooms as claimed in claim 2, wherein: the step 4 is further specifically as follows: the synchronization service carries out data deserialization on the received serialized data to obtain Mysql client, elastic search client, Redis client or mq client, and writes the data into corresponding elastic search service, Rabbitmq service, Redis service and Mysql service in a remote server.
4. A device for synchronizing service data among multiple machine rooms is characterized in that: the method comprises the following steps: business services and synchronization services;
the intrusion module is provided with a data synchronization frame, and uniformly takes over a synchronization mode through a code intrusion server side;
the serialization module is used for intercepting the data into a queue of the business service and serializing the data if the data is modified;
the sending module is used for sending the serialized data to the synchronous service by the business service;
and the synchronization module is used for performing data deserialization operation on the received serialized data by the synchronization service and writing the data deserialized data into a remote server.
5. The method for service data synchronization between multiple rooms as claimed in claim 4, wherein: the data synchronization framework is packaged with an Elasticissearch component, a Rabbitmq component, a Redis component and a Mysql component, and is used for providing Elasticissearch service, Rabbitmq service, Redis service and Mysql service and providing an interface consistent with original sdk.
6. The method for service data synchronization between multiple rooms as claimed in claim 5, wherein: the synchronization module further specifically comprises: the synchronization service carries out data deserialization on the received serialized data to obtain Mysql client, elastic search client, Redis client or mq client, and writes the data into corresponding elastic search service, Rabbitmq service, Redis service and Mysql service in a remote server.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 3 when executing the program.
8. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010896795.4A CN112182095A (en) | 2020-08-31 | 2020-08-31 | Method, device, equipment and medium for synchronizing service data among multiple machine rooms |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010896795.4A CN112182095A (en) | 2020-08-31 | 2020-08-31 | Method, device, equipment and medium for synchronizing service data among multiple machine rooms |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112182095A true CN112182095A (en) | 2021-01-05 |
Family
ID=73924005
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010896795.4A Pending CN112182095A (en) | 2020-08-31 | 2020-08-31 | Method, device, equipment and medium for synchronizing service data among multiple machine rooms |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112182095A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108256002A (en) * | 2017-12-31 | 2018-07-06 | 广东欧珀移动通信有限公司 | Across computer room method of data synchronization, device, system and server |
| WO2019061720A1 (en) * | 2017-09-29 | 2019-04-04 | 网宿科技股份有限公司 | Data synchronization method and system |
| CN110309231A (en) * | 2019-07-12 | 2019-10-08 | 焦点科技股份有限公司 | A kind of method of data synchronization and system across computer room |
| CN110597910A (en) * | 2019-09-12 | 2019-12-20 | 聚好看科技股份有限公司 | Remote data synchronization method, device and system |
| CN110750594A (en) * | 2019-09-30 | 2020-02-04 | 上海视云网络科技有限公司 | Mysql-based real-time cross-network database synchronization method for incremental logs |
-
2020
- 2020-08-31 CN CN202010896795.4A patent/CN112182095A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019061720A1 (en) * | 2017-09-29 | 2019-04-04 | 网宿科技股份有限公司 | Data synchronization method and system |
| CN108256002A (en) * | 2017-12-31 | 2018-07-06 | 广东欧珀移动通信有限公司 | Across computer room method of data synchronization, device, system and server |
| CN110309231A (en) * | 2019-07-12 | 2019-10-08 | 焦点科技股份有限公司 | A kind of method of data synchronization and system across computer room |
| CN110597910A (en) * | 2019-09-12 | 2019-12-20 | 聚好看科技股份有限公司 | Remote data synchronization method, device and system |
| CN110750594A (en) * | 2019-09-30 | 2020-02-04 | 上海视云网络科技有限公司 | Mysql-based real-time cross-network database synchronization method for incremental logs |
Non-Patent Citations (2)
| Title |
|---|
| 李宁: ""一种异构关系型数据库间的数据同步方案"", 《福建电脑》 * |
| 范辉等: "《Python大数据基础与实战》", 31 July 2019, 西安电子科技大学出版社 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111506412B (en) | Airflow-based distributed asynchronous task construction and scheduling system and method | |
| CN109951331B (en) | Method, device and computing cluster for sending information | |
| CN110971655B (en) | Offline client playback and synchronization | |
| CN111614729B (en) | Method and device for controlling kubernetes container cluster and electronic equipment | |
| CN113742031B (en) | Node state information acquisition method and device, electronic equipment and readable storage medium | |
| CN108932157B (en) | Method, system, electronic device and readable medium for distributed processing of tasks | |
| CN111897633A (en) | Task processing method and device | |
| US20210081263A1 (en) | System for offline object based storage and mocking of rest responses | |
| CN109245908B (en) | Method and device for switching master cluster and slave cluster | |
| EP3713195B1 (en) | Log processing method, related device, and system | |
| CN111858045A (en) | Multitask GPU resource scheduling method, device, equipment and readable medium | |
| US11223522B1 (en) | Context-based intelligent re-initiation of microservices | |
| CN112905337A (en) | Software and hardware hybrid deployment MySQL cluster scheduling method and device | |
| CN115080101A (en) | Application management method, application subscription method and related equipment | |
| CN111064626A (en) | Configuration updating method, device, server and readable storage medium | |
| CN105430028A (en) | Service calling method, service providing method, and node | |
| CN112182095A (en) | Method, device, equipment and medium for synchronizing service data among multiple machine rooms | |
| CN110597600A (en) | Method for modifying application configuration parameters of distributed system and distributed system | |
| US10728323B2 (en) | Method and apparatus for operating infrastructure layer in cloud computing architecture | |
| CN115629909A (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN110661857B (en) | Data synchronization method and device | |
| CN110427260B (en) | Host job scheduling method, device and system | |
| CN113868184A (en) | Configuration method and system of master-slave architecture cache-based configuration center | |
| US9674307B1 (en) | Running mobile applications in a distributed client-server model using selective delegation of hardware and software contexts to the client | |
| CN116016555A (en) | Message synchronization method, device, equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |