CN114281827A - Method, device and equipment for real-time data synchronization and storage medium - Google Patents
Method, device and equipment for real-time data synchronization and storage medium Download PDFInfo
- Publication number
- CN114281827A CN114281827A CN202111618964.9A CN202111618964A CN114281827A CN 114281827 A CN114281827 A CN 114281827A CN 202111618964 A CN202111618964 A CN 202111618964A CN 114281827 A CN114281827 A CN 114281827A
- Authority
- CN
- China
- Prior art keywords
- data
- compensation
- synchronization
- mysql
- synchronization system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a method, a device, equipment and a storage medium for real-time synchronization of data, wherein the method comprises the following steps: creating a plurality of ES indexes based on a synchronization system; the synchronization system creates heterogeneous data based on the update data; the synchronous system writes the heterogeneous data into the corresponding ES index; the synchronous system sends each operation information of the ES to a compensation system; the compensation system determines a compensation target based on the operational information. The technical scheme of the invention realizes the real-time synchronization of Mysql and ES data, the data can be directly inquired from the ES index without being inquired through the Mysql whole-database full-table scanning, the data inquiry efficiency is improved, the massive data inquiry requirement is met, and the service processing and the data synchronization are completely decoupled; in addition, the timing comparison between heterogeneous data sources automatically compensates for various data losses caused by accidents, and ensures the integrity of data without manual intervention.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to a method, a device, equipment and a storage medium for real-time synchronization of data.
Background
The search engine elastic search (ES for short) can be used as a heterogeneous data source of Mysql, and makes up for the defects of Mysql library sub-database and sub-table query. The ES is an open source, distributed, high performance search engine, each of which can be searched as an index field. However, there are corresponding problems in using the ES, for example, how to synchronously store the data in Mysql in the ES in near real time, that is, a data consistency problem, and in the process of data synchronous storage, due to abnormal operation of each node program or network fluctuation, part of synchronous data may be lost, and when this scenario occurs, how to compensate for the lost data, and ensure data integrity.
Currently, real-time synchronization of Mysql and ES data is realized based on the following two ways: 1. the method is realized by calling a synchronous interface between applications; 2. by asynchronous message communication between applications. However, the first implementation manner has a high coupling degree, and the real-time service is affected no matter the interface caller application or the interface provider application has a system abnormality; even more, systemic risks may be incurred; a second implementation, although decoupling between applications is achieved; however, if the synchronization of Mysql and ES is realized by depending on the service application sending or consuming messages, a certain influence is also caused to the real-time service.
In addition, the existing heterogeneous data source synchronization scheme cannot automatically detect the difference between data sources and cannot avoid the problem of data loss caused by accidents.
Disclosure of Invention
The present invention is directed to solving, at least to some extent, one of the technical problems in the related art. Therefore, an object of the present invention is to provide a method, an apparatus, a device and a storage medium for real-time synchronization of data.
In order to solve the above technical problem, an embodiment of the present invention provides the following technical solutions:
a method of real-time synchronization of data, comprising:
creating a plurality of ES indexes based on a synchronization system;
the synchronization system creates heterogeneous data based on the update data;
the synchronous system writes the heterogeneous data into the corresponding ES index;
the synchronous system sends each operation information of the ES to a compensation system;
the compensation system determines a compensation target based on the operational information.
Optionally, the creating a plurality of ES indexes based on the synchronization system includes:
the synchronization system creates a plurality of ES indexes based on a data model of a Mysql data table and a query service requirement.
Optionally, the synchronization system creates heterogeneous data based on the updated data, including:
creating a first Kafka message queue based on the Mysql data table;
if the Mysql data table has data change, the Mysql generates a Binlog corresponding to the data change; the data change comprises data creation or data update;
the Mysql sends a Bilog event corresponding to the Bilog to the first Kafka message queue.
Optionally, the synchronization system creates heterogeneous data based on the updated data, and further includes:
the consumers of the first Kafka message queue monitor the first Kafka message queue in real time;
when the consumers of the first Kafka message queue monitor the Binlog event, the synchronization system analyzes the Binlog event to obtain an analysis result;
the synchronization system creates heterogeneous data based on the parsing result.
Optionally, the writing, by the synchronization system, the heterogeneous data into the corresponding ES index includes:
and writing the heterogeneous data into the corresponding ES index based on the query service requirement.
Optionally, the sending, by the synchronization system, each operation information of the ES to a compensation system includes:
the synchronization system generating an operation event based on each of the operation information of the ES;
the synchronization system sends the operational event to a second Kafka message queue of the compensation system.
Optionally, the determining, by the compensation system, a compensation target based on the operation information includes:
the consumers of the second Kafka message queue monitor the second Kafka message queue;
when the consumer of the second Kafka message queue monitors the operation event;
the compensation system stores the operation event into an ES operation information table of Mysql;
refluxing the ES operation information table to acquire reflux data;
the compensation system determines the compensation target based on the backflow data.
Optionally, the compensation system determines the compensation target based on the backflow data, including:
the compensation system assembles the backflow data to obtain assembly data; wherein the assembly data is consistent with a structure of data of the ES;
the compensation system stores the assembly data into a reflux data table of the Mysql;
the compensation system determines the compensation target based on the backflow data table.
Optionally, the compensation system determines the compensation target based on the backflow data, including:
presetting a time interval;
comparing the information of the ES operation data table with the information of the reflux table at preset time intervals;
if the information of the ES operation data table is inconsistent with the information of the reflux table, acquiring difference data;
the compensation system determines the compensation target based on the difference data.
Optionally, after the compensation system determines the compensation target based on the difference data, the compensation system further includes:
and the compensation system stores the compensation target to the ES index according to the query service requirement.
The embodiment of the invention also provides a device for real-time synchronization of data, which comprises:
a first creation module for creating a plurality of ES indexes based on a synchronization system;
a second creation module for the synchronization system to create heterogeneous data based on the update data;
a writing module, configured to write the heterogeneous data into the corresponding ES index by the synchronization system;
a sending module, configured to send, by the synchronization system, each operation information of the ES to a compensation system;
a compensation module for the compensation system to determine a compensation target based on the operational information.
Embodiments of the present invention also provide an electronic device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, the processor implementing the method as described above when executing the computer program.
Embodiments of the present invention also provide a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed, controls an apparatus in which the computer-readable storage medium is located to perform the method as described above.
The embodiment of the invention has the following technical effects:
according to the technical scheme, real-time synchronization of Mysql and ES data is realized, the data can be directly inquired from the ES index without being inquired through Mysql full-database full-table scanning, the data inquiry efficiency is improved, the massive data inquiry requirement is met, and the service processing and the data synchronization are completely decoupled; in addition, the timing comparison between heterogeneous data sources automatically compensates for various data losses caused by accidents, and ensures the integrity of data without manual intervention.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
FIG. 1 is a flow chart of a method for real-time data synchronization according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an execution flow of a synchronization system according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart illustrating an implementation of a compensation system according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an apparatus for real-time data synchronization according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.
To facilitate understanding of the embodiments by those skilled in the art, some terms are explained:
(1) ES: elasticsearch, search server.
(2) Mysql: a relational database management system.
(3) Kafka: a high throughput distributed publish-subscribe messaging system that can handle all the activity flow data of a consumer in a website; these data are typically addressed by handling logs and log aggregations due to throughput requirements.
(4) Binlog: and the binary log file is used for recording data update or potential update of mysql.
(5) And (3) DataX: a heterogeneous data source offline synchronization tool.
(6) BLCS: boundary Layer Control System.
(7) Producer: the generator of the message.
(8) Consumer: a consumer of the message.
(9) And (4) Broker: each kafka instance (or each kafka server node) is a Broker.
(10) DC: a data center operating system.
(11) ODPS: open Data Processing Service, a big Data computing Service.
(12) RDS: relational Database Service.
As shown in fig. 1, an embodiment of the present invention provides a method for real-time synchronization of data, including:
step S1: creating a plurality of ES indexes based on a synchronization system;
specifically, the creating of the ES index based on the synchronization system includes:
the synchronization system creates a plurality of ES indexes based on a data model of a Mysql data table and a query service requirement.
In an actual application scenario, an ES index is created based on a data model of the Mysql data table and an actual query service requirement, the ES index is used as heterogeneous data of the Mysql, an ES index field can supplement and convert data of the Mysql data table, and according to an actual service scenario, the data can be divided according to cold and hot search data to create a plurality of ES indexes.
Step S2: the synchronization system creates heterogeneous data based on the update data;
specifically, the creating, by the synchronization system, heterogeneous data based on the updated data includes:
creating a first Kafka message queue based on the Mysql data table;
if the Mysql data table has data change, the Mysql generates a Binlog corresponding to the data change; the data change comprises data creation or data update;
the Mysql sends a Bilog event corresponding to the Bilog to the first Kafka message queue.
In a practical application scenario, a first Kafka message queue is created based on the Mysql data table, and when data creation or update occurs on the Mysql data table, the Binlog generated by the Mysql is sent to the corresponding first Kafka message queue as an event by the synchronization system.
Specifically, the synchronization system creates heterogeneous data based on the update data, and further includes:
the consumers of the first Kafka message queue monitor the first Kafka message queue in real time;
when the consumers of the first Kafka message queue monitor the Binlog event, the synchronization system analyzes the Binlog event to obtain an analysis result;
the synchronization system creates heterogeneous data based on the parsing result.
The synchronization system assembles the analysis result into heterogeneous data according to rules, for example, assembles the analysis result into heterogeneous data according to rules such as policy data and insured persons.
In an actual application scenario, the synchronization system is responsible for monitoring the Binlog message generated by the Mysql in the first Kafka message queue, analyzing the Binlog message according to rules, assembling the Binlog message into heterogeneous data and synchronously storing the heterogeneous data into the ES index.
Binlog is a binary log of MySQL that records all Data Definition Language (DDL) and Data Manipulation Language (DML) statements, except data query statements, and records in the form of events. Therefore, when the Mysql data table is created and updated, the Bilog is generated, and the consistency of the Mysql and the ES data source can be ensured as long as the ES can successfully record the heterogeneous data assembled by each Bilog; and the generation of the Binlog and the event sending are basically near real-time operations, and zero delay of Mysql and ES synchronization can be realized.
Step S3: the synchronous system writes the heterogeneous data into the corresponding ES index;
specifically, the writing, by the synchronization system, the heterogeneous data into the corresponding ES index includes:
and writing the heterogeneous data into the corresponding ES index based on the query service requirement.
After writing the ES index successfully, the synchronization system sends the ES operation information of this time as an event to a second Kafka message queue of the compensation system, so as to facilitate subsequent data compensation.
In an actual application scenario, a synchronization system may have a situation that the ES cannot be guaranteed to successfully record heterogeneous data assembled by each Binlog;
for example: the method and the device can cause accidents due to abnormal program operation or network fluctuation and the like, and a compensation system is further arranged according to the embodiment of the invention.
Step S4: the synchronous system sends each operation information of the ES to a compensation system;
specifically, the step of sending each operation information of the ES to a compensation system by the synchronization system includes:
the synchronization system generating an operation event based on each of the operation information of the ES;
the synchronization system sends the operational event to a second Kafka message queue of the compensation system.
According to the embodiment of the invention, a synchronous system is constructed, the Bilog generated by Mysql master-slave replication is monitored, the Bilog is analyzed and assembled into heterogeneous data according to rules to be stored in the ES index; the Binglog generation is basically close to real time, the real time of data synchronization can be ensured, and the service processing and data synchronization are completely decoupled by monitoring the Binglog of Mysql by an independent synchronization system.
As shown in fig. 2, the above-mentioned embodiment of the present invention can be implemented based on the following implementation manners:
(1) the synchronization system creates a first Kafka message queue based on the Mysql data table, and when the Mysql data table generates data creation or updating, the Binlog generated by the Mysql based on the producer can be sent to the corresponding first Kafa message queue as an event;
(2) the Broker1 and the Broker2 buffer and process data of the first Kafa message queue.
(3) The consumer monitors the message queue in real time, analyzes the event when monitoring the Binlog event, and assembles the analysis result into heterogeneous data according to rules;
(4) and the synchronous system writes the heterogeneous data into the corresponding ES index according to the query service requirement.
Step S5: the compensation system determines a compensation target based on the operational information.
Specifically, the determining, by the compensation system, a compensation target based on the operation information includes:
the consumers of the second Kafka message queue monitor the second Kafka message queue;
when the consumer of the second Kafka message queue monitors the operation event;
the compensation system stores the operation event into an ES operation information table of Mysql;
refluxing the ES operation information table to acquire reflux data;
the compensation system determines the compensation target based on the backflow data.
The embodiment of the invention is independent of a synchronous system, and also constructs a compensation system, which can regularly reflux the data of Mysql in a certain time period to the Mysql reflux flow table, assemble the reflux data into the data consistent with the ES data structure, then compare the reflux data with the ES data, and compensate the lost data based on the comparison result.
Specifically, the determining, by the compensation system, the compensation target based on the backflow data includes:
the compensation system assembles the backflow data to obtain assembly data; wherein the assembly data is consistent with a structure of data of the ES;
the compensation system stores the assembly data into a reflux data table of the Mysql;
the compensation system determines the compensation target based on the backflow data table.
In an actual application scenario, the Mysql data table is periodically reflowed by using DataX, the reflowed data is assembled into data consistent with the ES data structure according to rules, and the data is stored in the Mysql reflow table in Mysql.
Specifically, the determining, by the compensation system, the compensation target based on the backflow data includes:
presetting a time interval;
comparing the information of the ES operation data table with the information of the reflux table at preset time intervals;
if the information of the ES operation data table is inconsistent with the information of the reflux table, acquiring difference data;
the compensation system determines the compensation target based on the difference data.
In an actual application scene, comparing the ES operation information table with the Mysql reflux table according to a certain preset time interval, and finding the difference between the ES operation information table and the Mysql reflux table;
the compensation system is responsible for refluxing Mysql data, storing the successfully stored data of the ES in real time, comparing the Mysql refluxing data with the successfully stored data of the ES at preset time intervals, and automatically compensating missing data in the ES to ensure that the Mysql and the ES data are consistent in the time period; the preset time interval may be any time period such as 1 hour, 30 minutes, 10 minutes, 5 minutes, and the like, and may even be real-time, and the compensation system is not particularly limited.
Specifically, after the compensation system determines the compensation target based on the difference data, the compensation system further includes:
and the compensation system stores the compensation target to the ES index according to the query service requirement.
Wherein, after acquiring the difference data, the compensation system can determine the compensation target and store the compensation target to the ES index in the same manner.
For example, after the compensation system acquires the difference data, the difference data is sent to the synchronization system; the synchronization system creates a compensated ES index based on the difference data; the Mysql data table shows data change based on the difference data, and then Mysql generates a compensation Binlog corresponding to the data change; the Mysql sends a compensation Binlog event to the first Kafka message queue; the consumers of the first Kafka message queue monitor the first Kafka message queue in real time; when the consumers of the first Kafka message queue monitor the compensation Binlog event, the synchronization system analyzes the compensation Binlog event to obtain a compensation analysis result; the synchronization system creates compensation heterogeneous data based on the compensation analysis result; and the synchronous system writes the compensation heterogeneous data into the corresponding compensation ES index based on the query service requirement.
The embodiment of the invention realizes the real-time synchronization of Mysql and ES data, the data can be directly inquired from the ES index without being inquired through Mysql full-table scanning, the data inquiry efficiency is improved, the massive data inquiry requirement is met, and the service processing and the data synchronization are completely decoupled; in addition, the timing comparison between heterogeneous data sources automatically compensates for various data losses caused by accidents, and ensures the integrity of data without manual intervention.
As shown in fig. 3, the embodiment of the present invention can be implemented based on the following implementation manners:
(1) the compensation system performs data reflux based on ODPS, wherein RDS is executed after being delayed for 1 hour and is executed for 1 time per hour; meanwhile, the ES data summarization (ES operation information table summarization) is performed according to 1 hour, and 1 time is performed in each hour;
(2) comparing the reflux data with the collected ES operation information table every hour;
(3) confirming whether there is difference, if there is no difference, ending the compensation process, if there is difference, sending the difference result in message form;
(4) enabling the Kafa data table to have data updating based on the message sent in the step (3), and finally generating a compensation Binlog event;
(5) analyzing the compensation Binlog event by the DC to obtain compensation heterogeneous data;
(6) the compensation system writes the compensated heterogeneous data into the corresponding compensated ES index.
As shown in fig. 4, an embodiment of the present invention further provides an apparatus 400 for real-time synchronization of data, including:
a first creating module 401 for creating a plurality of ES indexes based on a synchronization system;
a second creating module 402, configured to create heterogeneous data based on the update data by the synchronization system;
a writing module 403, configured to write the heterogeneous data into the corresponding ES index by the synchronization system;
a sending module 404, configured to send, by the synchronization system, each operation information of the ES to a compensation system;
a compensation module 405 for the compensation system to determine a compensation target based on the operational information.
Optionally, the creating a plurality of ES indexes based on the synchronization system includes:
the synchronization system creates a plurality of ES indexes based on a data model of a Mysql data table and a query service requirement.
Optionally, the synchronization system creates heterogeneous data based on the updated data, including:
creating a first Kafka message queue based on the Mysql data table;
if the Mysql data table has data change, the Mysql generates a Binlog corresponding to the data change; the data change comprises data creation or data update;
the Mysql sends a Bilog event corresponding to the Bilog to the first Kafka message queue. Optionally, the synchronization system creates heterogeneous data based on the updated data, and further includes:
the consumers of the first Kafka message queue monitor the first Kafka message queue in real time;
when the consumers of the first Kafka message queue monitor the Binlog event, the synchronization system analyzes the Binlog event to obtain an analysis result;
the synchronization system creates heterogeneous data based on the parsing result.
Optionally, the writing, by the synchronization system, the heterogeneous data into the corresponding ES index includes:
and writing the heterogeneous data into the corresponding ES index based on the query service requirement.
Optionally, the sending, by the synchronization system, each operation information of the ES to a compensation system includes:
the synchronization system generating an operation event based on each of the operation information of the ES;
the synchronization system sends the operational event to a second Kafka message queue of the compensation system.
Optionally, the determining, by the compensation system, a compensation target based on the operation information includes:
the consumers of the second Kafka message queue monitor the second Kafka message queue;
when the consumer of the second Kafka message queue monitors the operation event;
the compensation system stores the operation event into an ES operation information table of Mysql;
refluxing the ES operation information table to acquire reflux data;
the compensation system determines the compensation target based on the backflow data.
Optionally, the compensation system determines the compensation target based on the backflow data, including:
the compensation system assembles the backflow data to obtain assembly data; wherein the assembly data is consistent with a structure of data of the ES;
the compensation system stores the assembly data into a reflux data table of the Mysql;
the compensation system determines the compensation target based on the backflow data table.
Optionally, the compensation system determines the compensation target based on the backflow data, including:
presetting a time interval;
comparing the information of the ES operation data table with the information of the reflux table at preset time intervals;
if the information of the ES operation data table is inconsistent with the information of the reflux table, acquiring difference data;
the compensation system determines the compensation target based on the difference data.
Optionally, after the compensation system determines the compensation target based on the difference data, the compensation system further includes:
and the compensation system stores the compensation target to the ES index according to the query service requirement.
Embodiments of the present invention also provide an electronic device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, the processor implementing the method as described above when executing the computer program.
Embodiments of the present invention also provide a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed, controls an apparatus in which the computer-readable storage medium is located to perform the method as described above.
In addition, other configurations and functions of the apparatus according to the embodiment of the present invention are known to those skilled in the art, and are not described herein for reducing redundancy.
It should be noted that the logic and/or steps represented in the flowcharts or otherwise described herein, such as an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
In the description of the present invention, it is to be understood that the terms "central," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," "circumferential," and the like are used in the orientations and positional relationships indicated in the drawings for convenience in describing the invention and to simplify the description, and are not intended to indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and are therefore not to be considered limiting of the invention.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
In the present invention, unless otherwise expressly stated or limited, the terms "mounted," "connected," "secured," and the like are to be construed broadly and can, for example, be fixedly connected, detachably connected, or integrally formed; can be mechanically or electrically connected; they may be directly connected or indirectly connected through intervening media, or they may be connected internally or in any other suitable relationship, unless expressly stated otherwise. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
In the present invention, unless otherwise expressly stated or limited, the first feature "on" or "under" the second feature may be directly contacting the first and second features or indirectly contacting the first and second features through an intermediate. Also, a first feature "on," "over," and "above" a second feature may be directly or diagonally above the second feature, or may simply indicate that the first feature is at a higher level than the second feature. A first feature being "under," "below," and "beneath" a second feature may be directly under or obliquely under the first feature, or may simply mean that the first feature is at a lesser elevation than the second feature.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
Claims (13)
1. A method for real-time synchronization of data, comprising:
creating a plurality of ES indexes based on a synchronization system;
the synchronization system creates heterogeneous data based on the update data;
the synchronous system writes the heterogeneous data into the corresponding ES index;
the synchronous system sends each operation information of the ES to a compensation system;
the compensation system determines a compensation target based on the operational information.
2. The method of claim 1, wherein the creating a plurality of ES indices based on a synchronization system comprises:
the synchronization system creates a plurality of ES indexes based on a data model of a Mysql data table and a query service requirement.
3. The method of claim 2, wherein the synchronization system creates heterogeneous data based on the update data, comprising:
creating a first Kafka message queue based on the Mysql data table;
if the Mysql data table has data change, the Mysql generates a Binlog corresponding to the data change; the data change comprises data creation or data update;
the Mysql sends a Bilog event corresponding to the Bilog to the first Kafka message queue.
4. The method of claim 3, wherein the synchronization system creates heterogeneous data based on the update data, further comprising:
the consumers of the first Kafka message queue monitor the first Kafka message queue in real time;
when the consumers of the first Kafka message queue monitor the Binlog event, the synchronization system analyzes the Binlog event to obtain an analysis result;
the synchronization system creates heterogeneous data based on the parsing result.
5. The method of claim 2, wherein the synchronization system writes the heterogeneous data to the corresponding ES index, comprising:
and writing the heterogeneous data into the corresponding ES index based on the query service requirement.
6. The method of claim 1, wherein the synchronization system sends each operation information of the ES to a compensation system, comprising:
the synchronization system generating an operation event based on each of the operation information of the ES;
the synchronization system sends the operational event to a second Kafka message queue of the compensation system.
7. The method of claim 6, wherein the compensation system determines a compensation target based on the operational information, comprising:
the consumers of the second Kafka message queue monitor the second Kafka message queue;
when the consumer of the second Kafka message queue monitors the operation event;
the compensation system stores the operation event into an ES operation information table of Mysql;
refluxing the ES operation information table to acquire reflux data;
the compensation system determines the compensation target based on the backflow data.
8. The method of claim 7, wherein the compensation system determines the compensation objective based on the backflow data, comprising:
the compensation system assembles the backflow data to obtain assembly data; wherein the assembly data is consistent with a structure of data of the ES;
the compensation system stores the assembly data into a reflux data table of the Mysql;
the compensation system determines the compensation target based on the backflow data table.
9. The method of claim 8, wherein the compensation system determines the compensation objective based on the backflow data, comprising:
presetting a time interval;
comparing the information of the ES operation data table with the information of the reflux table at preset time intervals;
if the information of the ES operation data table is inconsistent with the information of the reflux table, acquiring difference data;
the compensation system determines the compensation target based on the difference data.
10. The method of claim 9, wherein the compensation system, after determining the compensation objective based on the difference data, further comprises:
and the compensation system stores the compensation target to the ES index according to the query service requirement.
11. An apparatus for real-time synchronization of data, comprising:
a first creation module for creating a plurality of ES indexes based on a synchronization system;
a second creation module for the synchronization system to create heterogeneous data based on the update data;
a writing module, configured to write the heterogeneous data into the corresponding ES index by the synchronization system;
a sending module, configured to send, by the synchronization system, each operation information of the ES to a compensation system;
a compensation module for the compensation system to determine a compensation target based on the operational information.
12. An electronic device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, the processor implementing the method of any one of claims 1 to 10 when executing the computer program.
13. A computer-readable storage medium, comprising a stored computer program, wherein the computer program, when executed, controls an apparatus in which the computer-readable storage medium is located to perform the method of any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111618964.9A CN114281827A (en) | 2021-12-27 | 2021-12-27 | Method, device and equipment for real-time data synchronization and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111618964.9A CN114281827A (en) | 2021-12-27 | 2021-12-27 | Method, device and equipment for real-time data synchronization and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114281827A true CN114281827A (en) | 2022-04-05 |
Family
ID=80876576
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111618964.9A Pending CN114281827A (en) | 2021-12-27 | 2021-12-27 | Method, device and equipment for real-time data synchronization and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114281827A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115840788A (en) * | 2023-02-21 | 2023-03-24 | 创意信息技术股份有限公司 | Method, device, terminal and storage medium for synchronizing MySql data to ES |
| CN116048840A (en) * | 2022-12-30 | 2023-05-02 | 北京白驹易行科技有限公司 | Multi-tenant-based order ES high-speed writing method and system |
| CN116108110A (en) * | 2023-04-17 | 2023-05-12 | 南方电网数字电网研究院有限公司 | Kafka message consumption compensation method, system and medium |
-
2021
- 2021-12-27 CN CN202111618964.9A patent/CN114281827A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116048840A (en) * | 2022-12-30 | 2023-05-02 | 北京白驹易行科技有限公司 | Multi-tenant-based order ES high-speed writing method and system |
| CN115840788A (en) * | 2023-02-21 | 2023-03-24 | 创意信息技术股份有限公司 | Method, device, terminal and storage medium for synchronizing MySql data to ES |
| CN115840788B (en) * | 2023-02-21 | 2023-04-28 | 创意信息技术股份有限公司 | Method, device, terminal and storage medium for synchronizing MySql data to ES |
| CN116108110A (en) * | 2023-04-17 | 2023-05-12 | 南方电网数字电网研究院有限公司 | Kafka message consumption compensation method, system and medium |
| CN116108110B (en) * | 2023-04-17 | 2023-08-15 | 南方电网数字电网研究院有限公司 | Kafka message consumption compensation method, system and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114281827A (en) | Method, device and equipment for real-time data synchronization and storage medium | |
| US10360193B2 (en) | Method and apparatus for smart archiving and analytics | |
| CN107402963B (en) | Search data construction method, incremental data pushing device and equipment | |
| US10372492B2 (en) | Job-processing systems and methods with inferred dependencies between jobs | |
| US8805852B2 (en) | Automatic query execution plan management and performance stabilization for workloads | |
| CN111539633A (en) | Service data quality auditing method, system, device and storage medium | |
| US9311175B2 (en) | Method and system for processing log information | |
| US8266273B2 (en) | Realtime process history server | |
| CN103440290A (en) | Big data loading system and method | |
| JP6633642B2 (en) | Method and device for processing data blocks in a distributed database | |
| CN111522703A (en) | Method, apparatus and computer program product for monitoring access requests | |
| CN113779094B (en) | Batch-flow-integration-based data processing method and device, computer equipment and medium | |
| US20130086133A1 (en) | Method and apparatus for file revision tracking | |
| CN107357809A (en) | Magnanimity platform Hisoon real-time database metric data access system | |
| CN111930862A (en) | SQL interactive analysis method and system based on big data platform | |
| CN109947828B (en) | Method and device for processing report data | |
| CN111913937A (en) | Database operation and maintenance method and device | |
| CN115391457B (en) | Cross-database data synchronization method, device and storage medium | |
| CN116186175A (en) | Data warehouse system and unified model table creation method | |
| CN115033639A (en) | Method and related device for generating relation graph for data sharing among clusters | |
| Cui et al. | Monitoring and control of unstructured manufacturing big data | |
| CN111259082B (en) | Method for realizing full data synchronization in big data environment | |
| CN116860871B (en) | Metadata standardization method, device and storage medium for unified processing source data | |
| CN109359014A (en) | A kind of computer operation condition monitoring method, system and storage medium | |
| CN115599868B (en) | Data real-time synchronous processing method, system, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |