CN113792094A - Data synchronization system, method, device and medium - Google Patents
Data synchronization system, method, device and medium Download PDFInfo
- Publication number
- CN113792094A CN113792094A CN202110989209.5A CN202110989209A CN113792094A CN 113792094 A CN113792094 A CN 113792094A CN 202110989209 A CN202110989209 A CN 202110989209A CN 113792094 A CN113792094 A CN 113792094A
- Authority
- CN
- China
- Prior art keywords
- database
- module
- transaction data
- etl
- transaction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000004458 analytical method Methods 0.000 claims abstract description 28
- 238000001914 filtration Methods 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 6
- 230000007246 mechanism Effects 0.000 claims description 6
- 239000002253 acid Substances 0.000 claims description 5
- 238000004140 cleaning Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 5
- 230000001360 synchronised effect Effects 0.000 claims description 5
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure relates to a data synchronization system, method, device and medium, the system comprising: the system comprises a MySQL module, a log analysis module, a message receiving module and an ETL loading module; MySQL utilizes binlog to record the logic log file updated by the source database in real time; wherein, the logic log file is composed of a series of transaction events; the log analysis module analyzes the logic log file based on a preset analysis rule to obtain a transaction data stream; the message receiving module writes the transaction data streams from different source end databases into a cache file according to the designated cache path; and the ETL loading module reads the transaction data stream from the cache file, performs ETL processing on the transaction data stream to obtain an SQL statement in a general format, and loads the SQL statement into the target end database. The method and the device can efficiently realize the backup of the different libraries and better meet the complex data exchange requirement of a service system.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a data synchronization system, method, device, and medium.
Background
Most data synchronization products are mainly developed by large database manufacturers aiming at database products of the companies, and are not necessarily suitable for other database management systems. Taking MySQL database data synchronization as an example, MySQL master-slave replication clusters rely on the master server writing updates to a binary log file and maintaining an index of the file to track log cycles while sending these logs to the slave servers for update by the slave servers. Therefore, the database of the source end and the database of the target end are required to be MySQL databases to use the master-slave configuration scheme, which brings a problem to data exchange between service systems.
Disclosure of Invention
To solve the above technical problem or to at least partially solve the above technical problem, the present disclosure provides a data synchronization system, method, device, and medium.
The present disclosure provides a data synchronization system, comprising: the system comprises a relational database management system MySQL, a log analysis module, a message receiving module and an ETL loading module;
the MySQL utilizes binlog to record a logic log file updated by a source end database in real time; wherein the logical log file is composed of a series of transaction events;
the log analysis module analyzes the logic log file based on a preset analysis rule to obtain a transaction data stream;
the message receiving module writes the transaction data streams from different source end databases into a cache file according to a designated cache path;
and the ETL loading module reads the transaction data stream from the cache file, performs ETL processing on the transaction data stream to obtain an SQL statement in a general format, and loads the SQL statement into a target end database.
Optionally, the log analysis module performs data cleaning on the logic log file according to the ACID attribute of the transaction in the analysis process; wherein the data cleansing comprises: checking data consistency and processing invalid and missing values.
Optionally, the log analysis module filters the logic log file according to a preset filtering rule; wherein the filtering rules are used to select the logical log files that need to be synchronized.
Optionally, the ETL loading module obtains a mapping rule corresponding to the type of the target-side database by reading a preset configuration file; and carrying out ETL processing on the transaction processing data flow according to the mapping rule to obtain an SQL statement in a general format.
Optionally, the ETL loading module configures a Java database to connect JDBC; and the ETL loading module uniformly loads the SQL statements into the target end database in the JDBC mode.
Optionally, the binlog records data based on a line mode of a synchronization mechanism.
Optionally, the target-side database is a non-MySQL type database.
The present disclosure provides a data synchronization method, including:
the MySQL utilizes binlog to record a logic log file updated by a source end database in real time; wherein the logical log file is composed of a series of transaction events; analyzing the logic log file based on a preset analysis rule to obtain a transaction data stream; writing the transaction data streams from different source end databases into a cache file according to a designated cache path; and reading the transaction data stream from the cache file, carrying out ETL processing on the transaction data stream to obtain an SQL statement in a general format, and loading the SQL statement into a target end database.
The present disclosure provides an electronic device, the electronic device including: a processor; a memory for storing the processor-executable instructions; the processor is used for reading the executable instructions from the memory and executing the instructions to realize the method.
The present disclosure provides a computer-readable storage medium having stored thereon a computer program for executing the above-mentioned method.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:
the embodiment of the disclosure provides a data synchronization system, a method, a device and a medium, wherein the system comprises: the system comprises a MySQL module, a log analysis module, a message receiving module and an ETL loading module; MySQL utilizes binlog to record a logic log file with Event as a unit in real time; the log analysis module analyzes the logic log file based on a preset analysis rule to obtain a transaction data stream with higher integrity; the message receiving module caches transaction data flow; the ETL loading module can efficiently carry out ETL processing on the transaction data flow to obtain the SQL sentences in the general format, and load the SQL sentences into the target end database. According to the technical scheme, the analysis and transmission of the logic log file take Event as a unit, the transmission quantity of data is reduced, and a complex rollback mechanism is not required to be processed, so that the database data synchronization can be efficiently carried out based on the ETL under the condition of ensuring the consistency and the integrity, and the efficiency of the data synchronization among the databases is improved; SQL sentences analyzed by the binlog based on MySQL are uniformly loaded at a target end in a JDBC mode, so that the purpose of heterogeneous library backup is achieved, and the complex data exchange requirement of a service system is well met.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present disclosure, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic structural diagram of a data synchronization system according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of a log parsing module according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of a message receiving module according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of an ETL loading module according to an embodiment of the present disclosure;
fig. 5 is a flowchart of a data synchronization method according to an embodiment of the disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.
At present, most data synchronization products require that databases of a source end and a target end are MySQL databases to use the master-slave configuration scheme, which brings a problem to data exchange between service systems. In order to improve the above problem, embodiments of the present disclosure provide a data synchronization system, method, device, and medium. The data synchronization system is used for realizing an ETL (Extract-Transform-Load, data warehouse technology)) data synchronization method based on MySQL database operation log online analysis, and under the condition of ensuring data consistency and integrity, the ETL is used for efficiently synchronizing the database data through the log online analysis technology, so that the purpose of different-library backup is achieved, and the problem of complex data exchange requirements of a service system is well met. For ease of understanding, the embodiments of the present disclosure are described in detail below.
Referring to fig. 1, a schematic structural diagram of a data synchronization system is shown, which includes: MySQL100 (relational database management system), a log parsing module 200, a message receiving module 300, and an ETL loading module 400.
In this embodiment, MySQL100 uses binlog (binary log) to record the logical log file updated by the source database in real time. Wherein, the logic log file is composed of a series of events; the source database is a MySQL100 type database.
The binlog in this embodiment records data based on the line mode of the synchronization mechanism. During specific implementation, a binlog recording function of MySQL100 is started and modified into a row (row) mode, and the binlog updates data based on the row mode of a synchronization mechanism and reflects data change of a row level in real time.
The log parsing module 200 receives the logic log file updated by the binlog record, and then parses the logic log file based on a preset parsing rule to obtain a transaction data stream. In this embodiment, the log parsing module 200 parses formats of the logical log file, such as a database name, a table name, an event type, a timestamp, an event number, a field name, and a corresponding field value, according to a fixed byte reading manner and a parsing rule, obtains a data set of an operation instruction and an operation result of a source database by a user, and determines a transaction data stream based on the obtained data set.
The log parsing module 200 communicates the resulting transaction data stream to the message receiving module 300. The message receiving module 300 writes the transaction data streams from different source databases into the cache file according to the designated cache path.
The ETL loading module 400 reads the transaction data stream from the cache file, performs ETL processing on the transaction data stream to obtain an SQL statement in a general format, and loads the SQL statement into the target end database. The target-side database is a non-MySQL 100-type database, such as a database that may include: NoSQL database of HBase, Kylin, or ES.
For ease of understanding, the following embodiments describe the data synchronization system described above in detail.
And starting a binlog recording function of the MySQL100, modifying the binlog recording function into a row mode, and recording the updated logic log file of the source database in real time by the MySQL100 by using the binlog. In the row mode, only data records with changes (such as modification or update) and data records after changes need to be recorded in the binlog, so that the log content of the row can clearly record the details of each data change, and is easy to understand; and the problems that the storage process and function under certain specific conditions and the calling and starting of trigger cannot be copied correctly do not occur.
Referring to fig. 2, the log parsing module 200 reads the following configuration rules: analyzing the rule and the filtering rule, analyzing the logic log file according to the read configuration rule, and then transmitting the transaction data stream obtained after analysis to the message receiving module 300 of the target end.
In order to improve the data quality of the transaction data stream obtained by the log parsing module 200, in an embodiment, the log parsing module 200 may perform data cleaning on the logic log file according to the ACID attribute of the transaction during parsing the logic log file according to a preset parsing rule. In general, the ACID properties of a transaction are atomicity, consistency, isolation, and durability, respectively.
In implementation, the log parsing module 200 may add a transaction integrity check point, and perform data cleaning on the logical log file through the transaction integrity check point and according to the ACID attribute of the transaction; the data cleansing may include: checking data consistency and processing invalid and missing values. In the embodiment, a check point of transaction integrity is added in the analysis process, and under the conditions of power failure and restarting of a failed database, reading of dirty data and repeated data is realized through data cleaning, repeated transactions and data with incomplete transactions are actively filtered, and the transaction integrity of business data is guaranteed.
In another embodiment, before parsing the logic log file, the log parsing module 200 may further filter the logic log file according to a preset filtering rule; wherein the filtering rules are used to select logical log files that need to be synchronized. By configuring the filtering rules, the user can pertinently select the logic log files to be synchronized and filter the logic log files which do not need to be sent, so as to reduce the data volume to be analyzed and further reduce the pressure of the data volume to be sent on the network.
The message receiving module 300 writes the transaction data streams from different source databases into the cache file according to the designated cache path.
Referring to fig. 3, the message receiving module 300 receives transaction data streams from different source databases, and writes the transaction data streams into a local cache file for data loading according to a specified cache path. The embodiment stores the received transaction data stream in a cache manner to reduce network pressure, accelerate network sending and receiving response time, reduce the influence of database blockage at the target end on data reception, and fundamentally solve the problem of data blockage between the source end and the target end caused by synchronous database data storage.
Referring to fig. 4, in an implementation manner of performing ETL processing on a transaction data stream, an ETL loading module 400 obtains a mapping rule corresponding to a type of a target-side database by reading a preset configuration file; and carrying out ETL processing on the transaction processing data flow according to the mapping rule to obtain the SQL statement in the general format. In this embodiment, the ETL loading module 400 performs online analysis on the transaction data stream, and processes the transaction data stream into a standard SQL statement in a universal format, so as to achieve a data backup function between heterogeneous databases.
The ETL loading module 400 configures JDBC (Java Database Connectivity); the ETL loading module 400 uniformly loads SQL statements into the target-side database in JDBC, and finishes loading data in batch.
In summary, the data synchronization system provided in the embodiment of the present disclosure includes: the system comprises a MySQL module, a log analysis module, a message receiving module and an ETL loading module; MySQL utilizes binlog to record a logic log file with Event as a unit in real time; the log analysis module analyzes the logic log file based on a preset analysis rule to obtain a transaction data stream with higher integrity; the message receiving module caches transaction data flow; the ETL loading module can efficiently carry out ETL processing on the transaction data flow to obtain the SQL sentences in the general format, and load the SQL sentences into the target end database. According to the technical scheme, the analysis and transmission of the logic log file take Event as a unit, the transmission quantity of data is reduced, and a complex rollback mechanism is not required to be processed, so that the database data synchronization can be efficiently carried out based on the ETL under the condition of ensuring the consistency and the integrity, and the efficiency of the data synchronization among the databases is improved; SQL sentences analyzed by the binlog based on MySQL are uniformly loaded at a target end in a JDBC mode, so that the purpose of heterogeneous library backup is achieved, and the complex data exchange requirement of a service system is well met.
The data synchronization system provided in the above embodiment is used to implement the data synchronization method shown in fig. 5, and the method may include the following steps:
step S502, MySQL uses binlog to record the logic log file updated by the source database in real time; wherein, the logic log file is composed of a series of transaction events;
step S504, analyzing the logic log file based on a preset analysis rule to obtain a transaction data stream;
step S506, writing the transaction data streams from different source end databases into a cache file according to the designated cache path;
step S508, reading the transaction data stream from the cache file, performing ETL processing on the transaction data stream to obtain an SQL statement in a generic format, and loading the SQL statement into the target end database.
The method provided by the embodiment has the same implementation principle and technical effect as the system embodiment, and for the sake of brief description, reference may be made to the corresponding content in the system embodiment for the part where the method embodiment is not mentioned.
Further, this embodiment also provides an electronic device, which includes: a processor; a memory for storing the processor-executable instructions; and the processor is used for reading the executable instruction from the memory and executing the instruction to realize the data synchronization method.
Further, the present embodiment also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used for executing the data synchronization method.
The computer program product of the data synchronization system, the data synchronization method, the data synchronization device, and the data synchronization medium provided in the embodiments of the present disclosure includes a computer readable storage medium storing a program code, where instructions included in the program code may be used to execute the method described in the foregoing method embodiments, and specific implementations may refer to embodiments of the system and the method, and are not described herein again.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A data synchronization system, comprising: the system comprises a relational database management system MySQL, a log analysis module, a message receiving module and an ETL loading module;
the MySQL utilizes binlog to record a logic log file updated by a source end database in real time; wherein the logical log file is composed of a series of transaction events;
the log analysis module analyzes the logic log file based on a preset analysis rule to obtain a transaction data stream;
the message receiving module writes the transaction data streams from different source end databases into a cache file according to a designated cache path;
and the ETL loading module reads the transaction data stream from the cache file, performs ETL processing on the transaction data stream to obtain an SQL statement in a general format, and loads the SQL statement into a target end database.
2. The system of claim 1, wherein the log parsing module performs data cleaning on the logical log file according to ACID attributes of transactions during parsing; wherein the data cleansing comprises: checking data consistency and processing invalid and missing values.
3. The system of claim 1, wherein the log parsing module filters the logical log file according to a preset filtering rule; wherein the filtering rules are used to select the logical log files that need to be synchronized.
4. The system according to claim 1, wherein the ETL loading module obtains the mapping rule corresponding to the type of the target-side database by reading a preset configuration file; and carrying out ETL processing on the transaction processing data flow according to the mapping rule to obtain an SQL statement in a general format.
5. The system of claim 1, wherein the ETL loading module configures Java database connectivity JDBC;
and the ETL loading module uniformly loads the SQL statements into the target end database in the JDBC mode.
6. The system of claim 1, wherein the binlog performs data logging based on a row mode of a synchronization mechanism.
7. The system according to claim 1, wherein the target-side database is a non-MySQL type database.
8. A method of data synchronization, comprising:
the MySQL utilizes binlog to record a logic log file updated by a source end database in real time; wherein the logical log file is composed of a series of transaction events;
analyzing the logic log file based on a preset analysis rule to obtain a transaction data stream;
writing the transaction data streams from different source end databases into a cache file according to a designated cache path;
and reading the transaction data stream from the cache file, carrying out ETL processing on the transaction data stream to obtain an SQL statement in a general format, and loading the SQL statement into a target end database.
9. An electronic device, characterized in that the electronic device comprises:
a processor;
a memory for storing the processor-executable instructions;
the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of claim 8.
10. A computer-readable storage medium, characterized in that the storage medium stores a computer program for performing the method as claimed in claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110989209.5A CN113792094A (en) | 2021-08-26 | 2021-08-26 | Data synchronization system, method, device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110989209.5A CN113792094A (en) | 2021-08-26 | 2021-08-26 | Data synchronization system, method, device and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113792094A true CN113792094A (en) | 2021-12-14 |
Family
ID=78876446
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110989209.5A Pending CN113792094A (en) | 2021-08-26 | 2021-08-26 | Data synchronization system, method, device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113792094A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114528266A (en) * | 2022-01-28 | 2022-05-24 | 苏州浪潮智能科技有限公司 | Method, system, device and storage medium for implementing database and cache consistency |
| CN114880346A (en) * | 2022-04-22 | 2022-08-09 | 苏州浪潮智能科技有限公司 | Data processing method, related assembly and accelerated processor |
| CN115525680A (en) * | 2022-09-21 | 2022-12-27 | 京信数据科技有限公司 | Data processing job scheduling method and device, computer equipment and storage medium |
| CN116578655A (en) * | 2023-07-06 | 2023-08-11 | 舟谱数据技术南京有限公司 | Data transmission system and control method thereof |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106709043A (en) * | 2016-12-30 | 2017-05-24 | 江苏瑞中数据股份有限公司 | Data synchronous loading method based on database log |
| CN108228756A (en) * | 2017-12-21 | 2018-06-29 | 江苏瑞中数据股份有限公司 | Data based on the PG databases of daily record analytic technique to Hadoop platform synchronize clone method |
| CN108228755A (en) * | 2017-12-21 | 2018-06-29 | 江苏瑞中数据股份有限公司 | The data of MySQL database based on daily record analytic technique to Hadoop platform synchronize clone method |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
-
2021
- 2021-08-26 CN CN202110989209.5A patent/CN113792094A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106709043A (en) * | 2016-12-30 | 2017-05-24 | 江苏瑞中数据股份有限公司 | Data synchronous loading method based on database log |
| CN108228756A (en) * | 2017-12-21 | 2018-06-29 | 江苏瑞中数据股份有限公司 | Data based on the PG databases of daily record analytic technique to Hadoop platform synchronize clone method |
| CN108228755A (en) * | 2017-12-21 | 2018-06-29 | 江苏瑞中数据股份有限公司 | The data of MySQL database based on daily record analytic technique to Hadoop platform synchronize clone method |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114528266A (en) * | 2022-01-28 | 2022-05-24 | 苏州浪潮智能科技有限公司 | Method, system, device and storage medium for implementing database and cache consistency |
| CN114528266B (en) * | 2022-01-28 | 2024-01-26 | 苏州浪潮智能科技有限公司 | Method, system, device and storage medium for realizing database and cache consistency |
| CN114880346A (en) * | 2022-04-22 | 2022-08-09 | 苏州浪潮智能科技有限公司 | Data processing method, related assembly and accelerated processor |
| CN114880346B (en) * | 2022-04-22 | 2024-01-23 | 苏州浪潮智能科技有限公司 | Data processing method, related assembly and acceleration processor |
| CN115525680A (en) * | 2022-09-21 | 2022-12-27 | 京信数据科技有限公司 | Data processing job scheduling method and device, computer equipment and storage medium |
| CN116578655A (en) * | 2023-07-06 | 2023-08-11 | 舟谱数据技术南京有限公司 | Data transmission system and control method thereof |
| CN116578655B (en) * | 2023-07-06 | 2023-09-15 | 舟谱数据技术南京有限公司 | Data transmission system and control method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113792094A (en) | Data synchronization system, method, device and medium | |
| US8108343B2 (en) | De-duplication and completeness in multi-log based replication | |
| AU2016250260B2 (en) | Backup and restore in a distributed database utilizing consistent database snapshots | |
| US11256715B2 (en) | Data backup method and apparatus | |
| CN107391628B (en) | Data synchronization method and device | |
| CN107402963B (en) | Search data construction method, incremental data pushing device and equipment | |
| US10671642B2 (en) | Copying data changes to a target database | |
| US11604597B2 (en) | Data processing method and apparatus | |
| WO2022063284A1 (en) | Data synchronization method and apparatus, device, and computer-readable medium | |
| US8572022B2 (en) | Automatic synchronization conflict resolution | |
| US7702698B1 (en) | Database replication across different database platforms | |
| US8924365B2 (en) | System and method for range search over distributive storage systems | |
| CN110209735B (en) | Database backup method, database backup device, computing device, and storage medium | |
| CN109710388A (en) | Method for reading data, device, electronic equipment and storage medium | |
| CN110765206A (en) | Data synchronization system, method, device, equipment and storage medium | |
| CN113868028A (en) | Method for replaying log on data node, data node and system | |
| CN109446262B (en) | Data aggregation method and device | |
| CN111930788B (en) | Processing method, device and equipment of operation request, readable storage medium and system | |
| CN111737331B (en) | Transaction consistency processing method and system for database and object storage | |
| CN113760902A (en) | Data splitting method, device, equipment, medium and program product | |
| CN112612647B (en) | Log parallel replay method, device, equipment and storage medium | |
| CN112685431B (en) | Asynchronous caching method, device, system, electronic equipment and storage medium | |
| CN112084161B (en) | Database-based data processing method and device and readable storage medium | |
| CN114116732B (en) | Transaction processing method and device, storage device and server | |
| Donselaar | Low latency asynchronous database synchronization and data transformation using the replication log. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |