CN115168307A - Data synchronization method, system, equipment and storage medium supporting breakpoint continuous transmission - Google Patents
Data synchronization method, system, equipment and storage medium supporting breakpoint continuous transmission Download PDFInfo
- Publication number
- CN115168307A CN115168307A CN202210869514.5A CN202210869514A CN115168307A CN 115168307 A CN115168307 A CN 115168307A CN 202210869514 A CN202210869514 A CN 202210869514A CN 115168307 A CN115168307 A CN 115168307A
- Authority
- CN
- China
- Prior art keywords
- data
- nodes
- node
- target database
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a data synchronization method, a system, equipment and a storage medium for supporting breakpoint continuous transmission, wherein the technical scheme is as follows: acquiring log file data of a source database; establishing a table according to the log file data; writing corresponding data nodes in the data of the table according to the data characteristics of the data in the table; transmitting the data written with the data nodes to a target database, judging whether information fed back by the target database and receiving the data nodes is received or not under the condition that the data nodes are transmitted in the process of transmitting the data written with the data nodes to the target database, and storing the position information and the corresponding storage time of the data nodes under the condition that the information fed back by the target database is received; the method and the device have the advantages that after the breakpoint, the data which are not successfully transmitted can be rapidly transmitted again, the data which are transmitted before are prevented from being synchronized again, and the efficiency of data synchronization is greatly improved.
Description
Technical Field
The present invention relates to the field of data transmission technologies, and in particular, to a data synchronization method, system, device and storage medium supporting breakpoint continuous transmission.
Background
The existing big data platform usually adopts off-line data synchronization, scenes needing historical data recalculation exist in each business system, the quantity of the historical data is huge, if the historical data is still resynchronized once in an off-line mode, a large amount of computing resources are occupied, the data synchronization range is difficult to define, certain errors exist between the historical data stored in a data warehouse and actual business data, the data accuracy cannot support business requirements under certain scenes, and in addition, due to other requirements on real-time performance such as business or large screens, the requirement on the real-time performance of the data is provided for the platform.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a data synchronization method, a system, equipment and a storage medium supporting breakpoint continuous transmission, which have the functional advantages that after a breakpoint, data which is not successfully transmitted can be rapidly retransmitted, and the data which is transmitted before is prevented from being resynchronized.
The technical purpose of the invention is realized by the following technical scheme:
a data synchronization method supporting breakpoint continuous transmission comprises the following steps:
acquiring log file data of a source database;
establishing a table according to the log file data;
writing corresponding data nodes in the data of the table according to the data characteristics of the data in the table;
transmitting data written with data nodes to a target database, judging whether information fed back by the target database and receiving the data nodes is received or not under the condition that the data nodes are transmitted in the process of transmitting the data written with the data nodes to the target database, and storing the position information and the corresponding storage time of the data nodes under the condition that the information fed back by the target database is received;
receiving a breakpoint continuous transmission request, and determining the request time of the breakpoint continuous transmission request;
determining a target data node corresponding to the storage time closest to the time point of the target data node according to the request time;
and determining data which is not transmitted to the target database according to the target data node and the data written with the data node, and continuously transmitting the data to the target database.
Optionally, the data characteristics include: data peaks and data troughs; the writing of the corresponding data node in the data of the table according to the data characteristics of the data in the table includes:
judging whether the data in the table has a data wave crest or not, and writing a corresponding data node at the data wave crest under the condition that the data wave crest exists;
and judging whether the data in the table has data troughs or not, and writing corresponding data nodes in the data troughs under the condition that the data troughs exist.
Optionally, the data feature further includes: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the writing of the corresponding data node in the data of the table according to the data characteristics of the data in the table further comprises:
judging whether the data consumption time required by the data between the adjacent data nodes in the table is larger than a preset time threshold, calculating to obtain the data length according to the preset time threshold and the real-time network transmission speed under the condition that the data consumption time required by the data between the adjacent data nodes is larger than the preset time threshold, adding the data length to the previous data node in the adjacent data nodes to obtain a data writing point, and writing the data writing point into the corresponding data node.
Optionally, the creating a table according to the log file data includes:
analyzing the log file data to obtain a database name, a table name, an operation type, a main key and all field values;
and establishing a table according to the database name, the table name, the operation type, the main key and all field values.
Optionally, after the creating the table according to the log file data, the method further includes:
dividing data in the table into a plurality of data rows according to the primary key;
and dividing each data line into a plurality of data blocks according to the preset configuration.
Optionally, the transmitting the data written with the data node to the target database includes:
sequentially reading all data blocks in each data row according to a time stamp, wherein the time stamp is acquired from log file data;
under the condition that the front end of the data block is read, marking a first binlog position at the front end of the data block, wherein the first binlog position is position information of the front end of the data block acquired from log file data;
under the condition that the rear end of the data block is read, marking a second binlog position at the rear end of the data block, wherein the second binlog position is position information of the rear end of the data block acquired from log file data;
and transmitting the data block marked with the first binlog position and the second binlog position to a target data end.
Optionally, the method further includes:
when data change exists in a source database in the process of transferring data written with a data node to a target database, determining a data row corresponding to the changed data according to a primary key corresponding to the changed data, comparing the position of the changed data with all first binlog positions and all second binlog positions of the data row to determine a data block corresponding to the changed data, and changing the data of the data block corresponding to the changed data according to the primary key corresponding to the changed data.
A data synchronization system that supports breakpoint resuming, comprising:
the data acquisition module is used for acquiring log file data of a source database;
the data tabulation module is used for tabulating according to the log file data;
the node writing module is used for writing corresponding data nodes in the data of the table according to the data characteristics of the data in the table;
the transmission judging module is used for transmitting the data written with the data nodes to a target database, judging whether information fed back by the target database and receiving the data nodes is received or not under the condition that the data nodes are transmitted in the process of transmitting the data written with the data nodes to the target database, and storing the position information of the data nodes and the corresponding storage time under the condition that the information fed back by the target database is received;
the receiving request module is used for receiving a breakpoint resume request and determining the request time of the breakpoint resume request;
the node determining module is used for determining a target data node corresponding to the storage time closest to the time point according to the request time;
and the data continuous transmission module is used for determining the data which is not transmitted to the target database according to the target data node and the data written with the data node, and continuously transmitting the data to the target database.
A computer device comprising a memory storing a computer program and a processor implementing the steps of the method described above when executing the computer program.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method.
In conclusion, the invention has the following beneficial effects: after the breakpoint, the data which is not successfully transmitted can be rapidly retransmitted, so that the data which is transmitted before is prevented from being resynchronized, and the data synchronization efficiency is greatly improved; compared with the division of data in fixed sections, the data transmission method has the advantages that the data are divided according to the characteristics of the data, the data can be dynamically determined by taking the number of data as a unit for processing, and the method is more suitable for real-time synchronous transmission of the data.
Drawings
FIG. 1 is a schematic flow chart of a data synchronization method supporting breakpoint continuous transmission according to the present invention;
FIG. 2 is a block diagram of a data synchronization system supporting breakpoint resuming provided in the present invention;
fig. 3 is an internal structural diagram of a computer device in the embodiment of the present invention.
Detailed Description
In order to make the objects, features and advantages of the present invention more comprehensible, embodiments accompanying figures are described in detail below. Several embodiments of the invention are presented in the drawings. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.
In the present invention, unless explicitly specified or limited otherwise, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.
The invention is described in detail below with reference to the figures and examples.
The invention provides a data synchronization method supporting breakpoint continuous transmission, as shown in fig. 1, including:
In practical applications, steps 100 to 400 are normal transmission steps executed under the condition that data transmission is normal, that is, data transmission is not interrupted due to a failure such as network interruption or power failure, and after a data transmission interruption (that is, a breakpoint) occurs and then a data transmission task is restarted, that is, a breakpoint resume request is issued, steps 500 to 700 are executed. According to the data synchronization method supporting breakpoint continuous transmission, after the breakpoint, the data which are not successfully transmitted can be rapidly retransmitted, the data which are transmitted before are prevented from being resynchronized, and the data synchronization efficiency is greatly improved.
Further, the data features include: data peaks and data troughs; the writing of the corresponding data node in the data of the table according to the data characteristics of the data in the table includes:
judging whether the data in the table has a data wave crest or not, and writing a corresponding data node at the data wave crest under the condition that the data wave crest exists; specifically, in the case of a plurality of data peaks, writing a data node corresponding thereto at each data peak;
judging whether the data in the table has data wave troughs or not, and writing corresponding data nodes in the data wave troughs under the condition that the data wave troughs exist; specifically, in the case where there are a plurality of data troughs, the data node corresponding thereto is written at each data trough.
In practical application, compared with the division of data in fixed segmentation, the data is divided according to the characteristics of the data by writing the corresponding data nodes at the data wave crest and the data wave trough, so that the method is more suitable for real-time synchronous transmission of the data.
Further, the data features further include: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the writing of the corresponding data node in the data of the table according to the data characteristics of the data in the table further comprises:
judging whether the data consumption time required by the data between the adjacent data nodes in the table is larger than a preset time threshold, wherein the preset time threshold represents a time period, and under the condition that the data consumption time required by the data between the adjacent data nodes is larger than the preset time threshold, namely, the data consumption time between the adjacent data nodes in the data nodes determined according to the data wave crests and the data wave troughs is too long, if a data breakpoint occurs between the adjacent data nodes, efficient data synchronization is not facilitated, therefore, the data length is obtained by calculating according to the preset time threshold and the real-time network transmission speed, the data writing point is obtained by adding the data length to the previous data node in the adjacent data nodes, and the corresponding data node is written in the data writing point.
Specifically, the data consumption time is determined according to the real-time network transmission speed, and then the data transmission time between adjacent data nodes is controlled within the preset time threshold according to the data consumption time and the preset time threshold, so that the data transmission time can be dynamically determined by taking how much data as a unit for processing, and after a data transmission breakpoint occurs, the data transmitted before is not required to be retransmitted from a target data node for too long time, and the data synchronization efficiency is further improved.
Further, the building a table according to the log file data includes:
analyzing the log file data to obtain a database name, a table name, an operation type, a main key and all field values;
and building a table according to the database name, the table name, the operation type, the main key and all field values.
Further, after the creating a table according to the log file data, the method further includes:
dividing data in the table into a plurality of data rows according to the primary key;
and dividing each data line into a plurality of data blocks according to the preset configuration.
In practical application, the data in the table is divided into a plurality of data rows according to the main key, in the process of data synchronization, the data in the plurality of data rows can be transmitted simultaneously, multiple concurrent transmission of the data is realized, and therefore the effect of data synchronous transmission can be further improved.
Further, the transmitting the data written with the data node to the target database includes:
sequentially reading all data blocks in each data row according to a time stamp, wherein the time stamp is acquired from log file data;
under the condition that the front end of the data block is read, marking a first binlog position at the front end of the data block, wherein the first binlog position is position information of the front end of the data block acquired from log file data;
under the condition that the rear end of the data block is read, marking a second binlog position at the rear end of the data block, wherein the second binlog position is position information of the rear end of the data block acquired from log file data;
and transmitting the data block marked with the first binlog position and the second binlog position to a target data end.
Further, still include:
under the condition that the source database has data change in the process of transmitting the data written with the data nodes to the target database, the data change comprises the following steps: the method comprises the steps of data adding, data deleting and data modifying, wherein a data row corresponding to changed data is determined according to a main key corresponding to the changed data, the position of the changed data is compared with all first binlog positions and all second binlog positions of the data row to obtain a comparison result, a data block corresponding to the changed data is determined according to the comparison result, under the condition that the position of the changed data is located between a first binlog position and a second binlog position corresponding to the first binlog position according to the comparison result, the data block corresponding to the changed data is determined to be the first binlog position and the data block corresponding to the first binlog position, the data of the data block corresponding to the changed data is changed according to the main key corresponding to the changed data, and if the changed data is added, deleted or modified, corresponding adding, deleting or modifying operation is carried out on the data in the corresponding database.
Through the data block, the mark of the first binlog position and the mark of the second binlog position, the position searching speed of the changed data in the table is accelerated, the real-time updating of the data can be realized, and the data synchronization efficiency is improved.
According to the data synchronization method supporting breakpoint continuous transmission, after the breakpoint, the data which is not successfully transmitted can be rapidly retransmitted, so that the data which is transmitted before is prevented from being retransmitted, and the data synchronization efficiency is greatly improved; compared with the division of data in fixed sections, the data consumption time is determined by writing corresponding data nodes at data wave crests and data wave troughs and by the real-time network transmission speed, and then the data transmission time between adjacent data nodes is controlled within the preset time threshold according to the data consumption time and the preset time threshold, so that the data are divided according to the characteristics of the data, the data can be dynamically determined by taking the data as a unit for processing, and the method is more suitable for real-time synchronous transmission of the data.
As shown in fig. 2, the present invention further provides a system for building and distributing an installation package of an application, including:
the data acquisition module 10 is configured to acquire log file data of a source database;
a data table building module 20, configured to build a table according to the log file data;
a node writing module 30, configured to write a corresponding data node in the data in the table according to the data characteristics of the data in the table;
the transmission judging module 40 is configured to transmit the data written with the data node to a target database, judge whether to receive information of the data node received by feedback from the target database when the data written with the data node is transmitted to the target database, and store the location information of the data node and corresponding storage time when the information of the data node received by feedback from the target database is received;
a receiving request module 50, configured to receive a breakpoint resume request and determine a request time of the breakpoint resume request;
a node determining module 60, configured to determine, according to the request time, a target data node corresponding to a storage time closest to the time point of the request time;
and a data continuous transmission module 70, configured to determine, according to the target data node and the data written with the data node, data that is not transmitted to the target database, and continuously transmit the data to the target database.
For specific limitations of the building and distributing system of the installation package of the application, reference may be made to the above limitations of the building and distributing method of the installation package of the application, which are not described herein again. The modules of the installation package building and distribution system of the application can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and the internal structure thereof may be as shown in fig. 3. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of building and distributing an installation package of an application.
Those skilled in the art will appreciate that the architecture shown in fig. 3 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, there is provided a computer device comprising a memory storing a computer program and a processor implementing the following steps when the processor executes the computer program:
acquiring log file data of a source database;
establishing a table according to the log file data;
writing corresponding data nodes in the data of the table according to the data characteristics of the data in the table;
transmitting data written with data nodes to a target database, judging whether information fed back by the target database and receiving the data nodes is received or not under the condition that the data nodes are transmitted in the process of transmitting the data written with the data nodes to the target database, and storing the position information and the corresponding storage time of the data nodes under the condition that the information fed back by the target database is received;
receiving a breakpoint continuous transmission request, and determining the request time of the breakpoint continuous transmission request;
determining a target data node corresponding to the storage time closest to the time point according to the request time;
and determining data which is not transmitted to the target database according to the target data node and the data written with the data node, and continuously transmitting the data to the target database.
In one embodiment, the data features include: data peaks and data troughs; the writing of the corresponding data node in the data of the table according to the data characteristics of the data in the table includes:
judging whether the data in the table has a data wave crest or not, and writing a corresponding data node at the data wave crest under the condition that the data wave crest exists;
and judging whether the data in the table has data troughs or not, and writing corresponding data nodes in the data troughs under the condition that the data troughs exist.
In one embodiment, the data feature further comprises: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the writing of the corresponding data node in the data of the table according to the data characteristics of the data in the table further comprises:
judging whether the data consumption time required by the data between the adjacent data nodes in the table is larger than a preset time threshold, calculating to obtain the data length according to the preset time threshold and the real-time network transmission speed under the condition that the data consumption time required by the data between the adjacent data nodes is larger than the preset time threshold, adding the data length to the previous data node in the adjacent data nodes to obtain a data writing point, and writing the data writing point into the corresponding data node.
In one embodiment, said building a table from said log file data comprises:
analyzing the log file data to obtain a database name, a table name, an operation type, a main key and all field values;
and building a table according to the database name, the table name, the operation type, the main key and all field values.
In one embodiment, after said building a table according to said log file data, further comprising:
dividing data in the table into a plurality of data rows according to the primary key;
and dividing each data line into a plurality of data blocks according to the preset configuration.
In one embodiment, the transmitting the data written with the data node to the target database includes:
sequentially reading all data blocks in each data line according to a time stamp, wherein the time stamp is obtained from log file data;
under the condition that the front end of the data block is read, marking a first binlog position at the front end of the data block, wherein the first binlog position is position information of the front end of the data block acquired from log file data;
under the condition that the rear end of the data block is read, marking a second binlog position at the rear end of the data block, wherein the second binlog position is position information of the rear end of the data block acquired from log file data;
and transmitting the data block marked with the first binlog position and the second binlog position to a target data end.
In one embodiment, further comprising:
when data change exists in a source database in the process of transferring data written with a data node to a target database, determining a data row corresponding to the changed data according to a primary key corresponding to the changed data, comparing the position of the changed data with all first binlog positions and all second binlog positions of the data row to determine a data block corresponding to the changed data, and changing the data of the data block corresponding to the changed data according to the primary key corresponding to the changed data.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, performs the steps of:
acquiring log file data of a source database;
establishing a table according to the log file data;
writing corresponding data nodes in the data of the table according to the data characteristics of the data in the table;
transmitting data written with data nodes to a target database, judging whether information fed back by the target database and receiving the data nodes is received or not under the condition that the data nodes are transmitted in the process of transmitting the data written with the data nodes to the target database, and storing the position information and the corresponding storage time of the data nodes under the condition that the information fed back by the target database is received;
receiving a breakpoint continuous transmission request, and determining the request time of the breakpoint continuous transmission request;
determining a target data node corresponding to the storage time closest to the time point according to the request time;
and determining data which is not transmitted to the target database according to the target data node and the data written with the data node, and continuously transmitting the data to the target database.
In one embodiment, the data features include: data peaks and data troughs; the writing of the corresponding data node in the data of the table according to the data characteristics of the data in the table includes:
judging whether the data in the table has a data wave crest or not, and writing a corresponding data node at the data wave crest under the condition that the data wave crest exists;
and judging whether the data in the table has data troughs or not, and writing corresponding data nodes in the data troughs under the condition that the data troughs exist.
In one embodiment, the data feature further comprises: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the writing of the corresponding data node in the data of the table according to the data characteristics of the data in the table further comprises:
judging whether the data consumption time required by the data between the adjacent data nodes in the table is larger than a preset time threshold, calculating to obtain the data length according to the preset time threshold and the real-time network transmission speed under the condition that the data consumption time required by the data between the adjacent data nodes is larger than the preset time threshold, adding the data length to the previous data node in the adjacent data nodes to obtain a data writing point, and writing the data writing point into the corresponding data node.
In one embodiment, said building a table from said log file data comprises:
analyzing the log file data to obtain a database name, a table name, an operation type, a main key and all field values;
and building a table according to the database name, the table name, the operation type, the main key and all field values.
In one embodiment, after said building a table according to said log file data, further comprising:
dividing data in the table into a plurality of data rows according to the primary key;
and dividing each data line into a plurality of data blocks according to the preset configuration.
In one embodiment, the transmitting the data written with the data node to the target database includes:
sequentially reading all data blocks in each data row according to a time stamp, wherein the time stamp is acquired from log file data;
under the condition that the front end of the data block is read, marking a first binlog position at the front end of the data block, wherein the first binlog position is position information of the front end of the data block acquired from log file data;
under the condition that the rear end of the data block is read, marking a second binlog position at the rear end of the data block, wherein the second binlog position is position information of the rear end of the data block acquired from log file data;
and transmitting the data block marked with the first binlog position and the second binlog position to a target data end.
In one embodiment, further comprising:
when data change exists in a source database in the process of transferring data written with a data node to a target database, determining a data row corresponding to the changed data according to a primary key corresponding to the changed data, comparing the position of the changed data with all first binlog positions and all second binlog positions of the data row to determine a data block corresponding to the changed data, and changing the data of the data block corresponding to the changed data according to the primary key corresponding to the changed data.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct Rambus Dynamic RAM (DRDRAM), and Rambus Dynamic RAM (RDRAM), among others.
All possible combinations of the technical features in the above embodiments may not be described for the sake of brevity, but should be considered as being within the scope of the present disclosure as long as there is no contradiction between the combinations of the technical features.
Claims (10)
1. A data synchronization method supporting breakpoint continuous transmission is characterized by comprising the following steps:
acquiring log file data of a source database;
establishing a table according to the log file data;
writing corresponding data nodes in the data of the table according to the data characteristics of the data in the table;
transmitting data written with data nodes to a target database, judging whether information fed back by the target database and receiving the data nodes is received or not under the condition that the data nodes are transmitted in the process of transmitting the data written with the data nodes to the target database, and storing the position information and the corresponding storage time of the data nodes under the condition that the information fed back by the target database is received;
receiving a breakpoint continuous transmission request, and determining the request time of the breakpoint continuous transmission request;
determining a target data node corresponding to the storage time closest to the time point according to the request time;
and determining data which is not transmitted to the target database according to the target data node and the data written with the data node, and continuously transmitting the data to the target database.
2. The data synchronization method supporting breakpoint resume according to claim 1, wherein the data characteristics include: data peaks and data troughs; the writing of the corresponding data node in the data of the table according to the data characteristics of the data in the table includes:
judging whether the data in the table has a data wave crest or not, and writing a corresponding data node at the data wave crest under the condition that the data wave crest exists;
and judging whether the data in the table has data troughs or not, and writing corresponding data nodes in the data troughs under the condition that the data troughs exist.
3. The data synchronization method supporting breakpoint resume according to claim 2, wherein the data characteristics further include: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the writing of the corresponding data node in the data of the table according to the data characteristics of the data in the table further comprises:
judging whether the data consumption time required by the data between the adjacent data nodes in the table is larger than a preset time threshold, calculating to obtain the data length according to the preset time threshold and the real-time network transmission speed under the condition that the data consumption time required by the data between the adjacent data nodes is larger than the preset time threshold, adding the data length to the previous data node in the adjacent data nodes to obtain a data writing point, and writing the data writing point into the corresponding data node.
4. The method for data synchronization supporting breakpoint resume according to claim 1, wherein the creating a table according to the log file data includes:
analyzing the log file data to obtain a database name, a table name, an operation type, a main key and all field values;
and building a table according to the database name, the table name, the operation type, the main key and all field values.
5. The method for data synchronization supporting breakpoint resume according to claim 4, further comprising, after said creating a table according to the log file data:
dividing data in the table into a plurality of data rows according to the primary key;
and dividing each data line into a plurality of data blocks according to the preset configuration.
6. The method for synchronizing data supporting breakpoint resuming according to claim 4, wherein the transmitting the data written with the data node to the target database includes:
sequentially reading all data blocks in each data row according to a time stamp, wherein the time stamp is acquired from log file data;
under the condition that the front end of the data block is read, marking a first binlog position at the front end of the data block, wherein the first binlog position is position information of the front end of the data block acquired from log file data;
under the condition that the rear end of the data block is read, marking a second binlog position at the rear end of the data block, wherein the second binlog position is position information of the rear end of the data block acquired from log file data;
and transmitting the data block marked with the first binlog position and the second binlog position to a target data end.
7. The data synchronization method supporting breakpoint resume according to claim 6, further comprising:
when the data of the source database is changed in the process of transferring the data written into the data node into the target database, determining a data row corresponding to the changed data according to a primary key corresponding to the changed data, comparing the positions of the changed data with all first binlog positions and all second binlog positions of the data row to determine a data block corresponding to the changed data, and changing the data of the data block corresponding to the changed data according to the primary key corresponding to the changed data.
8. A data synchronization system supporting breakpoint resuming, comprising:
the data acquisition module is used for acquiring log file data of a source database;
the data tabulation module is used for tabulating according to the log file data;
the node writing module is used for writing corresponding data nodes in the data of the table according to the data characteristics of the data in the table;
the transmission judging module is used for transmitting the data written with the data nodes to a target database, judging whether information fed back by the target database and receiving the data nodes is received or not under the condition that the data nodes are transmitted in the process of transmitting the data written with the data nodes to the target database, and storing the position information of the data nodes and the corresponding storage time under the condition that the information fed back by the target database is received;
the receiving request module is used for receiving a breakpoint resume request and determining the request time of the breakpoint resume request;
the node determining module is used for determining a target data node corresponding to the storage time closest to the time point according to the request time;
and the data continuous transmission module is used for determining the data which is not transmitted to the target database according to the target data node and the data written with the data node, and continuously transmitting the data to the target database.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210869514.5A CN115168307A (en) | 2022-07-22 | 2022-07-22 | Data synchronization method, system, equipment and storage medium supporting breakpoint continuous transmission |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210869514.5A CN115168307A (en) | 2022-07-22 | 2022-07-22 | Data synchronization method, system, equipment and storage medium supporting breakpoint continuous transmission |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115168307A true CN115168307A (en) | 2022-10-11 |
Family
ID=83497050
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210869514.5A Pending CN115168307A (en) | 2022-07-22 | 2022-07-22 | Data synchronization method, system, equipment and storage medium supporting breakpoint continuous transmission |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115168307A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117591345A (en) * | 2024-01-18 | 2024-02-23 | 柏科数据技术（深圳）股份有限公司 | Ceph-specific data synchronization method, device, equipment and storage medium |
-
2022
- 2022-07-22 CN CN202210869514.5A patent/CN115168307A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117591345A (en) * | 2024-01-18 | 2024-02-23 | 柏科数据技术（深圳）股份有限公司 | Ceph-specific data synchronization method, device, equipment and storage medium |
| CN117591345B (en) * | 2024-01-18 | 2024-05-14 | 柏科数据技术（深圳）股份有限公司 | Ceph-specific data synchronization method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020151181A1 (en) | Cross-platform data updating method and apparatus based on block chain, and computer device | |
| CN108965383B (en) | File synchronization method and device, computer equipment and storage medium | |
| CN111475517B (en) | Data updating method, device, computer equipment and storage medium | |
| CN100461136C (en) | Method and system for maintaining data coherency in multiprocessor system | |
| CN112632375B (en) | Session information processing method, server and storage medium | |
| CN112256433B (en) | Partition migration method and device based on Kafka cluster | |
| CN115168307A (en) | Data synchronization method, system, equipment and storage medium supporting breakpoint continuous transmission | |
| CN112947856A (en) | Memory data management method and device, computer equipment and storage medium | |
| CN111930850A (en) | Data verification method and device, computer equipment and storage medium | |
| CN116521091B (en) | Data reading method, device, equipment, data transmission system and storage medium | |
| CN111752972A (en) | Data association query method and system under key-value storage mode based on RocksDB | |
| CN111767297A (en) | Big data processing method, device, equipment and medium | |
| CN115098519A (en) | Data storage method and device | |
| CN111694806A (en) | Transaction log caching method, device, equipment and storage medium | |
| CN110765125B (en) | Method and device for storing data | |
| CN110795031A (en) | Data deduplication method, device and system based on full flash storage | |
| CN113704027B (en) | File aggregation compatible method and device, computer equipment and storage medium | |
| CN109254997B (en) | Data synchronization method, system, computer device and readable storage medium | |
| CN112783866A (en) | Data reading method and device, computer equipment and storage medium | |
| CN113157738A (en) | In-pile data cache synchronization method and device, computer equipment and storage medium | |
| CN108376104B (en) | Node scheduling method and device and computer readable storage medium | |
| CN114077610A (en) | Data publishing method and related device | |
| CN115334000B (en) | Data transmission method, device, equipment and medium based on multi-control storage system | |
| CN110716923A (en) | Data processing method, data processing device, node equipment and storage medium | |
| US20230045119A1 (en) | Data access method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |