CN115599870A - Data synchronization method based on fusion of stock data and incremental data of message queue - Google Patents
Data synchronization method based on fusion of stock data and incremental data of message queue Download PDFInfo
- Publication number
- CN115599870A CN115599870A CN202211609848.5A CN202211609848A CN115599870A CN 115599870 A CN115599870 A CN 115599870A CN 202211609848 A CN202211609848 A CN 202211609848A CN 115599870 A CN115599870 A CN 115599870A
- Authority
- CN
- China
- Prior art keywords
- data
- information
- data table
- stock
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization method based on fusion of stock data and incremental data of a message queue, which belongs to the technical field of big data and comprises the steps of establishing data source information and storing the data source information into a database; pulling a data table to be synchronized in a data source as a table list to be synchronized, acquiring data source information corresponding to the data table and the data table, and establishing a mapping relation between the data table and a target data table; creating an increment synchronous task, acquiring increment data and sending the increment data to a message queue according to a mapping relation; establishing a stock synchronous task, asynchronously acquiring stock data and sending the stock data to a message queue according to a mapping relation; and creating a data real-time access task, and writing the incremental data or stock data in the message queue into a target data table corresponding to the target database. The invention adopts the message queue to fuse stock synchronization and increment synchronization data, and only one link is used to simultaneously meet the requirements of stock synchronization and increment synchronization, thereby reducing the operation difficulty and the implementation cost of data synchronization.
Description
Technical Field
The invention belongs to the technical field of big data, and particularly relates to a data synchronization method based on fusion of stock data and incremental data of a message queue.
Background
In order to solve data synchronization, offline synchronization tools like sqoop, datax and button, and incremental synchronization tools like canal and debezium appear in the industry. Both types of synchronization tools play a very important role in data synchronization, but they each have their drawbacks.
The offline synchronization tool is used for synchronizing offline stock data, if the offline synchronization tool is used for incremental data synchronization, a scheduling tool is used for scheduling synchronization tasks with small time granularity at regular time, and in this way, the data delay is large, and the offline synchronization tool cannot be applied to data of fields without data change time. The incremental synchronization tool mainly achieves quasi-real-time data synchronization by monitoring a database change log such as mysql-binlog, but the incremental synchronization tool cannot be applied to stock data synchronization. Generally, an incremental synchronization tool is used in combination with an offline synchronization tool, a snapshot is established in a database during initialization, incremental synchronization is started to write changes behind the snapshot into a target library in an incremental synchronization manner, and then the offline synchronization tool is used to synchronize inventory data corresponding to the snapshot to the target library.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: a data synchronization method based on fusion of inventory data and incremental data of a message queue is provided to solve at least some technical problems.
In order to realize the purpose, the technical scheme adopted by the invention is as follows:
the data synchronization method based on the fusion of the stock data and the incremental data of the message queue comprises the following steps:
step 1, creating data source information and storing the data source information in a database;
step 2, pulling a data table to be synchronized in a data source as a table list to be synchronized, acquiring data source information corresponding to the data table and the data table, and establishing a mapping relation between the data table and a target data table;
step 3, creating an increment synchronous task, acquiring increment data and sending the increment data to a message queue according to a mapping relation;
step 4, establishing a stock synchronous task, asynchronously obtaining stock data and sending the stock data to a message queue according to a mapping relation;
and 5, creating a data real-time access task, and writing the incremental data or stock data in the message queue into a target data table corresponding to the target database.
Further, in step 1, the data source information at least includes a data source, and source database information, message queue information, a subject prefix and target database information corresponding to each data source, and the target database information at least includes a database type, database link information and a database name.
Further, in step 2, the mapping relationship between the data table and the target data table is as follows: the method comprises the steps of obtaining a data table in a database and a theme prefix corresponding to the data table, establishing a mapping relation between a target data table in a target database and the data table and a theme, and storing the mapping relation into a set map.
Further, the step 3 comprises: step 31, selecting a data table to be synchronized in the table list to be synchronized, constructing an increment synchronization list, comparing the data table in the increment synchronization list with a target data table in a target database, and newly building a new target data table or modifying an existing target data table; step 32, checking whether a cdc task corresponding to the data table to be synchronized exists, and if not, establishing a corresponding cdc task newly; step 33, the cdc task monitors the change of the data table to be synchronized, then generates and analyzes a change log, generates a change record and establishes a first json file; step 34, judging whether the data table to be synchronized is changed, if so, generating the latest schem information, registering to obtain the current schem version, storing the current schem version into a set map and generating schem total information corresponding to the data table; step 35, obtaining a current schem version from the set map and writing the current schem version into a first json file to form incremental data; and step 36, sending the incremental data and the name of the corresponding data table to a message queue sending module, and the message queue sending module obtains the corresponding theme according to the mapping relation and sends the incremental data to the theme corresponding to the message queue.
Further, in step 32, when a cdc task is newly created, loading a corresponding task template according to the database type corresponding to the data table to be synchronized, and at least filling source database information, data table information to be synchronized, and message queue information corresponding to the data table in the task template; in step 34, the schem total information at least includes a schem version number, and corresponding schem information and a data table.
Further, the step 4 comprises: step 41, selecting a data table to be synchronized in the table list to be synchronized, constructing a stock synchronization list, and initializing the stock synchronization list to be in a state to be executed; step 42, asynchronously obtaining a stock synchronization list in a state to be executed, and obtaining source database information of data source information corresponding to a data table in the stock synchronization list; and 43, selecting a corresponding stock synchronization mode according to the source database type of the source database information, and inquiring and synchronizing stock data in batches through a main key of the data table.
Further, the method for querying and synchronizing inventory data comprises: step 431, inquiring a data table of the source database, acquiring schema information, registering the current schema information and returning to the corresponding schema version number; step 432, establishing a second json file, and writing the returned schema version number into the second json file to form stock data; and 433, sending the stock data and the name of the corresponding data table to a message queue sending module, wherein the message queue sending module obtains a corresponding theme according to the mapping relation and sends the stock data to the theme corresponding to the message queue.
Further, the step 5 comprises: step 51, setting a data access module of a data real-time access task, and consuming message queue information; and step 52, setting a data processing module of the data real-time access task, and processing data source information.
Further, the step 52 includes: step 521, setting target database information; and 522, selecting a corresponding data processing template according to the database type of the target database information, analyzing the incremental data or stock data in the message queue, and writing the incremental data or the stock data into a corresponding target data table according to the mapping relation.
Further, the step 522 includes the following processes: step A, loading a schema version and corresponding schema information into a set map based on the schema total information; b, analyzing the incremental data or stock data, acquiring a corresponding schema version number, and acquiring corresponding schema information from the set map; step C, checking whether the obtained schema information is consistent or compatible with the schema information of the target data table; and D, when the verification results are consistent or compatible, calling a data writing module to write the incremental data or the stock data into the target data table according to the schema information and the database type of the target data table.
Compared with the prior art, the invention has the following beneficial effects:
the invention adopts the message queue to fuse stock synchronization and increment synchronization data, and only one link is used to simultaneously meet the requirements of stock synchronization and increment synchronization, thereby reducing the operation difficulty and the implementation cost of data synchronization. The incremental data and the stock data of the invention use the same set of schema information, and when the structure of the source data table changes, the change is sensed in time to deal with the change of the data structure, thereby adjusting the target data table to achieve the purpose of compatibility. And the stock data can be synchronized at any time, so that the requirement that the stock data is required to be synchronized to complete the target data table when the real-time data is lost or wrong due to various faults is met, and the requirement of data consistency is met.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
FIG. 2 is a diagram of mapping relationship generation according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
As shown in fig. 1, the data synchronization method based on fusion of stock data and incremental data of a message queue, provided by the invention, comprises the following steps:
step 1, creating data source information and storing the data source information into a database;
step 2, pulling a data table to be synchronized in a data source as a table list to be synchronized, acquiring data source information corresponding to the data table and the data table, and establishing a mapping relation between the data table and a target data table;
step 3, creating an increment synchronous task, acquiring increment data and sending the increment data to a message queue according to a mapping relation;
step 4, establishing a stock synchronous task, asynchronously obtaining stock data and sending the stock data to a message queue according to a mapping relation;
and 5, creating a data real-time access task, and writing the incremental data or stock data in the message queue into a target data table corresponding to the target database.
The invention adopts the message queue to fuse stock synchronization and increment synchronization data, and only one link is used to simultaneously meet the requirements of stock synchronization and increment synchronization, thereby reducing the operation difficulty and the implementation cost of data synchronization. The incremental data and the stock data of the invention use the same set of schema information, and when the structure of the source data table changes, the change is sensed in time to deal with the change of the data structure, thereby adjusting the target data table to achieve the purpose of compatibility. And the stock data can be synchronized at any time, so that the requirement that the stock data is required to be synchronized to complete the target data table when the real-time data is lost or wrong due to various faults is met, and the requirement of data consistency is met.
Step 1 of the present invention is an information set of each data source, and the created data source information at least includes a data source, and source database information, message queue information, a subject prefix, and target database information corresponding to each data source. The data source information is presented in the form of an information table, as shown in table 1. The target database information includes at least a database type, database link information, and a database name.
The information table of the data source information is shown in table 1:
table 1 information table of data source information
Step 2 of the invention is the pulling of the data table to be synchronized and the establishment of the mapping relation between the data table and the target data table. And pulling a data table to be synchronized in real time from a data source to construct a table list to be synchronized, wherein the table list to be synchronized can be used as an entrance for synchronizing incremental data and stock data, and data source information corresponding to the data table and the data table in the table list to be synchronized is obtained. A mapping method of a mapping relationship between a data table and a target data table is shown in fig. 2, and includes acquiring a subject prefix corresponding to the data table and the data table in a database, and establishing a mapping relationship between the target data table and the data table and a subject in the target database, that is, each data table has a data source, a subject and a target data table which are in one-to-one correspondence, and storing the mapping relationship into a set map. The name of the subject is the name of the prefix of the subject plus the name of the data table plus the lower horizontal line character, and the name of the target data table is the Ods character plus the upper and lower horizontal line character plus the name of the data table.
The mapping relationship table of the target data table, the data table and the subject is shown in table 2:
TABLE 2 mapping relationship Table
Step 3 of the present invention is incremental data generation based on the message queue. The step 3 comprises the following steps:
step 31, selecting a data table to be synchronized in the table list to be synchronized, constructing an incremental synchronization list, comparing the data table in the incremental synchronization list with a target data table in a target database, and newly building a new target data table or modifying an existing target data table, specifically: if the target database does not have the target data table corresponding to the data table, a corresponding target data table is newly built in the target database, if the target data table corresponding to the target database is in the target database, the structures of the two tables are verified again, if the structures are different, the target data table is modified, and the modification mode includes but is not limited to field increasing, field type modifying and the like;
step 32, checking whether a cdc task corresponding to the data table to be synchronized exists, if not, establishing a corresponding cdc task newly, and when establishing a cdc task newly, loading a corresponding task template according to the database type corresponding to the data table to be synchronized, and filling at least source database information, data table information to be synchronized, message queue information and subject prefix corresponding to the data table in the task template; the source database information at least comprises a database address, an account and a password; the data table information not only comprises a data table and a corresponding data source, but also comprises a main key of the data table; the message queue information at least comprises a sending data interval;
step 33, the cdc task monitors the change of the data table to be synchronized, then generates and analyzes a change log, generates a change record and establishes a first json file; the change of the data table comprises the addition, deletion and modification of data, and the corresponding change record is the addition record, the modification record and the deletion record;
step 34, judging whether the data table to be synchronized is changed, if so, generating the latest schem information, registering to obtain the current schem version, storing the current schem version into a set map and generating the total schem information corresponding to the data table; the set map is used for maintaining the table name of the data table to be synchronized and the version number of the schema version, and the schema management module persists the generated schema total information into the database and obtains the schema information corresponding to the data record by processing the schema version with subsequent data; the schem total information at least comprises a schem version number, corresponding schem information and a data table, and is presented in the form of an information table, and the information table of the schem total information is shown in table 3;
table 3 information table of schem total information
Step 35, obtaining a current schem version from the set map and writing the current schem version into a first json file to form incremental data, and if the set map does not have a schem version, executing step 34 to register schem;
step 36, sending the incremental data and the name of the corresponding data table to a message queue sending module, and the message queue sending module obtains a corresponding theme according to the mapping relation and sends the incremental data to the theme corresponding to the message queue; and in the process of sending the incremental data to the corresponding theme, the incremental data are also distributed to the corresponding subarea by hash modulo according to the primary key of the filling data table.
Step 4 of the present invention is the generation of inventory data based on the message queue. Step 4 comprises the following steps:
step 41, selecting a data table to be synchronized in a table list to be synchronized, constructing a stock synchronization list, and initializing the stock synchronization list to be in a state to be executed;
step 42, asynchronously obtaining a stock synchronization list of a state to be executed, and obtaining source database information of data source information corresponding to a data table in the stock synchronization list;
and 43, selecting a corresponding stock synchronization mode according to the source database type of the source database information, and inquiring and synchronizing stock data in batches through a main key of the data table. The stock synchronization mode includes but is not limited to jdbc and REST API.
The method for querying and synchronizing inventory data comprises the following steps: step 431, querying a data table of a source database and acquiring schema information, registering current schema information and returning to a corresponding schema version number, if the schema information exists, returning to the existing schema version number, and if the schema information does not exist, newly building a schema version and returning to the newly built schema version number; step 432, establishing a second json file, and writing the returned schema version number into the second json file to form stock data; and 433, sending the stock data and the name of the corresponding data table to a message queue sending module, wherein the message queue sending module obtains a corresponding theme according to the mapping relation and sends the stock data to the theme corresponding to the message queue, and in the process of sending the stock data to the theme corresponding to the message queue, the stock data is further distributed to a corresponding partition by hash modulo according to a main key for filling the data table.
Step 5 of the invention is the fusion of stock data and incremental data in the message queue. The step 5 comprises the following steps:
step 51, setting a data access module of a data real-time access task, consuming message queue information, wherein the message queue information comprises a message queue address, a subject to be consumed and a target data table corresponding to the subject, and the subject to be consumed is automatically generated according to a mapping relation;
and step 52, setting a data processing module of the data real-time access task, and processing data source information.
Said step 52 comprises: step 521, setting target database information; and 522, selecting a corresponding data processing template according to the database type of the target database information, analyzing the incremental data or stock data in the message queue, and writing the incremental data or the stock data into a corresponding target data table according to the mapping relation.
Said step 522 comprises the following processes: step A, loading a schema version and corresponding schema information into a set map based on the schema total information; b, analyzing the incremental data or stock data, acquiring a corresponding schema version number, and acquiring corresponding schema information from the set map; step C, checking whether the obtained schema information is consistent or compatible with the schema information of the target data table; and D, when the verification results are consistent or compatible, calling a data writing module to write the incremental data or the stock data into the target data table according to the schema information and the database type of the target data table.
The message queue according to the invention is preferably kafka.
Finally, it should be noted that: the above embodiments are only preferred embodiments of the present invention to illustrate the technical solutions of the present invention, but not to limit the technical solutions, and certainly not to limit the patent scope of the present invention; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention; that is, the technical problems to be solved by the present invention, which are not substantially changed or supplemented by the spirit and the concept of the main body of the present invention, are still consistent with the present invention and shall be included in the scope of the present invention; in addition, the technical scheme of the invention is directly or indirectly applied to other related technical fields, and the technical scheme of the invention is included in the patent protection scope of the invention.
Claims (10)
1. The data synchronization method based on fusion of stock data and incremental data of the message queue is characterized by comprising the following steps of:
step 1, creating data source information and storing the data source information into a database;
step 2, pulling a data table to be synchronized in a data source as a table list to be synchronized, acquiring data source information corresponding to the data table and the data table, and establishing a mapping relation between the data table and a target data table;
step 3, creating an increment synchronous task, acquiring increment data and sending the increment data to a message queue according to a mapping relation;
step 4, establishing a stock synchronous task, asynchronously obtaining stock data and sending the stock data to a message queue according to a mapping relation;
and 5, creating a data real-time access task, and writing the incremental data or stock data in the message queue into a target data table corresponding to the target database.
2. The message queue-based data synchronization method for fusing inventory data and incremental data according to claim 1, wherein in step 1, the data source information at least includes data sources, and source database information, message queue information, subject prefix and target database information corresponding to each data source, and the target database information at least includes database types, database link information and database library names.
3. The message queue-based data synchronization method based on fusion of stock data and incremental data of claim 2, wherein the mapping relationship between the data table and the target data table is as follows: the method comprises the steps of obtaining a data table in a database and a theme prefix corresponding to the data table, establishing a mapping relation between a target data table in the target database and the data table and a theme, and storing the mapping relation into a set map.
4. The message queue-based data synchronization method with fusion of inventory data and incremental data according to claim 3, wherein the step 3 comprises: step 31, selecting a data table to be synchronized in the table list to be synchronized, constructing an incremental synchronization list, comparing the data table in the incremental synchronization list with a target data table in a target database, and creating a new target data table or modifying an existing target data table; step 32, checking whether a cdc task corresponding to the data table to be synchronized exists, and if not, establishing a corresponding cdc task newly; step 33, the cdc task monitors the change of the data table to be synchronized, then generates and analyzes a change log, generates a change record and establishes a first json file; step 34, judging whether the data table to be synchronized is changed, if so, generating the latest schem information, registering to obtain the current schem version, storing the current schem version into a set map and generating the total schem information corresponding to the data table; step 35, obtaining a current schem version from the set map and writing the current schem version into a first json file to form incremental data; and step 36, sending the incremental data and the name of the corresponding data table to a message queue sending module, and the message queue sending module obtains the corresponding theme according to the mapping relation and sends the incremental data to the theme corresponding to the message queue.
5. The method according to claim 4, wherein in step 32, when a cdc task is newly created, a corresponding task template is loaded according to the database type corresponding to the data table to be synchronized, and at least source database information, data table information to be synchronized, and message queue information corresponding to the data table are filled in the task template; in the step 34, the schem total information at least includes a schem version number, and corresponding schem information and a data table.
6. The message queue-based data synchronization method with fusion of inventory data and incremental data according to claim 5, wherein the step 4 comprises: step 41, selecting a data table to be synchronized in the table list to be synchronized, constructing a stock synchronization list, and initializing the stock synchronization list to be in a state to be executed; step 42, asynchronously obtaining a stock synchronization list in a state to be executed, and obtaining source database information of data source information corresponding to a data table in the stock synchronization list; and 43, selecting a corresponding stock synchronization mode according to the source database type of the source database information, and inquiring and synchronizing stock data in batches through a main key of the data table.
7. The message queue-based data synchronization method fusing inventory data and incremental data according to claim 6, wherein the method for querying and synchronizing inventory data comprises: step 431, inquiring a data table of the source database, acquiring schema information, registering the current schema information and returning to the corresponding schema version number; step 432, establishing a second json file, and writing the returned schema version number into the second json file to form stock data; and 433, sending the stock data and the name of the corresponding data table to a message queue sending module, wherein the message queue sending module obtains a corresponding theme according to the mapping relation and sends the stock data to the theme corresponding to the message queue.
8. The message queue-based data synchronization method with fusing inventory data and incremental data according to claim 7, wherein the step 5 comprises: step 51, setting a data access module of a data real-time access task, and consuming message queue information; and step 52, setting a data processing module of the data real-time access task, and processing data source information.
9. The message queue-based data synchronization method with fusing inventory data and delta data according to claim 8, wherein the step 52 comprises: step 521, setting target database information; and 522, selecting a corresponding data processing template according to the database type of the target database information, analyzing the incremental data or stock data in the message queue, and writing the incremental data or the stock data into a corresponding target data table according to the mapping relation.
10. The message queue-based data synchronization method with fusing inventory data and delta data of claim 9, wherein said step 522 comprises the following process: step A, loading a schema version and corresponding schema information into a set map based on the schema total information; b, analyzing the incremental data or stock data, acquiring a corresponding schema version number, and acquiring corresponding schema information from the set map; step C, checking whether the obtained schema information is consistent or compatible with the schema information of the target data table; and D, when the verification results are consistent or compatible, calling a data writing module to write the incremental data or stock data into the target data table according to the schema information and the database type of the target data table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211609848.5A CN115599870B (en) | 2022-12-15 | 2022-12-15 | Data synchronization method based on fusion of stock data and incremental data of message queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211609848.5A CN115599870B (en) | 2022-12-15 | 2022-12-15 | Data synchronization method based on fusion of stock data and incremental data of message queue |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115599870A true CN115599870A (en) | 2023-01-13 |
| CN115599870B CN115599870B (en) | 2023-04-07 |
Family
ID=84854200
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211609848.5A Active CN115599870B (en) | 2022-12-15 | 2022-12-15 | Data synchronization method based on fusion of stock data and incremental data of message queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115599870B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115905300A (en) * | 2023-03-14 | 2023-04-04 | 云账户技术(天津)有限公司 | Method and device for fusion capture of stock and incremental data of TiDB (database data base) |
| CN116089537A (en) * | 2023-04-07 | 2023-05-09 | 江西省智能产业技术创新研究院 | Incremental data synchronization method, system, computer and storage medium |
| CN116361391A (en) * | 2023-03-30 | 2023-06-30 | 中电云数智科技有限公司 | Method and device for detecting and repairing structural abnormality of data synchronization table |
| CN116701717A (en) * | 2023-08-04 | 2023-09-05 | 杭州悦数科技有限公司 | Graph database data importing method and system |
| CN116975159A (en) * | 2023-09-25 | 2023-10-31 | 云筑信息科技(成都)有限公司 | Incremental data synchronization processing method |
| CN117076431A (en) * | 2023-10-13 | 2023-11-17 | 云筑信息科技(成都)有限公司 | Method for migrating system upgrade data |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103561349A (en) * | 2013-11-06 | 2014-02-05 | 上海文广科技（集团）有限公司 | Integrated content management system based on internet television services and management method of integrated content management system |
| AU2015201041A1 (en) * | 2008-03-04 | 2015-03-19 | Apple Inc. | Data synchronization protocol |
| CN107391635A (en) * | 2017-06-30 | 2017-11-24 | 北京奇虎科技有限公司 | Data synchronous system and method |
| CN110647579A (en) * | 2019-08-16 | 2020-01-03 | 北京百度网讯科技有限公司 | Data synchronization method and device, computer equipment and readable medium |
| CN110737720A (en) * | 2019-09-06 | 2020-01-31 | 苏宁云计算有限公司 | DB2 database data synchronization method, device and system |
| CN111339103A (en) * | 2020-03-13 | 2020-06-26 | 河南安冉云网络科技有限公司 | Data exchange method and system based on full fragmentation and incremental log analysis |
| CN112988702A (en) * | 2019-12-12 | 2021-06-18 | 天翼电子商务有限公司 | Heterogeneous data source real-time data transmission method and system, storage medium and terminal |
| CN114036119A (en) * | 2021-09-30 | 2022-02-11 | 河海大学 | Data synchronization method based on button and database log |
| CN115470217A (en) * | 2022-11-14 | 2022-12-13 | 云筑信息科技(成都)有限公司 | Method for solving change response problem of data bin model in real time |
-
2022
- 2022-12-15 CN CN202211609848.5A patent/CN115599870B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU2015201041A1 (en) * | 2008-03-04 | 2015-03-19 | Apple Inc. | Data synchronization protocol |
| CN103561349A (en) * | 2013-11-06 | 2014-02-05 | 上海文广科技（集团）有限公司 | Integrated content management system based on internet television services and management method of integrated content management system |
| CN107391635A (en) * | 2017-06-30 | 2017-11-24 | 北京奇虎科技有限公司 | Data synchronous system and method |
| CN110647579A (en) * | 2019-08-16 | 2020-01-03 | 北京百度网讯科技有限公司 | Data synchronization method and device, computer equipment and readable medium |
| CN110737720A (en) * | 2019-09-06 | 2020-01-31 | 苏宁云计算有限公司 | DB2 database data synchronization method, device and system |
| CN112988702A (en) * | 2019-12-12 | 2021-06-18 | 天翼电子商务有限公司 | Heterogeneous data source real-time data transmission method and system, storage medium and terminal |
| CN111339103A (en) * | 2020-03-13 | 2020-06-26 | 河南安冉云网络科技有限公司 | Data exchange method and system based on full fragmentation and incremental log analysis |
| CN114036119A (en) * | 2021-09-30 | 2022-02-11 | 河海大学 | Data synchronization method based on button and database log |
| CN115470217A (en) * | 2022-11-14 | 2022-12-13 | 云筑信息科技(成都)有限公司 | Method for solving change response problem of data bin model in real time |
Non-Patent Citations (2)
| Title |
|---|
| A.J. LEE 等: "The EVE approach: view synchronization in dynamic distributed environments" * |
| 冼啟铭: "大数据量下异构数据库的数据同步系统的设计与实现" * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115905300A (en) * | 2023-03-14 | 2023-04-04 | 云账户技术(天津)有限公司 | Method and device for fusion capture of stock and incremental data of TiDB (database data base) |
| CN116361391A (en) * | 2023-03-30 | 2023-06-30 | 中电云数智科技有限公司 | Method and device for detecting and repairing structural abnormality of data synchronization table |
| CN116089537A (en) * | 2023-04-07 | 2023-05-09 | 江西省智能产业技术创新研究院 | Incremental data synchronization method, system, computer and storage medium |
| CN116701717A (en) * | 2023-08-04 | 2023-09-05 | 杭州悦数科技有限公司 | Graph database data importing method and system |
| CN116701717B (en) * | 2023-08-04 | 2023-10-27 | 杭州悦数科技有限公司 | Graph database data importing method and system |
| CN116975159A (en) * | 2023-09-25 | 2023-10-31 | 云筑信息科技(成都)有限公司 | Incremental data synchronization processing method |
| CN116975159B (en) * | 2023-09-25 | 2024-02-23 | 云筑信息科技(成都)有限公司 | Incremental data synchronization processing method |
| CN117076431A (en) * | 2023-10-13 | 2023-11-17 | 云筑信息科技(成都)有限公司 | Method for migrating system upgrade data |
| CN117076431B (en) * | 2023-10-13 | 2024-03-12 | 云筑信息科技(成都)有限公司 | Method for migrating system upgrade data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115599870B (en) | 2023-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115599870B (en) | Data synchronization method based on fusion of stock data and incremental data of message queue | |
| JP4397385B2 (en) | Method implemented as computer processor instructions stored in computer and storage medium readable by computer | |
| CN105701159B (en) | A kind of data synchronization unit and method | |
| CN107992537B (en) | Service attribute transmission method, device, computer equipment and storage medium | |
| KR20150118975A (en) | System and methods for multi-user cax editing conflict management | |
| CN107886296B (en) | Collaborative auditing method between heterogeneous PDM systems | |
| CN110113406B (en) | Distributed computing service cluster system | |
| CN108319617A (en) | Determine the method, apparatus and method for handover control, device of database principal and subordinate's difference | |
| CN105183672B (en) | A kind of adaptive interface call method and system | |
| CN112597249A (en) | Synchronous distribution and storage method and system for service data | |
| CN112364049A (en) | Data synchronization script generation method, system, terminal and storage medium | |
| CN114285695B (en) | Communication method, device, apparatus, system and storage medium | |
| CN107465725B (en) | Heterogeneous long transaction processing system and method based on client information control system | |
| CN115048190B (en) | Virtual machine snapshot management method, system and computer readable medium | |
| CN115827657A (en) | Database data synchronization method, storage medium and device | |
| CN109711816A (en) | Cooperative institution's approaches to IM, device, computer equipment and storage medium | |
| CN111797166B (en) | Method and device for synchronizing quasi-real-time resume data, electronic equipment and medium | |
| CN114331391A (en) | Workflow synchronous updating method and workflow synchronous updating system based on global state updating | |
| CN114020368A (en) | Information processing method and device based on state machine and storage medium | |
| CN110935168B (en) | Distributed ID generation method, device and equipment for global uniform architecture | |
| US20200304449A1 (en) | E-mail display device and non-transitory computer readable medium storing program | |
| CN111506668A (en) | Robot cluster intelligent data synchronization method and system | |
| CN111159179A (en) | Table partitioning method, related device and computer readable storage medium | |
| CN110413589A (en) | Approaches to IM and platform based on interspace file system | |
| CN112559548B (en) | Data synchronization system and method for message middleware |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |