CN112445863A - Real-time data synchronization method and system - Google Patents
Real-time data synchronization method and system Download PDFInfo
- Publication number
- CN112445863A CN112445863A CN202011370254.4A CN202011370254A CN112445863A CN 112445863 A CN112445863 A CN 112445863A CN 202011370254 A CN202011370254 A CN 202011370254A CN 112445863 A CN112445863 A CN 112445863A
- Authority
- CN
- China
- Prior art keywords
- data table
- operational data
- log file
- mysql database
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000003860 storage Methods 0.000 claims abstract description 18
- 206010028980 Neoplasm Diseases 0.000 claims description 40
- 201000011510 cancer Diseases 0.000 claims description 40
- 230000008859 change Effects 0.000 claims description 23
- 238000004364 calculation method Methods 0.000 claims description 12
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000007726 management method Methods 0.000 description 9
- 238000004458 analytical method Methods 0.000 description 3
- 230000006399 behavior Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a real-time data synchronization method and a real-time data synchronization system, wherein the method comprises the following steps: acquiring a Binlog log file in a MySQL database in real time by adopting a Canal, and analyzing the Binlog file, wherein the Binlog file comprises a first operational data table; sending the Binlog log file to a Kafka queue for caching in a json format; acquiring Binlog log files cached on a Kafka queue into Hive bins in real time by adopting SreamSets; snapshot storage is carried out on the operational data table in the MySQL database at intervals of a preset period, and a second operational data table is obtained; and performing Merge operation on the first operational data table stored in the Hive number bin according to the second operational data table, and updating the first operational data table. The invention can update the operational data in the Hive bins in time and improve the efficiency of data synchronization.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a real-time data synchronization method and system.
Background
Today with IOE removed (IBM's mini-servers, Oracle databases, EMC storage devices removed), and a hugging source, OLTP (On-Line Transaction Processing) business data from many Internet companies is stored in the MySQL database. MySQL is a relational database management system that keeps data in different tables instead of putting all the data in one large repository, which increases speed and flexibility. As a main source of the structured data of the big data platform, the method mainly adopts an off-line batch T +1 batch running mode for collection at present.
Although the scheme is simple to implement, with the development of services, the time spent on offline full batch running is longer and longer, and the time requirement of downstream warehouse production cannot be met; a large amount of data is directly selected from the MySQL database, so that the MySQL database is greatly influenced, slow query is easily caused, and normal service on a service line is influenced; because the syntax of Hive does not support updating and deleting MySQL language, the data updated or deleted in MySQL can not be supported well, once the structure of the MySQL database table is changed, ETL (Extract-Transform-Load) and Hive tables in the big data platform can not be sensed and adjusted timely, and data quality difference is caused.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a method and system for real-time data synchronization.
A real-time data synchronization method comprises the following steps: acquiring a Bilog log file in a MySQL database in real time by adopting a Canal, and analyzing the Bilog log file, wherein the Bilog log file comprises a first operational data table; sending the Binlog log file to a Kafka queue cache in a json format; acquiring Binlog log files cached on a Kafka queue into Hive bins in real time by adopting SreamSets; snapshot storage is carried out on the operational data table in the MySQL database at intervals of a preset period, and a second operational data table is obtained; and performing Merge operation on the first operational data table stored in the Hive number bin according to the second operational data table, and updating the first operational data table.
In one embodiment, the obtaining and analyzing a Binlog log file in a MySQL database in real time by using Canal includes: the Canal sends a MySQL slave protocol to the MySQL database; receiving feedback information returned by the MySQL database; generating a data synchronization request based on a MySQL dump protocol and sending the data synchronization request to the MySQL database; and receiving the binlog log file returned by the MySQL database.
In one embodiment, the Canal includes: a cancer manager, a cancer server, a cancer client and a cancer standby; the cancer manager is connected with the cancer server and the cancer client; the cancer server and the cancer client are connected with the cancer standby through a ZooKeeper system; the Canal server is used for acquiring the Binlog log file; the Canal client is used for analyzing the Binlog log file and sending the Json format of the Binlog log file to a Kafka queue for caching.
In one embodiment, after sending the Binlog log file to the Kafka queue buffer in json format, the method further includes: and the real-time calculation engine consumes the data in the Kafka queue in real time, performs aggregate statistical calculation, and stores the calculation result in the storage engine.
In one embodiment, the obtaining and analyzing a Binlog log file in a MySQL database in real time by using Canal further includes: configuring and monitoring DDL operation of a MySQL database end; the change operations of the libraries, tables and fields obtained by analyzing the Binlog log file are landed in a MySQL database table in real time; the change operation is sent to a management end in a mail or short message mode in real time; and the management terminal rechecks whether the change operation is reasonable, and if not, the change operation is rolled back or prevented.
A system for real-time synchronization of data, comprising: the file acquisition module is used for acquiring and analyzing a Binlog log file in a MySQL database in real time by adopting Canal, wherein the Binlog log file comprises a first operational data table; the file sending module is used for sending the Binlog log file to a Kafka queue cache in a json format; the file acquisition module is used for acquiring the Binlog log files cached on the Kafka queue to Hive bins in real time by using StreamSets; the snapshot storage module is used for carrying out snapshot storage on the operational data table in the MySQL database at intervals of a preset period to obtain a second operational data table; and the Merge operation module is used for performing Merge operation on the first operational data table stored in the Hive number bin according to the second operational data table and updating the first operational data table.
Compared with the prior art, the invention has the advantages and beneficial effects that:
1. acquiring a Binlog log file in a MySQL database in real time by adopting Canal, sending the Binlog file to a Kafka queue in a json format, and acquiring the Binlog file cached on the Kafka queue to a Hive number bin in real time by adopting SreamSets, so that the Binlog file can be quickly acquired on the premise of not influencing the MySQL database;
2. snapshot storage is carried out on the operational data table in the MySQL database at intervals of a preset period, and a second operational data table is obtained; according to the second operational data table, Merge operation is carried out on the first operational data table stored in the Hive number bin, the first operational data table is updated, the operational data table in the Hive number bin can be updated in time, and the data synchronization efficiency is improved.
Drawings
FIG. 1 is a flow chart illustrating a method for real-time synchronization of data according to an embodiment;
FIG. 2 is a flow diagram that illustrates the real-time collection of a Binlog log file, under an embodiment;
FIG. 3 is a block diagram of a real-time data synchronization system according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings by way of specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In one embodiment, as shown in fig. 1, there is provided a real-time data synchronization method, comprising the steps of:
and S101, acquiring and analyzing a Bilog log file in the MySQL database in real time by adopting a Canal, wherein the Bilog log file comprises a first operational data table.
The Canal is a source opening project, provides incremental data subscription and consumption based on database incremental log analysis, and mainly supports MySQL at present.
The Binlog (Binary log, Binary log file) log file records all operations of insertion, deletion and modification in the MySQL database, and data stored in the MySQL database can be copied through the Binlog log file.
The operational data table stores data extracted directly from the business system and is stored in the form of a distributed file.
And step S102, sending the Binlog log file to a Kafka queue cache in a json format.
The Kafka is a distributed log system based on Zookeeper coordination, and the Binlog log file is sent to the Kafka queue, so that the transaction behavior of a front-end user can be effectively monitored, and the Binlog log file can be consumed by a lower-end system conveniently.
And step S103, acquiring the Binlog log file cached on the Kafka queue to Hive number bins in real time by using streams.
The Streamsets are a large data real-time acquisition and ETL (extract, transform, load) tool, can achieve data acquisition and circulation without writing a line of codes, and achieves design and timing task scheduling of data Pipelines (pipeline) through a dragging type visual interface. Therefore, the Binlog log files in the Kafka queue can be collected into Hive bins in real time through StreamSets.
And step S104, snapshot storage is carried out on the operational data table in the MySQL database at intervals of a preset period, and a second operational data table is obtained.
Specifically, after a first operational data table in the MySQL database is acquired, snapshot storage is performed on the operational data table in the MySQL database at intervals of a predetermined period, and a second operational data table is acquired, so that the first operational data table can be updated correspondingly for the second operational data table; and the operational data table in the Hive data bin can be updated in time according to the change in the MySQL database.
Step S105, performing a Merge operation on the first operational data table stored in the Hive number bin according to the second operational data table, and updating the first operational data table.
Specifically, the Merge operation includes an UPDATE statement and an INSERT statement, the first operational data table is queried according to the second operational data table, the UPDATE operation is performed on the connection condition matching, the INSERT operation is performed which cannot be matched, all work can be completed through one-time full-table scanning, and the execution efficiency is higher than that of the UPDATE and the INSERT statements.
Specifically, the quick update of the operational data in the Hive number bin can be realized through the Merge operation, so that the operational data in the Hive number bin can be always consistent with the operational data in the MySQL database.
In the embodiment, the method comprises the steps of acquiring a Binlog log file in a MySQL database in real time by adopting Canal, sending the Binlog log file to a Kafka queue in a json format, and acquiring the Binlog log file cached on the Kafka queue to a Hive number bin in real time by adopting SreamSets, so that the Binlog log file can be quickly acquired on the premise of not influencing the MySQL database; snapshot storage is carried out on the operational data table in the MySQL database at intervals of a preset period, and a second operational data table is obtained; according to the second operational data table, Merge operation is carried out on the first operational data table stored in the Hive number bin, the first operational data table is updated, the operational data table in the Hive number bin can be updated in time, and the data synchronization efficiency is improved.
Wherein, step S101 specifically includes: sending a MySQL slave protocol to a MySQL database by the Canal; receiving feedback information returned by the MySQL database; generating a data synchronization request based on the MySQL dump protocol and sending the data synchronization request to a MySQL database; and receiving the binlog log file returned by the MySQL database.
Specifically, the Canal sends a MySQL slave protocol to the MySQL database, which is disguised as MySQL slave, thereby being capable of realizing connection and data transmission with the MySQL database.
Wherein, because the Binlog log file of MySQL supports three formats: the Statement, Row and Mixed formats, but only the Binlog log file in the Row format records the changed data, so before acquiring the Binlog log file, the Binlog log file in the MySQL database needs to be adjusted to the Row format, which is convenient for realizing data update acquisition service.
Wherein, step S101 further includes: configuring and monitoring DDL operation of a MySQL database end; the change operation of the library, the table and the field obtained by analyzing the Binlog log file is landed in a MySQL database table in real time; the change operation is sent to a management end in a mail or short message mode in real time; and the management terminal rechecks whether the change operation is reasonable, and if not, the change operation is rolled back or prevented.
In particular, DDL operations include Create, enter, Drop, and the like operations.
Specifically, the change operation of the library, the table and the field obtained by analyzing the Binlog log file is landed in a MySQL database table in real time, the change information can be sent to a database management end in a mail or short message mode in real time, and a manager at the management end can check whether the change operation is reasonable or not and roll back or prevent the change operation when the unreasonable change operation occurs.
In addition, related personnel of the big data platform can contrast the change operation, screen mapping logic and related data storage Hive tables related to the ETL of the big data platform, and timely make contrast adjustment, so that data difference between a source service system and a big data platform is avoided.
In one embodiment, as shown in FIG. 2, a Canal includes: a cancer manager, a cancer server, a cancer client and a cancer standby; the cancer manager is connected with the cancer server and the cancer client; the cancer server and the cancer client are connected with the cancer standby through a ZooKeeper system; the Canal server is used for acquiring a Binlog log file; the Canal client is used for analyzing the Binlog log file and sending the json format of the Binlog log file to the Kafka queue for caching.
Wherein, the cancer manager is a cancer manager, the cancer server is a cancer server, the cancer client is a cancer client, and the cancer standby is a cancer standby terminal.
The solid arrows represent control flows, and the dashed arrows represent data flows. The Canal server acquires the Bilog log file from the MySQL database, and sends the Bilog log file to the Kafka queue for caching through the Canal client.
Specifically, when the Canal server needs to start a certain actually-operated data queue, it needs to first perform an attempt start judgment on the Zookeeper, and the creation is successful and the data queue is allowed to be started by performing the judgment through the creation node; and a data queue that is not successfully created will be in standby. The method of the cancer client is similar to the method of the cancer server, and the method of the Zookeeper preempting the node is also used for controlling.
The method comprises the steps that with the expansion of business scale, the condition of sub-databases and sub-tables of the MySQL database is more and more, data of each business department are stored in different MySQL databases of different hosts in a scattered mode according to the fact that the lock meets the difference of scene functions, the data are scattered, different examples can be configured for the MySQL database, analysis of Binlog log files is conducted through different clients, analysis results are combined into one group, and then the data fall into one MySQL database, and therefore the purpose of data integration is achieved.
After step S102, the method further includes: and the real-time calculation engine consumes the data in the Kafka queue in real time, performs aggregate statistical calculation, and stores the calculation result in the storage engine.
Specifically, in actual production, in order to fully satisfy timeliness of service flow, the front-end system is only responsible for landing collected various user behaviors and interactive service data into the MySQL database, but not writing into a message queue, such as a Kafka queue; after the Binlog log file is cached in the Kafka queue, a subsequent real-time calculation engine can consume data in the Kafka queue in real time, various aggregation statistical calculations can be performed, and a final calculation result can be stored in Kudu (storage engine), so that effective monitoring on transaction behaviors of a front-end user can be achieved, and transaction results of various indexes can be statistically displayed in real time.
As shown in fig. 3, there is provided a real-time synchronization system 30 for data, comprising: a file acquiring module 31, a file sending module 32, a file collecting module 33, a snapshot storing module 34, and a Merge operating module 35, where:
the file acquisition module 31 is configured to acquire and analyze a Binlog log file in a MySQL database in real time by using Canal, where the Binlog file includes a first operational data table;
the file sending module 32 is configured to send the Binlog log file to a Kafka queue cache in a json format;
the file acquisition module 33 is used for acquiring the Binlog log files cached on the Kafka queue to Hive bins in real time by using StreamSets;
a snapshot storage module 34, configured to perform snapshot storage on the operational data table in the MySQL database at intervals of a predetermined period, and acquire a second operational data table;
the Merge operation module 35 is configured to perform a Merge operation on the first operational data table stored in the Hive number bin according to the second operational data table, and update the first operational data table.
In one embodiment, the file obtaining module 31 is further configured to send a MySQL slave protocol to the MySQL database by Canal; receiving feedback information returned by the MySQL database; generating a data synchronization request based on the MySQL dump protocol and sending the data synchronization request to a MySQL database; and receiving the binlog log file returned by the MySQL database.
The file acquisition module 31 is also used for configuring and monitoring the DDL operation of the MySQL database end; the change operation of the library, the table and the field obtained by analyzing the Binlog log file is landed in a MySQL database table in real time; the change operation is sent to a management end in a mail or short message mode in real time; and the management terminal rechecks whether the change operation is reasonable, and if not, the change operation is rolled back or prevented.
The foregoing is a more detailed description of the present invention that is presented in conjunction with specific embodiments, and the practice of the invention is not to be considered limited to those descriptions. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (6)
1. A real-time data synchronization method is characterized by comprising the following steps:
acquiring a Bilog log file in a MySQL database in real time by adopting a Canal, and analyzing the Bilog log file, wherein the Bilog log file comprises a first operational data table;
sending the Binlog log file to a Kafka queue cache in a json format;
acquiring Binlog log files cached on a Kafka queue into Hive bins in real time by adopting SreamSets;
snapshot storage is carried out on the operational data table in the MySQL database at intervals of a preset period, and a second operational data table is obtained;
and performing Merge operation on the first operational data table stored in the Hive number bin according to the second operational data table, and updating the first operational data table.
2. The real-time data synchronization method according to claim 1, wherein the acquiring and analyzing Binlog log files in MySQL database in real time by using Canal comprises:
the Canal sends a MySQL slave protocol to the MySQL database;
receiving feedback information returned by the MySQL database;
generating a data synchronization request based on a MySQL dump protocol and sending the data synchronization request to the MySQL database;
and receiving the binlog log file returned by the MySQL database.
3. The real-time synchronization method of data according to claim 1, wherein said Canal comprises: a cancer manager, a cancer server, a cancer client and a cancer standby;
the cancer manager is connected with the cancer server and the cancer client; the cancer server and the cancer client are connected with the cancer standby through a ZooKeeper system; the Canal server is used for acquiring the Binlog log file; the Canal client is used for analyzing the Binlog log file and sending the Json format of the Binlog log file to a Kafka queue for caching.
4. The real-time data synchronization method of claim 1, wherein after sending the Binlog log file to the Kafka queue buffer in json format, the method further comprises:
and the real-time calculation engine consumes the data in the Kafka queue in real time, performs aggregate statistical calculation, and stores the calculation result in the storage engine.
5. The real-time data synchronization method according to claim 1, wherein the acquiring and analyzing Binlog log files in MySQL database in real time by using Canal further comprises:
configuring and monitoring DDL operation of a MySQL database end;
the change operations of the libraries, tables and fields obtained by analyzing the Binlog log file are landed in a MySQL database table in real time;
the change operation is sent to a management end in a mail or short message mode in real time;
the management end checks whether the change operation is reasonable,
if not, the change operation is rolled back or blocked.
6. A system for real-time synchronization of data, comprising:
the file acquisition module is used for acquiring and analyzing a Binlog log file in a MySQL database in real time by adopting Canal, wherein the Binlog log file comprises a first operational data table;
the file sending module is used for sending the Binlog log file to a Kafka queue cache in a json format;
the file acquisition module is used for acquiring the Binlog log files cached on the Kafka queue to Hive bins in real time by using StreamSets;
the snapshot storage module is used for carrying out snapshot storage on the operational data table in the MySQL database at intervals of a preset period to obtain a second operational data table;
and the Merge operation module is used for performing Merge operation on the first operational data table stored in the Hive number bin according to the second operational data table and updating the first operational data table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011370254.4A CN112445863B (en) | 2020-11-30 | 2020-11-30 | Data real-time synchronization method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011370254.4A CN112445863B (en) | 2020-11-30 | 2020-11-30 | Data real-time synchronization method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112445863A true CN112445863A (en) | 2021-03-05 |
| CN112445863B CN112445863B (en) | 2024-06-18 |
Family
ID=74738637
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011370254.4A Active CN112445863B (en) | 2020-11-30 | 2020-11-30 | Data real-time synchronization method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112445863B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113064906A (en) * | 2021-04-21 | 2021-07-02 | 杭州天谷信息科技有限公司 | Binlog log data adaptive migration method and system |
| CN113220757A (en) * | 2021-04-24 | 2021-08-06 | 上海钢银科技发展有限公司 | Real-time warehouse automatic ETL method, system, equipment and computer storage medium |
| CN113297216A (en) * | 2021-05-17 | 2021-08-24 | 中国人民解放军63920部队 | Real-time storage method for space flight measurement and control data |
| CN113468170A (en) * | 2021-06-02 | 2021-10-01 | 上海赛可出行科技服务有限公司 | System for automatically realizing real-time data synchronization |
| CN113742392A (en) * | 2021-09-17 | 2021-12-03 | 中国银行股份有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN113821407A (en) * | 2021-09-15 | 2021-12-21 | 浙江网新恩普软件有限公司 | Storm distributed real-time computing method and system |
| CN115269563A (en) * | 2022-09-26 | 2022-11-01 | 北京奥星贝斯科技有限公司 | Method and device for performing log analysis on database system |
| CN115470217A (en) * | 2022-11-14 | 2022-12-13 | 云筑信息科技(成都)有限公司 | Method for solving change response problem of data bin model in real time |
| CN117555699A (en) * | 2024-01-11 | 2024-02-13 | 杭州剑齿虎信息技术有限公司 | LCK real-time acquisition system |
| CN117633104A (en) * | 2023-11-06 | 2024-03-01 | 湖北华中电力科技开发有限责任公司 | Method and device for realizing supervision work order collaboration through incremental data channel |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107330003A (en) * | 2017-06-12 | 2017-11-07 | 上海藤榕网络科技有限公司 | Method of data synchronization, system, memory and data syn-chronization equipment |
| US20180210808A1 (en) * | 2017-01-25 | 2018-07-26 | Verizon Patent And Licensing Inc. | System and methods for application activity capture, error identification, and error correction |
| CN108804237A (en) * | 2017-05-05 | 2018-11-13 | 北京京东尚科信息技术有限公司 | Data real-time statistical method, device, storage medium and electronic equipment |
| CN109829009A (en) * | 2018-12-28 | 2019-05-31 | 北京邮电大学 | Configurable isomeric data real-time synchronization and visual system and method |
| CN110521171A (en) * | 2017-03-28 | 2019-11-29 | 思科技术公司 | The stream cluster parsing for monitoring and managing for application performance |
| CN110597914A (en) * | 2019-09-18 | 2019-12-20 | 北京思维造物信息科技股份有限公司 | Data transmission system, method, device and equipment |
| CN111209344A (en) * | 2020-02-07 | 2020-05-29 | 浪潮软件股份有限公司 | Data synchronization method and device |
-
2020
- 2020-11-30 CN CN202011370254.4A patent/CN112445863B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180210808A1 (en) * | 2017-01-25 | 2018-07-26 | Verizon Patent And Licensing Inc. | System and methods for application activity capture, error identification, and error correction |
| CN110521171A (en) * | 2017-03-28 | 2019-11-29 | 思科技术公司 | The stream cluster parsing for monitoring and managing for application performance |
| CN108804237A (en) * | 2017-05-05 | 2018-11-13 | 北京京东尚科信息技术有限公司 | Data real-time statistical method, device, storage medium and electronic equipment |
| CN107330003A (en) * | 2017-06-12 | 2017-11-07 | 上海藤榕网络科技有限公司 | Method of data synchronization, system, memory and data syn-chronization equipment |
| CN109829009A (en) * | 2018-12-28 | 2019-05-31 | 北京邮电大学 | Configurable isomeric data real-time synchronization and visual system and method |
| CN110597914A (en) * | 2019-09-18 | 2019-12-20 | 北京思维造物信息科技股份有限公司 | Data transmission system, method, device and equipment |
| CN111209344A (en) * | 2020-02-07 | 2020-05-29 | 浪潮软件股份有限公司 | Data synchronization method and device |
Non-Patent Citations (2)
| Title |
|---|
| R. SHREE: ""KAFKA: The modern platform for data management and analysis in big data domain"", 《2017 2ND INTERNATIONAL CONFERENCE ON TELECOMMUNICATION AND NETWORKS (TEL-NET)》, 31 December 2017 (2017-12-31), pages 1 - 5 * |
| 杨兴文: ""AETA 数据采集系统的设计与实现"", 《北京大学学报(自然科学版)》, 31 July 2020 (2020-07-31), pages 600 - 606 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113064906A (en) * | 2021-04-21 | 2021-07-02 | 杭州天谷信息科技有限公司 | Binlog log data adaptive migration method and system |
| CN113220757A (en) * | 2021-04-24 | 2021-08-06 | 上海钢银科技发展有限公司 | Real-time warehouse automatic ETL method, system, equipment and computer storage medium |
| CN113297216A (en) * | 2021-05-17 | 2021-08-24 | 中国人民解放军63920部队 | Real-time storage method for space flight measurement and control data |
| CN113297216B (en) * | 2021-05-17 | 2024-06-11 | 中国人民解放军63920部队 | Real-time warehousing method for aerospace measurement and control data |
| CN113468170A (en) * | 2021-06-02 | 2021-10-01 | 上海赛可出行科技服务有限公司 | System for automatically realizing real-time data synchronization |
| CN113821407B (en) * | 2021-09-15 | 2023-08-01 | 浙江浙大网新软件产业集团有限公司 | Storm distributed real-time computing method and system |
| CN113821407A (en) * | 2021-09-15 | 2021-12-21 | 浙江网新恩普软件有限公司 | Storm distributed real-time computing method and system |
| CN113742392A (en) * | 2021-09-17 | 2021-12-03 | 中国银行股份有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN115269563A (en) * | 2022-09-26 | 2022-11-01 | 北京奥星贝斯科技有限公司 | Method and device for performing log analysis on database system |
| CN115269563B (en) * | 2022-09-26 | 2022-12-09 | 北京奥星贝斯科技有限公司 | Method and device for performing log analysis on database system |
| CN115470217A (en) * | 2022-11-14 | 2022-12-13 | 云筑信息科技(成都)有限公司 | Method for solving change response problem of data bin model in real time |
| CN117633104A (en) * | 2023-11-06 | 2024-03-01 | 湖北华中电力科技开发有限责任公司 | Method and device for realizing supervision work order collaboration through incremental data channel |
| CN117555699A (en) * | 2024-01-11 | 2024-02-13 | 杭州剑齿虎信息技术有限公司 | LCK real-time acquisition system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112445863B (en) | 2024-06-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112445863B (en) | Data real-time synchronization method and system | |
| EP3602341B1 (en) | Data replication system | |
| US10210221B2 (en) | System and method for distributed database query engines | |
| US6009271A (en) | Method of retrieving data from a relational database | |
| US10296629B2 (en) | Server supporting a consistent client-side cache | |
| CN107835983A (en) | Backup-and-restore is carried out in distributed data base using consistent database snapshot | |
| CN111125260A (en) | Data synchronization method and system based on SQL Server | |
| CN111506556A (en) | Multi-source heterogeneous structured data synchronization method | |
| JP2014512608A (en) | Method and system for loading data into a temporary data warehouse | |
| CN111930768B (en) | Incremental data acquisition method, incremental data transmission method, incremental data acquisition device, incremental data transmission device and computer storage medium | |
| EP2767912A2 (en) | In-memory real-time synchronized database system and method | |
| CN107180113A (en) | A kind of big data searching platform | |
| US20180032567A1 (en) | Method and device for processing data blocks in a distributed database | |
| CN110161930B (en) | Data monitoring system and data monitoring method thereof | |
| CN118331962A (en) | Data processing method, device, storage medium and database based on time schedule | |
| CN111209314A (en) | System for processing massive log data of power information system in real time | |
| CN107004036B (en) | Method and system for searching logs containing a large number of entries | |
| CN116383207A (en) | Data tag management method and device, electronic equipment and storage medium | |
| CN116594982A (en) | Flow number bin construction method based on rule engine and Clickhouse | |
| CN111752918A (en) | Historical data interaction system and configuration method thereof | |
| CN110941536A (en) | Monitoring method and system, and first server cluster | |
| CN113986656B (en) | Power grid data safety monitoring system based on data center platform | |
| CN115470192A (en) | Method for collecting and managing metadata of database | |
| CN115098486A (en) | Real-time data acquisition method based on customs service big data | |
| CN105205168A (en) | Exposure system based on Redis database and operation method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |