CN111008246A - Database log synchronization method and device, computer equipment and readable storage medium - Google Patents
Database log synchronization method and device, computer equipment and readable storage medium Download PDFInfo
- Publication number
- CN111008246A CN111008246A CN201911171078.9A CN201911171078A CN111008246A CN 111008246 A CN111008246 A CN 111008246A CN 201911171078 A CN201911171078 A CN 201911171078A CN 111008246 A CN111008246 A CN 111008246A
- Authority
- CN
- China
- Prior art keywords
- log
- binlog
- storage module
- synchronization
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000004458 analytical method Methods 0.000 claims abstract description 18
- 238000004590 computer program Methods 0.000 claims description 9
- 238000013075 data extraction Methods 0.000 claims description 6
- 238000007405 data analysis Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 abstract description 10
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides a database log synchronization method, a database log synchronization device, computer equipment and a readable storage medium, wherein the method comprises the following steps: simulating slave nodes of MySQL by pseudo slave node synchronization service, and extracting binlog logs from the MySQL; storing, by the pseudo slave node synchronization service, the extracted binlog log in a storage module of the processing engine; and reading and analyzing the binlog log from the storage module through the analysis service, and storing an analysis result into a target storage. The scheme ensures that the whole binlog log synchronization process of MySQL can be completed in the processing engine through programs, avoids introducing additional components such as third-party synchronization tools, message queues or middleware such as JMQ and the like, and simultaneously can avoid the cooperation of multiple components in the binlog log synchronization process of MySQL, thereby being beneficial to simplifying the processing flow.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a database log synchronization method and apparatus, a computer device, and a readable storage medium.
Background
Real-time data analysis shows that direct query of a MySQL (database) table or frequent data extraction can generate great performance pressure on the MySQL, so that the currently adopted method is to synchronize the binlog of the MySQL, analyze the binlog and store the binlog in a target for storage, and avoid generating the performance pressure on the MySQL.
For example, in the conventional Binlog log synchronization processing flow, as shown in fig. 1, it is necessary to read Binlog using a synchronization tool (e.g., Maxwall or Canal), and then synchronize the Binlog to a message queue or middleware such as JMQ (Java message queue) to buffer data, and a data processing engine (e.g., a big data computing engine such as flash, Spark, etc.) consumes the Binlog of the message queue, and parses the Binlog and saves the Binlog to a target repository (e.g., Hbase, elastic search, etc.). The processing flow needs a third-party synchronization tool, middleware such as a message queue or JMQ, and a data processing engine to cooperate with each other, so that the flow is relatively complicated.
Disclosure of Invention
The embodiment of the invention provides a database log synchronization method, which aims to solve the technical problem of complicated flow in the prior art of binlog log synchronization. The method comprises the following steps:
simulating slave nodes of MySQL by pseudo slave node synchronization service, and extracting binlog logs from the MySQL;
storing, by the pseudo slave node synchronization service, the extracted binlog log in a storage module of the processing engine;
and reading and analyzing the binlog log from the storage module through the analysis service, and storing an analysis result into a target storage.
The embodiment of the invention also provides a database log synchronization device, which is used for solving the technical problem of complicated flow in the binlog log synchronization in the prior art. The device includes:
the data extraction unit is used for simulating slave nodes of MySQL through pseudo slave node synchronization service and extracting binlog logs from the MySQL;
a storage unit for storing the extracted binlog log in a storage module of the processing engine through the pseudo slave node synchronization service;
and the data analysis unit is used for reading the binlog log from the storage module through the analysis service, analyzing the binlog, and storing an analysis result into the target storage.
The embodiment of the invention also provides computer equipment which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor realizes the random database log synchronization method when executing the computer program so as to solve the technical problem of complicated flow existing in binlog log synchronization in the prior art.
The embodiment of the invention also provides a computer readable storage medium, which stores a computer program for executing the arbitrary database log synchronization method, so as to solve the technical problem of complicated processes in the prior art for the synchronization of the binlog logs.
In an embodiment of the present invention, there is provided the above-mentioned database log synchronization method executed in a processing engine, the slave node of MySQL is simulated by the pseudo slave node synchronization service, the binlog log is extracted from MySQL, the binlog log is prevented from being extracted by a third-party synchronization tool, storing the extracted binlog log in a storage module of the processing engine by the pseudo slave node synchronization service, avoiding introducing middleware such as message queue or JMQ, reading binlog log from the storage module by analysis service, analyzing, storing the analysis result in the target storage, the whole binlog log synchronization process of MySQL can be completed in the processing engine by programs, so that additional components such as third-party synchronization tools, message queues or middleware such as JMQ and the like are avoided, meanwhile, the cooperation of multi-party components can be avoided in the synchronization process of the binlog of MySQL, and the simplification of the processing flow is facilitated.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principles of the invention. In the drawings:
FIG. 1 is a schematic diagram of a prior art binlog log processing flow;
FIG. 2 is a flowchart of a database log synchronization method according to an embodiment of the present invention;
fig. 3 is a schematic flowchart of a database log synchronization method according to an embodiment of the present invention;
FIG. 4 is a block diagram of a computer device according to an embodiment of the present invention;
fig. 5 is a block diagram of a database log synchronization apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the following embodiments and accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
In an embodiment of the present invention, a database log synchronization method is provided, which may be executed in a processing engine, as shown in fig. 2, and includes:
step 202: simulating slave nodes of MySQL by pseudo slave node synchronization service, and extracting binlog logs from the MySQL;
step 204: storing, by the pseudo slave node synchronization service, the extracted binlog log in a storage module of the processing engine;
step 206: and reading and analyzing the binlog log from the storage module through the analysis service, and storing an analysis result into a target storage.
It can be known from the flow shown in fig. 2 that, in the embodiment of the present invention, the above-mentioned database log synchronization method operating in the processing engine is proposed, a slave node of MySQL is simulated by the pseudo slave node synchronization service, a binlog log is extracted from MySQL, the binlog log is prevented from being extracted by a third-party synchronization tool, the extracted binlog is stored in the storage module of the processing engine by the pseudo slave node synchronization service, introduction of middleware such as a message queue or JMQ is avoided, finally, the binlog log is read from the storage module by the parsing service and parsed, and the parsed result is stored in the target storage, so that the whole binlog log synchronization process of MySQL can be completed in the processing engine by a program, introduction of additional components such as middleware such as a third-party synchronization tool, a message queue or JMQ is avoided, and cooperation of multiple components can be avoided in the binlog synchronization process of MySQL, is beneficial to simplifying the processing flow.
In specific implementation, the pseudo slave node synchronization service can use an interface component MySQL-binlog-connector-java of a third party to simulate a slave node of MySQL, and extract and analyze a binlog log from the MySQL. The pseudo slave node synchronization service can monitor the binlog log by adopting a data synchronization thread according to options of a host, a port, an account password, a table name, the index of the binlog, the offset of the binlog and the like of MySQL, simulate slave nodes of MySQL and further synchronize the binlog from a Master node.
In specific implementation, in order to ensure the performance of data synchronization, in this embodiment, as shown in fig. 3, the storing the extracted binlog log in the storage module of the processing engine by the pseudo slave node synchronization service includes:
storing the extracted binlog log in a storage module of the processing engine in a manner of pre-writing a log by the pseudo slave node synchronization service. The pseudo slave node synchronization service stores the extracted binlog log in a storage module of the processing engine in a log pre-writing mode, at the moment, no data is processed, the subsequent analysis service reads the binlog log from the storage module and analyzes the binlog, the introduction of middleware such as a message queue/JMQ and the like is avoided, and frequent data pulling from the binlog of MySQL is also avoided, because the direct consumption of data from MySQL can generate unnecessary operations of disk data reading, network IO and the like, so that the performance can be ensured, for example, when the data is abnormal or the binlog is consumed again, the data can be consumed from the storage module of the processing engine in the log pre-writing mode at first, and if the data in the log pre-writing mode is lost, the data is consumed from the binlog of the MySQL directly, and the log is prevented from being synchronized each time. Specifically, the storage module of the processing engine may be a local disk, a Hadoop distributed file system, or the like.
In specific implementation, in order to further ensure performance and stability of data extraction and analysis, in this embodiment, a checkpoint is further added in the binlog log synchronization method of MySQL, for example, as shown in fig. 3, the method further includes:
storing a synchronization offset of a binlog log in a storage module in a checkpoint by the pseudo slave node synchronization service each time the extracted binlog log is stored in the storage module of the processing engine;
each time a binlog log is read from a storage module, a parsing offset of the binlog log in the storage module is saved in a checkpoint by the parsing service.
In particular implementations, the synchronization offset may be the maximum offset (e.g., offset) in the current storage module when the extracted binlog log is stored in the storage module; the parse offset may be the maximum offset (e.g., offset) in the current storage module when the binlog log is read from the storage module. Specifically, each log message corresponds to an offset.
In a specific implementation, each time the extracted binlog log is stored in the storage module of the processing engine, after the synchronization offset of the binlog log in the storage module is saved in the checkpoint by the pseudo slave node synchronization service, the extracted binlog can be extracted based on the last saved synchronization offset (offset increment) in the next round of data extraction. Each time a binlog log is read from a storage module, a next round of data parsing may start from a last saved parsing offset (offset position) after the parsing offset of the binlog log in the storage module is saved in a checkpoint by the parsing service.
In specific implementation, in the process of storing the synchronization offset and analyzing the offset, the method for obtaining the initial offset may be: and using a MySQL brinlog tool of MySQL to specify options such as start and end time and further obtain the index and offset of the initial binlog.
Information such as the location of the binlog can be specified:
val optionsMap＝Map(
"host" - > "127.0.0.1"// specifying hostname
"port" - > "3306"// designated port
"userName" - > "root" -// specifying a user name
"password" - > "123456",// specifying password
"binlogindex" - > "4",// index specifying binlog
"offset" - > "4"// offset specifying a log
)
Pseudo code example:
in specific implementation, the analysis service can read the binlog log from the pre-written log (a local disk, a Hadoop distributed file system and the like) by means of an interface component mysql-binlog-connector-java of a third party and analyze the binlog, the operation type of analyzing each log record can be insertion, updating, deleting and the like, and the operation type is updated into the log record and then written into the target storage.
Specifically, the generated parsing result is an execution SQL statement of MySQL, such as an insert, update, delete statement, and the like. Related operations may be performed on insert, update, delete, etc. statements, respectively, based on downstream storage. Downstream storage needs to perform operations such as insert and update, so downstream storage can be Kudu, elastic search, Hbase, Tidb, etc.
In this embodiment, a computer device is provided, as shown in fig. 4, and includes a memory 402, a processor 404, and a computer program stored on the memory and executable on the processor, and the processor implements any of the above database log synchronization methods when executing the computer program.
In particular, the computer device may be a computer terminal, a server or a similar computing device.
In the present embodiment, there is provided a computer-readable storage medium storing a computer program for executing any of the database log synchronization methods described above.
In particular, computer-readable storage media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable storage medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
Based on the same inventive concept, the embodiment of the present invention further provides a database log synchronization apparatus, as described in the following embodiments. Because the principle of the database log synchronization device for solving the problems is similar to the database log synchronization method, the implementation of the database log synchronization device can refer to the implementation of the database log synchronization method, and repeated parts are not repeated. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 5 is a block diagram of a structure of a database log synchronization apparatus according to an embodiment of the present invention, and as shown in fig. 5, the apparatus includes:
a data extraction unit 502, configured to extract binlog logs from MySQL by simulating slave nodes of MySQL through a pseudo slave node synchronization service;
a storage unit 504 for storing the extracted binlog log in a storage module of the processing engine through the pseudo slave node synchronization service;
and a data parsing unit 506, configured to read the binlog log from the storage module through the parsing service, parse the binlog, and store a parsing result in the target storage.
In one embodiment, the storage unit is configured to store the extracted binlog log in a storage module of the processing engine in a manner of pre-writing a log through the pseudo slave node synchronization service.
In one embodiment, the above apparatus further comprises:
an offset storage unit for saving a synchronization offset of the binlog log in the storage module in a checkpoint by the pseudo slave node synchronization service every time the extracted binlog log is stored in the storage module of the processing engine; each time a binlog log is read from a storage module, a parsing offset of the binlog log in the storage module is saved in a checkpoint by the parsing service.
In one embodiment, the synchronization offset is the maximum offset in the current storage module when the extracted binlog log is stored in the storage module; the parsing offset is the maximum offset in the current storage module when the binlog log is read from the storage module.
The embodiment of the invention realizes the following technical effects: the binlog log is extracted from the MySQL by simulating the slave node of the MySQL through the pseudo slave node synchronization service, the binlog log is prevented from being extracted through a third-party synchronization tool, the extracted binlog log is stored in a storage module of the processing engine through the pseudo slave node synchronization service, middleware such as a message queue or JMQ is prevented from being introduced, the binlog log is read from the storage module through the analysis service and analyzed, and an analysis result is stored in a target storage, so that the whole binlog log synchronization process of the MySQL can be completed in the processing engine through a program, extra components such as middleware such as a third-party synchronization tool, a message queue or JMQ are prevented from being introduced, and meanwhile, the cooperation of multi-party components can be avoided in the binlog log synchronization process of the MySQL, and the processing flow is simplified.
It will be apparent to those skilled in the art that the modules or steps of the embodiments of the invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, embodiments of the invention are not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes may be made to the embodiment of the present invention by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A database log synchronization method, the method operating in a processing engine, the method comprising:
simulating slave nodes of MySQL by pseudo slave node synchronization service, and extracting binlog logs from the MySQL;
storing, by the pseudo slave node synchronization service, the extracted binlog log in a storage module of the processing engine;
and reading and analyzing the binlog log from the storage module through the analysis service, and storing an analysis result into a target storage.
2. The database log synchronization method of claim 1, wherein storing the extracted binlog log in a storage module of the processing engine by the pseudo slave node synchronization service comprises:
storing the extracted binlog log in a storage module of the processing engine in a manner of pre-writing a log by the pseudo slave node synchronization service.
3. The database log synchronization method of claim 1 or 2, further comprising:
storing a synchronization offset of a binlog log in a storage module in a checkpoint by the pseudo slave node synchronization service each time the extracted binlog log is stored in the storage module of the processing engine;
each time a binlog log is read from a storage module, a parsing offset of the binlog log in the storage module is saved in a checkpoint by the parsing service.
4. The database log synchronization method of claim 3, wherein the synchronization offset is a maximum offset in a current storage module when the extracted binlog log is stored in the storage module; the parsing offset is the maximum offset in the current storage module when the binlog log is read from the storage module.
5. A database log synchronization apparatus, comprising:
the data extraction unit is used for simulating slave nodes of MySQL through pseudo slave node synchronization service and extracting binlog logs from the MySQL;
a storage unit for storing the extracted binlog log in a storage module of a processing engine through the pseudo slave node synchronization service;
and the data analysis unit is used for reading the binlog log from the storage module through the analysis service, analyzing the binlog, and storing an analysis result into the target storage.
6. The database log synchronization apparatus of claim 5, wherein the storage unit is configured to store the extracted binlog log in a storage module of the processing engine in a pre-written log manner by the pseudo slave node synchronization service.
7. The database log synchronization apparatus of claim 5 or 6, further comprising:
an offset storage unit for saving a synchronization offset of the binlog log in the storage module in a checkpoint by the pseudo slave node synchronization service every time the extracted binlog log is stored in the storage module of the processing engine; each time a binlog log is read from a storage module, a parsing offset of the binlog log in the storage module is saved in a checkpoint by the parsing service.
8. The database log synchronization apparatus of claim 7, wherein the synchronization offset is a maximum offset in a current storage module when the extracted binlog log is stored in the storage module; the parsing offset is the maximum offset in the current storage module when the binlog log is read from the storage module.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the database log synchronization method of any of claims 1 to 4 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the database log synchronization method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911171078.9A CN111008246B (en) | 2019-11-26 | 2019-11-26 | Database log synchronization method, device, computer equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911171078.9A CN111008246B (en) | 2019-11-26 | 2019-11-26 | Database log synchronization method, device, computer equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111008246A true CN111008246A (en) | 2020-04-14 |
| CN111008246B CN111008246B (en) | 2024-04-19 |
Family
ID=70113799
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911171078.9A Active CN111008246B (en) | 2019-11-26 | 2019-11-26 | Database log synchronization method, device, computer equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111008246B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112015821A (en) * | 2020-09-08 | 2020-12-01 | 香港中文大学（深圳） | Database synchronization method, terminal and storage medium |
| CN112100147A (en) * | 2020-07-27 | 2020-12-18 | 杭州玳数科技有限公司 | Method and system for realizing real-time acquisition from Bilog to HIVE based on Flink |
| CN112162904A (en) * | 2020-09-25 | 2021-01-01 | 同程网络科技股份有限公司 | Order change process integration method, order change process extraction method, order change process integration device and order change process extraction device |
| CN112948378A (en) * | 2021-02-04 | 2021-06-11 | 上海中通吉网络技术有限公司 | Data processing method, device and equipment based on HBase |
| CN114328750A (en) * | 2021-12-31 | 2022-04-12 | 北京发现角科技有限公司 | Method and device for synchronizing service data with ODS (oxide dispersion strengthened) layer |
| CN117032950A (en) * | 2023-07-10 | 2023-11-10 | 企迈科技有限公司 | Real-time data transparent transmission method and system based on log |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120030172A1 (en) * | 2010-07-27 | 2012-02-02 | Oracle International Corporation | Mysql database heterogeneous log based replication |
| WO2016101752A1 (en) * | 2014-12-22 | 2016-06-30 | 北京奇虎科技有限公司 | Method and device for data synchronization |
| CN109753531A (en) * | 2018-12-26 | 2019-05-14 | 深圳市麦谷科技有限公司 | A kind of big data statistical method, system, computer equipment and storage medium |
-
2019
- 2019-11-26 CN CN201911171078.9A patent/CN111008246B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120030172A1 (en) * | 2010-07-27 | 2012-02-02 | Oracle International Corporation | Mysql database heterogeneous log based replication |
| WO2016101752A1 (en) * | 2014-12-22 | 2016-06-30 | 北京奇虎科技有限公司 | Method and device for data synchronization |
| CN109753531A (en) * | 2018-12-26 | 2019-05-14 | 深圳市麦谷科技有限公司 | A kind of big data statistical method, system, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 何小川;: "MySQL数据库主从复制的实现", 广东通信技术 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100147A (en) * | 2020-07-27 | 2020-12-18 | 杭州玳数科技有限公司 | Method and system for realizing real-time acquisition from Bilog to HIVE based on Flink |
| CN112100147B (en) * | 2020-07-27 | 2024-06-07 | 杭州玳数科技有限公司 | Method and system for realizing real-time acquisition from Binlog to HIVE based on Flink |
| CN112015821A (en) * | 2020-09-08 | 2020-12-01 | 香港中文大学（深圳） | Database synchronization method, terminal and storage medium |
| CN112015821B (en) * | 2020-09-08 | 2023-11-07 | 香港中文大学（深圳） | Database synchronization method, terminal and storage medium |
| CN112162904A (en) * | 2020-09-25 | 2021-01-01 | 同程网络科技股份有限公司 | Order change process integration method, order change process extraction method, order change process integration device and order change process extraction device |
| CN112948378A (en) * | 2021-02-04 | 2021-06-11 | 上海中通吉网络技术有限公司 | Data processing method, device and equipment based on HBase |
| CN114328750A (en) * | 2021-12-31 | 2022-04-12 | 北京发现角科技有限公司 | Method and device for synchronizing service data with ODS (oxide dispersion strengthened) layer |
| CN117032950A (en) * | 2023-07-10 | 2023-11-10 | 企迈科技有限公司 | Real-time data transparent transmission method and system based on log |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111008246B (en) | 2024-04-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111008246B (en) | Database log synchronization method, device, computer equipment and readable storage medium | |
| CN110147411B (en) | Data synchronization method, device, computer equipment and storage medium | |
| CN109460349B (en) | Test case generation method and device based on log | |
| CN111339041B (en) | File analysis and storage method and device and file generation method and device | |
| CN106202235B (en) | Data processing method and device | |
| CN110275916B (en) | Data operation record storage method, system, device and equipment | |
| CN111209344A (en) | Data synchronization method and device | |
| CN112182104A (en) | Data synchronization method, device, equipment and storage medium | |
| CN110362547B (en) | Method and device for encoding, analyzing and storing log file | |
| US20160117376A1 (en) | Synchronizing object in local object storage node | |
| CN113704790A (en) | Abnormal log information summarizing method and computer equipment | |
| CN111367948B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN115390847A (en) | Log processing method and device, computer readable storage medium and terminal | |
| CN113468196B (en) | Method, apparatus, system, server and medium for processing data | |
| CN109460389B (en) | OpenResty-based log recording method | |
| CN114398217A (en) | Massive simulation waveform data slicing method for FPGA parallel simulation | |
| CN112000971B (en) | File permission recording method, system and related device | |
| CN112883088B (en) | Data processing method, device, equipment and storage medium | |
| CN111047427A (en) | Data reporting method, device, server and storage medium | |
| CN113625952B (en) | Object storage method, device, equipment and storage medium | |
| CN107391708B (en) | Method and device for storing performance statistics file in storage system | |
| CN115757639A (en) | Data source synchronization method and device, electronic equipment and storage medium | |
| CN111459411B (en) | Data migration method, device, equipment and storage medium | |
| CN115391457A (en) | Cross-database data synchronization method and device and storage medium | |
| CN110287172B (en) | Method for formatting HBase data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |