CN111770145A - One-way network bilateral data synchronization system and method based on log analysis - Google Patents
One-way network bilateral data synchronization system and method based on log analysis Download PDFInfo
- Publication number
- CN111770145A CN111770145A CN202010553458.5A CN202010553458A CN111770145A CN 111770145 A CN111770145 A CN 111770145A CN 202010553458 A CN202010553458 A CN 202010553458A CN 111770145 A CN111770145 A CN 111770145A
- Authority
- CN
- China
- Prior art keywords
- log
- intranet
- extranet
- antboe
- service database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 27
- 230000002146 bilateral effect Effects 0.000 title claims abstract description 16
- 238000012544 monitoring process Methods 0.000 claims abstract description 7
- 230000002457 bidirectional effect Effects 0.000 abstract description 4
- 238000004891 communication Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 231100000989 no adverse effect Toxicity 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a log analysis-based one-way network bilateral data synchronization system and a log analysis-based one-way network bilateral data synchronization method, which comprise an intranet service database, an intranet AntBOE, an extranet message queue, an extranet AntBOE and an extranet service database, wherein the intranet service database has a log opening function, and reads the log to obtain the log information, the intranet AntBOE has the function of configuring the connection of the intranet and intranet service databases, and generates timing operation on the external network AntBOE by calling an external network interface, the internal network AntBOE can produce timing operation by configuring the connection of the internal and external network service databases, and packages the timing operation into a message to be pushed to an extranet message queue, the intranet AntBOE acquires the message by monitoring the message queue, unpacking the message to update data of an internal network service database, wherein the external network service database has a log starting function and reads a log to acquire log information; the invention realizes the bidirectional synchronization of the database data of the internal and external network unidirectional network.
Description
Technical Field
The invention relates to a data synchronization system, in particular to a log analysis-based one-way network bilateral data synchronization system and a log analysis-based one-way network bilateral data synchronization method.
Background
With the development of big data, the security of data information becomes more and more important, and for the protection of data, governments or enterprises often store data in an intranet database environment with higher security level and isolated from the internet or public network environment. Once the data is generated, there will be shared demand, regardless of territory, regardless of network environment. As data becomes more and more demanding. But services such as applications, interfaces and the like deployed in public networks and open to the public cannot call, read and update the data stored in the intranet database due to the physical environment isolation and the limitation of network one-way communication. At the moment, partial data which is low in sensitivity and can be shared openly needs to be stored on a public network database for use. The problem of how to maintain the intra-and extranet data consistency is faced at this time. The following data synchronization methods are currently available in the face of this situation.
1. A snapshot method: a Snapshot (Snapshot) is an instant image of a storage object in a database at a certain time. By defining a snapshot or similar method for the synchronization object, its current image can be used as the content of the updated copy. The method can only synchronize the whole database, cannot selectively synchronize data, and has poor real-time performance.
2. The trigger method comprises the following steps: the method can affect the performance of the source database, and the bidirectional synchronization can be triggered repeatedly and circularly and can only be used for unidirectional synchronization.
Time stamping method: the method adds a time stamp field for each table needing synchronization, and the original application system records the modification time of each record in the column. The method does not affect the running efficiency of the original application, but needs to make large adjustment on the original system, and can not capture the operation data change caused by the application system, and is difficult to process the deleted record synchronization.
Disclosure of Invention
The present invention provides a log analysis-based system and method for synchronizing bilateral data in a unidirectional network, so as to solve the problems in the background art.
In order to achieve the purpose, the invention provides the following technical scheme:
a log analysis-based one-way network bilateral data synchronization system comprises an intranet service database, an intranet ANtBOE, an extranet message queue, an extranet ANtBOE and an extranet service database, wherein the intranet service database has a log opening function and reads a log to acquire log information, the intranet ANtBOE has a function of configuring connection of the intranet and extranet service databases and generates timing operation in the extranet ANtBOE by calling an extranet interface, the intranet ANtBOE can produce timing operation by configuring connection of the intranet and extranet service databases and packages the timing operation into a message to be pushed to the extranet message queue, the intranet ANtBOE acquires the message by monitoring the message queue and unpacks the message to update data in the intranet service database, the extranet service database has a log opening function and reads the log to acquire log information, and the extranet ANtBOE can produce timing operation through the log information acquired by the extranet service database, and packaging the timing operation into a message and pushing the message to an extranet message queue, wherein the extranet AntBOE acquires the message by monitoring the message queue and unpacks the message to update data of an extranet service database.
A log analysis-based one-way network bilateral data synchronization method comprises the following steps:
the method comprises the following steps: an intranet service database starts a log function, an extranet service database starts the log function, and intranet ANtBOE configures intranet and extranet service data connection;
step two: the intranet ANtBOE generates timing operation to read and analyze the intranet service database logs, packages the timing operation into messages and pushes the messages to an extranet message queue; the external network AntBE generates timing operation to read and analyze the log of the external network service database, and packages the log into a message to be pushed to an external network message queue;
step three: the timing operation of the intranet ANtBOE monitors the extranet message queue to obtain the message pushed by the extranet ANtBOE, unpacks the message and updates the data of the intranet service database; and the timing operation of the external network AntBE monitors the external network message queue to acquire the message pushed by the internal network AntBE, unpacks the message and updates the data of the external network service database.
As a further scheme of the invention: and recording the current time when the timing operation in the step two reads the log, and reading the log from the last recording time when the log is read next time.
As a further scheme of the invention: the timing operation can calculate the time interval according to the last reading time and judge whether to read the online log or to archive the log.
As a still further scheme of the invention: the synchronization data is updated by selecting a different database user than the data generated to prevent two-way cycle repeat synchronization.
Compared with the prior art, the invention has the beneficial effects that:
1. the invention realizes the bidirectional synchronization of the database data of the internal and external network unidirectional network.
2. The invention has no adverse effect on the performance of the database based on the database log analysis mode.
3. The invention shortens the synchronization real-time to the second level based on the message queue communication.
4. According to the invention, the synchronous data 0 is lost due to two data backups of the data log and the message queue, and 100% of unexpected data such as power failure can be restored.
5. The cluster service of the invention ensures that synchronization cannot be stopped due to accidents such as downtime and the like.
Drawings
Fig. 1 is a block diagram of a system and method for one-way network bilateral data synchronization based on log parsing.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, in an embodiment of the present invention, a log-parsing-based one-way network two-sided data synchronization method includes an intranet service database, an intranet AntBee, an extranet message queue, an extranet AntBee, and an extranet service database, where the intranet service database has a log starting function and reads a log to obtain log information, the intranet AntBee has a function of configuring connection between the intranet and the extranet service database and generates a timing job on the extranet AntBee by invoking an extranet interface, the intranet AntBee can generate the timing job by configuring connection between the intranet and the extranet service database and packages the timing job into a message to be pushed to the extranet message queue, the intranet AntBee obtains the message by monitoring the message queue and unpacks the message to update data of the intranet service database, the extranet service database has a log starting function and reads the log to obtain log information, the external network AntBOE can produce timing operation through the log information acquired by the external network service database, packages the timing operation into messages and pushes the messages to the external network message queue, acquires the messages through monitoring the message queue, and unpacks the messages to update data of the external network service database.
A log analysis-based one-way network bilateral data synchronization method comprises the following steps:
the method comprises the following steps: an intranet service database starts a log function, an extranet service database starts the log function, and intranet ANtBOE configures intranet and extranet service data connection;
step two: the intranet ANtBOE generates timing operation to read and analyze the intranet service database logs, packages the timing operation into messages and pushes the messages to an extranet message queue; the external network AntBE generates timing operation to read and analyze the log of the external network service database, and packages the log into a message to be pushed to an external network message queue;
step three: the timing operation of the intranet ANtBOE monitors the extranet message queue to obtain the message pushed by the extranet ANtBOE, unpacks the message and updates the data of the intranet service database; and the timing operation of the external network AntBE monitors the external network message queue to acquire the message pushed by the internal network AntBE, unpacks the message and updates the data of the external network service database.
And recording the current time when the log is read by the timing operation in the step two, and reading the log from the last recording time when the log is read next time, so as to prevent repeated reading.
The timing operation can calculate the time interval according to the last reading time and judge whether to read the online log or to archive the log.
The synchronization data is updated by selecting a different database user than the data generated to prevent two-way cycle repeat synchronization.
The bidirectional synchronization of the database data of the internal and external network unidirectional network is realized.
The system has no adverse effect on the performance of the database based on a database log analysis mode; the system can shorten the synchronization real-time property to the second level based on the message queue communication; the data log and the message queue of the system are subjected to data backup twice, so that synchronous data are zero lost, and unexpected data such as power failure can be restored by 100%; the cluster service of the system ensures that synchronization cannot be stopped due to accidents such as downtime and the like.
The system supports the database table selected by synchronization and customization, supports the database statement execution user selected by synchronization and customization, supports the database table attribution user selected by synchronization and customization, supports the database execution statement type selected by synchronization and customization, and supports the customized synchronization time interval; synchronization can be realized by simple configuration at a WEB end.
Although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that various changes in the embodiments and/or modifications of the invention can be made, and equivalents and modifications of some features of the invention can be made without departing from the spirit and scope of the invention.
Claims (5)
1. A one-way network bilateral data synchronization system based on log analysis comprises an intranet service database, an intranet AntBOE, an extranet message queue, an extranet AntBOE and an extranet service database, and is characterized in that: the intranet ANtBOE has a function of starting the log and reads the log to obtain log information, the intranet ANtBOE has a function of configuring connection of the intranet and the extranet service databases and generates timing operation on the extranet ANtBOE by calling an extranet interface, the intranet ANtBOE can produce timing operation by configuring connection of the intranet and the extranet service databases and packages the timing operation into messages to be pushed to an extranet message queue, the intranet ANtBOE acquires the messages by monitoring the message queue and unpacks the messages to update data of the intranet service database, the extranet service database has a function of starting the log and reads the log to acquire the log information, the extranet ANtBOE can produce the timing operation through the log information acquired by the extranet service database and packages the timing operation into messages to be pushed to the extranet message queue, and the extranet ANtBOE acquires the messages by monitoring the message queue, and unpack the message to update data to the extranet service database.
2. A log analysis-based one-way network bilateral data synchronization method is characterized in that: the method comprises the following steps:
the method comprises the following steps: an intranet service database starts a log function, an extranet service database starts the log function, and intranet ANtBOE configures intranet and extranet service data connection;
step two: the intranet ANtBOE generates timing operation to read and analyze the intranet service database logs, packages the timing operation into messages and pushes the messages to an extranet message queue; the external network AntBE generates timing operation to read and analyze the log of the external network service database, and packages the log into a message to be pushed to an external network message queue;
step three: the timing operation of the intranet ANtBOE monitors the extranet message queue to obtain the message pushed by the extranet ANtBOE, unpacks the message and updates the data of the intranet service database; and the timing operation of the external network AntBE monitors the external network message queue to acquire the message pushed by the internal network AntBE, unpacks the message and updates the data of the external network service database.
3. The log parsing-based one-way network bilateral data synchronization method according to claim 2, wherein: and recording the current time when the timing operation in the step two reads the log, and reading the log from the last recording time when the log is read next time.
4. The log parsing-based one-way network bilateral data synchronization method according to claim 3, wherein: the timing operation can calculate the time interval according to the last reading time and judge whether to read the online log or to archive the log.
5. The log parsing-based one-way network bilateral data synchronization method according to claim 2, wherein: the synchronization data is updated by selecting a different database user than the data generated to prevent two-way cycle repeat synchronization.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010553458.5A CN111770145A (en) | 2020-06-17 | 2020-06-17 | One-way network bilateral data synchronization system and method based on log analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010553458.5A CN111770145A (en) | 2020-06-17 | 2020-06-17 | One-way network bilateral data synchronization system and method based on log analysis |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111770145A true CN111770145A (en) | 2020-10-13 |
Family
ID=72720818
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010553458.5A Pending CN111770145A (en) | 2020-06-17 | 2020-06-17 | One-way network bilateral data synchronization system and method based on log analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111770145A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113360475A (en) * | 2021-06-18 | 2021-09-07 | 广州中爆数字信息科技股份有限公司 | Data operation and maintenance method, device and equipment based on intranet terminal and storage medium |
| CN114911872A (en) * | 2022-07-18 | 2022-08-16 | 北京智芯微电子科技有限公司 | Intranet and extranet data synchronization method, device and system, extranet server and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9294340B1 (en) * | 2014-10-07 | 2016-03-22 | Google Inc. | Systems and methods for updating data across multiple network architectures |
| CN105516094A (en) * | 2015-11-27 | 2016-04-20 | 蓝网科技股份有限公司 | Industrial computer based internal-external network data exchange method and apparatus |
| CN107579971A (en) * | 2017-08-31 | 2018-01-12 | 上海欣方智能系统有限公司 | A kind of intranet and extranet method for interchanging data based on security platform |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN108897895A (en) * | 2018-07-12 | 2018-11-27 | 浪潮软件集团有限公司 | Device and method for realizing cross-gateway Mysql database synchronization based on cloud message service platform |
| CN111221664A (en) * | 2019-12-24 | 2020-06-02 | 视联动力信息技术股份有限公司 | Operation log processing system |
-
2020
- 2020-06-17 CN CN202010553458.5A patent/CN111770145A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9294340B1 (en) * | 2014-10-07 | 2016-03-22 | Google Inc. | Systems and methods for updating data across multiple network architectures |
| CN105516094A (en) * | 2015-11-27 | 2016-04-20 | 蓝网科技股份有限公司 | Industrial computer based internal-external network data exchange method and apparatus |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN107579971A (en) * | 2017-08-31 | 2018-01-12 | 上海欣方智能系统有限公司 | A kind of intranet and extranet method for interchanging data based on security platform |
| CN108897895A (en) * | 2018-07-12 | 2018-11-27 | 浪潮软件集团有限公司 | Device and method for realizing cross-gateway Mysql database synchronization based on cloud message service platform |
| CN111221664A (en) * | 2019-12-24 | 2020-06-02 | 视联动力信息技术股份有限公司 | Operation log processing system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113360475A (en) * | 2021-06-18 | 2021-09-07 | 广州中爆数字信息科技股份有限公司 | Data operation and maintenance method, device and equipment based on intranet terminal and storage medium |
| CN114911872A (en) * | 2022-07-18 | 2022-08-16 | 北京智芯微电子科技有限公司 | Intranet and extranet data synchronization method, device and system, extranet server and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020147392A1 (en) | Method and system for data synchronization between databases | |
| WO2019154394A1 (en) | Distributed database cluster system, data synchronization method and storage medium | |
| US10621049B1 (en) | Consistent backups based on local node clock | |
| US8626778B2 (en) | System and method for conversion of JMS message data into database transactions for application to multiple heterogeneous databases | |
| CN112507029B (en) | Data processing system and data real-time processing method | |
| CN110569269A (en) | data synchronization method and system | |
| CN106503158B (en) | Data synchronization method and device | |
| CN109298978B (en) | Recovery method and system for database cluster of specified position | |
| US9971779B2 (en) | Automated data intake system | |
| CN111770145A (en) | One-way network bilateral data synchronization system and method based on log analysis | |
| CN112039701B (en) | Interface call monitoring method, device, equipment and storage medium | |
| CN111432025A (en) | Cloud edge cooperation-oriented distributed service directory management method and system | |
| CN108228755A (en) | The data of MySQL database based on daily record analytic technique to Hadoop platform synchronize clone method | |
| CN114048217A (en) | Incremental data synchronization method and device, electronic equipment and storage medium | |
| CN111222089A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN112269823A (en) | Method and system for realizing synchronization of PostgreSQL incremental data | |
| CN112328702A (en) | Data synchronization method and system | |
| Jannes et al. | OWebSync: Seamless synchronization of distributed web clients | |
| CN114661823A (en) | Data synchronization method and device, electronic equipment and readable storage medium | |
| CN113486095A (en) | Civil aviation air traffic control cross-network safety data exchange management platform | |
| CN110417892B (en) | Message analysis-based data replication link optimization method and device | |
| CN114860782B (en) | Data query method, device, equipment and medium | |
| CN114500289B (en) | Control plane recovery method, device, control node and storage medium | |
| CN113239005B (en) | I and IV area data synchronization method and device for power monitoring system | |
| WO2022193855A1 (en) | Task state updating method and apparatus, device, and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201013 |