CN111400394A - Distributed database real-time synchronization method across safety regions - Google Patents
Distributed database real-time synchronization method across safety regions Download PDFInfo
- Publication number
- CN111400394A CN111400394A CN201910801533.2A CN201910801533A CN111400394A CN 111400394 A CN111400394 A CN 111400394A CN 201910801533 A CN201910801533 A CN 201910801533A CN 111400394 A CN111400394 A CN 111400394A
- Authority
- CN
- China
- Prior art keywords
- data
- real
- time
- synchronization
- log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 15
- 238000002955 isolation Methods 0.000 claims abstract description 26
- 238000001914 filtration Methods 0.000 claims abstract description 14
- 238000011084 recovery Methods 0.000 claims abstract description 14
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 230000001360 synchronised effect Effects 0.000 claims description 3
- 230000003993 interaction Effects 0.000 claims description 2
- 238000005065 mining Methods 0.000 abstract description 2
- 238000012544 monitoring process Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 4
- 238000013316 zoning Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Bioethics (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a distributed relational database log-based cross-safe-area data real-time synchronization method which comprises the steps of obtaining, analyzing, converting and synchronizing incremental logs of a distributed relational database in real time. The method realizes real-time acquisition of incremental data through a subscription mode in a multi-tenant mode, realizes performance load and data isolation through the tenant mode, guarantees performance and data safety, analyzes the incremental logs in real time and converts the incremental logs into proprietary protocol data, realizes real-time synchronization of the data through a safety interface of an isolation device, guarantees information safety, and guarantees the reliability of the data through a data recovery point mechanism. The invention has a perfect data check point mechanism, a flexible data filtering rule and a flexible data synchronization exception handling mechanism, can realize the real-time synchronization of the data from the external network to the internal network, and lays a foundation for the real-time data monitoring, the cross-data associated service query analysis and the data value mining under the internal and external network integrated deployment mode.
Description
Technical Field
The invention belongs to the technical field of data synchronization, and particularly relates to a distributed database real-time synchronization aspect across safe regions.
Background
With the rapid development of computer and database technologies, the application range of databases is more and more extensive, and meanwhile, databases have moved from centralized to distributed, and data simultaneously exist on different servers with scattered positions, so that the high-reliability operation of enterprise systems is ensured.
The internal and external network integrated independent deployment mode is a traditional large-centralized large-scale system construction mode, unified support of an external internet mode and an internal local area network mode is realized on the basis, information safety is realized through a 'zoning and zoning' mode, under the internal and external network integrated mode, safety isolation gateway equipment exists between an information external network and an information internal network, sensitive data such as client certificates and addresses are stored in the information internal network and the information internal network respectively according to the sensitivity degree of the data, and non-sensitive data such as user accounts are stored in the information external network.
Disclosure of Invention
The purpose of the invention is as follows: the invention aims to solve the problem of data real-time synchronization of cross-isolation device equipment in an internal and external network integrated mode, and is based on a real-time increment log of a distributed database, a data recovery point mechanism and a multi-tenant mechanism, the real-time synchronization of external information network data to an information internal network is realized by analyzing and decrypting the distributed data log and converting the distributed data log into a special data specification of the isolation device equipment, the real-time performance of data synchronization is realized by a subscription mode, and the safety of the data is realized by the special data specification of the isolation device equipment.
The technical scheme is as follows: in order to achieve the above object, the method for real-time synchronization of distributed databases across secure areas according to the present invention comprises the following steps:
(1) acquiring real-time incremental log data, and solving the problems of instantaneity of acquisition of incremental data of a distributed database and multi-tenant channels;
(2) converting real-time incremental log data, and solving the problem of private protocol conversion of a distributed database log decryption and isolation device;
(3) synchronizing real-time incremental data, and decrypting data synchronization and data recovery point mechanism from an external network to an internal network.
The real-time incremental log data acquisition in the step (1) comprises the following steps:
(1.1) evaluating the data scale of a business database table, making a multi-tenant configuration strategy, and configuring a multi-tenant real-time data increment channel in a distributed database;
(1.2) modifying data synchronization parameter configuration, including tenant name, tenant address, user name, password, abnormal retry times, filtering rules, black and white lists and the like, and configuring according to a real-time data synchronization channel mode of each tenant;
and (1.3) starting a real-time data increment acquisition program, reading a data recovery point log, acquiring a synchronization point, and subscribing and consuming the increment log of the distributed database in real time.
The real-time incremental log data conversion in the step (2) comprises the following steps:
(2.1) acquiring tenant real-time incremental log data in real time, and filtering the database log data which does not need to be synchronized according to configured filtering rules;
(2.2) analyzing a real operation data table of log data aiming at the logs needing to be processed in real time in an incremental manner, directly filtering the data table logs in a blacklist, and turning the data needing to be processed to the next step;
(2.3) acquiring the name, the column type, the character set, the column value, whether encryption is performed or not and the encryption type of the newly added data log, processing the time type according to standard time, and performing decryption processing according to the step (2.6) for the encrypted column;
(2.4) aiming at the newly added data log, acquiring an updated data table name, an updated column type, a character set, a value before the updated column, a value after the updated column, whether encryption and an encryption type are required or not, processing the time type according to standard time, and carrying out decryption processing aiming at the encrypted column in the step (2.6);
(2.5) acquiring a deleted data table name, a data table primary key name, a primary key value and the like aiming at the deleted data log;
(2.6) respectively executing decryption according to the decryption types corresponding to the encryption types aiming at the data of the encryption columns, and processing the decrypted data as the content of the proprietary protocol message of the isolation device;
and (2.7) converting the analyzed message content into a message in the professional SQ L protocol format of the isolation device and synchronizing in real time.
The real-time incremental log data synchronization in the step (3) comprises the following steps:
(3.1) calling an isolation device equipment interface, and establishing connection with an intranet database through an isolation device;
(3.2) calling a SQ L protocol message special for the isolation device to call a data transmission interface of the isolation device, so as to realize real-time synchronization of data;
and (3.3) recording the data recovery point log after the interface call is successful, continuously processing the next incremental data log, calling the interface call failure according to the configured failure rule, and stopping the tenant synchronization program if the interface call failure still fails after the exception handling strategy is completed.
The database is deployed in an information internal and external network mode as a whole, and interaction is carried out according to a proprietary data protocol of the isolation setting device.
The synchronization parameter configuration in the step (1) comprises a tenant name, a tenant address, a user name, a password, abnormal retry times, a filtering rule and a black and white list, and is configured according to a real-time data synchronization channel mode of each tenant;
the invention ensures the real-time performance of data synchronization through a data subscription mechanism of a distributed database; the data synchronization reliability under the conditions of database exception, network exception or synchronization program exception is ensured by adopting a data recovery point log-based mode; through a multi-tenant subscription mode, tenant division is carried out according to the data scale of a service data table, and the system can still quickly respond under a high-performance and load mode; the real-time synchronization of data from an information outer network to an information inner network is realized through the specific JDBC specification of the isolation device equipment, and the safety of the data is ensured; in order to ensure the data security of the information external network, the invention automatically realizes data decryption and synchronization aiming at the data encrypted by the external network.
Has the advantages that: compared with the prior art, the invention has the following remarkable advantages: 1. a set of complete and feasible solution is provided for the synchronization of the distributed databases across the safe regions, and the safety protocol of the isolation device is used for ensuring the safety and reliability of data transmission; 2. a multi-tenant data synchronization mechanism can be realized, and the real-time data synchronization efficiency under the conditions of high performance and load is guaranteed; 3. the personalized data encryption and decryption mechanism is realized, and the method can flexibly adapt to various types of data encryption and decryption; 4. the method has a perfect data check point mechanism, a flexible data filtering rule and a flexible data synchronization exception handling mechanism, can realize real-time synchronization from an external network to an internal network, and lays a foundation for data real-time monitoring, cross-data associated service query analysis and data value mining in an internal and external network integrated deployment mode.
Drawings
FIG. 1 is a schematic view of the overall structure;
FIG. 2 is a flow diagram of real-time log data processing;
FIG. 3 is a flow chart for obtaining a real table name;
FIG. 4 is a data decryption flow diagram;
fig. 5 is a schematic diagram of data recovery point usage.
Detailed Description
The following detailed description of specific embodiments of the invention refers to the accompanying drawings.
Fig. 1 is a schematic diagram of an overall architecture of the present invention, in which an internal and external network deployment mode is adopted in the overall system, the external network is composed of a multi-tenant subscription, real-time incremental log data acquisition, analysis, synchronization, a proprietary data protocol and a data recovery point log, and each component interacts with each other through an interface, and data load is realized through the multi-tenant subscription mode, data security is realized through the proprietary data protocol, and data reliability is guaranteed through the data recovery point log.
FIG. 2 is a flow chart of real-time data processing according to the present invention, which mainly comprises the following steps:
step 1: acquiring an incremental data log in real time through a multi-tenant subscription mode, judging whether a log database meets filtering requirements or not, directly filtering the data which does not meet the requirements, and carrying out next processing on the data which meets the requirements;
step 2: judging whether the data table is in the range of the synchronous blacklist or not aiming at the database data which is not filtered, if so, filtering, not carrying out next processing in the range, and acquiring the real table name of the data table in the judging process, wherein the acquired real table name is shown in figure 3;
step 3, acquiring the operation type of the real-time incremental data log, judging whether the operation type is DM L (Insert, Update and Delete), and skipping if the operation type is DD L;
step 4, acquiring log types of the real-time incremental data DM L, and respectively processing according to different types;
and 5: analyzing the column name, the type, the encryption type and the data field value of the acquired data aiming at the newly added class data; analyzing and acquiring column names before and after updating, data values before and after updating, whether encryption is performed, encryption types, main keys and the like aiming at the updated data; analyzing and acquiring a deleted primary key column and a primary key value aiming at the deleted class data; processing the analyzed data, and decrypting the data needing to be decrypted, wherein the data decryption process is shown in fig. 4;
step 6, converting the analyzed data according to a proprietary data protocol (NDS SQ L) of the isolation device into an NDS SQ L statement supported by the isolation device;
and 7: and calling the NDS driving interface to submit the processed proprietary data protocol message, updating a data recovery point when the processing is successful and failed, wherein the data recovery point has the function as shown in FIG. 5.
Claims (6)
1. A method for real-time synchronization of distributed databases across secure areas, comprising the steps of:
(1) acquiring real-time incremental log data, and configuring a multi-tenant real-time data incremental channel for a distributed database;
(2) converting real-time incremental log data, and converting the decrypted database log into a professional SQ L protocol format message of the isolation device;
(3) and synchronizing real-time incremental data, decrypting data synchronization from the external network to the internal network and recording data recovery point logs.
2. The method for real-time synchronization of distributed databases across secure areas according to claim 1, wherein the real-time incremental log data acquisition in step (1) comprises the following steps:
(1.1) evaluating the data scale of a business database table and making a configuration strategy;
(1.2) modifying data synchronization parameter configuration;
and (1.3) starting a real-time data increment acquisition program, reading a data recovery point log and acquiring a synchronization point.
3. The method for real-time synchronization of distributed databases across secure areas according to claim 1, wherein the real-time incremental log data conversion in step (2) comprises the following steps:
(2.1) acquiring tenant real-time incremental log data in real time, and filtering the database log data which does not need to be synchronized according to configured filtering rules;
(2.2) analyzing a real operation data table of log data aiming at the logs needing to be processed in real time in an incremental manner, directly filtering the data table logs in a blacklist, and turning the data needing to be processed to the next step;
(2.3) acquiring the name, the column type, the character set, the column value, whether encryption is performed or not and the encryption type of the newly added data log, processing the time type according to standard time, and performing decryption processing according to the step (2.6) for the encrypted column;
(2.4) aiming at the newly added data log, acquiring an updated data table name, an updated column type, a character set, a value before the updated column, a value after the updated column, whether encryption and an encryption type are required or not, processing the time type according to standard time, and carrying out decryption processing aiming at the encrypted column in the step (2.6);
(2.5) acquiring a deleted data table name, a data table primary key name, a primary key value and the like aiming at the deleted data log;
(2.6) respectively executing decryption according to the decryption types corresponding to the encryption types aiming at the data of the encryption columns, and processing the decrypted data as the content of the proprietary protocol message of the isolation device;
and (2.7) converting the analyzed message content into a message in the professional SQ L protocol format of the isolation device and synchronizing in real time.
4. The method for real-time synchronization of distributed databases across secure enclaves according to claim 1, wherein the real-time incremental log data synchronization in step (3) comprises the following steps:
(3.1) calling an isolation device equipment interface, and establishing connection with an intranet database through an isolation device;
(3.2) calling a data transmission interface of the isolation device by adopting an SQ L protocol message special for the isolation device to realize real-time synchronization of data;
and (3.3) recording the data recovery point log after the interface call is successful, continuously processing the next incremental data log, calling the interface call failure according to the configured failure rule, and stopping the tenant synchronization program if the interface call failure still fails after the exception handling strategy is completed.
5. The method of claim 1, wherein the database synchronization is integrally deployed in an intranet-extranet mode, and interaction is based on a proprietary data protocol of an isolation setting device.
6. The method for real-time synchronization of distributed databases across secure domains according to claim 1, wherein the synchronization parameter configuration in step (1) includes tenant name, tenant address, user name, password, abnormal retry number, filtering rule and black and white list, and is configured according to a real-time data synchronization channel mode of each tenant.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910801533.2A CN111400394B (en) | 2019-08-28 | 2019-08-28 | Distributed database real-time synchronization method crossing security areas |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910801533.2A CN111400394B (en) | 2019-08-28 | 2019-08-28 | Distributed database real-time synchronization method crossing security areas |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111400394A true CN111400394A (en) | 2020-07-10 |
| CN111400394B CN111400394B (en) | 2023-11-07 |
Family
ID=71432079
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910801533.2A Active CN111400394B (en) | 2019-08-28 | 2019-08-28 | Distributed database real-time synchronization method crossing security areas |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111400394B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000741A (en) * | 2020-08-25 | 2020-11-27 | 中国南方电网有限责任公司 | Intranet and extranet data exchange system, method, device, computer equipment and medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130031613A1 (en) * | 2011-07-26 | 2013-01-31 | Salesforce.Com, Inc. | Secure access to customer log data in a multi-tenant environment |
| CN106709043A (en) * | 2016-12-30 | 2017-05-24 | 江苏瑞中数据股份有限公司 | Data synchronous loading method based on database log |
-
2019
- 2019-08-28 CN CN201910801533.2A patent/CN111400394B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130031613A1 (en) * | 2011-07-26 | 2013-01-31 | Salesforce.Com, Inc. | Secure access to customer log data in a multi-tenant environment |
| CN106709043A (en) * | 2016-12-30 | 2017-05-24 | 江苏瑞中数据股份有限公司 | Data synchronous loading method based on database log |
Non-Patent Citations (1)
| Title |
|---|
| 曹静;: "基于物理隔离的分布式数据库同步" * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000741A (en) * | 2020-08-25 | 2020-11-27 | 中国南方电网有限责任公司 | Intranet and extranet data exchange system, method, device, computer equipment and medium |
| CN112000741B (en) * | 2020-08-25 | 2024-05-03 | 中国南方电网有限责任公司 | Internal and external network data exchange system, method, device, computer equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111400394B (en) | 2023-11-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10917417B2 (en) | Method, apparatus, server, and storage medium for network security joint defense | |
| EP1955159B1 (en) | Log collection, structuring and processing | |
| US8032489B2 (en) | Log collection, structuring and processing | |
| US20110314148A1 (en) | Log collection, structuring and processing | |
| US20130159723A1 (en) | Methods, apparatus and systems for monitoring locations of data within a network service | |
| US20170181054A1 (en) | Managed access graphical user interface | |
| CN104504014A (en) | Data processing method and device based on large data platform | |
| CN102065416B (en) | Method, device and system for formatting logs | |
| US20150234910A1 (en) | Lifecycle management and provisioning system for unified communications | |
| CN111400394B (en) | Distributed database real-time synchronization method crossing security areas | |
| CN115270182A (en) | Power grid project closed-loop control file management system | |
| CN113810366A (en) | Website uploaded file safety identification system and method | |
| CN112423017B (en) | Channel packaging system | |
| CN113778709B (en) | Interface calling method, device, server and storage medium | |
| CN114024719B (en) | Medical information safety management system based on blockchain technology | |
| CN112838933B (en) | Information synchronization method, equipment and storage medium in network traffic analysis | |
| CN105631559A (en) | Enterprise information management system | |
| GB2546464A (en) | Communication apparatus, method for controlling communication between different types of devices, | |
| CN112069255A (en) | Method and device for synchronizing internal and external network database data | |
| CN111917584A (en) | Data security exchange system and heterogeneous data conversion method | |
| CN110933025A (en) | Multi-source heterogeneous data cross-domain synchronous shared storage method, device, equipment and medium | |
| CN116401128B (en) | Big data-based information operation and maintenance management system | |
| CN116029729B (en) | Cross-link method and system based on dynamic access application link management contract mode | |
| CN102026177B (en) | Monitoring method and system | |
| EP3591556A1 (en) | Automated security assessment of information systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: No. 19 Chengxin Avenue, Moling Street, Jiangning District, Nanjing City, Jiangsu Province, 211106 Patentee after: Nanjing Nanrui Ruizhong Data Co.,Ltd. Country or region after: China Patentee after: NARI Group Corp. Address before: No.180 software Avenue, Yuhuatai District, Nanjing City, Jiangsu Province, 210012 Patentee before: CHINA REALTIME DATABASE Co.,Ltd. Country or region before: China Patentee before: NARI Group Corp. |
|
| CP03 | Change of name, title or address |