CN111400394B - Distributed database real-time synchronization method crossing security areas - Google Patents
Distributed database real-time synchronization method crossing security areas Download PDFInfo
- Publication number
- CN111400394B CN111400394B CN201910801533.2A CN201910801533A CN111400394B CN 111400394 B CN111400394 B CN 111400394B CN 201910801533 A CN201910801533 A CN 201910801533A CN 111400394 B CN111400394 B CN 111400394B
- Authority
- CN
- China
- Prior art keywords
- data
- real
- time
- log
- synchronization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 13
- 238000002955 isolation Methods 0.000 claims abstract description 22
- 238000001914 filtration Methods 0.000 claims abstract description 14
- 238000011084 recovery Methods 0.000 claims abstract description 14
- 238000012545 processing Methods 0.000 claims description 15
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 230000001360 synchronised effect Effects 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 3
- 230000003993 interaction Effects 0.000 claims description 2
- 238000005065 mining Methods 0.000 abstract description 3
- 238000012544 monitoring process Methods 0.000 abstract description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000013316 zoning Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a real-time synchronization method of data across a safe area based on a distributed relational database log. According to the method, the incremental data is acquired in real time in a subscription mode in a multi-tenant mode, performance load and data isolation are realized in the tenant mode, performance and data safety are guaranteed, the incremental log is analyzed in real time and converted into proprietary protocol data, data real-time synchronization is realized through a safety interface of the isolation device, information safety is guaranteed, and data reliability is guaranteed through a data recovery point mechanism. The invention has perfect data check point mechanism, flexible data filtering rule and data synchronization exception handling mechanism, can realize the real-time synchronization from the external network to the internal network data, and lays a foundation for the real-time monitoring of the data in the internal and external network integrated deployment mode, the query analysis of the cross-data correlation service and the mining of the data value.
Description
Technical Field
The invention belongs to the technical field of data synchronization, and particularly relates to a real-time synchronization aspect of a distributed database crossing a security area.
Background
Along with the rapid development of computer and database technologies, the application range of databases is becoming wider and wider, and meanwhile, databases have been distributed from centralized, and data exist on different servers with scattered locations at the same time, so that the high-reliability operation of enterprise systems is ensured.
The integrated independent deployment mode of the internal and external networks is a traditional large-scale centralized system construction mode, unified support of the external Internet and the internal local area network is achieved on the basis, information security is achieved through a 'zoning and domain division' mode, safety isolation gatekeeper devices exist between the internal and external networks in the integrated mode of the internal and external networks, sensitive data such as client certificates, addresses and the like are stored in the internal and external networks respectively according to the sensitivity of the data, non-sensitive data such as user accounts and the like are stored in the internal and external networks respectively, and in the deployed mode of the internal and external networks, how to achieve relevant service query analysis and data value mining based on the whole data are difficult to develop.
Disclosure of Invention
The invention aims to: the invention aims to solve the problem of real-time data synchronization of cross-isolation device equipment in an internal and external network integrated mode, and based on a real-time incremental log, a data recovery point mechanism and a multi-tenant mechanism of a distributed database, the real-time incremental log is analyzed and decrypted and converted into a special data specification of the isolation device equipment, the real-time synchronization of information external network data to an information internal network is realized, the real-time performance of the data synchronization is realized through a subscription mode, and the safety of the data is realized through the special data specification of the isolation device equipment.
The technical scheme is as follows: in order to achieve the above objective, the method for real-time synchronization of a distributed database across a secure area according to the present invention comprises the following steps:
(1) Acquiring real-time incremental log data, and solving the problems of real-time performance and multi-tenant channels of acquisition of incremental data of a distributed database;
(2) Converting real-time increment log data, and solving the problem of proprietary protocol conversion of a distributed database log decryption and isolation device;
(3) And synchronizing the real-time incremental data, and decrypting the data synchronization from the external network to the internal network and the data recovery point mechanism.
The step (1) of acquiring the real-time incremental log data comprises the following steps:
(1.1) evaluating the data scale of a business database table and formulating a multi-tenant configuration strategy, and configuring a multi-tenant real-time data increment channel in a distributed database;
(1.2) modifying data synchronization parameter configuration including tenant name, tenant address, user name, password, abnormal retry number, filtering rule, blacklist and the like, and configuring according to a real-time data synchronization channel mode of each tenant;
and (1.3) starting a real-time data increment acquisition program, reading a data recovery point log and acquiring a synchronization point, and subscribing and consuming an increment log of the distributed database in real time.
The conversion of the real-time incremental log data in the step (2) comprises the following steps:
(2.1) acquiring tenant real-time incremental log data in real time, and filtering database log data which does not need synchronization according to configured filtering rules;
(2.2) analyzing a real operation data table of log data aiming at the log which needs to be processed in real time in an increment way, directly filtering the log of the data table in a blacklist, and transferring the data which needs to be processed to the next step;
(2.3) aiming at the newly added data log, acquiring the name, the column type, the character set, the column value, whether encryption and encryption type of the newly added data table, processing the time type according to standard time, and carrying out decryption processing aiming at the encryption column by referring to the step (2.6);
(2.4) for the newly added data log, acquiring an updated data table name, an updated column type, a character set, an updated column pre-update value, an updated column post-update value, whether encryption and encryption types are performed, processing the time types according to standard time, and performing decryption processing for the encryption columns by referring to the step (2.6);
(2.5) aiming at the deleted data log, acquiring a deleted data table name, a data table primary key name, a primary key value and the like;
(2.6) respectively executing decryption by the decryption type corresponding to the encryption type aiming at the data of the encryption column, wherein the decrypted data is used as the content of the protocol message special for the isolation device;
and (2.7) converting the analyzed message content into a professional SQL protocol format message of the isolation device and synchronizing in real time.
The step (3) of synchronizing the real-time incremental log data comprises the following steps:
(3.1) calling an isolating device equipment interface, and establishing connection with an intranet database through an isolating device;
(3.2) calling a proprietary SQL protocol message of the isolation device to call a data transmission interface of the isolation device, so as to realize real-time synchronization of data;
and (3.3) recording a data recovery point log after the interface call is successful, continuing to process the next incremental data log, calling the interface call failure according to the configured failure rule, and stopping the tenant synchronization program after the exception handling strategy is finished and still fails.
The database synchronization is deployed in an information intranet-extranet mode as a whole, and interaction is based on a proprietary data protocol of the isolation setting device.
The synchronous parameter configuration in the step (1) comprises tenant names, tenant addresses, user names, passwords, abnormal retry times, filtering rules and black-white lists, and is configured according to a real-time data synchronous channel mode of each tenant;
the invention ensures the real-time performance of data synchronization through the data subscription mechanism of the distributed database; the data synchronization reliability under the conditions of abnormal database, abnormal network or abnormal synchronization program is ensured by adopting a mode based on a data recovery point log; according to the multi-tenant subscription mode, tenant division is carried out according to the data scale of the service data table, so that the system can still respond quickly under the high-performance and load modes; the real-time synchronization of data from the information external network to the information internal network is realized through the specific JDBC specification of the isolation device equipment, so that the safety of the data is ensured; in order to ensure the safety of the information external network data, the invention automatically realizes data decryption and synchronization aiming at the data encrypted by the external network.
The beneficial effects are that: compared with the prior art, the invention has the remarkable advantages that: 1. a set of complete and feasible solution is provided for the synchronization of the distributed databases crossing the safety area, and the safety protocol of the isolation device is used for guaranteeing the safety and reliability of data transmission; 2. the multi-tenant data synchronization mechanism can be realized, and the real-time data synchronization efficiency under the conditions of high performance and load is ensured; 3. the personalized data encryption and decryption mechanism is realized, and the method can be flexibly suitable for encrypting and decrypting various types of data; 4. the method has a perfect data check point mechanism, a flexible data filtering rule and a data synchronization exception handling mechanism, can realize real-time synchronization of the data from the external network to the internal network, and lays a foundation for data real-time monitoring, cross-data correlation service query analysis and data value mining in an internal and external network integrated deployment mode.
Drawings
FIG. 1 is a schematic overall structure;
FIG. 2 is a flow chart of real-time log data processing;
FIG. 3 obtains a flow chart of the real table names;
FIG. 4 is a data decryption flow chart;
FIG. 5 is a schematic diagram of data recovery point usage.
Detailed Description
Specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
Fig. 1 is a schematic diagram of an overall architecture of the present invention, in which an internal and external network deployment mode is adopted in the overall system, the external network is composed of multi-tenant subscription, real-time incremental log data acquisition, analysis, synchronization, proprietary data protocol and data recovery point logs, the components interact through interfaces, data load is realized through the multi-tenant subscription mode, data security is realized through the proprietary data protocol, and data reliability is ensured through the data recovery point logs.
FIG. 2 is a flow chart of the real-time data processing according to the present invention, which mainly comprises the following steps:
step 1: acquiring an incremental data log in real time through a multi-tenant subscription mode, judging whether a log database meets the filtering requirement, directly filtering the data which does not meet the requirement, and carrying out the next processing on the data which meets the requirement;
step 2: judging whether the data table is in the synchronous blacklist range or not according to unfiltered database data, if so, filtering, and not carrying out the next processing in the range, wherein the real table name of the data table needs to be acquired in the judging process, and the real table name is acquired as shown in fig. 3;
step 3: acquiring the operation type of the real-time incremental data log, judging whether the operation type is DML (Insert, update and Delete), and skipping if the operation type is DDL;
step 4: acquiring the type of the real-time incremental data DML log, and respectively processing according to different types;
step 5: aiming at the newly added class data, analyzing and acquiring the column name, type, encryption or non-encryption of the data and the value of a data field; for updating the data, column names before and after updating, data values before and after updating, whether encryption, encryption type, primary key and the like are analyzed and acquired; aiming at the deletion type data, analyzing and acquiring a deletion main key column and a main key value; processing the analyzed data, and decrypting the data to be decrypted, wherein the data decryption flow is shown in fig. 4;
step 6: aiming at the analyzed data, converting according to the proprietary data protocol (NDS SQL) of the isolation device into an NDS SQL statement supported by the isolation device;
step 7: and calling the NDS drive interface to submit the processed proprietary data protocol message, and updating the data recovery point when the processing is successful and unsuccessful, wherein the function of the data recovery point is shown in figure 5.
Claims (1)
1. A real-time synchronization method of a distributed database crossing a safety area is characterized in that the database synchronization is wholly deployed by adopting an information intranet-extranet mode, and interaction is based on a proprietary data protocol of an isolation setting device, and the method comprises the following steps:
(1) Acquiring real-time incremental log data, configuring a multi-tenant real-time data incremental channel with a distributed database, and comprising the following steps:
(1.1) evaluating the data scale of a business database table and formulating a configuration strategy;
(1.2) modifying the data synchronization parameter configuration;
(1.3) starting a real-time data increment acquisition program, reading a data recovery point log and acquiring a synchronization point;
the synchronous parameter configuration comprises tenant names, tenant addresses, user names, passwords, abnormal retry times, filtering rules and black-white lists, and is configured according to a real-time data synchronous channel mode of each tenant;
(2) Converting real-time increment log data, converting the decrypted database log into a professional SQL protocol format message of the isolation device, comprising the following steps:
(2.1) acquiring tenant real-time incremental log data in real time, and filtering database log data which does not need synchronization according to configured filtering rules;
(2.2) analyzing a real operation data table of log data aiming at the log which needs to be processed in real time in an increment way, directly filtering the log of the data table in a blacklist, and transferring the data which needs to be processed to the next step;
(2.3) aiming at the newly added data log, acquiring the name, the column type, the character set, the column value, whether encryption and encryption type of the newly added data table, processing the time type according to standard time, and carrying out decryption processing aiming at the encryption column by referring to the step (2.6);
(2.4) for the update data log, acquiring an update data table name, an update column type, a character set, an update column pre-update value, an update column post-update value, whether encryption and encryption type are performed, processing the time type according to standard time, and performing decryption processing for the encryption column by referring to the step (2.6);
(2.5) aiming at the deleted data log, acquiring a deleted data table name, a data table primary key name and a primary key value;
(2.6) respectively executing decryption by the decryption type corresponding to the encryption type aiming at the data of the encryption column, wherein the decrypted data is used as the content of the protocol message special for the isolation device;
(2.7) converting the analyzed message content into a professional SQL protocol format message of the isolation device and synchronizing in real time;
(3) Synchronizing real-time incremental data, decrypting data synchronization from an external network to an internal network, and recording a data recovery point log, wherein the method comprises the following steps of:
(3.1) calling an isolating device equipment interface, and establishing connection with an intranet database through an isolating device;
(3.2) calling a data transmission interface of the isolation device by adopting an SQL protocol message special for the isolation device to realize real-time synchronization of data;
and (3.3) recording a data recovery point log after the interface call is successful, continuing to process the next incremental data log, calling the interface call failure according to the configured failure rule, and stopping the tenant synchronization program after the exception handling strategy is finished and still fails.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910801533.2A CN111400394B (en) | 2019-08-28 | 2019-08-28 | Distributed database real-time synchronization method crossing security areas |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910801533.2A CN111400394B (en) | 2019-08-28 | 2019-08-28 | Distributed database real-time synchronization method crossing security areas |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111400394A CN111400394A (en) | 2020-07-10 |
| CN111400394B true CN111400394B (en) | 2023-11-07 |
Family
ID=71432079
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910801533.2A Active CN111400394B (en) | 2019-08-28 | 2019-08-28 | Distributed database real-time synchronization method crossing security areas |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111400394B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000741B (en) * | 2020-08-25 | 2024-05-03 | 中国南方电网有限责任公司 | Internal and external network data exchange system, method, device, computer equipment and medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130031613A1 (en) * | 2011-07-26 | 2013-01-31 | Salesforce.Com, Inc. | Secure access to customer log data in a multi-tenant environment |
| CN106709043A (en) * | 2016-12-30 | 2017-05-24 | 江苏瑞中数据股份有限公司 | Data synchronous loading method based on database log |
-
2019
- 2019-08-28 CN CN201910801533.2A patent/CN111400394B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130031613A1 (en) * | 2011-07-26 | 2013-01-31 | Salesforce.Com, Inc. | Secure access to customer log data in a multi-tenant environment |
| CN106709043A (en) * | 2016-12-30 | 2017-05-24 | 江苏瑞中数据股份有限公司 | Data synchronous loading method based on database log |
Non-Patent Citations (1)
| Title |
|---|
| 曹静 ; .基于物理隔离的分布式数据库同步.计算机工程.2008,(第16期),第186-187、190页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111400394A (en) | 2020-07-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022126968A1 (en) | Micro-service access method, apparatus and device, and storage medium | |
| CN110543464B (en) | Big data platform applied to intelligent park and operation method | |
| EP1955159B1 (en) | Log collection, structuring and processing | |
| US10296182B2 (en) | Managed access graphical user interface | |
| EP2580692A2 (en) | Query pipeline | |
| CN101820449A (en) | Cross-safety zone application service isolation platform | |
| CN101488965B (en) | Domain name filtering system and method | |
| CN106572087B (en) | Voice outbound system | |
| CN112632135A (en) | Big data platform | |
| CN108520004B (en) | Multi-tenant data source switching system based on method parameter section | |
| US20220138184A1 (en) | Object resolution among account-level namespaces for database platforms | |
| CN104504014A (en) | Data processing method and device based on large data platform | |
| US7917636B2 (en) | System and method for detecting unused accounts in a distributed directory service | |
| US20120254337A1 (en) | Mainframe Management Console Monitoring | |
| US11593354B2 (en) | Namespace-based system-user access of database platforms | |
| US20150234910A1 (en) | Lifecycle management and provisioning system for unified communications | |
| CN113590639A (en) | Data synchronization method between databases isolated by gatekeepers | |
| KR20040052569A (en) | Method and system for monitoring and securing a database | |
| CN113810366A (en) | Website uploaded file safety identification system and method | |
| WO2022026973A1 (en) | Account-level namespaces for database platforms | |
| CN111400394B (en) | Distributed database real-time synchronization method crossing security areas | |
| US20190066012A1 (en) | Enterprise customer website | |
| CA3155794A1 (en) | System and method for blockchain based backup and recovery | |
| CN112667586B (en) | Method, system, equipment and medium for synchronizing data based on stream processing | |
| Cisco | Working with Log Files |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: No. 19 Chengxin Avenue, Moling Street, Jiangning District, Nanjing City, Jiangsu Province, 211106 Patentee after: Nanjing Nanrui Ruizhong Data Co.,Ltd. Country or region after: China Patentee after: NARI Group Corp. Address before: No.180 software Avenue, Yuhuatai District, Nanjing City, Jiangsu Province, 210012 Patentee before: CHINA REALTIME DATABASE Co.,Ltd. Country or region before: China Patentee before: NARI Group Corp. |
|
| CP03 | Change of name, title or address |