CN111046434A - Method for realizing data desensitization based on canal - Google Patents
Method for realizing data desensitization based on canal Download PDFInfo
- Publication number
- CN111046434A CN111046434A CN201911319825.9A CN201911319825A CN111046434A CN 111046434 A CN111046434 A CN 111046434A CN 201911319825 A CN201911319825 A CN 201911319825A CN 111046434 A CN111046434 A CN 111046434A
- Authority
- CN
- China
- Prior art keywords
- module
- desensitization
- data
- canal
- slave
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
- G06F21/6254—Protecting personal data, e.g. for financial or medical purposes by anonymising data, e.g. decorrelating personal data from the owner's identification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Storage Device Security (AREA)
Abstract
The invention belongs to the technical field of data desensitization, and particularly relates to a method for realizing data desensitization based on canal, which comprises a data desensitization system based on canal, wherein the data desensitization system based on canal comprises a master module, the master module is connected with a binlog module, the binlog module is connected with an expression filtering module, the expression filtering module is connected with a slave common data module and a canal module, the canal module is connected with a desensitization framework module, the desensitization framework module comprises a canal client side packaging module, a desensitization pipeline module and a database writing module, and the database writing module is connected with a slave desensitization data module; the method for realizing data desensitization based on canal comprises the following steps: s1: firstly, determining a mysql data source to be desensitized in a master module; s2: transmitting a part of data which does not need desensitization to a slave common data module through an expression filtering module; s5: and transmitting a part of desensitization data needing to be filtered to a slave desensitization data module by the canal client encapsulation module through a desensitization rule through an expression filtering module.
Description
Technical Field
The invention belongs to the technical field of data desensitization, and particularly relates to a method for realizing data desensitization based on canal.
Background
With the rise of data mining, data warehouse construction and data security play an important role, and once privacy or other sensitive data leakage occurs, property, reputation, personal safety and legal benefits of data subjects (clients, employees and companies) are seriously damaged. Data desensitization is an effective way to solve the problem, and although a great number of data desensitization schemes exist in the industry at present, the technical threshold is high, the cost is high, and the operation is difficult.
Disclosure of Invention
In order to reduce technical difficulty, save cost and desensitize in real time, the invention provides a desensitizing technology implementation method which comprises the following steps: canal data desensitization method. On the basis of the existing mysql data source, master-slave separation of the database is realized, the data needing desensitization is subjected to desensitization treatment by using canal and then synchronized to the slave library, and finally the slave library is used as a basic data source for data analysis. In order to achieve the technical purpose, the technical scheme adopted by the invention is as follows:
a method for realizing data desensitization based on canal comprises the steps of realizing a data desensitization system based on canal, wherein the data desensitization system based on canal comprises a master module, the master module is connected with a binlog module, the binlog module is connected with an expression filtering module, the expression filtering module is connected with a slave common data module and a canal module, the canal module is connected with a desensitization framework module, the desensitization framework module comprises a canal client side packaging module, a desensitization pipeline module and a database writing module, the canal client side packaging module is connected with the canal module, the database writing module is connected with the slave desensitization data module, and the slave desensitization data module and the slave common data module form a slave module;
the method for realizing data desensitization based on canal comprises the following steps:
s1: firstly, determining a mysql data source, a detailed table and a field to be desensitized in a master module; then determining a desensitization rule through a desensitization pipeline module; constructing master-slave environment and canal service of mysql;
s2: transmitting a part of data which does not need desensitization to a slave common data module through an expression filtering module;
s5: and transmitting a part of desensitization data needing to be filtered to a slave desensitization data module by the canal client encapsulation module through a desensitization rule through an expression filtering module.
In a preferred embodiment of the invention, the desensitization pipeline module comprises a plurality of desensitization treatment modules.
As a preferred embodiment of the present invention, the desensitization rule includes data replacement: replacing a true value with fictional data; truncation, encryption, concealment or invalidation: replacing the truth value with 'invalid' or '#'; randomization: replacing the true value with random data; offsetting: changing the digital data by random shifting; character subchain shielding: creating a custom mask for specific data; customizing a processing rule: custom processing rules written using external programs are supported.
The invention has the beneficial effects that:
1. the technical threshold is low, and mysql is master-slave;
2. the real-time performance is strong, data insertion and updating are updated in real time, and offline is not needed;
3. no commercial cost exists, and mysql and canal are open-source products;
4. the desensitization index of an application company is flexibly adjusted, and custom expansion is supported;
5. abundant extended functions: such as timed triggers, automatic mail delivery, three-party interface notifications, etc.
Drawings
The invention is further illustrated by the non-limiting examples given in the accompanying drawings;
FIG. 1 is a schematic structural diagram of an embodiment of a method for performing data desensitization based on canal according to the present invention.
Detailed Description
In order that those skilled in the art can better understand the present invention, the following technical solutions are further described with reference to the accompanying drawings and examples.
As shown in fig. 1, the method for implementing data desensitization based on canal of the present invention includes implementing a data desensitization system based on canal, where the data desensitization system based on canal includes a master module, the master module is connected with a binlog module, the binlog module is connected with an expression filter module, the expression filter module is connected with a slave common data module and a canal module, the canal module is connected with a desensitization framework module, the desensitization framework module includes a canal client encapsulation module, a desensitization pipeline module and a database write-in module, the canal client encapsulation module is connected with the canal module, the database write-in module is connected with a slave desensitization data module, and the slave desensitization data module and the slave common data module constitute a slave module;
the method for realizing data desensitization based on canal comprises the following steps:
s1: firstly, determining a mysql data source, a detailed table and a field to be desensitized in a master module; then determining a desensitization rule through a desensitization pipeline module; constructing master-slave environment and canal service of mysql;
s2: transmitting a part of data which does not need desensitization to a slave common data module through an expression filtering module;
s5: transmitting a part of desensitization data to be filtered to a slave desensitization data module from the canal client encapsulation module through a desensitization rule through an expression filtering module; and finally, completing the complete replication of the normal data and the desensitized data to completely replicate the master data and the slave data.
Wherein the desensitization pipeline module comprises a plurality of desensitization processing modules.
Wherein the desensitization rule includes data replacement: replacing a true value with fictional data; truncation, encryption, concealment or invalidation: replacing the truth value with 'invalid' or '#'; randomization: replacing the true value with random data; offsetting: changing the digital data by random shifting; character subchain shielding: creating a custom mask for specific data; customizing a processing rule: custom processing rules written using external programs are supported.
In this embodiment, the master module: is a master library of mysql;
binlog module: storing the write operation record of the mysql master library, and the slave library can replay the write operation of the master library through the file to complete data synchronization;
a slave module: one slave library representing mysql;
slave common data block: a set of dependent tables that do not require data desensitization;
slave desensitization data block: a set of related tables that require desensitization, such as a table that holds basic information for the user;
a canal module: the data synchronization middleware can analyze the binlog file, can format and record the binlog file as an event and is used for self-defining operation;
an expression filtering module: an expression can be set to specify which tables are scanned to realize data desensitization, and which tables are not required to be desensitized to directly synchronize the slave libraries;
desensitizing the frame module: an integrated environment for data desensitization;
desensitization framework-canal client encapsulation module: receiving a write operation event from canal and pushing a message to the desensitization pipeline;
desensitization frame-desensitization conduit module: the desensitization treatment blocks are assembled one by one and desensitized by running water. For example, desensitization and disorder are carried out firstly, and then special characters are replaced;
desensitization framework-database write module: write desensitized data to desensitization data correlation table from bank slave.
The foregoing embodiments are merely illustrative of the principles of the present invention and its efficacy, and are not to be construed as limiting the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.
Claims (3)
1. A method for performing data desensitization based on canal, comprising: the data desensitization system based on canal is realized, and comprises a master module, wherein the master module is connected with a binlog module, the binlog module is connected with an expression filtering module, the expression filtering module is connected with a slave common data module and a canal module, the canal module is connected with a desensitization framework module, the desensitization framework module comprises a canal client encapsulation module, a desensitization pipeline module and a database writing module, the canal client encapsulation module is connected with the canal module, the database writing module is connected with a slave desensitization data module, and the slave desensitization data module and the slave common data module form a slave module;
the method for realizing data desensitization based on canal comprises the following steps:
s1: firstly, determining a mysql data source, a detailed table and a field to be desensitized in a master module; then determining a desensitization rule through a desensitization pipeline module; constructing master-slave environment and canal service of mysql;
s2: transmitting a part of data which does not need desensitization to a slave common data module through an expression filtering module;
s5: and transmitting a part of desensitization data needing to be filtered to a slave desensitization data module by the canal client encapsulation module through a desensitization rule through an expression filtering module.
2. A method of achieving data desensitization based on canal according to claim 1, wherein: the desensitization pipeline module comprises a plurality of desensitization processing modules.
3. A method of achieving data desensitization based on canal according to claim 2, wherein: the desensitization rule includes data replacement: replacing a true value with fictional data; truncation, encryption, concealment or invalidation: replacing the truth value with 'invalid' or '#'; randomization: replacing the true value with random data; offsetting: changing the digital data by random shifting; character subchain shielding: creating a custom mask for specific data; customizing a processing rule: custom processing rules written using external programs are supported.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911319825.9A CN111046434A (en) | 2019-12-19 | 2019-12-19 | Method for realizing data desensitization based on canal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911319825.9A CN111046434A (en) | 2019-12-19 | 2019-12-19 | Method for realizing data desensitization based on canal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111046434A true CN111046434A (en) | 2020-04-21 |
Family
ID=70238015
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911319825.9A Pending CN111046434A (en) | 2019-12-19 | 2019-12-19 | Method for realizing data desensitization based on canal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111046434A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106599713A (en) * | 2016-11-11 | 2017-04-26 | 中国电子科技网络信息安全有限公司 | Database masking system and method based on big data |
| CN107291926A (en) * | 2017-06-29 | 2017-10-24 | 搜易贷（北京）金融信息服务有限公司 | A kind of binlog analysis methods |
| CN108228621A (en) * | 2016-12-15 | 2018-06-29 | 上海祈贝健康管理咨询有限公司 | A kind of method of strange land real-time synchronization SQL data |
-
2019
- 2019-12-19 CN CN201911319825.9A patent/CN111046434A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106599713A (en) * | 2016-11-11 | 2017-04-26 | 中国电子科技网络信息安全有限公司 | Database masking system and method based on big data |
| CN108228621A (en) * | 2016-12-15 | 2018-06-29 | 上海祈贝健康管理咨询有限公司 | A kind of method of strange land real-time synchronization SQL data |
| CN107291926A (en) * | 2017-06-29 | 2017-10-24 | 搜易贷（北京）金融信息服务有限公司 | A kind of binlog analysis methods |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Jørgen Hole | Anti-fragile ICT systems | |
| Lichtenberg et al. | The role of the media in risk communication | |
| CN111885040A (en) | Distributed network situation perception method, system, server and node equipment | |
| CN107895122B (en) | Special sensitive information active defense method, device and system | |
| CN109033268A (en) | Method of data synchronization, device, equipment and storage medium | |
| CN103957172B (en) | A kind of inside and outside network physical isolation network data automatic switch-board | |
| CN110019502A (en) | Synchronous method, Database Systems and equipment between primary database and standby database | |
| CN108810127A (en) | Disaster recovery method based on block chain and device | |
| CN117009483A (en) | Method, device and equipment for generating question-answering service and readable storage medium | |
| CN110059280A (en) | A kind of information issuing method based on block chain | |
| CN114077518A (en) | Data snapshot method, device, equipment and storage medium | |
| CN116167085A (en) | Data desensitization method and device | |
| CN103106200A (en) | Synchronization system of non-relational type database and double-writing synchronization method | |
| CN111046434A (en) | Method for realizing data desensitization based on canal | |
| US9749452B2 (en) | Contact person display processing method and mobile terminal | |
| CN111177785A (en) | Desensitization processing method for private data of enterprise-based business system | |
| CN104462342A (en) | Synchronous processing method and device for database snapshots | |
| CN110968896A (en) | Method for realizing data desensitization based on canal | |
| Liu et al. | Heritage matters in crisis informatics: How information and communication technology can support legacies of crisis events | |
| CN105303122B (en) | The method that the locking of sensitive data high in the clouds is realized based on reconfiguration technique | |
| Cárdenas et al. | Digital outburst: The expression of a social crisis through online social networks | |
| Rinaldi | Post-Western World | |
| CN203233445U (en) | High security internal network information safety system | |
| CN108089944A (en) | A kind of system to guarantee data integrity under the conditions of database failure | |
| Livingstone | The End of'Responsible Gambling': Reinvigorating Gambling Studies |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |