CN109657496A - A kind of big data static database desensitization system and method for the full mirror image of zero-copy - Google Patents
A kind of big data static database desensitization system and method for the full mirror image of zero-copy Download PDFInfo
- Publication number
- CN109657496A CN109657496A CN201811563203.6A CN201811563203A CN109657496A CN 109657496 A CN109657496 A CN 109657496A CN 201811563203 A CN201811563203 A CN 201811563203A CN 109657496 A CN109657496 A CN 109657496A
- Authority
- CN
- China
- Prior art keywords
- desensitization
- data
- task
- database
- going
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Medical Informatics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
It mainly include system management module, data source management module, data desensitization task execution module and desensitization configuration management module four module the invention discloses a kind of big data static database of the full mirror image of zero-copy desensitization system, wherein data desensitization task execution module is the core of whole system.The big data static database of the kind full mirror image of zero-copy of the present invention one desensitizes system, and the system management module being managed including the basic function to system, the data desensitization task execution module for realizing configuration and execution and monitoring to desensitization task, for desensitization task provides the desensitization configuration management module on configuration basis at the data source management module being managed to the source database address of system and destination address.The invention also discloses a kind of big data static database desensitization methods of the full mirror image of zero-copy.
Description
Technical field
The present invention relates to the interleaving techniques field of computer technology and information security, specifically a kind of full mirror image of zero-copy
The method and system of big data static database desensitization.
Background technique
Now, the development of social informatization and networking leads to the growth of data explosion formula, with various industries business
Fast development, have accumulated in these business production systems it is a large amount of to personal, enterprise, the relevant sensitive data of government.Actual
The truthful data that operation system is directly used in exploitation, test, outsourcing and other nonproductive environment, easily causes sensitive data
Leakage.Therefore, most users carry out transformation of data by desensitization rule to sensitive information, in fact by means of data desensitization technology
The protection of existing privacy-sensitive data.
Currently, the data desensitization mode that user uses mainly includes following several:
1. the data based on database command desensitize
Data desensitization mode based on database command is the sql command for directlying adopt database, to needing in database
The field of desensitization is blocked or is replaced, to achieve the purpose that data desensitize.This method is simple to operate, but exists very
More drawbacks, specific as follows:
1) when configuring desensitization rule by database command, user needs to be well understood by the sensitive data of database, and one
Denier user omits sensitive data, will cause sensitive data leakage；
2) when the table and field substantial amounts in database, user needs to take a significant amount of time configuration desensitization rule, time-consuming
And effort；
3) based on database command setting desensitization rule, the resource of database can be occupied, database performance is impacted.
4) in the case where developing and testing environment, the mode based on database command cannot keep the business pass of data after desensitization
Connection property and consistency influence the normal use of desensitization data.
2. the data desensitization based on data landing
Data desensitization mode based on data landing usually uses data extraction tool by the data pick-up of user first
To local, later after locally carrying out sensitive data discovery, data desensitization is carried out to sensitive data using desensitization algorithm.Which
It can satisfy the consistency and relevance demand of timing of the user under nonproductive environment, increment desensitization and desensitization data, still
It there is also some problems, specific as follows:
1) the data desensitization based on data landing is that data are drawn into local from production environment, when data volume is bigger
When, data pick-up causes serious influence to the performance of database, or even will affect the normal use of customer data base；
2) data based on data landing, which desensitize, stores the creation data of extraction to local, increases the wind of leaking data
Danger can cause huge loss once storage equipment is stolen to user；
3) based on the data desensitization mode of data landing when carrying out database timed increase desensitization task, to promote desensitization
Speed can do related change to customer data base, may influence the performance of customer data base；
4) the data desensitization mode based on data landing only considers the data desensitization that increment is timed to full library, but simultaneously
It does not support the desensitization of table level timed increase and the timed increase of view rank to desensitize, is unable to satisfy whole scenes of user demand.
From the foregoing, it will be observed that there is also following challenges for existing data desensitization mode:
1) how database performance to be influenced to support the data of timed increase to desensitize in the smallest situation；
2) the data desensitization mode that how data are taken with the full mirror image of zero-copy, guarantees that data are not landed, reduces sensitive number
According to the risk of leakage；
3) how the case where guaranteeing to desensitize rate, customer data base is not changed, realizes the desensitization task of timing with increment；
4) database level, table level, the desensitization of the data of view rank how to be realized, the database after retaining whole library grade desensitization is about
Beam relationship and view meet the diversified scene demand of user；
5) after how guaranteeing whole library grade data desensitization, the business association and consistency of data.
Summary of the invention
To solve the above-mentioned problems, the present invention provides a kind of big data static database desensitization system of full mirror image of zero-copy
And method.
A kind of big data static database desensitization system of the full mirror image of zero-copy, carries out pipe including the basic function to system
The system management module of reason, is realized the data source management module being managed to the source database address of system and destination address
The data desensitization task execution module of configuration and execution and monitoring to desensitization task provides configuration basis for desensitization task
Desensitize configuration management module.
Wherein, the system management module includes role management module, user management module, device management module, cluster
Management module.The data source management module includes registration, modification, deletion, inquiry, enabling and the disabling function to source database
The source database address management module being managed；The data source management module further includes the storage address to data after desensitization
Registration, modification, deletion, inquiry, enabling and disabling be able to carry out the destination address management module of management.The data desensitization is appointed
Business execution module includes the whole library grade static data desensitization module configured to the data desensitization task of whole library grade, to table level
The view that data desensitization task was configured the desensitization of table level static data module, is configured to the data of view grade desensitization task
The data desensitization task monitoring module of figure grade static data desensitization module, all data desensitization tasks of monitoring.The desensitization configuration
Management module include sensitive classification system management module, data desensitization policy management module, customized sensitive words segment management module,
Data desensitization algorithm management module, customized sensitive data management module, log query and analysis module.
The full mirror image big data static database desensitization method of a kind of zero-copy of the invention, using a kind of full mirror image of zero-copy
Big data static database desensitization system, desensitizes to data, includes the following steps:
S1, database is obtained, goes to step s2；
S2, input database data source information carry out database data Login, go to step s3；
S3, configuration data desensitization task, and start desensitization task, go to step s4；
S4, data are synchronous, go to step s5；
S5, monitoring objective library whether successful connection, if so, going to step s6；If it is not, going to step s7；
S6, judgement desensitization task type simultaneously select task, if desensitization task type is whole library grade desensitization, execute whole library grade
Desensitization task；If the task type that desensitizes executes table level desensitization task for table level desensitization；Drag desensitization task type de- for view grade
It is quick, then execute view grade desensitization task；
S7, desensitization task execution failure, terminate process.
Wherein, database data Login is carried out to include the following steps:
The source database data source information that y1, input are registered, goes to step y2；
Whether the information inputted in y2, judgment step y1 is correct, if it is not, source database data source information registration failure, turns
To step y3；If so, going to step y4；
Y3, data source registration terminate；
Y4, the table structure for obtaining database, go to step y5；
Y5, constraint database relation information is obtained, goes to step y6；
Y6, data base view information is obtained, goes to step y7；
Y7, database data sample is extracted using random sampling algorithm, goes to step y8；
Y8, sensitive data identification is carried out to sample using intelligent self-adaptive sensitive data identification method, goes to step y9；
Y9, output sensitive data recognition result, go to step y10；
Y10, the success of source database data source registration.
Data, which synchronize, to be included the following steps:
T1, system receive sync database event, and log-on data synchronous task goes to step t2；
T2, database is connected automatically, go to step t3；
T3, judge whether database connection succeeds, if so, step t4 is gone to, if it is not, database connection failure, goes to step
Rapid t10；
T4, database structure is obtained, goes to step t5；
T5, the database structure that step t4 is got is compared with table structure when data source registration, goes to step
t6；
Whether the comparison structure in t6, judgment step t5 is consistent, if so, going to step t10；If it is not, going to step t7；
Table structure when t7, more new registration, goes to step t8；
T8, the sensitive data based on multithreading find that mode finds the sensitive data in updated database automatically, turn
To step t9；
T9, data synchronize success；
T10, end.
Whole library grade desensitization task is executed to include the following steps:
Z11, desensitization task configuration information is obtained, executes desensitization task, goes to step z12；
Z12, the synchronous process of data is executed, goes to step z13；
Whether successful connection, if it is not, desensitization task execution fails, is tied if so, going to step z14 for z13, detection object library
Beam exits；
Z14, judge whether desensitization task is whole library grade desensitization, if so, step z15 is gone to, if it is not, it is de- to go to other types
It is quick；
Z15, object library is judged with the presence or absence of relationship, if so, step z16 is gone to, if it is not, going to step z17；
Z16, delete target base relation, go to step z17；
Z17, make task distribution to source database, go to step z18；
Z18, data desensitization rule is obtained, goes to step z19；
Z19, judge whether target database table structure and source database are consistent, if so, going to step z110；If it is not, going to step
Rapid z114；
Z110, judge whether desensitization task is timed increase task, if so, step z111 is gone to, if it is not, going to step
z113；
Z111, it desensitizes to the data of increment, and is appended to object library, go to step z112；
Z112, judge whether addition succeeds, if so, going to step z115；If it is not, going to step z114；
Z113, deletion table identical with desensitization task object library table, go to step z114；
Z114, sensitive data discovery, desensitization and importing are carried out to table again, goes to step z115；
Z115, write-in relationship and view.
Table level desensitization task is executed to include the following steps:
Z21, desensitization task configuration information is obtained, executes desensitization task, goes to step z22；
Z22, the synchronous process of data is executed, goes to step z23；
Whether successful connection, if it is not, desensitization task execution fails, is tied if so, going to step z24 for z23, detection object library
Beam exits；
Z24, judge whether desensitization task is table level desensitization, if so, step z25 is gone to, if it is not, it is de- to go to other types
It is quick；
Z25, make task distribution to source database, go to step z26；
Z26, data desensitization rule is obtained, goes to step z27；
Z27, judge whether target database table structure and source database are consistent, if so, going to step z28；If it is not, going to step
z212；
Z28, judge whether desensitization task is timed increase task, if so, step z29 is gone to, if it is not, going to step
z211；
Z29, it desensitizes to the data of increment, and is appended to object library, go to step z210；
Z210, judge whether addition succeeds, if so, going to step z213；If it is not, going to step z212；
Z211, deletion table identical with desensitization task object library table, go to step z212；
Z212, sensitive data discovery, desensitization and importing are carried out to table again, goes to step z213；
Z213, write-in relationship and view.
View grade desensitization task is executed to include the following steps:
Z31, desensitization task configuration information is obtained, executes desensitization task, goes to step z32；
Z32, the synchronous process of data is executed, goes to step z33；
Whether successful connection, if it is not, desensitization task execution fails, is tied if so, going to step z34 for z33, detection object library
Beam exits；
Z34, judge whether desensitization task is the desensitization of view grade, if so, step z35 is gone to, if it is not, it is de- to go to other types
It is quick；
Z35, make task distribution to source database, go to step z36；
Z36, data desensitization rule is obtained, goes to step z37；
Z37, judge whether target database table structure and source database are consistent, if so, going to step z38；If it is not, going to step
z312；
Z38, judge whether desensitization task is timed increase task, if so, step z39 is gone to, if it is not, going to step
z311；
Z39, it desensitizes to the data of increment, and is appended to object library, go to step z310；
Z310, judge whether addition succeeds, if so, going to step z313；If it is not, going to step z312；
Z311, deletion table identical with desensitization task object library table, go to step z312；
Z312, sensitive data discovery, desensitization and importing are carried out to table again, goes to step z313；
Z313, write-in relationship and view.
The invention proposes a kind of method and systems of the big data static database of the full mirror image of zero-copy desensitization, effectively
Solve the problems, such as that current big data static state desensitization exists.In data desensitization, data are extracted in batches, to supporting data zero-copy technique,
And data are not landed, and ensure that the safety of data in data desensitization；By way of distributed type assemblies, it is de- to improve data
The speed of quick task processing；Database performance is being influenced to support the data of timed increase to desensitize in the smallest situation；Guaranteeing
The case where rate that desensitizes, does not change customer data base, realizes the desensitization task of timing with increment；It is desensitized skill by research data base
Art realizes database level, table level, the desensitization of the data of view rank, and can retain the constraint relationship of database, and it is more to meet user
The scene demand of sample；By consistency desensitization algorithm, the consistency and business of data after the data desensitization of whole library grade ensure that
Relevance.
Detailed description of the invention
Fig. 1 is a kind of big data static database desensitization system architecture schematic diagram of the full mirror image of zero-copy of the invention；
Fig. 2 is a kind of big data static database desensitization method main flow signal of the full mirror image of zero-copy of the invention
Figure；
Fig. 3 is a kind of big data static database desensitization method data source registration process of the full mirror image of zero-copy of the invention
Schematic diagram；
Fig. 4 is that a kind of synchronous process of big data static database desensitization method data of the full mirror image of zero-copy of the invention is shown
It is intended to；
Fig. 5 is a kind of whole library grade data desensitization of big data static database desensitization method of the full mirror image of zero-copy of the invention
Flow diagram；
Fig. 6 is a kind of big data static database desensitization method table level data desensitization stream of the full mirror image of zero-copy of the invention
Journey schematic diagram；
Fig. 7 is a kind of big data static database desensitization method view grade data desensitization of the full mirror image of zero-copy of the invention
Flow diagram.
Specific embodiment
In order to better understand the present invention, the invention will now be described in detail with reference to the accompanying drawings.
A kind of big data static database desensitization system of the full mirror image of zero-copy mainly includes system management module, data source
Management module, data desensitization task execution module and desensitization configuration management module four module, wherein data desensitize task execution
Module is the core of whole system.The system as shown in Figure 1, a kind of big data static database of the full mirror image of zero-copy of the present invention desensitizes
System, the system management module that is managed including the basic function to system, to the source database address of system and destination address
The data source management module that is managed realizes that the data of configuration and execution and monitoring to desensitization task desensitize task execution
Module provides the desensitization configuration management module on configuration basis for desensitization task.Specifically, the system management module includes angle
Color management module, user management module, device management module, cluster management module.Role management module executes the angle to system
Color such as increases, deletes, looking into, changing at the operation, and is role authorization.User management module execution increases the operation user of system, deletes, looking into, changing
It Deng operation, and is the authorization of user progress role and data source.
The data source management module includes registration, modification, deletion, inquiry, enabling and the disabling function to source database
The source database address management module being managed；The data source management module further includes the storage address to data after desensitization
Registration, modification, deletion, inquiry, enabling and disabling be able to carry out the destination address management module of management.Source database address pipe
It manages module and realizes the functions such as registration, modification, deletion, inquiry, enabling and the disabling of source database.Destination address management module is realized
To the functions such as the registration, modification, deletion, inquiry of the storage address of data after desensitization, enabling and disabling, including target database
Address administration and folder address management.
Data desensitization task execution module includes that the whole library grade that is configured to the data desensitization task of whole library grade is quiet
State data desensitization module, the table level static data desensitization module that the data of table level desensitization task is configured, to view grade
The data desensitization of view grade static data desensitization module, all data desensitization tasks of monitoring that data desensitization task is configured is appointed
Business monitoring module.Database desensitization task execution module realizes matching to the data desensitization task of whole library grade, table level and view grade
It sets, execute and monitors, be the core of whole system.The data desensitization task of whole library grade is matched in whole library grade static data desensitization
It sets, configuration information includes timing, increment, sensitive data discovery, subset extraction, target output address etc..Table level static data is de-
Quick module configures the data desensitization task of table level, and configuration information includes timing, increment, sensitive data discovery, subset pumping
It takes, target output address etc..View grade static data desensitization module configures the data desensitization task of view grade, matches confidence
Breath includes timing, increment, sensitive data discovery, subset extraction, target output address etc..Institute is monitored according to desensitization task monitoring module
There is data desensitization task, supports to operate modification, deletion and stopping of task etc..
The desensitization configuration management module include sensitive classification system management module, data desensitization policy management module, from
It defines sensitive words segment management module, data desensitization algorithm management module, customized sensitive data management module, log query and divides
Analyse module.Desensitization configuration management is the basis of data desensitization task configuration, including sensitive classification system management, data desensitization strategy
Management, customized sensitive field, customized sensitive data, data desensitization algorithm management and log inquiry and analysis.Sensitivity point
Class system management module supports user to check the sensitive classification system of system default, supports the self-built sensitive classification system of user；Number
The increasing of the data desensitization strategy based on sensitive classification system is supported according to desensitization policy management module, the operation such as deletes, looks into, changes；It makes by oneself
Adopted sensitive words segment management module supports sensitive field and its affiliated sensitive kind in user's self-defining data library；It is customized quick
Feel data management module and supports the customized sensitive data of user and its Substitution Rules；Data desensitization algorithm management module supports user
Self-defining data desensitizes algorithm, and the operation such as is increased to customized algorithm, delete, look into, changing；Log query and analysis module are remembered
Recording system log and business diary, inquire log and are statisticallyd analyze.
As shown in Fig. 2, a kind of full mirror image big data static database desensitization method of zero-copy of the invention, using one kind zero
Full mirror image big data static database desensitization system is copied, desensitizes, includes the following steps: to data
S1, database is obtained, goes to step s2；
S2, input database data source information carry out database data Login, go to step s3；
S3, configuration data desensitization task, and start desensitization task, go to step s4；
S4, data are synchronous, go to step s5；
S5, monitoring objective library whether successful connection, if so, going to step s6；If it is not, going to step s7；
S6, judgement desensitization task type simultaneously select task, if desensitization task type is whole library grade desensitization, execute whole library grade
Desensitization task；If the task type that desensitizes executes table level desensitization task for table level desensitization；Drag desensitization task type de- for view grade
It is quick, then execute view grade desensitization task；
S7, desensitization task execution failure, terminate process.
Wherein, data source registration is the core of source database management, the part by extract source database data structure,
The constraint relationship and sensitive data discovery, supporting database desensitization task, as shown in figure 3, progress database data Login includes
Following steps:
The source database data source information that y1, input are registered, goes to step y2；
Whether the information inputted in y2, judgment step y1 is correct, if it is not, source database data source information registration failure, turns
To step y3；If so, going to step y4；
Y3, data source registration terminate；
Y4, the table structure for obtaining database, go to step y5；
Y5, constraint database relation information is obtained, goes to step y6；
Y6, data base view information is obtained, goes to step y7；
Y7, database data sample is extracted using random sampling algorithm, goes to step y8；
Y8, sensitive data identification is carried out to sample using intelligent self-adaptive sensitive data identification method, goes to step y9；
Y9, output sensitive data recognition result, go to step y10；
Y10, the success of source database data source registration.
The synchronous basis as data desensitization task of data, data, which synchronize, can effectively ensure that data desensitization Mission Success is held
Row, includes the following steps: as shown in figure 4, data synchronize
T1, system receive sync database event, and log-on data synchronous task goes to step t2；
T2, database is connected automatically, go to step t3；
T3, judge whether database connection succeeds, if so, step t4 is gone to, if it is not, database connection failure, goes to step
Rapid t10；
T4, database structure is obtained, goes to step t5；
T5, the database structure that step t4 is got is compared with table structure when data source registration, goes to step
t6；
Whether the comparison structure in t6, judgment step t5 is consistent, if so, going to step t10；If it is not, going to step t7；
Table structure when t7, more new registration, goes to step t8；
T8, the sensitive data based on multithreading find that mode finds the sensitive data in updated database automatically, turn
To step t9；
T9, data synchronize success；
T10, end.
It is that minimum particle size carries out the configuration of data desensitization task and holds that the data desensitization of whole library grade, which is using entire database,
Row includes the following steps: as shown in figure 5, executing whole library grade desensitization task
Z11, desensitization task configuration information is obtained, executes desensitization task, goes to step z12；
Z12, the synchronous process of data is executed, goes to step z13；
Whether successful connection, if it is not, desensitization task execution fails, is tied if so, going to step z14 for z13, detection object library
Beam exits；
Z14, judge whether desensitization task is whole library grade desensitization, if so, step z15 is gone to, if it is not, it is de- to go to other types
It is quick；
Z15, object library is judged with the presence or absence of relationship, if so, step z16 is gone to, if it is not, going to step z17；
Z16, delete target base relation, go to step z17；
Z17, make task distribution to source database, go to step z18；
Z18, data desensitization rule is obtained, goes to step z19；
Z19, judge whether target database table structure and source database are consistent, if so, going to step z110；If it is not, going to step
Rapid z114；
Z110, judge whether desensitization task is timed increase task, if so, step z111 is gone to, if it is not, going to step
z113；
Z111, it desensitizes to the data of increment, and is appended to object library, go to step z112；
Z112, judge whether addition succeeds, if so, going to step z115；If it is not, going to step z114；
Z113, deletion table identical with desensitization task object library table, go to step z114；
Z114, sensitive data discovery, desensitization and importing are carried out to table again, goes to step z115；
Z115, write-in relationship and view.
The data of table level desensitize, and are the configuration and execution that data desensitization task is carried out using table as minimum particle size, such as Fig. 6 institute
Show, executes table level desensitization task and include the following steps:
Z21, desensitization task configuration information is obtained, executes desensitization task, goes to step z22；
Z22, the synchronous process of data is executed, goes to step z23；
Whether successful connection, if it is not, desensitization task execution fails, is tied if so, going to step z24 for z23, detection object library
Beam exits；
Z24, judge whether desensitization task is table level desensitization, if so, step z25 is gone to, if it is not, it is de- to go to other types
It is quick；
Z25, make task distribution to source database, go to step z26；
Z26, data desensitization rule is obtained, goes to step z27；
Z27, judge whether target database table structure and source database are consistent, if so, going to step z28；If it is not, going to step
z212；
Z28, judge whether desensitization task is timed increase task, if so, step z29 is gone to, if it is not, going to step
z211；
Z29, it desensitizes to the data of increment, and is appended to object library, go to step z210；
Z210, judge whether addition succeeds, if so, going to step z213；If it is not, going to step z212；
Z211, deletion table identical with desensitization task object library table, go to step z212；
Z212, sensitive data discovery, desensitization and importing are carried out to table again, goes to step z213；
Z213, write-in relationship and view.
The data of view grade desensitize, and are the configuration and execution that data desensitization task is carried out using view as minimum particle size, such as Fig. 7
It is shown, it executes view grade desensitization task and includes the following steps:
Z31, desensitization task configuration information is obtained, executes desensitization task, goes to step z32；
Z32, the synchronous process of data is executed, goes to step z33；
Whether successful connection, if it is not, desensitization task execution fails, is tied if so, going to step z34 for z33, detection object library
Beam exits；
Z34, judge whether desensitization task is the desensitization of view grade, if so, step z35 is gone to, if it is not, it is de- to go to other types
It is quick；
Z35, make task distribution to source database, go to step z36；
Z36, data desensitization rule is obtained, goes to step z37；
Z37, judge whether target database table structure and source database are consistent, if so, going to step z38；If it is not, going to step
z312；
Z38, judge whether desensitization task is timed increase task, if so, step z39 is gone to, if it is not, going to step
z311；
Z39, it desensitizes to the data of increment, and is appended to object library, go to step z310；
Z310, judge whether addition succeeds, if so, going to step z313；If it is not, going to step z312；
Z311, deletion table identical with desensitization task object library table, go to step z312；
Z312, sensitive data discovery, desensitization and importing are carried out to table again, goes to step z313；
Z313, write-in relationship and view.
The invention proposes a kind of method and systems of the big data static database of the full mirror image of zero-copy desensitization, effectively
Solve the problems, such as that current big data static state desensitization exists.In data desensitization, data are extracted in batches, to supporting data zero-copy technique,
And data are not landed, and ensure that the safety of data in data desensitization；By way of distributed type assemblies, it is de- to improve data
The speed of quick task processing；Database performance is being influenced to support the data of timed increase to desensitize in the smallest situation；Guaranteeing
The case where rate that desensitizes, does not change customer data base, realizes the desensitization task of timing with increment；It is desensitized skill by research data base
Art realizes database level, table level, the desensitization of the data of view rank, and can retain the constraint relationship of database, and it is more to meet user
The scene demand of sample；By consistency desensitization algorithm, the consistency and business of data after the data desensitization of whole library grade ensure that
Relevance.
The foregoing is only a preferred embodiment of the present invention, but scope of protection of the present invention is not limited thereto,
Anyone skilled in the art within the technical scope of the present disclosure, according to the technique and scheme of the present invention and its
Inventive concept is subject to equivalent substitution or change, should be covered by the protection scope of the present invention.
Claims (11)
- The system 1. a kind of big data static database of full mirror image of zero-copy desensitizes, which is characterized in that including the basis to system System management module that function is managed, the data source control that the source database address of system and destination address are managed Module, realization are provided to the data desensitization task execution module of configuration and execution and the monitoring of desensitization task, for desensitization task Configure the desensitization configuration management module on basis.
- The system 2. a kind of big data static database of the full mirror image of zero-copy according to claim 1 desensitizes, feature exist In the system management module includes role management module, user management module, device management module, cluster management module.
- The system 3. a kind of big data static database of the full mirror image of zero-copy according to claim 1 desensitizes, feature exist In the data source management module includes that registration, modification, deletion, inquiry, enabling and the disabling function to source database carry out pipe The source database address management module of reason；The data source management module further includes the note to the storage address of data after desensitization Volume, modification, deletion, inquiry, enabling and disabling are able to carry out the destination address management module of management.
- The system 4. a kind of big data static database of the full mirror image of zero-copy according to claim 1 desensitizes, feature exist In the data desensitization task execution module includes the whole library grade static data configured to the data desensitization task of whole library grade It is desensitization module, the table level static data desensitization module that the data of table level desensitization task is configured, de- to the data of view grade The data desensitization Mission Monitor of view grade static data desensitization module, all data desensitization tasks of monitoring that quick task is configured Module.
- The system 5. a kind of big data static database of the full mirror image of zero-copy according to claim 1 desensitizes, feature exist In, the desensitization configuration management module include sensitive classification system management module, it is data desensitization policy management module, customized quick Feel field management module, data desensitization algorithm management module, customized sensitive data management module, log query and analysis mould Block.
- 6. a kind of full mirror image big data static database desensitization method of zero-copy, which is characterized in that any using claim 1-5 A kind of full mirror image big data static database desensitization system of zero-copy, desensitizes to data, includes the following steps:S1, database is obtained, goes to step s2；S2, input database data source information carry out database data Login, go to step s3；S3, configuration data desensitization task, and start desensitization task, go to step s4；S4, data are synchronous, go to step s5；S5, monitoring objective library whether successful connection, if so, going to step s6；If it is not, going to step s7；S6, judgement desensitization task type simultaneously select task, if desensitization task type is whole library grade desensitization, execute whole library grade desensitization Task；If the task type that desensitizes executes table level desensitization task for table level desensitization；Drag desensitization task type for the desensitization of view grade, Then execute view grade desensitization task；S7, desensitization task execution failure, terminate process.
- 7. a kind of big data static database desensitization method of the full mirror image of zero-copy according to claim 6, feature exist In progress database data Login includes the following steps:The source database data source information that y1, input are registered, goes to step y2；Whether the information inputted in y2, judgment step y1 is correct, if it is not, source database data source information registration failure, goes to step Rapid y3；If so, going to step y4；Y3, data source registration terminate；Y4, the table structure for obtaining database, go to step y5；Y5, constraint database relation information is obtained, goes to step y6；Y6, data base view information is obtained, goes to step y7；Y7, database data sample is extracted using random sampling algorithm, goes to step y8；Y8, sensitive data identification is carried out to sample using intelligent self-adaptive sensitive data identification method, goes to step y9；Y9, output sensitive data recognition result, go to step y10；Y10, the success of source database data source registration.
- 8. a kind of big data static database desensitization method of the full mirror image of zero-copy according to claim 7, feature exist In data, which synchronize, to be included the following steps:T1, system receive sync database event, and log-on data synchronous task goes to step t2；T2, database is connected automatically, go to step t3；T3, judge whether database connection succeeds, if so, step t4 is gone to, if it is not, database connection failure, goes to step t10；T4, database structure is obtained, goes to step t5；T5, the database structure that step t4 is got is compared with table structure when data source registration, goes to step t6；Whether the comparison structure in t6, judgment step t5 is consistent, if so, going to step t10；If it is not, going to step t7；Table structure when t7, more new registration, goes to step t8；T8, the sensitive data based on multithreading find that mode finds the sensitive data in updated database automatically, go to step Rapid t9；T9, data synchronize success；T10, end.
- 9. a kind of big data static database desensitization method of the full mirror image of zero-copy according to claim 8, feature exist In executing whole library grade desensitization task and include the following steps:Z11, desensitization task configuration information is obtained, executes desensitization task, goes to step z12；Z12, the synchronous process of data is executed, goes to step z13；Whether successful connection, if it is not, desensitization task execution fails, terminates to move back if so, going to step z14 for z13, detection object library Out；Z14, judge whether desensitization task is whole library grade desensitization, if so, step z15 is gone to, if it is not, going to other types desensitization；Z15, object library is judged with the presence or absence of relationship, if so, step z16 is gone to, if it is not, going to step z17；Z16, delete target base relation, go to step z17；Z17, make task distribution to source database, go to step z18；Z18, data desensitization rule is obtained, goes to step z19；Z19, judge whether target database table structure and source database are consistent, if so, going to step z110；If it is not, going to step z114；Z110, judge whether desensitization task is timed increase task, if so, step z111 is gone to, if it is not, going to step z113；Z111, it desensitizes to the data of increment, and is appended to object library, go to step z112；Z112, judge whether addition succeeds, if so, going to step z115；If it is not, going to step z114；Z113, deletion table identical with desensitization task object library table, go to step z114；Z114, sensitive data discovery, desensitization and importing are carried out to table again, goes to step z115；Z115, write-in relationship and view.
- 10. a kind of big data static database desensitization method of the full mirror image of zero-copy according to claim 9, feature exist In execution table level desensitization task includes the following steps:Z21, desensitization task configuration information is obtained, executes desensitization task, goes to step z22；Z22, the synchronous process of data is executed, goes to step z23；Whether successful connection, if it is not, desensitization task execution fails, terminates to move back if so, going to step z24 for z23, detection object library Out；Z24, judge whether desensitization task is table level desensitization, if so, step z25 is gone to, if it is not, going to other types desensitization；Z25, make task distribution to source database, go to step z26；Z26, data desensitization rule is obtained, goes to step z27；Z27, judge whether target database table structure and source database are consistent, if so, going to step z28；If it is not, going to step z212；Z28, judge whether desensitization task is timed increase task, if so, step z29 is gone to, if it is not, going to step z211；Z29, it desensitizes to the data of increment, and is appended to object library, go to step z210；Z210, judge whether addition succeeds, if so, going to step z213；If it is not, going to step z212；Z211, deletion table identical with desensitization task object library table, go to step z212；Z212, sensitive data discovery, desensitization and importing are carried out to table again, goes to step z213；Z213, write-in relationship and view.
- 11. a kind of big data static database desensitization method of the full mirror image of zero-copy according to claim 10, feature It is, executes view grade desensitization task and include the following steps:Z31, desensitization task configuration information is obtained, executes desensitization task, goes to step z32；Z32, the synchronous process of data is executed, goes to step z33；Whether successful connection, if it is not, desensitization task execution fails, terminates to move back if so, going to step z34 for z33, detection object library Out；Z34, judge whether desensitization task is the desensitization of view grade, if so, step z35 is gone to, if it is not, going to other types desensitization；Z35, make task distribution to source database, go to step z36；Z36, data desensitization rule is obtained, goes to step z37；Z37, judge whether target database table structure and source database are consistent, if so, going to step z38；If it is not, going to step z312；Z38, judge whether desensitization task is timed increase task, if so, step z39 is gone to, if it is not, going to step z311；Z39, it desensitizes to the data of increment, and is appended to object library, go to step z310；Z310, judge whether addition succeeds, if so, going to step z313；If it is not, going to step z312；Z311, deletion table identical with desensitization task object library table, go to step z312；Z312, sensitive data discovery, desensitization and importing are carried out to table again, goes to step z313；Z313, write-in relationship and view.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811563203.6A CN109657496B (en) | 2018-12-20 | 2018-12-20 | Zero-copy full-mirror-image big data static database desensitization system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811563203.6A CN109657496B (en) | 2018-12-20 | 2018-12-20 | Zero-copy full-mirror-image big data static database desensitization system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109657496A true CN109657496A (en) | 2019-04-19 |
| CN109657496B CN109657496B (en) | 2022-07-05 |
Family
ID=66115360
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811563203.6A Active CN109657496B (en) | 2018-12-20 | 2018-12-20 | Zero-copy full-mirror-image big data static database desensitization system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109657496B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110532799A (en) * | 2019-07-31 | 2019-12-03 | 平安科技（深圳）有限公司 | Data desensitization control method, electronic device and computer readable storage medium |
| CN111177785A (en) * | 2019-12-31 | 2020-05-19 | 广东鸿数科技有限公司 | Desensitization processing method for private data of enterprise-based business system |
| CN111858546A (en) * | 2020-06-22 | 2020-10-30 | 网联清算有限公司 | Data processing method, device and system |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120272329A1 (en) * | 2007-11-15 | 2012-10-25 | International Business Machines Corporation | Obfuscating sensitive data while preserving data usability |
| CN106407843A (en) * | 2016-10-17 | 2017-02-15 | 深圳中兴网信科技有限公司 | Data desensitization method and data desensitization device |
| CN106529329A (en) * | 2016-10-11 | 2017-03-22 | 中国电子科技网络信息安全有限公司 | Desensitization system and desensitization method used for big data |
| CN106599713A (en) * | 2016-11-11 | 2017-04-26 | 中国电子科技网络信息安全有限公司 | Database masking system and method based on big data |
| CN106778351A (en) * | 2016-12-30 | 2017-05-31 | 中国民航信息网络股份有限公司 | Data desensitization method and device |
| CN107403111A (en) * | 2017-08-10 | 2017-11-28 | 中国民航信息网络股份有限公司 | HIVE data desensitization method and device |
| CN107441317A (en) * | 2016-05-30 | 2017-12-08 | 王停 | It is a kind of to be used for the special Chinese medicinal formulae for reporting the treatment of constitution allergic rhinitis |
| CN207489017U (en) * | 2017-10-23 | 2018-06-12 | 中恒华瑞（北京）信息技术有限公司 | Data desensitization system |
-
2018
- 2018-12-20 CN CN201811563203.6A patent/CN109657496B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120272329A1 (en) * | 2007-11-15 | 2012-10-25 | International Business Machines Corporation | Obfuscating sensitive data while preserving data usability |
| CN107441317A (en) * | 2016-05-30 | 2017-12-08 | 王停 | It is a kind of to be used for the special Chinese medicinal formulae for reporting the treatment of constitution allergic rhinitis |
| CN106529329A (en) * | 2016-10-11 | 2017-03-22 | 中国电子科技网络信息安全有限公司 | Desensitization system and desensitization method used for big data |
| CN106407843A (en) * | 2016-10-17 | 2017-02-15 | 深圳中兴网信科技有限公司 | Data desensitization method and data desensitization device |
| CN106599713A (en) * | 2016-11-11 | 2017-04-26 | 中国电子科技网络信息安全有限公司 | Database masking system and method based on big data |
| CN106778351A (en) * | 2016-12-30 | 2017-05-31 | 中国民航信息网络股份有限公司 | Data desensitization method and device |
| CN107403111A (en) * | 2017-08-10 | 2017-11-28 | 中国民航信息网络股份有限公司 | HIVE data desensitization method and device |
| CN207489017U (en) * | 2017-10-23 | 2018-06-12 | 中恒华瑞（北京）信息技术有限公司 | Data desensitization system |
Non-Patent Citations (3)
| Title |
|---|
| ATTILAX: "《Atitit 数据库视图与表的wrap与层级查询规范》", 《HTTPS://BLOG.CSDN.NET/ATTILAX/ARTICLE/DETAILS/69661939》 * |
| DR NITIN: "《How to desensitize data in Excel》", 《HTTPS://EFFICIENCY365.COM/2014/11/09/HOW-TO-DESENSITIZE-DATA-IN-EXCEL/》 * |
| 陈学亮: "《数据仓库测试环境数据搬运方法研究》", 《计算机光盘软件与应用》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110532799A (en) * | 2019-07-31 | 2019-12-03 | 平安科技（深圳）有限公司 | Data desensitization control method, electronic device and computer readable storage medium |
| CN110532799B (en) * | 2019-07-31 | 2023-03-24 | 平安科技（深圳）有限公司 | Data desensitization control method, electronic device and computer readable storage medium |
| CN111177785A (en) * | 2019-12-31 | 2020-05-19 | 广东鸿数科技有限公司 | Desensitization processing method for private data of enterprise-based business system |
| CN111858546A (en) * | 2020-06-22 | 2020-10-30 | 网联清算有限公司 | Data processing method, device and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109657496B (en) | 2022-07-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106164859B (en) | Equipment strategy manager | |
| CN105206114B (en) | Fly control, license, security maintenance method and apparatus, server, aircraft | |
| Ulusoy et al. | GuardMR: Fine-grained security policy enforcement for MapReduce systems | |
| CN104484617B (en) | A kind of Access and control strategy of database method based on many strategy fusions | |
| US8863276B2 (en) | Automated role adjustment in a computer system | |
| CN109657496A (en) | A kind of big data static database desensitization system and method for the full mirror image of zero-copy | |
| CN102088379B (en) | Detecting method and device of client honeypot webpage malicious code based on sandboxing technology | |
| CN104956376B (en) | Using the methods and techniques with equipment control in virtualized environment | |
| CN103839003B (en) | Malicious file detection method and device | |
| CN108683652A (en) | A kind of method and device of the processing attack of Behavior-based control permission | |
| CN106529329A (en) | Desensitization system and desensitization method used for big data | |
| US20170279840A1 (en) | Automated event id field analysis on heterogeneous logs | |
| CN107004089A (en) | Malware detection method and its system | |
| US20080313556A1 (en) | Access control system with rules engine architecture | |
| CN112564988B (en) | Alarm processing method and device and electronic equipment | |
| CN107430660A (en) | For the method and system for the anonymous mass-rent of automation for characterizing equipment behavior | |
| CN103765430A (en) | Data leak prevention system and method | |
| CN105378648A (en) | Self-provisioning access control | |
| CN106936812B (en) | File privacy disclosure detection method based on Petri network in cloud environment | |
| CN104462937B (en) | Operating system peripheral access permission control method based on users | |
| US10726054B2 (en) | Extraction of policies from natural language documents for physical access control | |
| CN111813497A (en) | Container environment anomaly detection method, device, medium and computer equipment | |
| CN109587151A (en) | Access control method, device, equipment and computer readable storage medium | |
| CN107480553A (en) | A kind of data exploration system, method, equipment and storage medium | |
| RU2645265C2 (en) | System and method of blocking elements of application interface |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |