CN109726572A - Data management-control method, device, equipment, computer storage medium and system - Google Patents
Data management-control method, device, equipment, computer storage medium and system Download PDFInfo
- Publication number
- CN109726572A CN109726572A CN201811626565.5A CN201811626565A CN109726572A CN 109726572 A CN109726572 A CN 109726572A CN 201811626565 A CN201811626565 A CN 201811626565A CN 109726572 A CN109726572 A CN 109726572A
- Authority
- CN
- China
- Prior art keywords
- data
- stored
- sensitive
- sensitive data
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention discloses a kind of data management-control method, device, equipment, computer storage medium and systems.Wherein, data management-control method includes: to determine the corresponding sensitive data information of data to be stored, and store data and its corresponding sensitive data information to be stored in storing data；When reading data, obtains sensitive data and read rule, data to be read and the corresponding sensitive data information of data to be read, and rule, data to be read and its corresponding sensitive data information are read according to sensitive data, generate output data.According to embodiments of the present invention, the data of big data platform can be managed, improves the safety of data.
Description
Technical field
The invention belongs to technical field of data security more particularly to a kind of data management-control method, device, equipment, computer
Storage medium and system.
Background technique
Data are carried out currently, big data platform mainly passes through " interface message processor (IMP)-agency service-big data platform component " mode
Control.
Interface message processor (IMP) is common platform, operation maintenance personnel only need login interface machine authorization account can to big data platform into
Row operation management, and the permission of each authorization account is generally identical.But, it is understood that there may be more people share same authorization account
Situation leads to the specific identity for being difficult to confirm the operation maintenance personnel for implementing operation management operation, so that can not carry out to operation maintenance personnel
It calls to account.
In addition, when downloaded from big data platform/export data when, can not know the circulation after data fall on interface message processor (IMP)
Situation, if data are directly downloaded on user terminal, it is most likely that cause the risk of data leak.Also, due to interface
Machine connects big data platform component by way of agency service, also can not be before agency service call operation order to O&M
Management operation is managed in real time, can equally bring security risk.
Summary of the invention
The embodiment of the present invention provides a kind of data management-control method, device, equipment, computer storage medium and system, can
The data of big data platform are managed, the safety of data is improved.
On the one hand, the embodiment of the present invention provides a kind of data management-control method, comprising:
In storing data, the corresponding sensitive data information of data to be stored is determined, and store the number to be stored
According to and its corresponding sensitive data information；
When reading data, it is corresponding to obtain sensitive data reading rule, data to be read and the data to be read
Sensitive data information, and rule, the data to be read and its corresponding sensitive data are read according to the sensitive data
Information generates output data.
Further, it is determined that before the corresponding sensitive data information of data to be stored, further includes:
If the data to be stored are greater than preset data amount, the data to be stored are carried out at distributed computing
Reason.
Further, it is determined that the corresponding sensitive data information of data to be stored includes:
If the data to be stored are structural data, determined using structured data analysis component described to be stored
The corresponding sensitive data information of data；
If the data to be stored are unstructured data, determined using unstructured data analytic unit described wait deposit
The corresponding sensitive data information of the data of storage.
Further, to include in the data to be stored include the corresponding sensitive data information of the data to be stored
Data Position and the sensitive data in the data to be stored of sensitive data, the sensitive data grade.
Further, the data to be stored are stored and its corresponding sensitive data information includes:
The data to be stored are stored to database；
The corresponding sensitive data information of the data to be stored is stored to signature library.
Further, obtaining the method that sensitive data reads rule includes:
Obtain the user identifier for reading data；
According to the user identifier, inquires the sensitive data corresponding with the user identifier and read rule.
Further, the sensitive data read rule include sensitive data is carried out Fuzzy Processing and to sensitive data into
Row encryption.
Further, if it is that sensitive data is encrypted that the sensitive data, which reads rule, when reading data,
Further include:
Obtain permission modification instruction；
It is modified and is instructed according to the permission, decrypt the sensitive data being encrypted in the output data, and regenerate defeated
Data out.
On the other hand, the embodiment of the invention provides a kind of data control device, described device includes:
Data storage cell is configured in storing data, determines the corresponding sensitive data information of data to be stored,
And store the data and its corresponding sensitive data information to be stored；
Data outputting unit is configured to when reading data, obtain sensitive data read rule, data to be read and
The corresponding sensitive data information of the data to be read, and according to the sensitive data read rule, data to be read and
Its corresponding sensitive data information generates output data.
Another aspect, the embodiment of the invention provides a kind of data management and control devices, the equipment includes: processor and deposits
Contain the memory of computer program instructions；
The processor realizes above-mentioned data management-control method when executing the computer program instructions.
In another aspect, being deposited in the computer storage medium the embodiment of the invention provides a kind of computer storage medium
Computer program instructions are contained, the computer program instructions realize above-mentioned data management-control method when being executed by processor.
In another aspect, the embodiment of the invention provides a kind of data managing and control systems, comprising:
Database, for storing data；
User terminal, for issuing the instruction letter for reading data to the data database storing or from the database
Breath；
Data manage module, connect respectively with the user terminal and the database communication, for according to the user
The described instruction information that terminal issues executes the data management-control method as described in claim 1-8 any one.
Further, the database includes at least HDFS database, Hive database and HBase database.
Data management-control method, device, equipment, computer storage medium and the system of the embodiment of the present invention, for big number
It is managed according to the data of platform, the sensitive data information of data to be stored can be first identified in storing data, then together
When store data and its corresponding sensitive data information to be stored, when reading data according to sensitive data read rule, to
The data of reading and its corresponding sensitive data information generate output data, so that operation maintenance personnel is only exported when reading data
By carrying out treated output data to sensitive data, the Information Security of big data platform is improved.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, will make below to required in the embodiment of the present invention
Attached drawing is briefly described, for those of ordinary skill in the art, without creative efforts, also
Other drawings may be obtained according to these drawings without any creative labor.
Fig. 1 is the flow diagram of data management-control method provided by one embodiment of the present invention；
Fig. 2 is the flow diagram of the specific method of determining sensitive data information provided by one embodiment of the present invention；
Fig. 3 is the flow diagram of the specific method provided by one embodiment of the present invention for reading data；
Fig. 4 is the structural schematic diagram of data control device provided by one embodiment of the present invention；
Fig. 5 is the hardware structural diagram of data management and control devices provided in an embodiment of the present invention；
Fig. 6 is the structural schematic diagram of data managing and control system provided by one embodiment of the present invention；
Fig. 7 is the process signal that the data in HDFS database are read using the data managing and control system of the embodiment of the present invention
Figure；
Fig. 8 is the process signal that the data in Hive database are read using the data managing and control system of the embodiment of the present invention
Figure；
Fig. 9 is the process signal that the data in HBase database are read using the data managing and control system of the embodiment of the present invention
Figure.
Specific embodiment
The feature and exemplary embodiment of various aspects of the invention is described more fully below, in order to make mesh of the invention
, technical solution and advantage be more clearly understood, below in conjunction with drawings and the specific embodiments, the present invention is carried out further detailed
Description.It should be understood that specific embodiment described herein is only configured to explain the present invention, it is not configured as limiting this hair
It is bright.To those skilled in the art, the present invention can be in the case where not needing some details in these details
Implement.The description of embodiment is preferably managed just for the sake of being provided by showing example of the invention of the invention below
Solution.
It should be noted that, in this document, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation
In any actual relationship or order or sequence.Moreover, the terms "include", "comprise" or its any other variant are intended to
Non-exclusive inclusion, so that the process, method, article or equipment including a series of elements is not only wanted including those
Element, but also including other elements that are not explicitly listed, or further include for this process, method, article or equipment
Intrinsic element.In the absence of more restrictions, the element limited by sentence " including ... ", it is not excluded that including
There is also other identical elements in the process, method, article or equipment of the element.
In order to solve prior art problem, the embodiment of the invention provides a kind of data management-control method, device, equipment, meter
Calculation machine storage medium and system.Data management-control method is provided for the embodiments of the invention first below to be introduced.
Fig. 1 shows the flow diagram of data management-control method provided by one embodiment of the present invention.As shown in Figure 1, should
Data management-control method includes:
S110, in storing data, determine the corresponding sensitive data information of data to be stored, and store described wait store
Data and its corresponding sensitive data information；
S120, when reading data, obtain sensitive data and read rule, data to be read and the data to be read
Corresponding sensitive data information, and rule, the data to be read and its corresponding sensitivity are read according to the sensitive data
Data information generates output data.
The embodiment of the present invention can first be identified in storing data wait deposit for managing to the data of big data platform
Then the sensitive data information of the data of storage stores data and its corresponding sensitive data information to be stored simultaneously, is reading
Rule, data to be read and its corresponding sensitive data information are read according to sensitive data when data and generate output data, is made
Operation maintenance personnel when reading data, only output improves big data and puts down by carrying out treated output data to sensitive data
The Information Security of platform.
In embodiments of the present invention, sensitive data may include the personal information of platform user, such as name, phone number
Code, mailbox, address, bank's card number etc..Therefore, it is exported again after handling sensitive information to the operation maintenance personnel for reading data,
The data being likely to occur after can preventing data from being obtained by operation maintenance personnel leak, and further increase the safety of data, ensure that
The individual privacy information of platform user.
In the step S110 of the embodiment of the present invention, before determining the corresponding sensitive data information of data to be stored, also
Include:
If data to be stored are greater than preset data amount, the data for treating storage carry out distributed computing processing.Even to
The data of storage are greater than preset data amount, then illustrate the biggish data of Data Data to be stored, it is therefore desirable to first treat and deposit
The data of storage carry out distributed computing (Map Reduce, MR) processing, generate MR data file, then to the MR data file into
The identification of row sensitive data information.
If data to be stored are not more than preset data amount, illustrate that data to be stored belong to lesser data at this time, it can
Directly to carry out the identification of sensitive data information.
It should be noted that in embodiments of the present invention, preset data amount is configured as needed.
In the step S110 of the embodiment of the present invention, the specific side of the corresponding sensitive data information of data to be stored is determined
Method may include:
If data to be stored are structural data, determine that data to be stored are corresponding using structured data analysis component
Sensitive data information；Wherein, structured data analysis component may include structural data analytics engine.
If data to be stored are unstructured data, data to be stored are determined using unstructured data analytic unit
Corresponding sensitive data information；Wherein, unstructured data analytic unit may include unstructured data analytics engine.
Therefore, in embodiments of the present invention, side corresponding with structural data and unstructured data can be utilized respectively
Method determines the corresponding sensitive data information of data to be stored, and guarantees the recognition efficiency and accuracy of sensitive data.
In embodiments of the present invention, to include in data to be stored include the corresponding sensitive data information of data to be stored
Data Position and sensitive data in data to be stored of sensitive data, sensitive data grade.
Wherein, the method for identifying the sensitive data for including in data to be stored may include: to be identified using sensitive data
Rule identifies the sensitive data for including in data to be stored, specifically, sensitive data recognition rule can for it is preset just
Then expression formula or data dictionary.After identifying the sensitive data for including in the data wait store, the available sensitivity number
According to locating Data Position in data to be stored, for example, the sensitive data specifically which field of which file which
In a byte.
In other embodiments of the present invention, sensitive data information can also include the type of sensitive data.Wherein, sensitive number
According to type can be associated with the grade of sensitive data.
It should be noted that in embodiments of the present invention, sensitive data recognition rule, which can according to need, to be added, repairs
Change and deletes.
In the step S110 of the embodiment of the present invention, data and its corresponding sensitive data packet to be stored are stored
It includes:
Data to be stored are stored to database, the corresponding sensitive data information of data to be stored is stored to label
Library.Wherein, database can be the database of big data platform, and signature library can be Solr database.In the embodiment of the present invention
In, data to be stored and its corresponding sensitive data information can be stored respectively, and only by data to be stored
It is stored in big data platform, and its corresponding sensitive data information is stored in individual Solr database, can be not take up big
The database resource of data platform, and since Solr database is index data base, search efficiency can be improved.
Fig. 2 shows the signals of the process of the specific method of determining sensitive data information provided by one embodiment of the present invention
Figure.As shown in Fig. 2, determining that the specific method of sensitive data information may include:
Step S201, data to be stored are obtained；
Step S202, it determines whether data to be stored are greater than preset data amount, if more than preset data amount, then executes step
Rapid S203 thens follow the steps S204 if being not more than preset data amount；
Step S203, data to be stored are handled and create MR file；
Step S204, determine whether data to be stored are structural data, if structural data, are thened follow the steps
S205, if not structural data, thens follow the steps S206；
Step S205, the corresponding sensitive data information of data to be stored is determined using structured data analysis component, so
Step S207 is executed afterwards；
Step S206, the corresponding sensitive data information of data to be stored is determined using unstructured data analytic unit,
Then step S207 is executed；
Step S207, determine whether to find sensitive data, if discovery sensitive data, thens follow the steps S208, if not finding,
Then end processing；
Step S208, the grade of the sensitive data of discovery is determined using sensitive data hierarchy rules；
Step S209, Data Position of the sensitive data in data to be stored is determined；
Step S210, the number by the sensitive data, sensitive data that include in data to be stored in data to be stored
According to the grade of position and sensitive data as sensitive data information preservation to signature library.
It in embodiments of the present invention, can should if being abnormal during above-mentioned determination sensitive data information
The information of data to be stored is recorded in sensitive data identification exception table, terminates identification, and directly by the number to be stored
According to storing to database.Then for having stored in the data in database, it can be spaced, redefine at predetermined time intervals
Sensitive data identifies the sensitive data information in the corresponding data of information in exception table, and sensitive data information will be determined
The corresponding information of data is deleted from sensitive data identification exception table.
In the step S120 of the embodiment of the present invention, the specific method for obtaining sensitive data reading rule may include:
Obtain the user identifier for reading data；
According to user identifier, inquires sensitive data corresponding with user identifier and read rule.
The reading permission of the corresponding operation maintenance personnel of the user identifier can be determined, thus according to reading according to user identifier
Weighting limit inquires corresponding sensitive data and reads rule.
In other embodiments of the present invention, the grade of sensitive data and the type of sensitive data can also be obtained, then root
According to user identifier, the type of the grade of sensitive data and sensitive data, determine that the corresponding operation maintenance personnel of the user identifier is corresponding
Sensitive data reads rule.
In embodiments of the present invention, it may include Fuzzy Processing being carried out to sensitive data and to quick that sensitive data, which reads rule,
Sense data are encrypted.Wherein, carrying out Fuzzy Processing to sensitive data is to be blurred sensitive data, for example, only showing
Show the individual characters in sensitive data, this processing be one kind can not inversely processing, when the data that operation maintenance personnel is read are by mould
It can not be initial data by the data convert by other operations after the data for pasting processing.Sensitive data is encrypted
It is to encode sensitive data using the encryption rule of setting, sensitive data is shown in display by the coding, this
Processing can be set to that reversible treatment may be set to be can not inversely processing.When encryption is reversible treatment, can pass through
The data convert is initial data by other operations.
Fig. 3 shows the flow diagram of the specific method provided by one embodiment of the present invention for reading data.Such as Fig. 3 institute
Show, if sensitive data reads rule to carry out reversible encryption to sensitive data, when reading data, is exported when generating
After data, the method for reading data can also include:
S320, permission modification instruction is obtained；
S330, instruction is modified according to permission, decrypts the sensitive data being encrypted in output data, and regenerate output number
According to.
At this point, operation maintenance personnel can issue the authority application for being directed to the output data to the administrative staff of big data platform,
If administrative staff ratify the authority application, permission modification instruction can be issued.It is instructed at this point, can both be modified according to the permission
The sensitive data being encrypted in output data is decrypted, and regenerates output data using original data to be read.
Fig. 4 shows the structural schematic diagram of data control device provided by one embodiment of the present invention.As shown in figure 4, should
Data control device includes:
Data storage cell 410 is configured in storing data, determines the corresponding sensitive data letter of data to be stored
Breath, and store data and its corresponding sensitive data information to be stored；
Data outputting unit 420 is configured to when reading data, is obtained sensitive data and is read rule, number to be read
Rule, data to be read and its right are read according to sensitive data information corresponding with data to be read, and according to sensitive data
The sensitive data information answered generates output data.
The embodiment of the present invention can first be identified in storing data wait deposit for managing to the data of big data platform
Then the sensitive data information of the data of storage stores data and its corresponding sensitive data information to be stored simultaneously, is reading
Rule, data to be read and its corresponding sensitive data information are read according to sensitive data when data and generate output data, is made
Operation maintenance personnel when reading data, only output improves big data and puts down by carrying out treated output data to sensitive data
The Information Security of platform.
In embodiments of the present invention, if data storage cell 410 can be further configured to data to be stored and be greater than
Preset data amount after the data progress distributed computing processing for treating storage, then determines the corresponding sensitive number of data to be stored
It is believed that breath.
In embodiments of the present invention, it is if data storage cell 410 can also be further configured to data to be stored
Structural data determines the corresponding sensitive data information of data to be stored using structured data analysis component；If wait store
Data be unstructured data, utilize unstructured data analytic unit to determine the corresponding sensitive data letter of data to be stored
Breath.
In embodiments of the present invention, to include in data to be stored include the corresponding sensitive data information of data to be stored
Data Position and sensitive data in data to be stored of sensitive data, sensitive data grade.
In embodiments of the present invention, data outputting unit 420 can be further configured to obtain the user for reading data
Mark, and according to user identifier, it inquires sensitive data corresponding with user identifier and reads rule.
In embodiments of the present invention, data outputting unit 420, which can also be further configured to work as, generates output data
Afterwards, permission modification instruction is obtained, and is modified and is instructed according to permission, decrypts the sensitive data being encrypted in output data, and again
Generate output data.
Fig. 5 shows the hardware structural diagram of data management and control devices provided in an embodiment of the present invention.
It may include processor 501 and the memory 502 for being stored with computer program instructions in data management and control devices.
Specifically, above-mentioned processor 501 may include central processing unit (CPU) or specific integrated circuit
(Application Specific Integrated Circuit, ASIC), or may be configured to implement implementation of the present invention
One or more integrated circuits of example.
Memory 502 may include the mass storage for data or instruction.For example it rather than limits, memory
502 may include hard disk drive (Hard Disk Drive, HDD), floppy disk drive, flash memory, CD, magneto-optic disk, tape or logical
With the combination of universal serial bus (Universal Serial Bus, USB) driver or two or more the above.It is closing
In the case where suitable, memory 502 may include the medium of removable or non-removable (or fixed).In a suitable case, it stores
Device 502 can be inside or outside synthesized gateway disaster tolerance equipment.In a particular embodiment, memory 502 is nonvolatile solid state
Memory.In a particular embodiment, memory 502 includes read-only memory (ROM).In a suitable case, which can be
ROM, programming ROM (PROM), erasable PROM (EPROM), the electric erasable PROM (EEPROM), electrically rewritable of masked edit program
The combination of ROM (EAROM) or flash memory or two or more the above.
Processor 501 is by reading and executing the computer program instructions stored in memory 502, to realize above-mentioned implementation
Any one data management-control method in example.
In one example, data management and control devices may also include communication interface 503 and bus 510.Wherein, as shown in figure 5,
Processor 501, memory 502, communication interface 503 connect by bus 510 and complete mutual communication.
Communication interface 503 is mainly used for realizing in the embodiment of the present invention between each module, device, unit and/or equipment
Communication.
Bus 510 includes hardware, software or both, and the component of data management and control devices is coupled to each other together.Citing comes
It says rather than limits, bus may include accelerated graphics port (AGP) or other graphics bus, enhance Industry Standard Architecture (EISA) always
Line, front side bus (FSB), super transmission (HT) interconnection, the interconnection of Industry Standard Architecture (ISA) bus, infinite bandwidth, low pin count
(LPC) bus, memory bus, micro- channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express
(PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association part (VLB) bus or other conjunctions
The combination of suitable bus or two or more the above.In a suitable case, bus 510 may include one or more
Bus.Although specific bus has been described and illustrated in the embodiment of the present invention, the present invention considers any suitable bus or interconnection.
The data management and control devices can execute the data management-control method in the embodiment of the present invention, to realize in conjunction with above-mentioned attached
Scheme the data management-control method and device of description.
In addition, the embodiment of the present invention can provide a kind of computer storage in conjunction with the data management-control method in above-described embodiment
Medium is realized.Computer program instructions are stored in the computer storage medium；The computer program instructions are held by processor
Any one data management-control method in above-described embodiment is realized when row.
Fig. 6 shows the structural schematic diagram of data managing and control system provided by one embodiment of the present invention.As shown in fig. 6, should
Data managing and control system includes:
Database 610, for storing data；
User terminal 620, for issuing the command information for reading data to data database storing or from database；
Data manage module 630, communicate to connect respectively with user terminal and database, for what is issued according to user terminal
Command information executes above-mentioned data management-control method.
In embodiments of the present invention, database 610 can be the database of big data platform, include at least HDFS data
Library, Hive database and HBase database.
In embodiments of the present invention, the network terminal (Web Termianl), Web are installed on user terminal 620
Termianl is the webpage client that can simulate operation management operation interface.Network set may be implemented in Web Termianl
The transmission of word (WebSocket) connection request is connect, the number that keyboard input, window change events and WebSocket are returned is monitored
User is output to according to stream, and by standard output (stdout) data flow and standard error output (stderr) data flow of distal end
Terminal 620 is simultaneously shown as standard output and standard error.
Therefore, the embodiment of the present invention is due to having used webpage client, it can avoid operation maintenance personnel to client according to
Rely, and can be realized the permission control to operation maintenance personnel, makes operation maintenance personnel in the case where not changing operating habit, it can also be right
The operational order that operation maintenance personnel executes is recorded and is controlled.
In embodiments of the present invention, data control module 630 may include file isolation module 631, data outputting module
632 and data memory module 633.
Wherein, data outputting module 632 can be made of, user terminal Docker finger daemon (Docker Daemon)
Web Termianl can be with far call Docker Daemon api interface.Therefore, the embodiment of the present invention can eliminate line
Difference between environment upper, under line, guarantees the consistency and standardization of the environment of modules.At the same time it can also accurately
CPU, the resources such as memory are distributed for modules, ensure that will not influence each other between modules.
In embodiments of the present invention, data memory module 633 may include sensitive data recognition unit 6301, sensitive data
Storage unit 6302 and recognition rule storage unit 6303.
Specifically, sensitive data recognition unit 6301 is used for according to the sensitive number stored in recognition rule storage unit 6303
The corresponding sensitive data information of data to be stored is determined according to recognition rule, and data to be stored are stored to big data platform
Database 610, sensitive data information is stored to sensitive data storage unit 6302.It is stored in data outputting module 632
Sensitive data reads rule, after data outputting module 632 obtains data and its sensitive data information to be read, Ke Yigen
Rule, which is read, according to sensitive data generates output data.Output data includes the output data of file stream and the output number of data flow
According to, wherein the output data of file stream is buffered in file isolation module 631, and the output data of data flow is directly transferred to Web
Termianl is shown.
The file isolation module 631 of the embodiment of the present invention can by the generation of filestream data with downloading mutually separate, will be literary
The use of part flow data is merged with downloading, and filestream data is allow to go directly operation maintenance personnel, realize " who using whose downloading,
Who download who be responsible for " filestream data security management and control, safety responsibility is implemented into specific operation maintenance personnel, to substantially mention
Rise the safety of core data.
In embodiments of the present invention, operation maintenance personnel can input (stdin) data flow big data platform by standard and deposit
It stores up data and issues write-in, read or modify request.When to 610 storing data of database, sensitive data recognition unit
6301 determine sensitive data information by parsing stdin data flow.
Therefore, in embodiments of the present invention, operation maintenance personnel can be used by using the Web on user terminal 620
Termianl far call data control module 630 to carry out operation management operation to the database 610 of big data platform, is transporting
While dimension personnel do not change operating habit, it can be remembered by the operational order that Web Termianl executes operation maintenance personnel
Record and control.In addition, when operation maintenance personnel reads data, it can also be as needed directly to the sensitive number in data to be read
According to taking safeguard measure.
In embodiments of the present invention, the sensitive data stored in data outputting module 632, which reads rule, can also define use
The restrictive condition of family access sensitive data.Wherein, primary access can have multiple restrictive conditions, may include to sensitive data
Grade, the type of sensitive data, sensitive data Data Position, whether disclose sensitive data, whether send warning information etc..On
The multiple restrictive conditions stated can define corresponding sensitive data by combination and read rule: allow access sensitive data
Or denied access sensitive data, allow sensitive data to be encrypted or Fuzzy Processing when access sensitive data, allows to visit
Ask warning information etc. whether is sent when sensitive data.
In the following, being described in detail according to the reading data process of Fig. 7-9 pairs of different types of databases.
Fig. 7 is the process signal that the data in HDFS database are read using the data managing and control system of the embodiment of the present invention
Figure.
HDFS database can provide storage and read functions to HADOOP data, and may include HADOOP-
NameNode node and HADOOP-DataNode node.When database is HDFS database, management file system files
The agency (DataNode-Agent) of the node of the agency (NameNode-Agent) and storing data of the node of metadata information
It may be constructed data outputting module 632.
As shown in fig. 7, the detailed process that data managing and control system reads the data in HDFS database includes:
Step S701, the NameNode of file etc. is requested to send out for the first time to NameNode-Agent by user terminal initiation
It include Sasl message in the NameNode request sent.
Step S702, after NameNode-Agent receives NameNode request, judge whether it has Sasl message, if
Have, parses Sasl message.Wherein, NameNode-Agent can carry out Sasl several times with user terminal during parsing Sasl message
The interaction of message.
Step S703, NameNode-Agent carries out kerberos certification after Sasl message is parsed.
Step S704, when kerberos authenticate by after NameNode-Agent will create Sasl client, and by
Sasl client sends Sasl message to HADOOP-NameNode.
Step S705, HADOOP-NameNode parses Sasl message after receiving Sasl message.
Step S706, after HADOOP-NameNode is parsed Sasl message, kerberos certification is carried out.
Step S707, HADOOP-NameNode return authentication result is to NameNode-Agent.
Step S708, after NameNode-Agent receives authentication result, judge whether to authenticate successfully, certification achievement is then called
The access control interface of BDS, checking whether allows to access.
Step S709, NameNode-Agent is by access control and after allowing access, and return authentication result is to user's end
End.
Step S710, user terminal sends RPC and requests to NameNode-Agent after authenticating successfully.
Step S711, NameNode-Agent carries out RPC request analysis after receiving RPC request.
Step S712, NameNode-Agent authenticates the operation of parsing, checks whether there is operation file or catalogue
Permission etc..
Step S713, NameNode-Agent re-assemblies RPC request, and the RPC request after assembling is sent to HADOOP-
NameNode。
Step S714, HADOOP-NameNode carries out dissection process after receiving RPC request, returns to RPC response message.
Step S715, if necessary to operation file, such as the read-write of file, the RPC that HADOOP-NameNode is returned is responded
In include data block information, the address and end of the HADOOP-DataNode in NameNode-Agent replacement data block message
Mouth is the address and port of DataNode-Agent.
Step S716, NameNode-Agent returns to RPC response message to user terminal.
Step S717, user terminal receives direct request DataNode-Agent after RPC response, and the read-write for sending data is asked
It asks.
It step S718,, basis if opening kerberos after DataNode-Agent receives the request of user terminal
Encryption key obtains encrypting traffic.
Step S719, DataNode-Agent is written and read request protocol parsing, tells read-write type.
Step S720, read-write requests are transmitted to HADOOP-DataNode by DataNode-Agent.
Step S721, HADOOP-DataNode carries out protocol analysis and returns response data to after being written and read
DataNode-Agent。
Step S722, DataNode-Agent is according to judging whether to need to carry out Fuzzy Processing or encryption to sensitive data
Processing, and handled according to judging result.
Step S723, treated data by file isolation module are returned to user terminal by DataNode-Agent.
Fig. 8 is the process signal that the data in Hive database are read using the data managing and control system of the embodiment of the present invention
Figure.
Hive database includes HiveSever2 node.When database is Hive database, HiveSever2 node
BDS Hive agency (BDS Hive-Agent) may be constructed data outputting module 632.
As shown in figure 8, the detailed process that data managing and control system reads the data in Hive database includes:
Step S801, user terminal sends solicited message to BDS Hive-Agent.
Step S802, after BDS Hive-Agent receives solicited message, big data platform system configuration is read, judgement is
No unlatching kerberos certification, thens follow the steps S803 if opening, thens follow the steps S804 if being not turned on.
Step S803, BDS Hive-Agent carries out kerberos authentication, if S804 is thened follow the steps by certification, if not
S806 is thened follow the steps by certification.
Step S804, BDS Hive-Agent, which accesses, controls certification, if S805 is thened follow the steps by certification, if not
S806 is thened follow the steps by certification.
Step S805, BDS Hive-Agent carries out SQL authentication, if S807 is thened follow the steps by authentication, if not passing through
Authentication thens follow the steps S806.
Step S806, BDS Hive-Agent returns to the response data of prompt insufficient permission to user terminal, then executes
Step S816.
Step S807, BDS Hive-Agent determines read whether enable protecting sensitive data function, executes if enabling
Step S808 thens follow the steps S809 if not enabling.
Step S808, BDS Hive-Agent carries out sensitive data inspection, by inspection result write parameters, and regenerates
Solicited message.
Step S809, solicited message is sent to HiveSever2 by BDS Hive-Agent.
Step S810, HiveSever2 handles solicited message.
Step S811, HiveSever2 is to BDS Hive-Agent returning response data.
Step S812, BDS Hive-Agent parses the HiveSever2 response data returned, by response data
TBase is written.
Step S813, BDS Hive-Agent determines read whether enable protecting sensitive data function again, if enabling
Step S814 is executed, thens follow the steps S815 if not enabling.
Step S814, BDS Hive-Agent carries out Fuzzy Processing or encryption to sensitive data, regenerates sound
Answer data.
Step S815, response data is back to user terminal.
Step S816, user terminal receives response data.
Fig. 9 is the process signal that the data in HBase database are read using the data managing and control system of the embodiment of the present invention
Figure.
HBase database may include HBase node and Zookeeper node.When database is HBase database,
The agency (HBase-Agent) of HBase node and the agency (Zookeeper-Agent) of Zookeeper node may be constructed number
According to output module 632.
As shown in figure 9, the detailed process that data managing and control system reads the data in HBase database includes:
Step S901, Zookeeper-Agent is connected by user terminal first, requests linking objective HBase-Agent.
Step S902, after Zookeeper-Agent receives request, request message is parsed.
Step S903, Zookeeper-Agent forwards request message to Zookeeper.
Step S904, Zookeeper is to Zookeeper-Agent feedback response information.
Step S905, Zookeeper-Agent parses Zookeeper message after receiving response message, and Zookeeper is disappeared
Address information in breath is substituted for the address information of HBase-Agent.
Step S906, modified response message is fed back to user terminal by Zookeeper-Agent.
Step S907, user terminal receives the connection established after response message with HBase-Agent, send Sasl message to
HBase-Agent。
Step S908, HBase-Agent carries out it to parse and repeatedly be handed over user terminal after receiving Sasl message
Mutually.
Step S909, kerberos certification is carried out after the parsing of HBase-Agent completion Sasl message.
Step S910, HBase-Agent return authentication result is to user terminal.
Step S911, user terminal sends RPC and requests to HBase-Agent.
Step S912, HBase-Agent parses RPC request after receiving request.
Step S913, HBase-Agent is parsed to access to control inspection and operate to the operation of request after RPC is requested and be reflected
Power, the request to access control or authentication is not met return to refusal request message to user terminal.
Step S914, HBase-Agent re-assemblies RPC request.
Step S915, HBase-Agent pseudo subscriber terminal sends Sasl message and carries out kerberos certification to HBase.
Step S916, HBase is to HBase-Agent return authentication result.
Step S917, HBase-Agent sends the RPC re-assemblied and requests to HBase.
Step S918, HBase returns response data to HBase-Agent according to the RPC request received.
Step S919, after HBase-Agent receives the response data of HBase, if opening protecting sensitive data function,
Fuzzy Processing or encryption are carried out to sensitive data, and data are reassembled into response data by treated.
Step S920, the response data re-assemblied is returned to user terminal by HBase-Agent.
The data managing and control system of the embodiment of the present invention is used for towards the data file that need to frequently obtain big data platform or needs
It is related to the operation maintenance personnel of sensitive data access, the restoring files and security protection of an exclusive individual private possession can be provided for it
Platform.
The data managing and control system of the embodiment of the present invention forces operation maintenance personnel that personal account access big data can only be used flat
Platform solves the problems, such as that more people share account and can not call to account, and can pass through of parsing, sensitive data to operational order
Match and the access of national treasury mode, realizes the real-time control of operational order.
The data managing and control system of the embodiment of the present invention has big data control platform by setting file isolation module
The center of storage and the circulation of exclusive filestream data meets the control requirement to the subsequent circulation of filestream data.
The performance of the data managing and control system of the embodiment of the present invention is more excellent, system load is lower, can be more fully sharp
Use system resource.
It should be clear that the invention is not limited to specific configuration described above and shown in figure and processing.
For brevity, it is omitted here the detailed description to known method.In the above-described embodiments, several tools have been described and illustrated
The step of body, is as example.But method process of the invention is not limited to described and illustrated specific steps, this field
Technical staff can be variously modified, modification and addition after understanding spirit of the invention, or suitable between changing the step
Sequence.
Functional block shown in structures described above block diagram can be implemented as hardware, software, firmware or their group
It closes.When realizing in hardware, it may, for example, be electronic circuit, specific integrated circuit (ASIC), firmware appropriate, insert
Part, function card etc..When being realized with software mode, element of the invention is used to execute program or the generation of required task
Code section.Perhaps code segment can store in machine readable media program or the data-signal by carrying in carrier wave is passing
Defeated medium or communication links are sent." machine readable media " may include any medium for capableing of storage or transmission information.
The example of machine readable media includes electronic circuit, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), soft
Disk, CD-ROM, CD, hard disk, fiber medium, radio frequency (RF) link, etc..Code segment can be via such as internet, inline
The computer network of net etc. is downloaded.
It should also be noted that, the exemplary embodiment referred in the present invention, is retouched based on a series of step or device
State certain methods or system.But the present invention is not limited to the sequence of above-mentioned steps, that is to say, that can be according in embodiment
The sequence referred to executes step, may also be distinct from that the sequence in embodiment or several steps are performed simultaneously.
The above description is merely a specific embodiment, it is apparent to those skilled in the art that,
For convenience of description and succinctly, the system, module of foregoing description and the specific work process of unit can refer to preceding method
Corresponding process in embodiment, details are not described herein.It should be understood that scope of protection of the present invention is not limited thereto, it is any to be familiar with
Those skilled in the art in the technical scope disclosed by the present invention, can readily occur in various equivalent modifications or substitutions,
These modifications or substitutions should be covered by the protection scope of the present invention.
Claims (13)
1. a kind of data management-control method characterized by comprising
In storing data, determine the corresponding sensitive data information of data to be stored, and store the data to be stored and
Its corresponding sensitive data information；
When reading data, it is corresponding quick to obtain sensitive data reading rule, data to be read and the data to be read
Feel data information, and rule, the data to be read and its corresponding sensitive data information read according to the sensitive data,
Generate output data.
2. data management-control method according to claim 1, which is characterized in that determine the corresponding sensitive number of data to be stored
It is believed that before breath, further includes:
If the data to be stored are greater than preset data amount, distributed computing processing is carried out to the data to be stored.
3. data management-control method according to claim 1, which is characterized in that determine the corresponding sensitive number of data to be stored
It is believed that breath includes:
If the data to be stored are structural data, the data to be stored are determined using structured data analysis component
Corresponding sensitive data information；
If the data to be stored are unstructured data, determined using unstructured data analytic unit described to be stored
The corresponding sensitive data information of data.
4. data management-control method according to claim 1, which is characterized in that the corresponding sensitive number of the data to be stored
It is believed that breath includes the sensitive data for including, the sensitive data in the data to be stored in the data to be stored
The grade of Data Position and the sensitive data.
5. data management-control method according to claim 1, which is characterized in that the storage data and its correspondence to be stored
Sensitive data information include:
The data to be stored are stored to database；
The corresponding sensitive data information of the data to be stored is stored to signature library.
6. data management-control method according to claim 1, which is characterized in that obtain the method packet that sensitive data reads rule
It includes:
Obtain the user identifier for reading data；
According to the user identifier, inquires the sensitive data corresponding with the user identifier and read rule.
7. data management-control method according to claim 1, which is characterized in that it includes to quick that the sensitive data, which reads rule,
Sense data carry out Fuzzy Processing and sensitive data are encrypted.
8. data management-control method according to claim 7, which is characterized in that if the sensitive data reads rule as to quick
Sense data are encrypted, when reading data, further includes:
Obtain permission modification instruction；
It is modified and is instructed according to the permission, decrypt the sensitive data being encrypted in the output data, and regenerate output number
According to.
9. a kind of data control device, which is characterized in that described device includes:
Data storage cell is configured in storing data, determines the corresponding sensitive data information of data to be stored, and deposit
The storage data and its corresponding sensitive data information to be stored；
Data outputting unit is configured to when reading data, is obtained sensitive data and is read rule, data to be read and described
The corresponding sensitive data information of data to be read, and rule, data to be read and its right are read according to the sensitive data
The sensitive data information answered generates output data.
10. a kind of data management and control devices, which is characterized in that the equipment includes: processor and is stored with computer program and refers to
The memory of order；
The processor realizes the data control as described in claim 1-8 any one when executing the computer program instructions
Method.
11. a kind of computer storage medium, which is characterized in that be stored with computer program in the computer storage medium and refer to
It enables, the data control side as described in claim 1-8 any one is realized when the computer program instructions are executed by processor
Method.
12. a kind of data managing and control system characterized by comprising
Database, for storing data；
User terminal, for issuing the command information for reading data to the data database storing or from the database；
Data manage module, connect respectively with the user terminal and the database communication, for according to the user terminal
The described instruction information of sending executes the data management-control method as described in claim 1-8 any one.
13. data managing and control system according to claim 12, which is characterized in that the database includes at least HDFS data
Library, Hive database and HBase database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811626565.5A CN109726572A (en) | 2018-12-28 | 2018-12-28 | Data management-control method, device, equipment, computer storage medium and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811626565.5A CN109726572A (en) | 2018-12-28 | 2018-12-28 | Data management-control method, device, equipment, computer storage medium and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109726572A true CN109726572A (en) | 2019-05-07 |
Family
ID=66296680
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811626565.5A Pending CN109726572A (en) | 2018-12-28 | 2018-12-28 | Data management-control method, device, equipment, computer storage medium and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109726572A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112232050A (en) * | 2020-10-13 | 2021-01-15 | 中国平安人寿保险股份有限公司 | Method, equipment, terminal and readable medium for generating congratulatory newspaper |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110126281A1 (en) * | 2009-11-20 | 2011-05-26 | Nir Ben-Zvi | Controlling Resource Access Based on Resource Properties |
| CN102143158A (en) * | 2011-01-13 | 2011-08-03 | 北京邮电大学 | Data anti-leakage method based on trusted platform module (TPM) |
| CN104796290A (en) * | 2015-04-24 | 2015-07-22 | 广东电网有限责任公司信息中心 | Data security control method and data security control platform |
| CN106649587A (en) * | 2016-11-17 | 2017-05-10 | 国家电网公司 | High-security desensitization method based on big data information system |
| CN108289095A (en) * | 2018-01-02 | 2018-07-17 | 诚壹泰合（北京）科技有限公司 | A kind of sensitive data storage method, apparatus and system |
-
2018
- 2018-12-28 CN CN201811626565.5A patent/CN109726572A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110126281A1 (en) * | 2009-11-20 | 2011-05-26 | Nir Ben-Zvi | Controlling Resource Access Based on Resource Properties |
| CN102143158A (en) * | 2011-01-13 | 2011-08-03 | 北京邮电大学 | Data anti-leakage method based on trusted platform module (TPM) |
| CN104796290A (en) * | 2015-04-24 | 2015-07-22 | 广东电网有限责任公司信息中心 | Data security control method and data security control platform |
| CN106649587A (en) * | 2016-11-17 | 2017-05-10 | 国家电网公司 | High-security desensitization method based on big data information system |
| CN108289095A (en) * | 2018-01-02 | 2018-07-17 | 诚壹泰合（北京）科技有限公司 | A kind of sensitive data storage method, apparatus and system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112232050A (en) * | 2020-10-13 | 2021-01-15 | 中国平安人寿保险股份有限公司 | Method, equipment, terminal and readable medium for generating congratulatory newspaper |
| CN112232050B (en) * | 2020-10-13 | 2024-04-09 | 中国平安人寿保险股份有限公司 | Method, device, terminal and readable medium for generating greeting report |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109190410B (en) | Log behavior auditing method based on block chain in cloud storage environment | |
| US9679005B2 (en) | Client computer for querying a database stored on a server via a network | |
| CN108509805A (en) | Data encrypting and deciphering and desensitization runtime engine and its working method | |
| CN112134956A (en) | Distributed Internet of things instruction management method and system based on block chain | |
| US11038835B2 (en) | Systems and methods for managing domain name information | |
| JP2022006164A (en) | Method, device, electronic device, computer-readable storage media and computer program for processing user request | |
| CN109376021A (en) | The response method and server that interface calls | |
| CN116436682A (en) | Data processing method, device and system | |
| CN103971059A (en) | Cookie local storage and usage method | |
| CN111212031A (en) | Control method and device for interface access frequency, electronic equipment and storage medium | |
| CN108370312A (en) | Encryption device, retrieval device, encipheror, search program, encryption method and search method | |
| CN112364022B (en) | Information deduction management method, device, computer equipment and readable storage medium | |
| CN109726572A (en) | Data management-control method, device, equipment, computer storage medium and system | |
| US20140283080A1 (en) | Identifying stored vulnerabilities in a web service | |
| CN114500347B (en) | Method and system for formalized verification of security interconnection protocol | |
| CN110232570A (en) | A kind of information monitoring method and device | |
| CN109905408A (en) | Network safety protection method, system, readable storage medium storing program for executing and terminal device | |
| CN105825247B (en) | A kind of card reader and data transmission method | |
| CN109565502A (en) | It is used for transmission storage equipment, data transmission set and the method for data | |
| CN114282186A (en) | Method and system for safely exporting large file data | |
| CN113542238A (en) | Risk judgment method and system based on zero trust | |
| CN108108310A (en) | A kind of data processing method, device and server | |
| KR20210027038A (en) | Proxy apparatus and method for processing information executed on proxy apparatus | |
| US20220309599A1 (en) | System and method for authorizing transfer requests of physical locations | |
| Horodelski et al. | A representative fragment method of analyzing complex systems of smart contracts |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190507 |
|
| RJ01 | Rejection of invention patent application after publication |